Slow loading or not responding applications can be a disaster for both, enterprises and consumers. The latter will most likely never return if they experience unsatisfied response times, and the former will lose money due to less users buying their products online.
Successful organizations understood that reactivity is a bad advice when it comes to reliability of their business applications. Therefore, they integrated load testing into their development chain and eliminate bottlenecks on pre-production stages. This paper provides an overview on the general load-testing topic, followed by best practices and fundamental steps in a load-testing project.
Terms and Definition
Load testing is a method of testing that verifies whether the system can handle a particular load within agreed boundaries. Software testers often use the term load and performance testing in a similar fashion. However, their objectives are different. The focus of load testing is more on simulation of the user or data volumes. End-to-end response times are not relevant.
Performance testing includes the simulation of realistic load patterns, verification of boundaries and also, also end-to-end response time measurement. It covers load-testing aspects but also pays high attention to realistic user and transaction load, repeatability and perceived response times.
Stress testing involves load test elements, but the focus is more on the identification of breaking points by testing the application beyond the limits of normal operation.
In recent years software development methods moved in the agile direction. Developers and test engineers start early in the lifecycle with automated component level performance checks.
The system under test is often not fully integrated. Mocks are used for simulation of missing interfaces, which simply return the expected results. The basic idea behind is the creation of a performance baseline, comparison of service response times with previous builds and identification of bottlenecks early in the development lifecycle.
Testing on component level is essential but it does not give confidence that the new system can handle the real production load pattern within agreed boundaries. End-to-End Performance tests are the ideal setting when it comes to verification of non-functional requirements. In this type of tests, engineers implement test scripts, which simulate relevant use cases from an end user perspective. The injected user and transaction volume reflects actual or future growth patterns and provides confidence that the system under test can handle the load within agreed boundaries.
It is not enough that applications can deal with regular user and transaction volumes. Customers expect that a business application remains stable and within acceptable speed thresholds even in high peak load situations. There is nothing worse than an application, which can't handle the unexpected user volume because those users will leave your site and you will lose financial revenue. Therefore, engineers set up tests to verify how the system behaves under peak load situations.
They specify mainly the max number of users and the time over which the ramp up and the steady state load should be in an application. The goal is to identify the breaking points of the application under test.
There are those nasty implementation failures, which hold objects back from being freed up. Sometimes those issues are hard to tackle and occur just if the load is injected over a long period. A simulation of a production load testing pattern over 8 or 12 hours will help to verify the reliability of the application under test. This test is essential and should be part of every project.
In the worst case, several users hit a certain service exactly at the same time. Customers do not care about this and expect that the system can handle such situations accurately. Parallel requests of a pore implemented service can lead to system crashes or wrong responses. Experienced test engineers execute at least one concurrency test for all new services to test how they behave when being accessed in parallel.
Testing what an application needs to do is already a fundamental step in software development processes. Verification of non-functional aspects such as response time is still an afterthought. This section provides key reasons why organizations should integrate load and performance test in their delivery chain.
Performance and load testing reduces the risk of severe outages or system failures dramatically. Even if hundreds of testers conducted thousands of manual tests on a new application, this is no guarantee that the system can handle the expected load pattern. The most application works fine under normal load or data volume conditions. Depending on the design and implementation decisions the end-to-end response times often rise when the new system needs to handle a certain number of concurrent user requests.
Up-to-date users expect high-speed applications, and with the growing service portfolio, the power goes more and more to the users. The response time of a business application decides whether you hold your customers or lose commercial revenue. Optimized high-speed applications will make customers happy and satisfied users will often return.
Companies invest thousand of dollars in search engine optimization and neglect an even more important aspect, the actual speed of their application. Maybe this is still not well known, but search engines rate applications with low speed or pure application design down. Obviously, it is better to improve the response time prior making investments in search engine optimizations.
Hardware sizing is often guesswork with a bad end. Oversized infrastructure is waste of money and a shortage in system resources can have a bad impact on user experience. Performance and load tests eliminate this try and error exercises. Simply simulate the expected load on the new system and verify if system resource utilization is within the recommended boundaries is enough for an appropriate investment in IT infrastructure.
Starting with load testing as part of your performance strategy without any guidance can be tricky. The set of hard-won experience and forward looking ideas below provides a way to perform that is more likely to succeed.
Meaningful non-functional requirements are must criterion. It is not enough to guess the number of users and just simulate a stress test on the new system. Clarify the performance requirements early in the analysis phase, consider those in design decisions and conduct load and performance tests to verify the same at pre-production stages.
Test early and test often is essential when it comes to load and performance testing. The majority of performance breakdowns root on failures in application design. Therefore, it is wise to consider performance validation early in the development lifecycle.
Sizing of the load and performance test environment is essential for reliable test results. Response times of applications vary with the load. Therefore, never try to simulate 50 percent of the load on a 50 percent production like environment. Sure, it is better than no test at all, but the chance is very high that the speed of this application is worse than expected. Therefore, simulate the full load on a smaller environment is better than the other way around.
Don't underestimate the relevance of an appropriate load injection platform. Many criteria's such as flexibility, license costs, hardware requirements, and technology support should be considered when it comes to choosing the load and performance test solution. Cloud-based load injection platforms are more and more popular because they allow a quick start with the implementation and execution of relevant performance tests.
Performance testing is more a journey than a destination because small changes in configuration, user or data volumes can have an enormous impact on end-to-end response times. Therefore, execute load tests regularly to verify the impact of changes in software, hardware or data volumes.
Make experienced performance engineering specialistâ€™s part of the initial setup because they are familiar with typical pitfalls and can speed up the way to quick results from your investment in load and performance tests.
Several steps need to be considered before implementing and effective load testing performance strategy. This section provides insights to the most important aspects involved.
Business analysts and requirement experts specify the performance expectations for the new system. They outline regular and peak load patterns including transaction rates, user volumes and response time expectations.
Engineers review nonfunctional requirements and develop an appropriate performance test approach. Responsibilities, test data, environments, load patterns, relevant use cases, scheduled test runs, thresholds and success criteria will be specified. The service owner approves this test approach.
Test teams agree about activities on the used environment. Load and performance tests will be executed in exclusive slots to avoid the negative impact of background activity.
Engineers implement the necessary test scripts. Some load testing solutions provide a script recorder, which allows convenient test script creation without any programming knowledge. The tester just executes the use case manually while the recorder captures the interactions and creates the test script. With open source tools, the script implementation often requires programming skills.
Configure and Execute Test
The application owner enables system resource monitoring while test staff compiles their scenarios. They setup component level tests to run on a daily basis and end-to-end performance test scenarios to run on the fully integrated production like environment.
After the test execution, test staff collects all results including response time and system resource metrics. In best case, no performance criteria have been violated. However, in most load and performance tests bottlenecks were identified, defects need to be raised, and root-cause analysis starts.
If the performance is not acceptable corrective actions must be conducted. Engineers lay out their test approach, discuss actual results with developers, infrastructure, database, network and IT architect teams. Those virtual teams decide on tuning measures, implements corrective actions and repeat the test until performance is acceptable.
All things considered, load testing is an excellent investment in performance that gives confidence that the new or changed system performs within the agreed boundaries.