“Home page is currently unavailable”; “Your transaction cannot be processed now. Please try again later”; these are some of the error messages usually end-users encounter during peak business hours. The most annoying message is “HTTP Error 503 – Service unavailable”. Enterprise business organizations, predominantly verticals like Banking, Retail, Insurance, Telecommunications, and Healthcare applications have a critical role in day-to-day activities in terms of transactions. Even small amount of timely loss impacts their business growth and the stock market value.
Major US bank introduced a new $5 debit rule which charges monthly fee from their customers. A day after the rule implemented, users were unable to access their bank accounts and they can see only temporarily unavailable error. Major US Retail chain’s website crashes again and the outage extended for 2.5 hours which impacts on their stock market price.
During above instances, enterprises won’t explain the root cause of the issues nor their technical problems. But it is a crystal clear that any organizations want to avoid the outage/downtime during the business hours.
Before deploying the code into production environment, it is important to validate the functional and non-functional aspects of the application. Performance testing is one of the important types of testing under non-functional testing group.
Performance testing is the super set term. According to IEEE definition, performance testing is conducted to evaluate the compliance of a system or component with specified performance requirements.
Load testing is the subset of performance testing. Load testing validates the applications’ capacity. E.g. how many users can do transactions concurrently, how many orders application can process per minute etc. the process of inject the demand/load on the application and measuring its response.
How much pounds of weight you can lift in gym? 200 pounds, 300 pounds, 1000 pounds? Every person has a breaking point, mentally, physically and intellectually. The same principal applies in load testing as well.
How much load can server handle at a specific point of time? 100 users, 200 users, 10K users? Every server has a breaking point.
The main objective of load testing is to study the behaviors of the application under test (AUT) under specific load circumstances.
Load testing helps engineers to understand the applications’ behavior and availability.
Now we will see about load testing process and how to study the behaviors of the application during load test. Following are the key phases of load testing any application.
Identifying critical scenarios is an inception and critical phase in load test process. Following are the various factors play a vital role in this phase: expertise in the application domain, business rules and requirements, technology etc.
Following the important metrics that needs to be collected and monitored:
Also there are different kinds of metrics which needs to be identified such as Client-side metrics, server-side metrics, business metrics, network metrics etc.
Following are the examples of critical scenarios for banking applications
|S. No||Business Scenario||Metrics|
|1||Login||Response time should be < 2 seconds for 500 Vusers|
|2||Fund Transfer||Response time should be 100 Funds Transfers should happen per minutes|
|3||Apply for Credit Card||Response time should be < 2 seconds for 300 VUsers200 Credit Card application should get processed in 2 minutes|
Workload model helps test engineers to design the test execution as per real time scenario. In production environment how the application is getting utilized. In order to inject production like pattern in the test, effective work load model design is required.
Below is the one of the workload model type which will be used for load testing. Graph will be plotted Users vs. Time. There are three phases: Ramp-up, Steady State and Ramp-Down.
When effective work load design is available, next step is to start generating the scripts. First step in generating scripts is to record your business transactions. Load testing tools supports recording feature which enables test engineers to record their actions.
Once the business transaction is recorded, immediate is to enhance the scripts. Following points should be considered in script enhancement phase:
Once the script is enhanced, it is mandate to perform unit testing to baseline the test script.
Script execution is a critical phase in load testing process. In this phase the defined load will be injected to the server and its behavior will be recorded and analyzed. Poor load injection will not help in analyzing the applications’ performance.
Before injecting load to the server, following points should be consider:
Test Execution Phases
The test execution consists of pre-initiation, execution and post testing activities.
The pre-initiation activities include
Following activities performed during test execution
The Post Test activities include
Identifying performance bottlenecks is a blend of art and mathematics. Basic knowledge about statistical terms and correlation of various charts is required for an effective analysis. Following are the key steps gets involve in report analysis:
Provide the following details in your final report: Average, Minimum, Maximum, Mean, Median, Mode, Standard Deviation, 90th / 95th percentile, Error percentage, Throughput, Utilization of CPU, Memory, Disk and Network and Histogram analysis (if required). Apart from above metrics, you can add additional performance counters to your results.
This is the final phase of load testing process. Results will be compared with industry standard benchmarks. If there is a conflict, root cause analysis should be performed with the help of developers/architects and the necessary recommendations should be provided.
Also it is important to document all the learning and issues encountered during load testing for future reference.
It is important to define and validate the load testing entry criteria before start testing. Following are the points which needs to considered:
All the project teams such as development team, environment team etc. should be aware of performance testing cycles to keep the servers in quiet mode during testing
All user processes that consume more memory and CPU need to be turned off from all load generator consoles
After successful testing gets completed, following points should be considered:
Load test data plays a significant role in load testing. The databases in the performance test environment should have data volumes similar to production. For applications which are in production already, a copy of the latest data should be taken from production to the performance testing.
For applications that are yet to be in production, effort should be spent for test data set up. Privileges and access permissions should be provided.
Risks and Mitigations should be in place to avoid chaos during load testing. Following are the points should be considered:
|Issues related to Performance testing tool||Performance testing execution to be delayed||Contact tool vendors support|
|Issues with test data||Performance testing execution to be delayed||Contact test data team|
|Unstable Application||Performance testing execution to be delayed||Critical flag should be raised with clients|
|Lack of availability of required resources||Performance testing execution to be delayed||Notify clients in advance|
|Major changes promoted post performance testing||Additional effort is required||Re-run the performance testing suite with modified scripts or accept the impact|
Load Testing helps to study the behaviors of application under the specified load. Performance bottlenecks which causes performance degrade can be identified during load testing. Following are the basic methodology for load testing: identifying critical scenarios, effective work load model, test script design with enhancements, load simulation, collecting the performance metrics and benchmarking & recommendations.
This post is also available in: French