What is capacity Testing – a definition
It determines how many users (or) transactions (or) hits a system can take up per unit time while meeting the performance SLAs of the system. It is basically done from a planning objective. Business teams constantly look forward for growth and scalability and they want their systems such that those could meet the increasing user demands. Hence while increasing business and adding users, the team should be aware of the system capacity so that the user experience is not impacted while meeting the growth objective.
Benefits of capacity test
- Helps in demand and capacity planning
- Beneficial in cases where prioritization is required on resource expenses
- Can serve as a benchmark while comparison multiple solution providers providing similar solutions
- Ensures that the performance and security vulnerabilities are not exposed while testing the app under high load
Difference between capacity and stress tests
- In case of stress test the system is overstressed with a target of breaking the system. However in case of capacity test, the system is loaded only till the point that SLAs are met.
- In case of stress test, tests are performed even above user levels that do not meet the performance SLA. In case of capacity test, as soon as SLA point is reached, testing is stopped.
- Capacity test value (number of simultaneous users) is less than stress test corresponding value.
Capacity testing using QTest
This is how you can use QTest to effectively plan the capacity test:
- In the planning phase, understand the application, the test requirements and contemplate on the Performance test acceptance criteria
- Create prototypes with different performance and understand with the teams (for each prototype) weather the performance was acceptable / manageable and needs improvement. Remember the resources are constrained and hence you should try to meet the business and end user needs in minimum server availability
- Discuss the acceptance criteria and the feedback obtained from end users with the development and the business teams and finalize the SLAs
- Once the SLAs are finalized, complete the other planning and infrastructure setup activities
- Based on which test to perform (QTest on Cloud / QTest on your in-house infrastructure), get the appropriate license
- Then using QTest wizard set the user count (or transaction count per unit time) to minimum business expectation and execute the test.
- Run this test for 30 mins and then proceed on result analysis and comparison of the results with the SLAs as identified earlier.
o If all SLAs are met, then increase the user count by 10-15 % more users and run another test and analyze results, thus proceed iteratively.
o If all SLAs are not met, then try to fine tune the application, work with performance engineers and achieve end results within the SLAs as identified. (Note – A lot of time, teams also consider it appropriate to revise the SLAs at this step since it is at this point that they get more clarity on how much budget is required to improve a specific parameter of performance).
It is essential to fine tune the application at this point since the minimum user load also doesn’t meet the business SLAs.
- Once repeated rounds of tests are performed and the user load is identified at which the system no longer meets the performance SLAs, that user load and results at all different loads (as performed) are then documented in result report
- Perform the exercise again (at least once) to validate if similar results are obtained and if the system shows a consistent behavior
o If similar results are obtained and same load (as that in the earlier exercise) is determined as the capacity of the system, then call this load as the system capacity.
o If not, fine tune the application to obtain consistency in it.
- QTest tool is extremely helpful in such tests because result analysis becomes a cake walk with QTest analysis tool. Also result comparison across multiple tests is quite easy with QTest.
Things to include in Capacity test Result Report
The following counters should be included in the result report with focus on testing at different user loads (increasing user loads), till the application stops meeting the SLA. Results of the load test with load one step higher than the point where SLAs are not met, is essential to be included in the report. Also 3-4 rounds of tests are done to validate the Capacity value.
- Response time numbers : Min, Max & Avg
- Transaction failure rate
- Transaction execution percentage
- Error percentage
- Server utilization at different load levels and variance in the numbers
This post is also available in: French