Performance & Load Testing

Resources Center

Performance test results: What to report?

by Swaraj Gupta

In the days of AGILE, where quick application development, test, fix and releases are happening sharing a static report for a performance test is not the norm. A more and modern approach to what to be reported has taken shape. The project managers and business sponsors need more than a report to be presented. They need a thorough analysis and conclusions based on those. They need information on what has gone wrong and which component to be looked into to resolve the issue noticed. They need data to back the test findings and they need total value out of the tests. At the same time there are other stakeholders involved in the test. These are the tech team members who play a very critical role in further fine tuning the app. To satiate them, one needs to present them with not just the response time numbers and the transaction failure rates, but also trends of those over earlier releases, individual data points associated with those, failure points, data associated with failures, information on activities happening on the servers during the failures. In this article, I am trying to highlight what in performance test result report matters and how is that helpful for the business, the tech folks and the customers.

 

 

Transaction related items:

For each transaction following items should be included in the result report :

  • Average response time
  • Transaction failure rate
  • Minimum response time
  • Maximum response time
  • Standard deviation of response time
    >
  • 95% response time
  • Execution as a % of total transaction execution

Comparison over earlier tests (for earlier releases or same release):

  • Avg response time of this release Vs earlier releases
  • Transaction failure rate of this release Vs earlier releases
  • 95% response time of this release Vs earlier releases
  • Execution % of this release Vs earlier releases

Client side counters: Following are client-side graphs, if included, help in analysis –

  • Hits per sec over the test
  • Number of concurrent users over the test
  • Transactions per sec over the test
  • Error per sec over the test
  • Average throughput graphs
  • Total hits graphs

Server related counters:

  • Processor:
    • % processor time / total,
    • % disk time
    • Processor queue lenght
    • Disk queue length
    • Interruptions/sec
    • Thread count
  • Memory:
    • Available bytes,
    • pages / sec,
    • page faults / sec
  • Network:
    • bytes / sec
    • not found errors / sec
    • anonymous users / sec
  • Active server pages:
    • Requests / sec,
    • Memory allocated,
    • requests queued,
    • requests errored – rejected

Error details:

  • Error snapshot
  • Form data for the transaction that failed
  • Time on server when transaction failed
  • Any loss of data in db
  • Performance counters on app & db servers at the time of the failure
  • Steps to reproduce the transaction that failed

Things that matter to business:

  • If the issues observed in earlier releases were resolved?
  • If issue wasn’t resolved, was there any performance improvement at least that was observed?
  • How the servers were utilized during the peak load? Can the capacity be reduced?
  • How long the issue fixation will take?
  • Are the SLAs realistic and achievable?
  • Is there any change in schedule / SLAs that need to be taken into account?

This post is also available in: French


About Swaraj Gupta

Swaraj is a performance, automation and functional test expert who has worked on variety of desktop and mobile applications. The major areas that he focuses on are - functionality, usability, performance and consistency of application behavior. He manages the entire performance testing cycle of the projects that he is responsible for and works on multiple such engagements simultaneously. He has worked in variety of different business domains that include - Hi tech consulting, Financial services, management consulting, auditing services, e commerce, e learning, etcT

Learn more about QTest
AgileLoad.com
Agile Software Security


More Blog