Mobile revolution has very quickly brought the entire world under its purview and mobility has caught the world. Also every business has some or the other way of getting benefit out of this revolution. Every business (and now-a-days every revenue generating arm of a business) has recognized some or the other way of generating more revenues and building business.
Not just that, if someone is not following what the market is adopting so quickly, there is a great chance that he would lose the competitive advantage. This is primarily the reason why banking firms, real estate agencies, FMCG companies, durable goods companies all are trying to get into the mobile space. It is a growing market which business gurus say is still in its very early state.
While making investments into Mobile, while building your Mobile app, there might come a day when you would want to scale up your app / would want to improve the performance of your app. That would be the day when you would want to do performance testing of your app. While doing this testing if you follow following 5 principles you won’t have to worry a lot –
Your business / app may have multiple functions and features that the end users would be using. Testing all of these features is never a requirement (and shouldn’t be targeted) in performance testing. However it is essential to determine the key business flows that the end users are going through and the most frequent business transactions happening in the system. Taking that as the base, the performance testing scenarios and performance testing test cases (further the transaction names) should be determined. This helps in improving the performance of the entire application with only 20% (usually top 20% of business transactions are considered as the key business transactions as selected for performance test).
The load on each of these modules then should further be obtained from the production server logs and while designing the performance test the load on those different modules should also be divided such that a production like load distribution is also achieved. Subsequently a production like scenario must be created (in terms of load on other components of the infrastructure and the production like network) and performance testing is done.
As we know that the processing capability of mobile devices be it tabs or your touchscreen mobile smartphones is much lesser than that of your laptops or your workstations. Also there is a good chance that your laptop is less mobile than the smartphones and thus may have better network availability. Hence resource availability for mobile apps is much less than that of the desktop / web apps. However the performance expectation for the end user may not be different for different apps. They would still want their ‘mobile google’ to work as quick and efficiently as their ‘desktop-google’. Hence identifying the right set of factors that impact performance is really crucial. Smartphones, network bandwidth availability, OS are some of the key factors that impact the performance of the system directly. Also there are zillions of combinations possible if these three factors are to be considered in different scenarios. Hence finding those right combinations (or combination) is extremely crucial while testing the app for its performance.
Best approach is to understand your end users and notice their behavior and go for the most frequent behavior (most frequently used phones, network scenarios / OS). We haven’t forgotten how quickly Blackberry and Nokia lost its market share in last 2-3 years hence it is extremely necessary that you keep on updating your set of testing devices and combinations that you create as the market evolves and grows.
It is extremely essential to achieve a business load as good as that of the production load. Coz it is possible that an issue observed at a higher level volume may not be visible at a lower load. To reproduce the issues that might come up in production, it is essential to have production like load on the system.
Essentially the test team targets to load the system with production like load through backend mechanisms like sending messages / API requests on the servers using some tools and tries to see how the system behaves in such a scenario by validating the behavior by running automated scripts on real devices. This helps them capture realistic numbers on real devices in production like scenario and reproduce the issues (as could have been observed in production if the app wouldn’t have been tested)
User conditions differ and hence the required scenarios to reproduce those situations. Depending on the mobile set (device) and the operating system, the app may perform differently and produce different results. Hence while validating the system on different types of devices, it is also essential to validate them in different network conditions too. End users may use WiFi or 2G or 3G and testing the app under different conditions would definitely produce different results. As a part of the test team, it is your responsibility to reproduce those scenarios and obtain strange behavior and reproduce issues.
Remember you don’t just need to acquire customers; you also need to retain them.
Try to include some key counters in your result report while sharing the same with the team. Along with response time and failure rates, it is essential to mention the client side and the server side counters too. These would include –
Client side – Hits / sec, transactions / sec, errors/sec,
Server side – CPU and Mem utilization, etc.
DB side – Response time taken by key queries, Response time of queries that takes most time to respond, etc.
It is also essential to include the network related counters in the report since network plays a key role in mobile performance.
This post is also available in: French