Performance Testing

10060 Socket Error

noreply@blogger.com (Adam Brown) — Thu, 25 Feb 2010 11:17:00 +0000

To load and performance testers this will be a familiar message, however it often causes much confusion.
This message is seen under the following conditions:

The server has run out of socket connections
The injector has run out of socket connections
Socket connections are timing out
There is a network policy limiting the number of connections per machine

Useful tools:

The netstat command will list all active connections on the machine.

The regedit command (Registry Editor) and the following parameters (if you're not going to use Qutium Daemon to manage the chnages):

\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\Tcpip\Parameters\MaxUserPort
\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\Tcpip\Parameters\TcpTimedWaitDelay

Remedies:

Load injector: If the injector machine has not been optimised to be a load injector, use the Quotium Daemon to adjust the registry settings for:

MaxUserPort - this is 5000 as standard in Windows 7, you can increase this to 65534. This is true for any windows OS. The parameter controls the maximum port number that is used when an applicaiton requests any available port, by increasing this value you increase the number of sockets available to applications on the machine.
TcpTimedWaitDelay - this is set to 240 by deault in windows machines, this can be set between 30 - 300. Reduce the value to increase the turn around time for available sockets. The paramter controls the length of time a connection will stay in the TIME_WAIT state when being closed. While a connection is in the TIME_WAIT state,t he socket pair cannot be closed.

Other rememdies:

It may be that the server is overloaded - in this case you need to speak to the applciation architects

It may be that network policy is stopping you open a large number of connections from a single workstation machine that you are using as an injector. Try installing QTest on multiple machines and distributing the load across all of these machines.

Cloud Computing: What could this mean for testers?

noreply@blogger.com (Adam Brown) — Thu, 21 Jan 2010 17:36:00 +0000

A problem for test managers, especially those involved in performance testing is that there are not the platforms available on which to run the number of test systems that they would like to, such as a UAT system for each build, or a system for performance testing. Those involved in performance testing will often have to pitch a very good case in order to get a like for like staging environment for performance testing. Many test departments are denied this and performance testing is done out of hours or after UAT is complete in the stages before go live.

Sure, most IT departments now have some kind of virtualisation, however even to get a virtualised server often internal procurement is involved, as there are storage, resource utilisation and other capacity questions to be answered. In the cloud a server can be provisioned within typically less than 5 minutes and cost just cents to run. The cloud seems like a very good option as a test platform.

Companies such as Microsoft and Oracle have given organisations the option to use their premises, kit and infrastructure as testing platforms when very large performance tests have been a necessity, now with the cloud this is no longer essential as there is a more cost effective, straight forward option available.

Of course the cloud is a shared resource and is a different proposition to a purely hosted environment; there are issues such as performance due to the fact that cloud machines are essentially virtual machines running in data centres on shared physical machines (only with a nice web interface or API to facilitate management of these virtual machines). However these issues can be overcome. For example to performance test a cloud based system, it may be necessary to run several tests at different times of the day to ensure that an even set of results is achieved. We saw this when running the cloud based test shown in our video here: www.youtube.com/user/Quotium. The second time we ran the test we found that results were different to the first time we ran the test.

The cloud provides great resilience due to its distributed nature, vastly reduced costs and simplicity. Assuming that issues surrounding the cloud are not a problem for an organisation, testers within that organisation could in fact lead their own organisation into a cloud computing environment – could this coin a new term – Test Driven Infrastructure? If a cloud infrastructure is used as a test environment then it would follow suit that the business would then start to look at the cloud as an alternative to their hosted or in house solutions and the testers would have shown the way.

As a company we have found that the cloud answers many of our needs including the ability to have a scalable load platforms and international presences that before would have been difficult and expensive for us to reach.

If you'd like to try cloud computing for yourself just google 'amazon ec2', get your credit card ready (it'll only cost you a few cents or possibly dollars - just remember to terminate the instance when your finnished otherwise the dollars might add up) and start an EC2 instance. You then simply use a terminal services client to connect to it and use it as you would any other VM.

Poll: How much time & effort to deliver a Performance Test using an Open Source tool vs. a reputable Commercial tool?

noreply@blogger.com (Adam Brown) — Tue, 10 Nov 2009 15:25:00 +0000

It's a LinkedIn Poll so you have to be a member.
http://polls.linkedin.com/poll-results/65174/ojyrq
It'll be interesting to see the results!

Common HTTP Response Codes seen in Load Testing

noreply@blogger.com (Adam Brown) — Mon, 26 Oct 2009 09:58:00 +0000

A complete list of HTTP response codes can be found on here Wikipedia and at w3.org.
What's below is not a comprehensive list but some commonly seen HTTP response codes seen when load testing and where possible some reasons for seeing them.

400 Bad Request
Officially: The request could not be understood by the server due to malformed syntax. The client SHOULD NOT repeat the request without modifications.

When / Why: The request in the script may be badly formatted. The best thing to do is check if the request has any parameters that you have edited, of so check them by debugging.

HTTP 401 Unauthorised
Officially: The request requires user authentication. The response MUST include a WWW-Authenticate header field (section 14.47) containing a challenge applicable to the requested resource. The client MAY repeat the request with a suitable Authorization header field (section 14.8). If the request already included Authorization credentials, then the 401 response indicates that authorization has been refused for those credentials. If the 401 response contains the same challenge as the prior response, and the user agent has already attempted authentication at least once, then the user SHOULD be presented the entity that was given in the response, since that entity might include relevant diagnostic information. HTTP access authentication is explained in "HTTP Authentication: Basic and Digest Access Authentication"

When / Why: Typically when load testing many users are generated from the same machine. In a Microsoft example the browser can use the current Windows logon credentials as credentials for the web site using NTLM. Therefore if the login is different as does not have the same authorisation as the original login (from the recording phase) during the execution the server may deny access with 401.
In the script you should use:
SetAuthentication(UserNameVariable,PasswordVariable,DomainVariable);

HTTP 403 Forbidden
Officially: The server understood the request, but is refusing to fulfill it. Authorization will not help and the request SHOULD NOT be repeated. If the request method was not HEAD and the server wishes to make public why the request has not been fulfilled, it SHOULD describe the reason for the refusal in the entity. If the server does not wish to make this information available to the client, the status code 404 (Not Found) can be used instead.

When / Why: If you do not see the error when manually browsing but do see it when running a script check that the script is recent. If there has been aconfiguration change on the server you may see this message: For example if if at one time a server hosted the site and now no longer does so and can't or won't provide a redirection to the new location it may send a HTTP 403 back rather than a more meaningful message.
Check that authentication is correctly set up in the script - see HTTP 401.
Also check that the browser you are simulating is allowed as a security policy can ban certain types of traffic from a server.

HTTP 404 Not Found
When / Why: You may find that when interacted with manually the web site does not appear to throw any HTTP 404 messages.
When running a load test script you then may see 404's in the response codes. This can be because the object (probably a page component) that the tool requested does not exist on the server - it may be a .gif or a .js java script or similar that is referenced by the page yet does not exist on the server. In a browser this would simply appear as an empty image place holder in the case of an image for example, or in the case of a java script you may even see nothing if the java script is redundant. As the tool specifically requests the object (as the browser does) it will of course log the 404 message if the object cannot be found.

HTTP 407 Proxy Authentication Required
Officially: This code is similar to 401 (Unauthorized), but indicates that the client must first authenticate itself with the proxy. The proxy MUST return a Proxy-Authenticate header field (section 14.33) containing a challenge applicable to the proxy for the requested resource. The client MAY repeat the request with a suitable Proxy-Authorization header field (section 14.34). HTTP access authentication is explained in "HTTP Authentication: Basic and Digest Access Authentication"

When / Why: The script was probably captured with the browser already pointing to a proxy server set (see browser network settings). See the information on HTTP 401 for reasons why this might happen.
Typically it's best to avoid running a load test through a proxy server, especially if production load will not be routed through that proxy server. Ways to avoid the proxy server are to remove the part of the script that states use of the proxy server (often internal applications are available even while bypassing the proxy server), if that doesn't work - move the injection point to a location in the network where the proxy server can be bypassed (perhaps the same VLAN as the web server).

HTTP 500 Internal Server Error
Officially: The server encountered an unexpected condition which prevented it from fulfilling the request.

When / Why: You'll often see this after an HTTP POST statement and it usually means that the post statement has not been formed correctly.
There can be a number of reasons for this including the request being badly formed by the tool - or at least not formed as expected by the server. More typically it's because the POSTed form values are incorrect due to incorrect correlation / parameterisation of form variables.
For example: In a .Net application a very large __VIEWSTATE value is passed between the browser and server with each POST, this is a way to maintain state and puts the onus on state ownership on the browser rather than the server. This can have performance issues which I won't go into here. If this value is not parameterised correctly in the script (there can be more than one __VIEWSTATE) then the server can be confused (sent erroneous requests) and respond with a 500 Internal Server Error.
A 500 error usually originates from the application server part of the infrastructure.
It's not just .Net parameters that can cause this. Items such as badly formed dates, incorrectly formatted fields and badly formatted strings (consider replaced spaces with + characters) and so on can all cause HTTP 500 errors.

HTTP 503 Service Unavailable
Officially: The server is currently unable to handle the request due to a temporary overloading or maintenance of the server. The implication is that this is a temporary condition which will be alleviated after some delay. If known, the length of the delay MAY be indicated in a Retry-After header. If no Retry-After is given, the client SHOULD handle the response as it would for a 500 response.

When / Why: Typically this will be due to the allowed number of concurrent connections on the server and is usually down to a configuration or license setting. For example IIS running on a non server version of Windows is limited to 10 concurrent connections - after this point it will deliver a 503 message. There is a temptation in load testing to overload the application under test, so it's worth revisiting your non functional requirements - will the production server ever see this number of concurrent connections?

References - w3.org

How do you determine the number of concurrent users you will need in your test?

noreply@blogger.com (Adam Brown) — Fri, 02 Oct 2009 11:36:00 +0000

It's quite a common question and sometimes a very difficult one to answer -
How do you estimate how many virtual users you should simulate in a performance test?
However if you have the following information you can make an estimate.

You need to know:

a: How many visits your site gets or how many visits you're expecting per day (choose a relevant day - e.g. if you're a business to business web site choose a weekday, if your a retail site perhaps a weekend day would be more appropriate)

b: The average time a user stays on the site

c: The ratio between a busy period and a slack period

d: How many minutes in a day (or at least how many minutes you consider the site to be active in a day)

You can then apply the following rule:
( a x b x c) / d = concurrent users

For example: you have a site which sees 30,000 visits each day, the average time a user spends on the site is 10 minutes and busy periods are 10 times busier than slack periods. This means that the concurrent user rate is (30,000 x 10 x 10) / 1440 = 2083 concurrent users.

I won't claim to have come up with this formula myself - it's from Performance Planning and Deployment with Content Management Server - a paper about MS's CMS system. The formula will apply to any web site, not just those hosted on these servers of course.

Should you automate your tests?

noreply@blogger.com (Adam Brown) — Tue, 29 Sep 2009 16:51:00 +0000

Most IT Managers behave as external software engineering companies with their own organizations.

How the IT department gets involved within the company’s business issues is fundamental. Indeed they can be involved in many tasks such as:
- Defining the business needs
- Developing the applications
- Choosing the appropriate solutions on the market
- Setting and managing appropriate infrastructures
- Managing application & infrastructure resources everyday

What’s more the SLA’s between the IT department and the business departments strengthen this role of internal provider of services.

Once an application is developed or acquired it is obvious that its functionalities and its performance must be qualified and validated.

This document deals with the relevance to automate – or not – functional tests and performance tests in this context. It also deals with the application & infrastructure monitoring during the production cycle.

Functional Tests

Your goal is to check that the application meets the specified requirements. Then throughout the application lifecycle the regression tests aim at validating that the application still runs despite its evolutions. Beyond the unit tests achieved by the developers or the project managers, an independent department dedicated to validation / qualification / testing should certify that the applications run properly and meet the requirements specified.

So the first question is: Should you automate those functional tests or not? The second question: Will you be more productive so that your initial investment in functional test automation is justified? Everybody knows that automatic tests are 5 to 7 times more expensive than manual tests. They can be even more expensive when the first testing campaigns are realised by teams with poor experience in automation tools whereas a team dedicated to validation & qualification with a good experience in programming could reduce the costs.

Thus this investment will be recouped more or less quickly depending on the frequency of the minor changes included in the maintenance of the applications. The word “minor” is extremely important, indeed any major change to the application architecture or the GUI can impact the test scripts considerably. Sometimes you even have to rewrite them all.

Several contexts fully justify functional test automation. For instance software editors have to maintain several versions of their products at the same time, and this in several languages, for several system environments, for several browsers. Moreover they use several patches everyday. In this case test automation is vital – it is an integral part of the lifecycle of the products.

In the context of a specific application developed internally, it is obvious that you have to compare the initial costs – acquiring solutions, training testers and creating scripts of course – with the benefits of your test automation given the frequency of the tests made on minor changes.

Performance Tests

In this case your goal is to check that the response times experienced by end-users meet the specified requirements. Respecting those requirements is vital when the response times have a strong impact on the business goals or the user’s work. For instance if your application permits to place an order online, the global order management can take more time because of initial bad response times, and the sales figures can be reduced because the data entry component is unavailable.

Whereas it’s easy to make functional tests manually, you hardly can make load tests manually. You would have to ask dozens or hundreds of end-users to stop their work to make the tests, they would have to be synchronised on a common clock, to make the same tasks at the same time, such as to click or to type in data. Each task should be repeated whenever a correction or a change would be done.

This approach asks such a lot of energy and resources that it is incompatible with most companies.

Several solutions are offered on the market, they are absolutely reliable and they let you obtain the same results without requiring real end-users. They permit you to create scripts in half a day if you work in HTTP environments.

The cost of load test automation is 10 to 30 times less important than the cost of functional test automation. It is all the more recouped rapidly that it permits to test applications but also to validate infrastructure changes – server, network, RDBM S, system versions, browsers, etc…

Application monitoring during the production cycle

For business teams the quality of the services delivered by the IT department can be measured through 3 main criteria:
• Availability: is the application available?
• Response times: do the response times fulfil the business requirements? As outlined above response times can have a strong impact on the productivity and even on the sales figures.
• Accuracy: the end-user needs quick responses, and above all accurate responses, not “404 page not found” results for instance.

An application – when it is delivered to the business teams - implies that complex logistics is set up including most of the time: application server, database server, web server, router, firewall, provider, etc…When only one component of this chain is defective, even if the problem is local, it implies that at least one of the criteria seen above is not respected. From the IT Department’s viewpoint everything seems to work, but the end-user does not agree.

As a consequence you must check that the components of your applications are ok from both the infrastructure viewpoint and the end-user viewpoint. What’s more this correlating approach will permit you to react before the business departments suffer problems sometimes. At last a personalized map will show the deterioration processes thus it will allow locating them geographically and alert the concerned teams immediately.

by Daniel Melloul, Quotium

Low cost performance testing

noreply@blogger.com (Adam Brown) — Tue, 29 Sep 2009 08:59:00 +0000

Performance testing has typically been an expensive business with the pricing standard set by Mercury (now HP) with their LoadRunner product. This has seen a proliferation over the last few years of tools vendors of this type in the market.
Recently we launched a new type of license - a pay as you go license which makes load and performance testing available at a very digestible cost.
It's been very interesting for consultancy companies to enable them to provide performance testing as a service under their own brand.
For now we're not going to publish information about this on the web site as many ideas in this market are simply copied, if you'd like to know more then please contact us at ukinfo@quotium.com, or come and see us at the free UKCMG show in London, or SQS, also in London (see our events pages).

Performance Testing Approaches

noreply@blogger.com (Adam Brown) — Thu, 05 Mar 2009 13:49:00 +0000

Nearly all performance testing is based on protocol simulation. This can be a neat, manageable, efficient way to performance test. Notably most tools in this market have the capability to performance test HTTP based applications. Some tools make a dogs dinner of script creation (usually open source), some are elegant & effective (usually commercial), and some hide everything under the covers including errors & mistakes etc and take a script-less approach (also usually commercial and in my view always best avoided).
HTTP lends itself very well to protocol level testing and most enterprise applications will use HTTP in some form or another. Of course not all applications use HTTP, there are still many that use other protocols such as ODBC, SQLNet, RMI and more. There are tools that can capture these protocols I can think of at least three, however in my experience the parameterisation of these scripts can be a daunting task for even the most seasoned automation expert. It can be time consuming, require expensive specialists and can fail. Also if the application is re-written, then the script often has to be too, causing further expense.
It leads me to ask the following question:
Could it be better to have the machines deal with the complexities of protocol chit chat and performance test from the GUI?

Here’s my view and I hope this will spurn further discussion.
Pro’s: Saves time & effort, eradicates errors in scripts, makes regression testing for performance less of a head ache, reduces the need for specialists, independent of protocol therefore can be a one fits all approach.
Con’s: Needs a lot of hardware (terminal servers and / or virtualisation), each piece of hardware / virtual hardware needs a license so can be costly, can be clunky in execution, synchronisation issues become more prevalent.

Are we in a situation now where hardware is so cheap and available that we should be looking at farms of servers to help us with our performance testing, rather than relying on a neat but expensive script based approach?

How to define performance testing requirements

noreply@blogger.com (Adam Brown) — Thu, 05 Mar 2009 13:01:00 +0000

It really depends on the kind of business the application is supposed to fulfil the requirements of.
In my experience most of the time there are two types of organisations:
- The type that roughly know how their users are going to behave
- The type who have no idea what to expect from their users
It's really easy to start to calculate the requirements for the type of organisation that has a captive set of users. Take for example a call centre for an insurance firm. They know they have to be able to deal with X sales enquiries every day. They also know that Y clerks will be connected to the application on a typical or busy day. You then know the concurrent user rate and the transaction rate. Therefore the application must be able to support this business scenario. X users and a Y transaction rate.
You have to do this kind of evaluation for every type of scenario that is deemed high risk or high volume, typically there will be 5-30 scenarios.
For the organisation that knows nothing about their users such as an organisation creating a new web site / application and opening it to a new market then you have to start with what you can do. That might be to find out how many transactions the application can support per hour under a given load, thereby finding the limits of the application – you’ll be surprised how few web apps do actually scale as I’m sure people who use proper tools have found out. *Note: Beware the tools market (open source & cheap) – I’ve seen some real horror stories using these tools relating to false positives. I’m definitely not saying use LoadRunner (in fact I’d actively avoid it) but do look around and run some evaluations.

It happens to the best of us!

noreply@blogger.com (Adam Brown) — Wed, 25 Feb 2009 10:26:00 +0000

Today this story made it to the general press (Metro) - even google make mistakes.

Application performance testing: Modeling load testing scripts

noreply@blogger.com (Adam Brown) — Mon, 16 Feb 2009 15:07:00 +0000

by Alberto Saynes - originally posted on blog.quotiumtechnoloies.com 23rd July 2008
Automated Load testing is used to seek the answers to many different types of questions at different stages in the development and deployment of an application. Before a first deployment in production, these questions might be:
“What load will my application support for a given level of performance?”
“How will my application perform in time under nominal load?”
“How will my application react to spikes in load levels?”
“What changes do I need to make to reach a given level of performance at higher load levels?”
“What do I need to do to increase the performance of my application?”
“How does the new version perform as compared to the last one?”“How would the application react to loss of IT resources?”…
Once the application is in production, the questions are more pressing and the time to resolve performance issues is critical. The main questions at this stage are
“Why is the application not delivering the expected performances?”
“What needs to be done to quickly remedy performance issues in production?”
These two sets of questions are of course very much related. However, the application performance management tools currently available on the market do not make it easy to leverage the pre-production test runs and results when in production or to feedback production performance data to better design further load testing runs. A first level of response to this issue can be offered by using the same performance monitoring tool both in pre and post deployment in order to quickly identify and debug performance issues. A more comprehensive way to address this issue is to use integrated products to link the load tests executed pre-production with the application performance monitoring set in place in production. Very few companies offer this type of integrated solution. Whatever the load testing and performance monitoring tools, one key element allowing to leverage any load test in production is the simulation of real-life loads, a result of successful script Modeling.
All load testing tools provide a more or less elaborate script recording function. Recording scripts however is only the starting point. In order to simulate useful real life loads, each script must be modeled to include user-related profiles, variables, and actions. How modeling is supported by the script editor included in a load testing tool is a critically important factor in productivity and in the overall value of your load testing activity.
Modeling includes all script modification and customization that will lead to different user behavior based on which virtual user executes that script. It also makes it possible for a virtual user to react appropriately to dynamic application responses. Modeling techniques might include algorithms developed to reflect statistical behavior observed in production, Variables introduced to make user actions dependant on dynamic application responses or other techniques. However, for most comprehensive web applications, some modeling is actually necessary for the script to function at all. One of the trickiest and most time consuming scripts modeling activity involves dealing with Session Identifiers.
Session identifiers are used by the application to uniquely identify a client browser, while background (server-side) processes are used to associate the session identifier with a level of access. Thus, once a client has successfully been authenticated to the web application, the session identifiers can be used as a stored authentication voucher so that the client does not have to retype their login information with each page request. Manually dealing with session IDs can be a lengthy script edition task (even with powerful parsing and editing functions), and it is prone to errors.
Models: High end Load testing tools supports the concept of model in their script editor. Models are sets of script edition rules rules which can be created by the user and re-used anytime to automatically edit any scripts with a simple click. For the most common session ID modeling requirements, the editor should come loaded with pre-configured, ready to use models. These should include one click automated modeling support for application such as:
.NET applications (__VIEWSTATE, __EVENTARGUMENT,… variables),J2EE applications (jsessionid)SAP applications (sap-wd-cltwndid, sap-wd-appwndid variables)Siebel applications (SWEC, _sn, SWETS,… variables)Epiphany applications (SGloginRequest,… variables).
Etc…
Using these readily available models considerably speeds up the script modeling process, reduces the numbers of editing errors or omissions, and saves precious time to focus on application specific modeling so that the simulated load reflects real-life use of the application.

Snow causes web site & service failure?

noreply@blogger.com (Adam Brown) — Mon, 16 Feb 2009 11:59:00 +0000

Before I go on I'd like to make it clear that neither I nor the company I work for are in anyway related to National Rial or their web site and that they are simply, being used an example.

During the recent snow in London this monthI became one of probably many furstrated mobile web users.

Most mornings I use my phone to check the satus of the train service running from my local station. Orange provide a pretty good service for this - in fact they feed this service directly from the National Rail web site, however as transport on that fateful February (2nd of) morning was seriously diminished, it seems everyone had the same idea and subsequently saturated the service with requests for information.

I then tried to access the national rail enquiries web site and found of course that that too was unavailable. In fact normal service did not resume until later that afternoon. I found this to remain the case for the following few days every morning - an obvious effect of the internet activity of the commuting public.

So what can be done about this?

Effective performance testing will improve the performance of the site under test by enableing technical resources to adjust tuning ettings in the applciation and therefore make it as lean as possible meaning more people can access the site and have a satisfactory response from it at one time. However if the whole country suddenly demands the web site we'll probably still have a problem in these rare cases. Perhaps a solution might be to use a queueing mechanism similar to that that we see in call centres - giving everyone a chance to use the system. Either way performance testing and tuning will ensure that the application is the best ist can be.

Application level monitoring from an end user perspective would alert operatives that the site is not available from the moment that it's down, additionally as soon as the site starts to suffer with poor response times or internittent

It's on my wish list to speak to National Rail!

How to performance test a public web site.

noreply@blogger.com (Adam Brown) — Mon, 16 Feb 2009 10:53:00 +0000

One of the hardest things to estimate is what kind of load your web site should be expected to stand up to – especially if the web site or web application is a new one. In essence what we’re trying to do is work out non-functional requirements before running the test.

Really there should be a requirements specification that the web application was built to. Something like:
“Response time must remain below 7 seconds”
“The application must be able to handle 3,000 concurrent users.”
“The application must be able to handle 500 transactions in one hour.”
The test then has to be run with 3000 concurrent users at peak, and while at that peak should be able to complete 500 business transactions in one hour. All the time page response time should never exceed 7 sends.

But what can be done if this kind of information does not exist?

We had this challenge when a customer needed a performance test running but had no idea of how many users the site should support or how many transaction he might expect in peak times.The customer had some statistics from the TV company they were advertising the web site through. GMTV told them that after a similar campaign the TV company themselves ran they saw 25,000 unique registrations on their web site in one day.

At the time nothing was known about the performance capabilities of the web infrastructure the customer had so the following approach was taken.
1: Evaluate what the system is currently capable of
This was done using our cloud internet load testing service & it was found the application could deal with only a few hundred users while under a load of a few thousand transactions per hour, it was deemed that the application was unusable if page response time was greater than 12 seconds. The same method was repeated every time the application was tuned for performance.
2: Establish a support mechanism in case the number of concurrent users the system could support was breached
A simple queuing system was developed which in effect increased the capability of the system to handle several thousand concurrent users.
The outcome was that the applications performance testing exercise was highly valuable provide. Primarily performance testing enabled performance tuning exercises to be run making the web application as lean and as fast as possible with current infrastructure. Additionally it provided the known limits of the application enabling the metrics for the queuing system to be specified.

Performance Testing Blog

noreply@blogger.com (Adam Brown) — Tue, 10 Feb 2009 13:50:00 +0000

Here it begins - my performance testing blog!
I'm going to try to update this frequently - whether that's monthly (at least) or weekly yet remain s to be seen.
It should be (I hope) interesting and relevant and where I get permission from customers and partners I'll include some case studies and short stories.

So let's start with my 1st entry: What is performance testing?
We can quote ISEB’s definitions, what we find in Wikipedia or what the current buzzword using conference speaker is saying which are all valuable (perhaps not so much with the latter), but where would be the value in repeating already stated definitions? I’m going to start with what I see as the definition of performance testing & you can decide if it’s valuable or not.

Performance Testing from my experience 90% of the time refers to evaluating the performance of applications deployed via web / HTTP technology. The other 10% of cases may be deployed via other protocols such as Citrix (ICA), Oracle (OCI), Oracle Forms (OCX?), SAP (proprietary), Remote Desktop (RDP) & bespoke (usually some for of TCP / socket connection) and many more.
Either way in all cases the application whose performance is to be measured is critical to the business, in fact it’s usually a core application envisaged to be used every day at high volume. Often these are internal applications such as financial reporting software, content management systems, timesheet systems and so on. If not internal applications the company itself may in reality be the web site application, take the example of an on line betting company, or a trading company like ebay. The companies performance (in $/£/€ terms) is directly related to the performance of the web site. Alternatively the company may rely solely on the web site as it’s point of contact with the public, take an events company – the web site may be used to register athletes for a run, it may be used to sell tickets for a concert.
So if performance testing is so closely related to business performance what is it we really need to measure? Well if we go back to what business’ main goals are, they relate to transactions – business transactions. So in fact most of the time the application under test has to support a given transaction rate to be able to support the business. It’s not as simple as that though as the transaction rate has to be maintained while a certain number of real users are using the application, additionally the application has to remain stable (up and running) and usable (i.e. not tortoise / snail like).
So take a timesheet application for example, end of the week or month a global company has 10,000 employees all needing to submit timesheets. Human nature dictates that people generally leave this to the last minute. However due to time zones there is a 24 hour windows for the timesheets to be submitted and the biggest country has 4,000 employees. Worst case scenario is that at 5pm that office decides to submit all timesheets in that hour, this means the application will need to deal with 4000 timesheet submissions in 1 hour, if you have the luxury of having historic data (perhaps from a previous system) you may know that you typically see no more than 400 users accessing the system at one time.
This means we can say that we need to simulate 400 concurrent users and run the test for at least an hour at a timesheet submission rate of 10,000 per hour.
What about public web sites where access is simply not controlled – that’s pretty difficult but I’ve heard stories about how companies deal with that.
This entry’s pretty long now so I’ll talk about that next time.