Preparing a Benchmark

To test the performance of your application, you need to run it on some input data. Preparing good data for performance testing can be a difficult task. For many server-type applications this may involve setting up several machines, and creating an “artificial” load. Constructing the benchmark itself is probably the single most time-consuming step in application tuning, but when done correctly, it can be worth the effort.

Generally, you want the benchmark to resemble as much as possible the real load that the application will face when finally deployed.

There are several mistakes that are often made at this stage. To avoid the most common ones, remember the following:

Make the benchmark repeatable; this is key to the systematic approach to performance tuning.
Do not create unrealistic situations which could speed up or slow down your application considerably. For example:
- Do not direct all the traffic to just a few records in the database which may cause exceptionally good performance because of caching, or exceptionally bad performance because of lock contention.
- Do not run all clients from one machine. This might not exercise your application properly because of the performance bottleneck on the client side.
- Do not use too much invalid data; it may never get to the components of the application that do the real work.
- Do not use a different environment (like the JVM, or options) for the performance testing than the one to be used for the real deployment.
Size your benchmark appropriately:
- If the benchmark runs only briefly, the profile data will contain too much “startup noise” from the effects caused by class loading, dynamic libraries loading, bytecode compilation and optimization, and the application initializations (database connections, threads, and data structures).
- If the benchmark runs for too long, it will be very time consuming to repeat the run, and you'll be tempted to make several changes to your application between the consecutive profiling runs, which is not recommended.
As your application grows in terms of performance, you may need to scale up your benchmark as well.

Preparing a Benchmark

Technical documentation

» Table of Contents

» Glossary

» Index