Wednesday, March 24, 2010

Why We Run Performance Tests - Part 1

One of the wonders - and dangers - of performance testing is that you can run it forever. You can look at latency, or scalability, or throughput. You can change all kinds of variables about size of the system, what kind of traffic is coming through, system configuration and size, etc. It's no wonder some people make a full time job of it! Then there are the rest of us. We don't have a dedicated performance team or a dedicated tester. We do performance tests along with all the other tests we do. This model is pretty darn common, too.

So when you're restricted by time, as many of us are, you have to narrow your tests. In order for them to still provide value, you can't really explicitly hit every variable. So you have to optimize your tests to provide the information you're looking for. Let's say, just for example, that we've decided to run performance tests based on aggregate read and write throughput to the system. What would we do with those kinds of test results? To phrase it another way, why do we run these performance tests, and how do they affect what tests we run?

Marketing numbers

Why Do We Want This?
Marketing needs a nice fast number to put up against all of our competitors. This number usually represents the fastest possible aggregate ingest we could ever hope to see in the field under ideal circumstances. This will go in analyst briefing materials, brochures, and other marketing materials, usually with phrasing like "speeds of up to X MB/s". Typically these get run every time something substantial changes, like new hardware, or a major new feature, or a big performance push. It doesn't necessarily happen with every release.

How Does This Change Our Test?
This test is all about optimization to get a big number. Based on field experience, past performance tests, discussions with dev, we're looking for the fastest configuration we can find and the fastest data we can find. Empty system? Large system or small system? Number of clients used? Amount, type or size of data ingested? All of these things are geared toward getting a fast number. Don't ever cheat. Don't ever do something unsupported. Your numbers may be checked and you'll need to be able to show how you did it and why that's a conceivable number. But beyond that, optimize away.

Prevent Client Slowdowns

Why Do We Want This?
When we put a release into a field, you want to be sure that it doesn't hurt your existing customers. Just like we try to avoid functional regressions, we try to avoid performance regressions. The goal of this testing is to ensure that customers get a performance experience at least as good as what they have today. These performance tests need to be run on each release.

How Does This Change Our Test?
To give confidence that we haven't hurt our customers, we need to be doing approximately what our customers are doing. We won't be a perfect imitation, but the bounds need to be about the same. This means a typical customer hardware configuration, a typical software configuration (e.g., compression enabled), typical number of clients writing typical numbers of files. One test probably won't cover it, if your product is being used in several ways. Create a performance test for each basic use case in the field.

There are other purposes to performance tests, so we'll continue tomorrow.

No comments:

Post a Comment