Thursday, March 25, 2010

Why We Run Performance Tests - Part 2

Performance tests are one of those things that can be run forever. Unfortunately, we don't have forever. So given restricted time and restricted resources (machines, network, people), we should only run performance tests that actually provide some value. Yesterday we talked about some of the outward-facing needs filled by performance tests. What about inward-facing needs?

Fail Fast Deterioration or Improvement

Why Do We Want This?
When we do functional testing, we look for things to fail quickly. When we break something, we want feedback as soon as possible that we've broken it, so we can fix it before we go too far down a bad path. The same theory applies to system performance (however we've chosen to measure it). Maybe that refactoring made the code a lot cleaner but also got you a performance bump; wouldn't you like to know so you can do more of that kind of refactoring? Maybe that memory-saving innovation actually killed your performance; it's probably better to find out before you get too attached to it or even worse, have to go mucking with it at the very end of a release cycle.

How Does This Change Our Test?
This test is all about rapid feedback of areas likely to change. These tests should be as short as possible so they can be run on checkin, or daily, or at worst weekly. Automated reporting is useful here, just like it is in a checkin or smoke test for functional tests. For this kind of performance test, change is what's important. Making the tests run consistently is more important than things like upgrading hardware or trying different things. Also consider testing at a lower level than full system (i.e., component performance tests) to pick up more subtle performance changes.


Support Performance Work

Why Do We Want This?
The time will come eventually when we want to actually improve the performance of the system. At this point, there is a need to more fully profile the system as it performs under different loads, find bottlenecks, and change the system implementation to make it run faster under those interesting loads.

How Does This Change Our Test?
This test is mostly about finding bottlenecks and places for optimization. The goal here is to make it easy to run tests, because there will be a lot of different variables, a lot of speculation, and a lot of "okay, let's try this new build". For these tests, look at running with profilers, different load characteristics, etc. Given all the flexibility here, the way to handle these performance tests is to bound them on some non-test variable. Usually, these will be either time-bound or speed-bound. For example, the charter might be "we'll spend two weeks on performance improvement" or "we need to get to Y MB/s". Consider dedicating a team member and some resources (machines, etc) to the effort until the charter is finished.


There are a lot of other reasons to run performance tests; these are some of the most common I've seen. In any event, your tests will be much more effective if you can describe why you're doing them and what you will do with the results. These posts have provided examples for performance tests, but it works for other tests, too, including functional tests, usability tests, testability tests, even security testing. Use your needs to inform your design; you'll waste a lot less time and make everyone involved a lot happier.

5 comments:

  1. Just 1 comment... I hope that you start doing the Performance tests on a stable system. Any thoughts on how to keep doing performance tests on an application that's prone to change every fortnight???

    ReplyDelete
  2. How you address a rapidly changing application is going to depend on the kinds of tests you're interested in. As with any tests, you end up with two choices:
    1. finish your tests before the app changes under you
    2. stick with a given build for a bit

    For the tests I've described here, unless you're actually releasing these builds to customers, it's probably okay to skip a few if you really need longer tests. If it's truly not okay (you're releasing that frequently), then you've got to get your tests down far enough in duration that you can run them as frequently as your releases demand. So you automate, cut corners, or play tricks like using a continually upgraded system for these tests to get your elapsed time down.

    ReplyDelete
  3. Comments on Part I and II

    Hi Catherine,

    You bring data transmission speed ("Y Mb/s") here and there as a typical performance characteristic. Wouldn't be more pragmatic to look after response time at the front-end, and CPU usage + RAM usage at the back-end?

    Thanks,
    Albert

    ReplyDelete
  4. Albert, depending on what you're looking for, sure, you might measure other things like CPU utilization, or database connectivity, or whatever. MB/s just happens to be a good first place for the system I'm usually working on. Other times, it might be average page load time, if it's a basic webapp, etc.

    ReplyDelete
  5. Well, I guess the thinner is client the more important is connection speed.

    But for a web app I'd still clearly distinguish response time and load time in my reports.

    Thanks,
    Albert

    ReplyDelete