Tuesday, January 26, 2010

Reverse Your Tests

We have a set of tests we run every night. They generally run in the same order:
  • Test A
  • Test B
  • Test C
Often this works just fine. However, sometimes bad things happen. Test B might hang, and Test C just doesn't get run at all. Maybe there's a power failure halfway through Test A, so that night you don't get Test B or Test C.

Over time, you accumulate more runs of Test A than of Test B, and more of Test B than of Test C. You're exercising some code more than others now. Any defects that are shown by Test C are likely to remain hidden for longer than defects shown by Test A, simply because you have fewer opportunities to discover them. In the short urn this isn't a big deal, but in the long run, your testing starts to get lopsided.

If you're not getting through all your tests for one reason or another, try one simple thing to smooth out your coverage:

Reverse your tests.

For a while, run your tests in this order:
  • Test C
  • Test B
  • Test A
It's hardly perfect, but with most test systems this is pretty simple. In our case, it was a matter of adding a call to reverse(@tests).

Running your tests backwards actually provides several benefits:
  • It runs your least-run tests first for a while and evens out the frequency of coverage a bit. You probably won't run Test A quite as much, but you'll have more runs on Test C and you'll flush out problems there. When the pendulum has swung far enough, change order again.
  • It tells you whether your tests are really as isolated as you thought. If there are dependencies between tests, then you'll start seeing breakage and failures due to those (now missing) dependencies.
  • It's change. Maybe this one is just me, but even little changes like this make everything I'm doing just a bit more interesting.
Watch your tests, and watch how often you really run them. If it starts looking lopsided, it's time to change things up. It's time to reverse your tests.

2 comments:

  1. Some tests harnesses would allow you to run your tests in random order.

    Where this is possible, I like to do that.

    ReplyDelete
  2. Catherine, you are right, executing test cases in variations often quite productive in finding hidden issues.

    Nice idea :)

    ReplyDelete