Wednesday, October 27, 2010

Beware the Matrix

Matrices are very common in testing. Here's one we use at one of my clients:

It looks simple, and it really is. We have our tests written. All we have to do is run the test in a loop for each of the squares in our matrix. Hooray!

There's a catch. (There's always a catch.)

In this case, the catch is that each test takes about 5 hours to run, and we don't really have the machines to spare to run 100 different configurations at 5 hours each.

That's the problem with matrices; they seem like such a good idea, and they're really easy to design as test cases. Unfortunately, they're often something that simply can't be completed within your constraints, and even presenting a matrix like you see above generates an expectation of completeness on the part of anyone who looks at it. If I show this to my boss, for example, his response will be, "Great! Let me know when we can see the results." Unless he's willing to spend a whole lot of money on machines (and we do have a budget), the answer is "No time soon, and frankly there are better ways to spend our time."

Depending on what's in your matrix, it might be a good candidate for combinatorial testing. In particular, if your equivalence boundaries are accessible, consider using this technique. If your needs are different, then consider flexing along one variable at a time or simply skipping steps. In my example above, we're testing performance, so we don't necessarily care about every single step along the way. We can run with the extremes and a few values out of the middle. Only if something "looks funny" (that's a technical term!) do we need to go back and fill in the blank. In the end, we wound up running every other cache size and only at two chunk sizes. From that we learned that cache size didn't seem to make a major difference, but chunk sizes did. Test complete, information gleaned; the total cost of the effort was 10 tests - 10% of the cost of the entire matrix.

Sometimes you really do need to fill in the whole matrix, and when that happens, get cranking and start filling in boxes. Just make sure before you present a matrix that you really do need the entire matrix. If you only need part of a matrix, only show your test consumers (dev, management, whoever) the part of the matrix that you actually need and that you actually intend to test.

As with all tests, when you're looking at a test matrix, ask yourself what you're intending to learn from running these tests. Then run only the tests that give you the learning you need. Skip the rest; there are plenty of other things to do!

No comments:

Post a Comment