Friday, February 26, 2010

Still a Test

Tests are easy to recognize. They run in a test framework. They're named things like testFrobble or FrobbleTest. They do assertions and spit out results, generally some variant of "pass" and "fail".

There are other things that are tests as well, though. Even if they aren't named according to convention, or if they run outside of your test framework, they might still be tests. A test has a pretty broad definition:

A test is anything that exercises the system under test and which produces results that are analyzed.

Let's parse it out:

Exercises the system under test: To test a system, the test has to use the system. It could be humans (click through that GUI); it could be a sample program that uses the system's API; it could be a unit test that runs a single method deep in the system with various inputs. The only real requirement here is that it actually touch the system in some way.

Produces results: The test has to accomplish something. In the case of automated tests, this is often an assertion that something has an expected value or is in an expected state. In other cases it may be a more indirect result. For example, a script might use the system to get some information, then reinstall a machine based on that information. The successfully reinstalled machine is an indirect result; the system probably provided the right information if you ended up in the right place.

Are analyzed: This is the test version of "if a tree falls in the forest...". In order for it to be an actual test, you have to look at the results and do some analysis. Sometimes this is as simple as checking for the "pass" word and agreeing with it. Other times this involves digging through logs or output to make sure the system is getting exercised in the intended way, and to interpret the resulting behavior.

As long as it meets these three criteria, it's a test. It may not be the most effective test in the world, and it may not be the most efficient test in the world, but it's a test. So when you're looking at the tests you do, make sure you look at all of them, not only the ones that appear in your test plan, or that are named testFrobble.

No comments:

Post a Comment