Tuesday, December 8, 2009

Stress, Don't Break

"We just got a new build, let's break it!"


You have a piece of software to test. Destruction is not the point. The point is to anticipate and subject the system to things that might happen, even if "might happen" is an extreme circumstance. If the system happens to break under those circumstances, then it's great you found it, so you can react appropriately. If it doesn't, then you know something about the system anyway. And that's good, too. What you're really doing in this case is attempting to stress the system. You're looking for the edges and the holes.

Every system has rules. Some are explicit (e.g., don't install more than 5 of these on a single server) and others are implicit (e.g., it's never going to send packets across the network at faster than wire speed). These rules should help guide your exploration of the system. You may use them to do boundary value analysis and testing, or to design a load test, or to decide at what points you want to measure latency, or whether fuzz testing is going to be worth the time it takes. In some cases, it's possible to break the rules (e.g., you might be able to install 6 of these on a single server), and that's probably a legitimate bug. It shouldn't usually be the starting point of a test; generally it's the end point.

However, if you have to break the rules to break the system, then your goal has changed from "gather information about a system in order to help make intelligent decisions about it" to "KILL!". This misalignment of goals negates some of your testing. You're likely to go far down paths that don't provide a lot of information, just because it's a way to break the system. Here's a hint: you may be acting like a bit of a jerk about it, too.

So don't be a jerk. Exercise a system, yes, but keep in mind that your goal is information and stress, not breakage. System breakage or lack thereof is a side effect of information, not the other way around.

No comments:

Post a Comment