Monday, May 17, 2010

Bug Rhythm

Projects have heartbeats. They have their own rhythms and sequences, and these can be used to make predictions. A rough measure of stability can be gained, for example, just by looking at bugs over time.

For example, we have an automated test suite that runs every night. Over time, a pattern will emerge that can be roughly graphed like this:

The project starts. Then there's a few bad nights as many tests fail, then it gets better. Later in the project there's another spike, but smaller, followed by improvement. This is followed by another spike, smaller than the second, and so on. Each spike is an instance of destabilizing changes being checked in. Sometimes they're new tests, sometimes they're new features. Sometimes they're manual tests, sometimes automated. Sometimes the problems being pointed out by the failures are issues with new features, sometimes regressions, sometimes a bug that existed but wasn't exposed before, and sometimes a combination of all three.

The specifics don't matter. I don't particularly care what caused each bug logged here; it will be different every time. I'm more interested in the rhythm, in the pattern.

By stepping back we can see that over the course of a release, we destabilize as we take on big things, then we restabilize. More changes going in destabilize a bit and then restabilize again, with an overall trend downward - toward stability and toward a point at which we find far fewer bugs. We can also look at the periodicity of the spikes to figure out how long and how frequent they are. Look at this pattern over enough releases and you can start to see what an overall release rhythm looks like.

Maybe you tend to have four or five spikes, and then you stop finding things. Well, if you're just done with spike 3, you can expect one or two more destabilizing changes, and then you'll probably be able to release. You can even put a rough timeline on it, based on past releases and the patterns they showed.

This is a heuristic, not an absolute metric, but it's a good way to get a handle on a release. Pull back from the details and let the pattern tell you what is likely to happen.

No comments:

Post a Comment