Wednesday, February 10, 2010

A Use for Counting Bugs

For the most part, proposing a metric comparing number of bugs found to number of bugs fixed will get you laughed at. After all, bugs are not alike, and a dozen teeny bugs will look far worse under a "bugs found/bugs fixed" metric than one bug that crashes the system and causes data loss on its way down. In practice, often the one crash is more important than the dozen teeny bugs. Plus, a "bugs found/bugs fixed" metric is incredibly easy to game. Find an error in a page header? That's a bug per page right there! Woo hoo! Numbers goin' up (and this isn't helping anybody).


Not using a metric to measure tester or developer performance doesn't make it totally useless. Check out this chart, which tracks number of bugs found (red) and number of bugs fixed (green) over thirty days:

There are a few things this chart actually is useful for:
  • Charting overall volume. If you find 25 bugs in one day, is that a lot or not too many? Well, looking at our chart, we see that we've never found more than 8 bugs in a given day. Unless something has changed on the project (more people, different focus, new test technique), then you've found a lot of bugs. Now, stop congratulating yourself and figure out what's going on. Did we just do a big checkin? New feature? Refactoring gone wrong?
  • Project Stability. A chart with a lot of spikes like this one is often an indicator of instability in a project. A lot of things are getting fixed, and the find rate is keeping up. This doesn't tell you why the project isn't stable, but it's an early sign you should go look at what's going on. Maybe it's simply mid-development and a lot of new feature are coming in. Maybe there's something wrong with your process or code base and you have a lot of people breaking things as they fix things. Perhaps the fixes are exposing more problem areas, in which case you may simply be working through a nasty code base.
It's rare that looking at any number - including number of bugs found or number of bugs fixed - will give you a lot of answers. Looking at these kinds of trends, though, can give you clues about where to look in a project to see what's going on. Don't throw the baby out with the metrics bathwater; take a look and see what clues you can find.

No comments:

Post a Comment