Thursday, November 4, 2010

Expected Bug Levels

I had a conversation with my boss today that started like this:
Him: "Gosh, I've noticed a lot of bugs lately. The bug count is higher and there are lots of new ones."
Me: "Yup."
Him: "Should I be worried?"

Now that's an interesting question. The correct answer changes based on when we have this conversation. (In this case, we're just after feature freeze, so no, he shouldn't be worried.)

Every project I've worked on has a natural bug flow, like a tide in the ocean. Sometimes there are more bugs found, sometimes fewer. With small variations, it usually goes like this:

  • Project start: Early on in a project (or release), there isn't much new. Most of the work being done is analysis and coding on things that simply aren't checked in yet. The average number of bugs found in this period is pretty low.
  • Mid-Development Cycle: As some features start to get checked in, and the testers get some of their tests done, the bug count starts to creep up. Some bugs have been introduced, and some are being fixed, but the focus of the work is generally still on feature implementation, and the number of bugs found only comes up a little.
  • Feature Freeze: Regardless of whether feature freeze (also known as "a first pass at each feature has been done but it isn't baked yet") is formal or informal, there comes a point where pretty much all the basic implementation is complete. At this point the code is just starting to play together, and the volume of code being checked in is pretty high. The number of new bugs logged also goes way up; this phase often represents a high point in terms of quantity of bugs found. Note that this isn't inherently bad - we've got a lot of things and now we have to get them all to play together. It's just a relative high point, and it's to be expected.
  • Polishing: After the features are basically implemented and before release, the number of bugs found and number of bugs open generally trends down. Integration points are smoothed out, bugs that were ignored because someone was heads down in a feature are fixed, and the entire product solidifies toward something release-able.
This isn't a metric, and it isn't a value judgement. It's simply a project heartbeat. Seeing "lots" of bugs logged - regardless of what "lots" means to your project - might be okay or it might be a big problem, depending on where you are in a project. Take a look at your defect tracking system: it's one place that might help you see whether your project is behaving normally or if there's something going on that warrants investigation.

No comments:

Post a Comment