Tuesday, March 17, 2009

Aging Reports

When we talk about measuring engineering - both development and QA - there is pretty much always a need to measure ourselves. There are about as many ways to do this as there are to build software, and you have to be very careful about how you implement it. Any metric can, and probably will, be gamed. So you want to make sure that gaming the metric produces results that you find desirable.

One of the metrics I often like to consider is an aging report.

What Is It?
An aging report is simply a measurement of how long something takes from identification to conclusion. Most of the time, this is considered to apply to defects, but it can also be applied to backlog items or stories, if you're interested in measuring that. The key point here is that you have to be sure you're measuring things the group can control; to measure the team as a whole, look at all states combined. To measure individual groups, look at only states which that team controls.

Setting this up for a defect tracking system:
  • Identify each state in your defect tracking system, and which states you are interested in measuring
  • For each item, you are going to need to know how long it spent in each interesting state. How long was it in the "opened" or "reopened" states? How long was it in the "Ready for QA" state?
  • Keep in mind that a bug may be in some states more than once. You need to count the sum of all these times.
  • You're going to want a script for this if your defect tracking system doesn't do it for you.

Setting this up for stories or backlog items:
  • This is basically the same as for defect tracking systems, except that you may have to define your own states and change how you actually get the measurements out of the system.

Once you have this set up, you need to consider the stats:
  • Largest, smallest, and median show you both outliers and also an average that you can look at for trends.
  • Like most statistics, this is a rolling average, usually over the last 30 days.

So our defect aging report winds up showing us, for example, "average time bugs were Ready for QA but unverified over the last 30 days". Substitute other values as you need!

What's It Good For?
This is a metric intended to show how quickly we can push things through our process. A presupposition is that speed is good when resolving issues or implementing backlog items. While it is most often good for bugs because that's relatively easy to track, this can be used to measure any time a single group has responsibility for making progress. The real trick here is to look for trends rather than a specific value; you want to see the number go down as engineers get faster working through things.

As with all metrics, there are some downsides:
  • If you don't count total time spent in a given state (e.g., if a bug gets reopened, you need to count both open times), then there is a tendency for things to be ... eagerly moved.
  • If your development effort is very spikey.

Note that this is one of an occasional series on metrics.


  1. You say above the following:
    "Any metric can, and probably will, be gamed. So you want to make sure that gaming the metric produces results that you find desirable."

    It seems to me that the defect aging metric being gamed (i.e. people pushing defects through to verification faster by skipping tests they should be running) is not desirable. Maybe I am missing something?

    I agree that you have to be very careful about what you measure, particularly if you tie that measurement to someone's performance.

  2. Brian, nice catch! I hadn't thought of that one. If you missed something, then I am, too.

    I'll have to think about that.