I've been at STPCon this week, giving a few presentations and going to several presentations. One of the more controversial sessions there was on metrics, and how to use metrics to get to "success" (whatever measure of success you're using). The most interesting part to me was the discussion afterward.
You see, metrics sound like they ought to be awesome: we can measure what we're doing and then we'll know whether we get better or not! In practice, they tend to be pretty much unrelated to things that actually help us and actually help the business, and they are either neutral or they can do more harm than good. I'm not quite ready to throw this baby out with the bathwater yet, though.
Metrics are a time for
Not a time for
There are three main types of metrics that you can use:
- internal metrics
- process-oriented metrics
- business-oriented metrics
Internal metrics are the ones you don't tell anyone outside of engineering about. They're the things you track to measure your own performance. Internal metrics include checks on how accurate your estimates are: "On this last project, we were on average 20% over our estimates." or "The five last releases, we found a lot of bugs in the whizbang component relative to all the other components. Maybe we should go looking at that one a bit more". Internal metrics can be great information providing tools. Some of them are really only useful to point out potential problem areas in a one-time look, while others you can use for years. The important part of internal metrics is they provide information to engineering and don't have a lot of relevance to other departments. These are the things you go find when you say, "I wonder what..." and don't need to put up on a big project dashboard.
Process metrics are the ones that can be the most dangerous, and they're really common. These are metrics like "number of bugs found per story" or "number of bugs per thousand lines of non-comment code". The problem with process metrics is that they measure how you do things, not what you accomplish. Given that most people are trying to game metrics, gaming these changes how you do things, but not necessarily what you do.
Business metrics are the ones that measure effect on customers. Revenue is an example of a business metric, as is % of returning versus new customers, or brand value (Coke is worth over $66B). These are the metrics that are directly tied to the success of your business. Bringing these metrics into test is often somewhat difficult since better testing rarely can be tied directly to new customers (indirectly, of course, the overall quality of your product affects how many customers you bring in and how many you keep). However, if there is an issue will cause you to lose customers, or to be publicly embarrassed and hurt your brand value, the effects of that issue can be tied to these business metrics.
Don't throw metrics out entirely. Just be careful that your metric is limited in scope to ways that it will be useful, or that it can actually measure what you affect for the company's overall purpose. Throw the rest of it away - you'll have more time to spend actually testing.