Tuesday, May 6, 2008

Measurement and Effectiveness

There are a lot of ways to measure how productive the people who work for you are. In the end, any system can be gamed, and whatever system you come up with will eventually be manipulated to make someone look good. The trick to measuring productivity is to make your measurements actually reflect reality.

The Good
Some ways to measure this are really good. Typically these are a complete pain to gather, but if you're really interested in measuring this on a quantifiable rather than a gut level, you've got to start digging hard.
  • On Time Ratio. Measure how on time the person is with tasks and projects. The idea is to over time average how far off the person is. For example, someone who completes a 10-day project in 9 days has an on-time ratio of -10% for that project. Someone who does the same in 11 days has a +10% on-time ratio. The idea is that over time, for productive and effective employees this should approach 0. That is, the person should, over a number of projects, prove to be accurate. Note that someone with a high negative number (so someone who is generally finishing early) is not necessarily effective; they're just padding estimates.
  • Management Ratio. I run a very self-directed team; my title is Lead, not Manager and there's a reason for that. I don't spend all of my time managing someone. Sure, things come up, but it shouldn't be a constant. The management ratio defines how much time you spend with each person performing management duties. This is a comparative matrix. If you're spending 0 time with a person, then something is wrong and you the lead probably aren't helping him be as effective as he could be. However, spending too much time with someone is a drain on the whole team. So, measure how much time you spend on management activities with each member of your team. If some team members are taking inordinate amounts of time, then there's a problem. The specific definition of "inordinate" will vary a little bit, but anything over twice your average is probably bad.
The good thing about these metrics is that gaming them is exactly what you want. You want your tester to try to game the system by making estimates really accurate. You want the person working for you to game the system by not requiring a lot of hands-on management.

The Bad
Bad measurements tend to be really easy to gather, and they look like they're telling people a lot. In reality, they are incredibly easy to game without actually helping the team.
  • Number of bugs found
  • Number of bugs verified
  • Number of test cases written
  • Code coverage
  • Run length of automated tests
All of these items fall short because improving or increasing them does not have an inherently positive effect on the employee, the team, or the system under test. Interestingly, these measurements tend to be automatically gathered and tend to masquerade under the fancy-sounding term "metrics".

The Ugly
Some metrics treat employees like real cogs in a wheel. I hope by now everyone has given up on these:
  • How many hours you work.
  • How many modules you have tested.
  • How much you talk in team meetings.
This is both dumb and short-sighted. Just because I can see someone doesn't make him effective.

Ultimately, a good measurement of an effective employee gets back to gaming the system. Accept that whatever system you use will be gamed. The tricky part, then is to set up the system such that gaming it produces desirable results.

What other measurements or incentives do you use?


  1. I not sure if "productivity" is the right term to describe or measure what we do as testers.

    Testing is more like an investigation, thinking, critical analysis - much like the way scientists, investigators, artists, philisophers and other creative thinkers work.

    The productvity is best applied to tasks that are routine, repetitive, mundane (assembly line) kind of work. Testing does not fit in this context ....

    Chris MacMohon suggested the term "performence" (like that of artists) to describe the work of testers ....


    Shrini Kulkarni

  2. I've actually never heard of that measurement of productivity. If I had to define the term, I'd say it was fairly simple:

    Productivity is the amount of benefit received for the effort expended. We can be productive when washing dishes or other mundane tasks. We can be productive when tracking down a really hard bug.

    I don't think that I agree that productivity doesn't apply (or can't be measured) for testing. On the contrary, part of my job as a QA Lead is to help the people who work for me to be more productive. Sure, it's harder to measure sometimes, but it's still productivity.