Monday, April 25, 2011

ROI of a Single Test?

A coworker asked me this question the other day:

"How do I calculate the ROI of the coverage of testFooStress versus the time to develop and run it?"

It's an interesting question on several levels. Let's break it down a bit:
  1. Why is he asking? Why about this test in particular?
  2. What does ROI mean here?
  3. This is a stress test, so coverage is a bit of a funny word to use. Note to self: poke at this one a bit.
  4. There's an obvious implied measurement here of "cost" as "development and run time". Note to self: poke at this one a bit, too.
Okay, first and foremost, this person has an agenda. A general curiosity about, say, the ROI of tests, would not include the name of the specific test. Therefore, there's something about this particular test that is bugging this guy. First we need to find out what it is, since this conversation is really about that agenda and is only nominally about the ROI or the coverage or the cost of this particular test.

After some conversation, it came out that he was most worried that we weren't stressing the right thing, or possibly not stressing it enough, and that it was an expensive test in terms of number of hours it took to run each night. That's a fair concern. It's not exactly ROI, but it's legitimate.

Let's say, though, that we really were talking about ROI. How do we calculate that?

There are many different kinds of ways to slice and dice this. One of the most simple is the ROI of choosing to automate a test, which can be calculated roughly like this:

number of runs to recoup cost = cost of automation / (cost of manual run - cost of automatic run)

This, of course, assumes that your automated test accomplishes the same things your manual test accomplishes. I should note that this is only sometimes true.

Beyond that things get rather more complicated. It's rarely useful to talk about a single test in isolation, and it's rarely useful to talk about an ROI simply because so often the return is unmeasurable. The return is things like defects that never hit the field, for example, and that's not really something we can measure very well. If we're proceeding down this path, we want to talk about more of a general cost benefit analysis.

We can take as a given that we have X time, Y people, and Z machines (or other resources) to accomplish stuff. Stuff can be development of software, or tests, or documentation, or a myriad of other things. To do any given thing will take some subset of our time, people, and machines. So we do something, then we do something else, then we do a third thing, and repeat until we're out of time, people, and/or machines.

The trick is to pick the things that are going to give us the most benefit for our cost. A benefit is anything good that happens as a result of doing X. Some benefits are tangible: "If we build the widget control feature, we'll get 1000 more customers paying $1000 each!". Some benefits are best described as the reduction of pain: "If we set up a second web server and a load balancer, we won't have as much downtime." Some benefits aren't directly measurable: "If we change the web site from blue and white to green we'll look totally hip and modern!"

Things that increase benefit:
  • features that our customers have asked for
  • features that the market has asked for
  • features that will make us attractive to customers or acquirers
  • features that improve our brand (e.g., being seen as a market leader by coming out with features before anyone else does)
  • less downtime
  • lower visible error rate
  • cleaner, friendlier UI
  • more consistent behavior
  • lower risk (regardless of whether that risk is eventually realized)
Things that increase cost:
  • longer or more difficult implementation
  • resultant increased complexity of the overall system
  • increased risk (again, regardless of whether it is realized)
  • longer or more difficult run time (e.g., a test now takes longer or uses more machines)
  • changing user metaphors (e.g., new UI)
  • more or harder problems seen in the field
You can come up with constructs to measure the cost-benefit tradeoff of doing something, even of running a single test. It would probably look something like this:

Benefit = (likelihood of finding a defect * efficiency at finding that defect) + (prevented) cost of that defect occurring in the field + (prevented) cost in reputation of finding that defect

Cost = development time + (run time * number of runs) + analysis time

However, all of these numbers are pretty much just guesses. Educated guesses, maybe, but guesses. In the end I have more luck with simply considering the costs and the benefits, and making a choice from those directly rather than running them through a contrived algorithm.

So can you calculate the ROI of a single test? (or feature or refactoring or whatever). Yes. It's probably more productive to think about it in more general terms, though, and to break it down to the fundamental question of: "is this worth what it'll cost me?" In the end, that's the important decision to make.

No comments:

Post a Comment