Monday, September 8, 2008

People Don't Remember "About"

Be very careful with estimates. 

It happens sometimes that a tester will be pressed for an estimate.

"How much improvement will this cause at our customer sites?"
"How many bugs are you going to find in this release?"
"What will be the impact of unanticipated customer requests on this release cycle?"

The proper answer to all of these is "I don't know", but that of course is insufficient. So the temptation is to give an estimate.

You say something like this:

"We're not totally sure since the customer's environment is a factor, but in test we've seen a 20% performance increase, so the customer should be something similar."
"Hmm... last release we found 152 bugs, and there are half as many feature points in this release... so, about 70 or 80."
"We get on average one unanticipated customer request a week, and that takes about 10% of the team's overall velocity. Over a 10 week release cycle, well, that's one man week."

Your audience heard something totally different. They heard:
"70 bugs"
"one man week"

In general people here what they want to hear, and in this case they're looking for comfort and certainty. So they're going to take the "hard facts" part of your estimate and that's what's going to stick in their mind. All the caveats and hedges you put in your answer - gone until prompted. To be fair, when prompted, most people will remember the subtleties, but it isn't what's springing to mind.

So how do we deal with this? How do we make sure we're not overpromising unintentionally?
  1. Don't give an estimate unless you're pretty darn sure. Characterize instead. Phrases like "significant improvement" or "fewer bugs than the last release" are accurate (although less precise) and tell people for the most part what they need to know.
  2. Be conservative. Don't put your best test run out there as a best-case scenario. Use your average run, or your run that most closely resembles your customer pattern.
  3. Be clear when you know versus when you're estimating. Sometimes you really will know the answer. In those cases, be very clear. Other times it's an estimate, and there you need to repeat the word "estimate" to make sure people get the distinction.
  4. Repeat. Repetition is an aid to memory. Keep repeating the assumptions and variables, and that will help your audience remember them.
Estimates are not inherently bad. On the contrary, they're a fact of life. Don't be afraid to estimate. Just tailor your message to people's what people here; be crisp, clear and just a bit repetitive, and everyone will get on the same page.

P.S. All numbers and estimates are totally made up.

1 comment:

  1. Hello Catherine,
    I liked your posting especially when I had recently a similar idea about this only I wrote it in some other words of course.

    How much don't we test?

    I think you gave some very good additions were we testers have to be aware of.
    Thanks for sharing your thoughts.

    With regards,