Friday, April 11, 2008

Eleven Million Copies of the Same Error

We have a great test infrastructure at work. We run tests automatically every night, and they do this*:
  • start the test run
  • do a build
  • pick which test suites to run
  • reserve machines for the test (from our lab - with 300+ machines, you want a reservation system!)
  • run the test suite
  • gather logs from each machine used in the test
  • parse the logs to figure out whether the test failed
  • updates a list of failing tests and count of tests run
This is great, and generally works quite well. We come in every morning and have access to a central summary of exactly what failed and why, with all the logs right there. The lab just hums along in the background, doing other things.

There's one issue here, and it's really a general QA issue. Sometimes we come in and we've had a lot of tests in a lot of suites fail.... and all the errors look the same. One common example of this is ssh failures. Sometimes we'll come in and see 100+ failures all with the error "could not ssh to . No route to host."

So. Is this one bug, or many bugs?

Arguments in favor of one bug:
  • No one likes duplicate effort. That goes for bugs, too.
  • If it's the same problem, it's almost certainly the same fix. One problem, one fix, one bug (has a nice ring to it, huh?).
  • Don't get more people that necessary looking at the same bug.
  • Helps recognize the extent of the problem.
Arguments in favor of many bugs:
  • Bugs are a lot easier to merge than to separate.
  • There is a risk you're lumping multiple problems together under one issue.
In general, I will log many bugs and reference them ("looks like bug 42321") from each bug. That way it's easy to merge later. I tend to be risk averse, so I'd rather have more work and lower risk than less work and higher risk. Once we get the signature of the problem (how to differentiate it from similar problems), then we can feel free to consolidate.

How do you handle similar errors?

* For those of you who work with me, yes I'm simplifying a bit.

No comments:

Post a Comment