Monday, October 27, 2008

Bug Verification Checklist

Verifying bugs is a bit of an art. It's also a time to make very very sure you're right. After all, there are only four possible scenarios:
  • You verify a bug and it's actually fixed. This is what we want to have happen.
  • You verify a bug and it's not fixed. This means you're going to find it in the field. Your customer will be unhappy AND you'll have egg on your face. Not good all around.
  • You kick a bug back to dev and it's not fixed. This is the second best scenario; a fix would have been better but, hey, at least we caught it. In the end, it's not much worse than finding the bug in the first place.
  • You kick a bug back to dev and it's actually fixed. This is where we waste time. It's a minor embarrassment and it erodes developer trust in you a bit (rather like finding a bug that's clearly not a bug).
So, with only one happy outcome, one mediocre income, and two chances for us the testers to embarrass ourselves, let's approach defect verification carefully.

Here's what I do to verify the bug:
  1. Make sure I'm running a build that has the fix in it. In particular when there are a number of branches this is something that needs double-checking. Rely on check-ins and build tags for this, not on bug comment time stamps.
  2. Repeat the steps that reproduced the issue and make sure the behavior is what I expected. This is the obvious part. I try the thing that broke before and see if the behavior has changed. If there's zero change in behavior (i.e., the exact same thing happens), I'm really suspicious - after all the fix attempt is likely to have at least modified the system behavior, even if the fix is complete.
  3. Make sure I got all the way through. I can't prove this bug is resolved unless I can prove I exercised the thing that used to cause the bug. A failure before we even get as far as the bug leaves me in a verification form of Shrodinger's Cat - I can't prove whether it was fixed or not!
  4. Look for markers of resolution. Often a bug fix will include a mark or a note that is a secondary way to know the bug was fixed. Usually this is in the form of a message that does appear in the log (XX complete) or that does not appear in the log (failure message "blah" does not appear). Look for the positive indicators of a fix - success message - in addition to the negative indicators of a fix - lack of prior error.
  5. Reread the bug. Think I got it? Great. I'm going to read the bug one more time, start to finish, especially with a really long bug. Maybe the behavior morphed over time, or maybe there is a reference to another problem that sometimes hides this problem. Maybe there's a reference to some documentation change that should be opened as a separate request. Once this bug is closed out, it's unlikely to be read again, so make sure you get everything out of it that you need.
Once you've done all that, only then can you mark the bug as verified or failed, depending on what you found.

No comments:

Post a Comment