Friday, June 19, 2009

Verifying a Bug

You found a bug, someone fixed the bug, and now it's sitting back here in your queue awaiting verification. A build containing the fix has rolled off the build system and you're all configured. Awesome! It's verify time...

One of a few things is going to happen here:

It all works great, and nothing else broke.
This is ideal. It's well and truly fixed, and your product is now just a bit better. Great. Mark the bug as verified and move on.

It works great, but something else broke.
This happens to the best of us. Sometimes the fix itself broke something else, and sometimes fixing this bug merely exposed one that had been hidden before. For example, before you couldn't collect logs at all, and now you can (fixed!), but it collects the wrong thing (whoops!). Either way, you're not quite there yet. However, the bug as reported is fixed. So mark the original bug as verified and open a new bug for the new (or newly exposed) issue. You get bonus points for linking the two issues, or at least putting a comment in both bugs that fixing defect X resulted in logging defect Y. Don't go reopening the bug; the behavior of the system has changed, and your defect tracking system should reflect that.

It's useful to keep track of when this occurs, because a true fix should be more complete. Particularly if fixes are actually breaking other things, that points to brittle code that could probably use some cleanup, or to a developer who maybe isn't as careful as he should be. Both of those are problems you want to find sooner rather than later.

It does the same thing it did before.
This also happens, but it's more unusual than a fix with unintended consequences. In this case, the first thing to check is that you really have the fix. Did you take a build from the right branch, and are you sure it contains the fix? After all, for someone to attempt a change and have exactly zero effect on the behavior of the system is rather unusual.

Assuming you do have the right build, take it back to the developer. Don't just bounce the bug if you can avoid it; this is a time for conversation. You may find that the developer didn't understand your initial bug report and made a different fix, or that there's a difference between the developer's setup and yours that's causing the problem to recur, or that a later checkin accidentally reverted the change. After you've talked the bug will probably go back to the developer, but it's worth the conversation first.

It does something different but still not right.
This falls into the "well, we tried" category. The behavior changed, so you know you're looking at a change, but it's just not there yet. This is very similar to the above category. Unless you're working with a truly disengaged developer, it's not too likely that they just didn't care enough to finish the fix. Generally any engineer worth their salt is going to at least run through your steps on his environment and make sure that works. It's more likely that you're looking at another issue, or a conflicting checkin, or a vague writeup in the first place (did you really put ALL the steps in to reproduce it?), or a difference in environment. Here, too, have the conversation before you just go playing ping pong with bugs.

Note that there's also a special case of defects found in automated tests, but there's a bit of a different heuristic for that.

Regardless of what happens, when you verify a bug, it's useful to consider both the system behavior and also the nature of the change in system behavior.

No comments:

Post a Comment