Wednesday, June 24, 2009

Over-Reproducing and Under-Reproducing

There's a fine line between helpful and obnoxious sometimes. Take reproducing a bug, for example. There are two exchanges you commonly hear:

Exchange 1:
QA: "I found a bug"
Dev: "Well, hmm... that shouldn't happen. And it didn't happen when I tried it. Can you reproduce it?"

Exchange 2:
QA: "I found a bug"
Dev: "Ok"
QA: "The bug still exists"
Dev: "We haven't fixed it yet"
QA: "Look! After 5 more builds the bug still exists"

These two exchanges show the opposite problem. 

In the first exchange, you have under-reproduced the bug. You didn't try it again from far enough back, and you may have missed something in your writeup. It may really be a bug, but there's more subtlety to it than you actually wrote up.

In the second exchange, you're over-reproducing the bug. If no one's tried fixing it yet, of course it's still going to be a problem. While it's true that sometimes dev will look at a bug and think it might be fixed, that's not really a common case. So what are you really accomplishing by looking for a bug that has likely not changed state? In practice, not much. You're pretty much wasting your time recovering old (and unchanged) ground, and you're being obnoxious. So don't.

It's occasionally a bit difficult to know whether you should spend time and effort reproducing a bug that hasn't been fixed yet, but in general, if your initial writeup is good (and you reproduced it then), it's probably not worth the effort unless something has changed that is likely to affect the bug.

Easy lesson for us all today: Reproduce a bug until you know how to make it happen and you understand it, then stop until something material changes.

2 comments:

  1. Hi Catherine,

    It's a very interesting and true post which you have posted.

    Its important that reproducibility of the bug should be nailed down before you flag it as 100% technically reproducible. But sometimes you find the occurrence of the bug quite random in the game/application and the dilemma is how much time should i spent on isolating or should i just mention what i did and where i found the error. This doesn't help Developer much as with the information you have won't help him reproduce the problem. So in such situation what does your experience suggest ?

    Rahul
    QC Lead Ubisoft.

    ReplyDelete
  2. Rahul, I think it's a balance. A lot of times it will depend on where you are in the release cycle, how bad the bug looks, how close you are to reproducing it, and your general relationship with development.

    In general, I would say:
    - Report early. You'll keep working on it (and say so in the bug you log!), but this way you've warned them it's coming. It's also possible that they may be able to suggest ways to track it down.
    - Ask for help. If it's a hard one to track down, go to dev and get them to speculate with you about what might be going on (or what can't possibly be involved), and maybe it'll point you in the right direction.
    - Stick it in the back of your mind. If you're really really stuck, give it a break and come back to it later.
    - Change techniques. Move to a different build, a different machine, a different analysis technique. Shake things up to give yourself a chance to reproduce the problem or at least find out more if it happens again.
    - Get some help with tracking. Maybe you can't reproduce it, but you can certainly work with dev to add logging or change logging so that if it happens again you can find out more information.

    ReplyDelete