Friday, January 22, 2010

Where Is the Line?

Take the following situation.

The product under test is a software library. We have also written a wrapper program that implements the library and provides some helper functions (such as the ability to run it from the command line, and some lightweight reporting). Lastly, we have some automated tests we have written that exercise the library directly as well as through the wrapper program.

We find a bug in the wrapper program. Here's the question of the day:

Should QA just fix it, or not?

Let's posit the following:
  • having the same person write and test code is generally to be avoided; two minds and viewpoints will find more of the problems than one alone
  • QA has the coding and technical skills to fix the problem
  • QA has source code access and can actually check in a fix (this is a useful part of fixing something!)
  • having a tester work in the code "taints" that code for that tester and means someone else needs to start
There are some arguments in favor of fixing it:
  • It reduces turnaround time and gets rid of the problem quickly.
  • The bug isn't strictly in the product, so we're not "tainting" the tester for that product. That is, the tester isn't now testing their own code.
  • In some sense, the wrapper script is test code, and that's completely okay for QA to fix (heck, we work with it constantly!).
There are some arguments against fixing it as well, though:
  • We have a wrapper script distinct from test code for a reason. In our case, the wrapper script is intended to be sample code as well, and that makes it product code.
  • We found a bug. Cool. Now our job is to report it and move on with more testing (that thing we do best).
  • Since we believe that two pairs of eyes should be on each piece of code, we'll need to bring in another tester for this piece of code.
I'm really on the fence with this one. In general, I try to take the middle road and get another set of eyes on the fix. This can be done by pairing with a developer to fix it, or by submitting a patch for review and inclusion. Whatever method you choose, before you go do that quick bug fix, be cognizant of what you're doing.

3 comments:

  1. Is the wrapper program part of the product under test?

    If so, then don't fix it.

    If not (and the assumption is that the wrapper is just part of your test artifcats), then fix it, assuming you know how to do so.

    ReplyDelete
  2. One huge pro for fixing it yourself is you are independent of the development team. As a test manager, my preference was to maintain our own test scripts so we didn't have to try and negotiate through development schedules and timelines. This shouldn't be the sole reason for basing a decision, but it is definitely a factor.

    ReplyDelete
  3. In this case, those developers fix "test code" so it's not clear to me that there's a border from the other side.

    Also, fixing it might be easier it than deciding how it should behave.

    In this case, I clearly think that fixing it was the right thing to do. And having other eyes on it was also a good idea.

    ReplyDelete