Monday, September 27, 2010

Defect Injection

So you wanna know how good you are.

You know how many bugs you find. You know how severe they are. You know how frequent they are. You know the ones that cause people to say, "oh, okay" and the ones that cause people to say, "Oh boy, am I glad that one didn't get out to the world!"

But you don't know much about the bugs you missed. Maybe you missed 10, maybe 1000. Maybe the ones you missed will be seen in the field, or maybe not. Maybe they're high severity, maybe they're low severity. That's a lot of maybes and a big black hole in your knowledge of testing the system.

Enter defect injection.

Defect injection is a technique designed to see how effective testers are and what kinds of things they find and they miss. Here's how it works:
  1. Go find your friendly local development team
  2. Ask them politely to insert 10-100 bugs throughout the code.
  3. Test as normal
  4. Go back to your friendly local development team and show them what you found
  5. Compare lists
  6. Ask your friendly local development team to fix all the bugs they inserted before shipping, please.
Basically, this is a test of your defect finding skills. Given a known list of bugs, it's possible to make statements about the bugs you missed. It starts to become possible to see patterns. For example, maybe you miss bugs in a given module, or maybe you miss race conditions because your tests tend to be really short.

If you have a willing development team, and you'd like to know what you find and what you don't in your particular situation, consider using defect injection. Just don't forget to remove the injected defects.... before you ship!


  1. I wish this had been longer. In principle, I think putting bugs into software deliberately is always a mistake.

    But I might change my mind to "almost always".

  2. Chris, step 6 - "undo the defects" - is very important :)

    What more would you want to know?

  3. I don't agree that this is actually a good way to measure.

    Deliberate bugs tend to be very different in their nature than real bugs; are they easier or harder than real bugs to find then?

    Does step 6 mean that they fix the bugs without consequences, or do you have to test again?

    Does finding all the bug mean that you are a good tester, that all bugs are removed or that the devlopers made bugs too easy to find.

    I can see it work as inspiring the tester and also as a training exercise among developers, to understand the nature of a true bug.

  4. The few times I've done things like this, it always has unexpected consequences. For example, the bug introduced deliberately actually causes more problems than anticipated. Or testers find significant bugs that have no relation to the bugs introduced.

    It just seems to me that software has enough potential bugs as it exists, there just doesn't seem any need to me to add more on purpose.

  5. It's absolutely risky. It's still something to know is available, even if you choose not to use it.

    A few ways to use it with less risk:
    1. do this whole thing on a branch you then throw away
    2. do it on a similar sample application

    Oh, and you're right, Chris. Sometimes testers find unrelated bugs, but if knowing you're being measured makes you look deeper, well that's an interesting thing to know. I'm not sure that's actually a good thing.