You know how many bugs you find. You know how severe they are. You know how frequent they are. You know the ones that cause people to say, "oh, okay" and the ones that cause people to say, "Oh boy, am I glad that one didn't get out to the world!"
But you don't know much about the bugs you missed. Maybe you missed 10, maybe 1000. Maybe the ones you missed will be seen in the field, or maybe not. Maybe they're high severity, maybe they're low severity. That's a lot of maybes and a big black hole in your knowledge of testing the system.
Enter defect injection.
Defect injection is a technique designed to see how effective testers are and what kinds of things they find and they miss. Here's how it works:
- Go find your friendly local development team
- Ask them politely to insert 10-100 bugs throughout the code.
- Test as normal
- Go back to your friendly local development team and show them what you found
- Compare lists
- Ask your friendly local development team to fix all the bugs they inserted before shipping, please.
Basically, this is a test of your defect finding skills. Given a known list of bugs, it's possible to make statements about the bugs you missed. It starts to become possible to see patterns. For example, maybe you miss bugs in a given module, or maybe you miss race conditions because your tests tend to be really short.
If you have a willing development team, and you'd like to know what you find and what you don't in your particular situation, consider using defect injection. Just don't forget to remove the injected defects.... before you ship!