The best way I know to figure that out is to look at our defect leakage rate.
What Is It?
Defect leakage is the number of bugs that are found in the field that were not found internally. There are a few ways to express this:
- total number of leaked defects (a simple count)
- defects per customer: number of leaked defects divided by number of customers running that release
- % found in the field: number of leaked defects divided by number of total defects found in that release
In theory, this can be measured at any stage - number of defects leaked from dev into QA, number leaked from QA into beta certification, etc. I've mostly used it for customers in the field, though.
What Is the Goal?
The goal of all of the defect leakage metrics is to identify what kinds of things our internal development and testing is missing. By looking at the rate we can see if we're getting closer to anticipating our customer's usage and environment, or farther.
Why Do I Like It?
Defect leakage is one of my (very few) pet metrics. I really like that it's a directly customer-affecting measurement. If a customer hits a bug we've never seen before we have to do the diagnosis, resolution, and workaround/fix deployment all on the customer's time. If a customer hits a bug we've seen before, even if it's not fixed, it's generally a shorter and less painful process to get a a workaround or a fix deployed at their site. I also like that it's a measurement of engineering as a whole, not just of QA. Lastly, measuring defect leakage requires you to really consider every customer incident, and that kind of introspection helps me see patterns in problems. It really points out where our coverage is lacking and, conversely, where our product is pretty good.
As with any metric, measuring defect leakage is not perfect. It certainly doesn't apply to all situations, and there are some flaws to be aware of.
- It's kind of a pain to report on. For each defect found in the field, you have to figure out if you found it internally at any point in the development process, and somehow keep a report of all this. It's not generally as simple as running a query against your defect tracking system. If you're low volume, this isn't a big problem. At higher volumes it gets a lot harder.
- It's not useful as a measurement of QA alone. To keep your defect leakage rate low, you have to have development, QA, product management, and support all working together. This is actually one of the things I like about it, but it makes it a bad measure of QA alone. If, for example, product management incorrectly guesses how customers will use the product, your leakage rate can go up even if QA's effectiveness is unchanged.