If a tree falls in the forest and no one's there to hear it, does it make a sound?
Or to put it in a more tester-oriented way:
If a bug exists and no one finds it, do we care?
My testing team has been together a while now, and we've settled into a lot of patterns: how we work with the automated test suites, how we approach testing in general, how we verify bugs, etc. Now it's time to start thinking about the impact of our work.
When we think about impact, there are several types of bugs:
- Bugs we find that also hit (or would hit) the field
- Bugs we don't find that hit the field
- Bugs we find that no hits in the field
- Bugs we don't find that no one hits in the field
The first of these - bugs we find that would be found in the field - is pretty much the normal scenario. We found something (and hopefully got it fixed) before we shipped it and before it affected customers.
The second of these - bugs that we don't find that are found in the field - are leaked defects.
The last two are sort of interesting. If a bug isn't going to be found in the field, what value is there in finding it internally? And who cares if you miss them?
On the one hand, a bug that won't affect your customers doesn't matter. If they're not going to hit it, well, who cares? On the other hand, just because your customers haven't hit it doesn't mean they won't hit it in the future.
The more subtle thing is what you can learn about the usage model by what your customers find. If you find a bug that, for example, a WebDAV interface doesn't work at all, and none of your customers find that bug, you can guess they don't use the WebDAV interface. At least, not currently. That's useful information.
Personally, I'm a data junkie. I'd rather know, pretty much always, even if the bugs aren't found by customers. That being said, if I know customers don't use a feature, there's value in testing that later or cutting corners there rather than in your most used feature. I think of it as a way of setting priorities: first I'm going to do the real basics; then I'm going to do the things my customers do; then I'm going to hit the other stuff.
It seems I do care if a bug exists and no one finds it! I suppose that means that a tree falling in the forest might actually be darn noisy, too.