Monday, February 14, 2011

What.... and What Not

I've been working on a project with a group recently, and part of it involves writing a test strategy. As we were working on this, we started discussing the things we would do:

I would do some unit testing because XYZ. I would do risk identification using method MNO.

As interesting as what we would do is what we considered and decided we would not do:
I wouldn't bother stress testing because ABC.

It's become clear that for any list of things we would do, there are actually three separate lists:
  • the things we would do
  • the things we would NOT do
  • the things we hadn't considered
If we write down only the things that we would do, then we can't distinguish between ideas we considered and rejected - things we would not do - and ideas we simply didn't think of. Sometimes we really did think of everything, but that's pretty rare. Better to include a brief summary of the things we rejected as well as a list of things we will do. That allows us to explain our reasoning, making our assumptions visible for validation. It also allows others to suggest things that we didn't think of.

When you're making a list of what you will do, make a list of what you will not do as well. It'll help you make sure your lists are more complete.


  1. This is a great idea! I always try to consider things that "no user would ever do" when writing test plans - but I love the idea of putting it out there and discussing it and then pushing further on to things we hadn't considered. You never know what will come out of a good brainstorming session! Great post!

  2. Great post... I've been doing something similar on my project, but in my test results: I record what I tested and what the results were, but also what I didn't test and why (out of time, not relevant, no test data, etc.). I love the idea of applying this to strategy, because it might trigger an idea from somebody else before or during testing ("wait, we didn't address this..." "wait, we SHOULD test that because...").

  3. Especially the things you won´t test (and WHY you don´t testt them) are -to my experience- important to list in a test plan/ strategy. This helps to avoid misunderstandings like "I thought this topic was covered by 'xyz' of the will-test-list".