Thursday, March 11, 2010

Situation vs Conundrum

There are two relatively common testing scenarios: situation-based, and conundrum-based.

Situation-based scenarios are those where you create a situation and see what happens. This is often what we think of when we say we're testing something. We decide to log in with spaces in the name and see what the system does (appropriate error message, let us in, blow up dramatically, etc). We create a degraded hardware scenario (translation: yank out a hard drive) and see how long it takes the system to send a notification while we induce heavy load. All of these are scenarios where we started with the situation, and we are looking to see what might happen.

To be effective in a situation-based scenario, a tester needs certain skills:
  • Creativity in imagining things that might happen (coming up with scenarios)
  • Ability to analyze a system to identify interesting actions and potential interactions
  • Ability to closely observe a system and identify resultant behaviors, even when they're not directly visible
Conundrum-based scenarios are those where something happened, and now we get to figure out what on earth went on. When you get a problem in from the field, or a bug reported to you, you're looking at a conundrum-based scenario. You have a user who was "just writing, like normal", and the system crashed. Or you have an error message in your logs that development assures you "shouldn't happen", and you don't know how to get there. In conundrum-based scenarios like this, you are given the result, and you need to describe the situation.

To be effective in a conundrum-based scenario, a tester needs a different set of skills:
  • Ability to analyze a single state and identify relevant facts (e.g., the core file, not the log file name)
  • Grep and other analysis skills
  • A mental model of the system to identify operations and other actions
  • Creativity in scenario imagination
Working with situations and working with conundrums requires complementary but slightly different skills. When working on a problem, figure out which one you're facing. You'll be far more effective if you know what skills to apply.


  1. Catherine,

    As usual, great post. It's impressive that (a) you create so many posts and (b) it's not quantity over quality - they're insightful.

    A couple quick reactions:

    1) You wrote: "To be effective in a situation-based scenario, a tester needs certain skills: Creativity in imagining things that might happen (coming up with scenarios)"

    Agreed, but there is still hope for testers who are relatively unimaginative or relatively inexperienced. Seek checklists and war stories from others. I came across Elisabeth Hendrickson's 2-page "cheat sheet" again today. It's at: of testing ideas today, borrowed those for some tests and found a bug in our Hexawise test case generating application. (It was her imagination and experience that enabled me to find it, not mine). Checklists FTW.

    2) With respect to effectiveness in the conundrum-based scenario, another good skill to have is the ability to ask the right questions. Apropos of nothing, isn't "conundrum" just a fun word?

    Thanks for blogging away at your breakneck pace.

  2. Justin, you totally caught me. This entire post came out of having a grand time with the word "conundrum". (My team had been presented with a fun one earlier in the week.)

    And checklists can certainly help those of us - like me - who are not naturally creative. I think there's a different kind of creativity for a tester than for, say, a painter. Thanks for the cheat sheet link, too. That's a good one.