Monday, December 1, 2008

Notice With Purpose

When we're testing, particularly when we're doing exploratory testing, it's a very intense level of interaction with the system. There's a lot going on with any reasonably-sized system, and we have to be watching for tiny bits of information in a sea of data. So we keep our senses on alert, watching logs,  GUIs, timing, messages in third-party systems that interact with our system....

Senses..... alert!

But noticing is only half the battle. We have to notice with purpose. Not everything that happens is a sign of a problem, so it's important to filter out things that are unimportant. For example, I can be testing a system and notice that another tester is writing to my system. Is that important? Probably not, really. So what's the workflow here?
  • Notice something. For example, someone else writing to my system. This is anything that jumps out at you - a log message, some GUI behavior, some metric.
  • Decide if you caused it. If I wrote data and then noticed that I wrote data, well, that's probably not interesting. If I was trying to cause an error to occur and it occurred, that's good to know but probably not something worth spending too much time on.
  • Decide if it's standard background tasks. Many systems have normal things they do, like log rolling, etc. That's also not likely to be interesting. Be careful here, though - a standard task at an unusual time or in an unusual way is NOT standard.
  • Write it down. It may not seem important now, but it's worth a quick note in your testing notes.
  • Pursue it if it has potential to be related. Here's the real trick. If you chase every odd thing you notice, you'll never get anywhere.  So notice, and only follow up if you know what you're going to do with the follow up. For example, if I've noticed that someone else is writing to my system, I'm not going to follow up if what I'm testing is user creation in the admin. Writing simply doesn't matter to that (at least, I'm 99.3% certain of that). I will follow up if what I'm testing is data deletes and I'm seeing odd things going on.

There's simply too much information for it all to be currently relevant. This applies across many disciplines. In exploratory testing we call it mission-based tests, or directed tests. In meetings it's often called the parking lot. In  management we call it exception reporting.

Noticing things is good.... that's how you find the subtle bugs. But noticing everything will just leave you overwhelmed; you simply can't pursue it all. So notice, decide why you noticed, and move on with or without that thing. Notice... with purpose.

No comments:

Post a Comment