Thursday, July 17, 2008

Don't Fear the Checklist

I was at CAST 2008 earlier this week and was listening to Cem Kaner's presentation about checklists and their adaptation to software testing. He made several points, and I found myself nodding my head and agreeing (I'm paraphrasing, for the record!):
  • Checklists describe the goal, not the steps.
  • Checklists provide room for people to think; scripts deny testers the flexibility to see the whole picture.
  • Scripts only check for what we thought to explicitly verify; checklists allow you to adapt your verification to what you're seeing in the system.
Amen, brother! I've hired thinking testers. Let's let 'em think!




Checklists are scary!

Wait a minute. Checklists are tools that allow us to hint to testers and increase coverage while providing the freedom for them to think about and dive into a system. That's good!

The problem with checklists is twofold: (1) how do you know when you hit a specific test case?; and (2) How do you document what you actually tested? I'm not going to talk about the second of these; that's a whole separate blog post. But how do you make sure you hit a specific test case?

Let's step back for a minute and talk about why we might want to be certain we hit a specific test case. I can think of two things:
  1. We had a bug here and we need to be sure we haven't reintroduced it.
  2. It's a particularly sensitive action/configuration/setting and we don't want to go to our biggest (or most important, or loudest) customer without being very sure that this will work.
Our checklist doesn't cover that. Our checklist is goal-oriented, not configuration or specific step-oriented. And that's why checklists are scary.

Now, what can we do about it?

There are a lot of options, ranging from "hope you hit it every time" to "forget checklists! scripts are safer!". Instead, we simply acknowledge the realities of the situation. These specific things are important, so we need to write them down in the test plan. They don't have to be in our checklists; they can be elsewhere. They just need to be there. And then we need to run these specific things. They can be manual or automated, as long as they get run.

So go ahead, use checklists where you want testers to actually dig in and think, and use scripts where specificity really matters. Take the benefits of your checklists and eliminate the areas that are scary by scripting them away. After all...

Checklists aren't scary!

No comments:

Post a Comment