Thursday, March 19, 2009

No Substitute for a Real Client

We testers spend a good portion of our working lives trying to anticipate users. What will they do? How much load will they throw at us? What do they expect to see?

No matter how good a guesser you are, you, the tester, are probably wrong:
  • You understand the internal workings of your product, so you know what to steer around and what to do. 
  • Your job is to think about this product all day every day. For most users, your product is probably just a tool, and is not the actual point of their day.
  • You've probably had a feature or area of the product explained to you, possibly multiple times (hey, sometimes I don't get it the first time around!). Your customer might have gone to training... maybe... once... a few years ago on an older version of the product.
So.... gloom and doom, we're not going to succeed. But we can get closer, and we can do better. We can always do better.

First and foremost, any time you get to talk to a customer or watch a customer work, do it. I got a chance to visit a customer this week, and it was really eye opening. They had all these crazy theories that I would never have thought of, and some of 'em were really good ideas.

Second, consider that each usage model is composed of several component behaviors. Maybe those behaviors are things you can (or do) anticipate and try yourself.  I've talked about this approach before - looking at behaviors and considering usage from there, rather than looking at usage and considering behaviors. I call this approach testing in the small. The net result is that you may not have considered this precise use case, but you've probably gotten through parts of it, so you're not as far off as it seems.

And last, consider what your customers are telling you indirectly. Do you get lots from client systems? We do, at least, when there's a problem. Let's look at the logs and build up a picture of the usage. How many clients are using this simultaneously? What kinds of operations are they performing? Don't have this kind of info? That's something that ought to be added, and right quick. It's like watching a movie instead of  live action, but it gets you closer.

And in the end when we're testing something, we're not the real client. But we can always be better. We can always get closer.

No comments:

Post a Comment