Monday, September 15, 2008

Knowing Makes It Okay

Working with beta clients is a tricky business.

On the one hand, I don't like betas:
  • The software isn't ready for clients yet; that's why it's not released!
  • Beta clients have expectations that their feedback will be incorporated in that release - which may or may not be feasible or necessary depending on the feedback.
  • It's really hard to have a beta client actually follow the beta test plan (even if they wrote it!), so you can't be sure how much coverage you really got.
  • Poorly run beta clients are basically like asking support to look at an early release - scattershot and unfocused, and therefore of diminished value.
On the other hand, I love betas:
  • Client environments give us improved test coverage.
  • Clients who want early access tend to be very willing to work with support/QA/etc throughout the process.
  • Better to find out if something's really wrong before we ship, rather than after! At least it limits the damages to only beta customers.
A lot of the value of the beta comes in how it's run. First and foremost, this is not totally in your control. You need buy-in from your client, your account rep (or whoever is performing in that function), and also QA, development, product management, and the release team.

The second most important thing you can do to make a beta go well is to know the system. Having bugs is fine, even expected. If you tell the client about those bugs, it is far better than if the client tells you about those bugs. Imagine these two conversations:

Conversation 1: Knowledge is Power
You: "Here's your software, and here's the list of known issues. Let us know what you see!"
Client: "Thanks."
Client: "Gosh, I saw some of these, and that bug 14 you mentioned is really a doozy. It's getting fixed, right?"
You: "Yup. We're working on it and it'll be fixed before we ship. We thought it was bad, too."

Conversation 2: Going in Blind
You: "Here's your software, and here are the issues we've found so far. Let us know what you see!"
Client: "Thanks."
Client: "Here's MY list of bugs. You guys didn't find any of these! Look at my #14 - it's ridiculous! How could you not have found it? What are ya'll doing over there?"
You: "We'll take a look."

At this point, your client has lost confidence not in the software itself, but in your company's ability to produce good software. You've shown you can't find major problems, and now the client can easily go down the path of wondering what else you might have missed. Loss of confidence ensues.

It's hard, going into a beta, to mention all the problems you've found, but it puts the client on your side. The client will find problems - probably including some you didn't find - and that's okay. As long as the client knows about what you're finding, and you're in this together, the bugs won't reflect badly on you or on your development process. You don't have to be perfect, just forthright.

Should you choose to do a beta, the first thing I would say is that there will be problems, but knowing makes it okay. Once both you and your client understand that, you can do the beta and make your software stronger.

No comments:

Post a Comment