My ultimate goal is predictable releases: predictable quality, predictable timelines, predictable experiences. I don't want a perfect release (I'd be holding my breath a long time for that one!); I want incremental improvement and I want all the customers - dev, marketing, support, our clients - to feel safe with releases. Basically: "No surprises, please!"
Big bugs found in the release cycle - whether real or not - are surprises. We can (and should!) work to avoid them, but in the meantime, let's plan for them so they're not as surprising.
How do we handle these release scares?
- Allow some time for them to happen. Yes, pad your schedule a bit.
- Create a reporting structure that allows for rapid handling of bugs late in the process. Standups, automatic reporting (can your defect tracking system send emails?), and proactively talking to affected groups can make handling these things a lot easier and - yes - more reliable
How do we avoid these release scares?
- Test early. The earlier we test, the earlier we will find the major issues that we presume are there.
- Measure and improve coverage. This is a whole other blog post that I haven't thought through well enough yet.
- Release carefully. Don't put your release on your biggest, weirdest customer first. Put it on the customer site that has usage that closely matches your test usage, and on the customer that doesn't regularly overload the system. You want a "friendly use case" first. Once you've seen it work there you can start expanding who gets it.
Release scares will happen. If we can't prevent them, we can at least handle them. And that's better than constantly being surprised.