Thursday, September 17, 2009

Extended Smoke Test Estimation

As we were planning the next release cycle, the QA team where I work was looking at ways we can improve our testing this time around. One of the things that came up is that we were going too deep in certain areas at first, and not hitting the breadth of the product early enough. Thus, it was a bit uncomfortably late in the test cycle before we did much peeking at some of the more obscure features. Granted, we hadn't found anything bad at the last minute, but it did feel like maybe we should make sure we touched everything a bit earlier just for comfort.

So we decided to do an extended smoke test. This isn't a huge change for us, just a restructuring of the test plan to encourage us to touch a feature, move on to another feature, and come back for a deep dive on the feature after we'd gotten some breadth.

But how long should we spend on this extended smoke test? At some point we have to go deeper, and deeper takes a while, so if we spend too long going broad we risk some obscure problem in a really commonly-used feature.

We made a couple decisions early on:
  • This needed to be timeboxed.
  • We were going to be making it up a bit as we went along, so the modified test plan would be a guideline, not a rule.
And then we worked our way into an estimate. First, we went around the room getting gut feels on four questions:
  • What is a smoke test?
  • How long should a smoke test take?
  • How long do you think the whole test cycle is?
  • How many hours a day do you actually spend testing, on average?
Looking at this, we had the following:

Basically, we were all agreed on what a smoke test was, and that the extended smoke test should take about 1 / 8 of the time of the entire test cycle. Further, we figured that the team was getting one man day in a calendar day of testing done (tester 4 is on another project, hence the zero).

Our actual test cycle this time is 8 weeks, so we had to fit this into 5 days.

We've split our test plan up into 15 sections, which means we need to cover 3 sections a day. Now we're getting somewhere - that we can check to see if we're on track. Some sections are bigger than others, so figure "small" sections can have 1-2 hours of smoke test, and "large" sections can have 2-3 hours of smoke test.

And that's how we got to the extended smoke test estimate.

Caveat time: This is what made sense for us, and is intended only as an example. Feel free to modify it to suit your needs.


  1. Isn't "extended smoke test" an oxymoron?

    I thought the goal of a smoske test was to have a short list of "sanity check" -- simply to see if the most basic functionality is still intact in the build -- 30 minutes' work, not 8 hours of focused testing.

  2. Joe: I agree that a smoke test is basically a sanity check to make sure nothing major is broken. I hesitate to timebox it, though - smoke testing a basic corporate website would take me a lot less time than smoke testing the space shuttle!

    What we're actually doing in this case is a bit more than a smoke test, so we happen to call it an "extended smoke test". At first we tried calling it "initial pass" or "first pass", but that was confusing, and extended smoke test is something everyone in our group understands. I really don't think the particular name matters too much, as long as everyone has a common understanding of it.

    (What name would you suggest?)

  3. It seems to me that you're talking about degrees or levels of coverage. In the Rapid Software Testing course, we talk about three levels:

    a) Smoke and sanity; does this product even work well enough to be tested? We sometimes call this Level 1 coverage.

    b) The common, the core, and the critical; can this product handle the stuff it must handle? That is, can it work? We sometimes call this Level 2 coverage.

    c) The complex, the challenging, the harsh, the extreme, the exceptional. That is, will it work? Naturally, that question is not definitive; we'll never know for sure that it will work, no matter what gets thrown at it. But we can push it hard and increase confidence that it will handle the complex and messy world outside. We call this Level 3 coverage.

    What you're describing sounds like something more than Level 1 and less than Level 2 coverage, so we'd call it Level 1+, or Level 2- coverage. But I don't see anything intrinsically wrong with calling it an extended smoke test. People should call stuff by whatever name makes sense for them.

    ---Michael B.