As we were planning the next release cycle, the QA team where I work was looking at ways we can improve our testing this time around. One of the things that came up is that we were going too deep in certain areas at first, and not hitting the breadth of the product early enough. Thus, it was a bit uncomfortably late in the test cycle before we did much peeking at some of the more obscure features. Granted, we hadn't found anything bad at the last minute, but it did feel like maybe we should make sure we touched everything a bit earlier just for comfort.
So we decided to do an extended smoke test. This isn't a huge change for us, just a restructuring of the test plan to encourage us to touch a feature, move on to another feature, and come back for a deep dive on the feature after we'd gotten some breadth.
But how long should we spend on this extended smoke test? At some point we have to go deeper, and deeper takes a while, so if we spend too long going broad we risk some obscure problem in a really commonly-used feature.
We made a couple decisions early on:
- This needed to be timeboxed.
- We were going to be making it up a bit as we went along, so the modified test plan would be a guideline, not a rule.
And then we worked our way into an estimate. First, we went around the room getting gut feels on four questions:
- What is a smoke test?
- How long should a smoke test take?
- How long do you think the whole test cycle is?
- How many hours a day do you actually spend testing, on average?
Looking at this, we had the following:
Basically, we were all agreed on what a smoke test was, and that the extended smoke test should take about 1 / 8 of the time of the entire test cycle. Further, we figured that the team was getting one man day in a calendar day of testing done (tester 4 is on another project, hence the zero).
Our actual test cycle this time is 8 weeks, so we had to fit this into 5 days.
We've split our test plan up into 15 sections, which means we need to cover 3 sections a day. Now we're getting somewhere - that we can check to see if we're on track. Some sections are bigger than others, so figure "small" sections can have 1-2 hours of smoke test, and "large" sections can have 2-3 hours of smoke test.
And that's how we got to the extended smoke test estimate.
Caveat time: This is what made sense for us, and is intended only as an example. Feel free to modify it to suit your needs.