Tuesday, April 12, 2011

Sessions for Long Tests

I've been working with a program that is primarily focused on data ingest. We send a lot of data through the system and perform tests and analysis based on what happens during and after that data ingest. One of the testers came to me recently and said, "I've been wanting to try session-based test management, but I can't figure out how to do it when our tests take so long!"

Now, I've used session-based test management techniques in the past, but it was for a very different application. So I googled it. The first four examples I found while googling "Session-based testing" and "session-based testing example" were:
  • Testing the new file and open file features of an unspecified program
  • Creating a test coverage list and feature checklist for a win32 application based on its user manual
  • Analyzing a view menu in a map application
  • Testing a bookstore web application
Shoot. Every single one of these is a GUI application that offers very quick turnaround time. Now, the GUI part isn't a big deal; that's easy to accommodate. We have other ways of getting information into and out of our system. The turnaround time is rather harder, though. You see, with all of the examples above, the time lapse between deciding what an interesting test would be, performing the test, and seeing the results is measured in seconds. It's fast.

For us, deciding to perform a test frequently results in several hours before we get the results to analyze. For example, we might want to test what happens when we load the system from the full on-disk file and run data through it when that data has certain characteristics. At a 170MB/s average read rate from disk, the step "load the system from the full on-disk file" takes about 1000 seconds (a bit over 16 minutes). Depending on the characteristics of the data we're interested in, that step can take minutes (20GB of data or so) to hours (multiple TB of data). Only then do we get to analyze our results.

So how do we structure a session around that?

Let's define what the problems really are:
  • Long wait periods (for humans) in our tests mean it's hard to focus on one test or one session at a time
  • Expensive tests mean that we can't do many of them inside a typical session window of and hour or two
  • We frequently find the most interesting information when we compare results across sessions.
Now, I can't say we've fully solved the problem. Frankly, our attempts so far have very much been trying to put a square peg in a round hole, and this may not be the technique for us. However, we're trying not to throw the baby out with the bath water. And there are some benefits to structuring what we approach and when so we don't succumb to death by analysis.

Some things we have learned:
  • Separate gathering data from analysis. We focus on setting up test runs and configuring our data one day, then let the tests run overnight. The next day we separately analyze the output from the previous night's runs, from each other's runs, and from other data we've already gathered. By making analysis an explicit later step, we free ourselves to look at a broader spectrum of data, and we shorten the feedback loop on this part of the testing. Charting and calculating on existing data is much faster than gathering it.
  • Go wide. We don't run one test at a time; we run as many as we can get hardware for. That forces us to think a little bit before we run off just poking at the system, but we get more information per unit of time spent. And if a test does something unusual, we have some near peers to compare it to. (Ask me how that helped us track down a time skew issue in the lab that had nothing to do with our software but that had been bugging us for weeks!)
So are we doing session-based testing? Nope, not in the strict sense. The separation of design and analysis sort of kills that. But it works for us, and it works for the system we work on.

No comments:

Post a Comment