Thursday, July 30, 2009

A Different Test

Sometimes the test you do.... is not the test you intended to do.

Case in point:
Today I went to do a simple upgrade test. On a given platform in a given configuration, upgrade from the most recent released version to the new version that's coming out. Simple.

I completely forgot that we'd tested a "power event" on that system earlier this morning. (Note: for "power event" read "a tester went in and pulled the plug on the power strip for that entire system"). The system handled the "power event" just fine, but I hadn't yet gone in and completed the products startup validation process.

So my system was not in a good state when it started the upgrade. And the upgrade failed exactly like it was supposed to.

Not exactly the test I intended to do, but it was a perfectly valid test. So what's the moral of the story?

If you're attempting at test, and it does something unexpected, take a quick look. You may have done a different test.

Now off to do the test I meant to do originally!


  1. I can't tell you how many times I've found bugs that way. I sometimes wish there was an equivalent of a fuzzing tool for machine states. Oh, wait, there's Holodeck, I just can't afford it...

  2. There are also various tricks we can use to fuzz things in homegrown test utilities. They're not necessarily the fanciest things in the world, but you can do a lot of basic tests with a random generator and a list of plausible values.