"Word Game" Analysis

You know that word game where you change one letter at a time to turn a word into another word? Each middle step must also be a word, and you can only change one thing at a time.

Like this:

HATE

HAVE

HOVE

LOVE

Turns hate into love (yeah, I know, it's a cliche, but it was an easy one).

When you're trying to track down a problem and your initial analysis is getting you nowhere, it's time to start eliminating variables. To do this in a systematic manner, try the word game. How do you get from their config to your config? One change at a time... oh, and each one must be valid (a "word" in our analogy).

For example, dev and QA were seeing very different results on the same test. So how do we get from dev's environment to QA's? Let's lay out our "word" from dev to QA:

Version       Fullness    Type             Data                Encoding     Size

HEAD          0%             from code    10 GB file A    M                 4 node

.

.

.

.

.

4.2               75%             from CD     35 GB file B      E                 16 node

Now we have a map. Instead of randomly trying configurations, we're going to walk from dev to QA, changing one thing at a time and seeing what the results of our tests are. It's not about trying every configuration. It's about giving yourself a structure so you can hone in on a solution with as few different steps as possible. It's a way to look for progress and identify areas that are likely to cause a problem.

Let's look at our "word" again, this time with test results:

Version     Fullness     Type            Data                 Encoding     Size              Result
HEAD       0%               from code   10 GB file A     M                4 node          15
4.2             0%               from code   10 GB file A     M                4 node          15

4.2             0%               from code   10 GB file A     E                 7 node           13

4.2             0%               from code   35 GB file B     E                 7 node           12.5

4.2             0%               from CD      35 GB file B     E                 7 node           8  
4.2             75%              from CD     35 GB file B      E                 7 node           7.2
4.2             75%              from CD     35 GB file B      E                16 node          7.2

Looking at our output, we now start to get an idea of what areas are actually making a difference in performance. In our example, it turned out that installing from CD gave you a bit of a different configuration than running from code. Fix the CD installer (the code was correct) and the results for CD-based configurations started matching the code-based configurations.

Use this word game technique when you're facing a lot of variables and no real indications as to what is important and what is not. You'll never have enough time to run all the tests you can think of, so get some structure and start narrowing down the list of possible problem variables.

* For the record, the results I used in the example are made up. The example itself is real, though.