Very true. I know I've been guilty of the "I don't...

2009-07-06T20:58:21.914-04:00

Very true. I know I've been guilty of the "I don't know what to do. Let's reproduce it again" trap. If we can't say what's changed, well, we haven't really defined our hypothesis and our experiment well enough.

The only point I would dispute is the one about variable control. While it's a laudable goal, depending on how much the system that you're working with is under your control, you may get close (or not so close) to that level of control. Sometimes you simply have to deal with several variables changing at once, even though we'd like to be able to change just one.

I've also run into this cycle a few times (no coin...

2009-07-06T20:30:16.447-04:00

I've also run into this cycle a few times (no coincidence, since we work at the same place!). Here's something I do to stop unproductive circling in it's tracks (it's sort of an extension of "Explain why"):

1. Generate several experimental hypotheses up front, and rule out the ones that are already contradicted by the limited data you already have. Keep thinking things through until you you have a few mutually exclusive hypotheses that you can neither confirm nor reject with the current data. Hopefully some of them are actually plausible; it would help to know which are more plausible than others.

2. Figure out what observations you'd need to make in order to distinguish one hypothesis from the other. Will "reproducing" the issue and getting more logs give you the observational data you need, or are you just stalling? If another experiment won't give you the data you need, why bother running it? Maybe you need to find other, possibly indirect and imprecise measurements that are only roughly correlated with what you're trying to measure (e.g., I can't measure how fast the system is, but I can tell how often the user takes a coffee break).

3. Design the experiment and make sure you're prepared to make the observations you need.

4. Given what you *think* you know about the system, make predictions about what *should* happen. In other words, surface your assumptions early on. That way, when you look at the data later and notice one funny datapoint ("eh? where did that extra latency between requests 55923 and 55924 come from?"), it will catch your attention and let you either generate more hypotheses or design a more controlled experiment.

5. Carefully control your variables and run the experiment.

6. Look at the data -- have you figured out the issue yet?

7. Iterate. Even if you didn't nail the issue on the first try, the progress you're making will be obvious and will help to limit frustration.

The Scientific Method -- ain't it great? More often than not, you'll get to the root of the problem in only one or two iterations.

Comments on Abakas: AGAIN!?

Very true. I know I've been guilty of the "I don't...

I've also run into this cycle a few times (no coin...