Friday, December 18, 2009

Cold Systems

It's unusually cold here in Boston. Yes, it's winter in the northeast US, but still, even for that it's cold! I was standing waiting for my bus this morning - shifting from side to side - and trying to think about something other than, "I should have worn thicker socks". And it occurred to me that testing a cold system is something that I don't do enough of.

A cold system is a system that's not really doing anything. In a storage system, for example, there are no reads, writes, management modifications, etc. occurring. If a busy system is hot with activity, then a cold system is one in which very little is going on. (I know I'm stretching the metaphor here, but I had to get from socks to something else, and I went from socks to hibernation to sluggish to systems that aren't doing much - it's really interesting inside my head sometimes!)

So when we test a cold system, what are we looking for?
  • Background processes. Many systems have background cleanup procedures, indexing policies, and other things that occur at system idle. When the system is cold, it's much easier to see these run and look for more subtle patterns that might get lost in system activity. Look for slow memory leaks, timing patterns, background processes "taking over" when they're not drowned out by other things, overruns and other loop-type problems, etc.
  • Startup issues. When you first start using a system, there might be lags or errors that you can't see very well when many things start at once. Now's your chance to see them in a state much closer to isolation.
  • Spindown problems. Without activity to sustain them, certain things might spin down too far. For example, your disks might autonegotiate themselves slower (green drives do this) if they don't meet activity thresholds, and then the few requests you are getting might have slower response times - oops!

Testing a cold system is about looking for patterns that get lost when a system is busy. It's about finding the signal that gets lost in the noise. So slow down, cool your system off, and see what new things you can see.

Oh, and hopefully we'll get some snow soon to make the cold worth it!

No comments:

Post a Comment