Wednesday, August 13, 2008

Independent Verification

Sometimes I get into a situation where we see a behavior in the logs that indicate a client writing to us is doing things a certain way. Now, the first thing you have to know is that we log every request as it comes into the system. We write down the NFS operation, the volume, and the filehandle (you don't have to know what these things are, exactly - basically what the client wants us to do, the location of the file, and the name of the file to do whatever-it-is on) - and then we process it. We use these logs a lot when debugging and often find things that are interesting. For example, we may find that a client is doing three setattr operations when most clients do one. Or we may find that the client "just doing writes" immediately reads back everything.

This is great internally, and it's really helpful with client profiling. Sometimes it even finds a problem on the client side - consistently trying to create files before creating their containing directories, for example.

And there's the rub. Everyone's happy to believe our logs when they point out things in our system. When the problem is on the system that is a client to us, all of a sudden we find that our logs are insufficient. The support team for the client software doesn't believe that they're not manipulated in some way (don't get me started on politics!). I can't really blame them; they don't know our system. So to provide independent verification, we turn to WireShark. A quick trace shows our logs are accurate, and then we move on to solving the real problem.

The moral of the story is:

Just because you know something doesn't mean everyone knows it. Be prepared to provide some form of independent verification.

No comments:

Post a Comment