Monday, January 10, 2011

Faint Signals in Code Change Noise

We run tests every night. They run around the lab and do upgrades, ingest data, shutdown and restart and lots of other things. We come in the next morning and analyze anything that did something unexpected. In general it works quite well.

There's an interesting phenomenon that took me a little while to notice: weekends. For the most part "nothing changes" from Friday night to Monday morning. There are exceptions, of course, but most weekends there are no changes in the lab (hardware failure or the like) or changes in the code base. So with no changes, then weekends should be very consistent, right? Friday night's test rum should look the same as Sunday night's run.

That turns out to be a good source of information. Because everything should be near identical then variance becomes really interesting. We can see performance changes or intermittent bugs because the noise caused by changing code and changing environments is much less. All software gives out faint signals and with smaller noise we can listen.

If you have the opportunity to do the same thing in the same way, take it! Your software has things to tell you, even when nothing is changing.

No comments:

Post a Comment