Monday, April 14, 2008

Magic Numbers

One of the more common test techniques is boundary value analysis. Basically, you find the edges of allowable input and test each of the boundaries and also off by one values. For example, if you were testing a text field that allowed you to enter valid months, you would test:
  • 0 (invalid but off by 1)
  • 1 (lowest allowable value)
  • 12 (highest allowable value)
  • 13 (one above the highest allowable value)
That's all well and good, and there are lots of techniques for determining your boundaries when you have various inputs.

Defining Magic Numbers
I assert, however, that inside your application are other boundaries, often hidden from the user. These are what I think of as the magic numbers in your application. Some of these magic numbers are imposed by external forces (time is a big one, for example). Others are contained within your application. Magic numbers are the values around which your application is sensitive.

Finding Magic Numbers
Finding some magic numbers can be very easy, but others are deeply hidden. To find magic numbers, consider the ways your application can be sensitive. Common areas of sensitivity are:
  • Time. Background processes - cleanup, auditing, notifications, etc - are huge culprits here. Think of things that happen every n hours, minutes, days.
  • Size. This is usually a scalability control in place. Common examples are file-system or database areas - log rolling, indexes on tables over size n, creating a new directory on the file system when an older directory has more than n files in it.
  • Frequency. This one is a little less obvious, and is sometimes lower-level than most of your application - garbage collection, background processes triggered every 5 messages your system handles, etc.
Universal Magic Numbers
Some magic numbers are outside your application and apply to almost anything. These are units of time. Your application probably won't be sensitive to all of these, but it will very likely be sensitive to some of them:
  • Leap Years. February 29th is a common breaking point.
  • Daylight Savings Time. Switching on or off daylight savings time is a common place where things break; skipped or repeated jobs are likely.
  • Time Zones. Look for something to be off by your distance from GMT (or UTC). I'm GMT-4, currently, so I look for times that are 4 hours in the future.
  • Max Directory Sizes. This will vary by OS, but try looking to see if we're using a file system directory generator that rolls on you.
  • Max File Sizes. This will vary by OS also, but look to see if the OS is rolling files on you.
  • Min File Sizes. This one is a lot more rare, but watch that you don't get issues with an empty file. Occasionally you'll find they handle differently.
  • Every n Ticks. This one is really rare, but it's worth looking at. Check for things that happen every n clock cycles. When the CPU gets really busy these may start to vary a little bit.

Application-Specific Magic Numbers
In addition to the external forces acting on your application, there are likely internal magic numbers. These are the heartbeats of your application, and if you find them, you can stress them. These will vary by application, but look for some of these:
  • Log rolling. Be sure to check this as you're changing log levels.
  • Background processes. Cleanup, notification, and auditing are common ones.
  • Heartbeats.
  • Areas that interact with the file system. Message processing, FTP handling, etc.
  • Timeouts. Don't forget to include the timeouts of other programs you might be using - SSH, etc.
Now What?
Once you've found your magic numbers, you can apply boundary value analysis techniques to stress them. That's not always easy, but it's a good source of some really nasty and subtle bugs.

Happy hunting!

No comments:

Post a Comment