Wednesday, April 7, 2010

Runaway Automation

Test automation is a great thing. I'm absolutely for test automation in general. Like any tool, though, it should be used carefully.

"Automated tests are cheap to run" is a danger phrase.

It's generally true. Compared to a single manual run of a test, a single automated run of the same test probably takes less time, even accounting for a human looking at the result. It also frees the human up to do new and exciting tests. However, even if one automated test is cheap to run, that doesn't mean it's cheap forever.

The slippery slope thought process goes something like this:
- I found a bug and they fixed it! I'll write a test to prove it.
- I can automate this test. I'll add it.
- I should really write a test to show this bug, and when it's fixed we can check it in to prove it.
- New feature! Let's see what tests we need to feel good about our abilities to refactor it.
- (Fast forward 3 or 4 years)

By now your automated test suite is suffering from Rampant Automation. One test is cheap, but you don't have one. By now you have thousands. And for that reason automated tests are no longer cheap. They probably run for hours on end, take a significant amount of time to maintain and analyze, and use several machines.

All this isn't to say that you shouldn't use automated tests. Please feel free. Just make sure that you keep an eye on the totality of your automated tests and make sure they're not getting away from you. It's up to you to decide how much time and machine time you will put into automated tests. When you get close, it's time to clean up your automated tests, find ways to speed things up, remove tests that aren't helpful, and/or evaluate whether you really do want to automate the new test(s) you're considering.

Automated tests are great. Runaway automation is not. Keep an eye out!


  1. Excellent points.

    Note also that "cheap" is a statement about the cost of (running) the test. That's orthogonal to the value of the test. That should be considered too. If checking has high value, even a high cost might be worthwhile.

    Finally, opportunity cost always lurks. No matter how cheap something is, anything you do with it--creating it, running it, evaluating it, maintaining it, explaining it--takes time away from other stuff you could be doing. Is what you're doing now more valuable than what you could be doing? How do you know? How would you know?

    ---Michael B.

  2. If I understand your point correctly it's that it's very easy to leave tests alone once written vs actually re-evaluating if their cost (human time + running costs) are actually worth their current benefit.

  3. I like the idea of allocating time and machine capacity for test automation. Say, one hour max for a nightly build. When you exceed that, it's time to throw something out or make it run faster. Also perhaps limit the number of tests or amount of code you have in your test automation suite.

    Another danger phrase for test automation is "Let's automate UI testing to save time". Building a large body of tests on top of a UI can be a mistake - it's incredibly time consuming to keep those tests passing as the UI evolves. It's better to develop an API below the UI to access the business logic an automate on top of that. An added bonus is that such an API can be useful for other things than testing as well.