Cost of Running Automated Tests

One of the cliches of testing is that developing automated tests is expensive, but running them is free, or nearly so. That all sounds great, but doing anything eventually costs money. So, how free is it really?

The example test I use is the cost of running one of our weekly tests. This is an automated test that gets run once a week and confirms that with 7 servers we can write a lot of data and track the rate. Let's look at the cost of actually running this thing. I should note that all costs are general and based on Boston averages.

Test Duration:    3 days (72 hours)

Machines Used: 51 (50 servers and one client)

Total Machine Hours: 3672

Test Run Frequency:   once a week (52 times a year)

Machine Lifespan: 3 years

Buying Machines

A server costs about $3000 and lasts for 3 years. This server spends about 3 days a week running this test. It spends about 3 days a week running other tests. It spends about 1 day a week idle (not continuously, of course!). So, the cost of this particular test is about 40% of the cost of the server. We can say that this test will run once a week for the entire life of the server. So our cost per test run is as follows:

$3000 * 40% = $1200 (cost of the server for running this test)

$1200 / 3 (years) / 52 (times per year) = $7.69

So it costs $7.69 per test run per server to run this test.

Cost of Infrastructure

Servers use a lot of electricity. There's also additional electricity for cooling, etc. Then there's the cost of use of the network, etc. Our infrastructure costs us about $13.33 per server per month for power, cooling, lights, it's share of infrastructure (switches and whatnot), etc.

$13.33 * 40% = $5.33 (cost of the electricity to run this test)

$5.33 / 4 (times per month) = $1.33

So it costs $1.33 per test run per server to power/cool/etc this test.

Cost of Administration

This is an internal system, not a production system. However, if the servers go down, development basically stops. So we put some pretty good administrators on it. These servers take about 2 hours to set up (including checking, racking, installing, adding to the network, etc). Then they take about an hour a month of maintenance. So we've got 36 hours of maintenance plus 2 hours to set up over the life of the server. A Linux administrator gets about $65,000/year salary (or about $32.50 an hour) in Boston.

38 hours *40% * $32.50/hour = $494 (cost over three years to administer this server)

$494 / 3 (years) / 52 (times per year) = $3.17 per test run

Totals

So the actual cost per test run is as follows:

Server: $7.69

Infrastructure: $1.33

Administration: $3.17

Subtotal Per Server Per Test Run: $12.19

7 servers per test run: $85.33 per test run

So our "free or nearly free" automated test is costing us a little over $85 each time we run it. Manual tests require a lot of the same things, so generally this is worth it. Just make sure you check your numbers when so that your "should I automate" calculations include the true costs of running the automated tests.