This lesson applies to test infrastructure as well.
What We Do In the Real World
One of the things we have to test is an upgrader for our system. Here's roughly what happens:
- upload the upgrader package
- run an upgrade prep script
- run the upgrader
- wait while the upgrader checks the health of the system
- wait while the upgrader puts the system in maintenance mode
- wait while the upgrader deploys packages to the system
- wait while the upgrader reboots each machine in the system
- wait while the upgrader reapplies specific configurations
- wait while the upgrader puts the system back into normal mode
- check everything
What We Do In Test
The good news is that we have a number of automated tests for this very process. They run automatically and assert that various things that are supposed to change during an upgrade change. They even assert that various things to be preserved across upgrade are preserved.
However, in our automated tests, we're exercising upgrade with just a few small changes. We deploy packages just a bit differently, for example. We have some extra shares mounted to get at test code, etc. So it's an upgrade test... almost.
Is It Legitimate?
So is our automated upgrade test legitimate? Can we really say we've gathered some useful information?
I think it's a pipe dream to say that your test automation is going to be exactly like your manual test, or to say that either of those is going to be exactly like what will happen in the field. Instead, we have to strive for best achievable.
The goal isn't necessarily the same. The goal is to understand the differences and account for them. I'm not going to worry, for example, that my upgrader runs on a system that faces south when most of our customers deploy with systems facing east; that's a difference I'm willing to accept. I understand that package deployment changes mean I may be able to install a package in my automated test that would never install in the field, so I cover that with manual tests that more closely mimic the real world. So what's the lesson here?
Understand the changes you're making - intentionally or unintentionally - and do not expect your tests to provide information in areas that you've increased your risk by making a change. Each test has it's purpose; a change from the field simply informs and influences what that test's purpose can be.
* Note that what I've shorthanded here as "automated tests" means "test scripts that run from a cron job every night on the latest code and make assertions without human intervention." Human intervention is required to identify the cause of any failures or errors.