Tuesday, September 18, 2012

Automate What?

I was in a meeting the other day, talking about an upcoming project. I made some offhand comment about needing to make sure we allowed for a continuous integration and test automation environment when we were sizing our hardware needs. The response was immediate: "but automation never works!" After a bit of discussion, it emerged that this team had never attempted automation at any level under the GUI, and their GUI automation was, well, not good.

Automation at the GUI level is a legitimate thing to do. It's far from the only automation you can do. Consider all the other options:

  • Automated deployment. Manual scripts or with tools - this is an entire discipline in itself. And it doesn't have to deploy to production; automated deployment to test or staging environments is also useful.
  • Unit tests. Yes, this is test automation, too.
  • API tests. Automation of a user interface, jut not a graphical user interface.
  • Integration tests. You can test pieces together without testing the whole thing or without using the GUI.

So here's my automation heuristic:
Automate as much as you can as low as possible.

Let's break that down a bit. "Automate as much as you can" means just that. Not everything is automate-able, and not everything should be automated. If you have a GUI, testing that the colors go nicely together is probably not automate-able - so don't do it. Do that manually. If you have a method that takes inputs and produces outputs, you can probably automate the test that confirms it works with different kinds of inputs and outputs. If automating something involves massive architectural changes or fails one in three times randomly, then you're not going to maintain it and it's either not automate-able or simply broken.

(On a somewhat related code, the ability to automate something frequently correlates with how modular and maintainable an architecture it has. Hard-coded configuration parameters, nonexistent interfaces, etc. make code less testable and also a lot harder to maintain!)

"As low as possible" means that you should target your automation as close to the code you're trying to affect (deploy or test, for example) as you can. If you're testing that a method does the right thing, use a unit test. Sure, it might be possible to test that same method through a GUI test but the overhead will make it a lot slower and a lot more brittle. The team I was talking with was right about one thing: GUI tests tend to be brittle. If you can go below the GUI - either through an API or using an injected test framework (think xUnit tests) - then you can avoid the brittleness.

Yay automation! But even more yay, appropriate automation!


  1. Nice write-up, could not agree more.
    I work in an agile environment where we do a lot of pair programming. We began applying these principals a few years ago and have had quite a bit of success. However we quickly realized that the traditional line between the Dev and Test roles becomes a grey area and ultimately a hindrance to building our products.
    What are your thoughts on how to deal with different roles with this type of approach to testing?

  2. Anonymous, thanks for the comment. I find the line between dev and test to be very blurry. Some people with the title "developer" are great testers, and some people with the title "tester" write great code. I approach it more as a "tasks a person can do". If the team as a whole is balancing the tasks people do, then we're doing well. If not, we need to change the mix of the team - by adding people, moving people, or training. I try not to get too fussy about a "tester" adding a button to a page because the task was needed and she had time, or about a "developer" testing that button because the task was needed and she had time. To me, your title indicates the things you usually do, not the totality of what you can do.

  3. Hi

    I am also a tester and mainly I have done manual testing. So just one question how much automation is need in a project and is this prior need to automate project.

    But when I have started my testing career, I was told that manual testing is the mainly the real testing and when we come to conclusion of stability of product then we think about the automation testing.
    How much you are agreed to this concept

  4. your post have many thing to learn.

  5. abodeqa, I think that you should automate when doing it will provide information and when the cost of keeping the automation maintained drops to something reasonable. It's not really a rigid "manual first, automate second" approach or vice versa. Automation should be by value, not by time.