Tuesday, September 25, 2012

Clean Project Checklist

Some projects are harder than others. On a few of my projects, everything just feels harder than it should be. I can't find the classes I need to change; the tests on a clean checkout fail; configuration is manual and fiddly; getting set up to try your code is slow and unwieldy. They're just painful all around. Other projects are smooth and easy. There's a solid README for setting it up - and the README is short. Tests run well on clean checkout. Classes and methods are in the first place you look. The only hard part of these projects is any difficulty in actually solving the technical problem.

The main difference, I've discovered, is in the tooling around the projects. Some of the smooth projects are huge and involve different services and technologies. Some of the difficult projects are actually quite small and simple. When the tooling is clean and works, then the project is much more likely to be smooth. Over time I've come to look for a set of things that tend to indicate a clean project.

This is my clean project checklist:

  • Deployment and environment setup is automated using an actual automation tool. I don't care if it's Capistrano, Chef, Puppet, RightScale scripts or whatever. It just has to be something beyond a human and/or an SSH script. This includes creating a new box, whether it's a cloud box or a local machine in a data center somewhere. If it's scripted then I don't have to fiddle a lot manually, and it also tends to mean there's good isolation of configuration.
  • Uses Git. This makes the next item possible. It's also an indicator that the developers are using a toolchain that I happen to enjoy using and to understand.
  • Developers are branch-happy. Feature branches, "I might mess this up" branches, "just in case" branches - developers who are branch happy tend to think smaller. With that many branches, most don't survive for long! Smaller changes means smaller merging, and that makes me happy. It fits nicely with the way I work. I should note that I don't care how merging happens, either by merge or by pull request.
  • Has a CI system that runs all the automated tests. It might not run them all in one batch, and it might not run them all before it spits out a build, but it runs them. The key here is that automated tests run regularly on a system that is created cleanly regularly. This cuts down on tests that fail because they weren't updated, or issues that relate to data not getting properly put in the database (or other data stores).
  • Cares about the red bar. The build shouldn't be broken for long, and failing tests should be diagnosed quickly. In policing, it's called the broken windows problem: if you get used to small bad things then bigger bad things will happen. Don't let your project windows break.

I'm sure there's more, but what else am I missing?

Tuesday, September 18, 2012

Automate What?

I was in a meeting the other day, talking about an upcoming project. I made some offhand comment about needing to make sure we allowed for a continuous integration and test automation environment when we were sizing our hardware needs. The response was immediate: "but automation never works!" After a bit of discussion, it emerged that this team had never attempted automation at any level under the GUI, and their GUI automation was, well, not good.

Automation at the GUI level is a legitimate thing to do. It's far from the only automation you can do. Consider all the other options:

  • Automated deployment. Manual scripts or with tools - this is an entire discipline in itself. And it doesn't have to deploy to production; automated deployment to test or staging environments is also useful.
  • Unit tests. Yes, this is test automation, too.
  • API tests. Automation of a user interface, jut not a graphical user interface.
  • Integration tests. You can test pieces together without testing the whole thing or without using the GUI.

So here's my automation heuristic:
Automate as much as you can as low as possible.

Let's break that down a bit. "Automate as much as you can" means just that. Not everything is automate-able, and not everything should be automated. If you have a GUI, testing that the colors go nicely together is probably not automate-able - so don't do it. Do that manually. If you have a method that takes inputs and produces outputs, you can probably automate the test that confirms it works with different kinds of inputs and outputs. If automating something involves massive architectural changes or fails one in three times randomly, then you're not going to maintain it and it's either not automate-able or simply broken.

(On a somewhat related code, the ability to automate something frequently correlates with how modular and maintainable an architecture it has. Hard-coded configuration parameters, nonexistent interfaces, etc. make code less testable and also a lot harder to maintain!)

"As low as possible" means that you should target your automation as close to the code you're trying to affect (deploy or test, for example) as you can. If you're testing that a method does the right thing, use a unit test. Sure, it might be possible to test that same method through a GUI test but the overhead will make it a lot slower and a lot more brittle. The team I was talking with was right about one thing: GUI tests tend to be brittle. If you can go below the GUI - either through an API or using an injected test framework (think xUnit tests) - then you can avoid the brittleness.

Yay automation! But even more yay, appropriate automation!

Friday, September 14, 2012

Don't Forget the Joy

Work is, well, work. It's also an open secret that we're better workers when we have fun at work. We're more productive, we help the bottom line, and we're more likely to stay in a job longer if we enjoy our jobs. So let's roll up our sleeves and get to work, but don't forget the joy!

Some joy comes from the work itself. There is a thrill in solving a hard problem. Learning a new technique provides a quiet satisfaction. Finally understanding some concept - really understanding it! - puts a spring in my step. Looking at a demo or a customer report and knowing I built some feature they really love sends me home bragging. I know I'm not alone in quite simply loving what I do.

Other joy coms from the culture around the work. This is where your colleagues and your work environment come in. Joking around the coffee machine is just plain fun. Having your desk set up the way you like it - with your Star Wars figurines and footrest - makes the physical office more comfortable. We're people, after all, not automatons.

As an employer, there are things I can do to help make work more fun. Most of them aren't even very expensive! Consider these:

  • A training course - online or at a local university's extension school - for an employee who asks usually costs under $1000.
  • Letting someone go speak at a conference costs about 20 hours for prep and the cost of plane/hotel. (The conference itself is usually free for speakers.)
  • Letting the team choose a theme for sprint names and name their own sprints. Free and often hilarious.
  • Bringing in bagels or cookies or cake once or twice a month - either on a schedule or ad hoc, depending on your team's sense of routine - is surprisingly fun. Keeping snacks on hand accomplishes the same thing.
  • Don't do management by walking around. Be available and show up but don't hover. You don't want to be your employees' friend, but you don't want to be the heavy, either. Free, too.
Joy has a place at work. Encourage it.

Tuesday, September 11, 2012

Status Message Messaging

For those of us who use hosted services, status pages are an essential communication point. They're a way to indicate service health ("it's not us, it's you") and, when there are problems, they provide a forum for disseminating updates quickly and loudly. The "everything is okay" update is pretty easy. The "stuff's broken" update is a much more delicate thing to write. It has to reflect your relationship with your users, but also reflect the gravity of the situation.

Here's part of a status update Beanstalk published this morning:

"Sorry for the continued problems and ruining your morning." 

Oh man. That's an update you don't want to have to publish. To provide some context, we'll just say that Beanstalk has had a bad few days. Beanstalk provides git and subversion hosting; that makes them a high-volume service. Pushing, pulling, committing, checking in/out, etc. happen very frequently and, well, software teams on deadline are not known for being nice when their tools get in the way. The last few days have been hard on Beanstalk: they got hit by the GoDaddy attack, then had a problem with load on their servers traced to an internal problem, and finally are again having difficulties with their file servers that host repos. And you can see it in that status update. "[R]uining your morning" is the phrasing of someone who is seriously exasperated. That update does some things well: it shows they understand the problem is severe; and it reflects the increasing frustration users are likely experiencing. It's escalating, just like user's tempers are probably escalating. However, it goes too far for my taste. It reeks of frustration on the part of whoever wrote the update, and that's clearly not a good head space for actually solving the problem. It also implies a sense of fatalism. That update was at 9:23am - my morning might still be salvaged, if they can get the system back up relatively quickly. Don't give up, guys!

There's an art to writing the status update when things are going poorly. When I'm working with a team fixing a major problem, I'll dedicate someone to doing just that. They sit in the middle of the war room (or chat session or conference call) and do nothing but manage status communications. Let the people fixing the problem focus on fixing the problem. Hand off status to someone who will maintain a level head and write a good status update, considering:

  • Frequency of updates. At least every hour for most issues, and whenever something changes there should be a status update. Silence does not inspire confidence.
  • Location of updates. As many channels as possible are good. Use twitter, the status page, email or phone calls to really important customers, and any other tools at your disposal.
  • Tone of updates. This needs to match your general tone of communication (see the Twitter fail whale - still cute and fun, even in error!) but also show that you know your customers are getting frustrated.
  • Inspiring confidence. Providing enough information to look like you are getting a grip on the problem and will get it fixed is important. Providing a proper postmortem also helps inspire confidence.