Wednesday, August 31, 2011

Code Reviews: What I Look For

I'm currently working with a number of contractors, and all of us are working in the same code base. We each work on a separate feature branch, and when it's done, we merge to master and deploy to staging, and then to production.

One of the benefits of this is that it's easy for us to review each other's code. We just grab the feature branch and start looking. Usually we're developers reviewing each other's code, but sometimes we'll get a tester to take a look, or even one of our fearless marketing types (she's usually looking at language and naming).

Now, a lot of books and checklists will tell you useful and good things to look for:
  • relatively short methods
  • good encapsulation
  • conformance to style guidelines (get a tool to do this, if possible)
  • all the tests pass
  • error checking and input handling
  • readability and understandability
  • locations and placement (use of libraries, MVC conformance)
  • etc.
These are all great things. But wait.... there's more. We do code reviews because they sometimes offer us other benefits, like:
  • an example of an elegant way to solve a problem (if there's an elegant solution in that particular code snippet - and it's not uncommon!)
  • a basic understanding of how the new feature fits in to existing features. This is useful when we go to do another new feature later on - I'll probably remember that I should go look at this feature because it might be affected.
  • an interesting library (or gem or whatever your language calls it) that's worth looking at for other things
  • common code I might not otherwise have noticed was there for me to use
  • a chance for the developer who wrote the code to ask questions or point out worrisome areas
Bob Martin's Clean Code has a lot more information, and I highly recommend it. Then go forth and actually look at your code base. There's a lot of neat stuff in there!

Monday, August 29, 2011

Budgeting for Engineers

It's budget time (oh boy!). Budgeting is its own kind of fun, because it involves not only understanding where your past and current expenses are, but also guessing what your future expenses will be.

And that's hard.

There is an entire financial discipline in forecasting and budgeting, and I'm not going to get into all of it here. For many of us, budgeting isn't our day job, but it's something that most engineering managers wind up doing at some point. It almost always comes to us phrased as, "So how much money do you need for the next 6 months?"

First response: "I dunno. What do you want us to build in the next 6 months?"
Analysis: Completely true, and totally unhelpful.

More helpful (and still true!) response: "Sure, let's talk about assumptions, priorities, and constraints."

Some tips:
  • Figure out one thing that you will use as the basis of growth. This could be number of users, number of logins, number of credit card transactions, whatever. Base everything off that one number (how many devs you'll need, how much your hosting costs will be, etc.), and your budget will stay coherent.
  • Separate assumptions. Put them in a separate tab in the spreadsheet, and calculate everything you can from those assumptions. That way when you discover a bad assumption, it's easy to fix.
  • Put in all the detail you can think of. No matter how small and silly, put in all the details. Specify your defect tracking system and source control, not a line item for "dev tools". Yes, it will be a very long list.
  • Create a summary. Take the subtotals from your detail and use them to populate a summary version of the budget. This is what you'll show people, and then use the details when they start asking questions.
  • Add 10% for unexpected things. Something's going to happen that none of us thought of. Give yourself a buffer for that.
I'm an engineer. I spend a large chunk of my day on code, not on budgets. But the budget is important, and I owe it to myself and to my team to do it properly. A budget is a negotiation, and just like any negotiation, the more prepared you are, and the more prepared you look, the better you'll come out of it.

Here's hoping these tips can help other engineering managers get through budgets with as little angst as possible.

Wednesday, August 24, 2011

Make Starting Easy

I've just started doing some work with a new client, and one of the things I'm doing is helping the team get started with some basic testing. The team consists mostly of programmers out of academia - they can write some code but have never written commercial software. This is in no way a derogatory statement; it's simply the current state.

One of the attributes of this team is that they know there is this thing called "software testing" that sounds important, and they know that there is "automated testing" that sounds like it would be really useful, but they're not at all sure how to go about doing it. Again, not a problem; that's why I'm here.

Testing software is a big problem space. People spend entire careers in software testing, or in specialties within the field. That's intimidating. For those of you non-testers in the audience, saying, "teach me to test" is something like saying, "teach me to be a developer" or "teach me accounting". This team doesn't need to know everything about testing; they're not going to be career software testers. They're just trying to make sure that they ship a product they can be proud of, and to make sure that their system does what they expect.

So how do we get a team of eager and smart people into software testing?

Make it very very easy to start.

So we started by talking about testing, and noting that there were a lot of different kinds of tests and many different terms. And then we ignored all of those terms and kinds of tests. Oh, we'll get back to many of them over time, but for now, it's not important.

Starting is important.

Doing one test, understanding why that's a test, and learning what we will get out of doing that test: that's what's important. And that's all that's important right now.

So we started with the very basic core principle of testing: you want to know something about this system. A test is a way of formulating that as a question and finding an answer. That's it. For now, it's really that simple.

So we're starting with the questions that we understand, that we're already trying to answer. And we're talking about how to answer them effectively, and how to formulate the questions so that they are answerable. You know.... testing. We don't care if it's a unit test or a performance test or what technique we're using. We just care that we're getting information out of the system, and that the information is effective and useful.

When you're faced with a big learning area, whether you're the student or the teacher, don't panic, and don't try to digest it all at once. Recognize that the hardest thing is starting. Then make starting as easy as possible. Nuance, techniques, and skill can come later. For right now, just find a way to get started easily.

Monday, August 22, 2011

Jenkins and Git

I've been setting up Jenkins for a client. While setting up a project, I ran into a very strange problem in which Jenkins (or more precisely, the git plugin) couldn't find git.

Here's what I did:
1. install Jenkins
2. create a project (call it "Foo")
3. say, "whoops. forgot the Git plugin"
4. install the Git plugin through Jenkins
5. restart Jenkins
6. configure the repo in Foo
7. build

I kept getting the error:
Error trying to determine the git version: --version
Assume 1.6

It couldn't see the git binary. I set the git binary in the main Jenkins config to "git" and to "/usr/bin/git", with no change. I also confirmed that the jenkins user could run the git binary, which it could.

The Fix:
To fix it, I deleted the project and created a new one. After all that, apparently you have to install the Git plugin before you create the project.

Lesson learned.

Thursday, August 18, 2011

Who/What/When/How/Why of Testing

I've been working with a client who has a group of academic engineers. These are very smart people who haven't ever built a product or worked in commercial software. So they asked me, "Can you teach me testing?"

Well, yes. I can teach you about testing. It's going to take a while. Ultimately, most of these guys don't need to become software testers, though. They just need a framework for figuring out how to figure out if their system does what it ought to do. The more sophisticated test design can be handled by the actual test team. These engineers just have to get far enough that they can work effectively with the testers.

Here are the very basics of what I told them:

Who should test?
Anyone can test. QA engineers (aka testers aka QE engineers) are simply people who specialize in testing activities. That doesn’t mean they’re the only ones who can do it! Use QA engineers as sources of knowledge and ideas just like you’d use a software architect, or a build engineer. They’ll do a lot of testing, but with a little help, you can also test.

What should you test?
Since testing is about gathering relevant information about potential and/or actual behavior, many things can be tested. Frequently, portions of software and systems as a whole are tested. However, software designs, UI designs, hardware, and even requirements or documentation can be tested. In short, if you (or someone involved in a project) want information about something, test it!

When should you test?
The earlier you test, the more time you have to react to what you learn from testing. So if you can test part of something, that’s fine - it’s early feedback. Testing continues for as long as you will do something with the information. That means you can continue to monitor and test, even after you’ve shipped the software. After all, you can still use what you learn, just in the next release.

How should you test?
Pretty much anything is fair game in testing. There are some guidelines and techniques that we can cover, but if it gets the information you need in a repeatable and sound manner, then go for it.

Why should you test?
You should test to get information that you can use. If you don’t have a use for the information you’ll learn from a test, then don’t bother doing the test. If you don’t know what information you might get from a test, then you need to better define your test (translation: go talk to someone who tests for a living).

For many testers, this is "back to basics" information. For someone new to professional software engineering, though, basics are a great place to start!

Tuesday, August 16, 2011

Why Textaurant Runs Rails

Textaurant is a webapp. It displays, routes, and gathers information. It sends text messages, and we emphasize the user experience on mobile browsers. The number of technologies we could have used to build Textaurant is really quite large. We could have gone with Java, or .NET, or Python, or PHP, or any one of a number of languages and frameworks. Any one of them would have been valid and viable choices.

We went with Ruby on Rails. And here's why:
  • Speed
  • Support
  • People
Textaurant is a startup, and (surprise!) we have competitors. We beat our competitors by giving them better value faster. On the technology side, that means we give them more features, more reliably, faster. Development speed is a huge consideration. Ruby on Rails is a fast development framework. My team and I can get features out there quickly. We can tweak them quickly. That makes us responsive to our customers, which helps us get and keep happy users.

There's a lot of support for Ruby on Rails; it's a rich development ecosystem. We use Heroku for hosting, for example, because it's optimized for Rails applications. That means that we can provide a stable, scalable (within reason) environment that is optimized and secured for our application. And we get it all for just the cost of hosting. There is also a strong community that makes reusable gems for everything from sending text messages (thanks, Twilio) to authentication. That means I don't have to reinvent the wheel, and makes development faster. (I should note that we also try to give back to the community by creating, maintaining, and/or contributing to gems - it's only a rich ecosystem because we all contribute to it.)

Finally, we use Ruby on Rails because of the people. There are many developers who can write Rails applications, and many testers who can test this type of application. This gives us a wide talent pool to draw from, and also lets us honestly say to people who work with us that they're developing and maintaining a useful skill set that they can use in the future, for Textaurant or wherever they end up. (Tangent: I had a job early in my career in healthcare IT. We saw a lot of developer candidates who had spent five years learning and working in a custom language called Magic for another healthcare company. They didn't have much in the way of Java skills, and that hurt them in the job market; many of these candidates got passed over, even though they had industry experience. I don't want to do that to anyone who chooses to work with me.)

Will Textaurant use Ruby on Rails forever? I don't know. Like any technology choice, Ruby on Rails has strengths and weaknesses. At some point we may run into limitations of the language and/or the framework and make a change. For now, though, we're using the technology stack that best fits our needs, and hopefully that fact will never change.

Friday, August 12, 2011

Notes on Feature-Based Releases

Let's say for the sake of argument that we would like to do feature-oriented releases.

Why We Might Do So
There are a number of reasons we might choose to do a release based on its contents (features) rather than a date. These include, for example:
1. because a customer is blocked by a bug and cannot wait until the next release
2. because a customer or potential customer is blocked by a feature that is required to complete (or continue) their development
3. because we want to show a customer or potential customer that we can turn around requests quickly
4. because a feature is likely to need change and we need to provide an early copy to customers (some or all) to collect feedback

Notes on Speed
An implication behind feature-oriented releases is that they are likely to be more frequent than date-based releases. This is mostly because date-based releases generally include several features, and are typically larger overall change.

Number of Versions in Field
There are some downsides to doing feature-based releases. In particular, it winds up with far more releases in the field. In a situation like a hosted web application, where the company controls production, this isn't a problem. In a situation where software is shipped and upgraded by customers (consumer installed software, app-store-based applications, many enterprise applications and libraries), the number of releases in the field can be a concern. Too many releases in the field causes a major drag on future releases, mostly due to upgrade and similar tests having many more variables.

There are a few ways to handle this, from aggressive release end-of-lifing to support policies forcing upgrades. This is a bit of a delicate line to draw with customers, though.

Increasing Speed
Let's assume that the time to actually implement, debug, refine, fix, test, and release any one feature is roughly the same regardless of what type of release we're doing. The way to increase the speed of releases, then, is to reduce the other work that goes into a release. This other work can be characterized as:
1. integration of other features, in particular of multiple in-progress features
2. documentation, including release notes and updates to presentations, marketing materials
3. availability notification of releases (e.g., upload to a customer portal)
4. once-per-release validations and measurements (e.g., performance tests, or packaging validation)

Of these, by far the largest are numbers 1 (integration) and 4. Number 1 in particular is often a large unknown: two features that are fine alone may not integrate cleanly. When both are still in progress, they are likely to integrate even less cleanly. For number 4, with very frequent releases, it can often be necessary to perform these validations once per several releases, depending on what they are for (e.g., do in depth performance testing every 5th release or if there is some concern about the specific feature being released).

Use of Branches
One of the most common problems with a feature-based release is when there are multiple features in progress. Feature A may be complete and ready to go, but feature B isn't ready and the tree is unstable as a result. Feature A is in essence held hostage by feature B. For this reason, doing feature-based releases, we usually designate a "release branch" (typically main or head) and "feature branches" (one per feature). The workflow for a feature looks like this:
1. create a branch for that feature
2. do all the feature work
3. merge from the main release branch (to pick up any changes or other features that were finished while you were working on this one)
4. fix/test your feature
5. repeat 3 and 4 until complete
6. merge to the main branch
7. sanity check and release

This adds some branch management overhead, but pays off in the ability to keep a stable, releasable main code base.

When working with multiple releases and in particular with APIs where deprecating features or APIs is a consideration, special attention should be paid to how many releases occur before a deprecated element is removed. As with all release types, the goal is to balance simplification of development (by removing support for deprecated items) with customer ease (not having to rework their integration). One approach is to do periodic "roll up" releases that remove deprecations from previous releases. These are frequently the releases to which you will push your slow-upgrading customers, who are likely to skip interim releases.

For example, we might release like this:
4.1: new feature A
4.2: new feature B
4.3: deprecate old feature X
4.4: new feature D
5.0: roll up release, remove deprecated feature X
5.1: new feature E
5.2: deprecate feature Y
5.3: new feature F

Feature-based releases have a number of benefits. They're also a very bad idea for some situations and some groups. Take a look, figure out if it's right foryou, and good luck either way!

Wednesday, August 10, 2011

Some Thoughts on Heroku

I've recently started working with Heroku for the first time; we're hosting a Ruby on Rails app on a shared database there. I've had sites hosted on Amazon EC2 for years, so I'm used to working with virtual machines in cloud environments, but Heroku is a new twist for me. What follows are some early impressions.

What I Like:
  • Easy administration. Our traffic happens to be extremely spiky (Friday's evening traffic is approximately 100x Monday's noon traffic, for example), and the ability to add and drop capacity very quickly is huge. It's literally a GUI slider, so even the non-technical business type can do it when I'm not available.
  • Low systems administration overhead. There are a lot of things involved in successfully administering a system: making sure there's redundancy, setting up and maintaining (and testing restores on) backups, keeping the machines up to date, etc. This isn't entirely gone under Heroku, but it's significantly reduced.
  • Rollback. I've only had to roll back once (to make sure I knew how!), but it was pretty darn simple. And when you're rolling back a change, you're already in a negative frame of mind, so it's really helpful to at least have an easy way out.
What I Dislike:
  • No database access. We're on the shared database, so I can't actually log in and run a SQLquery. It's not a big deal most of the time, but it turns out that I'd become a little bit addicted to having this information when I look for data-related issues or want to see query plans to track down a performance bottleneck, etc. I don't like the feeling of being blind. I should note that this limitation does not exist if you're running on a dedicated database.
  • Deployment. See below.
  • Rollback. Yes, this one is on both lists! It's easy to roll back code. Rolling back things like database migrations is a whole different kettle of fish, and I have the same problems here that I do with deployment.
  • A general feeling of blindness. I always feel like I'm missing something when I use herokulogs, like it's harder to grep through. I don't know that it's actually any better or worse than less and grep on a machine, but it's taking some getting used to.
  • Worries about uptime. Just like everyone else, Heroku isn't perfect, and we have seen some downtime. It's pretty rare, but it's still worrisome.

A Note on Deployment:
With Heroku, you deploy using git. For me to deploy to staging, for example, I do this:
1. check out the staging branch (in our github account)
git checkout staging
2. merge whatever I need to
git merge blah
3. run the tests as a sanity check
bundle exec rake rcov
4. push my changes to github
git push origin HEAD
5. push my changes to heroku
git push heroku-staging staging:master

On the surface, this is pretty slick. I'm already using git for source control, so deployment is as easy as setting up a new remote and pushing to it. For anything that's more complicated than a simple git push, though, it gets a lot messier. Let's say I have a migration to run as part of this change. Then my push looks like this:
heroku maintenance:on && git push heroku-staging staging:master &&heroku rake db:migrate && heroku maintenance:off

Adding other things, like jammit, gets even messier. So overall, it sounds great, but I'm not a big fan. Things like chef and puppet and capistrano exist for a reason and are great to use, so I'm not sure we needed yet another way. I've wound up writing recipes to do most of the pushing for me so I don't miss a step, though, which helps.

Putting my money where my mouth is, are we sticking with Heroku? Yes, at least for now. The quick and easy scale-up and scale-down is huge, that trumps the quibbles. Oh, and the quibbles really are pretty minor.

Overall, Heroku is not for everyone, and I encourage anyone considering it to look very hard at things like cost, uptime, service levels, backup policies and timing, etc. Make an informed decision, but after my experience so far, I would at least consider Heroku next time.

Monday, August 8, 2011

Testers and Testing

I've been working on a variety of projects lately that are staffed very differently. One project has a dedicated test team, and a development team. One project has no dedicated tester, and the development team is terrified that they "don't know how to test any of this". Yet another team has no dedicated testers, and they figure they're doing all right without it.

Why such different teams? And why such different reactions?

I suggest that it is entirely to do with testers versus testing.

Testers are people filling roles.
Testing is an activity.

A tester is (ideally) someone who is dedicated to the study of testing, and to doing the testing activity better, more efficiently, more effectively, etc. That doesn't mean that only testers can do testing. That doesn't mean that testers must only do testing. It mostly means that someone who is a tester is more likely to do testing activities more thoroughly and efficiently than someone who is not a tester. It's a simple analogy to a database architect. Sure, they're probably going to build you a better data model faster, but you could also have another engineer create a data model. It's up to you to decide whether that activity (the data model, or the testing) is worth having a dedicated resource.

Be careful not to conflate the role (tester) with the activity (testing). It's perfectly acceptable for all the testing activities to be done by people with the formal role of developer, or project manager, or client. It's completely all right if a person with the formal role of tester does testing activities.

The important thing for a successful project is not that there is a dedicated tester. The important thing is that the testing activity is occurring.

As with any software development activity, decide how important it is, and staff accordingly. Don't build a team based on roles; build a team based on activities.

Thursday, August 4, 2011


Last night I was talking with a friend about a team and a manager who was asking for excessive status updates. The manager was new to the team, was asking for updates in two places (on a wiki and email), and the team was getting demoralized. So what's going on, and how do we fix it?

Let's step back a second, and talk about contracts. Most contracts between people are implicit contracts. They're implied promises of behavior and trust. To use a non-software example, when I cross the street at a light, I have an implicit contract with the driver that he won't run me over, and I won't stand in front of him when he has the right of way.

The same is true in organizations; there is an implied contract at all levels that, "if you give me the information I need and get out of my way, then I will provide acceptable work at a reasonable pace that matches your needs". Teams producing software make that contract with their team leads. Team leads make that contract with their managers, and so on up the line.

When an implicit contract is being violated, angst ensues. This is what's happening to the team we were discussing last night. Someone is violating the contract. It could be a lot of things:
  • Maybe the manager is asking for status too frequently or the wrong way (not staying out of the way)
  • Maybe the team isn't producing good work, or not at a reasonable pace
  • Maybe the team isn't producing work that meets expectations (garbage in? or possibly just going off in their own way)
  • Maybe the manager doesn't know how to produce good work (in her case, a productive team is her good work), and this is a proxy for it
  • Maybe the manager's manager is not staying out of the way
In other words, we don't yet know where the contract violation is. Just to throw an interesting twist into things, sometimes the contract violation happened in the past, and people are still nervous about it. It takes time to relearn trust after a violation, and this may be that time period.

Is there a problem? Absolutely.
Where's the problem? Don't know yet. Follow the contract, and we'll find it.

Look beyond the immediate complaint (too much status!), check the implicit contract, and fix the underlying problem. That's the way to productivity and great teams.

Wednesday, August 3, 2011

Work With the System

Most systems have intent. They - courtesy of their creators - have workflows, expected usage patterns, etc. At their core, the creators of the system had one or more ideas of how the system would be used, and they optimized the system for those uses and those ideas.

For example, an archive storage system I worked on was designed specifically for archiving data. This intended use had several manifestations: it was highly optimized for writing and not for reading; and data integrity was paramount, even at the cost of speed or price.

As another example, I worked on an electronic medical record system. It was designed to be very friendly for physician use and to be a "one stop shop" for patient medical data. This intended use meant that physician workflows (ordering prescriptions, for example), could be accomplished faster and withe fewer clicks than insurance workflows (reviewing payment data); we had optimized the system for physician use. The intended use also meant that we had many interfaces to other systems, but almost all of them were other clinical systems (labs, or radiology systems) and not to insurance payment systems.

Now, could you use the system for something outside its intended purpose? Yes, absolutely. To continue the example of the medical record system, it's entirely possible to deploy it in a medical billing office. It'll just be harder to use.

And that's today's lesson:

Work with your system, not against it.

Yes, it's frequently possible to use the system in ways the creators didn't intend. Most of the time it'll even mostly work. But it will be harder than it should be. You'll run across non-intuitive workflows. Performance will be a slower. You'll find odd bugs in things that most users will never see but that are important to your workflow. You'll get frustrated.

So before you go making a system do things that the creators didn't mean for it to do, try finding something that actually is intended to do what you want. If you can work with the system, you'll have a much easier time.

Monday, August 1, 2011

AST Meet the Candidates: Tonight

Tonight I'll be participating in a Twitter forum for AST candidates. Here's how it works:

When: August 1, 9:00pm Eastern time
Where: Twitter. Look for #ASTElect
Who: Four of the five candidates will be answering your questions: me, Matt Heusser, Michael Larsen, and Pete Walen
What: We'll answer any questions you might have. Ask us about AST. Ask us about testing. As us about anything you like!

I hope to see ya'll there!

Find me at: @cmpowell