Thursday, September 30, 2010

Rails Bundler

I've been working with Rails3, and in particular with the bundler feature. Bundler exists because dependency management is truly obnoxious, and bundler tries to make it easier.

I'll let the tutorial teach you how to set up bundler. Let's look at what we're striving for.

The ideal is that setting up a Ruby application is as simple as:
  • install your database
  • install the right Ruby
  • install the bundler gem
  • check out the source tree
  • configure config/database.yml
  • run "bundle install"
  • run "rake db:setup"
  • Go!
This is really cool. No. Really, it's that cool. It's much easier than clicking on a page and watching it go boom because of some dependency you forgot.

Here's what's going on:
You specify the gems you want in the Gemfile, including the version you want. Then you run "bundle install", and it creates a Gemfile.lock file with all the gems that you're using (including their versions).

This directly addresses the consistency of various environments issue, in that each person does a "bundler install" which retrieves the exact versions specified in the Gemfile.lock and makes them available to the application. This is much more reliable than simply saying "require 'mime-types'" and maybe getting version 1.16 or maybe getting 1.15 depending on what was installed on that developer's system or that server. The only ruby gem needed, in fact, is bundler itself - all others, including rails, are loaded using it in isolation of any other versions of those libraries installed on the system. You can have four versions of some gem installed, and it'll use exactly the one specified in Gemfile.lock.

There must be some downside:
The biggest drawback to the bundler arrangement we have now is that deployment to servers (or development environments) has a dependency on the availability of the server and specific versions specified in the project Gemfile. If that server(s) is down or the specific version required no longer available, the "bundle install" will fail.

The fix:
Run "bundle package", which loads the gem packages into your application. You can then check them in. No more external dependency (but your source tree is rather larger).

Now that's cool. If you're on Rails3, give bundler a shot.

This guest post is courtesy Dan Powell.

Wednesday, September 29, 2010

STPCon 2011 - Call for Proposals

Okay, confession: I didn't write this, but it's worth repeating:

Submit! Try your hand at speaking to other testers!

Seriously, if you have anything you like to rant about, that's probably a good topic. Put it together coherently and send it in. I believe STP pays the conference fee for speakers, so you'll even get to all the cool stuff and only have to pay plane and hotel - not a bad gig.


You are invited to submit a speaker proposal for one of the leading events in the software test and quality assurance profession – Software Test Professionals Conference 2011, March 22-24 in Nashville, TN at the Gaylord Opryland Hotel.

If you have a session topic that meet’s STP’s standards, please visit our website at:

STP needs you – active software test and quality assurance practitioners, consultants, professional speakers, trainers, and solution providers– to share your expertise with your peers at one of the many one hour breakout sessions that will be featured at this event.

STP’s conference program board will review and select from these submissions the topics that support existing and emerging trends facing our industry today. We are seeking session proposals with solid content featuring specific how-to’s, and valuable take-aways. We are interested in sessions from industry leaders who have a particular expertise they can share, one that sheds light on an issue, practice or process in our industry. Product pitches will not be considered a viable breakout session. Strong platform and presentation skills are a must!

It does take work to put together a good presentation, but the rewards are great! If selected to present at STP 2011, you will receive a complimentary registration to the conference. Speaking at a Software Test Professionals conference not only gives you and your organization exposure at the conference but also leading up to the conference. Speakers who are chosen to present will have their name, title, organization, photo and bio published on the STP 2011 Conference website and in the full conference brochure. Speakers are provided with additional opportunities to promote their expertise and/or business. For instance, conference speaker articles have been featured on our website and e-blasted to our community of 50,000 members. When the article is featured, we also send out social media alerts to the software testing community-at-large.

Submit your proposal to speak now:

For consideration, please complete the online proposal form by September 30, 2010. Feel free to submit more than one topic. We look forward to receiving your proposal and hope to see you in Nashville in March!

Pass this email along to any of your colleagues who you believe would be an outstanding speaker at STP’s 2011 Annual Conference and Expo.

Thank you,
STP 2011 Conference & Expo Team

Tuesday, September 28, 2010

A Note On Su

I just learned this yesterday, and thought I'd share.

On a Linux system, doing:
(sudo) su -

And doing
(sudo) su

Does different things. When you include the dash, it loads root's environment. When you skip the dash, it makes you root but leaves you in your standard user environment. This is obvious once you know what to look for, but it took me a minute to figure out when I had forgotten that dash.

Monday, September 27, 2010

Defect Injection

So you wanna know how good you are.

You know how many bugs you find. You know how severe they are. You know how frequent they are. You know the ones that cause people to say, "oh, okay" and the ones that cause people to say, "Oh boy, am I glad that one didn't get out to the world!"

But you don't know much about the bugs you missed. Maybe you missed 10, maybe 1000. Maybe the ones you missed will be seen in the field, or maybe not. Maybe they're high severity, maybe they're low severity. That's a lot of maybes and a big black hole in your knowledge of testing the system.

Enter defect injection.

Defect injection is a technique designed to see how effective testers are and what kinds of things they find and they miss. Here's how it works:
  1. Go find your friendly local development team
  2. Ask them politely to insert 10-100 bugs throughout the code.
  3. Test as normal
  4. Go back to your friendly local development team and show them what you found
  5. Compare lists
  6. Ask your friendly local development team to fix all the bugs they inserted before shipping, please.
Basically, this is a test of your defect finding skills. Given a known list of bugs, it's possible to make statements about the bugs you missed. It starts to become possible to see patterns. For example, maybe you miss bugs in a given module, or maybe you miss race conditions because your tests tend to be really short.

If you have a willing development team, and you'd like to know what you find and what you don't in your particular situation, consider using defect injection. Just don't forget to remove the injected defects.... before you ship!

Friday, September 24, 2010

Beware the Zone

Ah.... the zone. That fabled place where we are tuned in to our work, where we're humming along with our system, where synchronicity and productivity bloom.

Beware the Zone

A zone is a temporary communion with your work, yes. But a zone is also a circumscribed place, a place that by definition has boundaries and edges - like a tunnel. A zone is characterized as much by what you do not see as by what you do see.

Many of the bugs I've found, and most of the really serious behaviors I've been able to characterize have started as a distraction. They weren't part of the test I was doing, but rather something that made me say, "hmm, that's funny."

When you're in the zone, you tune out distractions. That means you aren't seeing the guys across the hall playing basketball with a trash can, sure. That also means you're not noticing the odd setting in the log file that isn't what it should be.

It feels great to be in the zone. So let yourself go into it. Just don't stay too long - and ask yourself formally what sidelines you should be pursuing. It takes time to learn, but make sure that anything you accomplish includes the question, "and what else interesting do I see?". If you ask it formally long enough, it'll enter your zone, and then you're really testing.

Leave the basketball players out, but try to make sure the log file oddities are within your zone. After all, we're testers; very little we do is narrow. So let's make sure our zone is good and wide.

Tuesday, September 21, 2010

Ease Into It

I'm working with a support engineer who has some scripting chops. He wants to dig into a much bigger code set now so he can continue improving his development practices. I'm all for this. He's expanding his skills, he's providing help with projects, and he seems to be having a good time doing it.


It's a big code base he's coming into, relative to the code he's been working on. Most of his background is 100 line scripts written to be used in isolation and mostly by himself. The test code base he's going into is about 250,000 lines, is object oriented, and is highly interdependent. It's a brand new environment.

So the first change I asked him to make in there was related to a report it spits out. This change is going to involve a lot of things he already knows how to do - string and file manipulation, searching, etc.

When I give someone a project that involves a lot of new learning, I try to make sure it has three attributes:
  • It's useful.
  • It's straightforward.
  • It relies on techniques he's already comfortable with.
We're trying to set this engineer up to succeed, and a comfortable engineer is much more likely to succeed. So we give him a project that helps him stretch his skills but that leaves him grounded in skills he feels comfortable with. The goal here is to introduce uncertainty only in the areas where there is active learning. Everything else - the requirements of the project, whether anyone will care, and language or other techniques - should be easy or familiar. This helps the engineer learn without increasing the frustration level too much.

Learning something large isn't easy. So ease into it - start from what you know, and change only what you need to learn. There you'll find a useful project and a good learning experience. Repeat that enough, and you'll discover a whole new world of skills.

Monday, September 20, 2010

Explain It To A Newbie

As we work with code, we form habits. Some of them are great habits - run the tests before checkin, for example. Some habits are bad habits - writing only unit tests and no acceptance tests until after all the code is written, for example. Many habits are neutral habits - using rspec, for example. In all cases, they're ruts we fall into.

I'm suspicious of ruts. When I stay in a rut too long I worry that I'm not doing things as efficiently as I could. Maybe there's some newer methods that make things faster/better/stronger. Sometimes switching doesn't make sense, and sometimes it does. But you have to think about your habits to even consider changing them. It's shark engineering - keep up with the tools or you'll die!

To evaluate my habits, I'll grab a friend who's new to the codebase and explain what's going on.

For example, this is a recent one (a good habit):
Me: "Run bundle install to grab all the dependencies"
Him: "What is bundle install do?"
Me: "Bundler is a Rails 3 utility that allows you to specify all the gems your project depends on and just install them as you need them."
Him: "Oh, that's pretty cool. Saves discovering dependencies manually or installing them weirdly."

Or another recent one (neutral, this time):
Me: "Yup, we're using rspec here"
Him: "Why rspec?"
Me: "Well, it's in Rails core so there's not much to install, and the feature set is broad enough I haven't found much I can't do yet."
Him: "Oh, well, okay."

And a third (oops - a bad one):
Me: "I'm loading these with a rake task."
Him: "That's seed data, right? Why not use seeds.rb?"
Me: "Well, I taught the person providing the data how to write yaml so it's easy."
Him: "Sure, but seeds.rb can run any Ruby, so you can load the yaml from there, and then a new developer will be able to use the standard method."

This was a bad practice - and I fixed it. He was right, after all.

The point here is that I'm explaining my practices to a newbie - someone who doesn't know the code but who is an engineer. That innocent question - "Why?" - forces me to revisit a habit and to justify it. If it's good, the habit survives. If it's bad, the habit generally doesn't survive the explanation - and then I know to change it.

So if you're worried about habits, find someone and start explaining yourself - it's amazing what you can learn.

Wednesday, September 15, 2010

The New Toys Stack

As I may have mentioned, I've just started a new project. That means we get to make technology choices. We get to pick what tools we'll use for testing, what language we'll use, what libraries (or gems or plugins) we use. It's fun!

It's also a chance to play with new toys. Wanting to try HAML? Here's my chance. Never really got to mess with Watir? Now I can.


It's easy to go too far. There are so many new things to try, and it's easy to get carried away. You wind up using a technology that dies before it gets off the ground. Or you'll wind up using technologies that are great but that don't actually add value.

So when I'm building my new toy stack I put in all the new things I want to try that meet two criteria:
  • I'm actually going to use it now. Not eventually. Not soon. Now.
  • It's moving. It's gaining features and being actively worked.
There are no guarantees when we pick a technology that it will do what we want, or that it will grow with us. But we can try to protect ourselves a little bit by looking for projects that are likely to be useful and that are likely to grow with us.

When you're starting a project, feel free to put some new toys in your stack. Just make sure that you'll be happy with your new toys.

Monday, September 13, 2010

Yeah, but I Hid It

I logged a bug this weekend on a project I've been working on. The bug was pretty simple: under certain circumstances, when you clicked on item n in the list, you would see a few of the children of item n-1. You could tell because they still had the icon from item n-1 next to each of the children.

And a patch was applied in under two hours.

I poked at it and discovered that I was still seeing children in the wrong place. Only now, they didn't have the icon.

So I emailed the developer to be sure that he had actually fixed it. His answer....

"No, but I hid the error"

I laughed for a few minutes, then I started to tease him mercilessly, then I stopped.

You see, this is demo software. His fix didn't fix anything, but it did prevent an error from showing up in the demo. And when the demo is scheduled for Monday morning, sometimes hiding a bug is okay. Eventually it should be fixed, but considering the circumstances, I have to agree with his call - the low risk incurred to hide the problem was worth it; the higher risk to fix the problem wasn't.

Given a problem, we can live with it, change it's behavior (hide it in this case), or fix it. Any of the three are options, and we'd do well to remember that. Depending on what's going on, the right answer might be any one of those three choices.

Friday, September 10, 2010

Help Yourself: Read the Error

I've been having a conversation with someone today who was trying to follow some instructions I'd written several months ago. Unfortunately, things had changed with the script.

The step as written said this:
./ --tag=KERMIT --target=/tmp/foo

The script had been updated since I wrote the instructions, and the guy using it got this error:
./ --tag=KERMIT --log=/tmp/foo
Unknown option: --target

Were the instructions wrong? Absolutely.

Should this be something that requires my intervention to solve? Nope.

Reading the error makes it pretty obvious what likely happened: the log option got renamed to logDir somewhere along the way. It's a pretty good idea to try using logDir instead of log - it's got a good chance of working.

Help yourself. Read the error and see if you can solve it. If you can't, go ask for help. But at least read the error and try to help yourself first.

Wednesday, September 8, 2010

Learning is Embarrassing

Learning is embarrassing. As I'm learning something, I mess up. Frequently. And I get a bit embarrassed about it. After all, I've been doing this for a while; I should know what I'm doing!

For example, recently I've been doing a little coding in C. I don't really speak C. At all. Fortunately, I was pairing with Ken, who possesses a large amount of patience. I did a lot of dumb things, like try to print a buffer with sprintf, or forget to define a function in a header file. And each time it totally and completely bombed, he'd say, "maybe we should do it like this", and show me the right way.

I was embarrassed. I'm sure he could have gotten our task done about four times faster if he'd just done it. And really, I'm a software engineer. I ought to be able to get it to compile! And work, even!

But here's the thing. Ken didn't mind that this task took a long time, because it was a good thing to have done and now I know a lot more than I did when we started. Next time I have to change something in our C code it'll be a lot easier for me, and therefore less work he has to do. I'm not exactly a C guru, but those hours he put in will be paid back with hours that Ken doesn't have to spend because I can. And that's learning. Ken wasn't embarrassed or disappointed.

The only one embarrassed is you.

Everyone else understands that you're learning. And they're glad you're learning something new. They're glad that one day in not too long you'll have figured this out and you'll be a real help.

It's easy to be embarrassed to be practicing a skill that you haven't quite got, or to be learning something new and kind of messing up along the way. But put that aside. You're learning, and that's the important part.

No one else expects you to be perfect. Stop expecting it of yourself.

Tuesday, September 7, 2010

Summarizing with Relative Measures

"It spewed a lot of data."
"I found many bugs."
"That took a long time."

I know I've said every single one of these things. Most testers, developers, support engineers, product managers, and pretty much everyone else involved in creating software has said something similar more than once. It's a common way to summarize something.

So what's the problem?

Every time I've said this (or heard someone say this), the first question is identical:
"How much is a lot?"
"How many bugs?"
"How long, specifically?"

It's about as common as saying, "How are you?" followed by "I'm well. How are you?".

The problem is that our measure is relative. "A lot" to you might be 100, but to someone else, it might be "10 million" before it's a lot. "A long time" to that kernel developer might be 10 milliseconds, but to the web developer, 10 milliseconds is just about nothing. When we use these relative measures, our audience needs to calibrate those measures to their own relative measures. So they ask for the real number, pull out their internal yardstick, and say, "yup, that's a lot" or "wow, that's not a lot".

Don't be afraid of relative measures; they're just fine to use. Just be sure that when you're providing a summary with a relative measure that you have the actual number handy - odds are you'll need it to make sure your relative is the same as your audience's relative.

Friday, September 3, 2010

Nothing in the Logs

We got an error message in our logs the other day that said:

Operation '' not permitted

After a bit of digging, we discovered that we're logging in ASCII, and for some reason this was attempting to do something that could not be represented in ASCII. When we finally got debugging to show us something, it was just junk that happened to render as a weird symbol.

This defect is now known internally as "the operation formerly known as Prince."

(For those of us playing along at home, the underlying issue was attempting to use the wrong portion of memory at the wrong time, with an odd result.)

Wednesday, September 1, 2010

Code Is Communication, Too

I recently started a new project. It's greenfield, very exciting, and best of all, it's my idea. Unfortunately, this also means that whatever specs need to exist, well, I'd better get to work on creating them!

For the basic idea, we've set up a Basecamp project, and I'm putting stories in there. The stories are high level, because we've all worked together for long enough we can communicate in shorthand. We can say, "let's build user login based on email and password" and understand that implies session timeout, hiding everything behind the login unless we specifically mention otherwise, etc. And then there are the nitpicky little details. What are the requirements for password strength? What are the error messages the user should see when they fail to login?

For those, we're communicating through code. Yes, code can be communication, too.

I'm a tester, so we're communicating through tests for these types of little details. After all, I'd have to write these tests anyway. So I just write them a little earlier, check them in (failing), and the person writing login simply makes them pass. For the record, I didn't come up with this idea. It's been around for years called test driven development and behavior driven development (and probably several other things).

I've learned a few things over the years that are helping to make this a success, though. It's far from guaranteed.

Communicate through code when:
  • The audience is engineers. No code is really English* so use code only when communicating with others who are comfortable in code - developers, testers, some analysts.
  • The requirements are small and concrete. If there is a lot of interaction or a bigger picture, then express it in some way that describes the bigger picture. This is almost certainly not a single test or a series of tests. Use tests to communicate password strength requirements; don't use them to define an interface between two systems (that one probably wants a data flow diagram to make it intelligible to others).
  • You're not co-located. When you're in Boston and the other person on the project is in San Francisco, you can't talk, and writing things down in code is easier than writing code and also something else to communicate all the details.
  • You want one location for information. You will eventually have the information in code anyway, in the form of implementation and tests. If you only put it in code then you'll save some time trying to maintain and reconcile multiple copies of the information.

Don't communicate through code when:
  • You're talking big picture. Vision is a large, shimmering, wonderful thing. Express it in words, drawings, videos, and other things that can show your excitement. Code is many things, but it doesn't convey excitement.
  • You need to communicate with outsiders. If they can't see your code, then they can't get information from it.
  • You need to communicate with non-coders. I don't speak German to people who don't speak German; they're simply not going to understand. The same thing applies to speaking code to people who can't read code.

Communicating through code is one of those things that sounds like a great idea. Often, it is a great idea. Sometimes, it's a pretty terrible idea. As with any communication, consider your audience, think about why you're communicating, and then decide if code is an appropriate communication mechanism.

* I had a conversation recently with James Bach and a few others about this specific topic. Code isn't English, no matter how many English words is uses and how much it really does kind of look like English. There are far more rules and restrictions in code than in English (or any other human language). Credit goes to James for helping me think about this one a little more.