Friday, January 28, 2011

"Anyone Can Do It"

I was lucky enough to come of age in a software world in which web development was just starting to become really accessible. My first computer dialed up to the Internet using Juno, and then we got AOL. By the time I was in college, we had people putting up websites and discovering the joys of the tag. Then came php, then ASP and then Java, VB, and more and more.

All of a sudden, web development was accessible. "Anyone could do it."

We're still in that world. Frameworks abound, tutorials are just a Google search away, and many computers come bundled with at least basic tools for simple web development. Now, no one's pretending that these kinds of tutorials and frameworks are going to launch the next space ship, but still, we've created a world where "anyone can be a developer".

This was the conversation I had with some friends - all of us software engineers - last week. And the second the phrase "anyone can do it" came out of our mouths, every single one of us cringed.

No why on earth would we do that?! After all, most of us have used these frameworks and languages. Heck, we still do! It's really convenient to stand up a quick Ruby on Rails application to accomplish some purpose. And it's not like some of these tools don't scale; just because the Java pet store tutorial exists doesn't mean Java can't (and doesn't) run some very large and complex applications.

But still, we all cringed.

In the end, I think the issue is with a coder versus a developer (or engineer; I use those two words interchangeably). There are a whole lot of people who can write code that I simply wouldn't hire to develop an application. Knowing how to code is important for anyone who works around computers all day, but there are a lot of other problems to solve when building something larger than a personal website (or a basic test script!).

After all, the coding part of software engineering doesn't have to be hard. Frameworks, libraries, tools and tutorials all make life a bit easier. It's understanding software that turns a coder into a developer. Knowing how a thread pool works lets a developer size it appropriately rather than just setting "10" because that's the default or that's what the tutorial says. Understanding the architecture of a system lets an engineer work on one part of it safely without having to sit down and digest thousands or millions of lines of code.

There's no major lesson here, other than yes, let's embrace the "anyone can do it" ethos. It will develop pique the interest of the next generation of great engineers. Just be aware that it gives you a coder; being an engineer is the next step you have to take, and you have to choose to take it on your own.

Tuesday, January 25, 2011

STPCon 2011 Early Bird Deadline

I feel like I just blinked and it was January 1, but here we are approaching the end of January. And that means spring conference season is just starting to gear up.

This spring I'm speaking at STPCon 2011 in March in Nashville, TN. I'm doing a session on "Agile in a Waterfall World". Scott Barber and I are also facilitating a pre-conference tutorial on the Business of Testing. (In my head that one translates to "so that's what they're thinking!")

I hope you'll come join us; it's going to be a great conference!

Early bird registration ends February 4th, and I think it gets you something like $400 off. So come!

I look forward to seeing you.

Monday, January 24, 2011

Failure Presentation

There are several phases in a bug's life. First, it is introduced. Then it is identified. Then it's fixed. Then verified. It all sounds nice and clean and simple.

It's not. We all know it's a bit messy in the middle steps. Bug identification, in particular can be quirky.

For example, I have a test we'll call "TestFoo". This is a piece of code that I run every night and that spits out a log file with some statistics and some checks. Over the past week, it has failed in the following ways:
  • One night it couldn't SSH to one of the machines involved, starting about 80% through its workflow
  • One night it expected a value X and got a value X-50.
  • One night it expected a value X and got a value X-48..... on a different branch of code.
So we have a bug! Or possibly two bugs.... or maybe even three bugs. It's kind of hard to tell based on the information we have currently.

What we have are failure presentations.

A failure presentation is the thing you see that is a problem. It is a presentation by the software of a state you don't expect. Often it indicates a bug is somewhere underneath there, but the failure itself is not the bug. Instead, the failure is a result of a bug. One bug may have many failure presentations. Several bugs may have the same failure presentation.

For example, the SSH failure I saw is a failure presentation. The underlying bug could be that the machine's hard drive died (aka not really a bug in our software at all). The underlying bug could be that our software crashed and caused a kernel panic so the machine is no longer responding to SSH requests. This is one failure presentation with several possible underlying causes.

The incorrect values I saw (one off by 50, one off by 48), are each failure presentations. They might be the same underlying bug with two slightly different presentations. Or they might be two different bugs.

The point is that until you understand the bug itself, and all the nasty things it might do, then all you can do is notice the way the failures present. The failure presentation can lead you to the bug, but don't be fooled into thinking you're there yet.

I find it particularly useful to distinguish failure presentations from bugs when working with development. I can log a bug, for example that says, "this is behavior blah in TestFoo. It looks like the failure in bug 123, which is TestBar, but has the same general behavior." By doing that, I have pointed out a possible link and helped narrow the bug, but I haven't made assumptions that they truly are the same thing. They simply present in a similar way. It's a relatively small but very useful distinction.

Thursday, January 20, 2011


For many projects, that glorious day comes when someone else actually uses your software. It's a really exciting day for the whole team; everyone from product management to development to test to support to implementations to management holds their breath for just a moment as the first user..... uses the software.

For enterprise projects this often takes the form of a pilot or an evaluation. For consumer projects it might be a beta test or simply sending it out into the world and letting someone download it. In either case, the feedback's gonna fly. There are many things you can do with this feedback and many responses that are completely legitimate. There's one thing you shouldn't do:

Try not to sound surprised when your "easy-to-use" software is actually easy to use.

If it fails, you can deal with it (how to respond is another post, but not uncommon). If it works, though, you want to inspire confidence. Being surprised that your software meets the claims you made does not inspire confidence.

Please avoid phrasing like, "I'm amazed how quickly you got it running" or "Oh, that's great; I'm relieved." If you accept that the claims you made have borne out, then your customers will be more confident, too.

Tuesday, January 18, 2011

Adaptive Naming

Every day we come up against new things. New techniques, new twists on existing techniques, new technologies, new procedures. Many of these things are adaptations of techniques, technologies, and procedures that existed elsewhere first. For example, today I got on the Silver Line to get to work. The Silver Line is a "subway line" here in Boston. It shows up along with the red, green, orange, and blue subway lines on the agency's website, etc. But there's a catch.

This is the Silver Line:

Yup, it's a bus. Now, we have bus lines in Boston, run by the same agency, but those show up in a different place on the website and run on different schedules. So I can say, "you call it a subway, but it's a bus!" all I like, and I will be correct. But I sure won't find the schedule.

This is a case of adaptive naming. I call it a subway because that is the name that has been chosen for it. It's technically not a subway - I don't think it goes underground - but "subway" is the commonly accepted term for the Silver Line, and I'll have a lot more luck getting information if I simply bow to that.

Adaptive naming goes on all the time. We call a software development process "kanban" or "lean" even though both those processes come from manufacturing. We talk about "user mode" even though a lot of programs run in that space that aren't explicitly requested by the user (cron, anyone?). The short fact is that there are a whole lot of new things out there and one of the ways we humans cope is by giving them names based on similar things. Thus, "kanban" in software is kind of like "kanban" in manufacturing, and our "silver line" bus is kind of like a subway train. It's not identical, but we've adapted the name.

It's useful to know when you're adapting a name, but don't be afraid of doing so. Similarity and pattern matching have helped humanity for thousands of years, and we can continue that similarity and pattern matching in what we call things, so it can help us understand each other for many years to come.

Friday, January 14, 2011

An Object Lesson in Unsafe Changes

Any change is risky. Some changes are more risky and others, and some changes have more visible risk than others. But every single change is a risk.

What follows is a true story.

We were working on a change that was a little bit hairy, fixing a bug in the way we handled some data on upgrade. We were able to write a test that showed the problem, and to fix it on the mainline code. Great, this was humming along nicely. Next step: merge it to a branch so we could get the fix into the next release.

The trouble is, we couldn't merge the test because it depended on some things that simply weren't in the branch. The fix itself merged onto the branch just fine, but we needed to find a way to test the fix on the branch. "Easy," I said, "we have a tool that prints out this information that already exists on the branch. We can use this tool and compare the data before and after upgrade. It's a bit tedious, but it'll be fine to get through this release."

As it turns out, it was almost that easy. We weren't printing all the information, but we were printing about 80% of it. So we just had to add a little bit to the tool, and do the merge, and we were off and running. We did the merge, added the extra printouts to the tool, ran a couple of quick tests, and checked it in. Simple!

(Here is where you should be hearing ominous music.)

The next morning, I came in to discover that:
  • the machine that runs the automated nightly tests was spewing out of memory errors and swapping. A lot.
  • one of the testers on my team was staring at his screen saying, "huh. That's weird."
But the only thing that got checked in on this branch was our change. Oh dear.

It didn't take long to figure out that the "simple printing change" had a bug in it. This small bug caused us to print out 13 MB of empty spaces every time we ran the test program in a particular configuration. Repeat that across the number of times we use this test program and all of a sudden it's using huge amounts of memory. Oh, and it's causing the tester's terminal to look like a bad '80s programming movie, with the cursor racing down his terminal.

The fix was trivial. But it was a reminder that nothing is safe. All we did, after all, was print something incorrectly some of the time, and we wreaked a little bit of localized havoc!

Fortunately, there weren't any long-term effects. It was a simple bug that we found almost immediately and were able to fix quickly. Most of the team was never affected and it certainly never hit the field. It did reiterate, though, why every change is a risk.

Mea culpa.

Tuesday, January 11, 2011

Big Is Not The Same As Comprehensive

I work a lot with companies doing evaluations of software. It generally goes something like this:
  • hear about shiny new technology
  • identify the internal need that the new shiny might fit
  • schedule a validation
  • put together a test plan
  • do the validation
  • write a report

And then life goes on. Now, there's risk in any validation. There are hidden agendas in any validation. However, there is one thing that I can almost always count on: the test plan will be large.

You see, no matter the real underlying agenda, the people soon the validation need to prove that they were thorough in order to successfully prove whatever point they're trying to make. So they have a large (and hopefully impressive looking) test plan.


A large test plan is not necessarily comprehensive.

Often, in fact, the test plan is limited in the things it does but does each of them in many configurations. For example, I recently reviewed a validation test plan that wanted to test every major UI workflow in each of 8 browsers. This is like a four week plan! It's huge! It doesn't at all address scaling or deployment or security or extensibility or code complexity or maintainability or .... in other words, it's big but not comprehensive. And plans like that aren't uncommon.

The fastest way I know to tell if your test plan is comprehensive at a glance is to see if it is a matrix. Matrices are great but generally filling out a matrix is not enough to tell you about all relevant aspects of a product.

So if you're the evaluator, check your validation test plan to make sure you're really covering multiple aspects of the software. and if you're the one being evaluated, don't necessarily fear a big plan: it may miss all your weak spots.

After all, bigger isn't always broader.

Monday, January 10, 2011

Faint Signals in Code Change Noise

We run tests every night. They run around the lab and do upgrades, ingest data, shutdown and restart and lots of other things. We come in the next morning and analyze anything that did something unexpected. In general it works quite well.

There's an interesting phenomenon that took me a little while to notice: weekends. For the most part "nothing changes" from Friday night to Monday morning. There are exceptions, of course, but most weekends there are no changes in the lab (hardware failure or the like) or changes in the code base. So with no changes, then weekends should be very consistent, right? Friday night's test rum should look the same as Sunday night's run.

That turns out to be a good source of information. Because everything should be near identical then variance becomes really interesting. We can see performance changes or intermittent bugs because the noise caused by changing code and changing environments is much less. All software gives out faint signals and with smaller noise we can listen.

If you have the opportunity to do the same thing in the same way, take it! Your software has things to tell you, even when nothing is changing.

Friday, January 7, 2011

Bus Test

Yesterday I asked someone to run a bus test. A what, now? Yup, a bus test.

Did you ever hear the saying, "What if you got run over by a bus?" It's a (rather morbid) way of saying that your hot-shot coworker, who is the only one who understands how foo works, may not be here tomorrow, and someone else had better know how to do it. You can talk about "what if so-and-so got run over by a bus" in several ways:
  • the bus factor: how many people know how to do X
  • the bus risk: how likely that a key person is going to leave
So what's a bus test? Simple:

A bus test is an exercise performed simply to show that someone can accomplish a task.

For example, at the office we have a bunch of data sets that we run to provide numbers for marketing and to analysts. (Yes, these do make the product look good. Surprise!) Right now, only one person, Bob*, knows what's in those data sets. He's the only one who can run the data sets in the right configuration to get the same output, and reproducing the output is important when an analyst is standing over you saying, "show me." He's also the only one who can describe the data sets to any level of detail (that's another thing those marketing and analyst types want). Our bus factor here is one. And that's not too good.

So we are running a bus test. Someone on my team will work from the existing documentation alone and will attempt to run the data sets and produce a report that matches the report we last sent to analysts. The only point of this is to prove that if Bob were to not come in tomorrow, then we could still produce the reports we need reliably.

(On a side note, talking about running a bus test leads to some really fun overheard conversations, going something like this, "No, you can't pair with Bob on this. After all, he might be kidnapped by aliens tonight. If you get stuck, then we know there's a problem and Bob will update the documentation.")

We don't have to run a bus test for everything. Many things are a product of institutional knowledge, or it's okay to have someone sit down and figure it out if necessary. For public things where consistency is key and it can't be contained entirely inside one runnable thing, though, consider a bus test.

* Name changed to protect the innocent.

Wednesday, January 5, 2011

How I Learned To Code

I wrote yesterday about writing code as a tool in the tester's toolbox. Phil brought up a great question: "That's nice, smartypants. How?" (I'm paraphrasing; he was more polite!). But he's right - how do you learn to code?

Learning to code is like learning anything else: people learn in different ways. Here's my story:

I left college working for a startup in open source software, and I didn't write any code. Time passed, the startup went out of business, I found a new job... and all of a sudden I find myself the sole dedicated tester at a software company. Oh, and they just bought a Quick Test Pro license.

This is where I started to learn to code.

First of all, I'm human. And I discovered that deploying all the projects to the server, setting up the configuration, and restarting the right services was (1) boring; and (2) error prone. A passing developer said, "why not just write a batch script to deploy it?" So I did. I and Google and lots of "echo this" and "why did it do that?" got the script written. And it worked. I was off to the races. Fire up QTP and.... well, shoot. Record/playback really doesn't work very well. The QTP help manual has a section that indicates you can modify your recordings. So I did. It was Java, but Google knows Java, too. I also got into the habit of stopping by developer desks and asking for help.

From there it was a fairly short step to "hey, I want a JUnit test that looks like that other JUnit test but does foo differently". From there to standalone programs for simple things was also a small step. Changing jobs and changing projects meant picking up new languages - Python, Perl, Ruby, Java - and new frameworks. As I learned, the code just got larger and larger (and eventually smaller and more efficient) and better and better structured.

To this day, I write code to solve simple tasks just to keep fresh during times when work doesn't have me coding. Sometimes it's writing a simple script to normalize my mp3 names, or writing a webapp for use in presenting a concept at a talk I'm giving at a conference. Oh, and I still ask people who are better coders than me to look at my code.

And that's how I got where I am.

So how does that help you? There are some things you can do to get started:
  1. Start small. Don't set out to "learn a language". Set out to solve a small problem.
  2. Don't worry about "good"; worry about "functional". "Good" will come as you get to be a stronger coder.
  3. Read a book or take a class if it'll help you. They're great for concepts and for the language to talk about coding. Keep in mind that these things supplement practice; they don't replace it.
  4. Don't be afraid to ask. Most people are really happy to help (unless they're behind on a deadline, and that's just a temporary state!).
  5. Break the problem down into small parts. The smaller the piece, the more likely you can solve it. Solve something small, string it together, refactor.
Give it a shot! The best way to learn to code is to start.

Tuesday, January 4, 2011

The Coding Tester

One of the things that most stands out to me when talking to testers is how variable their paths into test really are. There are testers who started in support and moved over. There are testers who were developers and decided test looked more fun. There are testers who got out of a computer science program in college, happened to land at testing job (usually with some sense of, "they want me, they really want me!" and never left). Some project managers and business analysts shift into testing. Every once in a while you'll get an end user who decides the software part is really fun and becomes a very knowledgable domain tester (ask me about the ex-nurse tester sometime - she was wickedly effective and incredibly good at convincing people to get the bugs she found fixed).

Of course, once you're a tester, how you got there is less relevant than your ability to do the job.* As systems change, the scope of skills we need to do our jobs changes, too. Most of us no longer have to worry about vacuum tubes or hardware wiring. More of us have to worry about APIs and load.

And here's my point: If you want to be a tester, know how to write reasonable code.

The systems I see today are getting both larger and smaller simultaneously. A lot of the software I see is moving toward the module UNIX (and variants) have been using for 30 years: many little components all working together. Every system is smaller, and does rather less - Twitter, for example, really only takes 140 characters in one input. Every system is also bigger, and combines with many more components - count, for example, the number of different programs that use Twitter's APIs.

When humans interact with a program, it's often through a GUI or a command line. That's great. But more and more, programs interact with other programs, and in order to test that, we're going to have to write some code.

We have to write code to:
  • test APIs (our "user" is another program; i.e., some code)
  • simulate things too big to do by hand (load testing, anyone?)
  • break down components and test each of them (mocking, etc.)
Now I'm not saying coding is the only thing we do, or that it's a solution to all problems. I'm saying it's a tool in our toolbox just like the ability to write a coherent email or to explain a difference in behavior. We're not going to get very far without it.

So do yourself a favor. Don't go out to become a developer, but do learn to write some code. It's another tool that your toolbox needs.

* There are many ideas about what a tester's job is and isn't. That's not the point of this post so I'm just not going to quibble about it.