Thursday, January 31, 2008

Push Versus Pull

I've written before about the queues I work with, and how I touch each of them every day. One of the things I'm playing with is whether I want to deal with the queue by pushing or by pulling.

If I go with a push method, all changes to each queue get emailed to me. If I go with a pull method, I go look at each queue.

Push Pros:
  • I never miss a change.
  • I can work my list when I'm not VPN'd in to the office.
Push Cons:
  • Significantly greater email
  • When there are multiple changes in the queue I get extraneous notifications
Pull Pros:
  • Keep all my queues in their place - in the queue, not in my email
  • Doesn't drown my other emails
Pull Cons:
  • Much higher chance I'll miss something, or be slow on something.
  • Can't get to it from outside the VPN
I haven't decided which I like better - push or pull. Any thoughts?

Wednesday, January 30, 2008

Overhead and Velocity

We've been refining our velocity measurements at work lately. Velocity for a dev team is relatively simple: any work performed on the implementation of a story is counted as part of the team's velocity. This includes design work, implementation, test implementation, documentation of the story, etc. This does not include fixing bugs found after acceptance, helping support, daily standups, etc.

Velocity for a QA team is a bit harder. To hew to the equivalent of the dev team's definition, we would count work performed to accept stories, and work performed on QA stories ("develop a user acceptance test plan", "certify our software with another vendor's software", etc). But is this really sufficient? We've modified the definition a bit to be the following:

QA velocity includes activities that are defining, performing, and analyzing the results of tests.

Things that are included in our velocity measurement:
  • Test plan development
  • Test automation
  • Performing manual tests
  • Going through automated test results to identify new and recurring issues
  • Accepting stories
  • Creating acceptance criteria for stories
Things that are not included in our velocity measurement:
  • Helping support diagnose a problem
  • Verifying bugs that are not part of stories (so if a bug became a story, it counts. If the bug didn't, then it doesn't count)
  • Standups
  • Attending other team planning meetings
In the end, velocity isn't about the absolute number. It isn't even about whether the number matches the dev team's number. It's about whether the number goes up or down over time. And it's about using the number to figure out how far down our queue we're going to get this iteration.

So don't sweat what goes in the number. Just define your velocity and go with it. Then watch how the number changes and how it works with your estimates.

Tuesday, January 29, 2008

Every Release Has Two Scares

In every release I've ever been involved with, there have been two scares:

The first scare occurs late in testing. This is the one where you think you've found a showstopper bug and there is definitely not time to get it fixed and retest sufficiently. Here's how to handle it:
  1. calm down
  2. reproduce it. If your testing is good, about 60% of the time you've just scared yourself and there isn't really a bug, or the bug isn't as big as it looked.
  3. take it to your leads, and do it fast. Your boss, your product manager, your dev lead all need to know. Sitting on the bug will only make them feel more blindsided when they do find out about it and then find out that you've known for a while.
  4. find a workaround. Do this after everyone knows there's an issue. This will help you understand how necessary it is to fix it.
  5. figure out whether it needs to be fixed. If so, start having the date slip discussion as you're figuring this out.

The second scare occurs within the first two weeks after you release. This is the one where the next guy down the line (support, implementations, or your client usually), thinks they've found a showstopper bug.... and you missed it. Handle this the same way you handled the first scare - calm down, make sure it's really a bug and not just people panicking, and then deal with it.

When you're planning a release, don't forget to plan in time for two scares - better to anticipate it and not need it than to need it and not be able to handle it.

Monday, January 28, 2008

Day 1, Project 1

I've been hiring a lot recently - we've gone from 1 to 4 people in less than 6 months (yipes!). One thing I'm slowly learning is how to deal with a person's first day.

  • Give them a project. This is the most important thing. It provides focus and a way to learn. Make sure it's not a dull backwater project, but it should be something that can take a few days.
  • Do the HR stuff. They'll be worried about their benefits, etc., if they don't just get it out of the way.
  • Tell them to "just read the manual". BORING.
  • Have them "play with the product". Also kind of dull and directionless. Useful, sure, but hard to do without some sort of map or guideline.
  • Ask them to "settle into their computer". For some people, this is a real project. For most testers, though, that means finding email, IM, the defect tracking system, the test plans, and source code. Done. This won't really take long.
If you expect people to hit the ground running, they will. So turn the treadmill on for 'em.

Thursday, January 24, 2008

The Day After

Today is the day after we have shipped from QA to deployment (they do some training, and then it's off to customers!). 

So what did the QA team do, this day after?

  1. went out to lunch together
  2. cleaned up the wiki page to reflect the release as complete instead of in progress
  3. burned archive CDs of the release
  4. archived the test plan (including test results)
  5. answered questions from dev, product management, deployment, sales, etc.
And that's about it.

If your team has worked hard through the release, there's probably a mountain of work still to be done. But take a day - just one day - as a breather. There needs to be a lull after release so you can release all the stress.

In the end you'll gain more for that than you'll lose with a few hours of inefficiency.

Wednesday, January 23, 2008

Mac, PC, and Linux

I recently got an email asking about using macs for testing, and I thought I'd share my response to him with everyone else. I know this would have been better timed if I'd gotten it up a bit closer to Macworld, but better late than never!

In my current job, I have the following personal machines:
  • MacBook Pro (15")
  • Debian Woody desktop
  • Windows 2003 Server on a 1U rack mount box (this one isn't on my desk, obviously!)
I use the macbook as my laptop/business stuff machine. It does email, web, some coding, etc. I use the Linux desktop for the majority of my testing and for a lot of my coding. There's some wiki maintenance and other browsing there as well. I use the Windows machine for Windows testing (CIFS shares, Active Directory, etc).

The development environment here is pretty much entirely Linux. The dev leads all have macs, and there are a few Windows and Solaris test machines, but otherwise it's all Linux all the time. The business side uses Windows exclusively.

Pros and Cons - Mac
The Mac has a lot going for it and has some significant downsides.
  • Easy to use. Most of the programs I install just plain work.
  • UNIX underpinnings make interacting with the dev environment easy. There's no extra SSH client to install or other utilities to configure, like there is on Windows. I just open a terminal and I have bash on a mac that works as well as bash on my Linux machine.
  • Pretty. The mac UI is pretty, right down to the Dashboard.
  • Battery life. The mac battery lasts at least twice as long as any battery in any Windows or Linux laptop I've ever had.
  • Tendency to hang. This is worse with Leopard than it was with Tiger. I've noticed that in particular when I have stale NFS mounts, Leopard will hang to the point that I have to hard reboot the machine.
  • Less than ideal interaction with Windows. This is true in particular of interaction with Exchange. For example, I get meeting notifications as emails with attached .ics files; it works pretty well, but it is an extra step. I also have trouble with line endings.
  • Office isn't as good as on Windows. I'm a power user of Word and Excel in particular, and the mac versions of these programs just don't measure up. In Word formatting gets a bit odd and margins in particular don't line up the same way. In Excel some calculations don't come across at all, only the results of the functions. (For the record, Pages and Numbers are even less compatible.)
Pros and Cons - Linux
The Linux machine I use is a good workhorse but takes a lot of fussing.
  • Incredibly configurable. If you want to configure it, you can. I can tweak this thing until it's completely and utterly mine.
  • Doesn't believe in crashing. This is definitely my most stable computer, even though it's a test machine and I do some pretty weird things to it.
  • Time-consuming. All that configuration is great, but I find that I have to do a lot more configuration to get programs to run well, and that takes time.
  • Sensitivity to hardware changes. With my Linux box, changing hardware is quite likely to require a new driver or something. It's definitely a lot less plug-and-play than my mac or my Windows machine.

Pros and Cons - Windows
I should note that this is Windows server for testing, not a Windows client OS.
  • Some things just can't be done on other environments. I just can't test the behavior of Windows Explorer on Linux or mac. That alone will keep me on Windows.
  • Very popular with our clients. A large chunk of the market is Windows client machines, and that has power.
  • Good hardware compatibility. I can swap out hardware on this box and rarely have to get my own drivers or other configuration changes. 
  • Doesn't fit well into our dev environment. Many of our helper scripts, etc. are shell scripts and simply won't run on Windows (without cygwin or the like). There are extra hoops we have to jump through here.
  • Logging and errors aren't as clear. It's harder sometimes to dig in and figure out what's going on because Windows logging is a bit lighter than on Linux or mac in many places.

In Conclusion
I'm not a platform zealot; I'm not going to start a flame war favoring mac or Windows or Linux. I will say that each OS is very different and each one works better in some environments than in others. There's a lot of talk about context-driven testing these days. Let's try context-driven environments for a while.

Tuesday, January 22, 2008

Keep Your Hand In

I'm a manager now, for all intents and purposes. My primary job is to build a team, help them grow their skillsets, interface with the rest of the company (how's that for management speak!) on behalf of QA, etc.

But I always test every release.

It's important as a manager to keep your hand in, to stay involved with the actual work your employees do.

There are a lot of advantages to making sure you're still a tester, just like you're also a manager:
  • Timeliness. If you can still test, you can be the hands that help your team through a crisis. If you're the one who caused that crisis by overpromising or misestimating, this is particularly important - you broke it, you darn well better help fix it. Even if the crisis is externally driven, your testing can make sure that your team still comes in on time and with the level of testing you need.
  • Respect. Your team will respect someone who can walk in their shoes, who understands the ins and outs of testing. This is why I'm not a big fan of having a manager come in with no test experience. They don't have to have been a tester, but they need to have worked very closely with testers.
  • BS Meter. You hope this never happens, but sometimes one of your team members will try to lead you astray. Often this is saying something was tested when it wasn't thorough enough, or overestimating how long a test will take. If you still test, your skills and instincts will be fresh enough that you can call them on it.
  • Keep Current. Testing, like all software engineering, changes fast. The techniques, the tools, the languages, the sensitivities of the systems under test: all that will be different in two years. The best way to keep up is to keep testing.
There are some disadvantages to being a tester manager, however:
  • Longer Hours. Keep in mind that testing is on top of managing your team, not instead of. You'll find yourself doing most of your testing off hours.
  • Bailouts. If your team feels like you'll bail them out to make a deadline, they may decide to let you. Make sure that you're not letting them slack off while you do their work for them.
All in all, I prefer to be a manager who still tests, not just a manager.

Monday, January 21, 2008

Snow Day!

In honor of Martin Luther King day, I spent the weekend at Bretton Woods, in New Hampshire. Snowshoeing, long snowy hikes, dogsledding, and cocoa by the fire wore me out!

So long and short, no blog today. Happy MLK day!

Thursday, January 17, 2008

Little Rewards Trump Big Rewards

I have a large enough team now that I have a bonus pool. I can distribute that pool however I like. There are two ways I could handle this:
  • Lump Sum Rewards. 
  • Small Rewards More Often
Lump Sum Rewards
The advantage of lump sum rewards is that it feels like a lot of money: "Wow! I got a $5,000 bonus this year!" (or whatever it is). This is the kind of reward that's (a) noticeable; and (b) something people will talk up inside and outside the company. It can substantially improve morale and your group's perceived status.

The bad news is that you can only afford to do this sort of thing once a year or so. So for 50 weeks out of the year you get nothing, and for 2 weeks out of the year you have a team happy with the glow of a windfall. Also, if you repeat this over enough years, your team will come to know and expect the bonus, and to compare it to last year's bonus. It takes more and more to get the same effect. When you're talking a $5000 bonus, giving $5100 doesn't look that much better. This can turn very expensive very quickly.

Small Rewards More Often
This is my preferred method of rewarding. I take the pool and split it into roughly monthly amounts. And then I spend it on a lot of different things, none of them large. 

One person on your team came in over the weekend to finish up that big project? Awesome. Here's a $50 gift card.

The team got a release out on time? Cool. Let's all go out to lunch one day. Total cost (depending on team size), about $250.

A couple of people on your team have been really helping out the more junior people? Let them know you notice: $100 each.

Random Thursday? Great. 6 pints of ice cream to perk up the afternoon. Or maybe USB keys for everybody.

Structuring rewards around non-monetary things, or around small monetary amounts keeps the rewards coming. It lets you tie work to reward very closely, since you don't have to wait until "bonus time" comes around. I find that I use my bonus pool up a lot less quickly this way, but I have a stash that I can use to instantly reward good work. Overall, the morale of my team goes up.

The downside to this is that your team winds up bragging about the quantity of rewards rather than this windfall that they got. It doesn't do as much to make the people outside your team jealous. However, if a potential team member comes along, there will be a story about good work getting noticed early and often, and that's the kind of long-term benefit I'm looking for.

Wednesday, January 16, 2008

The Underwritten Bug

In direct contrast to my earlier post about the overwritten bug, today we take a look the underwritten bug.

Writing a bug shouldn't be the beginning of a Sherlock Holmes novel. You don't want to force the developer to make and test assumptions. First, this makes things take a lot longer. Second, you only piss off your developer over time.

This, for example, is an underwritten bug:
Updated time overlaps status info

I ran the upgrader and saw this.

So what's bad about this bug?
  • It doesn't tell you what the problem is.
  • With GUI bugs, screenshots are rarely a bad idea.
  • It doesn't tell you what the desired behavior is.
  • If there are any special conditions required to get this bug to occur (e.g., browser version, certain time of day), they aren't mentioned here.
If you have to pick between an overwritten bug and an underwritten bug, choose an overwritten bug. However, the better way is to choose a well-written bug.

Tuesday, January 15, 2008

You Will Fail

One of the most difficult things about being a QA engineer is knowing that ultimately you will fail. There will be some bug that you don't find and that customers do.

So how do you handle it?

Set your sights 10% higher than you could possibly achieve. When you fail to achieve that, you will have still achieved a good goal, and your failures will be acceptable.

More on this one later. In the meantime, buck up!

Monday, January 14, 2008

Deadlines Don't Move

Yesterday was a snow day here in Boston. 600 schools closed, 7.0" of snow, a lot of people out. Our office was closed as well.


I walk to work, and I came in.


Deadlines don't move just because it snows.

Happy snow day, everybody!

Friday, January 11, 2008

Power of the Empty List

I generally work out of a series of lists. There are lists of bugs that are ready for QA, lists of stories that need acceptance, lists of tests that need to be automated, and emails in my inbox. This way everything I have to do is written down in at least one place. There are still a lot of lists, though, as I wrote about in an earlier blog entry. So how to handle it all?

The most efficient list is an empty list.

If I have a list with many items on it, I waste a lot of time going through the list. A simple example:

It takes 30 seconds, on average, to look at a bug and figure out what kind of test I need to do, if there's a question, if I'm waiting for a build, etc. If I have 50 bugs in my list, it takes 25 minutes just to figure out which bugs I can work on - and I have to do that every time I look at my list! So I only look at each bug once, and I put it into a different state depending on what the next action is. I've lost the 25 minutes, but only once. The incoming queue is then divided into several lists: "needs build X", "to answer", "test", etc. Each of those lists contain items for which I can either do all of them or none of them. Either way, I can now treat the list as a single item and save myself a lot of time.

Another example:
It takes about 60 seconds to look at an email and decide how to deal with it. I've seen people with thousands of emails in their Inboxes. At 60 seconds per read, it's no wonder people don't get much else done! So I deal with it as follows:
- if I can answer in under 5 minutes, I do it
- if it's informational, I file it (I use Lookout on Windows and Spotlight on the mac - searching later is easy if I really need to)
- if I can't deal with it, I put it on my calendar for a time when I will be able to deal with it, either because I'll have the information I need or because I'll have the time to handle the issue.
Every time I check mail, I try to get my inbox to 0, and I usually succeed. At most I'll have 10 or so emails in my inbox.

The less time you spend fussing with your work lists, the more time you have to work. So every time you look at a list, clear it out.

Thursday, January 10, 2008


I call myself a pragmatic tester, but it occurs to me that I've never actually explained what I mean.

Let's start with the basics: "pragmatic"
Dealing or concerned with facts or actual occurrences; practical.

Pragmatic QA is about doing what works. Shipping product well is paramount. How you do it is less important. Just get it done.
  • If you have to change your process, do it.
  • If you have to alter your test plan to account for new information, do it.
  • If you have to push back your test schedule to allow for the last minute "forgot it" feature, do it.
  • If you have to restart testing of an area because you just found a big bug, just let people know and reset expectations.

Oh, and do it all cheerfully.

Wednesday, January 9, 2008

Ingredients for a Solid Team

There are three things you need to build a team that will ship great software:
  1. Good coders. If your team isn't talented, nothing can save them. Your developers should be good coders; your testers should be good testers; your architects should be good designers, etc.
  2. Good System Understanding. You need to know the system from top to bottom, inside and out. Every member of your team should be able to describe the system in detail, from a user's (black box) perspective, all the way to the inner workings and interactions of the components. Your problems almost always lie at the outermost (black box) and innermost (component interaction) level. Note the semi-gratuitous zoom video describing a system and how your team should be able to traverse it!
  3. Commitment. If someone on your team says he will do something, then the entire team needs to be able to trust that commitment. A good team is one in which every member says something will get done and it does get done on time every time. Trust is only built by keeping your end of the bargain all the time.
Of these, the first two are mostly a matter of hiring the right people and helping them get to know your system. The last part is the hardest, and regularly dooms groups of really smart people. If you're lacking any of these three, you are shipping software in spite of your team, not because of it.

Tuesday, January 8, 2008

Tricks Only Work Once

As a tester, your tricks should only work once.

Every technique you apply is a "trick". It's a tool, or a way of looking at things, or a specific test design mechanism. Each trick should only work once. If it works multiple times, then you're not teaching your developers to properly learn from the bugs you find.

The good news is, this keeps you having to come up with new tricks. And new tricks find new (and old!) bugs.

Just a thought.

Monday, January 7, 2008

The Overwritten Bug

When you're writing a bug, detail is great. However, there is such a thing as drowning the bug reader in too much information. Your job as a QA engineer is to provide enough detail do diagnose and resolve the issue, and not more.

This, for example, is an overwitten bug:

Login error when I use a certain password

Here's how to reproduce the bug:
1. Log in to a Windows XP machine. Username "foo", password "bar".
2. Click the Start button
3. click the Internet option (note: this should open IE 7. If it doesn't call QA.)
4. Wait for the browser to open
5. Click in the address bar
6. type ""
7. Press enter
8. Wait for the page to load. You should see a login screen with a username field and a password field. The username field should be 2" long (at 1024x768), the password field should be as long as the username field. The username field should be labeled "Username". The password field should be labeled "Password". Both fields should be blank.
9. Click in the username field
10. Type the username "foo"
11. Click in the password field.
12. Type a bad password.
13. Click the Password button. Do not press enter! Click it. Be sure you click on the letters on the button, not on the background.
14. Wait for the error to load.
15. The login page shows with an error in read. The error says "Wrong".

Logs from the machine are attached. I could only get the last week.

So what's bad about this bug?
  • It's way way too long. It's hard to find the information that matters among all the noise.
  • It doesn't tell you what you need to know. What is the bug really reporting as the error?
  • It doesn't tell you what should happen. The expected behavior is undefined.
There's a lot out there about things you should put in a bug, but be careful to not put too much in.

Friday, January 4, 2008

Who Cares About Test Types?

I've been thinking about different kinds of tests and their audiences. What do these kinds of tests tell us, and to whom is each speaking?

Unit Tests
First, a working definition: unit tests exercise internals of the code, typically on a per-method level. These are also called white box tests.

These tests are useful for:
  • Identifying that your method does what it thinks it does. This is particularly useful when you are doing math or logic inside your method. It's not so useful when all your method does is pass along an input to an output.*
  • Confirming that your method takes what it needs to take an outputs the correct things. The act of writing a unit test will force you to think about your function, and that can catch out the silly mistakes like needing an input you don't have, etc.
  • Testing basic code practices. Unit tests are a great place to check that you handle nulls, empty strings, etc. properly.
System Tests
Again, a working definition: system tests use the system as your customers will, whether your customer is another team within the company or external, and whether your interface is an API, a GUI, a CLI, or whatever. These, ultimately, are the tests your customers care about.

  • Confirming that the system meets a customer's needs. These tests prove that when the customer does something, the proper thing comes out the other end. How you got there doesn't really matter; that you got there (in a reasonable time and with reasonable resource usage!) is what this test cares about.
  • Confirming that you can really handle external users who don't know the subtleties of implementation. Since users don't know the internals of your code, they'll send in odd values, missing values, null values, etc. These tests confirm that the system can handle that.
In the end, your customer only cares about system tests. They're what tells your customer whether the system will do what it's supposed to do. Unit tests tell you about how the system does what it does. These are tests that are for the developer, and provide a good cushion under which the system can grow and change.

* I know some people disagree with me on this, and it's certainly simpler from a code coverage perspective to unit test all methods. I don't think it's harmful, but it doesn't seem overly helpful to have a unit test that simply reimplements the method. Net net, if you have the time or if you do a lot of refactoring, then go for it. If you don't, then just be sure you test your high-risk methods first (just like all testing, riskiest first!).

Thursday, January 3, 2008

Bug of the Day

This is a bug I found today, and it might be my most amusing bug ever:

We have a feature that lets you pick a date. I logged in today and noticed that I couldn't pick today.... or yesterday. I couldn't pick any date in 2008.

Here's the code (note: real code not shown to prevent IP issues):

* Writes HTML code for the date form. Combo-boxes
* are shown for the year, month and day after which stuff should be done.
* A JavaScript calendar is also included.

private static void showDateForm (Locale locale) {
Calendar mycal = new calendar(locale)
//move time back a week
int nowyear = mycal.YEAR
int nowmonth = mycal.MONTH
int nowday = mycal.DAY

No, we don't know why they did this. Our best guess is that the date defaults to one week ago and they kind of screwed up the implementation of it.

I'm amused. :)

Wednesday, January 2, 2008

An Interview Question

Okay, let's get it out of the way....

First post of 2008!

Now on to the real post...

One thing that I look for in a QA engineer is an ability to understand the system under test on a number of levels. A successful QA engineer will be able to see the system as a whole and how its component parts may interact with each other. That same QA engineer will be able to describe the system as each component part sees it - imagine a package-level tour, with our friendly local QA engineer as a tour guide. This helps find bugs on a micro level (class X extends class Y, which allows for Z improper use) and bugs on a macro level (when component A goes down while component B is processing intake from the file system, then component A will not properly update component C and component C will eventually crash because it doesn't realize the file system has changed).

How do you find this out in an interview?

What Doesn't Work:
  • Asking about systems they've worked on. I simply don't know those systems well enough to state whether there's true knowledge there or not. It's hard to probe them with any detail.
  • Games. It can be effective if you ask someone to describe, say, Monopoly. The problem is when you run up against candidates who don't know the game.
What Does Work:
  • Ask them to describe how a common household object works. I like asking how a blender works, or how a fridge works. These are things that most people know but don't know the details of.
  • Ask them to describe a physical thing they can see but can't take apart. In this case I can hand them the object. I've done hair dryers and laptops. *
I'm still trying to figure out how to get it to apply to less mechanical systems and more software systems.

Any ideas?

* In a few cases where the item has been a non-computer or other non-job related item, I've had good candidates get offended. I'm not totally sure why, but it biases the candidate against us and me against the candidate. More and more I'm trying to avoid non-tech subjects as unfairly prejudicial.