The Embarrassment Test

When you release software, there's usually some quality bar that must be met. This is often expressed in terms of clean runs of automated tests, number and type of tests performed and passed, existence of supporting documentation, etc.

There's one more test: the embarrassment test. It's a simple test:

Will I be embarrassed if this build ships?

The embarrassment test gets to the heart of what you care about in a given release, showing the quality bar that matters to you.

For example, I'm working on a product that has simplicity of API as one of its core principles. So my embarrassment test includes looking at the API. Anything that seems sloppy, inconsistent, or kludgy is embarrassing. That's not something that would appear in a traditional release checklist, but it's something for which we will hold a release.

What's in your embarrassment test?

Testing Error Messages

There is a type of test in which the tester attempts to do something erroneous and makes sure that the software fails gracefully with a helpful error message. This could be anything from an invalid entry in a form field to a network connection error to an illegal operation.

The tests generally do something that should result in an error. Then they confirm that they get the expected error. There's one more step, though:

An error message is only complete if it provides the recipient with a course of action.

When an error occurs, the user hasn't been able to do whatever he was trying to do. To complete his task, the user must correct the error or find a way around it. The error message is the first guidance to doing this. The error message needs to be unambiguous and to correctly indicate how fatal it is.

For example, maybe a user enters the password "foo":

BAD: The password is invalid.

BETTER: Passwords must be at least 8 characters and contain one letter and one number.

Or maybe there was a network connection drop:

BAD: Error writing to server

BETTER: Connection lost to server foo. Please check your network settings and retry.

Or perhaps you just crashed a component of the system:

BAD: (Stack trace)

BETTER: A fatal error occurred in foo. Please contact support.

When you're testing (or writing, for that matter) error messages, make sure each error meets two criteria. It must properly throw an error, and it must be an error the user can correctly do something about.

Reopen vs Duplicate

Sometimes you'll find a bug that looks a lot like an existing bug. Maybe it's the same test. Maybe it shows the same symptom, even. For example, maybe you have a test that starts, connects to a remote server, gets some information, and checks that information. The test is failing with a "connection refused" error, and that sure looks like the "connection refused" error you saw last week.

At this point, you have two choices:

• Log it separately, and risk creating a duplicate
• Log it in the existing issue, and risk conflating two issues into one

Either way, you're taking a risk: of a duplicate, or of a false reopen. So what do you do?

Go with the risk of a duplicate, because duplicates are cheaper.

Let's say that risk is realized, and the bug is a duplicate. Now you have to close the bug as a duplicate, and reference the duplicate bug from the original. Annoying, sure, and a little bit of time wasted confirming that the bug is a duplicate, but in the grand scheme of things, not a big deal.

If you'd gone the other way, and reopened a bug, you might have realized the risk associated with that: maybe it's a different bug. In this case you have to open a new bug, comment/edit/delete the erroneous reporting in the old bug, close the old bug again, and move any referenced logs or other information. Oh yeah, and now the developer who closed the bug is just a little bit insulted that you thought his fix was incomplete.

If you're very sure that the same root cause underlies two failures, please go ahead and reopen the bug. If you're not sure - like our "connection refused" example, which could have several different causes ranging from network failure to configuration problem to server failure - go ahead and log a new bug. It's easier to find out you have a duplicate than to peel apart two bugs that got conflated into one.

Tidy

Normally, we run two versions of our software in the general lab pool at any point in time:

• HEAD. The unstable, next generation stuff.
• RELEASE. The stable version that's going out next or is just out.

Other things - experiments, prototypes, patches on older branches, etc. - are done on separate branches, but for the most part they don't run in the general lab pool. Because we have these two main items running in the pool, many of our utilities differentiate between head and release. This allows us to graph performance results on head separately from on release, for example.

All good so far.

But.

We've recently decided to run two releases simultaneously for some things. So now our lab pool looks like this:

• HEAD. The unstable, next generation stuff.
• BRANCH-A. The stable version that's going out next or is just out.
• BRANCH-B. The stable version going out just after branch A.

This means we need to update various utilities, graphing tools, etc. to handle two releases instead of one. The simple way to do this would be to do it in an additive manner.

Our test reporting tool used to look like this, for example:

if (head) {

do stuff

elsif (release) {

do other stuff

else {

you must be doing one of your wacky side projects

}

I could have done the simple cheating thing and made it look like this:

if (head) {

do stuff

elsif (release) {

do BRANCH A stuff

elsif (branch B) {

do BRANCH B stuff

else {

you must be doing one of your wacky side projects

}

It would have worked just fine. It would have been wrong.

When you're working in code, keep it tidy. Yes, test code, too.

I made it look like this:

if (head) {

do stuff

elsif (branch A) {

do BRANCH A stuff

elsif (branch B) {

do BRANCH B stuff

elsif (release) {

notify QA we missed a spot

else {

you must be doing one of your wacky side projects

}

That's tidier. I also added a todo and will go back in about a week and get rid of that release clause, because by then I'll know if we've missed anything.

Yes, it's test code. That's no reason to not be tidy about changing it. Keep your code clean, in test just as you do in the product.

Full Screen

Sometimes - particularly on a beautiful Friday afternoon when the tree outside my window is blooming - I have a bit of trouble focusing. I'd like to, but it's just so pretty out, and there are emails coming in and ... and... and...

Okay, time for forced focus. It's time to go full screen.

This is the world's simplest focus technique I know: just hide everything else.

I close the window blinds. I turn off the monitor on my other machine. Most importantly, I put the window I'm working in on full screen mode. That way I can't see anything but what I'm working on. It's similar to tools like WriteRoom, but in this case it's for my IDE.

When you really want to focus, but can't, consider going full screen. It almost always helps me.

STPCon Call For Proposals

The Software Test and Performance Conference for 2010 is in October, and the call for speaker proposals has just gone out. I'm on the conference board, and I'm really looking forward to a whole lot of variety in sessions; it should be a great conference!

The call for proposals follows:

You are invited to submit a speaker proposal for the leading event in the software test and quality assurance profession – STP 2010 Annual Conference and Expo (October 18–21 in Las Vegas, Nevada).

STP needs you – active software test and quality assurance practitioners, consultants, professional speakers, trainers, and solution providers – to share your expertise with your peers at one of the many breakout sessions that will be featured at this year's event.

We expect to receive more than 100 submissions for this year's event. STP's conference program board will review and select from these submissions the topics that support existing and emerging trends facing our industry today. We are seeking session proposals with solid content featuring specific how–to's, and valuable take-aways. We are interested in sessions from industry leaders who have a particular expertise they can share, one that sheds light on an issue, practice or process in our industry. Product pitches will not be considered a viable breakout session. Strong platform and presentation skills are a must!

It does take work to put together a good presentation, but the rewards are great! Being chosen to be part of the STP 2010 Annual Conference faculty is an honor that is fun and career enhancing. If selected to present at STP 2010, you will receive a complimentary registration to the conference in addition to the coveted industry expert status as an STP 2010 faculty member.

If you have a session topic that meet's STP's standards, please visit our website at: https://stpcollaborative.wufoo.com/forms/stp-2010-conference-expo-call-for-proposals/.

For consideration, please complete the online proposal form by May 5, 2010. Feel free to submit more than one topic. We look forward to receiving your proposal and hope to see you in Vegas in October!

Pass this email along to any of your colleagues who you believe would be an outstanding speaker at STP's 2010 Annual Conference and Expo.

Thank you,

STP 2010 Conference & Expo Team

Pre and Post Plan

I've had a lot of comments recently that have been variations on the theme:

"Requirements should be detailed, and should be set well before testers ever see the software."

I can see where that impulse comes from; it sure sounds like a nice world where we all - from product management to development to test to support - know what we're going to do and then do it and then go home and have a nice supper. News flash:

It's not going to happen.

This idea of preplanning is at best idealistic. It can also be dangerous, since it leaves you without the ability to handle changes after that early detailed planning. In the real world, you can expect to see some or all of the following:

• Some behavior that is entirely correct according to the requirements but has an odd consequence.
• Use cases in the field that you never imagined and that want something just a bit different than the requirements you specified.
• Requirements that seem detailed but leave out some point of clarity that doesn't get noticed until development or until test.
• Requirements that change due to some late-breaking information.
• Requirements that someone thought was going to satisfy a customer's need but don't quite actually do so.

All of these add up to two things: you will need to resolve ambiguity, and you will need to handle change in requirements. Fortunately, there are simple things you can do here:

• Communicate. Make sure there is a project meeting including everyone from product management to support at least once a week where you can talk through ambiguities and changes. Don't be shy about talking with other disciplines outside this meeting, too.
• Give Yourself Some Space. Changes and ambiguities generally mean re-testing or expanding your tests. Give yourself a little padding time for that.
• Test Use, Not Requirements. Make sure you test based on customer use cases in rather than the requirements that are derived from those cases. Requirements can be tested directly, but users ultimately care about how the system works as a whole more than about a single requirement.

On Reference Checks

When I hire someone, like many people, I do reference checks. When I call a reference, I get one of three replies:

• a positive review
• a negative review
• a review of faint praise

If I get a positive review, it's generally a good reference. This is the review you want, where the reviewer is almost gushing and wants to work with the candidate again. Look for words and phrases like "great", "best", and "would hire again if this person hadn't moved across the country".

If I get a negative review, I'm almost never going to hire the person. A negative reference says not only that the candidate has caused at least some problems, but also that the candidate cannot properly pick a reference. This person is a potential problem in the workplace and has shown bad personal judgement. Run away!

If I get a review with faint praise, then I'm not likely to hire. Terms and phrases like "fine", "good", and "middle team member" aren't explicitly bad, but certainly point out that the candidate wasn't a great contributor. Keep in mind that the person giving the reference is someone that the candidate specifically chose to show how good the candidate really is. If faint praise is the best the candidate could get, again, that creates a lot of doubt about the candidate.

There are no hard-and-fast rules with reference checks. But keep an eye out not only for negative reviews, but also for a candidate who gets damned with faint praise.

Detailed Requirements

Requirements are a lot of fun due to their sheer variety. "The system must support Active Directory" is a requirement. "Add a column to the database that indicates deleted or not" is a requirement. Both of these are perfectly good requirements - assuming someone actually wants them. However, your customer is probably going to look at you funny if you talk about internal system requirements (e.g., that deleted flag in the database), and your developer is probably going to ask for clarification if that Active Directory requirement shows up.

Because testers talk with so many different groups - customers, product management, development, support - it's useful to know how much detail to express in a requirement.

For example, the customer may have a requirement that users be grouped and reported on by company.

The product manager would express that requirement as needing "company" on the signup form and a report in the administration section called "Users by Company".

The designer would express the requirement by adding a field to the signup form (just under last name) and a simple list report.

The development team would express the requirement with several tasks, such as: add a field to the database and migration script for deployment; add a field to the signup form; add a report including logic to handle legacy users.

They're all requirements; they're all the same requirement. And they're all correct.

Requirements should be expressed to the granularity that the person expressing the requirement understands the workings of the system.

In a nutshell, when you're talking to someone, talk to them about requirements in a language and at a level of detail that person will understand.

Start With the Bug

Many times when we test, we start by looking at a feature or a module. We start with a thing we're trying to do:

• an action we're trying to take
• a configuration we're attempting to run
• etc.

This action then either completes successfully or it reveals a bug. Great!

Sometimes, though, it's useful to start with the bug. When approaching the system, conceive of a bug that you might find. Use your knowledge of the architecture of the system to seek out some potential issue. For example, you might believe that there could be a race condition in saving data and reporting. Then design a test that would reveal the bug if it's present.

Starting with the bug is a valuable technique for when only a narrow range of tests or conditions will expose the bug. You're unlikely to stumble across it because the conditions that trigger it are fewer, so you need to start with the bug and then go hunting for it. Think of starting with the bug when you're testing for deadlocks, race conditions, and security tests in particular.

Give it a shot. Maybe you'll get lucky and the bug you imagine doesn't exist... but maybe it will, and you'll have found it because you're looking for it.

One Day Away

Many of us are important in small ways. We're the ones who always check out last night's automation first thing in the morning. We're the ones who find a creative way to track down a bug. We're the ones who make sure support has been trained on the latest release. Sure, we're all testers, and anyone on the team can effectively test the product, but every one of us brings something special. One focus, or skill, or talent that the others simply don't even notice to do. It's just your thing that happens, because you do it.

Now take one day away.

No planning, no prep, no warning or pre-training. Just take one day away.

What would happen?

Something wouldn't go right, just for that one day. Testing would get done, and people would accommodate your absence, but your small importance would get missed. Would the world end? No, of course not.

Now extend that to a week away.

That "not right" begins to snowball a little bit. A week is long enough for people to notice that little thing you always did that's now not getting done. The world still isn't ending, but it's noticeable.

Now, this is just a thought exercise. But consider for a moment the disservice your team experienced because you were just gone. That thing - whatever it is - needs to be something that can survive you. Can you train someone to keep an eye on your one thing? Can you automate it? Can you make the need for that one thing go away?

You can't move up if you leave problems behind, even the slow problem of one missed thing building up over time. So think about your thing, and make it your team's thing.

Planning Patches

There are the releases you plan and the releases you don't. The releases you plan usually have nice numbers like "3.2" and "10.6". The ones you don't usually have names like "patch 2010.02.12". Patches are often a reaction to surprise. They're a quick enhancement that sales Really Truly Needs to close a deal this quarter. They're a bug fix for something you thought you'd tested for but apparently missed. They're a bug fix for something you totally never expected in a million years but that hit a major customer.

It's difficult to know what will trigger the need for a patch, since it's different each time. However, it's not nearly as difficult to know that you will need a patch. You can't say what's in it, but you can definitely say you'll need it. Think back over your last three minor releases? How many of them also had patch releases later? How many patches between a minor release and the next release? Now repeat for major releases. For this purpose, consider "major" to be a release with a lot of new things and/or change in it. Consider "minor" to be a release with a relatively small amount of new things. This may or may not map with what marketing wound up calling a major release or a minor release.

Usually, you'll find that big releases where you introduced a lot of new stuff have the most patches after them. Minor releases have fewer patches after them. It'll look something like this:

It's easy to think of patch releases as surprises, but really, they're like birthday presents. You know you're going to get some; you just don't know what will be in them. So plan ahead and leave yourself room in your schedule for a patch release. After all, it's coming.

Your Certainty

Some days find you way outside your comfort zone. That's the day you present at a brown-bag lunch... and you've never talked to more than 2 people at once. That's the day you find yourself on the phone with a client explaining a bug... and for the first time discovering that you can't properly describe everything because they don't know the architecture of the system. That's the day you volunteer for the performance task force... when you're aware that "performance" has something to do with speed but don't know much else.

Stepping outside your comfort zone is a great thing, and learning new things is fun. Those are stressful days, though. They're days when you will question whether you can do it. They're days when you will go home and feel like you have no ground to stand on.

Those are days when you need your certainty.

Most of us have things we do really well, and that we're confident in. Some of us are really good at expressing ourselves in writing. Some of us can find bugs based on the barest wisps of smoke in the system. Some of us are awesome at system design. Hold on to those things.

When you have a day in which you're way outside your comfort zone, try to find some time to do something you're confident in. Think of your knowledge as an anchor in your own personal sea. When you're lost at sea, if you can just touch your anchor, it'll help you keep floating (and with that I'll stop the analogy before it falls apart!).

Mixing the new knowledge you're acquiring with the old knowledge you have can help you learn the new stuff faster by staving off the feelings of being overwhelmed. It frees you to learn what you want or need to learn. Try new things, please. And when you're feeling lost, remember what you know, and learn the new knowledge off that solid base.

Runaway Automation

Test automation is a great thing. I'm absolutely for test automation in general. Like any tool, though, it should be used carefully.

"Automated tests are cheap to run" is a danger phrase.

It's generally true. Compared to a single manual run of a test, a single automated run of the same test probably takes less time, even accounting for a human looking at the result. It also frees the human up to do new and exciting tests. However, even if one automated test is cheap to run, that doesn't mean it's cheap forever.

The slippery slope thought process goes something like this:

- I found a bug and they fixed it! I'll write a test to prove it.

- I can automate this test. I'll add it.

- I should really write a test to show this bug, and when it's fixed we can check it in to prove it.

- New feature! Let's see what tests we need to feel good about our abilities to refactor it.

- (Fast forward 3 or 4 years)

By now your automated test suite is suffering from Rampant Automation. One test is cheap, but you don't have one. By now you have thousands. And for that reason automated tests are no longer cheap. They probably run for hours on end, take a significant amount of time to maintain and analyze, and use several machines.

All this isn't to say that you shouldn't use automated tests. Please feel free. Just make sure that you keep an eye on the totality of your automated tests and make sure they're not getting away from you. It's up to you to decide how much time and machine time you will put into automated tests. When you get close, it's time to clean up your automated tests, find ways to speed things up, remove tests that aren't helpful, and/or evaluate whether you really do want to automate the new test(s) you're considering.

Automated tests are great. Runaway automation is not. Keep an eye out!

Changing Style and Code

Code is dynamic. You write something, and then you add something to it, then you modify some behavior, then you delete that crufty thing that you're not hugely proud of. Over time it grows and changes.

Coding standards and techniques are dynamic. You start with some basics, then you add a few things. Then you decide that the brilliant style idea you had wasn't so brilliant, so you modify your technique. Over time it grows and changes.

The really fun part is that you've now got (organic) code and (organic) coding standards and (organic) style, and they're all of varying ages and applications. If you change a piece of code you haven't touched in a while, it might not conform to your current coding practices in a lot of ways, even if the change you're making is really small. We have a few options here:

1. Whenever we change a coding practice or style, we go through and update all the code.
2. Whenever we touch a piece of code, we update the entire file/class/method to use the current style.
3. We write all new or modified code in the current style, and don't worry about code you're not touching.
4. We give up on coding standards. Or give up on our code. Come to think of it, this one is a bad idea. Skip #4!

Any of the first three options is viable. If you have a huge code base, or frequent style guide changes, then option 1 starts to take an inordinately long time. Option 2 or option 3 may be more appropriate.

If you stick with a code base for any period of time longer than a few months, you'll likely see code changes, and you'll likely see style changes. When that happens, you'll need to decide how to deal with it. You have choices, just be sure to make your choice so that your entire team knows what to do as things change.

Two Sides

I was working on something last night and I was getting frustrated. Basically, I was writing a test of some code and having trouble distinguishing failure states. The code basically looked like this:

def method

begin

result = makeSOAPCall

rescue

result = "FAILED"

end

return result

end

In principle, this code isn't too shabby. It catches exceptions and handles them instead of blowing up, it returns a result that indicates a problem if there is one, etc. However, it has one major testability flaw: there's no way to see why you failed from a test. All you get to see is that it did fail or didn't.

My test was feeding values into the SOAP call, attempting to induce specific responses (successful and errors), and verifying that the parameters produced the results I expected. The frustration was that I couldn't see why something failed. So I was reduced to parsing through the logs where the actual error was recorded, and that was painful.

And then it occurred to me.

There are two sides to the equation. We - the collective team - owns both the code and the test code. We can make these reconcile. So I grabbed a developer and we changed the code so that the result object had an exception message on it which included the actual error. My test could simply handle that just like it handled seeing if the SOAP message succeeded or failed. Simpler test, and no loss of function in the product. Perfect.

It's easy to get caught up in needing to take the next step to get a piece of data out of a system you're testing. When it's hard, though, or brittle, it's worth asking if you can change the system to get the data out more easily. There are two sides to every contract; consider both of them.

The Yay Mail

Eventually things come to an end. You're testing away on a release or on a problem, and eventually you do everything you said you were going to do. Or eventually you reach the designated end date. By whatever criteria you've set, you're done.

Great. Next!

Not so fast. For better or worse, often people won't assume things are going to be done on time, or in the way that you said. They probably don't blame you for it, but they're going to try not to assume completion erroneously, so they'll err on the side of assuming you haven't finished on time or completely. So when you're done, tell someone.

Don't forget to send the yay mail.

When you've finished an effort of any large-ish size (e.g., a release), tell people about it. Maybe it's an email confirming you've done X, Y and Z on some date. Maybe it's an updated wiki page or a status in a project management system. It doesn't matter how you do it; just find a way to confirm for people that you're really done. That explicit closing out of a project keeps everyone on the same page, and then you can go on to the next thing. Celebrate just a little, then move on.

You deserve your yay.