Up To Date

I've been in more than one series of events like this:

"We're starting to put down some dates for the Sweetums* release."

"Okay. What's in it?"

"Stories X, Y, and Z. We've put a brief writeup of each in Jira.**"

(pause while we go look at the stories and estimate the test effort on each as well as the estimates on how many bugs will come through, etc.)

"All right, we're probably looking at 4 weeks between feature freeze and code freeze, and 2 weeks after code freeze for final verification."

And then time passes. We start coding tests, looking at early builds, etc. Development, well, develops.

Inevitably, something changes. We might get a new customer who really wants a feature, or we might find a bug that needs to be fixed for the next release, or we decide to drop a story that seemed important but turned out not to really matter. Net net, something changes. So we have another conversation:

"What's the new estimate for this release?"

"Umm.... is everything updated in Jira?"

"Yeah, except we're not doing story Y, and we're doing story W."

At this point the conversation needs to stop. We have a process. We have a place that defines what's in a release. Let's use it.

If you've defined a place to keep information - the contents of a release, bugs and what's been done on them, employee phone numbers - then that place should be kept up to date. It seems like a lot of work sometimes, to keep updating the information location. It's worth it, though. It saves a lot of other people time, and it works to train them to trust the information location. If it's up to date always, then people will come to trust it. You'll spend less time explaining status and more time working.

And less time on status and more time working sounds great to me.

* We name our releases after muppets where I work. This is Sweetums:

** We keep our stories in Jira so we can track them and do things like attach bugs to stories, etc.

An Error

What's wrong with this test code?

def testFailureMode

my result = runMyGreatTestCommandHere

if result.error

puts "Test PASSED. We wanted to make sure it errored properly"

else

puts "Test FAILED. Expected to error."

end

end

On the surface, it's not so bad. It runs a test that should produce an error, and makes sure an error is produced. Let's dig one step deeper.

We're testing a failure mode, which means we're expecting not just any error, but an error we induced. For example, let's say we're testing login with a bad password, and want to see that there's an error "incorrect password". However, this test would pass if any other error occurred. If the page never loaded, or returned a stack trace instead of a login page, this test would pass just fine. After all, it got an error!

When you're testing failure modes, make sure not only that you get an error, but that you get the error you meant to get.

Reboot Heuristic

Let's say you had to reboot your primary work computer, right now. How much would you lose?

How many windows do you have open with things half done?

How many emails partially written?

How many IMs to read? To respond to?

How many browser tabs with articles to read?

How many IDE windows with various code changes?

How many tests running in various terminals?

And now you have to reboot.

How many different tasks do you have to finish or at least get to a stopping spot before you could reboot?

Your primary workstation shows you how much of a multitasker you think you are. You're not really multitasking, though. You're simply not finishing things - maybe a lot of things.

I'm not particularly paper dependent - there are zero pieces of paper on my desk as I write this - so I use the reboot heuristic to remind myself to finish things. Ready?

If I have to touch more than three things before I would be willing to reboot, then I can't start something new until I've finished at least one.

The reboot heuristic is a reminder to myself that I need to finish things, not just start them. If I write an email and get partway through before I remember that test I wanted to start, that's okay. Repeat several times, though, and I'm spending more time switching between tasks than actually doing tasks. So three is my limit: when I'm doing more than three things, it's time to finish something.

Starting things is great. Finishing things is better. Multitasking is part of working these days, but not letting it get out of control - that's a heuristic I need.

On Telling Somebody

Testing is about information. We do things to systems in order to get information about the system. From there, we make educated guesses about the systems likely behavior in the future. For example, I might test an installation on a certain hardware configuration, and then from there make statements about the installer's likely behavior on the same hardware configuration at a customer site. Such statements probably take the form, "yup, it'll work", and they can be made explicitly or they can be made implicitly ("Pass").

Testing can be broken down into those two parts:

1. doing something and seeing what happens
2. predicting future events

If you haven't done something and seen what happens, you haven't tested. If you haven't told someone about it, you haven't tested.

It's easy to get caught up in the doing, but don't forget the communication side of it. Tell someone - in a test plan, in an email, in a green bar in your IDE, in conversation. You're not done until you've told someone.

"Why" Is a Question

"Why don't we test X?"

"Why do we wait to check in until the tests are done?"

All too often, we take these questions as a suggestion that we change our behavior. "Why don't you foo?" is something we take as "Do foo." "Why do we do bar?" is a question we take as "Don't do bar."

But it is a question, not a directive.

Sometimes the correct response really is, "No good reason. Let's change that." Sometimes, though, there is a reason why we do things. Maybe we don't test X because we're getting rid of that feature, or we don't have the hardware to do it. Maybe we do wait to check in until the tests are done to make sure our coverage stays high and we don't create technical debt.

Answer the question. Then change your behavior - but only if that's the right answer to the question.

Test Near Development

Test Driven Development (TDD) is a great idea. You write some tests that say, basically, "this is what should happen." Then you write some code to make it so. It's really rather like waving your magic wand (or keyboard).

Repeat this often enough, the idea goes, and you have yourself a pretty good system, and a pretty good amount of code coverage so you can refactor safely.

All great things.

But.

TDD is hard. It's really difficult to do once you get beyond the first little bit. After all, as you keep adding and modifying tests, every change starts to take a little longer. Not much longer, but enough you notice and start to lose focus. And if it takes more than 8 seconds for something to happen - say, your tests to finish - you're likely to lose focus. (See David Cornish and Dianne Dukette: The Essential 20).

The benefits of TDD, however, accrue as long as the tests exist when the feature exists. This holds true no matter if the feature is a single method or an entire new component. The goal of having tests to enable safe refactoring, to guide the implementation, is still met. We're just not dogmatic about product code vs test code: both product and tests are required; get there in the way that works for you as an engineer.

So on our team we practice test near development. We don't force people to write their tests first. Instead, we ensure that tests exist when the feature is complete. In practice, some engineers write a little code, then write some tests, then refactor a little code, then refactor some tests, then write some more code. The last thing written is usually the acceptance tests. We end up in the same place, but we've allowed the tests to come before, during, and after the development effort. I've taken to calling this "Test Near Development".

And you know what? It works for us.

Give it a shot sometime, if you're having trouble with TDD. It might be a gateway to help get you to full TDD, or it might just give you better test coverage that your team can sustain.

Asking

Software is a community phenomenon. For almost everything, no matter what you're doing, someone else out there is doing or has done something similar. Maybe they're not doing the same big-picture thing, but someone else is doing that small task you have before you now.

You can ask for help. And you're really very likely to get an answer, but only if you ask the right way.

Whether you ask your team a question, or send out a request to a mailing list, or hit up a support forum, or hop onto an IRC channel, you need to remember a few basic things:

1. Describe what you're trying to do. Give the user some context so they don't suggest something that makes no sense. Don't take more than about three sentences to do this, at most.
2. Indicate you've tried something. This does two things: (1) shows you're trying; and (2) prevents people from suggesting something you've tried.
3. Ask a question. Make sure you actually ask a question. Be as specific as possible in describing what you need to move forward.

Doing these things alone will greatly increase your chances of getting an answer. You've made it easier to understand your question, and therefore to answer it. Asking seems so easy, doesn't it?

No, We Shouldn't Document That

"We should document that" is not an uncommon phrase. It usually means "something is a little weird and our users might take it the wrong way, or maybe they already have taken it wrong." It's a place where you're doing something your users will find confusing or non-intuitive. Consider these examples:

• Our login page asks for "username" but we expect you to provide an email as your username. People keep clicking "Forgot username", so we should document that your username is the email you signed up with.
• When patching the high-availability system, support must patch the passive half of the pair first. Patching the active half first leads to system downtime. We should document that for support.
• We can register our system in DNS at installation time (go us!) but only if you provide the username and password of someone with permission to do so.

Every "we should document that" is a bug until proven otherwise.

If you have to document it, most of the time that means you've designed it or implemented it in a confusing way. So yes, please document it. Then log it as a bug and make it less confusing. Sometimes that's as simple as changing a label - "email" instead of "username". Sometimes it's harder, like making a high-availability configuration patchable from either half with transparent failover. Sometimes, it's even dependent on a third-party system, and involves moving the documentation into the product (e.g., "please enter the username and password for a user with permission to create DNS entries") rather than removing documentation entirely.

So no, we shouldn't document that. Instead, we should fix that. Let's quit with the post facto workarounds and start actually fixing the weirdness in our application up front.

Found Time

Unfortunately, often things take longer than planned. A meeting runs over, a quick bug fix takes hours, a feature that should have taken a few days takes nearly a week. Sometimes, though, the inverse is true. That hour-long meeting breaks up in 15 minutes. The bug fix that looked hard turned out to be a simple one-line change (and was elegant, too!). The time you would have spent is found time, and found time is a gift.

So what do you do with your found time?

You can go straight on to the next thing - your next backlog item, the next bug, the next meeting.

But you don't have to.

I use found time to do the nagging little things. These are things that are so unimportant you won't get to them normally. Nonetheless, they're small bumps in my day, things that don't go quite as smoothly as I wish. These are things that when I'm sitting at my desk, I won't schedule them because there's so much more important to do. But they're things that I run into every day.

I had a few hours today of found time. I used them to:

• Update a script that spit out information in rows, and I always wind up sending that information to other people in columns. I made it spit out columns. Total time saved: 30 seconds, tops, but boy was that annoying having to transpose it. No more transposing!
• Upgrade all my Rails gems. I'm just starting on a new project, and using the most current ones will be useful later on.
• Update the Abakas web site with some clearer services information.
• Install Transmit 4, which is a lot faster than the previous version.

Not one of these things was hugely important. Now that I've done a few of them, though, my day will be a bit smoother, and I'll be better able to concentrate on the truly important things.

Found time is a free gift. It's unscheduled and its yours. Use some of it for doing the nagging little things. Think of it as spring cleaning for your day!

My Current Test Stack

I've been doing some functional testing on web applications. The applications are relatively simple, written in Ruby on Rails, and provide the following user interfaces:

• a GUI
• SOAP and REST APIs

My current testing stack for functional tests is:

Rspec. I use this for testing models primarily, and some of the controllers.

Cucumber. This is where I do most of my testing of workflow, business logic, and integration between pieces. A lot of the tests take the form of "sign up through the GUI, register a machine through an API, upload user data through the API, view usage data through the GUI" (and variations thereon).

Selenium. I use this for the very few tests that cucumber can't handle because of JavaScript, rendering niceties, or other browser-type concerns.

I'm in the market to learn a new toy (I mean... tool!), so I'm turning it over to you, readers. What about this stack should I swap out and for what kinds of tests?

Analyze With Purpose

Analysis is a double-edged sword. While in theory a quest for understanding is laudable, understanding rarely actually fixes a problem. In the end, our focus has to be on fixing a problem; analysis is one step on the way.

Analysis is a means to an end, not an end by itself.

Instead, we must perform analysis with an eye toward what we want to accomplish. Don't say, "we'd like to understand behavior X". Instead say, "the customer is seeing behavior X and wants behavior Y. Let's figure out how those two behaviors are different under the covers."

Don't say, "that unexpected performance improvement should be analyzed." Instead say, "we'd like to make sure that the performance increase we're seeing isn't due to measurement error, some change in system input we haven't noticed, or caused by some transient state."

In other words, don't simply set analysis or even understanding as a goal. Instead, set a goal of what you will do with that analysis and that understanding. That way you know when to stop analyzing and when to start doing.

"Can't"

How many of us have heard this phrase?

"We can't do that!"

Or even better...

"You can't do that!"

Almost every time, the person (or us) is perfectly capable of doing that, whatever "that" is. What we really mean is, "please oh please don't do that!" or "I'd really rather not do that". We say "can't". We mean "won't".

The trouble is that "can't" turns it into a dare. "You can't do that!" "Oh yeah? Watch me!". Whoops.

Telling someone they can't do something often has two effects:

• Makes you sound rather whiny
• Makes the person more likely to do it if only to prove that yes, they can
• Makes the person more defensive of his position

Take away the dare. Not "you can't do that!" but "here's why this should be done another way". You're much more likely to prevent "that" from happening (whatever it is) if you can provide an alternative rather than a dare.

CAST 2010

I try not to do these shameless plugs too often, but I'm excited about this one.

I'm presenting a session at this year's CAST 2010 on Communicating with NonTesters. We'll talk about how to present information in a way that whoever you're talking to really understands it. Throughout the session we'll cover the dreaded "status report", the art of writing a defect explanation for a (probably angry) customer, and discuss how to write a risk assessment that doesn't make you look like you're panicking over nothing.

I'd love to see you there!

Here's the conference info:

Attend CAST 2010

The 5th annual Conference of the Association for Software Testing

August 2-4, 2010, Grand Rapids, Michigan, USA

"Skills in Testing"

About CAST

CAST reflects the AST's core mission: to build community amongst scholars, practitioners, and students for the advancement of the practice of software testing. In 2010, CAST aims to leverage peer collaboration to build an enhanced understanding of how various skills influence tester effectiveness.

CAST offers a unique opportunity to learn and confer with others that simply isn’t found at other conferences. Each scheduled session allocates time for facilitated “open season” discussions that encourage participants to question and challenge the presentation. What takes place in the hallways, at receptions, and during meals and lightning talks truly sets CAST apart; for many attendees, the greatest value is derived from the opportunity to discuss and delve into the topics that matter to them.

More information and Registration: www.CAST2010.org

Change Tactics

Given a problem, there are generally several ways to approach it. The problem might be an issue occurring at a customer site, or a test we want to conduct, or a scaling design goal, etc. In any of these situations, we have lots of choices in how we tackle it. We might start by looking at what changed to cause the issue to occur. We might begin with reviewing the architecture of massively scaling services similar to ours. We might use code review to identify possible flaws that we want to exercise with unit testing code.

All those are great, and worth trying.

But what if it's not working?

Is that "what changed?" question you've been asking yourself about a problem that showed up at a client site not getting you any answers? Did that massively scalable architecture that seems like it would provide some good ideas lead you down a rathole of trying to map your square peg to their round role?

Time to change tactics. If it's not working, stop doing it, and try something else.

So how do we know when to change it up versus when to pursue a given tactic?

As a rule of thumb, I try to give no more than 20% of my estimate to any given tactic.

Let's say, for example, that I'm designing a test of a new module. If I estimated that the whole test would take me 5 days, and after one day I'm nowhere, it's time to rethink my tactics. If it's only been an hour or two, well, I might just not have taken it far enough yet or given the tactic enough time. If it's been two days and I'm still nowhere, I'm ratholing.

As with any heuristic, there are exceptions. Sometimes it's worth spending longer on a particularly tricky, slow or likely tactic. Just make that a conscious decision. About 20% of the way through your overall estimate, stop. Ask yourself if you're really getting anywhere with this tactic. If not, time to change.

We'd all like our first tactics to work consistently. Often, they will. For those times it won't, pull yourself out of the rathole and change tactics.