"Technical"

"I'm looking for a technical tester." This is code.

This really means one of several things:

• I want someone who can code (developer-lite)
• I want someone who will create a robust automated test suite and tools to run it
• I want someone who knows what a mouse is and has a prayer of understanding a command line.
• I want someone who knows how to understand and manipulate a system, and to apply thoughts derived from computer science to a mental model of how it works.

These are all legitimate needs. It's up to you to figure out which one is your true "technical tester".

Testers who are primarily coders are great. They make wonderful tools engineers (really just a name for someone who writes code used by testers). They are great for testing APIs and other things where interaction with the system under test is done through code.

Testers who don't code much are fine. They can still think deeply about a system. Some of them are great with usability and user interface because they don't think about the code but about the system. Testers like this can also usually put together scripts and other helper code snippets when they need to as well.

Testers who don't code at all are fine. They're more limited in what they can do, but the tester who can't work on Linux is limited as well, just in a different way.

Don't tell me you need a technical tester. Tell me you need a tester who can do X, Y, and Z. Only if you're specific about your needs can I find you a tester who actually meets those needs.

I'd like to thank Matt Heusser for the inspiration for this post.

Fix Small Fix Big

Sometimes you walk into a situation that can only be termed a complete disaster. You've got angry clients and hostile relations between dev, test, support, and product management. You've got problems in production, and it takes a ridiculous amount of time even to build a feature to put into production. You've got rituals but no actual benefit from them.

There's so much to fix it's hard to know where to start.

Okay, deep breath time.

When faced with a huge problem, you have two objectives: to show that it's getting better by showing positive changes; and to make big changes to fix the underlying the problems in the mid-and longer term. In other words, fix something small, and fix something big.

It's important to fix something small because you need to show progress. Showing that something, anything, is improving helps keep morale up. It also shows that change can happen.

Fixing something big is how you ultimately clean up the whole mess. It will take rather longer, and probably look hopeless at a few points (it's not! keep going!), but in the end it's how you get out of it for good. Fixing something big also shows you're not afraid, and that big things can happen.

In the face of hopelessness, fix small and fix big. Tackling the disaster from both ends is the best way I know to get it properly resolved.

Take Advantage of Internal Users

Many applications are consumed by external users. Think everything from antivirus software to a storage system, etc. Other applications - or sometimes parts of external applications - are consumed only by people who work for the company that builds the software. These internal applications can be anything from a reporting module to an internal data analysis application, etc.

Knowing whether your application has internal only or at least some external users will help you figure out how defensive your coding and therefore your testing needs to be. For internal only applications, it's often possible to make the assumption that users will work in a more consistent way than if you have external users. Internal users have shared goals, training, purposes, and assumptions. In addition, if all users are internal users, the cost of error may be smaller. Of course, you need to check all of these assumptions. If they hold true, however, that implies that you can do less testing.

For example, if you have an API that is exposed for reporting, and the only reporting application is internal, you can be less defensive in the implementation of that API. You can, for example, assume that IDs will always be numbers without explicitly checking for it, as long as the consuming reporting application always uses IDs that are numbers.

Will this cause you to miss some problems? Absolutely. Not performing negative testing leaves you more vulnerable to problems caused when your users step off that positive path. However, that might be acceptable if you have more need to get features out quickly than to be able to handle change without explicit inter-group planning.

For applications used by external parties, you will generally have less notice of change and more variety in use cases seen in the field. Therefore, you need to do more testing that will expose problems in those scenarios - more negative testing.

As with most things in software development, how much you test off the happy path will depend on your specific circumstances. Just make sure you ask the questions so you can take advantage of your internal users wherever possible.

One Problem One Bug

A bug (defect, issue, whatever) logged in your defect tracking system should be about a single problem. Sounds straightforward. But it's sometimes hard to tell.

Let's look at some examples:

Failed Upgrade:

Problem Report: An upgrade failed because a hard drive failed mid-upgrade and then there were configuration issues. One problem or two?

Answer: One problem if the configuration issues were caused by the hard drive failure. Two problems if they were two separate problems that happened to occur during the same upgrade. Hard to tell.

Practically, what to do: Log this as one issue, and log separate child issues as you isolate separate problems.

Active Directory User Does Not Exist:

Problem Report: At two clients, an active directory user that the system was attempting to look up failed. In one case, the error returned was "INVALID_USER" and in another case, the error returned was "LOOKUP_FAILED". One problem or two?

Answer: One problem. In both cases, this was an issue where the system attempted to access Active Directory when AD was failing over and not servicing requests.

Practically, what to do: Log these as two separate bugs. It's not obvious they're the same issue. When it becomes clear that they are the same problem, mark one issue as a duplicate of the other and close it out (aka consolidate the two issues into one).

On a day to day basis, it doesn't really matter whether you log things in separate bugs or the same bug initially. What matters is that you stick with the "one problem, one bug" rule, even as your understanding changes. If that means marking bugs as duplicates, great. If that means logging new bugs and putting "this comes from bug X" in them, that's okay, too. The point is that your defect tracking system should reflect your current understanding of the problem, with only one problem in each ticket.

Reconsider Setup

Test cases - written or not - in general have a basic form:

• Prerequisites
• Setup
• Actual test
• Results
• Analysis

The first design of a test case considers all of these, explicitly or implicitly. As you design the test case, you think about how to get the software into an interesting situation (prerequisites and setup), what might happen or not happen (test), and look at what occurs when you actually try this (results). The next release, when you're doing a similar test - following a script or whether you simply want to explore the same thing - you don't tend to think as deeply. You assume the same prerequisites and setup, and you do the test. Much of the time, this is okay. In particular if you're doing regression testing where you're looking for lack of change, that's generally fairly safe.

However, it's not complete.

Something has changed every time you redo a test. It might not be obvious what it is, or it might be as straightforward as a new build you're testing. Either way, the entire test case deserves consideration. Don't just redo a test. Redo the prerequisites and the setup analysis, too. You'll find more that way.

Check In But Don't Run

When a system under test is a software library or other piece of software with no GUI, you find yourself writing a lot of code. If the means of using the system under test is an API, pretty much any test you do is going to be using the API.

You end up with a lot of automated tests, and the problem of "how much should I automate?" is greatly reduced. Whatever you write just has to be hooked in to the test running and reporting system, and the barrier to automation is pretty low. It's a lot of fun.

It's also dangerous.

Running tests willy nilly can give you a huge headache trying to create and maintain tests and test infrastructure, analyze and report on results. Not everything

There are three acceptable things to do with test code you write:

• Check it in as an automated test.
• Throw it away.
• Check the code in but don't run it.

The most obvious course of action once you've written a test is to check it in and make it part of your automation suite. This is often the right answer. Do this when the test is well-written, when the test is fairly cheap to run, when the test exercises an area of code not already covered by other automated tests.

It's also acceptable to throw code away. Think of doing this for quick scripts that aren't well-written (e.g., with hard coded paths and the like). Also consider throwing this away if the test didn't find anything interesting about the code - bug or not. Lastly, if the code that this tests is going away or changing, don't worry about testing it again; go ahead and delete that test code.

Lastly, there is a class of tests that you need to run, but only occasionally. These are tests that are too expensive (time or resources) to run in your general test infrastructure. However, they describe important things about the product, and you will need them again in the future. Scaling tests, for example, or performance tests using particular hardware, often fall into this category. In this case, write the test well and check in the code, but don't hook it up so it runs in your test infrastructure. It's okay to have these hand-run tests to ensure that your expensive tests run consistently when you do run them - manually.

Next time you have some test code, consider whether it's an automated test, a throwaway bit of code, or something you'll want again, just not tomorrow.

Bug Rhythm

Projects have heartbeats. They have their own rhythms and sequences, and these can be used to make predictions. A rough measure of stability can be gained, for example, just by looking at bugs over time.

For example, we have an automated test suite that runs every night. Over time, a pattern will emerge that can be roughly graphed like this:

The project starts. Then there's a few bad nights as many tests fail, then it gets better. Later in the project there's another spike, but smaller, followed by improvement. This is followed by another spike, smaller than the second, and so on. Each spike is an instance of destabilizing changes being checked in. Sometimes they're new tests, sometimes they're new features. Sometimes they're manual tests, sometimes automated. Sometimes the problems being pointed out by the failures are issues with new features, sometimes regressions, sometimes a bug that existed but wasn't exposed before, and sometimes a combination of all three.

The specifics don't matter. I don't particularly care what caused each bug logged here; it will be different every time. I'm more interested in the rhythm, in the pattern.

By stepping back we can see that over the course of a release, we destabilize as we take on big things, then we restabilize. More changes going in destabilize a bit and then restabilize again, with an overall trend downward - toward stability and toward a point at which we find far fewer bugs. We can also look at the periodicity of the spikes to figure out how long and how frequent they are. Look at this pattern over enough releases and you can start to see what an overall release rhythm looks like.

Maybe you tend to have four or five spikes, and then you stop finding things. Well, if you're just done with spike 3, you can expect one or two more destabilizing changes, and then you'll probably be able to release. You can even put a rough timeline on it, based on past releases and the patterns they showed.

This is a heuristic, not an absolute metric, but it's a good way to get a handle on a release. Pull back from the details and let the pattern tell you what is likely to happen.

Thrill of the Chase

It starts with a situation, and with some wiggle room.

"We're having this issue.... sometimes."

"We're having a problem... with some users."

"Clients are sending us screenshots that look like this but we can't make that problem happen."

The hunt is on. Assemble the data hounds!

Let's work through what we know:

• we have gotten more complaints for it on new users
• it hasn't occurred in our dev configuration, but it has in the client's test configuration
• we've gotten reports of the problem on at least Firefox and IE 7
• when it happens, it's always CSS styles that look funny or wrong

Now we can start to eliminate possibilities:

From there, we know to hit the test server on Firefox with a new user. I did that, and reproduced the problem.

NOW we're getting somewhere.

With a little help from Firebug, it became obvious quickly that the problem was that any user who was seeing the EULA (new users only) had some of the styles redefined outside the CSS in the EULA itself. A little more digging turned up the fact that whoever had created the EULA for the client's test system had done so by saving a Microsoft Word document as an html page with inline styles. Some of the style names overlapped, and there's our bug!

Often, I get my prey and another bug is understood. Sometimes, the hounds and I go home defeated.

Either way, embrace the thrill of the bug hunt. It's only part of what we do, but boy does it feel good to catch a devious defect.

Practicing Ceding

I wrote a few days ago about ceding control to your team. And I got a great response: "Okay. How?" The short answer is it depends on you. This is your need for control, so you have to figure out how to cede it.

The longer answer is that there are some things you can do that will likely help:

• Be aware
• Understand consequences explicitly
• Start small
• Trust publicly

The first key to changing any behavior, really, is to be aware of it. Watch yourself and check for times when you're asserting control. When that happens, ask if you really need to be in control of this thing in this moment. Watch out for thoughts like "I'll just do this myself!" or "if it needs to be done right..." or "Who knows how so-and-so will do it". These are indications you're taking control in an area where maybe you could delegate the task. Whenever you hear yourself thinking one of these phrases, stop, and go to the second key.

The second key is understanding consequences. Like many things, this is a two part question: (1) what is the likelihood of failure; and (2) what is the consequence of failure. In other words, treat this like a bug (you know how to do that!). Once you've noticed a decision where you can take control or delegate (see the first key above), you need to consider what happens if your team member fails, and how likely he is to fail. If the consequence of failure is high - customer embarrassment, public malfeasance, etc. - that's an argument for not letting the work go out without your oversight. If the consequence of failure is low - the test domain gets a name you wouldn't have picked - then it's a prime opportunity to delegate. Consider the same for likelihood of failure. And don't forget it's okay for people to fail, as long as the effects are controllable.

The third key is to start small. Not comfortable delegating an entire test plan to someone? Get them to write a section of it in an area of the product they know well. Really feel like the lab hardware allocation is make or break? Ask a team member for a proposal that you will review before it gets sent to IT for implementation. Start delegating contained tasks. Be a bit of a gatekeeper reviewing work for a while. This allows you to get an idea of what your team members can handle, and it lets you learn that you don't have to control for good things to happen. Build up from there to larger and larger tasks with less and less oversight.

Lastly, be explicit about the trust you're placing in the team. These are testers just like you. They're probably people you've worked with and developed a rapport with. So sit down and confess: "Hey, guys, I'm new at this management stuff. Let's help each other out." Going forward, it's okay to give a task to a team member and tell them, "I'm trusting you with this." Your team will almost certainly work to earn that trust.

Be patient with yourself. Giving up control is hard. You're going to take over and do too much. But you've given up control before; after all, you didn't personally test the entire product did you? Your team mates did a good chunk of it, and you trusted them then. Keep that trust, build that trust, and let them learn and grow. You and your entire team will be better for it.

The Humans

We made about three decisions:

1. We use Jira for defect tracking.
2. We allow scripts and utilities to log defects (e.g., when a hardware failure occurs, or when a machine isn't clean after a test)
3. All defects including escalations from the field, human reported issues, and machine reported issues are roughly the same regardless of origin. Any of them may lead to a high or low priority, or be against any one or more of the components of the software.
4. Sometimes all I remember about a bug is that a human reported it. (This one isn't a decision; it's just the way my mind works.)

Effectively, these decisions mean sometimes I'm trying to track down a bug that I remember and all I can remember about it is that someone in support or QA reported it, we've known about it for a while, and it's related to timezone settings. So I want to search Jira for an issue not reported by a machine with the phrase "timezone" in it.

Trouble is, Jira doesn't let you - at least through their GUI - create a search with only a NOT. I can't, for example say that the reporter is NOT automatedReporter (our Jira user for automated bug reporting). I can say it's X NOT Y, but I can't simply say NOT Y. So I made a group in Jira called "Humans" and I put everyone but automatedReporter in that group.

Which means that in my job, sometimes I'm searching for an issue found by a human.

For some reason, that cracks me up!

Ceding Control

I'm mentoring a new manager these days. She entered management in a fairly typical fashion: by being the go to tester on her team, the one with the communication skills, the team needed to grow, and - presto! - she's now the manager. For the most part, the change hasn't been too bad. She's coordinating the test priority, and helping keep focus on all the things the team is working on. She's talking about test with the development manager and going to requirements meetings to suggest ways to make the requirement more testable. All good.

There's one problem, though: ceding control.

A good individual contributor learns to trust herself and her work. When she does something, she knows that it'll be done right. This holds whether it's development, test, whatever. As a new manager, it's really easy to carry over that attitude, and to start to insist on how things should be tested and what should be said. Keeping control like this is a recipe for exhaustion.

Managers need to be able to cede control to their teams.

Yes, ultimately anything your team does - for better or worse - is your responsibility. But you can't do it all; it won't get done. You have to let your team help you, and they can do that most effectively when they're the ones in control, when they're not being micromanaged.

This is really hard.

Let's try an exercise. Take a piece of paper and draw a line down the middle of it. On the left side, write "must control" and on the right side put "Let it go". Now you're going to make a list of the kinds of things that you absolutely must own. You're also going to make a list of the things that you can let your team control. Then when something comes up, ask yourself if you really need to control it.

The point is that as you move into management, you take on different responsibilities. Part of that is letting your team grow to fill that void you're leaving. You took control and you didn't screw up; it's how you ended up in management. Give your team the same opportunity you had. Cede some control to them.

Status

When we want to show status, it generally winds up with a single overall status that is very brief. Further detail follows for those who scroll down, click through, or keep talking. There are a lot of ways to express that overall status. This is your chance to have fun with it!

The stop light is a classic way to express overall status. Green is good, yellow is "we might not make it", and red is "real trouble here".

You can expand into the carnival strength tester. Higher up means farther along!

Also consider the pain scale, borrowed from medicine. Red (more pain) is bad!

You have to look at status every day, and overall status quite often. Have some fun with it!

Good Times

I have a habit of saying "good times" when things aren't going so well but can't be helped. When it's bad or weird and the only thing left is to work through it, I'll say, "good times" and start working.

Yesterday I happened to say it in front of someone I barely knew. His response: "I don't think this is good."

That kind of laugh was enough to last me all day, after I explained "good times" to him!

You're Wrong!

Here's the thing:

People don't like being told they're wrong.

Most of the people I've encountered don't really like being told they're wrong. They're generally aware they're wrong sometimes. They generally like the idea of being better or more correct in principle. But the words "you're wrong" or "that's not right" still sting a bit. Heck, I feel that way, too; it's only human.

Nevertheless, we are all wrong sometimes, and it's useful to be able to provide and take that kind of feedback. There are some things that we can do when we're saying "you're wrong" to make it a little easier for the recipient:

• Address the problem, not the person. Don't say, "you're wrong". Say, "this sentence would be more accurate with XYZ".
• Provide solutions. Don't say, "this is wrong". Describe what would make it more correct.
• Indicate uncertainty. Sometimes we think something is wrong but we're not sure. Go ahead and say that. "I'm not sure" or "we should confirm" are valid phrases to use.

Being wrong sort of stinks, but it happens to all of us. We can hear feedback more clearly and be a little less embarrassed by our mistakes if we address them in a positive way that corrects the mistake without belittling the person.

Working Hours

One of the fun cliches of the knowledge-based professions is this:

"I don't care what hours you work; I just care that the job gets done."

Actually, though, I do care what hours you work. Like it or not, software development is a social profession. Design sessions with fellow team members, meetings with current and potential customers, assisting junior team members with coding problems: all of these things require you to be working at the same time as your colleagues. Now, that doesn't mean I'm going to watch you clock in at 9:00am (not 9:01!), or haggle over a long lunch sometimes.

I care that your working hours:

• are somewhat regular. It's helpful to be able to predict when you'll be around and communicating. That way I can just wait 15 minutes if you'll be in soon, or plan that meeting I really need you to attend before you leave for the day. Exceptions happen, and that's fine, as long as it's somewhat predictable.
• overlap with the rest of your team. They don't have to match exactly, but you have a team you work with, and it's a lot easier to work with them when you work at least some of the same time. The more overlap the better, but I'm usually looking for 4 hours or so.
• involve you working. Yes, what you get done matters. That part of the cliche is correct.

I really don't care that your working hours:

• start late or end early. If you want to show up at 11am, go for it. If you want to come in at 6:30 and leave at 3, that's fine, too.
• include time at home and time in the office. Want to go home, feed your kids dinner, and then get back to work? Fine by me.

Work hours are a tricky thing in some ways. There's not really a good reason for excessive clock watching, yet we do all have to work together and depend on each other, and work hours are part of that. So let's avoid the extremes. It's not about the hours you work; it's about the work you do, and that includes the work you do helping out your team. Be generally consistent, be there when your team needs you, get your work done, and then we'll move on to more interesting things!