Tuesday, November 30, 2010

The Woodchuck Factor

We've been writing our test plan for the next release, and the first draft contained a sentence that might as well have said:

Run the following tests:
  • Identify how much would a woodchuck chuck if a woodchuck could chuck wood, varying the amount of wood and the chuck factor.
  • Confirm that a woodchuck would chuck as much wood as a woodchuck could chuck, if a woodchuck could chuck wood.

Score one for incomprehensible sentences. The actual phrasing was of course, different, but it was the same effect: completely accurate and a good test, but expressed in a truly confusing way. We had hit the woodchuck factor.

The woodchuck factor is when a sentence in your documentation could be replaced with the woodchuck sentence and make about as much sense. (This is one of those things where we know it when we see it.)

Now first of all, these are the sentences that make me laugh, so that's a reason to look for woodchucks. But there's an actual problem here, too. It's really hard to understand, and it's fairly simple to misinterpret or simply skip because, well, that "doesn't make any sense" at first read.

If you hit the woodchuck factor, look for two underlying causes:
  • Overloaded Vocabulary. Are you using the same word as a noun and as a verb? Are you using the same word to describe multiple different things? Often we get away with describing two things with very similar terms, until we try to use them together - and then there's a woodchuck.
  • "Fancy" Phrasing. Woodchuck factors show up when you're trying to be "formal" or "fancy". Legal documents are notorious for this - "the party of the first part shall hereinafter be referred to as the First Part Party" - but it can happen to any of us. You can use formal language and still be declarative. Start with an example and generalize from there, but keep it simple.
Woodchucks are fixable. So the next time you find yourself reading a document and saying in your head, "How much wood would a woodchuck chuck...", laugh. Then fix it. Future readers will thank you.

Wednesday, November 24, 2010

Weekend Testing Americas Retrospective

On November 13th, I participated in the first Weekend Testing America. Weekend Testing is a challenge we take, getting together for a few hours to test an application.

Our application was a Dance Dance Revolution clone. First, I have a confession to make: I used to play that game on the PlayStation. And I was sort of okay. However, I was terrible at this game. I had a lot of fun! We paired up and figured out how to test together. It was certainly a target-rich environment, so the question wasn't how to find a bug. It was how to approach this application and how to do it with a remote partner. Joe Harter and I paired for this one.

He and I spent two hours together testing the application and talking about it with the other participants. Our charter was to pair and discover the application.

Things I noticed about pairing with a remote tester:
  • Use a different computer for talking than the one you're using to test.
  • Screen sharing rules.
  • Chat is a bit tough. It's harder even when you type at different speeds or in different ways (Joe was a type-a-lot-at-once guy, and I'm a frequent-short-sentences girl). When in doubt, calling each other helps.
  • Spend some time before you pair setting up sharing
Things I noticed about testing:
  • In a target-rich environment, double check your charter. It's really easy to stray.
  • Writing up bugs always takes far longer than you think it will, if you're interested in doing it properly.
The single biggest thing I noticed was the sheer diversity of view points. At one point I was sitting there saying, "That is so weird! It must be a bug!" and Joe was telling me, "Actually, I think a lot of video games are that way." Oh, really? Oops, good to know. During the post-test discussion, even more viewpoints came out. Some people spent a lot of time actually playing the game, some people were very bug focused, some people were very focused on comparative tests, and some people were simply overwhelmed.

If you at all have the ability to do one of these, please try Weekend Testing. Shaking up the routine, working with new people - it's amazing what a change in routine will do.

Tuesday, November 23, 2010

So You Wanna Make a Big Change

There's been a lot of buzz recently about "selling". Let's "sell" agile, and let's "sell" testing, and we're all really "selling brand You" when we look for a job (that last one was a particularly atrocious local news story on what had to be a slow news day).

I think it's great. After all, sales isn't really a dirty word. Sales is simply making people aware of the benefits and costs of something with the end results of asking them to commit to that thing. I'm sure there are more official definitions out there, but this is the one I like.

A lot of the things we're trying to sell, though, amount to a large cultural and strategic change. We want to sell someone on moving to an agile development methodology. We want to sell someone on providing testers with early access to a system. The costs of this type of change are high; it can easily take months or years to change a culture and it's inefficient while it's in progress. Cultural and strategic changes also rarely go unnoticed. Maybe if you're HugeCo and you start with one tiny project, then you can slip it under the radar. But most of us don't work for HugeCo, and most of us don't have a project that is our own domain that we can try new things with.

So let's assume we have to sell this cultural change. We're borrowing the sales verbiage, so let's borrow the techniques, too. Approaching it like it's a sales challenge, then we need to identify:
- the benefit of the change ("who wins")
- the cost of the change ("who pays")
- the people who - directly or indirectly - have to change along with it ("who cares")

From there, it's generally a matter of identifying the decision maker and decision influencers. The decision maker it the go/no-go person in the end, and it's probably the guy who's going to end up both getting the most benefit and absorbing the most cost. In the case of changing development methodologies, it's probably the head of engineering. It might be the entire executive committee.

Decision influencers are a lot more numerous. It's the sales guys who are counting on a release next quarter, and the marketing types who need a new feature for a trade show in 4 months. It's the support group who will need to handle different ways of getting customer hotfixes through development. And it's your software architects who have to implement this thing. The decision influencers are everyone who wins, everyone who pays, and everyone who cares.

Convince the influencers, and they'll help you convince the decision maker. But you have to hit most of the decision influencers, and you have to start selling. Start showing them why they should care, why they win, and why the cost is something they want to absorb because of the benefit they'll end up getting. Then you're really selling. And who knows - maybe you'll get the eventual decision in your favor.

Monday, November 22, 2010

Chicken Little Eats Risk

Software testing is highly prone to chicken little syndrome. It's relatively simple to translate "we don't know what would happen if..." to "think of all the terrible things that MIGHT happen if...". This isn't a testing problem, per se. Nervous engineers and managers come from all over an organization.

For example:
"We don't know what would happen if a hard drive failed while the system was doing X"

can easily turn into

"If a hard drive fails while the system is doing X, it might crash! And if that happens at our biggest customer, they might decide not to use the system any more because if it crashes they can't trust it! And then they might tell someone they're not using the system any more! And then our other customers might get nervous and decide to stop using our system because if major customers don't trust it then the system must be bad! And then we won't have any customers and we'll go out of business!"

(I should note that doomsday scenarios tend to include a whole lot of exclamation marks.)

"We don't know" means that we don't know terrible things won't happen. But we also don't know that terrible things will happen. This is where risk comes in. For example, if the thing we don't know much about is the static HTML help, then it's probably not too likely that terrible things will happen. If the thing that we don't know much about is the core algorithm of our product, then that's a bigger deal.

This is where testers come in. Testers, after all, are generally the ones who actually work with the software as an entire system frequently. They know which parts of the system are generally well understood, and which parts have a lot of unknowns. Often, that means testers are the first ones to say, "well, we don't really know what would happen if....". This is the point at which we get to affect chicken little syndrome, and we get to do it by describing what we do know.

The panic of chicken little syndrome comes from the perception of a vast area of potential problems; the bigger the grey area, the more likely it is that someone can think of a huge problem. So to avoid chicken littles, make sure you bound the area of unknown as much as possible.

Using our html help example from earlier, we can say, "We don't know much about the static help; we haven't looked at it yet this release." And we can stop and chicken little will come running through the office looking panicky. Or we can keep going, "However, we do know that this is all static html, so the likelihood of any side effects from issues or of any problems beyond the content is pretty small." By restricting the area of the unknown, we have started to state our risk, and we have also decreased the likelihood that anyone in our reporting circle will panic.

So go ahead and report what we know and what we don't know. Just be sure that you report the boundaries clearly. Placing your report in context makes all the difference between calm handling of uncertainty and potential panic.

Tuesday, November 16, 2010

The Forest, The Trees, and the Leaves: Part II

Yesterday, we talked about thinking about the effects of our work on the next biggest goal. When we work on the leaves, we must consider how they help the trees. When we work on the trees, we must consider how they help the forest.

Today an example of a leaf:

I have decided I want to host a brown-bag lunch for our engineering team, providing an overview of a new language I've been playing with.

This is a small thing. It's one lunch, probably an hour of everyone's time.

It's tempting to jump to the forest. How will one brown-bag lunch help the company's bottom line?

The short answer is: it doesn't.

One brown-bag lunch will do nothing at all for the company's earnings this quarter. Or next quarter. Or even next year.

However, one brown-bag lunch just might...
... inspire an attendee to write a little reporting prototype, just to play with the new app
which just might ...
... inspire a passing product manager to make that a new feature
which just might ...
... get the product a "most improved 2012!" writeup in a major industry magazine
which just might ...
... increase sales by 75% that quarter.

Now there's a lot of mights in there. It might not happen. But it might. And if we don't hold that brown bag lunch, then it definitely won't happen.

So our leaf - a brown bag lunch - has helped our tree - employee training. And because we helped our tree, we gave our forest a better chance.

Handle the leaf, and think about the tree. The forest will follow naturally from there.

Monday, November 15, 2010

The Forest, The Trees, and the Leaves: Part I

What's that old saying: "You can't see the forest for the trees"?

The truth is, we have to see the forest sometimes; sometimes we really need to be looking at the trees; and sometimes the leaves are the targets of our investigation.

If we consider that the smallest unit of work we can do, that is a leaf. A test session is a leaf. A bug verified is a leaf. A

The forest is what we commonly call "vision". The forest is the vision of the company and the ultimate goal: to make money for shareholders. When we look at the forest, we are attempting to directly answer the question, "What will X do to make money for the company?"

The trees are the in-between. The trees are the sum of our actions that together provide value for the business, and indirectly affect the company's vision. Often, we call the tree "strategy", and in a medium way it is. Employee training is a tree. Brand identity is also a tree. Customer service is a tree, too. Quality of product is a tree. There are many trees.

The leaves service the trees, the trees make up the forest. We don't ask what the leaf does for the forest; we ask what the leaf does for the tree. We ask what the tree does for the forest.

We often hear: "Don't do something unless it helps the company." I offer a corollary: "Consider how this helps the next biggest thing."

Over the next few days we'll talk about examples of considering the next biggest thing.

Tuesday, November 9, 2010

Manage People, Not Roles

Alan Page wrote recently wondering where the innovative thoughts in software testing really were (I'm paraphrasing; go read the blog post). In a comment, he mentioned that "The way we manage test teams (entire engineering teams, really) has to change."

He's right. We have to learn how to manage better.

I, like many managers, worked in my discipline (software test) and was promoted up to being a lead, then to being a manager. This makes me a technical manager; I understood the discipline of test very well. I understood the discipline of management rather less well.

Fortunately, I've learned a lot since I was first a manager. The single biggest lesson:

Manage people, not roles.

If I'm a Test Manager, there's a tendency to read it exactly that way: test first, manager second. When this happens, it's easy to approach every problem as a test problem. After all, test is the comfort zone, and manager is the new and slightly mysterious stuff. Things like team makeup become a matter of "ensuring coverage" by hiring different types. Things like correcting a team members behavior become "logging a bug" and "verifying the fix", which often translates to saying, "here's a problem. Let me know when you've fixed it." That might work when logging a bug for dev, but it isn't likely to work when you're dealing with someone who is frequently late on deadlines.

The better way is to realize that a Test Manager is a manager first. You happen to manage people who test. A Development Manager is a manager first, too; she just happens to manage people who develop. I'll give you three guesses what an Engineering Manager is first!

I don't manage testers. I manage people who happen to test for a living. People have different quirks than tests do. People need reinforcement and guidance and praise and correction and freedom to fail and encouragement to learn and a chance to succeed. This is true for developers, for testers, for support engineers, and probably for accountants and writers, too. (I'm guessing on those last two, but it seems reasonable).

My first responsibility is to the people who need to get the testing done, not to the tests that the people have to do.

If I can get the right people and give them the right environment, the testing will fall from there. Don't worry about the roles. Worry about the people. Get that right and everything else will follow.

Monday, November 8, 2010

Elevator Test: Part II

Apparently there are a lot of really awkward elevators in this world.

Here's an elevator button panel for us to test (courtesy of Robyn's Posterous):
Let's say the entire thing works correctly. I'm pretty sure I'm still going to log at least a few bugs. We've got usability and consistency problems galore.

Just because it functions doesn't mean there isn't a bug!

Friday, November 5, 2010

Goal vs Task

I met a friend last night out at a restaurant he'd never been to. To tell him where to go, I said, "from your office, walk down Boylston until you hit Dartmouth, and make a right. Then walk down Dartmouth and it'll be on your right." This was a mistake. You see, my friend wasn't coming from his office; he ran an errand on the way and all of a sudden my directions weren't very helpful. What I should have said was, "it's at the corner of Dartmouth and Clarendon." That way, no matter where he was, he could figure out the best way to get there. I told him what to do, not what my goal was.

This applies to testing, too. When the product manager asks me, "can you please run a performance test in configuration foo?", that's telling me what to do. It might be right, or I might get some element of it wrong and basically waste the test (and who has time to waste!). If instead, the product manager wants to "run a performance test to update this analyst with the amazing benefits of our new feature foo", that's telling me what you're trying to accomplish. With that I now know to run the exact same test as I did the last time we published results for that analyst - same data sets, same hardware, same system configuration - with the exception that I'll make sure feature foo is enabled. Because I understand what the product manager was seeking to accomplish, I'm much more likely to provide useful test results.

Don't tell me what you want to do. Tell me what you're trying to accomplish.

I'm much more likely to do the right thing, if I understand the end goal. So if I hear something that sounds like a task to me - that sounds like telling me what to do - I'll probably respond with something like, "Just so I make sure I get the information you want, can you tell me about what we're going to do with the results?" You save yourself a lot of time and test repetition that way.

Thursday, November 4, 2010

Expected Bug Levels

I had a conversation with my boss today that started like this:
Him: "Gosh, I've noticed a lot of bugs lately. The bug count is higher and there are lots of new ones."
Me: "Yup."
Him: "Should I be worried?"

Now that's an interesting question. The correct answer changes based on when we have this conversation. (In this case, we're just after feature freeze, so no, he shouldn't be worried.)

Every project I've worked on has a natural bug flow, like a tide in the ocean. Sometimes there are more bugs found, sometimes fewer. With small variations, it usually goes like this:

  • Project start: Early on in a project (or release), there isn't much new. Most of the work being done is analysis and coding on things that simply aren't checked in yet. The average number of bugs found in this period is pretty low.
  • Mid-Development Cycle: As some features start to get checked in, and the testers get some of their tests done, the bug count starts to creep up. Some bugs have been introduced, and some are being fixed, but the focus of the work is generally still on feature implementation, and the number of bugs found only comes up a little.
  • Feature Freeze: Regardless of whether feature freeze (also known as "a first pass at each feature has been done but it isn't baked yet") is formal or informal, there comes a point where pretty much all the basic implementation is complete. At this point the code is just starting to play together, and the volume of code being checked in is pretty high. The number of new bugs logged also goes way up; this phase often represents a high point in terms of quantity of bugs found. Note that this isn't inherently bad - we've got a lot of things and now we have to get them all to play together. It's just a relative high point, and it's to be expected.
  • Polishing: After the features are basically implemented and before release, the number of bugs found and number of bugs open generally trends down. Integration points are smoothed out, bugs that were ignored because someone was heads down in a feature are fixed, and the entire product solidifies toward something release-able.
This isn't a metric, and it isn't a value judgement. It's simply a project heartbeat. Seeing "lots" of bugs logged - regardless of what "lots" means to your project - might be okay or it might be a big problem, depending on where you are in a project. Take a look at your defect tracking system: it's one place that might help you see whether your project is behaving normally or if there's something going on that warrants investigation.

Wednesday, November 3, 2010

Manufacturing Software

Matt Heusser wrote a great riff on "Quality" as interpreted by American business. Go read it and come back.

Back? Great.

So here's the problem with many of the examples in Matt's piece: they're all about manufacturing. Taylor wasn't concerned with the software engineer; he was concerned with paper and steel manufacturing. Lean principles came from car factories. All these things are concerned with building something consistently and doing that as efficiently as possible. Innovation isn't intended to make something different or to make the product better; it's intended to make the product more consistent and to use less effort to make that product more consistent.

So now we apply this to software!

After all, software is about building the same thing over and over and simply doing it more efficiently and.... wait. That doesn't sound right at all.

This is the fundamental breakdown: as software professionals we seek innovation in the product itself, not innovation in the process used to create the product.

There is a process of manufacturing software. There is a compiler that manufactures bytecode. There is a library that calculates a hash. Those are manufacturing processes in the sense that the way we improve quality is to produce the same end result (the same bytecode, the same hash) more efficiently and more consistently. Taylor's "labor" - the workers actually doing the manufacturing - is all software for us now. We, the developers and the testers, we are Taylor's management.

We as management have responsibilities, too. It is our responsibility to seek better products for our workers (compilers) to build. It is our responsibility to seek out inefficiencies in the manufacturing process (are we using SHA-2 hashes when MD5 will do?) and to correct them. We are the ones who must conduct time and motion studies (only we generally call this optimizing the software or performance analysis).

So yes, go ahead and use a manufacturing analogy - just recognize that the humans are all on the "management" side of that analogy. (Yes, even those of us that don't have "manager" in our title.) Taylor wanted to make laborers into highly efficient drones with focused areas of expertise. In software, he has succeeded - a computer program makes a highly efficient drone with a focused area of expertise. It's time to move on; it's time to think about what we're building instead of how we're building it.

Monday, November 1, 2010

Developer and Tester Pay

When I'm building a development team, I'm looking for certain characteristics:
  • ability to learn quickly
  • logical thinking
  • familiarity with common patterns and techniques
  • knowledge of relevant tools
  • self-directed and able to solve problems on their own
When I'm building a test team, I'm looking for certain characteristics:
  • ability to learn quickly
  • logical thinking
  • familiarity with common patterns and techniques
  • knowledge of relevant tools
  • self-directed and able to solve problems on their own
Those look the same!

In many ways, testers and developers are really quite similar. A lot of the expectations we have for developers are the same as the expectations I have for testers.

So why would I pay my testers and my developers differently?

I'm hiring a software professional. Sometimes it's a software professional who happens to have expertise in front-end technologies; sometimes it's a software professional who happens to have experience with file systems; sometimes it's a software professional who happens to have experience with performance analysis. In all cases, I'm prepared to pay according to how rare that expertise is. Your average kernel developer is going to cost more than your average web developer. Your average performance tester is going to cost more than your average GUI tester.

Judge by the expertise you need, not by the title you're going to give. That will tell you how much you need to pay for the next software professional you hire.