Wednesday, March 31, 2010

On Simplicity

"There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult." - C.A.R. Hoare

This applies to software design, project design, and software test. The more simple it can be and still solve the problem, the better off you are. If you can write a test with 16 steps to show a bug, or a test with 4 steps to show the same bug, go with the four-step test. If you can put together a project plan on the wiki you already use or you can go buy project management software and learn it, go for the wiki.

Make it easy. Make it work. That is simplicity.

When things stop working it's generally fairly obvious. Something will break, or in the case of human processes, something will start to feel wrong. Functionality is a prerequisite; if it has to be more complex to work, then it has to be more complex.

Once it works, though, making it as simple as possible is the goal. It's hard to do, and it's scary to decide to skip something or to not do something. But as long as it works, ask yourself if you can do it more easily or with fewer steps.

It's hard but possible to be simple enough to obviously work. Let's see how close we can get.

Tuesday, March 30, 2010

Directness of Test

Sometimes when we write a system test, it looks like this:



Sure, we're ending up with the test we want - scratching our back - but it's hardly direct. We're having to work ourselves really hard to get to an end result. See that window? We're standing there, spraying water trying to make the back scratch. That seated gentleman? He's your web server. Only if he's working right will we get to the breaking glass that's the responsibility of our app server. And only if the app server is also working properly will we eventually scratch our database back. Phew!

Wouldn't it be better if we just did this?


If we want to test scratching our back, we should test scratching our back. System tests (aka end-to-end tests) have their place; it's useful to know that all the pieces line up together. If what you're testing, though, is some inner piece of functionality, you can choose to address it directly with tests.

Directly testing a component:
  • Allows us to provide better coverage with smaller changes (just move our hands around, not construct all sorts of variations)
  • Creates a test with fewer dependencies, which is therefore easier to run and maintain
  • Is more reliable because fewer things have to work to even get to the test we're interested in
  • Is more maintainable because it's less susceptible to changes in the behavior of the other parts of the system between us and the behavior we're addressing
So if you're looking to thoroughly exercise a component, test the component as directly as possible. Save your system tests for testing the entire system and how it works together.


Monday, March 29, 2010

See With Your Own Eyes

We ran into an interesting problem today. We have a new piece of hardware in house that we're running some tests on. It has been around long enough that we know things pretty much work, but not long enough that we've looked at everything we're interested in, or know all the details of. Someone asked, "Does this machine have one processor or two?"

Well, now, that's an interesting question. There are a number of different ways we can look at this:

cat /proc/cpuinfo | grep "processor"
That tells us how many processors we have, but processors here really means cores of any type, real or virtual. It could be two quad-core CPUs, or one quad-core CPU with hyperthreading enabled, or two dual-cores CPUs with hyperthreading. Better keep looking....

cat /proc/cpuinfo | grep "core id"
This one sounds good, at first. However, it returned us 0 1 2 3 0 1 2 3. Hmm... that could be one quad-core with hyperthreading, or two quad-cores. Shoot.

cat /proc/cpuinfo | grep "physical id"
This one might work. In theory it shows the socket into which the processor was put (0 or 1). Hope there's no bug there...

The problem with all of these is that we're not talking to the hardware; we're talking to the OS. That's a level of abstraction that introduces all sorts of possible bugs. Maybe we've configured the OS to use one CPU even though there are two physically present. Maybe we've disabled hyperthreading so its counting physical CPUs properly but we could have more.

Sometimes, the only way to be sure is to open the box and see with your own eyes (it turned out to be one quad-core processor with hyperthreading enabled). Believe your evidence, but check your tools with your own eyes just to confirm.

Thursday, March 25, 2010

Why We Run Performance Tests - Part 2

Performance tests are one of those things that can be run forever. Unfortunately, we don't have forever. So given restricted time and restricted resources (machines, network, people), we should only run performance tests that actually provide some value. Yesterday we talked about some of the outward-facing needs filled by performance tests. What about inward-facing needs?

Fail Fast Deterioration or Improvement

Why Do We Want This?
When we do functional testing, we look for things to fail quickly. When we break something, we want feedback as soon as possible that we've broken it, so we can fix it before we go too far down a bad path. The same theory applies to system performance (however we've chosen to measure it). Maybe that refactoring made the code a lot cleaner but also got you a performance bump; wouldn't you like to know so you can do more of that kind of refactoring? Maybe that memory-saving innovation actually killed your performance; it's probably better to find out before you get too attached to it or even worse, have to go mucking with it at the very end of a release cycle.

How Does This Change Our Test?
This test is all about rapid feedback of areas likely to change. These tests should be as short as possible so they can be run on checkin, or daily, or at worst weekly. Automated reporting is useful here, just like it is in a checkin or smoke test for functional tests. For this kind of performance test, change is what's important. Making the tests run consistently is more important than things like upgrading hardware or trying different things. Also consider testing at a lower level than full system (i.e., component performance tests) to pick up more subtle performance changes.


Support Performance Work

Why Do We Want This?
The time will come eventually when we want to actually improve the performance of the system. At this point, there is a need to more fully profile the system as it performs under different loads, find bottlenecks, and change the system implementation to make it run faster under those interesting loads.

How Does This Change Our Test?
This test is mostly about finding bottlenecks and places for optimization. The goal here is to make it easy to run tests, because there will be a lot of different variables, a lot of speculation, and a lot of "okay, let's try this new build". For these tests, look at running with profilers, different load characteristics, etc. Given all the flexibility here, the way to handle these performance tests is to bound them on some non-test variable. Usually, these will be either time-bound or speed-bound. For example, the charter might be "we'll spend two weeks on performance improvement" or "we need to get to Y MB/s". Consider dedicating a team member and some resources (machines, etc) to the effort until the charter is finished.


There are a lot of other reasons to run performance tests; these are some of the most common I've seen. In any event, your tests will be much more effective if you can describe why you're doing them and what you will do with the results. These posts have provided examples for performance tests, but it works for other tests, too, including functional tests, usability tests, testability tests, even security testing. Use your needs to inform your design; you'll waste a lot less time and make everyone involved a lot happier.

Wednesday, March 24, 2010

Why We Run Performance Tests - Part 1

One of the wonders - and dangers - of performance testing is that you can run it forever. You can look at latency, or scalability, or throughput. You can change all kinds of variables about size of the system, what kind of traffic is coming through, system configuration and size, etc. It's no wonder some people make a full time job of it! Then there are the rest of us. We don't have a dedicated performance team or a dedicated tester. We do performance tests along with all the other tests we do. This model is pretty darn common, too.

So when you're restricted by time, as many of us are, you have to narrow your tests. In order for them to still provide value, you can't really explicitly hit every variable. So you have to optimize your tests to provide the information you're looking for. Let's say, just for example, that we've decided to run performance tests based on aggregate read and write throughput to the system. What would we do with those kinds of test results? To phrase it another way, why do we run these performance tests, and how do they affect what tests we run?

Marketing numbers

Why Do We Want This?
Marketing needs a nice fast number to put up against all of our competitors. This number usually represents the fastest possible aggregate ingest we could ever hope to see in the field under ideal circumstances. This will go in analyst briefing materials, brochures, and other marketing materials, usually with phrasing like "speeds of up to X MB/s". Typically these get run every time something substantial changes, like new hardware, or a major new feature, or a big performance push. It doesn't necessarily happen with every release.

How Does This Change Our Test?
This test is all about optimization to get a big number. Based on field experience, past performance tests, discussions with dev, we're looking for the fastest configuration we can find and the fastest data we can find. Empty system? Large system or small system? Number of clients used? Amount, type or size of data ingested? All of these things are geared toward getting a fast number. Don't ever cheat. Don't ever do something unsupported. Your numbers may be checked and you'll need to be able to show how you did it and why that's a conceivable number. But beyond that, optimize away.


Prevent Client Slowdowns

Why Do We Want This?
When we put a release into a field, you want to be sure that it doesn't hurt your existing customers. Just like we try to avoid functional regressions, we try to avoid performance regressions. The goal of this testing is to ensure that customers get a performance experience at least as good as what they have today. These performance tests need to be run on each release.

How Does This Change Our Test?
To give confidence that we haven't hurt our customers, we need to be doing approximately what our customers are doing. We won't be a perfect imitation, but the bounds need to be about the same. This means a typical customer hardware configuration, a typical software configuration (e.g., compression enabled), typical number of clients writing typical numbers of files. One test probably won't cover it, if your product is being used in several ways. Create a performance test for each basic use case in the field.



There are other purposes to performance tests, so we'll continue tomorrow.

Tuesday, March 23, 2010

Being Reactive

How many times have all of us heard or uttered the following sentiment?

"We have to be proactive about this!"

The implication, of course, is that we have to get ahead of the situation, put a plan in place, and stop simply reacting to what's going on. If you're reacting, you're always going to be just a bit late. If you're proactive, though, you can anticipate things that are going to come up and have an answer or a solution for them. Laudable ideas, all of them!

But... (there's always a but)

There is a time to be reactive. Being purely proactive means you have a plan, and you will stick to that plan. Since we're note all great prognosticators of future events, our plan is likely to need some modifications as we move forward. We're going to have to react to events as they occur, making ourselves a bit reactive (gasp!).

This is not a call to stop planning or to stop being proactive. There is immense value in having answers ready when you need them because you've proactively gone out to get them. Your clients will feel much more comfortable if you can answer their questions and concerns readily because you've been proactive. Your colleagues and subordinates will be glad you anticipated the pending lab power outage and didn't leave them with a lot of half-built code and half-run tests. You'll be better able to handle crises when you've considered them and what you have to do beforehand.

Be proactive in your overall plan. Be reactive in the details.

Being proactive involves thinking about the future likelihood of events and preparing for them today. As those events unfold, though, keep looking and react to them. Your earlier preparation means you have the big pieces in place. Your reaction will make sure those pieces are tweaked to work as the situation actually warrants, and not trying to make something not quite right work.

So please, be proactive. But be reactive, too. The combination is what will get you ahead.

Monday, March 22, 2010

The Mall Door Problem

Going to the mall is sort of an interesting business. I'm not generally a shopper, per se, so when I got to a mall, it's usually to get to a store, get what I want to buy, and get out, preferably without snapping at anyone or getting lost along the way. Malls aren't really set up to facilitate this. Take a look at a fairly typical mall layout:




There are multiple levels, multiple entrances, different hallways, different orientations, and a mixed numbering/naming/symbol system. It's confusing.

...

That sounds a lot like a modern system, doesn't it? A modern piece of software has modules, classes, mixed naming/constants/variables, different interfaces, and multiple access points to each logical entity. It's confusing.

And somewhere in all of this is the module we're looking to exercise, and the problem we're looking to track down. So, our goal is to track down this problem or to exercise this module. Our goal is also to get in and out as cleanly and quickly as possible.

We need to find the mall door that provides the most direct route to the module (or store) we want. Thinking not only about our task or problem, but also thinking about how we get there will help us:
  • reduce our steps to reproduce
  • decrease setup time
  • improve the reliability of hitting the problem we're looking for (especially if it's a race condition or a deadlock)
So, how do we do this?

The actual solution to the mall door problem will be custom to your system. However, there are some principles that can be applied to a given module or area under consideration. Consider the following questions. At the point where you can answer "yes", you've achieved your goal of finding the closest mall door to your behavior. You win!
  • First, can the module be addressed directly? Are there unit tests, an exposed API or other means of access? Does this direct address show the problem or behavior I'm looking for?
  • If not, what other module might be interacting? Can I address just these two modules? Are there system tests, mocking, an API or other means of access? Does this show the problem I'm looking for?
  • If I can't get the problem or behavior with two modules, it's time to try three. Can I get any three modules to show the problem or behavior?
  • If I can't get the problem or behavior with three modules, then I'm looking at a system problem. So let's start from the outside and start throwing things away. Consider the system, and eliminate any APIs or entry points that do NOT show the problem. Keep throwing things out until the problem or behavior goes away. Then back up a step, and there's your mall door.

The goal is to approach a problem from its closest entry point. Over time, this goal will help make you check your assumptions about entry points. If getting to any problem or behavior has to be done with the entire system, well, that's about like having to walk all the way to one end of the mall to get inside, only to have to walk most of the way through it to get to your store. That walking is wasting time and wasting effort. Consider putting in a side door. Maybe the side door is a new testing interface. Maybe it's a mocking framework. What specifically it does will depend on your system. The point of finding the mall door is to consider how you can address components of your system in the most efficient way that still produces results, even if you have to change your components over time to do so. We call these testability requirements, and they're great changes to make. You won't find them until you start looking, though, so go ahead and seek the closest mall door.


Thursday, March 18, 2010

Patching the Problem Cause

What can we do to make sure this never happens again?

Usually this question gets asked after something bad and embarrassing happens. Maybe it's a problem at a customer site, or a release you have to retract because it just doesn't work in the field. Either way, you're now embarrassed, you have fixed the immediate problem, and you're looking to avoid being embarrassed again.

So you ask the question: "What can we do to make sure this never happens again?"

On the surface it sounds like a good question. Something went wrong, and it went wrong in a way that was more public than you'd like. How do we make sure that this doesn't go wrong in public again?

But... it may not be the right question to ask. It runs the risk of fighting yesterday's war and focusing too narrowly on the specific problem you just had. After all, this particular problem is already fixed. We have a clear underlying goal: to avoid being embarrassed in this way again. The real challenge is to fix not this problem but to prevent similar problems, too. Let's look at what kinds of things we may need to do to accomplish this goal.

  • Find the hole that let this out and close it. Do we have a gap in our regression tests? Are we squeezing design or implementation or test and this is a corner that got cut? This root cause analysis is essential to prevent similar-but-not-quite-the-same problems from occurring.
  • Add a test for this particular problem so we don't regress. This may or may not be worth the cost of the test (in time and in resources). Before we go running off to do this, it behooves us to consider whether we are prone to regressions and whether this is the type of problem that will respond to that. Memory leak in a component? It probably makes sense to run that component under Valgrind to detect memory leaks before we ship. Design flaw? A test is probably not the right place to fix it; it's way too late by then.
  • Look for other instances of the same problem. Sure, we found this problem, but what related or similar problem might we have missed? Maybe we found the SQL injection flaw in the signup form; that means it might be a good idea to look for the same flaw in the login form. Depending on the root cause of the problem, this may mean looking at a lot of other things or very little extra work.

It's understandable to want to prevent something bad from ever happening again. However, just patching the cause of the problem is not sufficient, so avoid the knee jerk reaction and do some root cause analysis on the reason the problem came into existence, not just the problem itself. This will give you more chance of preventing similar problems, and help make sure the problem you just had doesn't come back, either.

Wednesday, March 17, 2010

Apples, Oranges, and Strawberries

Many times we'll run tests in an attempt to compare two things. Maybe we're comparing our performance against that of a "comparable" (read: competitor's) product. Maybe we're comparing new software on cheaper hardware to see if we've lost speed in a "comparable" (read: offers our customers about the same thing) configuration. These comparisons are completely legitimate, but we need to be a bit careful with them.

Maybe we do want to compare apples to oranges. After all, they're both fruits. They're both round. They both come from trees. They both have seeds on the inside.

Apples are like oranges, if the thing we're trying to compare is seed location.

Maybe, though, we want to compare apples to strawberries. After all, apples and strawberries are both red. Perhaps we're most interested in making sure that whatever we compare is red. Or perhaps it's that they both have stems that come with the fruit. Maybe it's fruits with edible skin.





In order to make a valid comparison, you have to identify the characteristics that you really care to compare. If there are things that must stay the same (performance, say, or cost to the customer), then make sure your comparison respects those things.

Monday, March 15, 2010

Unreproducible Toyota

Last week, a California man caused a news sensation and a large traffic jam when his Prius went 90+ miles per hour on a California highway and only stopped when a police officer used his cruiser to slow the thing down. The car was tested over the weekend, and engineers were unable to reproduce the problem. Naturally, the news media is going with the claims that the man driving the car must have been lying. Gotta get those viewers!

I'm no automotive engineer, but I think we've all seen our share of problems that wound up being "unreproducible". Think of all the things it could be:

  • A race condition in software and/or hardware
  • A deadlock in software (does hardware have deadlocks?)
  • User error: the man didn't follow a procedure that could have stopped the car
  • User error: the man maliciously managed to get his break lights to turn on without actually attempting to break
  • Incorrect entry conditions: the problem only happens after the car has been driven for some period of time
  • Incorrect entry conditions: the problem only occurs when the car is at a certain speed and some other event (momentary breaking, gear shift, acceleration) also occurs
  • Incorrect entry conditions: the problem only happens when the seat is in a certain position (thus putting the angle of pedal pressure at some key spot)
I'm sure there are many many more possible explanations. Like many unreproducible issues, this one may never be definitively solved. It's an interesting exercise, though, to come up with ideas. What other possibilities can you think of?






Friday, March 12, 2010

Useless Phrases

Certain phrases add huge amounts of value. For example, "I know! We're not releasing the mutex in the finally!" is often a useful thing to say (assuming you're working with mutexes and try/catch/finally blocks). Other phrases don't have much value. Sometimes we're just speaking to fill the air, and that's both useless and makes us look kind of ill-prepared to actually make a contribution. Some examples:

Useless phrase: We're overthinking this.
Why it's useless: You're not offering a solution or a way to proceed. You're complaining.
Useful variation: Let's simplify this by going with X.

Useless phrase: Just get it done.
Why it's useless: Again, not a solution. A demand, and probably a silly one at that. There is an exception for this one where the non-doer is really an employee who spends more hours in a day on excuses than on actual work.
Useful variation: What are the barriers to doing X? Here's how we're going to work around each of them.

I'm on a mission to expunge useless phrases from my vocabulary. What other useless phrases do you know?

Thursday, March 11, 2010

Situation vs Conundrum

There are two relatively common testing scenarios: situation-based, and conundrum-based.

Situation-based scenarios are those where you create a situation and see what happens. This is often what we think of when we say we're testing something. We decide to log in with spaces in the name and see what the system does (appropriate error message, let us in, blow up dramatically, etc). We create a degraded hardware scenario (translation: yank out a hard drive) and see how long it takes the system to send a notification while we induce heavy load. All of these are scenarios where we started with the situation, and we are looking to see what might happen.

To be effective in a situation-based scenario, a tester needs certain skills:
  • Creativity in imagining things that might happen (coming up with scenarios)
  • Ability to analyze a system to identify interesting actions and potential interactions
  • Ability to closely observe a system and identify resultant behaviors, even when they're not directly visible
Conundrum-based scenarios are those where something happened, and now we get to figure out what on earth went on. When you get a problem in from the field, or a bug reported to you, you're looking at a conundrum-based scenario. You have a user who was "just writing, like normal", and the system crashed. Or you have an error message in your logs that development assures you "shouldn't happen", and you don't know how to get there. In conundrum-based scenarios like this, you are given the result, and you need to describe the situation.

To be effective in a conundrum-based scenario, a tester needs a different set of skills:
  • Ability to analyze a single state and identify relevant facts (e.g., the core file, not the log file name)
  • Grep and other analysis skills
  • A mental model of the system to identify operations and other actions
  • Creativity in scenario imagination
Working with situations and working with conundrums requires complementary but slightly different skills. When working on a problem, figure out which one you're facing. You'll be far more effective if you know what skills to apply.



Wednesday, March 10, 2010

Mining History

You have a hugely valuable resource in your defect tracking system. You have a history of the bugs you have found in the past, and that can tell you all kinds of things about the future. For example:
  • Looking at which modules had the most defects in the past can tell you to look at those modules early, because they're likely to produce defects.
  • Looking at which modules had a lot of defects in older releases, and then fewer defects over new releases can indicate that the module had underlying issues which are now fixed. Translation: spend a bit less time here.
  • Looking at where you find issues in the field, and which customers have problems. What about their use cases are we not accounting for properly?
  • Looking at where in a release issues get introduced: during development, right after feature completion, late in the test cycle, etc. Moving defect discovery up might become a goal, if you're finding bugs too late.
As always, research is better with a question in mind, but consider researching in your defect tracking system. You never know what you'll find!

Monday, March 8, 2010

Estimates, Not Ifs

"How long will this test take?"

It's a reasonable question. The answer is generally reasonable, too.

"It'll take about 3 hours to set it up, then the test itself should take a few more hours. So, it'll take about a day."

There is a silent "if" at the end of that answer, though. After all, a lot could extend the time:
"... if the I can get on the hardware I need, which is currently being used by another test."
"... if the bug that blocked this test is really fixed."
"... if I don't get pulled off to another meeting between now and then."

As testers, we get used to providing information, so it's tempting to outline the ifs as well as the estimate. Most of the time, you shouldn't.

A test estimate is just an estimate. As the estimator, it's your responsibility to account for non-optimistic scenarios: all those ifs. So for each if, you get to consider how likely a problem is to occur, what you'd do if it occurred, and what that means for how long it will take you to do the test. Maybe all of those things are extremely unlikely; maybe something blows up 80% of the time (ouch!).

In either case, your estimate should now look something like: "About two days, allowing for the usual hubbub of life, plus the few hour setup time, and leaving time to run the test itself."

When you're providing an estimate, keep in mind that you need to describe the estimate, not all the things you can think of that would go wrong. Account, don't over-elaborate.

Friday, March 5, 2010

Log Errors

Let's assume you have some tests that do automated logging. Let's further assume that you are looking for results in the logs and spitting out pass or fail information. This is great; you're doing rather well. Sometimes, though, it's good to just double check your work. Look beyond the pass/fail and see if anything else is going on.

Look for errors in your test and product logs periodically.

This doesn't have to be difficult; usually a simple find and grep will work. But go around the error Usually you'll find small things that can be easily cleaned up - failed attempts to copy files that you deleted from your repository, warnings about permissions when moving files that leave a few logs behind, etc. Occasionally you'll find really large things, although hopefully this is rare. A simple find and some cleanup, though, will help make sure your tests continue to do what you expect them to do, and continue to test what you expect them to test. Think of it as spring cleaning, tester-style.

Wednesday, March 3, 2010

Positive Negativity

One of the risks of being a tester or QA engineer is that you can come off as a Chicken Little if you're not careful. Part of your job is notifying others of risks, and of telling them about things that are going wrong, that have gone wrong, or that may go wrong. It's all a bit negative, really.

Testing isn't actually a negative profession, though. On the contrary, quite often the system does what you'll expect, and you're finding information that reduces risk. You're proving that the intent of the code matches the desired user experience. You're saving field and customer embarrassment when you do find bugs. These are all positive and good things.

Your tone is what will ultimately dictate how testers come across, and whether you're perceived as a fearful negative type or not. It's not as much about what you say as how you say it. Let's assume you want a positive tone (since after all, you want people to keep you around, and that's more likely to happen if they don't think of you as Dr. Doom And Gloom). There are ways you can say things that will help this. A few tricks are all it takes:
  • Start with the positive. What you say first sets the tone for the conversation. So start with something positive. For example, if it's a status update, start with the tests you did that worked, or showed better than expected results.
  • End with the positive. What you say last will shape what people remember about the conversation and particularly your attitude. So end with optimism. For example, if it's a status update, end with a test you will run rather than a risk you perceive.
  • Be concise and factual about the potentially negative. Present risks, bugs, and other "negative" things without hyperbole and without excessive rhetoric. For the most part, people you're talking with are going to be able to understand the consequences. For example, you might say, "we found this bug. It's a data loss bug, that could affect large customers X, Y and Z." You don't need to go further than that; it's pretty obvious that it should be fixed at that point. Adding rhetoric like, "and we could lose the customer over it" doesn't help any one's understanding of the issue, and only makes you look negative or scared.
  • Just the facts, ma'am. Don't say, "I'm worried that blah." Say, "Under X circumstances, blah may occur." The point is to be the bringer of information, not of fear.
Risk-oriented, possibilities, worry, and negative findings are all part of our business. That doesn't have to mean that we are worriers overly focused on the negative, or that we need to be perceived as such. We can't always control the information we have to present, but we can control how we present it. The calmer and more positive we appear, the less likely we'll be dismissed when something really is wrong. And that's what we want. So present the negative, but do it in a positive, effective way.

Tuesday, March 2, 2010

Interesting and Useful

On occasion, the following phrase will be heard in the QA area: "It would be interesting to know....". "It would be interesting to know what our performance is like in a degraded network situation." "It would be interesting to know what kinds of widgets can fit into our frobble." "It would be interesting to know how many angels can dance on the head of a pin."

All of these things would be interesting. For most of those things, you can design a test to show you the answer, too. That doesn't mean we're going to run the test.

For it to be worth running, a test has to be both interesting and useful. Knowledge for the sake of knowledge is great, but ultimately that's not what we're here for. We're here to ship software and to help make good decisions. Our tests need to further those goals, not just provide data.

When you come up with an interesting test idea, before you design and run the test, ask yourself, "What am I going to do with the results?" Until you can answer that, you're not ready to run the test. If you can't describe what you'll do with the outcome, it's likely that you don't really know what kind of outcome you're looking for. And if you can't describe it, you're not likely to actually see it when the test happens. You may measure widgets fitting into frobbles larger than you'd ever actually ship, or in a configuration that your customers don't have. You may wind up measuring the performance on a connection that your customers would never even consider using.

Identifying the usefulness of your test results will help you define a more precise test that is still interesting. Interesting is great. Useful is even better. Interesting and useful? Now you have a test worth doing.

Monday, March 1, 2010

Moving Goalposts

"Goals" are incredibly common, at least in the companies I've worked in and around. Usually there are the big corporate goals along the lines of "book $X in revenue this quarter". Your team probably also has goals as well: "Get our bug count under X" or "Have a week of clean test runs" or "Improve our estimates so the next 10 stories are within 10% of estimate." Having goals is great. Meeting goals is great. It's also not quite enough.

Once you've met your goal, it's time to move the goalpost.

For example, let's say you're trying to get your bug count under 50. When you meet that, it's time to move the goalpost. Celebrate your goal however you like (our team is very food-oriented, so it's usually muffins or cookies or something). Then move the goalpost. It was 50? Now get it under 30. Oh yeah, and stay there for some period of time.

When you meet your goals, go ahead and celebrate. Then move the goalpost. After all, goals aren't about easy; goals are about accomplishing better things.