Thursday, December 30, 2010

Feels Right and Is Right

I have a reasonably large stash of about 200 cookbooks at home. I'd been organizing them roughly by category (baking, Italian, fancy meals, etc.) but my collection was growing large enough I decided to turn the organization of my bookshelf over to the experts: the Library of Congress.

The Library of Congress catalog puts cookbooks in the TX category (T = Technology, X = Home economics. I'm not making that up.). So I sat down with my cookbooks and the online catalog lookup, and sorted and organized. When I was done, I had a completely correctly ordered bookshelf. It was great! It was perfec.... well, actually, that's funny.

There was a section that had six books in this order:
Northern Italian | Greek | Indian | Italian | Italian | Italian

According to the Library of Congress, that's correct (regional comes before whole country). According to me, that's confusing. So we have something that is right, but it doesn't feel right.

You guessed it: I moved the Northern Italian book to be next to its Italian brethren.

It's a useful thing to remember when we're testing. There's what the spec says, but we also need to consider whether the spec itself matches with the way the software will actually be used. When in doubt, and whenever it's possible, usable trumps correct. Every time.

Monday, December 27, 2010

Some Names Are Protected

This post falls into the category of "I knew that, but I didn't think about it."

I'm working on a project that is a small, fairly simple Rails application. Or seemed to be until I ran into this piece of code:
<%= Chart::TYPE_OPTIONS.each do |o| %>
<%= f.radio_button(:type, o) %>
<%= f.label :type, t("activerecord.attributes.chart.values.#{o}"), :value => o %>
<% end %>

This is on the new chart form, and it's one of several fields (but the rest aren't important). Everything was working fine except this one field; the field just wouldn't save. It kept throwing "Can't mass-assign protected attributes" warnings.

Anyone see the problem? This one took me a good 10 minutes.

Yup, "type" is a protected keyword. Don't name attributes "type".

I renamed it to "chart_type" and lo and behold everything works. I could probably have made type work, but in general I'd rather rename the attribute than try to figure out all the places I have to do something odd. The moral of today's story is: know your language and your framework well enough to know what words you just shouldn't use.

Thursday, December 23, 2010

Just How Much Do We Need?

Sometimes when we write a piece of software it runs for ages and ages and it get used a lot. Other times, we write very "one off" software. Migrations, data cleanup utilities, etc. can often fall into this category.

A friend is working at a company where they license television airing data (think a TV guide) from a third party. They then massage this data and put it into a database, then do stuff with it. they recently switched providers for this data, and are having to do a few one-time things to get the data to be clean throughout. For example, there was about a three-day overlap in provided data, so for three days they have a lot of duplicates (e.g., two database entries that say, "Rehab Wars aired at 8pm on Tuesday"). So they need a one-time job to reconcile these entries, get rid of the duplicates, and update references to this data.

So how much testing is needed?

Well, let's see. On the "we need to test a lot!" side:
  • television data has lots of quirks (weird characters, subtitles for shows, overlapping air times)
  • we can't take down production for this, so it's going to have to be done as other items are changing, which means we can't simply restore from backup if it fails. And undoing it is going to be non-trivial (aka pretty hard).
On the "eh, let's not be dumb about it but let's not go overboard side:"
  • it's a relatively small amount of data
  • we wrote it so that it simply stops on any error, and we're pretty aggressive about error checking (so the odds of it running amok and hurting things are fairly low)
  • we can do it by hand (aka someone in the db reconciling SQL) in about 2 days, so inflating the project much beyond that is kind of silly
I don't actually know what they decided to do in this case. I do know that they finished the project (presumably successfully).

So what tests would I actually run?

Here are the parameters of the project:
  • This is a single SQL script that will be run on a production database once.
  • The script will fail on any error or data mismatch (that it finds) and is restartable from the fail point. (allegedly, of course - you may want to test for this)
  • The script identifies duplicates by name and air time and merges their records in the database.
So now it's your turn: what tests would you run in this situation? What would you skip? Why?

I'll tell you tomorrow what I would do.

Monday, December 20, 2010

Why Look at Automated Results

A developer I know socially asked me today why QA looks at the output of the nightly automated tests. He said that at least at the places he worked: "development mostly looks at the output of the automated tests. After all, the tests say, ' I tried to do X and didn't' and that means development has something to go fix."

That's interesting. After all, where I work, QA looks at the output of the nightly automated tests and logs bugs (or updates them). But why? What does this exercise tell us?

I want to look at nightly results to:
  • identify regressions so they can be fixed
  • see how well new features (and their tests) work in the overall context of the system. Ever see module A checked in and completely - and accidentally - break module C? Often the automated tests show it.
  • as a catch to identify race conditions or other infrequent bugs that are more likely to be found the more often I run a given test that has a chance to expose the condition
  • look at coverage reports to see what hasn't been covered at all
  • find areas that I should be exploring more. If the, say, upgrade tests, break frequently, then I know they're probably fairly brittle and so I go explore that area of the code in my own testing.
All of those, except the last one, are things that could be done by developers or testers. But that last one - guidance as to what I should be looking at more deeply - that I'm not willing to give up. I get information out of looking, and that's worth the 15 minutes a day I spend on the task. Of course, development is welcome to look, too! More eyes will see more information.

Thursday, December 16, 2010

In Theory

How many times have we heard or said those infamous words: "in theory that shouldn't be a problem"?

That's nice, but I don't live in Theory. I live in Boston.

Now, the only mental leap is to prepend the phrase "in theory" to anything that says "shouldn't be X" or "should be Y". "Should" is a word that indicates uncertainty - and that's a place we're going to want to test! After all, Theory is a pretty empty place.

Tuesday, December 14, 2010

Correcting Vocabulary

Semantics: the words we use, the way we describe things, the jargon we employ. All these are hugely important and entirely trivial at the same time.

Good semantics is precision.
Bad semantics is off-putting.

Often the only difference between those is your audience.

For example, Michael Bolton talks fairly frequently about testing versus checking. He goes on at great length in several forums about how testing is a sapient learning process and checking is simply comparing X to Y (I'm paraphrasing; go read the article). That's all great, for the right audience. Making that distinction provides a learning opportunity for testers, which is wonderful.

And sometimes correcting that vocabulary is a really negative thing to do.

Here's a hint: your average Product Manager who wants to publish a test plan really doesn't care if it says checking or testing. Your average Implementation Manager who wants you to review a product evaluation plan really doesn't care that the client has provided a plan that says "test X" when it should say "check X". He just wants to know if your product is going to pass.

Remember that guy in school who was always correcting everyone? "You don't mean literally got pushed under the bus. You mean metaphorically." "Actually, the appendix does have a purpose; it's vestigial, not useless." He was usually right, but nobody liked being around him. He was so stuck in the details and in proving how smart he was that he completely missed the bigger picture - an expression of frustration, or comfort offered to a sick and worried friend. Don't be that guy.

I recently was reviewing an evaluation plan provided by a potential client, and it was written by someone who evaluates products for a living. He isn't trained as a tester; his background is development and research. His evaluation plan says things like, "test that the memory utilization is 1GB". That's probably a check. So did I send back a bunch of comments that describe what a check is and why these particular things were tests and not checks? Nope. It wouldn't have helped. He was going to do these things during the evaluation, regardless of what they were called (so why bother making him call them something else?). He's not going to apply that knowledge going forward; after all, this evaluation technique has worked for him for years. All I would have accomplished would be to send the message, "I'm scared of something here so I'm trying to throw you off your game by drowning you with vocabulary that isn't going to affect what you do and therefore really isn't important." Later, when we have a relationship and when there's an opportunity to talk about testing versus checking in an environment where he's open to learning, then I'll bring it up. But right now, it just looks like a snow job intended to make him feel dumb.

So when you go to correct someone's vocabulary - to say, for example, "You called that a test but it's really a check and here's why" - make sure you're doing it for the right reasons. Make sure it actually matters. If you feel the need to correct someone's vocabulary, ask yourself three questions first:
  • Does this person care whether he got this term wrong?
  • Will correcting him cause any positive behavioral change?
  • Am I doing this to make him think I'm smart or because there's an actual purpose?
Don't set out to prove you're smart. Set out to help. Accept that the English language is imprecise, and seek increased precision in vocabulary only where precision will help.

Tuesday, December 7, 2010

Tools-Oriented Test Plan

I've been reviewing a lot of test plans lately that, well, that weren't written by testers. They were written by fresh computer science graduates, or by developers for the most part. To be fair, I've also seen junior testers write test plans a lot like these.

They're the tools-oriented test plans. Basically, it's a test plan that says, roughly, "we're going to use these tools." The relatively good ones even say, "we're going to use the tools in these configurations.".

Reading a tools-oriented test plan is like asking your kid, "what are you going to play in the park?" and getting the answer, "there's a slide and a swing set and a monkey bars!"

Okay. That's nice. But what are you going to actually do when you get there?

The biggest problem with a tools-oriented test plan is that it doesn't describe what you're trying to accomplish. There are no success criteria. There are no intended areas of learning. There are no pass/fail or conformance criteria. Because it doesn't convey intent, you can't know if you've accomplished that intent. There's no way to know if you've succeeded.

The good news is that a tools-oriented test plan is often really good at making you aware of the existence of tools. It's also generally salvageable. You've written a lot about how you're going to accomplish things, so all you need to do is add and rework it to describe WHY you're going to use this tool and WHY these configurations are interesting or important. To get from a tools-oriented test plan to something usable, I do the following:
  • walk through each section and ask the question, "what is doing this going to tell me?"
  • make sure I put the answer to the question into the test plan
  • create a section called "tools"
  • put the tool name and a description of all of the relevant features into the tools section
  • replace all the tool info with a reference to the appropriate feature(s) in the tools section

But if you find yourself reading and saying, "I see what to do, but what will that tell me?", you've got a tools-oriented test plan. Tools are great, but you don't have to let them run you. Instead, put them in their proper place as servants to the information you need.

Monday, December 6, 2010

Quick Creativity Exercise

Sometimes we get into a rut. The team's the same for a while. The product's basically the same (sure, new features, but pretty much the same idea). The test strategy's the same. The customers are solving roughly the same problems with it. In other words, you have a stable team working on a mid-life product. In many ways this is a good thing. But it's a rut nonetheless.

As testers, we need to stay away from ruts in our thinking. There is an element of creativity in what we do, dreaming up uses, configurations, combinations, deployment strategies, and whatnot for the product. When the test team is in danger of a rut, I like to do a quick creativity exercise that I call "How Many?" (I'm sure there are other names for it, and it's highly unlikely I invented this, but I don't know who did.)

Here's how it works:
1. pick some small thing in an area you all know well
2. ask the team "how many ways can you accomplish this small thing?"
3. give everyone 5-15 minutes to brainstorm. Actually trying it is optional; the point is creativity, not necessarily that it would actually work.
4. compare notes
5. the person with the most things wins a small prize. We tend to put a congratulations message on that person's whiteboard or something (doesn't have to cost money or be a physical prize).

For example, we did one recently that was:
How many ways can you think of that a file might be removed from a perforce change?

Example answers:
  • the person owning the change deletes it from the file list
  • in doing a p4 resolve, the person might choose to skip the file
  • a trigger on submit might find a problem with that file and remove only it from the change (this, by the way, wouldn't be desirable, but it's possible)
  • the file might never have gotten added to the change in the first place
The game doesn't have to be about your product directly. (Hint: I'm not on a team that builds Perforce. We just use it for source code management) It should be about something work-relevant, though: your product, your tools, your customer's environment, etc.

If you're worried about a rut, give it a shot. Just a few minutes of explicit creativity can make a difference across tasks.

Thursday, December 2, 2010

Error Messages You Don't Hate

Getting an error in something you're using stinks. You have to stop what you're doing, figure out if you did something wrong or if there's a problem with the tool you're using, figure out a way around it, and only then can you get back to what you were actually trying to do in the first place.

How long this takes and how you feel about it partly depends on the error messages you get. Some are no help at all: "An error occurred. Please try again."
Some are incomprehensible: "Error 0x930432. Contact support."

But there's only one that made me laugh out loud yesterday:
"421 Timeout - try typing a little faster next time"

Go ahead and have fun with error messages. Be informative and a little irreverent. After all, when's the last time an error made you smile?

Wednesday, December 1, 2010

Team Jazz Band

We had the following conversation yesterday:

1: "Hey, look! The mem option is a signed int. Whoops!"
2: "Oh yeah, 3 just logged one for the scanner that the object option is signed"
1: "Cool. Let me see..... oooooh that one core dumped. Mine just threw a bad error message."
3 walks in with some hot dogs (it's lunch time)
2: "3 wins!"
visiting dev: "what does he win?"
3: "hot dogs, apparently"
visiting dev: "you got him hot dogs for a core dump?"
2: "oh! business idea! core dumps spit out hot dogs. We'll make a fortune in the food court!"
1: "sweet"
random other dev coming in for the day: "how'd nightly tests do last night?"
visiting dev: "nightly did great. Those humans are making up for the gap by logging lots of bugs."
1: "Yup, option testing on the different binaries. It's producing kind of a lot."
1: "Hey, 2, did you ever figure out when that performance drop happened?"
2: "I know it was after the 28th, because I just dug up the test results and they were faster then. So I wanna say it was in early November. Still digging."
1: "Visiting dev, I think the boss said you thought it was the 5th of November or so. Was that based on anything or just speculation?"
Visiting dev: "Boss's speculation, maybe. Although the 5th of November sounds familiar."

And the conversation continued on from there. I felt like hanging a sign on the wall: "Serious testing going on here."

Were we having fun? Definitely. We're the laughing-est group in engineering.
Precision language? Nope.
All work related? Nah, but most of it.
Were we testing? You bet.

I like to think of this as the jazz band state for the QA team. We're all working on separate things, and if you asked us what we were doing we would say:
1: Testing options on the reader binary
2. Trying to track down a performance regression
3.: Testing options on the scanner binary

But we're doing much more than that. We're creating ideas and bouncing off each other while continuing to work on our separate things. It's like loosely coupled pairing. We're not really pairing, but by keeping an ear open to each other's conversation, we get ideas to improve our own tests and we answer each others' questions.

It's a great flow state for us when we can get there. Our tests improve, our speed improves, and our rhythm improves. So whenever we're all in the office testing (no meetings or other interruptions), we try to work on similar problems in different areas. For example, today most of us were working on testing the options to a binary (--verbose, --mem, etc.). That way our ideas actually make sense to each other for immediate use. We turn from individuals playing our instruments into a jazz band, bouncing solos and harmonies off each other and making great music along the way.

Of course, like all musicians, sometimes we need to practice on our own. That's what the team quiet room is for; we put a computer in the team room and each of us can shut ourselves away for some quiet focus time. But the real magic happens when we come together and play a tune.

Give it a try and see what wonderful music ya'll can play together.