Monday, September 17, 2007

Get a Good Data Set

Creating good test data is underrated. In the end, no matter how manual or automated, how exploratory or scripted, how this or that, having good data to test against is paramount. It can turn "yeah, it works in THAT situation" into "absolutely. It works."


A good data set, easily loaded, makes testing a lot of different bug aspects and edge cases easy. You don't have to spend a lot of time doing data entry; it's all there, in your data set. The point is to keep testing and not have to stop to change your data.

So what characterizes a good data set?
- You can load it easily and pretty much all at once.
- You know what's in there.
- It's got lots of good stuff like special characters and different languages.
- It shows the edges of your situation (blogs with no posts and blogs with 100 posts in a day)
- It has missing data (people with no first name, for example)
- It doesn't have overlapping data. If you have to change underlying data mid-test, that's an opportunity to stop testing.

More on creating good data sets later.

Friday, September 14, 2007

"Is It Ready?"

It's an insidious question, but one day you will be asked: "So, is it ready?".

It's tempting to say Yes or No, but don't succumb. The truth is, you probably don't actually know the answer. Think off all the things you need to know to understand whether a release is ready:
- what defects are there
- what features are there
- what defects were fixed
- what defects you probably haven't found (at least what areas, types, and severities)
- the tolerance of your customers for the defects you know about or suspect
- the drivers behind and importance of the release date itself (maybe it's tied to a trade show, for example)
- the feasibility of a follow-up release for bugs
- whether customers WANT the release

So don't sell your company short. Provide the information you have, and get the information you don't. It takes more than one person to make a release!*

* If you're working alone, then obviously you play all the roles so you have all the information.

Thursday, September 13, 2007

Special Instance of Murphy's Law

The surest way to get a change in a component is to tell your team, "Let's go ahead and start testing this. It won't change between now and release."

Darn it.

Wednesday, September 12, 2007

A Test For Every Bug

I'm working on a project for which I'm the sole QA engineer. We release to production approximately once a week. It's a web app, and a relatively simple one - simple reads and writes. There are probably 30 pages in the app. The only difficult part about it is the release frequency.

Early on, I decided that since I was working alone and releasing often, automation was going to be my friend. So I instituted a rule: no bug gets closed without an automated test written. I made little attempt to automate general functional tests, although I did some as I had the time.

Today, 6 months later, we haven't had a single regression ship. There have been regressions, sure, but they've all been caught by the developers on the project (they have the ability to run the tests I've written). I realize this isn't rocket science, but chalk me up as a fan of this technique!

Tuesday, September 11, 2007

Unintended Consequences

To me, the hardest bugs aren't those that are errors in functionality (something that is simply not implemented correctly). The toughest bugs are the "unintended consequences". These are the bugs that come from implementing a feature and not thinking through how it fits in to the product as a whole.

Sometimes unintended consequences are small: I now have this setting in two places and I need to refresh one place when the user updates the other.

Other times unintended consequences are large: We store files on a file system and the website pulls directly from that file system. However, files below a certain size are stored in the record - too bad the website can't get to them, even if everything else can.

Either way, these are the bugs that almost always get to QA before they're discovered. These are the bugs that bite you right before release.

Preventing unintended consequences is really simple: have a system design and walk through every feature with that design in mind. Preventing unintended consequences is also very hard: you'll never think of everything a user will try. In the end, having an updated, written design is a good first step, even if it doesn't solve the problem entirely.

Down with unintended consequences!

Monday, September 10, 2007

Phrases That Make Your Jaw Drop

Overheard in a meeting today:

"Just tell the developers to not code bugs."

I didn't know what to say. Still don't.

Friday, September 7, 2007

My Toolkit

I wanted to share some of the tools in my basic testing toolkit. These are the things that I take with me to every job and that I couldn't imagine living without.

1. WireShark. This is a network trace tool. It records all the packets that go over your network card, including headers, packet contents, ACKs, etc. There's very good filtering available, and if you get totally lost, you can just hit "follow TCP stream" and get a nicely formatted request response set, all color-coded for you. This one's also good for a laugh; if you want to see a non-technical manager's hair stand on end, tell him you're using a packet sniffer!
2. SnagIt. Sometimes a picture really is worth a thousand words, and SnagIt makes getting a screenshot easy.
3. A scripting language. If I'm on windows, it's batch or VBScript. If I'm on *nix or mac, it's bash shell scripting. What really matters is that the script tools run on target systems without requiring installation of anything underlying (no JVMs, etc). This is for quick easy test scripts to deploy software, generate data, automate those repetitive tasks, etc.
4. FireBug. Want to know what your browser's really seeing? Check out FireBug. It's a FireFox add-on that times downloads, shows you the JavaScript that actually gets used, etc. This is a huge help for web development.
5. SQL Query Analyzer or Toad. Or whatever your database supports. If you can't see your database, you can't see what it actually thinks you have. This is great for getting to the source of data problems. For as long as we're storing things in databases, SQL will continue to be an essential language, and database querying without your app as an intermediary will continue to be an essential tool.

What are your absolutely essential tools?

Thursday, September 6, 2007

You Choose It, You Do It

I like to call my management style "you choose it, you do it".

In the end it's pretty simple. If you work for me, I let you drive. If I ask you for something, the first thing I'm going to ask is when you can have it done and what you need to do it. You tell me that, and I consider you committed.

Let's say I ask you to implement an automated installer for the QA environment. You tell me that it'll take a week and you need three test machines. Once I get you those three test machines, it's your job to make it happen in a week. If you can do it in 10 hours, great! If it takes you 80 hours, well, I guess you're working an 80 hour week.

In other words, come work for me and I'll never make you do anything you don't sign up for. Once you sign up for it, you get it done, no excuses.

Wednesday, September 5, 2007

Using Your Poor Man's Performance Lab

Yesterday I talked about setting up a performance analysis tier on the cheap. Now let's talk about how you can use it. For this example, let's assume a basic web application. You have three tiers: a web tier serving content, an app tier working on your business logic, and a database.

Step 1: Defining your test
First, define your "performance problem" (or potential performance problem). Are you testing load? Are your load times awful when there's a lot of data? Are your load times horrific even when it's just one user? Are you mostly interested in what will fail first? Based on this, write a simple test (on whiteboard, in code, whatever your preferred method is). It needs to be as simple as possible.

Step 2: Set up your test
You're going to need to be very certain that you're running the same test every time, so set up carefully. Load all the data you want in, then back up your database. Clear your browser caches on the client. Kick extraneous people off your network. Write your load scripts for background loading.

Step 3: Start your tracers
Turn on SQL Profiler and make sure you're logging queries. Turn on WireShark on each tier and start capturing. Launch FireFox and make sure you have the add-ons (FireBug and YSlow) enabled.

Step 4: Fire it up!
This is the part where the more hands helping the better. Start your volume tools (JMeter or MS ACT), if you are doing load tests. By hand, walk through yoru workflow with your reference client. Once you've done this, you can shut everything down.

Step 5: Gather your traces
Stop the SQL Profiler trace. Stop the WireShark traces and save them. Export your FireBug results. Make a single folder in a central spot and save everything from your test run.

Sure, it's not packaged up nicely for you, but you now have a record of exactly what your systems were doing. Analysis time!

Tuesday, September 4, 2007

Poor Man's Performance Lab

Every software project I've ever worked on has had "a performance problem". Eventually, you will have to do a performance profile of your application. So now what?

Sometimes you get to buy the big tools: OPTNET, Mercury Load Runner, etc. Sometimes you have a whopping (sarcasm alert!) $500 budget and a couple of machines you can repurpose. Here's my basic setup for performance profiling a web application on the cheap:

1. SQL Profiler (or the equivalent for your database).

2. WireShark installed on each tier

3. FireBug and YSlow installed on the client

4. JMeter (or MS ACT if you're a C# environment) on the client for generating load.

Total cost of software: $0

Your hardware needs will vary, but you will need at least enough to replicate the tiers you have in production. If production is a one-box setup, then you need one box in QA. If production has separate web tiers, app tiers, and database tiers, then you're looking at 3 or more boxes. Hopefully you have this already from your functional testing.

In another post we'll talk about how to use this setup effectively.