Tuesday, September 30, 2008

Stylish

I wrote yesterday about evaluating test code, and wanted to follow up about the use of style. Much of what I discussed was a case of right or wrong, will or won't scale, can or can't be maintained; it's fairly black or white. Things get grey very quickly, however, when we start talking about style guides. After all, who cares if you put a space between your arguments or not?

The short answer is that I care, and you should, too. After all, consistency matters.

Why do we care about consistency and style?
There are a number of reasons to care about consistency.
  • Consistent code is easier to read. Your eye starts to look for standard clues in formatting (quote blocks, loop layout, etc.), and when it's there you comprehend more quickly.
  • Crossing editors is easier. Not everyone is going to use the same tools, or even the same OS. Having style guidelines helps things work on everyone's environment, not just yours.
  • Bugs are in the inconsistencies. So you didn't lay your loop out with the braces in the normal location, so what? But then when someone changes your loop, they'll make the code consistent - and now you've got an extra brace and a bug. Sure, it's an easy bug to catch (thanks, compiler!), but it didn't have to be a bug in the first place. The same thing is true on more subtle levels (think array indices, for example).
What are the elements of a style guide?
The specific elements of your style guide will vary based on language, environment, age of code base and the personal preferences of the person creating/enforcing the code style. However, areas that are covered usually include:
  • Comment formatting. This applies to both contents and layout of comments. It is particularly important when you're using a doc generator.
  • Whitespace, indentation, and braces. These help in particular in mixed environments, where "tab" may not mean the same number of spaces everywhere, for example. Brace usage can, depending on language, allow for increased maintainability when moving from one-line clauses to multi-line clauses. Most of this formatting is about making code really fast to read, so you can simply assume the formatting and really see the code flow easily.
  • Naming. Method names, variable names, class names, constant names will all have a style so that it's easier to distinguish them.
  • Scoping. Making scope as narrow as possible is generally the guideline here.
  • Exception throwing and handling. How many exception types are there? Should you swallow the exception, throw it higher, leave it?
  • Overloading. When overloading is good versus when it's discouraged varies greatly.
  • File and method ordering within classes. Make things easier to find without using your IDE's "open declaration" feature (if your IDE has that).
Sometimes, even with a style guide, you'll find yourself violating the style. You may change styles and not immediately refactor all your code. That's okay - total consistency is not the goal. The goal of coding styles and style guides is to help you better read/write/maintain code as a whole team - current and future. Don't let your style run away with delivering value, but don't let delivering value today be so poorly styled that you can't keep and increase that value tomorrow.

And remember, your test code is code, too. Make it stylish!

Monday, September 29, 2008

Evaluating Test Code

One of the projects I'm working on is "auditing and evaluating" a set of existing test code. Basically, they want me to walk through the code and put together a list of items that are done wrong, areas that were missed, good practices to continue, etc. In short, it's a perfectly normal project.

So how do we approach a test code evaluation?

There are a few kinds of things to look for here:
  • Things that are flat out wrong. These just don't work. Examples include extensive use of timeouts, 
  • Things that are unmaintainable, etc. These are things that work now, but won't scale as the project continues. Unscalable infrastructure code is a huge culprit here. Unmaintainable code that's written very specifically is also a problem here.
  • Poor coverage. No matter how well-written your tests, if you're not testing enough, you're not done.
  • Stylistic things. Consistency of code layout, comment formats, etc. are big here. This is where things get a little more sensitive, since they're not likely to be wrong so much as picking one.
Obviously, there's a lot of depth to each of these. Leaving it at this level for the moment, though, there are land mines to watch for:
  • Make sure to distinguish between wrong and non-ideal. Both are good, 
  • Don't forget to consider changing standards. For example, in Rails two years ago it was all fixtures, then we went to foxy fixtures, now many people are avoiding fixtures, for example. If you're not current, don't take this kind of project.
  • Style matters. It's easy to say, "eh, whatever" about naming, spacing, etc., but it's a huge factor for maintaining your code over time.
These kinds of projects are really useful for checking your thinking about test code and test structure. However, they can also turn very political very quickly, since people have put a lot of effort into this code. So take them and document how to make things better, but do it politely.

Friday, September 26, 2008

Coincidences

We have a rather large (say 8500 or so) suite of tests that runs every night (and until about noon the next day). These tests are controlled by a single machine.

Yesterday we were having problems with that nightly test controller machine, specifically with the OS on it. So we backed up everything and reinstalled the machine. However, this wasn't going to be done in time for last night's run, so we grabbed another machine and set it up to run nightly. Being slightly paranoid, I logged in last night after the nightly run started, saw the processes chugging along, saw the log file filling with standard information, said, "great!", and went to bed.

This morning, we got in to discover that the nightly tests had not run.

Oh....

dear....

After digging in some more, we found that the nightly test suite hadn't run because the clean build it does (before it does any tests) had failed. Our machine was configured just fine; it was just a coincidence that we happened to switch test controller machines on a day when the build was going to fail.

The lesson of the day?

Coincidences really do occur.



P.S. I've left out a number of steps in our continuous integration/build/test process because they weren't relevant to this particular story.

Thursday, September 25, 2008

Clusters

It's a little weird, but sometimes I spend time going through old resolved bugs looking for patterns.* It doesn't really approach the level of structured analysis; I'm just looking to see what jumps out.

I'll pick sets of bugs all opened within a time period (a week, or a month) and just look at what their resolutions happened to be. I'll start with the obvious - x% were fixed, y% were marked unreproducible, etc. Then I look at specific resolutions - the comments in the tickets themselves.

So what am I looking for?
  • ssh failures (timeouts, failures to connect, etc)
  • mount/umount failures and other OS issues
  • tests that were fixed by extending their timeout periods
  • fixes in test assumptions or infrastructure rather than in code
  • bugs in certain layers of the code across different tests; these are sometimes not obvious when looking at the functional areas that the tests assert on directly
  • numbers of bugs marked "unreproducible"

No one ticket with this resolution comment would raise a red flag, but if you get a cluster of them, well, maybe there's something going on there. The increased timeouts may mean that the product is getting slower. SSH failures can be a sign of an overloaded network. Bugs in specific layers of the code may indicate an underlying weakness that's being masked by the layers above (or below) it. Not always, but sometimes I find that this is where the deep bugs are.

It feels kind of silly sometimes to simply scroll through resolved bugs, but that second level analysis shows patterns that a first level analysis simply missed. So I keep scrolling....



* Am I the only one who does this?

Wednesday, September 24, 2008

Lots of Data

I got some test results from a test run by a vendor. It was presented with the following statement: "There's a lot of data here."

Well, yes. The logs are quite large (several hundred MB for a 6 hour test). I don't know if they contain any actual information. After all, not all data is useful.

Data: A collection of facts and states.

Information: Data that's actually useful to the person charged with doing something about it.

Here's hoping there's information in those logs, and not just data!

Tuesday, September 23, 2008

Idea to Existence

It's easy to get caught up in process, even when process adds more overhead than you could possibly want. At the same time, it's also easy to go around process because it's cumbersome and then watch the system blow up in your face because it's become a mishmash of poorly understood ad hoc changes.

For example, we had this sequence of events occur today:
  1. Someone in dev sends an email to the testing team that says, "Hey! You're logging bugs, but we can't really diagnose them because debugging isn't turned up. Please remember to set the debug level higher when you're testing."
  2. I send an email back that says, "Whoops! We can do that. But I was thinking, if we can't debug it here, how are we going to debug it in the field unless we also turn debug up there?"
  3. A dev sends email back that says, "Good thought. Let's turn the debug level up by default. It only adds about 5-10% additional size to the logs."
  4. A totally different dev chimes in with an email that says, "Well, what if the logs roll? Maybe we should log at debug to one log and log at info to another log so we get duration and detail both."
  5. I send an email that says, "Works for me. We already have a script that interlaces logs from the various systems, so we can just put it together back here if we need to."
  6. A couple of devs pair on the change set and check it in to head.
(I should note that most of this is emails because some of us are in meetings... and maybe not paying full attention! That's a blog post for another day.)

Now, the first thing to say is bad engineers. We didn't follow the process, and this was an enhancement. We should have written a story, let the customer prioritize it, estimated it, and checked it in.

The second thing to say is that we had a cool idea that doesn't change the customer experience (except possibly to make debugging customer issues faster and easier), and we implemented it quickly. This kind of responsiveness and turnaround is a sign of a functional organization.

So what's the right thing here? How do we balance between following a process and getting things done?

There are a series of questions I ask myself when I'm in these scenarios where process maybe isn't happening:
  • Am I following the process?
  • Am I following the intent of the process?
  • Why am I going around the process?
If the answer to the first question is that I'm following the process, well, there's nothing more to be said. Hooray for the change!

If the answer to the first question is that I'm not following the process, I have to ask myself whether I'm following the intent of the process. That is, we've (hopefully) designed a process that is intended to get good software into the field as quickly as possible. If I have to violate the process to follow the intent of the process, well, something's not right. Either the process is broken - usually too much overhead - or my understanding is way off.

If the answer to the first two questions is no, then the answer to the third question becomes important. Sometimes we violate process because of a true customer emergency. Patches often fall into this category, and that's valid. (Our process should accommodate those scenarios, but that's also a separate blog post.) Other times we violate process because we just don't want to follow it; we don't want to go through the bother of making the story, etc. If that's the case, well, shame on us. We need to make the process less burdensome or revise our expectations.

How do you handle the transition from idea to existence when process maybe isn't your first instinct?

Monday, September 22, 2008

Assigning Blame

Blame is a funny thing.

It's pretty common, particularly in startups, to hear statements made that try to avoid blame entirely:

"Fault doesn't matter; we're not here to blame each other. We just all work together to get it done. After all, that's what's important."

As soon as you hit a really hairy situation, though, or the company grows, blame starts happening under the surface. Oh, sure, "blaming others is a waste of time", but it happens anyway.

So that leaves us with two questions:
  1. How do I know when blame starts to matter?
  2. What do I do about it?
Before we address whether we have blame and what to do about it, let's step back and talk about some of our assumptions going into it:
  • Blame is something we can avoid. We're all friends here. Actually, blame is a pretty inherent trait. Even Koko the gorilla blamed things on others (scroll down), as do children just learning to communicate. It's not something we really outgrow.
  • Blame is inherently bad. If you think about it, root cause analysis is really assigning blame within a software system. Blame there is great - "hey, we found it!". The same thing applies to humans. Blame is good if it provides insight into why a problem happened.
  • Blame helps it not happen again. Yes and no. If blame is found rather than assigned, then it can help figure out what went wrong (and preventive measures can then be implemented). Self-defensive blame isn't productive, though, and can hide the underlying problem.
So we've established that blame is going to happen, and that it can be productive but isn't necessarily productive. So the real question to ask ourselves is not whether there is blame (because most of the time, there is!) and what we can do to avoid it. The real question is whether that blame is productive. Let's swing back around to our questions from earlier.

How do I know when blame starts to matter?
Blame happens. Sometimes it's a very personal background thing (e.g., "Shoot! I logged a duplicate bug! How did I miss that one?"), and sometimes it can extend much further.  It starts to matter when it:
  • starts taking up time
  • is unresolved or unsatisfactorily resolved
  • causes working relationships to break down
Blame doesn't always have to be addressed. Only when it starts to matter does it need to be handled explicitly.  Blame isn't always obvious. Rarely in an average organization do you find an employee stalking the halls declaiming loudly that, "Shelia did it!" Instead, you have to look for the signs that blame is simmering underneath the surface:
  • meetings start coming out with much more detailed meeting notes and action items that so-and-so committed to such-and-such
  • you find people being talked about and hear second-hand
  • your boss starts asking you what the real story behind XYZ is
  • your subordinates start asking you what the real story behind XYZ is
  • previously informal conversations turn into emails or meetings
The goal of handling blame should always be to preserve working relationships and resolve the issue. Think of it as root cause analysis for the corporate system (rather than your software system).

What do I do about it?
Once we've identified that blame is going on and is starting to consume a company's time, it's time to do something about it. Specifically what you do will vary greatly depending on your position in the organization - the higher up you are the more rarely you should get involved, but the more effective you can be in finalizing a situation. That being said, there are always some things you can do:
  • Get the whole story. You may or may not know what incident(s) occurred, but get the whole story from several perspectives quietly before you attempt to do anything.
  • Study your own behavior. You or your team may have the blame here, or at least share it.
  • Figure out if it's productive or spinning. If blame assignment is really problem cause analysis, the best thing you can do is nothing. Let it work itself out. If it's going nowhere, that's when you have to step in.
  • Solve the problem, not the person. In all likelihood, the people involved have a lot of pride in their work, and don't like to screw up, much less be pointed out as being at fault. Make sure that any blame assignment centers on the problem, not the person or group. Talking about blame in terms of process rather than in terms of individual names can help.
  • Assign multi-fault. Just like many software issues have proximate and underlying causes, so do corporate incidents. Sure, maybe the proximate issue is that support restarted something they shouldn't have. But the underlying cause may be an issue with an outdated procedure, or a change that dev should have made to make that kind of restart safer or even unnecessary. When groups or teams share the blame, then they don't get out of balance - it's easier to move on when everyone took a little bit of the blow.
  • Change the tone. The tone of the conversation should not be "to assign blame". Rather, the tone of the conversation is "to find the root cause and contributing causes of the issue". 
  • Close it out. Once the problem is identified and addressed, don't mull over it. Part of your job is to simply not bring it up again; the issue is in the past and dwelling will only create resentment. Keep it in the back of your mind, in case of recurrence, but never tolerate the situation being flung in someone's face in a future situation. The words, "Well, you messed it up last time" should be reprimanded quickly and firmly.

Friday, September 19, 2008

Balancing Today and Tomorrow

There are a lot of things I work on that are for today:
  • analyze the results of last night's automated run
  • write acceptance criteria for this story
  • work on a beta test plan for a client
There are also things that I work on that don't help for today but will help me tomorrow:
  • interview job candidates
  • set up backup test machines
  • work on stories to get them ready for assignment to dev
It's really easy to work on the first list. These are things that are asking for your attention now. The trouble is, if you keep putting of the tomorrow list, nothing will ever get better. You'll still be running on the same treadmill, with the same problems.

So how do we balance these things? How do we do enough of tomorrow that things get better, while still getting today's stuff done?

Some days the today things will take over. On those days, all you can do is make one thing just a little bit better. But most days, most days I think we should be striving to work on tomorrow as well as today. There are a number of ways to do this:
  • devote half a day a week to "future work" (or whatever)
  • devote an hour a day to "future work" (or whatever time period)
  • get your future work done, then your today work. I don't think this is actually feasible.
  • get your today work done, then your future work. This one is a nice dream but rarely actually happens for me.
  • force your future work to be today work.
The last one has worked best for me. Basically, I force all of my future work to start being as important as today work by turning it into today work. For example, I make a public commitment to respond to all resume submissions within 48 hours. Now I get a resume, and it's "has to happen today" instead of "I should really look at resumes sometime". Combine that with my empty inbox desires, and it works out pretty well.

How do you handle the balance?

Wednesday, September 17, 2008

Status Holder

One of the things I wish for in our defect tracking system is a basic status field for my use only. Here's my use case:

We have a weekly escalation meeting in which we go through all the bugs that have come in from customers. In preparation for that meeting I basically go through each bug and note for myself the status of the issue. It might be "waiting to deploy patch at customer site" or "dev is working on this one", or something else.

The information isn't really useful to anyone else. It just helps me remember what's going on without having to load and scroll through every bug during the meeting.

Boy I wish our defect tracking system did this. I've never heard of one that does.

Tuesday, September 16, 2008

Want To Work With You

Working with people is pretty much inevitable. For some of us that's perfectly fine, for others it's a bit less natural (hey, some people like their focus time!). Eventually, though, you will end up in a situation where you're asking for something. Maybe you want an explanation, maybe you want a new feature in a product, maybe you want help debugging, whatever.

When you ask for help, the trick is to make people want to talk to you. You'll get much more - and better, and more timely - help when you set up the situation so people want to spend time with you. There are certain concrete things you can do to "grease the wheels" so to speak, and make sure that you're creating a relationship where help is freely offered.

Respect their schedule.
Remember, you're the one who needs something. The other person (or group) involved is granting you something, so you need to work with them. Come in early, leave late, etc. The goal is to inconvenience them as little as possible.

Don't tell them how to do their jobs.
A really fast way to make someone defensive is to ask a favor and then proceed to lecture them about how exactly they should go about doing it. Just don't do this. Don't tell a system architect how to design a solution that meets your perceived needs - you're leaving her guessing what on earth you're trying to accomplish. Don't tell a project manager which person to allocate to fulfilling your request - you're not letting them pick the best (and most available) resource. Your goal is to describe the change you wish to see made, not how that change should be made. Telling them how to fulfill your request limits their options and you will likely not get an ideal solution.

If I had to pick one thing, this would be the top thing that really can make people avoid you instead of want to help you.

Acknowledge that it may be your problem.
Noting up front that some request may be a result of your lack of understanding, or a mistake you may have made humbles you a bit. It also prevents your request from sounding accusatory. By acknowledging that you may have caused the issue, you change the tone of the conversation from "you screwed up" to "hey, here's this wacky thing. Help!" Again, it reduces defensiveness. Plus, when it really is your problem (as happens to all of us sometimes), there's a graceful way out for everyone.

Don't ask the same question twice.
Asking for help once is perfectly wonderful. Asking for the same thing a second time generally means you didn't listen the first time. And why on earth would someone want to help a guy who didn't listen the first time around? Note that this doesn't cause a problem if the repetitiveness of the question isn't apparent to either of you when you start.

Have something to offer.
Relationships are about reciprocity. Since you're asking for help, you need to offer something. This doesn't have to be tit-for-tat at all. Just be sure that you're willing to help when someone needs you. And it never hurts to bring in brownies, or mention to your helper's boss that you really appreciated the hand!

There are two halves to any interaction between two people. Your task is to understand your role and to make sure that your half of the interaction is done effectively and with the least possible negative impact on the person in the other side. Build your reputation, don't be the cause of your own problems, be precise, and temper your speech, and you will find help 

Monday, September 15, 2008

Knowing Makes It Okay

Working with beta clients is a tricky business.

On the one hand, I don't like betas:
  • The software isn't ready for clients yet; that's why it's not released!
  • Beta clients have expectations that their feedback will be incorporated in that release - which may or may not be feasible or necessary depending on the feedback.
  • It's really hard to have a beta client actually follow the beta test plan (even if they wrote it!), so you can't be sure how much coverage you really got.
  • Poorly run beta clients are basically like asking support to look at an early release - scattershot and unfocused, and therefore of diminished value.
On the other hand, I love betas:
  • Client environments give us improved test coverage.
  • Clients who want early access tend to be very willing to work with support/QA/etc throughout the process.
  • Better to find out if something's really wrong before we ship, rather than after! At least it limits the damages to only beta customers.
A lot of the value of the beta comes in how it's run. First and foremost, this is not totally in your control. You need buy-in from your client, your account rep (or whoever is performing in that function), and also QA, development, product management, and the release team.

The second most important thing you can do to make a beta go well is to know the system. Having bugs is fine, even expected. If you tell the client about those bugs, it is far better than if the client tells you about those bugs. Imagine these two conversations:

Conversation 1: Knowledge is Power
You: "Here's your software, and here's the list of known issues. Let us know what you see!"
Client: "Thanks."
Client: "Gosh, I saw some of these, and that bug 14 you mentioned is really a doozy. It's getting fixed, right?"
You: "Yup. We're working on it and it'll be fixed before we ship. We thought it was bad, too."

Conversation 2: Going in Blind
You: "Here's your software, and here are the issues we've found so far. Let us know what you see!"
Client: "Thanks."
Client: "Here's MY list of bugs. You guys didn't find any of these! Look at my #14 - it's ridiculous! How could you not have found it? What are ya'll doing over there?"
You: "We'll take a look."

At this point, your client has lost confidence not in the software itself, but in your company's ability to produce good software. You've shown you can't find major problems, and now the client can easily go down the path of wondering what else you might have missed. Loss of confidence ensues.

It's hard, going into a beta, to mention all the problems you've found, but it puts the client on your side. The client will find problems - probably including some you didn't find - and that's okay. As long as the client knows about what you're finding, and you're in this together, the bugs won't reflect badly on you or on your development process. You don't have to be perfect, just forthright.


Should you choose to do a beta, the first thing I would say is that there will be problems, but knowing makes it okay. Once both you and your client understand that, you can do the beta and make your software stronger.

Friday, September 12, 2008

What I Did Today

I've been thinking lately about tactics versus strategy, about getting through the day and the pile of "gotta do it now" versus the long-term changes we'd really like to make. On a separate note, I have been thinking about treating QA projects like development projects, and chunking them up intelligently. On top of all that, there's estimates, and how much time we should spend estimating versus just saying, "this is the most important thing we have to do now, so let's go do it.".

All of these things are about balance. They're about finding the mean that gets the most people happiest the quickest. In the end, I think it boils down to one small philosophy:

Every day, leave working having made one thing better.

I'm a fan of small philosophies. They're attainable, they're simple. You don't have to spend a lot of time selling people on your grandiose vision. It's just your contribution to your world - one small day at a time.

So every day, I go into the office. I go to meetings. I deal with bugs and fires. I test things.

And in that quiet time at the end of the day, I ask myself what one thing I made better today.

In the end, that adds up. And I can do it every single day.

So change your world with me, one small day at a time.

Thursday, September 11, 2008

Don't Say That!

There are some phrases I catch myself using, or that I catch others using, that are really sort of nasty turns of phrase. They're meant to mean one thing, but the actual thing the recipient hears is something totally different.

"Honestly, ...."
My first reaction to this is, "oh, and all the stuff before was dishonest?". Let's not go down that path!

"Let me be straight with you."
Again, an instance of "ahh, and we weren't being straight before".

"Yes." or "No."
This is ambiguous when answering a question. It could mean, "I agree", or it could mean, "You're incorrect. I answered in the affirmative." Clarify. Plus, one word responses tend to sound abrupt. Repeating the substance of the question helps connect you with your listener.

"We want to..."
Also ambiguous. You want to do something but you can't or won't? You want to and you're going to?

None of these are hard and fast rules, and I'm guilty of some of them, too. But over time, it really helps your overall work relationships to consider what your listener hears, not just what you say.

Wednesday, September 10, 2008

Chunking QA Projects

Bug fixes are easy. They're very focused. I work until the bug no longer exists.

New implementations (of QA code, anyway) are harder. There are a lot of stopping spots that are conceivable. How do you know when to say that you're done with this project? It's inefficient to say that we're going to do the whole thing before we check it in. It's also silly to check in something that flat out doesn't work. We have to chunk our projects just like software chunks functionality into releases.

So, what are our rules of QA project chunking?
  • getting something in is more important than getting all the features
  • continuing to add is perfectly acceptable
  • meet your current goal before you start your future goals
  • incomplete is better than not at all
Long story short, writing QA software is like writing any other software. Start with a small feature set, and keep adding. Finish your first feature set, then refactor and add new features. And whatever you do, don't let perfect be the enemy of good.

Tuesday, September 9, 2008

How Far Don't We Know

It's common in our line of work to not know things. Don't know test results, don't know how many bugs we'll find, don't know root cause of a problem we're working on, etc.

There are, however, multiple kinds of not knowing. The three major types are:
  • I don't know, but I have a path to the answer.
  • I don't know, but I know how to start looking for the answer.
  • I don't know, and I have no idea how to start.
What divides the types of not knowing is simply how far down the path you can see.

So concentrate not on what you don't know, but on the goal and how far you're can see to get there. Your goal is not to magically find an answer. Your goal is to see farther and farther down the path - until your answer appears.

Monday, September 8, 2008

People Don't Remember "About"

Be very careful with estimates. 

It happens sometimes that a tester will be pressed for an estimate.

"How much improvement will this cause at our customer sites?"
"How many bugs are you going to find in this release?"
"What will be the impact of unanticipated customer requests on this release cycle?"

The proper answer to all of these is "I don't know", but that of course is insufficient. So the temptation is to give an estimate.

You say something like this:

"We're not totally sure since the customer's environment is a factor, but in test we've seen a 20% performance increase, so the customer should be something similar."
"Hmm... last release we found 152 bugs, and there are half as many feature points in this release... so, about 70 or 80."
"We get on average one unanticipated customer request a week, and that takes about 10% of the team's overall velocity. Over a 10 week release cycle, well, that's one man week."

Your audience heard something totally different. They heard:
"20%"
"70 bugs"
"one man week"

In general people here what they want to hear, and in this case they're looking for comfort and certainty. So they're going to take the "hard facts" part of your estimate and that's what's going to stick in their mind. All the caveats and hedges you put in your answer - gone until prompted. To be fair, when prompted, most people will remember the subtleties, but it isn't what's springing to mind.

So how do we deal with this? How do we make sure we're not overpromising unintentionally?
  1. Don't give an estimate unless you're pretty darn sure. Characterize instead. Phrases like "significant improvement" or "fewer bugs than the last release" are accurate (although less precise) and tell people for the most part what they need to know.
  2. Be conservative. Don't put your best test run out there as a best-case scenario. Use your average run, or your run that most closely resembles your customer pattern.
  3. Be clear when you know versus when you're estimating. Sometimes you really will know the answer. In those cases, be very clear. Other times it's an estimate, and there you need to repeat the word "estimate" to make sure people get the distinction.
  4. Repeat. Repetition is an aid to memory. Keep repeating the assumptions and variables, and that will help your audience remember them.
Estimates are not inherently bad. On the contrary, they're a fact of life. Don't be afraid to estimate. Just tailor your message to people's what people here; be crisp, clear and just a bit repetitive, and everyone will get on the same page.



P.S. All numbers and estimates are totally made up.

Friday, September 5, 2008

Calm and Passive

There's a fine line between being calm and being passive.

Calm is good.

Passive is, well, disconcerting.

Strive for calm and engaged.

(Rediscovered sitting in a theatre tonight watching some very.... passive... patrons. I don' t THINK they were asleep, but it's hard to tell for sure.)

Thursday, September 4, 2008

Capacity Planning

Capacity planning is one of those areas that is in the no-man's-land that I call "things that are not my job but if done wrong have the ability to give my team a really miserable day".

Therefore I care deeply about capacity planning and specifically about how well the team that does this actually does it.

What Is Capacity Planning?
Capacity planning is the art of deciding how big/how fast/how reliable/how configured a system is required to meet the customer's needs. Basically, your customer wants to do X; what does the system that lets him do that look like?

Stated Requirements
Certain requirements will be stated explicitly. The user will say - sometimes after some prompting - that he needs to support 150 concurrent users, for example. The basic things almost all applications will need to consider are:
  • total number of users (peak usage)
  • average number of users (concurrent usage)
  • amount of data read, written, modified, deleted
  • hardware restrictions (if any)
  • windows of usage (e.g., what is time restricted?)
Derived Requirements
Once you have your list of stated requirements, there are certain requirements you can derive based on the inner workings of your system. These are things the customer doesn't know or doesn't state explicitly, but that are required to keep your system happy. For example, maybe your system works best when the disks on the server are no more than 70% full. Derived requirements are much more specific to your application, but tend to include things like:
  • data access patterns that will influence cache usage
  • available disk space and usage
  • power and cooling requirements
  • network configuration requirements
Calculations
Once you have your requirements capacity planning is a case of simple calculation. The only real trick here is to be conservative in your rounding. If you have a throughput requirement, round up. If you have a system speed, round down. What you're specifically calculating is dependent on your specific application.

For a web app, for example, we might say that a single web server can support 50 simultaneous users. If we expect to have 100 of 'em, well, we need 2 web servers. If we can support 100 simultaneous requests on our app server, and we expect to peak at 40 simultaneous page loads, each of which requires 10 requests, then we're going to need 4 app servers. And so on and so forth. The calculation isn't the hard part here. Once we've done all our calculations, apply the rule of conservatism and pick the largest overall environment. That's what we need.

In the end that's all capacity planning is. Consider your system variables, find out your load, and start calculating how much of everything you need. There's an art to asking the right questions that only comes with practice, but better to ask than to get caught by surprise after implementation!

Wednesday, September 3, 2008

Mail Ick

I've been using Mac Mail on a MacBook Pro, and it does the weirdest thing (aka I think this is a bug). Has anyone else seen this?

Setup:
First let me note that I'm on Leopard. Plugged into the network (wired), Mail is running and works fine. Sending and receiving mail is great.

Steps:
  1. Unplug from the network (say, go to a meeting)
  2. While unplugged, write a new message and hit send
Effects:
  • Message shows up in the On My Mac folder repeatedly (about once every 45 seconds)
  • The sending folder shows a spinner for 10 minutes or so
  • An error appears with the message: "The connection to the server "mail.example.com" on port 993 timed out."
This repeats indefinitely, and no other mail can be sent or received, even after the user plugs in. I keep getting that error message once every 2-3 minutes. The only way I can get it to stop is to blow away my Mail preferences file and reconfigure the account.

How frustrating.

Tuesday, September 2, 2008

Get a Toehold

There's a new QA engineer at work who just joined this week. He's a pretty typical hire for me - lots of talent, lots of potential, in just a bit over his head. He came to me Friday with a task he'd been given, and he just didn't know how to approach it. The problem was that he was starting at the beginning, didn't know how to do that part, and so was stuck.

The trick with a problem you can't start on is that you don't have to start at the beginning. Start anywhere you can get a toehold. From there you can work your way back to the beginning and forward to the end.

I don't care if you can't solve the whole problem now. Don't beat yourself up for it; that's a recipe for getting nowhere. Just solve the part of the problem you can do.

So, when you're faced with a small problem, just find yourself one small toehold and remember that any change you make is fine, as long as you make it just a bit better than it was when you started. Launder, rinse, repeat, and eventually you'll have nailed the whole problem.

Monday, September 1, 2008

Whither the Agile Tester?

Jason over at ParlezUML (boy did I probably get that capitalization wrong) writes about the difficulties he's having finding an Agile Developer-Tester.

Basically, he wants someone who can:
  1. write code (a developer)
  2. who understands test
  3. who can automate tests with
  4. who's done this before in a SCRUM/XP team
And he's having a hard time hiring someone.

So now we enter the realm of unsolicited advice.... 

First of all, people like this do exist. I've hired more than one in my career. Granted, I'm in Boston, MA, and that's not London, but I suspect they're out there. So, how do you find and then hire this elusive creature?

  • Understand if you're looking for a developer or a tester. Yes, we want both in one person, but that's not the way industry is currently structured (at least in the US), and ultimately you're going to have to figure out whether you work with a developer recruiter or a QA recruiter.
  • Look into pairing. One of the advantages of XP is the emphasis on pairing, and pairing someone who's stronger on development with someone who's stronger on test will eventually give you two developer-testers.
  • Yes, they are rare. There are a lot of software engineers out there, and probably only several thousand (note: this is a total guess) developer-testers. As a percentage of the total pool, that's pretty small. So be prepared to sift through a lot before you find the right candidate.
  • Phrase your ad in terms of problems. Part of the issue is that people self-identify as developers or as testers. However, many of the problems overlap. So emphasize the problems, not the title.
  • Get rid of the chip on your shoulder. Going in with an attitude that "the industry thinks testers are inferior to developers" reflects that attitude right on you, the hiring manager, regardless of whether you actually share that attitude.  A good developer-tester already knows the state of the industry and the relative rankings of dev, test, design, DBA, etc; there's no reason for you to project it, too. Embrace the future - where these distinctions are both less important and also less indicative of a relative hierarchy.
And good luck.