Thursday, December 10, 2009

Is That Good Or Bad?

Let's say that I have a program and a set of tests I've written. Go me! Now the time has come to change that program. So I go in there and I muck about, and then I run my tests.

All my tests pass. Is this good or bad?

Good. I was refactoring, and didn't intentionally make functional changes. My tests tell me that I didn't (at least, in the areas I'm testing).

Bad. I was refactoring, and I did break something. My tests just aren't good enough. Oh, and I'm blind to it until I find a problem later in the process (in QA or in production).

Good. I added a new feature and didn't have to touch any existing code. And apparently I really didn't.

Bad. I added a new feature that changed the behavior of an existing feature. And my tests didn't catch it. Looks like my tests aren't quite thorough enough.

Some of my tests fail. Is this good or bad?

Good. I was refactoring, screwed something up, and my (now regression) tests caught it. That's what they're there for.

Bad. I was refactoring, and I didn't mess anything up. My tests are too closely tied to the internals of my implementation.

Good. I added a new behavior that changed the behavior of my existing features. Tests affecting those features should fail.

Bad. I added a new feature that didn't change existing features but did change some of my test's assumptions (reference data in the database, number of fields in a form, etc). Looks like my tests are a bit brittle.

All of my tests fail. Is this good or bad?

Good. I was refactoring, and broke something that's really at the core of my app. My (now regression) tests caught it. That's what they're there for.

Bad. I was refactoring, and I broke something, but that's a whole lot of tests to wade through for a problem that just happens to be in step 1 of every tests (whoops! login!). My tests are too repetitive; I should refactor and introduce dependent tests so I don't just repeat the same failures.

Good. I added a feature that broke pretty much every other feature. My tests caught it.

Bad. I added a feature and my test infrastructure can't handle it. Need some more robustness there.



As with most things, a passing test or a failing test isn't inherently good or bad, and the same thing applies to suites. When tests pass, you don't know your code is right; you only know that it hasn't broken in a way your tests are looking for. When tests fail, you know something is wrong, but it might be your code or your test, and you'll have to dig to find out. Either way, take a red bar or a green bar for what it is - a piece of information that will guide your next efforts.


No comments:

Post a Comment