Monday, January 9, 2012

Positive Negative Assertions

I've been working with someone learning how to test, and we ran across our first negative test case. The method we were testing looked something like this:

new_car_color = paint_car(color)

You feed it the color you want your car, and it returns the color of the new car. If you feed it nothing, it returns the current color.

We'd been writing positive tests that looked something like this:

def test_paint_new_color
   color_i_want = "red"
   color_i_got = paint_car("red")
   color_i_want.should == color_i_got
end

def test_dont_paint
  color_i_want = "blue"
  color_i_got = paint_car("blue")
  color_i_want.should == color_i_got
  color_i_got = paint_car()
  color_i_want.should == color_i_got
end

So when we got to the first negative test, we talked about it and came up with this: "We want to test what happens when we try to paint a car a color that doesn't exist".

The junior tester then wrote up the following test:

def test_paint_no_color
    color_i_want = "not a color"
    color_i_got = paint_car("not a color")
    color_i_want.should != color_i_got
end

Well, it's a start. But our car could be any color as long as it's a color. Is that really what we want? What if we started red and the method painted the car blue whenever it got a color it didn't understand? We wouldn't have a clue!

Instead, we write the method like this:


def test_paint_no_color
  current_color = paint_car()
  color_i_got = paint_car("not a color")
  color_i_got.should == current_color
end

We make a positive assertion and confirm that the color of the car didn't change.


Even when we're doing negative tests, our assertions should be as positive as possible. We should assert that something is rather than asserting that something is not. After all there is a large range of things that fits inside "is not" - and we might not like all of them! There is a much smaller range of things that fits inside "is". So wherever possible, we should do positive assertions, even when we're testing a negative case.

2 comments:

  1. This is excellent post - when we think about response time objectives in performance testing, it's hard to state a negative case. Let's consider a login for concurrent user quantities:

    def test_response_time_login
    one_user_time = login("users=1","concurrent=true")
    lots_users_time = login("users=1000","concurrent=true")
    one_user_time == lots_users_time
    end

    In performance, it's not just a negative test case. The difference between 1 user response time and 1000 user response time is more interesting as a result than the absolute product of the comparison.

    Again - thanks for a great, visual post!

    ReplyDelete
  2. Mark, that's a good point. Knowing something - rather than knowing not something - gives you a much stronger base from which to draw conclusions/learn more/see problems/etc.

    ReplyDelete