Wednesday, March 17, 2010

Apples, Oranges, and Strawberries

Many times we'll run tests in an attempt to compare two things. Maybe we're comparing our performance against that of a "comparable" (read: competitor's) product. Maybe we're comparing new software on cheaper hardware to see if we've lost speed in a "comparable" (read: offers our customers about the same thing) configuration. These comparisons are completely legitimate, but we need to be a bit careful with them.

Maybe we do want to compare apples to oranges. After all, they're both fruits. They're both round. They both come from trees. They both have seeds on the inside.

Apples are like oranges, if the thing we're trying to compare is seed location.

Maybe, though, we want to compare apples to strawberries. After all, apples and strawberries are both red. Perhaps we're most interested in making sure that whatever we compare is red. Or perhaps it's that they both have stems that come with the fruit. Maybe it's fruits with edible skin.





In order to make a valid comparison, you have to identify the characteristics that you really care to compare. If there are things that must stay the same (performance, say, or cost to the customer), then make sure your comparison respects those things.

2 comments:

  1. Hi Catherine,

    I really prefer using One-to-Many (or Many-to-Many) relationship in comparison. That makes it more dynamic and versioned, just as our applications are.

    So, borrowing your fruits-under-test example:

    color(red_apple) in set {green, red, yellow} -> true

    color(red_strawberry) in set {green, yellow} -> false

    Note, that true or false doesn't really tell whether it's valid or not. It depends on the context.

    Thank you,
    Albert Gareev
    http://automation-beyond.com/

    ReplyDelete
  2. Albert: I can see that. Your "characteristics that must stay the same" become the set, and then you figure out if your proposed comparator is true or not. And you're right: meets criteria and valid might be two different things!

    ReplyDelete