Use the Official Tests

I've been working on some tests of a system that uses the NIST PIX PDQ Preconnectathon tests. In many ways, these tests have been simple. They all spin up a server (the system under test), receive SOAP messages, and assert on the responses the server sends. The NIST tests have made it very simple to see what to assert on. Each test shows the body of the message that the system sends, and lists the assertions it makes.

Simple, right?

I implemented these as cucumber tests, basically reimplementing the NIST tests so that they're a lot easier and faster to run. I use the same message bodies the NIST tests use, and I assert on everything the NIST test asserts on. Done!

Right?

Wrong.

I still run the tests against the NIST server periodically. Not because the cucumber tests I wrote are wrong or bad, but because they're incomplete. There are hidden assertions - things that the NIST tests look for that aren't part of the official test case.

For example, the first time I ran our system against the NIST server, our system threw an exception and failed, because the NIST messages had elements in the SOAP envelope that we didn't have, and that our system couldn't handle.

Wherever possible, use tests that are fast and convenient for you (in our example, the cucumber tests). Also check your work against the official tests as well, though (in our example, the NIST tests). They provide different value, and they're both useful - so don't forget about one or the other.