Simple, right?
I implemented these as cucumber tests, basically reimplementing the NIST tests so that they're a lot easier and faster to run. I use the same message bodies the NIST tests use, and I assert on everything the NIST test asserts on. Done!
Right?
Wrong.
I still run the tests against the NIST server periodically. Not because the cucumber tests I wrote are wrong or bad, but because they're incomplete. There are hidden assertions - things that the NIST tests look for that aren't part of the official test case.
For example, the first time I ran our system against the NIST server, our system threw an exception and failed, because the NIST messages had elements in the SOAP envelope that we didn't have, and that our system couldn't handle.
Wherever possible, use tests that are fast and convenient for you (in our example, the cucumber tests). Also check your work against the official tests as well, though (in our example, the NIST tests). They provide different value, and they're both useful - so don't forget about one or the other.
No comments:
Post a Comment