Oracles can take many forms:
- requirements specifications
- screen mockups
- other (competing or complimentary) products
- style guides (yours or someone else's)
- similar or related functions in the same application
- a previous or alternate version of the application being tested (think porting projects)
When you start to test your application, one of your first jobs is to find the oracle(s). This will help guide your future tests, bugs identified, and behavioral expectations.
Sometimes finding the oracle is easy; for example, if you're handed a requirements specification, then you've got at least one of your oracles. Other times, you'll have to get creative.
For example, a friend recently approached me with a problem:
"We're processing files, and we discovered that on this file server with 573million files, we detected 19 million ZIP files. Is that reasonable?"
This is a problem in search of an oracle. It's infeasible at that scale to hand-check all of the files. We're not yet sure we can trust our detection software, though; after all, that's what we're testing. So what's our oracle?
Seems hopeless. "Hopeless" is just another word for "needs creativity". We can do this.
Our problem revolves around the percentage of files on a file server that are of a certain type. There are two ways I think to go about this:
1. run one or more other file type detectors and see how much they agree with my results
2. find other file servers (or reports about other file servers) with a typical file type breakdown. Academic papers, storage vendors, and OS vendors are great sources for this kind of information.
What would you use for an oracle in this situation?