Blind

For the first time in a long time I'm working on testing a system in which I can't get at the logs. All I can see is what the end user sees - what data went in, and what the results output are.

It's immensely frustrating.

It's also hugely educational.

I hate being blind to what's going on under the covers because it makes me very dependent on the developers who can get at the logs. I'm unable to be as precise as I would like to be in reporting results, simply because I can't distinguish different behaviors. For example, if I get a response from the API that contains no results, then I don't know what happened. It could be any one of the following:

• if I fed it data that shouldn't have gotten results
• I called the API incorrectly
• it wasn't done yet, and I just needed to be a bit more patient
• an error or bug occurred

I have no way to tell whether there's a problem and what kind of problem it might be, which means I'm running to the developer for every test that doesn't return exactly what  I expect. Codependence is not normally my style!

On the other hand....

What I see is what external consumers of the API see. The customers using the product don't get to look at logs, either. So if there's not enough information to figure out what's going on, then customers may have the same problems that I'm having. I've learned a lot about the usability of the API, which it turns out was maybe not so good. And we've been fixing it, making it more transparent what's going on - whether it's bad calls, or data that doesn't end up with results, or a bug in the underlying system.

So even if you're not blind, like I am, spend some time pretending you are. Ignore your logs and your code traces. Being blind is sometimes as illuminating as seeing.