There's a classic film technique called "visual indirection" in which the viewer sees not the action itself, but the reaction of other things to that action. Sounds a lot like debugging a customer problem, doesn't it?
You can't actually see the thing itself. You can only see the effects it causes on other things.
So what do we do?
The first and most direct effect is to affect your logs - have the program write out what it thinks it's doing. Sometimes you'll want this only at debug levels, or other nuances, but this can be a help.
The second effect is on downstream components: because A did something odd, then B, which follows, may also behave abnormally. This is where knowing your system flow is crucial - if you can catch B's odd behavior then you can trace it back to A. This kind of effect lends itself to the Five Whys debugging method.
The third effect is on the component itself: because A did something odd, the next time A occurs it will do something odd (either the same odd thing or something else). To help see these, find a pattern, then find a violation of the pattern. Once you have your violation, look not only at the violation but at the last previous instance of that pattern.
The fourth effect is on concomitant components: because A did something odd, unrelated component C did something unexpected. Race conditions and deadlocks are classic examples of this. These are particularly difficult to pin down, and require you to model your system through time (a decidedly nontrivial task in an uncontrolled environment).
So when you're debugging, don't forget that you aren't looking at the problem, you're looking at the effects of the problem. Look around and you'll find your smoking gun.