When you get a system into test, it usually has quirks (don't they all?). This is perfectly normal, and so you go about logging bugs. And the bugs get fixed, and you move on. This is how software development is supposed to work.
Sometimes, however, the quirks don't go away. Fixing the quirks just exposes (or creates) more quirks. At this point, you keep logging bugs, and they keep getting fixed. At this point, you need to stop. Just put away the defect tracking system. If the system keeps getting tweaked, and quirks keep showing up, well, you may have a pig of a system on your hands.
And no matter how much you fix and polish and document and work around the quirks, in the end, you've still got....
This is the point you need to go back to development, and maybe even all the way back to the customer. Because now is when QA is calling for a fundamental rethink of the problem. Somewhere along the line, our design, or our UI, or our understanding of the problem just failed. And all the bugfixes in the world can't fix that.
Remember, you're testing not just the actual implementation, but the design and the problem definition that underly that implementation. Don't just keep applying makeup; call the pig a pig and let's get something a little better in there.