Our customers want a lot of reasonable things:
- information about the product's current state
- information about the severity and likelihood of an issues they have/might/hope they won't find
- test coverage information
- risk assessment of our products
Our customers also sometimes want a lot of things that are impossible:
- Zero defect releases
- The ability to "make up for engineering" by testing in some quality (darn it!)
And then there are the things that our customers want that are something of a grey area:
- a go/no go release vote
- something to measure how good their testers really are; a "standard" or "guideline" or "best practice" or "certification" or "metric" (the words are different; the desire is the same)
Telling your customer that they can have something they want is easy. The other two categories are much more difficult.
Telling your customer they can't have the impossible is not a fun conversation but, assuming you're dealing with a rational human being, generally goes fairly well. For example, if a customer asks for a zero defect release, you can explain the difference between known defects and potential defects, and describe your find rate and fix rate currently. Do some preparatory math: if your find rate is decreasing at 10% per week, and you're finding 10 bugs per week currently, then in 10 more weeks you will have a 0 find rate, probably. Do the same with the fix rate, add in the regression rate if necessary, and you will have a target date for zero known defects. Then you simply let your customer decide if they really want to ship with zero known defects, or if their release date is earlier than that can happen. Other examples are similar. The impossible requests are generally counter-able with logic and a few numbers.
The really hard part is when your customer is asking for something that's not out and out wrong, but that is a matter of opinion. A good example here is whether "QA votes in the release decision". There are a number of things going on with these grey areas:
- There's no tester consensus on this one. Some schools of thought say that test is an information providing function only and should not make a decision about the location of the software in the development life cycle (moving through any phase, including release). Other schools of thought embrace the notion of testers as a hub of information and therefore put the release decision squarely in the test corner.
- This isn't about something industry-wide. For every "testers are information providers" argument there's a "testers have a place at the release table" argument, and they're both valid. These are arguments about role and function within an organization, which means they're specific to that organization. What they do at "other company X" is only marginally relevant.
So what do we do? We're faced with kind of a difficult choice here.
Ultimately, what you do is up to how far you're willing to go, and will be dependent on your relationships with your customers, your own comfort level, and even where your future ambitions lie and how much you're willing to step outside your "tester" label.
You can choose to follow what your customer wants. Take your place at the release table and treat it like any other test. Gather information from all available sources, consult whatever oracle you have, and a make a decision. (In this case, the decision is whether to release, not whether to log the bug, but the underlying process is similar.) Stick as many caveats around it as you want, but ultimately keep pleasing your customer.
You can choose to dig your heels in and refuse to provide anything resembling a decision. Eventually they'll stop asking. Note that normally this costs you politically, but that may be okay with you.
You can try to talk your customer out of it. Explain your views; perhaps your customer will agree. If not, you've at least provided contextual information about how this might not be a good choice that your customer's making. (If you're in the test is information providing mindset, well, you've just providing information about their decision - we're starting to get a bit meta here!). The goal here is to be transparent and to think rationally about this decision. Describe why this might seem like a good idea and why you think it's a bad idea. But ultimately, your customer is making the decision here.
What you do about your customer's requests is up to you. If they're asking for something reasonable or something impossible, that's fairly straightforward. But when you're in a situation where they're asking for something grey, think long and hard before you answer. In the end, you're going to win some of the grey areas, and lose some of the grey areas. Your job is to make the choice transparent. In the end, this is a relationship, and transparency of decision is more important for the long-term relationship than any single decision.