Thursday, October 16, 2008

Child Bugs

Here's a dilemma I've been trying to figure out.

Sometimes we have multiple bugs that turn out to have the same root cause. And sometimes we know that they have the same root cause but we haven't yet fixed it. So the bug keeps happening. With a manual test, that's no big deal - we just don't log it again. But what do we do when it's an automated test and it just... keeps.... failing?

The Normal Case
If we're in a normal case, this is an isolated bug that only shows up in this same test. We just keep logging the failure into the same bug. This way, when we go back we can see how often it failed and if there were any other factors that affected the way in which it failed.

The Root Cause Case
It gets a bit more difficult when you have multiple tests failing. So now I've got two (or five or ten) bugs, and they have the same root cause. How do we handle this? We've got options:
  • Close one bug as a duplicate of another
  • Leave the bugs separate and just comment that they have the same root cause
  • Merge or otherwise combine the bugs
  • Create a bug for the root cause and link the different instances in (how you link is up to you)
We use the latter method and call it "creating a child bug". We log a new bug just for the root cause, and then mark all the other bugs as children. Then we keep logging errors into the child bugs.

The upside to this is that it's easy to see the root cause (and all analysis based on that) without cluttering it with test failures. The downside is that you have to go to each of the children to ensure that it really is fixed in all situations.

How do you handle this scenario?
Posted by at 7:43 PM

4 comments:

  1. Hi Catherine,

    We usually do that simply linking the related bugs. I think most bug tracking systems provide a friendly options of linking one bug with another, at least those which I have used for the past several years do that. And yes, it's important to link the related bugs to be able to close all those in time.

    I'm not sure if there is a bug-tracking system which provides the option of marking one bug as a "root" and linked as "child" though. Have you seen such one or do you use such developed locally? Otherwise, one can just put a comment that "the related bug below seems to be the root cause of the problem" I guess.

    By the way, how does marking a bug as "root" help you? Can you give an example?

    Thanks,
    -Konstantin
    ReplyDelete
  2. Hi Cathrine,

    I found your post quite interesting as it made an assumption about the test / bug process that we do not follow in our environment.

    Your post seems to assume that all test failures result in a defect being logged. In our environment there's a gap between executing automated tests and recording defects. This gap is closed by manual analysis of the test results. If a test failure has highlighted a new defect then we log a new defect. Often, of course, a test failure highlights a known defect that has not yet been fixed.

    We use a 'Results Signature' approach to reduce the results analysis effort. E.g. every verification point within a test (pass or fail) has a unique code. A test run will produce a 'Results Signature'. Focusing on the codes from the failed verification points (Fail Signature) we can check for an expected pattern of fails which would tell us the test has failed for a known defect, e.g. the test has behaved as expected. The failure was expected, yet no new failure points have occured in this run. The Results Signature process incorporates signatures for passes as well as fails to allow us to identify if all the expected verification points were executed. If a verification point were skipped for some reason (script error) it may not cause a test fail.

    We use an excel based tool to assist with the test results analysis. The tool highlights where Actual and Expected Results Signatures differ.
    ReplyDelete
  3. Konstantin,

    There is a system called RT (RequestTracker, but we just call it RT) that has the concept of child bugs. It lets you set "Parents" and "Children" in the same way you would set dependencies in other systems (the standard "depends on"). I'd never used another system that did this, but it's actually kind of useful.

    One example of how we use this is when we're trying to trace down problems that don't show up directly in tests. For example, we were recently testing out a new kernel (Linux system, specifically debian), and we noticed that a number of our tests started failing in really weird ways. The test wrote some data, read it back, and compared the two data sets for consistency, and the tests were failing with inconsistent data. We eventually traced it down to a bug in the kernel having to do with NFS writes - so that became our root bug. And then we linked all the different tests that were failing (the child bugs) into this "root" bug so we could see all the different ways it was failing. It helped us isolate that issue from other issues the test was having. Anything that wasn't linked was something that had to be addressed separately; anything that was linked just increased the priority of getting that kernel issue handled.
    ReplyDelete
  4. Glenn,

    That's an interesting point. We do log bugs for virtually every test failure, hang, or error. This is done by humans (at least currently), though. Basically, one person on the team is put on triage, and goes through the automated test results every morning, and logs the results, either into new bugs, existing bugs, or not logging them.

    We have a tendency to log a lot of things that would normally be considered "infrastructure issues" as defects, even if that's not what the test is specifically testing (e.g., if the machine rebooted itself mid-test, we'll log that). This is in large part because we maintain our own infrastructure, and so a bug like that doesn't matter for the specific test but points to some hardware that's going bad, or a bug in the underlying test infrastructure, or another test that's using machines it shouldn't. (I'll stop rambling now, but that's the basic reason.)

    The verification point (or function point, I guess?) signature idea is interesting. I'll have to give it some thought. It seems like it would be really handy for failures in assertions, but I'm not at all sure how it would handle failures not in assertions (like your example of skipped verification points, which we would consider a bug in the test).
    ReplyDelete

Subscribe to: Post Comments (Atom)