[libvirt] heisenbug in command.c
Jim Paris
jim at jtan.com
Fri Mar 16 20:52:50 UTC 2012
Serge Hallyn wrote:
> On 03/16/2012 11:50 AM, Eric Blake wrote:
> >On 03/16/2012 10:36 AM, Serge Hallyn wrote:
> >>Hi,
> >>
> >>It seems I've run into quite the heisenbug, reported at
> >>https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/922628
> >>
> >>It manifests itself as virPidWait returning status=4 for iptables (which
> >>should never exit with status=4).
> >
> >Maybe iptables isn't documented as exiting with $? of 4, but that's what
> >is happening. The libvirt code in question is quite clear that it
> >grabbed an accurate exit status from the child process.
> >
>
> Well, yes. I figured that either (1) iptables actually got -EINTR
> from the kernel and passed that along as its exit code, or (2)
> something went wrong with memory being overwritten in libvirt,
> however unlikely. Stranger things have happened. If (1), I was
> wondering if it was being ignored on purpose.
Why do you bring up EINTR at all? Just because EINTR is 4? That
seems very much unrelated.
This is from iptables:
enum xtables_exittype {
OTHER_PROBLEM = 1,
PARAMETER_PROBLEM,
VERSION_PROBLEM,
RESOURCE_PROBLEM,
XTF_ONLY_ONCE,
XTF_NO_INVERT,
XTF_BAD_VALUE,
XTF_ONE_ACTION,
};
So it looks like iptables is returning RESOURCE_PROBLEM (which could
explain why it's intermittent).
-jim
> > ret = virPidWait(cmd->pid, exitstatus ? exitstatus :&status);
> > if (ret == 0) {
> > cmd->pid = -1;
> > cmd->reap = false;
> > if (status) {
> > char *str = virCommandToString(cmd);
> > char *st = virCommandTranslateStatus(status);
> > virCommandError(VIR_ERR_INTERNAL_ERROR,
> > _("Child process (%s) status unexpected: %s"),
> > str ? str : cmd->args[0], NULLSTR(st));
> >
> >> But it's only been seen on two (very
> >>different) machines, and the slightest shifting of the winds makes it go
> >>away. Given how sneaky this bug appears to be, there's a slight
> >>temptation to have iptablesAddRemoveRule pass in a int* for status and
> >>better deal with the -EINTR. But I fear that might be papering over a
> >>worse race.
> >
> >I don't follow how you think there is a -EINTR being encountered in
> >libvirt.
>
> Yeah I don't really either.
>
> > I think you'd be better off investigating why iptables really
> >is exiting with status 4.
>
> Well, given what EINTR means, shouldn't src/util/iptables.c re-try
> the command if it gets that?
>
> Anyway I'll keep digging, but was wondering if anyone else has seen this.
>
> -serge
>
> --
> libvir-list mailing list
> libvir-list at redhat.com
> https://www.redhat.com/mailman/listinfo/libvir-list
More information about the libvir-list
mailing list