[Crash-utility] Missing PID 1 is crash problem with losing tasks

[Bob Montgomery]

(Was Re: [Crash-utility] mount cmd crashes crash)

On Thu, 2010-08-19 at 12:45 +0000, Dave Anderson wrote:
> ----- "Bob Montgomery" <bob.montgomery at hp.com> wrote:

> > > Yeah, it's not important to use the context of pid 1, but it just needs
> > > some context, and I had presumed that init would always exist.  I thought
> > > that the panic("Attempted to kill the idle task!") in do_exit() would
> > > prevent pid 1 from ever going away -- but apparently your kernel figured
> > > out how to do it elsewhere...  ;-)
> > 
> > That test is for PID 0, not PID 1 (at least on the kernel I'm
> > debugging.)  However, there is this also:
> > 
> >         if (unlikely(tsk == child_reaper))
> >                 panic("Attempted to kill init!");
> 
> That's the one I *meant*...   ;-)
> 
> > 
> > And child_reaper in the dump points to a task struct for init that isn't
> > in the ps listing.  Hmmm.  Maybe that part *is* interesting in this dump...

Well, I've been picking at this some more.  PID 1 is in the system, but
crash misses it when it's building its table of tasks in
refresh_hlist_task_table_v2().  In fact, on my particular dump, it loses
track of at least 3 processes. 

The attached patch changes that behavior.  It has to do with collisions
on the pid_hash table where an early item on the chain has a NULL task
pointer which causes the code to ignore subsequent items on that
collision chain.  I'm not sure what it means when the tasks[0].first
pointer in the struct pid is NULL, but that's what triggers the problem
and keeps crash from following the pid_chain pointer to the next struct
pid.  I am not confident that this whole area is correct yet, just
closer to correct than it was. 

These now appear in the ps output:

crash-5.0.6-fix2> ps 1 8144 998
   PID    PPID  CPU       TASK        ST  %MEM     VSZ    RSS  COMM
      1      0   1  ffff81012bd3c780  IN   0.0    6124    688  init
   8144   6257   0  ffff81011996e140  RU   0.7  108876  35016  mirrorclient
    998     11   0  ffff81012a9cd780  IN   0.0       0      0  [fc_dl_1]

where before:

crash-5.0.6-fix> ps 1 8144 998
ps: invalid task or pid value: 1

ps: invalid task or pid value: 8144

ps: invalid task or pid value: 998

This might have been some transition behavior of the pid hash design in
the kernel, because I've got two dumps based on 2.6.18 kernels that show
missing processes (this one had 3 out of 532, the other had 1 out of
146), but my new patched crash doesn't reveal any missing processes in
2.6.29 and newer dumps (I checked 4 dumps, with process counts ranging
from 362 to 926).  Only my recent 2.6.18 dump was lucky enough to be
missing PID 1, with me being lucky enough to try crash's mount command,
or we'd still not know about it :-)

The patch is simple, but has lots of lines because I moved the indent.

Bob Montgomery
Working at HP




-------------- next part --------------
A non-text attachment was scrubbed...
Name: lost_task.patch
Type: text/x-patch
Size: 2500 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/crash-utility/attachments/20100825/dd20b062/attachment.bin>