[Crash-utility] crash aborts with cannot determine idle task

[Dave Anderson]

 > While running crash-4.0-6.1 on a vmcore , crash is aborting with
 >
 > --------
 > crash: cannot determine idle task addresses from init_tasks[] or runqueues[]
 >
 > crash: cannot resolve "init_task_union"
 > -------
 >
 >
 > during startup. The kernel is later than 2.6.18 . The changelog
 > http://people.redhat.com/anderson/crash.changelog.html mentions that this
 > is possibly fixed in version 4.0-3.1 . Hence could you pls point me to the
 > patch that fixed this problem.
 >
 > thanks,
 > Chandru

That particular two-year-old patch simply recognized and dealt with the kernel
name change from "struct runqueue" to "struct rq":

--- kernel.c    2 Aug 2006 14:34:35 -0000       1.140
+++ kernel.c    2 Aug 2006 18:35:31 -0000       1.141
@@ -55,6 +55,7 @@
         int i;
         char *p1, *p2, buf[BUFSIZE];
         struct syment *sp1, *sp2;
+       char *rqstruct;

         if (pc->flags & KERNEL_DEBUG_QUERY)
                 return;
@@ -158,7 +159,15 @@
                                 &kt->__per_cpu_offset[0]);
                         kt->flags |= PER_CPU_OFF;
                 }
-               MEMBER_OFFSET_INIT(runqueue_cpu, "runqueue", "cpu");
+               if (STRUCT_EXISTS("runqueue"))
+                       rqstruct = "runqueue";
+               else if (STRUCT_EXISTS("rq"))
+                       rqstruct = "rq";
+
+               MEMBER_OFFSET_INIT(runqueue_cpu, rqstruct, "cpu");
+               /*
+                * 'cpu' does not exist in 'struct rq'.
+                */
                 if (VALID_MEMBER(runqueue_cpu) &&
                     (get_array_length("runqueue.cpu", NULL, 0) > 0)) {
                         MEMBER_OFFSET_INIT(cpu_s_curr, "cpu_s", "curr");
@@ -183,17 +192,17 @@
              "runq_siblings: %d: __cpu_idx and __rq_idx arrays don't exist?\n",
                                         kt->runq_siblings);
                 } else {
-                       MEMBER_OFFSET_INIT(runqueue_idle, "runqueue", "idle");
-                       MEMBER_OFFSET_INIT(runqueue_curr, "runqueue", "curr");
+                       MEMBER_OFFSET_INIT(runqueue_idle, rqstruct, "idle");
+                       MEMBER_OFFSET_INIT(runqueue_curr, rqstruct, "curr");
                         ASSIGN_OFFSET(runqueue_cpu) = INVALID_OFFSET;
                 }
-               MEMBER_OFFSET_INIT(runqueue_active, "runqueue", "active");
-               MEMBER_OFFSET_INIT(runqueue_expired, "runqueue", "expired");
-               MEMBER_OFFSET_INIT(runqueue_arrays, "runqueue", "arrays");
+               MEMBER_OFFSET_INIT(runqueue_active, rqstruct, "active");
+               MEMBER_OFFSET_INIT(runqueue_expired, rqstruct, "expired");
+               MEMBER_OFFSET_INIT(runqueue_arrays, rqstruct, "arrays");
                 MEMBER_OFFSET_INIT(prio_array_queue, "prio_array", "queue");
                  MEMBER_OFFSET_INIT(prio_array_nr_active, "prio_array",
                          "nr_active");
-               STRUCT_SIZE_INIT(runqueue, "runqueue");
+               STRUCT_SIZE_INIT(runqueue, rqstruct);
                 STRUCT_SIZE_INIT(prio_array, "prio_array");

                 /*

So that patch was required for 2.6.18.

When you say that the "kernel is later than 2.6.18", well, that doesn't
help me much.

Look at the crash function get_idle_threads() in task.c, which is where
you're failing.  It runs through the history of the symbols that Linux
has used over the years for the run queues.  For the most recent kernels,
it looks for the "per_cpu__runqueues" symbol.  At least on 2.6.25-rc2,
the kernel still defines them in kernel/sched.c like this:

   static DEFINE_PER_CPU_SHARED_ALIGNED(struct rq, runqueues);

So if you do an "nm -Bn vmlinux | grep runqueues", you should see:

   # nm -Bn vmlinux-2.6.25-rc1-ext4-1 | grep runqueues
   ffffffff8082b700 d per_cpu__runqueues
   #

I'm guessing that's not the problem -- so presuming that the symbol *does*
exist, find out why it's failing to increment "cnt" in this part of
get_idle_threads():

        if (symbol_exists("per_cpu__runqueues") &&
             VALID_MEMBER(runqueue_idle)) {
                 runqbuf = GETBUF(SIZE(runqueue));
                 for (i = 0; i < nr_cpus; i++) {
                         if ((kt->flags & SMP) && (kt->flags & PER_CPU_OFF)) {
                                 runq = symbol_value("per_cpu__runqueues") +
                                         kt->__per_cpu_offset[i];
                         } else
                                 runq = symbol_value("per_cpu__runqueues");

                         readmem(runq, KVADDR, runqbuf,
                                 SIZE(runqueue), "runqueues entry (per_cpu)",
                                 FAULT_ON_ERROR);
                         tasklist[i] = ULONG(runqbuf + OFFSET(runqueue_idle));
                         if (IS_KVADDR(tasklist[i]))
                                 cnt++;
                 }
         }

Determine whether it even makes it to the inner for loop, whether
the pre-determined nr_cpus value makes sense, whether the SMP flag
reflects whether the kernel was compiled for SMP, whether the PER_CPU_OFF
flag was set, what address was calculated, etc...

Dave