[Crash-utility] kernel-debuginfo & System.map

Fri Oct 10 13:38:36 UTC 2008

----- "Jun Koi" <junkoi2004 at gmail.com> wrote:

> Hi,
> 
> I have a question on using crash:
> - Is it imperative to have System.map to use crash? It seems not to
> me, if we have a kernel debuginfo file. Is that correct?

That's absolutely correct.

The *only* reason for using a System.map file is if the vmlinux
file that you are putting on the crash command line is not
exactly the same as the vmlinux that generated the dumpfile
or is running on the live system.  

> - Is it imperative to have a kernel debuginfo file to use crash
> (/boot/vmlinux-*)? Or it is OK to have System.map only?

Yes, you absolutely need a vmlinux file with debuginfo data.
Otherwise the crash session will kill itself immediately
with a "no debugging data available" error.  

By adding a System.map file to the command line, you are implying
that the kernel symbol values found in in the command line vmlinux
file do not match those in the dumpfile or live system.  So after
the "wrong" vmlinux file is passed to the embedded gdb module,
the crash utility goes into the gdb code throug a backdoor
and modifies all of the "minimal_symbol" data structures to
reflect the correct symbol values found in the System.map.

So for example, taking my live system, no arguments are necessary,
because the vmlinux file is found in a "known" location:

 $ crash

 crash 4.0-7.2
 Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008  Red Hat, Inc.
 Copyright (C) 2004, 2005, 2006  IBM Corporation
 Copyright (C) 1999-2006  Hewlett-Packard Co
 Copyright (C) 2005, 2006  Fujitsu Limited
 Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
 Copyright (C) 2005  NEC Corporation
 Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
 Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
 This program is free software, covered by the GNU General Public License,
 and you are welcome to change it and/or distribute copies of it under
 certain conditions.  Enter "help copying" to see the conditions.
 This program has absolutely no warranty.  Enter "help warranty" for details.

 GNU gdb 6.1
 Copyright 2004 Free Software Foundation, Inc.
 GDB is free software, covered by the GNU General Public License, and you are
 welcome to change it and/or distribute copies of it under certain conditions.
 Type "show copying" to see the conditions.
 There is absolutely no warranty for GDB.  Type "show warranty" for details.
 This GDB was configured as "x86_64-unknown-linux-gnu"...

       KERNEL: /usr/lib/debug/lib/modules/2.6.18-92.el5/vmlinux
     DUMPFILE: /dev/crash
         CPUS: 8
         DATE: Fri Oct 10 09:24:10 2008
       UPTIME: 4 days, 00:07:36
 LOAD AVERAGE: 0.25, 0.20, 0.18
        TASKS: 244
     NODENAME: crash.usersys.redhat.com
      RELEASE: 2.6.18-92.el5
      VERSION: #1 SMP Tue Apr 29 13:16:15 EDT 2008
      MACHINE: x86_64  (1995 Mhz)
       MEMORY: 1 GB
          PID: 9033
      COMMAND: "crash"
         TASK: ffff810038e397e0  [THREAD_INFO: ffff81001c5f8000]
          CPU: 4
        STATE: TASK_RUNNING (ACTIVE)

 crash> 

But I can (for no good reason) enter the vmlinux file and its
associated System.map file, and as a result you'll see the
"patching ##### gdb minimal_symbol values" message:

 $ crash /usr/lib/debug/lib/modules/2.6.18-92.el5/vmlinux /boot/System.map-2.6.18-92.el5

 crash 4.0-7.2
 Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008  Red Hat, Inc.
 Copyright (C) 2004, 2005, 2006  IBM Corporation
 Copyright (C) 1999-2006  Hewlett-Packard Co
 Copyright (C) 2005, 2006  Fujitsu Limited
 Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
 Copyright (C) 2005  NEC Corporation
 Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
 Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
 This program is free software, covered by the GNU General Public License,
 and you are welcome to change it and/or distribute copies of it under
 certain conditions.  Enter "help copying" to see the conditions.
 This program has absolutely no warranty.  Enter "help warranty" for details.

 GNU gdb 6.1
 Copyright 2004 Free Software Foundation, Inc.
 GDB is free software, covered by the GNU General Public License, and you are
 welcome to change it and/or distribute copies of it under certain conditions.
 Type "show copying" to see the conditions.
 There is absolutely no warranty for GDB.  Type "show warranty" for details.
 This GDB was configured as "x86_64-unknown-linux-gnu"...

 please wait... (patching 29117 gdb minimal_symbol values) 

   SYSTEM MAP: /boot/System.map-2.6.18-92.el5
 DEBUG KERNEL: /usr/lib/debug/lib/modules/2.6.18-92.el5/vmlinux (2.6.18-92.el5)
     DUMPFILE: /dev/crash
         CPUS: 8
         DATE: Fri Oct 10 09:25:27 2008
       UPTIME: 4 days, 00:08:54
 LOAD AVERAGE: 0.24, 0.20, 0.18
        TASKS: 244
     NODENAME: crash.usersys.redhat.com
      RELEASE: 2.6.18-92.el5
      VERSION: #1 SMP Tue Apr 29 13:16:15 EDT 2008
      MACHINE: x86_64  (1995 Mhz)
       MEMORY: 1 GB
          PID: 9267
      COMMAND: "crash"
         TASK: ffff81003b6450c0  [THREAD_INFO: ffff8100394d0000]
          CPU: 2
        STATE: TASK_RUNNING (ACTIVE)

 crash> 

The use of a System.map file is mainly a remnant of the "old days"
when distros typically didn't make a debuginfo vmlinux file
available.  So the kernel src.rpm would have to be rebuilt after
the fact, with -g added to the kernel Makefile's CFLAGS.  But
doing that would/could alter the symbol values in the rebuilt 
debuginfo vmlinux such that they wouldn't match up exactly to
those in the original vmlinux.

It should be that the data structure declarations in the "wrong" vmlinux
are presumed to be correct, or correct enough such that the crash session
will work.  If any crucial data structures that crash depends upon
have been modified, then all bets are off.

Dave