[Crash-utility] Module debugging using crash

[Dave Anderson]

Dheeraj Sangamkar wrote:
 >
 > Hi,
 > Is there a way to print type information about types defined in a loadable
 > kernel module?  Is there a way to add to the type information that crash gets
 > from the vmlinux file?
 >
 > This would ease debugging crashed module code be helping me print structures
 > defined in modules.
 >
 > Dheeraj

Well, that's what the "mod" command does, i.e., load the debuginfo data
of kernel modules into the embedded gdb module of the crash session.

Take the "ext3" module for example.  It declares this data structure in
the file "fs/ext3/super.c":

   static struct super_operations ext3_sops = {
           .alloc_inode    = ext3_alloc_inode,
           .destroy_inode  = ext3_destroy_inode,
           .read_inode     = ext3_read_inode,
           .write_inode    = ext3_write_inode,
           .dirty_inode    = ext3_dirty_inode,
           .delete_inode   = ext3_delete_inode,
           .put_super      = ext3_put_super,
           .write_super    = ext3_write_super,
           .sync_fs        = ext3_sync_fs,
           .write_super_lockfs = ext3_write_super_lockfs,
           .unlockfs       = ext3_unlockfs,
           .statfs         = ext3_statfs,
           .remount_fs     = ext3_remount,
           .clear_inode    = ext3_clear_inode,
           .show_options   = ext3_show_options,
   #ifdef CONFIG_QUOTA
           .quota_read     = ext3_quota_read,
           .quota_write    = ext3_quota_write,
   #endif
   };

or for example, this data structure type is defined in
"fs/ext3/xattr.h":

   struct ext3_xattr_header {
           __le32  h_magic;        /* magic number for identification */
           __le32  h_refcount;     /* reference count */
           __le32  h_blocks;       /* number of disk blocks used */
           __le32  h_hash;         /* hash value of all attributes */
           __u32   h_reserved[4];  /* zero right now */
   };

Without the ext3 module's debuginfo data, crash (and gdb) doesn't know
what they are, so if I try to print the "ext3_sops" symbolically,
this happens:

   crash> p ext3_sops
   p: gdb request failed: p ext3_sops
   crash> whatis ext3_sops
   whatis: gdb request failed: whatis ext3_sops
   crash>

And if I want to see what an ext3_xattr_header structure is defined
as:

   crash> struct ext3_xattr_header
   struct: invalid data structure reference: ext3_xattr_header
   crash>

But if the module's debuginfo data is loaded:

   crash> mod -s ext3
    MODULE   NAME                  SIZE  OBJECT FILE
   e08c4f80  ext3                123337 
/lib/modules/2.6.18-53.el5/kernel/fs/ext3/ext3.ko
   crash>

(or use "mod -S" to load the debuginfo data of all modules),
then the same commands that failed above work like so:

   crash> p ext3_sops
   ext3_sops = $4 = {
     alloc_inode = 0xe08b1c54 <ext3_alloc_inode>,
     destroy_inode = 0xe08b2e4a <ext3_destroy_inode>,
     read_inode = 0xe08ae0c9 <ext3_read_inode>,
     dirty_inode = 0xe08ae063 <ext3_dirty_inode>,
     write_inode = 0xe08ad696 <ext3_write_inode>,
     put_inode = 0,
     drop_inode = 0,
     delete_inode = 0xe08ad7af <ext3_delete_inode>,
     put_super = 0xe08b2f6d <ext3_put_super>,
     write_super = 0xe08b1c37 <ext3_write_super>,
     sync_fs = 0xe08b291d <ext3_sync_fs>,
     write_super_lockfs = 0xe08b295f <ext3_write_super_lockfs>,
     unlockfs = 0xe08b29ae <ext3_unlockfs>,
     statfs = 0xe08b2c55 <ext3_statfs>,
     remount_fs = 0xe08b2a21 <ext3_remount>,
     clear_inode = 0xe08b2e5a <ext3_clear_inode>,
     umount_begin = 0,
     show_options = 0xe08b2835 <ext3_show_options>,
     show_stats = 0,
     quota_read = 0xe08b449b <ext3_quota_read>,
     quota_write = 0xe08b45f5 <ext3_quota_write>
   }
   crash>

   crash> ext3_xattr_header
   struct ext3_xattr_header {
       __le32 h_magic;
       __le32 h_refcount;
       __le32 h_blocks;
       __le32 h_hash;
       __u32 h_reserved[4];
   }
   SIZE: 32
   crash>

The example above presumes the Red Hat split debuginfo module
format, where the "mod" command shows that the ext3 module
exists in:

   /lib/modules/2.6.18-53.el5/kernel/fs/ext3/ext3.ko,

but that ext3.ko module contains a link to its associated debuginfo
part, which is actually located in:

/usr/lib/debug/lib/modules/2.6.18-53.el5/kernel/fs/ext3/ext3.ko.debug

If your module.ko file is not split into two, and has been built
with -g, then you can load its debuginfo data crash like so:

   crash> mod -s yourmodule yourmodule.ko

Dave