[Crash-utility] Module debugging using crash

[Dheeraj Sangamkar]

Dave,
Thanks for pointing out this feature of crash.
It will be of great help.

Dheeraj

On Tue, Mar 4, 2008 at 2:49 AM, Dave Anderson <anderson at redhat.com> wrote:

>
> Dheeraj Sangamkar wrote:
>  >
>  > Hi,
>  > Is there a way to print type information about types defined in a
> loadable
>  > kernel module?  Is there a way to add to the type information that
> crash gets
>  > from the vmlinux file?
>  >
>  > This would ease debugging crashed module code be helping me print
> structures
>  > defined in modules.
>  >
>  > Dheeraj
>
> Well, that's what the "mod" command does, i.e., load the debuginfo data
> of kernel modules into the embedded gdb module of the crash session.
>
> Take the "ext3" module for example.  It declares this data structure in
> the file "fs/ext3/super.c":
>
>   static struct super_operations ext3_sops = {
>           .alloc_inode    = ext3_alloc_inode,
>           .destroy_inode  = ext3_destroy_inode,
>           .read_inode     = ext3_read_inode,
>           .write_inode    = ext3_write_inode,
>           .dirty_inode    = ext3_dirty_inode,
>           .delete_inode   = ext3_delete_inode,
>           .put_super      = ext3_put_super,
>           .write_super    = ext3_write_super,
>           .sync_fs        = ext3_sync_fs,
>           .write_super_lockfs = ext3_write_super_lockfs,
>           .unlockfs       = ext3_unlockfs,
>           .statfs         = ext3_statfs,
>           .remount_fs     = ext3_remount,
>           .clear_inode    = ext3_clear_inode,
>           .show_options   = ext3_show_options,
>   #ifdef CONFIG_QUOTA
>           .quota_read     = ext3_quota_read,
>           .quota_write    = ext3_quota_write,
>   #endif
>   };
>
> or for example, this data structure type is defined in
> "fs/ext3/xattr.h":
>
>   struct ext3_xattr_header {
>           __le32  h_magic;        /* magic number for identification */
>           __le32  h_refcount;     /* reference count */
>           __le32  h_blocks;       /* number of disk blocks used */
>           __le32  h_hash;         /* hash value of all attributes */
>           __u32   h_reserved[4];  /* zero right now */
>   };
>
> Without the ext3 module's debuginfo data, crash (and gdb) doesn't know
> what they are, so if I try to print the "ext3_sops" symbolically,
> this happens:
>
>   crash> p ext3_sops
>   p: gdb request failed: p ext3_sops
>   crash> whatis ext3_sops
>   whatis: gdb request failed: whatis ext3_sops
>   crash>
>
> And if I want to see what an ext3_xattr_header structure is defined
> as:
>
>   crash> struct ext3_xattr_header
>   struct: invalid data structure reference: ext3_xattr_header
>   crash>
>
> But if the module's debuginfo data is loaded:
>
>   crash> mod -s ext3
>    MODULE   NAME                  SIZE  OBJECT FILE
>   e08c4f80  ext3                123337
> /lib/modules/2.6.18-53.el5/kernel/fs/ext3/ext3.ko
>   crash>
>
> (or use "mod -S" to load the debuginfo data of all modules),
> then the same commands that failed above work like so:
>
>   crash> p ext3_sops
>   ext3_sops = $4 = {
>     alloc_inode = 0xe08b1c54 <ext3_alloc_inode>,
>     destroy_inode = 0xe08b2e4a <ext3_destroy_inode>,
>     read_inode = 0xe08ae0c9 <ext3_read_inode>,
>     dirty_inode = 0xe08ae063 <ext3_dirty_inode>,
>     write_inode = 0xe08ad696 <ext3_write_inode>,
>     put_inode = 0,
>     drop_inode = 0,
>     delete_inode = 0xe08ad7af <ext3_delete_inode>,
>     put_super = 0xe08b2f6d <ext3_put_super>,
>     write_super = 0xe08b1c37 <ext3_write_super>,
>     sync_fs = 0xe08b291d <ext3_sync_fs>,
>     write_super_lockfs = 0xe08b295f <ext3_write_super_lockfs>,
>     unlockfs = 0xe08b29ae <ext3_unlockfs>,
>     statfs = 0xe08b2c55 <ext3_statfs>,
>     remount_fs = 0xe08b2a21 <ext3_remount>,
>     clear_inode = 0xe08b2e5a <ext3_clear_inode>,
>     umount_begin = 0,
>     show_options = 0xe08b2835 <ext3_show_options>,
>     show_stats = 0,
>     quota_read = 0xe08b449b <ext3_quota_read>,
>     quota_write = 0xe08b45f5 <ext3_quota_write>
>   }
>   crash>
>
>   crash> ext3_xattr_header
>   struct ext3_xattr_header {
>       __le32 h_magic;
>       __le32 h_refcount;
>       __le32 h_blocks;
>       __le32 h_hash;
>       __u32 h_reserved[4];
>   }
>   SIZE: 32
>   crash>
>
> The example above presumes the Red Hat split debuginfo module
> format, where the "mod" command shows that the ext3 module
> exists in:
>
>   /lib/modules/2.6.18-53.el5/kernel/fs/ext3/ext3.ko,
>
> but that ext3.ko module contains a link to its associated debuginfo
> part, which is actually located in:
>
> /usr/lib/debug/lib/modules/2.6.18-53.el5/kernel/fs/ext3/ext3.ko.debug
>
> If your module.ko file is not split into two, and has been built
> with -g, then you can load its debuginfo data crash like so:
>
>   crash> mod -s yourmodule yourmodule.ko
>
> Dave
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/crash-utility/attachments/20080304/2c9df316/attachment.htm>