[Linux-cluster] Re: Hard lockups during file transfer to GNBD/GFS device

[David Brieck Jr.]

On 9/28/06, David Teigland <teigland at redhat.com> wrote:
>
> Could you try it without multipath?  You have quite a few layers there.
> Dave
>
>

Thanks for the response. I unloaded gfs, clvm, gnbd and multipath, the
reloaded gnbd, clvm and gfs. It was only talking to one of the gnbd
servers and without multipath. Here's the log from this crash. It
seems to have more info in it.

I'm kinda confused why it still has references to mulitpath though. I
unloaded the multipath module so I'm not sure why it's still in there.

Sep 29 09:39:26 db2 kernel: Unable to handle kernel NULL pointer
dereference at virtual address 00000000
Sep 29 09:39:26 db2 kernel:  printing eip:
Sep 29 09:39:26 db2 kernel: f882d427
Sep 29 09:39:26 db2 kernel: *pde = 00004001
Sep 29 09:39:26 db2 kernel: Oops: 0000 [#1]
Sep 29 09:39:26 db2 kernel: SMP
Sep 29 09:39:26 db2 kernel: Modules linked in: lock_dlm(U) gfs(U)
lock_harness(U) gnbd(U) mptctl mptbase dell_rbu nfsd exportfs lockd
nfs_acl parport_pc lp p
arport autofs4 i2c_dev i2c_core dm_round_robin dlm(U) cman(U) sunrpc
ipmi_devintf ipmi_si ipmi_msghandler iptable_filter iptable_mangle
iptable_nat ip_conntr
ack ip_tables md5 ipv6 dm_multipath joydev button battery ac uhci_hcd
ehci_hcd hw_random e1000 bonding(U) floppy sg dm_snapshot dm_zero
dm_mirror ext3 jbd dm
_mod megaraid_mbox megaraid_mm sd_mod scsi_mod
Sep 29 09:39:26 db2 kernel: CPU:    5
Sep 29 09:39:26 db2 kernel: EIP:    0060:[<f882d427>]    Not tainted VLI
Sep 29 09:39:26 db2 kernel: EFLAGS: 00010286   (2.6.9-42.0.2.ELhugemem)
Sep 29 09:39:26 db2 kernel: EIP is at journal_start+0x23/0x9e [jbd]
Sep 29 09:39:26 db2 kernel: eax: 00000000   ebx: 8ca9b300   ecx:
e1f0b400   edx: 00000042
Sep 29 09:39:26 db2 kernel: esi: e1f0bc00   edi: 1ef03000   ebp:
02325e78   esp: 1ef03bc0
Sep 29 09:39:26 db2 kernel: ds: 007b   es: 007b   ss: 0068
Sep 29 09:39:26 db2 kernel: Process rsync (pid: 20038,
threadinfo=1ef03000 task=d9f178b0)
Sep 29 09:39:26 db2 kernel: Stack: d406cde8 1ef03c00 00000031 f88a8c55
d406cde8 1ef03c00 0216fc5c d406cde8
Sep 29 09:39:26 db2 kernel:        0216fcf1 3d38f768 3d38f770 0000000a
02170076 00000080 00000080 00000080
Sep 29 09:39:26 db2 kernel:        bf756da8 8b255598 00000000 00000086
00000000 39ffe980 021700e3 02148548
Sep 29 09:39:26 db2 kernel: Call Trace:
Sep 29 09:39:26 db2 kernel:  [<f88a8c55>] ext3_dquot_drop+0x14/0x3b [ext3]
Sep 29 09:39:26 db2 kernel:  [<0216fc5c>] clear_inode+0xb4/0x102
Sep 29 09:39:26 db2 kernel:  [<0216fcf1>] dispose_list+0x47/0x6d
Sep 29 09:39:26 db2 kernel:  [<02170076>] prune_icache+0x193/0x1ec
Sep 29 09:39:26 db2 kernel:  [<021700e3>] shrink_icache_memory+0x14/0x2b
Sep 29 09:39:26 db2 kernel:  [<02148548>] shrink_slab+0xf8/0x161
Sep 29 09:39:26 db2 kernel:  [<0214952c>] try_to_free_pages+0xd1/0x1a7
Sep 29 09:39:26 db2 kernel:  [<02142f1d>] __alloc_pages+0x1b5/0x29d
Sep 29 09:39:26 db2 kernel:  [<02140e51>]
generic_file_buffered_write+0x1a1/0x533
Sep 29 09:39:26 db2 kernel:  [<0214156c>]
__generic_file_aio_write_nolock+0x389/0x3b7
Sep 29 09:39:26 db2 kernel:  [<021415d3>]
generic_file_aio_write_nolock+0x39/0x7f
Sep 29 09:39:26 db2 kernel:  [<02141736>] generic_file_write_nolock+0x84/0x99
Sep 29 09:39:26 db2 kernel:  [<f9009055>] gfs_glock_nq+0xe3/0x116 [gfs]
Sep 29 09:39:26 db2 kernel:  [<021204e9>] autoremove_wake_function+0x0/0x2d
Sep 29 09:39:26 db2 kernel:  [<f9029bac>] gfs_trans_begin_i+0xfd/0x15a [gfs]
Sep 29 09:39:26 db2 kernel:  [<f901d0fc>] do_do_write_buf+0x2a6/0x452 [gfs]
Sep 29 09:39:26 db2 kernel:  [<f901d3c3>] do_write_buf+0x11b/0x15e [gfs]
Sep 29 09:39:26 db2 kernel:  [<f901c31c>] walk_vm+0xd7/0x100 [gfs]
Sep 29 09:39:26 db2 kernel:  [<f901d4a7>] __gfs_write+0xa1/0xbb [gfs]
Sep 29 09:39:26 db2 kernel:  [<f901d2a8>] do_write_buf+0x0/0x15e [gfs]
Sep 29 09:39:26 db2 kernel:  [<f901d4cc>] gfs_write+0xb/0xe [gfs]
Sep 29 09:39:26 db2 kernel:  [<0215a52f>] vfs_write+0xb6/0xe2
Sep 29 09:39:26 db2 kernel:  [<0215a5f9>] sys_write+0x3c/0x62
Sep 29 09:39:26 db2 kernel: Code: <3>Debug: sleeping function called
from invalid context at include/linux/rwsem.h:43
Sep 29 09:39:26 db2 kernel: in_atomic():0[expected: 0], irqs_disabled():1
Sep 29 09:39:26 db2 kernel:  [<02120209>] __might_sleep+0x7d/0x88
Sep 29 09:39:26 db2 kernel:  [<0215537c>] rw_vm+0xe4/0x29c
Sep 29 09:39:26 db2 kernel:  [<f882d3fc>] new_handle+0x38/0x40 [jbd]
Sep 29 09:39:26 db2 kernel:  [<f882d3fc>] new_handle+0x38/0x40 [jbd]
Sep 29 09:39:26 db2 kernel:  [<021557f3>] get_user_size+0x30/0x57
Sep 29 09:39:26 db2 kernel:  [<f882d3fc>] new_handle+0x38/0x40 [jbd]
Sep 29 09:39:26 db2 kernel:  [<021061bb>] show_registers+0x115/0x16c
Sep 29 09:39:26 db2 kernel:  [<02106352>] die+0xdb/0x16b
Sep 29 09:39:26 db2 kernel:  [<02122a14>] vprintk+0x136/0x14a
Sep 29 09:39:26 db2 kernel:  [<0211b236>] do_page_fault+0x421/0x5f7
Sep 29 09:39:26 db2 kernel:  [<f882d427>] journal_start+0x23/0x9e [jbd]
Sep 29 09:39:26 db2 kernel:  [<0211cec9>] activate_task+0x88/0x95
Sep 29 09:39:26 db2 kernel:  [<0211d3f4>] try_to_wake_up+0x28e/0x299
Sep 29 09:39:26 db2 kernel:  [<0211ae15>] do_page_fault+0x0/0x5f7
Sep 29 09:39:26 db2 kernel:  [<f882d427>] journal_start+0x23/0x9e [jbd]
Sep 29 09:39:26 db2 kernel:  [<f88a8c55>] ext3_dquot_drop+0x14/0x3b [ext3]
Sep 29 09:39:26 db2 kernel:  [<0216fc5c>] clear_inode+0xb4/0x102
Sep 29 09:39:26 db2 kernel:  [<0216fcf1>] dispose_list+0x47/0x6d
Sep 29 09:39:26 db2 kernel:  [<02170076>] prune_icache+0x193/0x1ec
Sep 29 09:39:26 db2 kernel:  [<021700e3>] shrink_icache_memory+0x14/0x2b
Sep 29 09:39:26 db2 kernel:  [<02148548>] shrink_slab+0xf8/0x161
Sep 29 09:39:26 db2 kernel:  [<0214952c>] try_to_free_pages+0xd1/0x1a7
Sep 29 09:39:26 db2 kernel:  [<02142f1d>] __alloc_pages+0x1b5/0x29d
Sep 29 09:39:26 db2 kernel:  [<02140e51>]
generic_file_buffered_write+0x1a1/0x533
Sep 29 09:39:26 db2 kernel:  [<0214156c>]
__generic_file_aio_write_nolock+0x389/0x3b7
Sep 29 09:39:26 db2 kernel:  [<021415d3>]
generic_file_aio_write_nolock+0x39/0x7f
Sep 29 09:39:26 db2 kernel:  [<02141736>] generic_file_write_nolock+0x84/0x99
Sep 29 09:39:26 db2 kernel:  [<f9009055>] gfs_glock_nq+0xe3/0x116 [gfs]
Sep 29 09:39:26 db2 kernel:  [<021204e9>] autoremove_wake_function+0x0/0x2d
Sep 29 09:39:26 db2 kernel:  [<f9029bac>] gfs_trans_begin_i+0xfd/0x15a [gfs]
Sep 29 09:39:26 db2 kernel:  [<f901d0fc>] do_do_write_buf+0x2a6/0x452 [gfs]
Sep 29 09:39:26 db2 kernel:  [<f901d3c3>] do_write_buf+0x11b/0x15e [gfs]
Sep 29 09:39:26 db2 kernel:  [<f901c31c>] walk_vm+0xd7/0x100 [gfs]
Sep 29 09:39:26 db2 kernel:  [<f901d4a7>] __gfs_write+0xa1/0xbb [gfs]
Sep 29 09:39:26 db2 kernel:  [<f901d2a8>] do_write_buf+0x0/0x15e [gfs]
Sep 29 09:39:26 db2 kernel:  [<f901d4cc>] gfs_write+0xb/0xe [gfs]
Sep 29 09:39:26 db2 kernel:  [<0215a52f>] vfs_write+0xb6/0xe2
Sep 29 09:39:26 db2 kernel:  [<0215a5f9>] sys_write+0x3c/0x62
Sep 29 09:39:26 db2 kernel:  Bad EIP value.
Sep 29 09:39:26 db2 kernel:  <0>Fatal exception: panic in 5 seconds
Sep 29 09:42:17 db2 syslogd 1.4.1: restart.

Thanks again for your help.