[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

[Linux-cluster] kernel panic umounting GFS FS



well,
today, I try to unmount GFS form one node to update it (for the latest kernel).
All nodes had a kernel panic.




Here is the stack :


May 24 07:23:09 ancona kernel: CMAN: too many transition restarts - will die
May 24 07:23:09 ancona kernel: CMAN: we are leaving the cluster. Inconsistent cluster view
May 24 07:23:09 ancona kernel: WARNING: dlm_emergency_shutdown
May 24 07:23:09 ancona clurgmgrd[4604]: <warning> #67: Shutting down uncleanly
May 24 07:23:09 ancona kernel: WARNING: dlm_emergency_shutdown
May 24 07:23:09 ancona kernel: SM: 00000006 sm_stop: SG still joined
May 24 07:23:09 ancona kernel: SM: 01000008 sm_stop: SG still joined
May 24 07:23:09 ancona kernel: SM: 02000014 sm_stop: SG still joined
May 24 07:23:09 ancona kernel: SM: 0300000a sm_stop: SG still joined
May 24 07:23:09 ancona ccsd[3732]: Cluster manager shutdown. Attemping to reconnect... May 24 07:23:10 ancona kernel: dlm: dlm_unlock: lkid 947100ed lockspace not found
May 24 07:23:10 ancona kernel: nval 91ed0131 fr 8 r 8        2
May 24 07:23:10 ancona kernel: home (3942) req reply einval 921a00a3 fr 8 r 8 2 May 24 07:23:10 ancona kernel: home (3942) req reply einval 924b0156 fr 7 r 7 2
May 24 07:23:10 ancona kernel: home send einval to 7
May 24 07:23:10 ancona kernel: home (3942) req reply einval 934f0161 fr 1 r 1 2 May 24 07:23:10 ancona kernel: home (3942) req reply einval 90a603ad fr 8 r 8 2 May 24 07:23:10 ancona kernel: home (3942) req reply einval 92b600d0 fr 4 r 4 2 May 24 07:23:10 ancona kernel: home (3942) req reply einval 915b02a7 fr 5 r 5 2 May 24 07:23:10 ancona kernel: home (3942) req reply einval 935b0262 fr 5 r 5 2 May 24 07:23:10 ancona kernel: home (3942) req reply einval 922d0261 fr 5 r 5 2
May 24 07:23:10 ancona kernel: home send einval to 2
May 24 07:23:10 ancona kernel: home send einval to 8
May 24 07:23:10 ancona kernel: home (3942) req reply einval 92b00008 fr 7 r 7 2
May 24 07:23:10 ancona kernel: home send einval to 7
May 24 07:23:10 ancona kernel: home (3942) req reply einval 92ca0337 fr 1 r 1 2 May 24 07:23:10 ancona kernel: home (3942) req reply einval 932d0128 fr 1 r 1 2 May 24 07:23:10 ancona kernel: home (3942) req reply einval 9276022a fr 7 r 7 2 May 24 07:23:10 ancona kernel: home (3942) req reply einval 94a90311 fr 8 r 8 2 May 24 07:23:10 ancona kernel: home (3942) req reply einval 93ec0156 fr 8 r 8 2 May 24 07:23:10 ancona kernel: 3931 pr_start last_stop 0 last_start 6 last_finish 0 May 24 07:23:10 ancona kernel: 3931 pr_start count 7 type 2 event 6 flags 250
May 24 07:23:10 ancona kernel: 3931 claim_jid 4
May 24 07:23:10 ancona kernel: 3931 pr_start 6 done 1
May 24 07:23:10 ancona kernel: 3931 pr_finish flags 5a
May 24 07:23:10 ancona kernel: 3916 recovery_done jid 4 msg 309 a
May 24 07:23:10 ancona kernel: 3916 recovery_done nodeid 6 flg 18
May 24 07:23:10 ancona kernel: 3930 pr_start last_stop 6 last_start 8 last_finish 6 May 24 07:23:10 ancona kernel: 3930 pr_start count 8 type 2 event 8 flags 21a
May 24 07:23:10 ancona kernel: 3930 pr_start 8 done 1
May 24 07:23:10 ancona kernel: 3930 pr_finish flags 1a
May 24 07:23:10 ancona kernel:
May 24 07:23:10 ancona kernel: lock_dlm: Assertion failed on line 361 of file /builddir/build/BUILD/gfs-kernel-2.6.9-75/smp/src/dlm/lock.c May 24 07:23:10 ancona kernel: lock_dlm: assertion: "!error || (plock && error == -EINPROGRESS)"
May 24 07:23:10 ancona kernel: lock_dlm:  time = 2227212828
May 24 07:23:10 ancona kernel: home: error=-22 num=5,9641cab lkf=0 flags=84
May 24 07:23:10 ancona kernel:
May 24 07:23:10 ancona kernel: ------------[ cut here ]------------
May 24 07:23:10 ancona kernel: kernel BUG at /builddir/build/BUILD/gfs-kernel-2.6.9-75/smp/src/dlm/lock.c:361!
May 24 07:23:10 ancona kernel: invalid operand: 0000 [#1]
May 24 07:23:10 ancona kernel: SMP
May 24 07:23:11 ancona kernel: Modules linked in: autofs4 lock_dlm(U) gfs(U) lock_harness(U) dlm(U) cman(U) md5 ipv6 sunrpc arpt_mangle arptable_filter arp_tables dm_mirror dm_round_robin dm_multipath button battery ac ohci_hcd tg3 bonding(U) floppy sg st ext3 jbd dm_mod qla2300 qla2xxx scsi_transport_fc cciss sd_mod scsi_mod
May 24 07:23:11 ancona kernel: CPU:    1
May 24 07:23:11 ancona kernel: EIP:    0060:[<f8ae8611>]    Not tainted VLI
May 24 07:23:11 ancona kernel: EFLAGS: 00010246   (2.6.9-67.0.7.ELsmp)
May 24 07:23:11 ancona kernel: EIP is at do_dlm_unlock+0xa9/0xbf [lock_dlm]
May 24 07:23:11 ancona kernel: eax: 00000001 ebx: e82b5b80 ecx: f6cdbef0 edx: f8aed2d3 May 24 07:23:11 ancona kernel: esi: ffffffea edi: 00000000 ebp: f8a53000 esp: f6cdbeec
May 24 07:23:11 ancona kernel: ds: 007b   es: 007b   ss: 0068
May 24 07:23:11 ancona kernel: Process gfs_glockd (pid: 3939, threadinfo=f6cdb000 task=f70d19b0) May 24 07:23:11 ancona kernel: Stack: f8aed2d3 f8a53000 00000003 e82b5b80 f8ae88b2 f8b48ede 00000001 ea3e7874 May 24 07:23:11 ancona kernel: ea3e7858 f8b3ed63 f8b75e60 ed9fcac0 ea3e7858 f8b75e60 e6fcf8fc f8b3e257 May 24 07:23:11 ancona kernel: ea3e7858 00000001 ea3e7858 f8b3e30e ea3e7874 00000000 f8b3f5f2 00000000
May 24 07:23:11 ancona kernel: Call Trace:
May 24 07:23:11 ancona kernel: [<f8ae88b2>] lm_dlm_unlock+0x14/0x1c [lock_dlm]
May 24 07:23:11 ancona kernel:  [<f8b48ede>] gfs_lm_unlock+0x2c/0x42 [gfs]
May 24 07:23:11 ancona kernel: [<f8b3ed63>] gfs_glock_drop_th+0xf3/0x12d [gfs]
May 24 07:23:11 ancona kernel:  [<f8b3e257>] rq_demote+0x7f/0x98 [gfs]
May 24 07:23:11 ancona kernel:  [<f8b3e30e>] run_queue+0x5a/0xc1 [gfs]
May 24 07:23:11 ancona kernel:  [<f8b3f5f2>] gfs_glock_dq+0x15f/0x16e [gfs]
May 24 07:23:11 ancona kernel: [<f8b3f946>] gfs_glock_dq_uninit+0x8/0x10 [gfs] May 24 07:23:11 ancona kernel: [<f8b4251e>] gfs_inode_destroy+0x8e/0xbf [gfs] May 24 07:23:11 ancona kernel: [<f8b403c8>] gfs_reclaim_glock+0xa2/0x13c [gfs]
May 24 07:23:11 ancona kernel:  [<f8b32e05>] gfs_glockd+0x39/0xde [gfs]
May 24 07:23:11 ancona kernel:  [<c011e7b9>] default_wake_function+0x0/0xc
May 24 07:23:11 ancona kernel:  [<c02d8876>] ret_from_fork+0x6/0x14
May 24 07:23:11 ancona kernel:  [<c011e7b9>] default_wake_function+0x0/0xc
May 24 07:23:11 ancona kernel:  [<f8b32dcc>] gfs_glockd+0x0/0xde [gfs]
May 24 07:23:11 ancona kernel:  [<c01041f5>] kernel_thread_helper+0x5/0xb
May 24 07:23:11 ancona kernel: Code: 73 34 8b 03 ff 73 2c ff 73 08 ff 73 04 ff 73 0c 56 ff 70 18 68 ef d3 ae f8 e8 de a2 63 c7 83 c4 34 68 d3 d2 ae f8 e8 d1 a2 63 c7 <0f> 0b 69 01 1b d2 ae f8 68 d5 d2 ae f8 e8 8c 9a 63 c7 5b 5e 5f
May 24 07:23:11 ancona kernel:  <0>Fatal exception: panic in 5 seconds
May 24 07:23:11 ancona kernel: dlm: dlm_lock: no lockspace
May 24 07:23:12 ancona kernel: nval 91ed0131 fr 8 r 8        2
May 24 07:23:12 ancona kernel: home (3942) req reply einval 921a00a3 fr 8 r 8 2 May 24 07:23:12 ancona kernel: home (3942) req reply einval 924b0156 fr 7 r 7 2
May 24 07:23:12 ancona kernel: home send einval to 7
May 24 07:23:12 ancona kernel: home (3942) req reply einval 934f0161 fr 1 r 1 2 May 24 07:23:12 ancona kernel: home (3942) req reply einval 90a603ad fr 8 r 8 2 May 24 07:23:12 ancona kernel: home (3942) req reply einval 92b600d0 fr 4 r 4 2 May 24 07:23:12 ancona kernel: home (3942) req reply einval 915b02a7 fr 5 r 5 2 May 24 07:23:12 ancona kernel: home (3942) req reply einval 935b0262 fr 5 r 5 2 May 24 07:23:12 ancona kernel: home (3942) req reply einval 922d0261 fr 5 r 5 2
May 24 07:23:12 ancona kernel: home send einval to 2
May 24 07:23:12 ancona kernel: home send einval to 8
May 24 07:23:12 ancona kernel: home (3942) req reply einval 92b00008 fr 7 r 7 2
May 24 07:23:12 ancona kernel: home send einval to 7
May 24 07:23:12 ancona kernel: home (3942) req reply einval 92ca0337 fr 1 r 1 2 May 24 07:23:12 ancona kernel: home (3942) req reply einval 932d0128 fr 1 r 1 2 May 24 07:23:12 ancona kernel: home (3942) req reply einval 9276022a fr 7 r 7 2 May 24 07:23:12 ancona kernel: home (3942) req reply einval 94a90311 fr 8 r 8 2 May 24 07:23:12 ancona kernel: home (3942) req reply einval 93ec0156 fr 8 r 8 2 May 24 07:23:12 ancona kernel: 3931 pr_start last_stop 0 last_start 6 last_finish 0 May 24 07:23:12 ancona kernel: 3931 pr_start count 7 type 2 event 6 flags 250
May 24 07:23:12 ancona kernel: 3931 claim_jid 4
May 24 07:23:12 ancona kernel: 3931 pr_start 6 done 1
May 24 07:23:12 ancona kernel: 3931 pr_finish flags 5a
May 24 07:23:12 ancona kernel: 3916 recovery_done jid 4 msg 309 a
May 24 07:23:12 ancona kernel: 3916 recovery_done nodeid 6 flg 18
May 24 07:23:12 ancona kernel: 3930 pr_start last_stop 6 last_start 8 last_finish 6 May 24 07:23:12 ancona kernel: 3930 pr_start count 8 type 2 event 8 flags 21a
May 24 07:23:12 ancona kernel: 3930 pr_start 8 done 1
May 24 07:23:12 ancona kernel: 3930 pr_finish flags 1a
May 24 07:23:12 ancona kernel:
May 24 07:23:12 ancona kernel: lock_dlm: Assertion failed on line 432 of file /builddir/build/BUILD/gfs-kernel-2.6.9-75/smp/src/dlm/lock.c
May 24 07:23:13 ancona kernel: lock_dlm:  assertion:  "!error"
May 24 07:23:13 ancona kernel: lock_dlm:  time = 2227213341
May 24 07:23:13 ancona kernel: home: num=2,7a2ec26 err=-22 cur=-1 req=3 lkf=10000
May 24 07:23:13 ancona kernel:
May 24 07:23:13 ancona kernel: ------------[ cut here ]------------
May 24 07:23:13 ancona kernel: kernel BUG at /builddir/build/BUILD/gfs-kernel-2.6.9-75/smp/src/dlm/lock.c:432!
May 24 07:23:13 ancona kernel: invalid operand: 0000 [#2]
May 24 07:23:13 ancona kernel: SMP
May 24 07:23:13 ancona kernel: Modules linked in: autofs4 lock_dlm(U) gfs(U) lock_harness(U) dlm(U) cman(U) md5 ipv6 sunrpc arpt_mangle arptable_filter arp_tables dm_mirror dm_round_robin dm_multipath button battery ac ohci_hcd tg3 bonding(U) floppy sg st ext3 jbd dm_mod qla2300 qla2xxx scsi_transport_fc cciss sd_mod scsi_mod
May 24 07:23:13 ancona kernel: CPU:    0
May 24 07:23:13 ancona kernel: EIP:    0060:[<f8ae8798>]    Not tainted VLI
May 24 07:23:13 ancona kernel: EFLAGS: 00010246   (2.6.9-67.0.7.ELsmp)
May 24 07:23:13 ancona kernel: EIP is at do_dlm_lock+0x134/0x14e [lock_dlm]
May 24 07:23:13 ancona kernel: eax: 00000001 ebx: ffffffea ecx: ee2f5c34 edx: f8aed2d3 May 24 07:23:13 ancona kernel: esi: f8ae87b7 edi: f7e4cc00 ebp: e5fa9280 esp: ee2f5c30
May 24 07:23:13 ancona kernel: ds: 007b   es: 007b   ss: 0068
May 24 07:23:13 ancona kernel: Process httpd (pid: 10812, threadinfo=ee2f5000 task=f4f6b830) May 24 07:23:13 ancona kernel: Stack: f8aed2d3 20202020 32202020 20202020 20202020 32613720 36326365 f8b30018 May 24 07:23:13 ancona kernel: 00000246 e5fa9280 00000003 00000000 e5fa9280 f8ae8847 00000003 f8af0c80 May 24 07:23:13 ancona kernel: f8a53000 f8b48e9a 00000008 00000001 e78e546c e78e5450 f8a53000 f8b3ea9a
May 24 07:23:13 ancona kernel: Call Trace:
May 24 07:23:13 ancona kernel: [<f8b30018>] gfs_acl_validate_set+0x18/0x8d [gfs] May 24 07:23:13 ancona kernel: [<f8ae8847>] lm_dlm_lock+0x49/0x52 [lock_dlm]
May 24 07:23:13 ancona kernel:  [<f8b48e9a>] gfs_lm_lock+0x35/0x4d [gfs]
May 24 07:23:13 ancona kernel: [<f8b3ea9a>] gfs_glock_xmote_th+0x130/0x172 [gfs]
May 24 07:23:13 ancona kernel:  [<f8b3e159>] rq_promote+0xc8/0x147 [gfs]
May 24 07:23:13 ancona kernel:  [<f8b3e345>] run_queue+0x91/0xc1 [gfs]
May 24 07:23:13 ancona kernel:  [<f8b3f355>] gfs_glock_nq+0xcf/0x116 [gfs]
May 24 07:23:13 ancona kernel: [<f8b3f92b>] gfs_glock_nq_init+0x13/0x26 [gfs]
May 24 07:23:13 ancona kernel:  [<f8b42f44>] gfs_lookupi+0x321/0x3bf [gfs]
May 24 07:23:13 ancona kernel:  [<f8b564c8>] gfs_lookup+0x83/0xfb [gfs]


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]