[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [Linux-cluster] kernel panic - help!



David Teigland wrote:
On Wed, Jun 21, 2006 at 03:41:58PM -0300, German Staltari wrote:
Jun 21 14:59:17 qmail-be-04 kernel: CMAN: removing node qmail-be-02 from the cluster : Missed too many heartbeats Jun 21 14:59:23 qmail-be-04 kernel: CMAN: removing node qmail-be-01 from the cluster : No response to messages Jun 21 14:59:29 qmail-be-04 kernel: CMAN: removing node qmail-be-06 from the cluster : No response to messages Jun 21 14:59:39 qmail-be-04 kernel: CMAN: removing node qmail-be-03 from the cluster : No response to messages Jun 21 14:59:46 qmail-be-04 kernel: CMAN: removing node qmail-be-05 from the cluster : No response to messages
Jun 21 14:59:52 qmail-be-04 kernel: CMAN: quorum lost, blocking activity
Jun 21 14:59:52 qmail-be-04 kernel: CMAN: node qmail-be-04 has been removed from the cluster : No response to messages
Jun 21 14:59:52 qmail-be-04 kernel: CMAN: killed by NODEDOWN message
Jun 21 14:59:52 qmail-be-04 kernel: CMAN: we are leaving the cluster. No response to messages

This is what led to the gfs panic, the cluster shut down when it lost
contact with all the other nodes.

Dave

Ok, but this node lost contact with the cluster because all the other nodes get the same panic at the same time. We had another panic a few minutes ago... 3rd panic today... the same logs output...

Jun 21 16:13:55 qmail-be-01 kernel: lock_dlm: Assertion failed on line 357 of file /soft/kernel/cluster-1.02.00/gfs-kernel/src/dlm/lock.c
Jun 21 16:13:55 qmail-be-01 kernel: lock_dlm:  assertion:  "!error"
Jun 21 16:13:55 qmail-be-01 kernel: lock_dlm:  time = 951351
Jun 21 16:13:55 qmail-be-01 kernel: mstore008-004: error=-22 num=2,75c6db lkf=10000 flags=84
Jun 21 16:13:55 qmail-be-01 kernel:
Jun 21 16:13:55 qmail-be-01 kernel: ------------[ cut here ]------------
Jun 21 16:13:55 qmail-be-01 kernel: kernel BUG at /soft/kernel/cluster-1.02.00/gfs-kernel/src/dlm/lock.c:357!
Jun 21 16:13:55 qmail-be-01 kernel: invalid opcode: 0000 [#1]
Jun 21 16:13:55 qmail-be-01 kernel: SMP
Jun 21 16:13:55 qmail-be-01 kernel: Modules linked in: nfsd exportfs lockd nfs_acl sunrpc gfs lock_dlm lock_harness dlm cman dm_round_robin dm_multipath ipv6 ohci_hcd i2c_piix4 i2c_core e1000 sg ext3 jbd dm_mod qla2300 qla2xxx scsi_transport_fc mptspi mptscsih mptbase sd_mod scsi_mod
Jun 21 16:13:55 qmail-be-01 kernel: CPU:    6
Jun 21 16:13:55 qmail-be-01 kernel: EIP: 0060:[<f90254d8>] Tainted: GF VLI
Jun 21 16:13:55 qmail-be-01 kernel: EFLAGS: 00010296   (2.6.16.11-gds #1)
Jun 21 16:13:55 qmail-be-01 kernel: EIP is at do_dlm_unlock+0xd1/0xe5 [lock_dlm] Jun 21 16:13:55 qmail-be-01 kernel: eax: 00000004 ebx: 00000084 ecx: ffffebd8 edx: 00000000 Jun 21 16:13:55 qmail-be-01 kernel: esi: 00010000 edi: ffffffea ebp: ca4265c0 esp: d741eef4
Jun 21 16:13:56 qmail-be-01 kernel: ds: 007b   es: 007b   ss: 0068
Jun 21 16:13:56 qmail-be-01 kernel: Process gfs_glockd (pid: 1061, threadinfo=d741e000 task=d6b40550) Jun 21 16:13:56 qmail-be-01 kernel: Stack: <0>f902b673 f53267e0 ffffffea 00000002 0075c6db 00000000 00010000 00000084 Jun 21 16:13:56 qmail-be-01 kernel: 00000002 f9732000 00000003 ca4265c0 cec8a4ac f902552e f905e6b5 cec8a4dc Jun 21 16:13:56 qmail-be-01 kernel: cec8a4c8 cec8a4dc f9055f02 00000296 000000d0 f9732000 f9089ee0 c539f9c0
Jun 21 16:13:56 qmail-be-01 kernel: Call Trace:
Jun 21 16:13:56 qmail-be-01 kernel: [<f902552e>] lm_dlm_unlock+0x14/0x1c [lock_dlm] Jun 21 16:13:56 qmail-be-01 kernel: [<f905e6b5>] gfs_lm_unlock+0x2c/0x47 [gfs] Jun 21 16:13:56 qmail-be-01 kernel: [<f9055f02>] gfs_glock_drop_th+0x84/0x182 [gfs] Jun 21 16:13:56 qmail-be-01 kernel: [<f9054817>] run_queue+0x348/0x374 [gfs] Jun 21 16:13:56 qmail-be-01 kernel: [<f90541a4>] handle_callback+0xe6/0x120 [gfs] Jun 21 16:13:56 qmail-be-01 kernel: [<f905485e>] unlock_on_glock+0x1b/0x24 [gfs] Jun 21 16:13:56 qmail-be-01 kernel: [<f905441b>] gfs_reclaim_glock+0xbc/0x170 [gfs]
Jun 21 16:13:56 qmail-be-01 kernel:  [<c031db3e>] _spin_lock_irqsave+0x9/0xd
Jun 21 16:13:56 qmail-be-01 kernel:  [<f9047bca>] gfs_glockd+0xda/0xff [gfs]


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]