[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [Linux-cluster] kernel panic - help!



David Teigland wrote:
Any messages before this?  The best you could hope for with a SAN failure
is that all the cluster nodes withdraw gfs, allowing you to reboot them
without the panic.  So, the end result wouldn't be all that different than
the panics.
This is the log just before the panic:

Jun 21 14:59:17 qmail-be-04 kernel: CMAN: removing node qmail-be-02 from the cluster : Missed too many heartbeats Jun 21 14:59:23 qmail-be-04 kernel: CMAN: removing node qmail-be-01 from the cluster : No response to messages Jun 21 14:59:29 qmail-be-04 kernel: CMAN: removing node qmail-be-06 from the cluster : No response to messages Jun 21 14:59:39 qmail-be-04 kernel: CMAN: removing node qmail-be-03 from the cluster : No response to messages Jun 21 14:59:46 qmail-be-04 kernel: CMAN: removing node qmail-be-05 from the cluster : No response to messages
Jun 21 14:59:52 qmail-be-04 kernel: CMAN: quorum lost, blocking activity
Jun 21 14:59:52 qmail-be-04 kernel: CMAN: node qmail-be-04 has been removed from the cluster : No response to messages
Jun 21 14:59:52 qmail-be-04 kernel: CMAN: killed by NODEDOWN message
Jun 21 14:59:52 qmail-be-04 kernel: CMAN: we are leaving the cluster. No response to messages
Jun 21 14:59:52 qmail-be-04 kernel: WARNING: dlm_emergency_shutdown
Jun 21 14:59:52 qmail-be-04 fenced[17897]: process_events: service get event failed Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid lockspace 1000041 from 3 req 1 Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid lockspace 100003b from 3 req 3 Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid lockspace 100003b from 3 req 9 Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid lockspace 100003b from 3 req 9 Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid lockspace 100003b from 3 req 3 Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid lockspace 100003f from 3 req 3 Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid lockspace 100003f from 3 req 9 Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid lockspace 100003f from 3 req 3 Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid lockspace 100003f from 3 req 3 Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid lockspace 100003f from 3 req 9 Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid lockspace 100003f from 3 req 3 Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid lockspace 100003f from 3 req 9 Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid lockspace 100003f from 3 req 3
Jun 21 14:59:53 qmail-be-04 last message repeated 7 times
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid lockspace 100003d from 3 req 3
Jun 21 14:59:53 qmail-be-04 last message repeated 6 times
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid lockspace 1000041 from 3 req 3 Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid lockspace 100003b from 3 req 3 Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid lockspace 100003b from 3 req 3 Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid lockspace 100003f from 3 req 1 Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid lockspace 1000041 from 3 req 9 Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid lockspace 100003f from 3 req 9 Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid lockspace 1000041 from 3 req 1 Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid lockspace 100003f from 3 req 1 Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid lockspace 100003b from 3 req 1 Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid lockspace 100003b from 3 req 5
Jun 21 14:59:53 qmail-be-04 last message repeated 5 times
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid lockspace 100003d from 3 req 5 Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid lockspace 100003f from 3 req 5 Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid lockspace 100003d from 3 req 9 Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid lockspace 100003d from 3 req 5
Jun 21 14:59:53 qmail-be-04 last message repeated 3 times
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid lockspace 100003d from 3 req 9 Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid lockspace 100003d from 3 req 5
Jun 21 14:59:54 qmail-be-04 last message repeated 20 times
Jun 21 14:59:54 qmail-be-04 kernel: dlm: dlm_unlock: lkid 3b013d lockspace not found
Jun 21 14:59:54 qmail-be-04 kernel: store004-003 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore004-003 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore004-003 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore002-004 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore004-001 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore004-001 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore003-003 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore001-001 add_to_requestq cmd 3 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore003-003 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore001-003 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore004-001 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore004-004 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore002-003 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore003-004 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore004-001 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore002-002 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore004-003 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore002-004 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore002-002 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore001-003 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore002-003 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore003-002 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore004-004 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore004-001 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore003-003 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel:  type 2 event 282 flags 21a
Jun 21 14:59:54 qmail-be-04 kernel: 28975 pr_start 282 done 1
Jun 21 14:59:54 qmail-be-04 kernel: 28975 pr_finish flags 1a
Jun 21 14:59:54 qmail-be-04 kernel: 28956 pr_start last_stop 273 last_start 283 last_finish 273 Jun 21 14:59:54 qmail-be-04 kernel: 28956 pr_start count 5 type 2 event 283 flags 21a
Jun 21 14:59:54 qmail-be-04 kernel: 28956 pr_start 283 done 1
Jun 21 14:59:54 qmail-be-04 kernel: 28956 pr_finish flags 1a
Jun 21 14:59:54 qmail-be-04 kernel: 28957 pr_start last_stop 283 last_start 285 last_finish 283 Jun 21 14:59:54 qmail-be-04 kernel: 28957 pr_start count 6 type 2 event 285 flags 21a
Jun 21 14:59:54 qmail-be-04 kernel: 28957 pr_start 285 done 1
Jun 21 14:59:54 qmail-be-04 kernel: 28956 pr_finish flags 1a
Jun 21 14:59:54 qmail-be-04 kernel: 28992 pr_start last_stop 116 last_start 287 last_finish 116 Jun 21 14:59:54 qmail-be-04 kernel: 28992 pr_start count 4 type 2 event 287 flags 21a
Jun 21 14:59:54 qmail-be-04 kernel: 28992 pr_start 287 done 1
Jun 21 14:59:54 qmail-be-04 kernel: 28992 pr_finish flags 1a
Jun 21 14:59:55 qmail-be-04 kernel: 28975 rereq 2,1cec36 id e029f 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28975 pr_start last_stop 282 last_start 289 last_finish 282 Jun 21 14:59:55 qmail-be-04 kernel: 28975 pr_start count 5 type 2 event 289 flags 21a
Jun 21 14:59:55 qmail-be-04 kernel: 28975 pr_start 289 done 1
Jun 21 14:59:55 qmail-be-04 kernel: 28975 pr_finish flags 1a
Jun 21 14:59:55 qmail-be-04 kernel: 28992 pr_start last_stop 287 last_start 291 last_finish 287 Jun 21 14:59:55 qmail-be-04 kernel: 28992 pr_start count 5 type 2 event 291 flags 21a
Jun 21 14:59:55 qmail-be-04 kernel: 28992 pr_start 291 done 1
Jun 21 14:59:55 qmail-be-04 kernel: 28992 pr_finish flags 1a
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 2,2fe4b id a001e 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 2,3fd13 id 7007a 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 2,6faaf id 90009 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 5,2fdd6 id c0135 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 5,4fc8f id c023b 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 2,34a id 8011d 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 5,2fe4b id b03c3 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 5,6faaf id b001e 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 5,34a id 11000e 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 2,6faa8 id f0016 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 5,6faa8 id 1001a8 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 2,5fbd9 id f00e9 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 2,5fb4d id 802ac 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 5,5fbd9 id f0026 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 5,3fd13 id c009b 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 2,2fdd6 id 8001d 3,0
Jun 21 14:59:56 qmail-be-04 kernel: 28974 rereq 2,4fc8f id c0367 3,0
Jun 21 14:59:56 qmail-be-04 kernel: 28974 rereq 2,2fe40 id a01fd 3,0
Jun 21 14:59:56 qmail-be-04 kernel: 28974 pr_start last_stop 289 last_start 293 last_finish 289 Jun 21 14:59:56 qmail-be-04 kernel: 28974 pr_start count 6 type 2 event 293 flags 21a
Jun 21 14:59:56 qmail-be-04 kernel: 28975 rereq 5,7fa97 id 1502b3 3,0
Jun 21 14:59:56 qmail-be-04 kernel: 28975 rereq 2,3fcea id 702f3 3,0
Jun 21 14:59:56 qmail-be-04 kernel: 28975 rereq 2,5fc1e id 6015f 3,0
Jun 21 14:59:56 qmail-be-04 kernel: 28975 rereq 5,5fc1e id c01c1 3,0
Jun 21 14:59:56 qmail-be-04 kernel: 28975 rereq 2,2fdfa id c0362 3,0
Jun 21 14:59:56 qmail-be-04 kernel: 28975 rereq 5,2fdfa id c02ad 3,0
Jun 21 14:59:56 qmail-be-04 kernel: 28975 rereq 5,5fb4d id f01ce 3,0
Jun 21 14:59:56 qmail-be-04 kernel: 28975 rereq 2,7fa97 id d0293 3,0
Jun 21 14:59:56 qmail-be-04 kernel: 28975 rereq 5,3fcea id d02ad 3,0
Jun 21 14:59:56 qmail-be-04 kernel: 28974 pr_start 293 done 1
Jun 21 14:59:56 qmail-be-04 kernel: 28974 pr_finish flags 1a
Jun 21 14:59:56 qmail-be-04 kernel: 29174 pr_start last_stop 118 last_start 295 last_finish 118 Jun 21 14:59:56 qmail-be-04 kernel: 29174 pr_start count 4 type 2 event 295 flags 21a
Jun 21 14:59:56 qmail-be-04 kernel: 29174 pr_start 295 done 1
Jun 21 14:59:56 qmail-be-04 kernel: 29174 pr_finish flags 1a
Jun 21 14:59:56 qmail-be-04 kernel: 28991 pr_start last_stop 291 last_start 297 last_finish 291 Jun 21 14:59:56 qmail-be-04 kernel: 28991 pr_start count 6 type 2 event 297 flags 21a
Jun 21 14:59:56 qmail-be-04 kernel: 28991 pr_start 297 done 1
Jun 21 14:59:56 qmail-be-04 kernel: 28991 pr_finish flags 1a
Jun 21 14:59:56 qmail-be-04 kernel: 29174 pr_start last_stop 295 last_start 299 last_finish 295 Jun 21 14:59:57 qmail-be-04 kernel: 29174 pr_start count 5 type 2 event 299 flags 21a
Jun 21 14:59:57 qmail-be-04 kernel: 29174 pr_start 299 done 1
Jun 21 14:59:57 qmail-be-04 kernel: 29174 pr_finish flags 1a
Jun 21 14:59:57 qmail-be-04 kernel: 29175 pr_start last_stop 299 last_start 301 last_finish 299 Jun 21 14:59:57 qmail-be-04 kernel: 29175 pr_start count 6 type 2 event 301 flags 21a
Jun 21 14:59:57 qmail-be-04 kernel: 29175 pr_start 301 done 1
Jun 21 14:59:57 qmail-be-04 kernel: 29175 pr_finish flags 1a
Jun 21 14:59:57 qmail-be-04 kernel: 29192 pr_start last_stop 120 last_start 303 last_finish 120 Jun 21 14:59:57 qmail-be-04 kernel: 29192 pr_start count 4 type 2 event 303 flags 21a
Jun 21 14:59:57 qmail-be-04 kernel: 29192 pr_start 303 done 1
Jun 21 14:59:57 qmail-be-04 kernel: 29192 pr_finish flags 1a
Jun 21 14:59:57 qmail-be-04 kernel: 29408 pr_start last_stop 122 last_start 305 last_finish 122 Jun 21 14:59:57 qmail-be-04 kernel: 29408 pr_start count 4 type 2 event 305 flags 21a
Jun 21 14:59:57 qmail-be-04 kernel: 29408 pr_start 305 done 1
Jun 21 14:59:57 qmail-be-04 kernel: 29408 pr_finish flags 1a
Jun 21 14:59:57 qmail-be-04 kernel: 29458 pr_start last_stop 124 last_start 308 last_finish 124 Jun 21 14:59:57 qmail-be-04 kernel: 29458 pr_start count 4 type 2 event 308 flags 21a
Jun 21 14:59:57 qmail-be-04 kernel: 29458 pr_start 308 done 1
Jun 21 14:59:57 qmail-be-04 kernel: 29457 pr_finish flags 1a
Jun 21 14:59:57 qmail-be-04 kernel: 29191 pr_start last_stop 303 last_start 309 last_finish 303 Jun 21 14:59:57 qmail-be-04 kernel: 29191 pr_start count 5 type 2 event 309 flags 21a
Jun 21 14:59:57 qmail-be-04 kernel: 29191 pr_start 309 done 1
Jun 21 14:59:57 qmail-be-04 kernel: 29192 pr_finish flags 1a
Jun 21 14:59:57 qmail-be-04 kernel: 29408 pr_start last_stop 305 last_start 311 last_finish 305 Jun 21 14:59:57 qmail-be-04 kernel: 29408 pr_start count 5 type 2 event 311 flags 21a
Jun 21 14:59:57 qmail-be-04 kernel: 29408 pr_start 311 done 1
Jun 21 14:59:57 qmail-be-04 kernel: 29408 pr_finish flags 1a
Jun 21 14:59:57 qmail-be-04 kernel: 29192 pr_start last_stop 309 last_start 313 last_finish 309 Jun 21 14:59:57 qmail-be-04 kernel: 29192 pr_start count 6 type 2 event 313 flags 21a
Jun 21 14:59:57 qmail-be-04 kernel: 29192 pr_start 313 done 1
Jun 21 14:59:57 qmail-be-04 kernel: 29192 pr_finish flags 1a
Jun 21 14:59:58 qmail-be-04 kernel: 29458 pr_start last_stop 308 last_start 315 last_finish 308 Jun 21 14:59:58 qmail-be-04 kernel: 29458 pr_start count 5 type 2 event 315 flags 21a
Jun 21 14:59:58 qmail-be-04 kernel: 29458 pr_start 315 done 1
Jun 21 14:59:58 qmail-be-04 kernel: 29458 pr_finish flags 1a
Jun 21 14:59:58 qmail-be-04 kernel: 29409 pr_start last_stop 311 last_start 317 last_finish 311 Jun 21 14:59:58 qmail-be-04 kernel: 29409 pr_start count 6 type 2 event 317 flags 21a
Jun 21 14:59:58 qmail-be-04 kernel: 29409 pr_start 317 done 1
Jun 21 14:59:58 qmail-be-04 kernel: 29408 pr_finish flags 1a
Jun 21 14:59:58 qmail-be-04 kernel: 29458 pr_start last_stop 315 last_start 319 last_finish 315 Jun 21 14:59:58 qmail-be-04 kernel: 29458 pr_start count 6 type 2 event 319 flags 21a
Jun 21 14:59:58 qmail-be-04 kernel: 29457 rereq 2,2bd7b5 id 801e5 3,0
Jun 21 14:59:58 qmail-be-04 kernel: 29458 pr_start 319 done 1
Jun 21 14:59:58 qmail-be-04 kernel: 29457 pr_finish flags 1a
Jun 21 14:59:58 qmail-be-04 kernel:
Jun 21 14:59:58 qmail-be-04 kernel: lock_dlm: Assertion failed on line 357 of file /soft/kernel/cluster-1.02.00/gfs-kernel/src/dlm/lock.c
Jun 21 14:59:58 qmail-be-04 kernel: lock_dlm:  assertion:  "!error"
Jun 21 14:59:58 qmail-be-04 kernel: lock_dlm:  time = 2512697
Jun 21 14:59:58 qmail-be-04 kernel: mstore008-002: error=-22 num=2,7798a8 lkf=10001 flags=84

Is the second panic today :(
Thanks again
German


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]