[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

[Linux-cluster] <err> #48: Unable to obtain cluster lock: Invalid argument



Hi.

After running a cluster node in a production cluster since July, I got
the folllowing error:

<err> #48: Unable to obtain cluster lock: Invalid argument 

Which resulted in a reboot:

--clip--
Dec 27 02:50:31 pcn1 clurgmgrd[6217]: <err> #48: Unable to obtain
cluster lock: Invalid argument 
Dec 27 02:50:31 pcn1 clurgmgrd[6217]: <notice> Stopping service
service:p01 
Dec 27 02:50:34 pcn1 in.rdiscd[30325]: setsockopt (IP_ADD_MEMBERSHIP):
Address already in use
Dec 27 02:50:34 pcn1 in.rdiscd[30325]: Failed joining addresses 
Dec 27 02:50:38 pcn1 snmpd[15929]: error on subcontainer 'ia_addr'
insert (-1) 
Dec 27 02:50:38 pcn1 snmpd[15929]: error on subcontainer 'ia_addr'
insert (-1) 
Dec 27 02:50:38 pcn1 snmpd[15929]: error on subcontainer '' insert (-1) 
Dec 27 02:50:38 pcn1 snmpd[15929]: error on subcontainer '' insert (-1) 
Dec 27 02:50:45 pcn1 clurgmgrd[6217]: <notice> Service service:p01 is
recovering 
Dec 27 02:50:45 pcn1 clurgmgrd[6217]: <notice> Recovering failed service
service:p01 
Dec 27 02:50:45 pcn1 kernel: dlm: add_to_waiters error 1
Dec 27 02:50:45 pcn1 kernel: dlm: remove_from_waiters error
Dec 27 02:50:45 pcn1 kernel: dlm: rgmanager: receive_unlock_reply not on
waiters
Dec 27 02:50:45 pcn1 clurgmgrd[6216]: <crit> Watchdog: Daemon died,
rebooting... 
Dec 27 02:50:45 pcn1 kernel: md: stopping all md devices.
Dec 27 02:55:23 pcn1 syslogd 1.4.1: restart.
--clip--

Other members of the cluster noticed the missing member, fenced it,
failed services over, and back (when the missing node had rejoined):

--clip--
Dec 27 02:50:56 pcn2 openais[4588]: [TOTEM] The token was lost in the OPERATIONAL state. 
Dec 27 02:50:56 pcn2 openais[4588]: [TOTEM] Receive multicast socket recv buffer size (262142 bytes). 
Dec 27 02:50:56 pcn2 openais[4588]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
Dec 27 02:50:56 pcn2 openais[4588]: [TOTEM] entering GATHER state from 2. 
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] entering GATHER state from 11. 
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] Saving state aru 6a4 high seq received 6a4 
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] entering COMMIT state. 
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] entering RECOVERY state. 
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] position [0] member 10.3.0.10: 
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] previous ring seq 324 rep 10.3.0.10 
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] aru 6a4 high delivered 6a4 received flag 0 
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] position [1] member 10.3.0.12: 
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] previous ring seq 324 rep 10.3.0.10 
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] aru 6a4 high delivered 6a4 received flag 0 
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] position [2] member 10.3.0.13: 
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] previous ring seq 324 rep 10.3.0.10 
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] aru 6a4 high delivered 6a4 received flag 0 
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] position [3] member 10.3.0.14: 
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] previous ring seq 324 rep 10.3.0.10 
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] aru 6a4 high delivered 6a4 received flag 0 
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] position [4] member 10.3.0.15: 
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] previous ring seq 324 rep 10.3.0.10 
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] aru 6a4 high delivered 6a4 received flag 1 
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] position [5] member 10.3.0.16: 
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] previous ring seq 324 rep 10.3.0.10 
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] aru 6a4 high delivered 6a4 received flag 1 
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] Did not need to originate any messages in recovery. 
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] Storing new sequence id for ring 14c 
Dec 27 02:51:01 pcn2 openais[4588]: [CLM  ] CLM CONFIGURATION CHANGE 
Dec 27 02:51:01 pcn2 openais[4588]: [CLM  ] New Configuration: 
Dec 27 02:51:01 pcn2 openais[4588]: [CLM  ]     r(0) ip(10.3.0.10)  
Dec 27 02:51:01 pcn2 openais[4588]: [CLM  ]     r(0) ip(10.3.0.12)  
Dec 27 02:51:01 pcn2 openais[4588]: [CLM  ]     r(0) ip(10.3.0.13)  
Dec 27 02:51:01 pcn2 openais[4588]: [CLM  ]     r(0) ip(10.3.0.14)  
Dec 27 02:51:01 pcn2 openais[4588]: [CLM  ]     r(0) ip(10.3.0.15)  
Dec 27 02:51:01 pcn2 openais[4588]: [CLM  ]     r(0) ip(10.3.0.16)  
Dec 27 02:51:01 pcn2 openais[4588]: [CLM  ] Members Left: 
Dec 27 02:51:01 pcn2 openais[4588]: [CLM  ]     r(0) ip(10.3.0.11)  
Dec 27 02:51:01 pcn2 openais[4588]: [CLM  ] Members Joined: 
Dec 27 02:51:01 pcn2 openais[4588]: [SYNC ] This node is within the primary component and will provide se
rvice. 
Dec 27 02:51:01 pcn2 openais[4588]: [CLM  ] CLM CONFIGURATION CHANGE 
Dec 27 02:51:01 pcn2 openais[4588]: [CLM  ] New Configuration: 
Dec 27 02:51:01 pcn2 openais[4588]: [CLM  ]     r(0) ip(10.3.0.10)  
Dec 27 02:51:01 pcn2 openais[4588]: [CLM  ]     r(0) ip(10.3.0.12)  
Dec 27 02:51:01 pcn2 openais[4588]: [CLM  ]     r(0) ip(10.3.0.13)  
Dec 27 02:51:01 pcn2 openais[4588]: [CLM  ]     r(0) ip(10.3.0.14)  
Dec 27 02:51:01 pcn2 openais[4588]: [CLM  ]     r(0) ip(10.3.0.15)  
Dec 27 02:51:01 pcn2 openais[4588]: [CLM  ]     r(0) ip(10.3.0.16)  
Dec 27 02:51:01 pcn2 openais[4588]: [CLM  ] Members Left: 
Dec 27 02:51:01 pcn2 openais[4588]: [CLM  ] Members Joined: 
Dec 27 02:51:01 pcn2 openais[4588]: [SYNC ] This node is within the primary component and will provide se
rvice. 
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] entering OPERATIONAL state. 
Dec 27 02:51:01 pcn2 openais[4588]: [CLM  ] got nodejoin message 10.3.0.10 
Dec 27 02:51:01 pcn2 openais[4588]: [CLM  ] got nodejoin message 10.3.0.12 
Dec 27 02:51:01 pcn2 openais[4588]: [CLM  ] got nodejoin message 10.3.0.13 
Dec 27 02:51:01 pcn2 openais[4588]: [CLM  ] got nodejoin message 10.3.0.14 
Dec 27 02:51:01 pcn2 openais[4588]: [CLM  ] got nodejoin message 10.3.0.15 
Dec 27 02:51:01 pcn2 openais[4588]: [CLM  ] got nodejoin message 10.3.0.16 
Dec 27 02:51:01 pcn2 openais[4588]: [CPG  ] got joinlist message from node 3 
Dec 27 02:51:01 pcn2 openais[4588]: [CPG  ] got joinlist message from node 4 
Dec 27 02:51:01 pcn2 openais[4588]: [CPG  ] got joinlist message from node 5 
Dec 27 02:51:01 pcn2 openais[4588]: [CPG  ] got joinlist message from node 6 
Dec 27 02:51:01 pcn2 openais[4588]: [CPG  ] got joinlist message from node 100 
Dec 27 02:51:01 pcn2 openais[4588]: [CPG  ] got joinlist message from node 2 
Dec 27 02:51:01 pcn2 kernel: dlm: closing connection to node 1
Dec 27 02:51:01 pcn2 fenced[4614]: pcn1-hb not a cluster member after 0 sec post_fail_delay
Dec 27 02:51:01 pcn2 fenced[4614]: fencing node "pcn1-hb"
Dec 27 02:52:13 pcn2 fenced[4614]: fence "pcn1-hb" success
Dec 27 02:52:18 pcn2 ccsd[4541]: Attempt to close an unopened CCS descriptor (799075500). 
Dec 27 02:52:18 pcn2 ccsd[4541]: Error while processing disconnect: Invalid request descriptor 
Dec 27 02:52:20 pcn2 clurgmgrd[6262]: <notice> Taking over service service:p01 from down member pcn1-hb 
Dec 27 02:52:20 pcn2 clurgmgrd[6262]: <notice> Taking over service service:i01 from down member pcn1-hb 
Dec 27 02:52:20 pcn2 kernel: kjournald starting.  Commit interval 5 seconds
Dec 27 02:52:20 pcn2 kernel: EXT3 FS on dm-65, internal journal
Dec 27 02:52:20 pcn2 kernel: EXT3-fs: mounted filesystem with ordered data mode.
Dec 27 02:52:21 pcn2 clurgmgrd[6262]: <notice> Taking over service service:i13 from down member pcn1-hb 
Dec 27 02:52:21 pcn2 in.rdiscd[2158]: setsockopt (IP_ADD_MEMBERSHIP): Address already in use
Dec 27 02:52:21 pcn2 in.rdiscd[2158]: Failed joining addresses 
Dec 27 02:52:22 pcn2 kernel: kjournald starting.  Commit interval 5 seconds
Dec 27 02:52:22 pcn2 kernel: EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
Dec 27 02:52:22 pcn2 kernel: EXT3 FS on dm-14, internal journal
Dec 27 02:52:22 pcn2 kernel: EXT3-fs: recovery complete.
Dec 27 02:52:22 pcn2 kernel: EXT3-fs: mounted filesystem with ordered data mode.
Dec 27 02:52:24 pcn2 clurgmgrd[6262]: <notice> Service service:p01 started 
Dec 27 02:52:25 pcn2 last message repeated 2 times
Dec 27 02:52:27 pcn2 kernel: kjournald starting.  Commit interval 5 seconds
Dec 27 02:52:27 pcn2 kernel: EXT3 FS on dm-2, internal journal
Dec 27 02:52:27 pcn2 kernel: EXT3-fs: dm-2: 3 orphan inodes deleted
Dec 27 02:52:27 pcn2 kernel: EXT3-fs: recovery complete.
Dec 27 02:52:27 pcn2 kernel: EXT3-fs: mounted filesystem with ordered data mode.
Dec 27 02:52:29 pcn2 kernel: kjournald starting.  Commit interval 5 seconds
Dec 27 02:52:29 pcn2 kernel: EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
Dec 27 02:52:29 pcn2 kernel: EXT3 FS on dm-38, internal journal
Dec 27 02:52:29 pcn2 kernel: EXT3-fs: recovery complete.
Dec 27 02:52:29 pcn2 kernel: EXT3-fs: mounted filesystem with ordered data mode.
Dec 27 02:52:30 pcn2 in.rdiscd[3313]: setsockopt (IP_ADD_MEMBERSHIP): Address already in use
Dec 27 02:52:30 pcn2 in.rdiscd[3313]: Failed joining addresses 
Dec 27 02:52:32 pcn2 clurgmgrd[6262]: <notice> Service service:i13 started 
Dec 27 02:52:35 pcn2 kernel: kjournald starting.  Commit interval 5 seconds
Dec 27 02:52:35 pcn2 kernel: EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
Dec 27 02:52:35 pcn2 kernel: EXT3 FS on dm-26, internal journal
Dec 27 02:52:35 pcn2 kernel: EXT3-fs: recovery complete.
Dec 27 02:52:35 pcn2 kernel: EXT3-fs: mounted filesystem with ordered data mode.
Dec 27 02:52:37 pcn2 in.rdiscd[3833]: setsockopt (IP_ADD_MEMBERSHIP): Address already in use
Dec 27 02:52:37 pcn2 in.rdiscd[3833]: Failed joining addresses 
Dec 27 02:52:38 pcn2 clurgmgrd[6262]: <notice> Service service:i01 started 
Dec 27 02:53:25 pcn2 last message repeated 2 times
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] entering GATHER state from 11. 
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] Saving state aru c8 high seq received c8 
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] entering COMMIT state. 
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] entering RECOVERY state. 
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] position [0] member 10.3.0.10: 
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] previous ring seq 332 rep 10.3.0.10 
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] aru c8 high delivered c8 received flag 0 
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] position [1] member 10.3.0.11: 
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] previous ring seq 288 rep 10.3.0.11 
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] aru 9 high delivered 9 received flag 0 
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] position [2] member 10.3.0.12: 
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] previous ring seq 332 rep 10.3.0.10 
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] aru c8 high delivered c8 received flag 0 
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] position [3] member 10.3.0.13: 
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] previous ring seq 332 rep 10.3.0.10 
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] aru c8 high delivered c8 received flag 0 
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] position [4] member 10.3.0.14: 
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] previous ring seq 332 rep 10.3.0.10 
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] aru c8 high delivered c8 received flag 0 
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] position [5] member 10.3.0.15: 
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] previous ring seq 332 rep 10.3.0.10 
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] aru c8 high delivered c8 received flag 1 
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] position [6] member 10.3.0.16: 
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] previous ring seq 332 rep 10.3.0.10 
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] aru c8 high delivered c8 received flag 1 
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] Did not need to originate any messages in recovery. 
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] Storing new sequence id for ring 150 
Dec 27 02:55:26 pcn2 openais[4588]: [CLM  ] CLM CONFIGURATION CHANGE 
Dec 27 02:55:26 pcn2 openais[4588]: [CLM  ] New Configuration: 
Dec 27 02:55:26 pcn2 openais[4588]: [CLM  ]     r(0) ip(10.3.0.10)  
Dec 27 02:55:26 pcn2 openais[4588]: [CLM  ]     r(0) ip(10.3.0.12)  
Dec 27 02:55:26 pcn2 openais[4588]: [CLM  ]     r(0) ip(10.3.0.13)  
Dec 27 02:55:26 pcn2 openais[4588]: [CLM  ]     r(0) ip(10.3.0.14)  
Dec 27 02:55:26 pcn2 openais[4588]: [CLM  ]     r(0) ip(10.3.0.15)  
Dec 27 02:55:26 pcn2 openais[4588]: [CLM  ] Members Left: 
Dec 27 02:55:26 pcn2 openais[4588]: [CLM  ] Members Joined: 
Dec 27 02:55:26 pcn2 openais[4588]: [SYNC ] This node is within the primary component and will provide se
rvice. 
Dec 27 02:55:26 pcn2 openais[4588]: [CLM  ] CLM CONFIGURATION CHANGE 
Dec 27 02:55:26 pcn2 openais[4588]: [CLM  ] New Configuration: 
Dec 27 02:55:26 pcn2 openais[4588]: [CLM  ]     r(0) ip(10.3.0.10)  
Dec 27 02:55:26 pcn2 openais[4588]: [CLM  ]     r(0) ip(10.3.0.11)  
Dec 27 02:55:26 pcn2 openais[4588]: [CLM  ]     r(0) ip(10.3.0.12)  
Dec 27 02:55:26 pcn2 openais[4588]: [CLM  ]     r(0) ip(10.3.0.13)  
Dec 27 02:55:26 pcn2 openais[4588]: [CLM  ]     r(0) ip(10.3.0.14)  
Dec 27 02:55:26 pcn2 openais[4588]: [CLM  ]     r(0) ip(10.3.0.15)  
Dec 27 02:55:26 pcn2 openais[4588]: [CLM  ]     r(0) ip(10.3.0.16)  
Dec 27 02:55:26 pcn2 openais[4588]: [CLM  ] Members Left: 
Dec 27 02:55:26 pcn2 openais[4588]: [CLM  ] Members Joined: 
Dec 27 02:55:26 pcn2 openais[4588]: [CLM  ]     r(0) ip(10.3.0.11)  
Dec 27 02:55:26 pcn2 openais[4588]: [SYNC ] This node is within the primary component and will provide se
rvice. 
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] entering OPERATIONAL state. 
Dec 27 02:55:26 pcn2 openais[4588]: [CLM  ] got nodejoin message 10.3.0.10 
Dec 27 02:55:26 pcn2 openais[4588]: [CLM  ] got nodejoin message 10.3.0.11 
Dec 27 02:55:26 pcn2 openais[4588]: [CLM  ] got nodejoin message 10.3.0.12 
Dec 27 02:55:26 pcn2 openais[4588]: [CLM  ] got nodejoin message 10.3.0.13 
Dec 27 02:55:26 pcn2 openais[4588]: [CLM  ] got nodejoin message 10.3.0.14 
Dec 27 02:55:26 pcn2 openais[4588]: [CLM  ] got nodejoin message 10.3.0.15 
Dec 27 02:55:26 pcn2 openais[4588]: [CLM  ] got nodejoin message 10.3.0.16 
Dec 27 02:55:26 pcn2 openais[4588]: [CPG  ] got joinlist message from node 100 
Dec 27 02:55:26 pcn2 openais[4588]: [CPG  ] got joinlist message from node 2 
Dec 27 02:55:26 pcn2 openais[4588]: [CPG  ] got joinlist message from node 3 
Dec 27 02:55:26 pcn2 openais[4588]: [CPG  ] got joinlist message from node 4 
Dec 27 02:55:26 pcn2 openais[4588]: [CPG  ] got joinlist message from node 5 
Dec 27 02:55:26 pcn2 openais[4588]: [CPG  ] got joinlist message from node 6 
Dec 27 02:55:35 pcn2 kernel: dlm: connecting to 1
--clip--

--clip--
Dec 27 02:55:24 pcn1 ccsd[4132]: Starting ccsd 2.0.69: 
Dec 27 02:55:24 pcn1 ccsd[4132]:  Built: Jun 27 2007 15:21:32 
Dec 27 02:55:24 pcn1 ccsd[4132]:  Copyright (C) Red Hat, Inc.  2004  All rights reserved. 
Dec 27 02:55:24 pcn1 ccsd[4132]: cluster.conf (cluster name = mappi-primary, version = 109) found. 
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] AIS Executive Service RELEASE 'subrev 1324 version 0.80.2' 
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] Copyright (C) 2002-2006 MontaVista Software, Inc and contribu
tors. 
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] Copyright (C) 2006 Red Hat, Inc. 
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] AIS Executive Service: started and ready to provide service. 
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] Using default multicast address of 239.192.46.199 
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] openais component openais_cpg loaded. 
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] Registering service handler 'openais cluster closed process g
roup service v1.01' 
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] openais component openais_cfg loaded. 
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] Registering service handler 'openais configuration service' 
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] openais component openais_msg loaded. 
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] Registering service handler 'openais message service B.01.01'
 
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] openais component openais_lck loaded. 
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] Registering service handler 'openais distributed locking serv
ice B.01.01' 
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] openais component openais_evt loaded. 
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] Registering service handler 'openais event service B.01.01' 
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] openais component openais_ckpt loaded. 
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] Registering service handler 'openais checkpoint service B.01.
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] openais component openais_amf loaded. 
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] Registering service handler 'openais availability management 
framework B.01.01' 
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] openais component openais_clm loaded. 
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] Registering service handler 'openais cluster membership servi
ce B.01.01' 
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] openais component openais_evs loaded. 
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] Registering service handler 'openais extended virtual synchro
ny service' 
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] openais component openais_cman loaded. 
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] Registering service handler 'openais CMAN membership service 
2.01' 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] Token Timeout (10000 ms) retransmit timeout (495 ms) 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] token hold (386 ms) retransmits before loss (20 retrans) 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] join (60 ms) send_join (0 ms) consensus (4800 ms) merge (200 
ms) 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] downcheck (1000 ms) fail to recv const (50 msgs) 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] seqno unchanged const (30 rotations) Maximum network MTU 1500
 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] window size per rotation (50 messages) maximum messages per r
otation (17 messages) 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] send threads (0 threads) 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] RRP token expired timeout (495 ms) 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] RRP token problem counter (2000 ms) 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] RRP threshold (10 problem count) 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] RRP mode set to none. 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] heartbeat_failures_allowed (0) 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] max_network_delay (50 ms) 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] HeartBeat is Disabled. To enable set heartbeat_failures_allow
ed > 0 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] Receive multicast socket recv buffer size (262142 bytes). 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] The network interface [10.3.0.11] is now up. 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] Created or loaded sequence id 284.10.3.0.11 for this ring. 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] entering GATHER state from 15. 
Dec 27 02:55:26 pcn1 openais[4143]: [SERV ] Initialising service handler 'openais extended virtual synchr
ony service' 
Dec 27 02:55:26 pcn1 openais[4143]: [SERV ] Initialising service handler 'openais cluster membership serv
ice B.01.01' 
Dec 27 02:55:26 pcn1 openais[4143]: [SERV ] Initialising service handler 'openais availability management
 framework B.01.01' 
Dec 27 02:55:26 pcn1 openais[4143]: [SERV ] Initialising service handler 'openais checkpoint service B.01
.01' 
Dec 27 02:55:26 pcn1 openais[4143]: [SERV ] Initialising service handler 'openais event service B.01.01' 
Dec 27 02:55:26 pcn1 openais[4143]: [SERV ] Initialising service handler 'openais distributed locking ser
vice B.01.01' 
Dec 27 02:55:26 pcn1 openais[4143]: [SERV ] Initialising service handler 'openais message service B.01.01
' 
Dec 27 02:55:26 pcn1 openais[4143]: [SERV ] Initialising service handler 'openais configuration service' 
Dec 27 02:55:26 pcn1 openais[4143]: [SERV ] Initialising service handler 'openais cluster closed process 
group service v1.01' 
Dec 27 02:55:26 pcn1 openais[4143]: [SERV ] Initialising service handler 'openais CMAN membership service
 2.01' 
Dec 27 02:55:26 pcn1 openais[4143]: [CMAN ] CMAN 2.0.69 (built Jun 27 2007 15:21:36) started 
Dec 27 02:55:26 pcn1 openais[4143]: [SYNC ] Not using a virtual synchrony filter. 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] Creating commit token because I am the rep. 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] Saving state aru 0 high seq received 0 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] entering COMMIT state. 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] entering RECOVERY state. 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] position [0] member 10.3.0.11: 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] previous ring seq 284 rep 10.3.0.11 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] aru 0 high delivered 0 received flag 0 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] Did not need to originate any messages in recovery. 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] Storing new sequence id for ring 120 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] Sending initial ORF token 
Dec 27 02:55:26 pcn1 openais[4143]: [CLM  ] CLM CONFIGURATION CHANGE 
Dec 27 02:55:26 pcn1 openais[4143]: [CLM  ] New Configuration: 
Dec 27 02:55:26 pcn1 openais[4143]: [CLM  ] Members Left: 
Dec 27 02:55:26 pcn1 openais[4143]: [CLM  ] Members Joined: 
Dec 27 02:55:26 pcn1 openais[4143]: [SYNC ] This node is within the primary component and will provide se
rvice. 
Dec 27 02:55:26 pcn1 openais[4143]: [CLM  ] CLM CONFIGURATION CHANGE 
Dec 27 02:55:26 pcn1 openais[4143]: [CLM  ] New Configuration: 
Dec 27 02:55:26 pcn1 openais[4143]: [CLM  ]     r(0) ip(10.3.0.11)  
Dec 27 02:55:26 pcn1 openais[4143]: [CLM  ] Members Left: 
Dec 27 02:55:26 pcn1 openais[4143]: [CLM  ] Members Joined: 
Dec 27 02:55:26 pcn1 openais[4143]: [CLM  ]     r(0) ip(10.3.0.11)  
Dec 27 02:55:26 pcn1 openais[4143]: [SYNC ] This node is within the primary component and will provide se
rvice. 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] entering OPERATIONAL state. 
Dec 27 02:55:26 pcn1 openais[4143]: [CLM  ] got nodejoin message 10.3.0.11 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] entering GATHER state from 11. 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] Saving state aru 9 high seq received 9 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] entering COMMIT state. 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] entering RECOVERY state. 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] position [0] member 10.3.0.10: 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] previous ring seq 332 rep 10.3.0.10 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] aru c8 high delivered c8 received flag 0 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] position [1] member 10.3.0.11: 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] previous ring seq 288 rep 10.3.0.11 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] aru 9 high delivered 9 received flag 0 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] position [2] member 10.3.0.12: 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] previous ring seq 332 rep 10.3.0.10 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] aru c8 high delivered c8 received flag 0 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] position [3] member 10.3.0.13: 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] previous ring seq 332 rep 10.3.0.10 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] aru c8 high delivered c8 received flag 0 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] position [4] member 10.3.0.14: 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] previous ring seq 332 rep 10.3.0.10 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] aru c8 high delivered c8 received flag 0 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] position [5] member 10.3.0.15: 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] previous ring seq 332 rep 10.3.0.10 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] aru c8 high delivered c8 received flag 1 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] position [6] member 10.3.0.16: 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] previous ring seq 332 rep 10.3.0.10 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] aru c8 high delivered c8 received flag 1 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] Did not need to originate any messages in recovery. 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] Storing new sequence id for ring 150 
Dec 27 02:55:26 pcn1 openais[4143]: [CLM  ] CLM CONFIGURATION CHANGE 
Dec 27 02:55:26 pcn1 openais[4143]: [CLM  ] New Configuration: 
Dec 27 02:55:26 pcn1 openais[4143]: [CLM  ]     r(0) ip(10.3.0.11)  
Dec 27 02:55:26 pcn1 openais[4143]: [CLM  ] Members Left: 
Dec 27 02:55:26 pcn1 openais[4143]: [CLM  ] Members Joined: 
Dec 27 02:55:26 pcn1 openais[4143]: [SYNC ] This node is within the primary component and will provide se
rvice. 
Dec 27 02:55:26 pcn1 openais[4143]: [CLM  ] CLM CONFIGURATION CHANGE 
Dec 27 02:55:26 pcn1 openais[4143]: [CLM  ] New Configuration: 
Dec 27 02:55:26 pcn1 openais[4143]: [CLM  ]     r(0) ip(10.3.0.10)  
Dec 27 02:55:26 pcn1 openais[4143]: [CLM  ]     r(0) ip(10.3.0.11)  
Dec 27 02:55:26 pcn1 openais[4143]: [CLM  ]     r(0) ip(10.3.0.12)  
Dec 27 02:55:26 pcn1 openais[4143]: [CLM  ]     r(0) ip(10.3.0.13)  
Dec 27 02:55:26 pcn1 openais[4143]: [CLM  ]     r(0) ip(10.3.0.14)  
Dec 27 02:55:26 pcn1 openais[4143]: [CLM  ]     r(0) ip(10.3.0.15)  
Dec 27 02:55:26 pcn1 openais[4143]: [CLM  ]     r(0) ip(10.3.0.16)  
Dec 27 02:55:26 pcn1 openais[4143]: [CLM  ] Members Left: 
Dec 27 02:55:26 pcn1 openais[4143]: [CLM  ] Members Joined: 
Dec 27 02:55:26 pcn1 openais[4143]: [CLM  ]     r(0) ip(10.3.0.10)  
Dec 27 02:55:26 pcn1 openais[4143]: [CLM  ]     r(0) ip(10.3.0.12)  
Dec 27 02:55:26 pcn1 openais[4143]: [CLM  ]     r(0) ip(10.3.0.13)  
Dec 27 02:55:26 pcn1 openais[4143]: [CLM  ]     r(0) ip(10.3.0.14)  
Dec 27 02:55:26 pcn1 openais[4143]: [CLM  ]     r(0) ip(10.3.0.15)  
Dec 27 02:55:26 pcn1 openais[4143]: [CLM  ]     r(0) ip(10.3.0.16)  
Dec 27 02:55:26 pcn1 openais[4143]: [SYNC ] This node is within the primary component and will provide se
rvice. 
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] entering OPERATIONAL state. 
Dec 27 02:55:26 pcn1 openais[4143]: [CMAN ] quorum regained, resuming activity 
Dec 27 02:55:26 pcn1 openais[4143]: [CLM  ] got nodejoin message 10.3.0.10 
Dec 27 02:55:26 pcn1 openais[4143]: [CLM  ] got nodejoin message 10.3.0.11 
Dec 27 02:55:26 pcn1 openais[4143]: [CLM  ] got nodejoin message 10.3.0.12 
Dec 27 02:55:26 pcn1 openais[4143]: [CLM  ] got nodejoin message 10.3.0.13 
Dec 27 02:55:26 pcn1 openais[4143]: [CLM  ] got nodejoin message 10.3.0.14 
Dec 27 02:55:26 pcn1 openais[4143]: [CLM  ] got nodejoin message 10.3.0.15 
Dec 27 02:55:26 pcn1 openais[4143]: [CLM  ] got nodejoin message 10.3.0.16 
Dec 27 02:55:26 pcn1 openais[4143]: [CPG  ] got joinlist message from node 100 
Dec 27 02:55:26 pcn1 openais[4143]: [CPG  ] got joinlist message from node 2 
Dec 27 02:55:26 pcn1 openais[4143]: [CPG  ] got joinlist message from node 3 
Dec 27 02:55:26 pcn1 openais[4143]: [CPG  ] got joinlist message from node 4 
Dec 27 02:55:26 pcn1 openais[4143]: [CPG  ] got joinlist message from node 5 
Dec 27 02:55:26 pcn1 openais[4143]: [CPG  ] got joinlist message from node 6 
Dec 27 02:55:26 pcn1 ccsd[4132]: Initial status:: Quorate 
Dec 27 02:55:35 pcn1 kernel: dlm: got connection from 100
Dec 27 02:55:35 pcn1 kernel: dlm: got connection from 2
Dec 27 02:55:35 pcn1 kernel: dlm: got connection from 3
Dec 27 02:55:35 pcn1 kernel: dlm: got connection from 5
Dec 27 02:55:35 pcn1 kernel: dlm: got connection from 6
Dec 27 02:55:35 pcn1 kernel: dlm: got connection from 4
Dec 27 02:55:35 pcn1 clvmd: Cluster LVM daemon started - connected to CMAN
Dec 27 03:01:04 pcn1 clurgmgrd[5515]: <notice> Starting stopped service service:i03 
Dec 27 03:01:04 pcn1 clurgmgrd[5515]: <notice> Starting stopped service service:i15 
[etc]
--clip--

Now I tried googling around for the mysterious error message #48, and couldn't
find any info. What might've been up?


--Janne
-- 
Janne Peltonen <janne peltonen helsinki fi>


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]