[Linux-cluster] <err> #48: Unable to obtain cluster lock: Invalid argument
Janne Peltonen
janne.peltonen at helsinki.fi
Wed Jan 2 11:37:35 UTC 2008
Hi.
After running a cluster node in a production cluster since July, I got
the folllowing error:
<err> #48: Unable to obtain cluster lock: Invalid argument
Which resulted in a reboot:
--clip--
Dec 27 02:50:31 pcn1 clurgmgrd[6217]: <err> #48: Unable to obtain
cluster lock: Invalid argument
Dec 27 02:50:31 pcn1 clurgmgrd[6217]: <notice> Stopping service
service:p01
Dec 27 02:50:34 pcn1 in.rdiscd[30325]: setsockopt (IP_ADD_MEMBERSHIP):
Address already in use
Dec 27 02:50:34 pcn1 in.rdiscd[30325]: Failed joining addresses
Dec 27 02:50:38 pcn1 snmpd[15929]: error on subcontainer 'ia_addr'
insert (-1)
Dec 27 02:50:38 pcn1 snmpd[15929]: error on subcontainer 'ia_addr'
insert (-1)
Dec 27 02:50:38 pcn1 snmpd[15929]: error on subcontainer '' insert (-1)
Dec 27 02:50:38 pcn1 snmpd[15929]: error on subcontainer '' insert (-1)
Dec 27 02:50:45 pcn1 clurgmgrd[6217]: <notice> Service service:p01 is
recovering
Dec 27 02:50:45 pcn1 clurgmgrd[6217]: <notice> Recovering failed service
service:p01
Dec 27 02:50:45 pcn1 kernel: dlm: add_to_waiters error 1
Dec 27 02:50:45 pcn1 kernel: dlm: remove_from_waiters error
Dec 27 02:50:45 pcn1 kernel: dlm: rgmanager: receive_unlock_reply not on
waiters
Dec 27 02:50:45 pcn1 clurgmgrd[6216]: <crit> Watchdog: Daemon died,
rebooting...
Dec 27 02:50:45 pcn1 kernel: md: stopping all md devices.
Dec 27 02:55:23 pcn1 syslogd 1.4.1: restart.
--clip--
Other members of the cluster noticed the missing member, fenced it,
failed services over, and back (when the missing node had rejoined):
--clip--
Dec 27 02:50:56 pcn2 openais[4588]: [TOTEM] The token was lost in the OPERATIONAL state.
Dec 27 02:50:56 pcn2 openais[4588]: [TOTEM] Receive multicast socket recv buffer size (262142 bytes).
Dec 27 02:50:56 pcn2 openais[4588]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Dec 27 02:50:56 pcn2 openais[4588]: [TOTEM] entering GATHER state from 2.
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] entering GATHER state from 11.
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] Saving state aru 6a4 high seq received 6a4
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] entering COMMIT state.
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] entering RECOVERY state.
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] position [0] member 10.3.0.10:
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] previous ring seq 324 rep 10.3.0.10
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] aru 6a4 high delivered 6a4 received flag 0
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] position [1] member 10.3.0.12:
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] previous ring seq 324 rep 10.3.0.10
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] aru 6a4 high delivered 6a4 received flag 0
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] position [2] member 10.3.0.13:
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] previous ring seq 324 rep 10.3.0.10
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] aru 6a4 high delivered 6a4 received flag 0
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] position [3] member 10.3.0.14:
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] previous ring seq 324 rep 10.3.0.10
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] aru 6a4 high delivered 6a4 received flag 0
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] position [4] member 10.3.0.15:
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] previous ring seq 324 rep 10.3.0.10
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] aru 6a4 high delivered 6a4 received flag 1
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] position [5] member 10.3.0.16:
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] previous ring seq 324 rep 10.3.0.10
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] aru 6a4 high delivered 6a4 received flag 1
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] Did not need to originate any messages in recovery.
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] Storing new sequence id for ring 14c
Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] CLM CONFIGURATION CHANGE
Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] New Configuration:
Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.10)
Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.12)
Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.13)
Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.14)
Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.15)
Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.16)
Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] Members Left:
Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.11)
Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] Members Joined:
Dec 27 02:51:01 pcn2 openais[4588]: [SYNC ] This node is within the primary component and will provide se
rvice.
Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] CLM CONFIGURATION CHANGE
Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] New Configuration:
Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.10)
Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.12)
Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.13)
Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.14)
Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.15)
Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.16)
Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] Members Left:
Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] Members Joined:
Dec 27 02:51:01 pcn2 openais[4588]: [SYNC ] This node is within the primary component and will provide se
rvice.
Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] entering OPERATIONAL state.
Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] got nodejoin message 10.3.0.10
Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] got nodejoin message 10.3.0.12
Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] got nodejoin message 10.3.0.13
Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] got nodejoin message 10.3.0.14
Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] got nodejoin message 10.3.0.15
Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] got nodejoin message 10.3.0.16
Dec 27 02:51:01 pcn2 openais[4588]: [CPG ] got joinlist message from node 3
Dec 27 02:51:01 pcn2 openais[4588]: [CPG ] got joinlist message from node 4
Dec 27 02:51:01 pcn2 openais[4588]: [CPG ] got joinlist message from node 5
Dec 27 02:51:01 pcn2 openais[4588]: [CPG ] got joinlist message from node 6
Dec 27 02:51:01 pcn2 openais[4588]: [CPG ] got joinlist message from node 100
Dec 27 02:51:01 pcn2 openais[4588]: [CPG ] got joinlist message from node 2
Dec 27 02:51:01 pcn2 kernel: dlm: closing connection to node 1
Dec 27 02:51:01 pcn2 fenced[4614]: pcn1-hb not a cluster member after 0 sec post_fail_delay
Dec 27 02:51:01 pcn2 fenced[4614]: fencing node "pcn1-hb"
Dec 27 02:52:13 pcn2 fenced[4614]: fence "pcn1-hb" success
Dec 27 02:52:18 pcn2 ccsd[4541]: Attempt to close an unopened CCS descriptor (799075500).
Dec 27 02:52:18 pcn2 ccsd[4541]: Error while processing disconnect: Invalid request descriptor
Dec 27 02:52:20 pcn2 clurgmgrd[6262]: <notice> Taking over service service:p01 from down member pcn1-hb
Dec 27 02:52:20 pcn2 clurgmgrd[6262]: <notice> Taking over service service:i01 from down member pcn1-hb
Dec 27 02:52:20 pcn2 kernel: kjournald starting. Commit interval 5 seconds
Dec 27 02:52:20 pcn2 kernel: EXT3 FS on dm-65, internal journal
Dec 27 02:52:20 pcn2 kernel: EXT3-fs: mounted filesystem with ordered data mode.
Dec 27 02:52:21 pcn2 clurgmgrd[6262]: <notice> Taking over service service:i13 from down member pcn1-hb
Dec 27 02:52:21 pcn2 in.rdiscd[2158]: setsockopt (IP_ADD_MEMBERSHIP): Address already in use
Dec 27 02:52:21 pcn2 in.rdiscd[2158]: Failed joining addresses
Dec 27 02:52:22 pcn2 kernel: kjournald starting. Commit interval 5 seconds
Dec 27 02:52:22 pcn2 kernel: EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
Dec 27 02:52:22 pcn2 kernel: EXT3 FS on dm-14, internal journal
Dec 27 02:52:22 pcn2 kernel: EXT3-fs: recovery complete.
Dec 27 02:52:22 pcn2 kernel: EXT3-fs: mounted filesystem with ordered data mode.
Dec 27 02:52:24 pcn2 clurgmgrd[6262]: <notice> Service service:p01 started
Dec 27 02:52:25 pcn2 last message repeated 2 times
Dec 27 02:52:27 pcn2 kernel: kjournald starting. Commit interval 5 seconds
Dec 27 02:52:27 pcn2 kernel: EXT3 FS on dm-2, internal journal
Dec 27 02:52:27 pcn2 kernel: EXT3-fs: dm-2: 3 orphan inodes deleted
Dec 27 02:52:27 pcn2 kernel: EXT3-fs: recovery complete.
Dec 27 02:52:27 pcn2 kernel: EXT3-fs: mounted filesystem with ordered data mode.
Dec 27 02:52:29 pcn2 kernel: kjournald starting. Commit interval 5 seconds
Dec 27 02:52:29 pcn2 kernel: EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
Dec 27 02:52:29 pcn2 kernel: EXT3 FS on dm-38, internal journal
Dec 27 02:52:29 pcn2 kernel: EXT3-fs: recovery complete.
Dec 27 02:52:29 pcn2 kernel: EXT3-fs: mounted filesystem with ordered data mode.
Dec 27 02:52:30 pcn2 in.rdiscd[3313]: setsockopt (IP_ADD_MEMBERSHIP): Address already in use
Dec 27 02:52:30 pcn2 in.rdiscd[3313]: Failed joining addresses
Dec 27 02:52:32 pcn2 clurgmgrd[6262]: <notice> Service service:i13 started
Dec 27 02:52:35 pcn2 kernel: kjournald starting. Commit interval 5 seconds
Dec 27 02:52:35 pcn2 kernel: EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
Dec 27 02:52:35 pcn2 kernel: EXT3 FS on dm-26, internal journal
Dec 27 02:52:35 pcn2 kernel: EXT3-fs: recovery complete.
Dec 27 02:52:35 pcn2 kernel: EXT3-fs: mounted filesystem with ordered data mode.
Dec 27 02:52:37 pcn2 in.rdiscd[3833]: setsockopt (IP_ADD_MEMBERSHIP): Address already in use
Dec 27 02:52:37 pcn2 in.rdiscd[3833]: Failed joining addresses
Dec 27 02:52:38 pcn2 clurgmgrd[6262]: <notice> Service service:i01 started
Dec 27 02:53:25 pcn2 last message repeated 2 times
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] entering GATHER state from 11.
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] Saving state aru c8 high seq received c8
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] entering COMMIT state.
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] entering RECOVERY state.
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] position [0] member 10.3.0.10:
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] previous ring seq 332 rep 10.3.0.10
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] aru c8 high delivered c8 received flag 0
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] position [1] member 10.3.0.11:
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] previous ring seq 288 rep 10.3.0.11
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] aru 9 high delivered 9 received flag 0
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] position [2] member 10.3.0.12:
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] previous ring seq 332 rep 10.3.0.10
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] aru c8 high delivered c8 received flag 0
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] position [3] member 10.3.0.13:
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] previous ring seq 332 rep 10.3.0.10
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] aru c8 high delivered c8 received flag 0
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] position [4] member 10.3.0.14:
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] previous ring seq 332 rep 10.3.0.10
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] aru c8 high delivered c8 received flag 0
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] position [5] member 10.3.0.15:
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] previous ring seq 332 rep 10.3.0.10
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] aru c8 high delivered c8 received flag 1
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] position [6] member 10.3.0.16:
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] previous ring seq 332 rep 10.3.0.10
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] aru c8 high delivered c8 received flag 1
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] Did not need to originate any messages in recovery.
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] Storing new sequence id for ring 150
Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] CLM CONFIGURATION CHANGE
Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] New Configuration:
Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.10)
Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.12)
Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.13)
Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.14)
Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.15)
Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] Members Left:
Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] Members Joined:
Dec 27 02:55:26 pcn2 openais[4588]: [SYNC ] This node is within the primary component and will provide se
rvice.
Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] CLM CONFIGURATION CHANGE
Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] New Configuration:
Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.10)
Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.11)
Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.12)
Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.13)
Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.14)
Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.15)
Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.16)
Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] Members Left:
Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] Members Joined:
Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.11)
Dec 27 02:55:26 pcn2 openais[4588]: [SYNC ] This node is within the primary component and will provide se
rvice.
Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] entering OPERATIONAL state.
Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] got nodejoin message 10.3.0.10
Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] got nodejoin message 10.3.0.11
Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] got nodejoin message 10.3.0.12
Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] got nodejoin message 10.3.0.13
Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] got nodejoin message 10.3.0.14
Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] got nodejoin message 10.3.0.15
Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] got nodejoin message 10.3.0.16
Dec 27 02:55:26 pcn2 openais[4588]: [CPG ] got joinlist message from node 100
Dec 27 02:55:26 pcn2 openais[4588]: [CPG ] got joinlist message from node 2
Dec 27 02:55:26 pcn2 openais[4588]: [CPG ] got joinlist message from node 3
Dec 27 02:55:26 pcn2 openais[4588]: [CPG ] got joinlist message from node 4
Dec 27 02:55:26 pcn2 openais[4588]: [CPG ] got joinlist message from node 5
Dec 27 02:55:26 pcn2 openais[4588]: [CPG ] got joinlist message from node 6
Dec 27 02:55:35 pcn2 kernel: dlm: connecting to 1
--clip--
--clip--
Dec 27 02:55:24 pcn1 ccsd[4132]: Starting ccsd 2.0.69:
Dec 27 02:55:24 pcn1 ccsd[4132]: Built: Jun 27 2007 15:21:32
Dec 27 02:55:24 pcn1 ccsd[4132]: Copyright (C) Red Hat, Inc. 2004 All rights reserved.
Dec 27 02:55:24 pcn1 ccsd[4132]: cluster.conf (cluster name = mappi-primary, version = 109) found.
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] AIS Executive Service RELEASE 'subrev 1324 version 0.80.2'
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] Copyright (C) 2002-2006 MontaVista Software, Inc and contribu
tors.
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] Copyright (C) 2006 Red Hat, Inc.
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] AIS Executive Service: started and ready to provide service.
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] Using default multicast address of 239.192.46.199
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] openais component openais_cpg loaded.
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] Registering service handler 'openais cluster closed process g
roup service v1.01'
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] openais component openais_cfg loaded.
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] Registering service handler 'openais configuration service'
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] openais component openais_msg loaded.
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] Registering service handler 'openais message service B.01.01'
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] openais component openais_lck loaded.
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] Registering service handler 'openais distributed locking serv
ice B.01.01'
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] openais component openais_evt loaded.
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] Registering service handler 'openais event service B.01.01'
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] openais component openais_ckpt loaded.
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] Registering service handler 'openais checkpoint service B.01.
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] openais component openais_amf loaded.
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] Registering service handler 'openais availability management
framework B.01.01'
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] openais component openais_clm loaded.
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] Registering service handler 'openais cluster membership servi
ce B.01.01'
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] openais component openais_evs loaded.
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] Registering service handler 'openais extended virtual synchro
ny service'
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] openais component openais_cman loaded.
Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] Registering service handler 'openais CMAN membership service
2.01'
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] Token Timeout (10000 ms) retransmit timeout (495 ms)
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] token hold (386 ms) retransmits before loss (20 retrans)
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] join (60 ms) send_join (0 ms) consensus (4800 ms) merge (200
ms)
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] downcheck (1000 ms) fail to recv const (50 msgs)
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] seqno unchanged const (30 rotations) Maximum network MTU 1500
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] window size per rotation (50 messages) maximum messages per r
otation (17 messages)
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] send threads (0 threads)
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] RRP token expired timeout (495 ms)
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] RRP token problem counter (2000 ms)
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] RRP threshold (10 problem count)
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] RRP mode set to none.
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] heartbeat_failures_allowed (0)
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] max_network_delay (50 ms)
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] HeartBeat is Disabled. To enable set heartbeat_failures_allow
ed > 0
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] Receive multicast socket recv buffer size (262142 bytes).
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] The network interface [10.3.0.11] is now up.
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] Created or loaded sequence id 284.10.3.0.11 for this ring.
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] entering GATHER state from 15.
Dec 27 02:55:26 pcn1 openais[4143]: [SERV ] Initialising service handler 'openais extended virtual synchr
ony service'
Dec 27 02:55:26 pcn1 openais[4143]: [SERV ] Initialising service handler 'openais cluster membership serv
ice B.01.01'
Dec 27 02:55:26 pcn1 openais[4143]: [SERV ] Initialising service handler 'openais availability management
framework B.01.01'
Dec 27 02:55:26 pcn1 openais[4143]: [SERV ] Initialising service handler 'openais checkpoint service B.01
.01'
Dec 27 02:55:26 pcn1 openais[4143]: [SERV ] Initialising service handler 'openais event service B.01.01'
Dec 27 02:55:26 pcn1 openais[4143]: [SERV ] Initialising service handler 'openais distributed locking ser
vice B.01.01'
Dec 27 02:55:26 pcn1 openais[4143]: [SERV ] Initialising service handler 'openais message service B.01.01
'
Dec 27 02:55:26 pcn1 openais[4143]: [SERV ] Initialising service handler 'openais configuration service'
Dec 27 02:55:26 pcn1 openais[4143]: [SERV ] Initialising service handler 'openais cluster closed process
group service v1.01'
Dec 27 02:55:26 pcn1 openais[4143]: [SERV ] Initialising service handler 'openais CMAN membership service
2.01'
Dec 27 02:55:26 pcn1 openais[4143]: [CMAN ] CMAN 2.0.69 (built Jun 27 2007 15:21:36) started
Dec 27 02:55:26 pcn1 openais[4143]: [SYNC ] Not using a virtual synchrony filter.
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] Creating commit token because I am the rep.
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] Saving state aru 0 high seq received 0
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] entering COMMIT state.
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] entering RECOVERY state.
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] position [0] member 10.3.0.11:
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] previous ring seq 284 rep 10.3.0.11
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] aru 0 high delivered 0 received flag 0
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] Did not need to originate any messages in recovery.
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] Storing new sequence id for ring 120
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] Sending initial ORF token
Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] CLM CONFIGURATION CHANGE
Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] New Configuration:
Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] Members Left:
Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] Members Joined:
Dec 27 02:55:26 pcn1 openais[4143]: [SYNC ] This node is within the primary component and will provide se
rvice.
Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] CLM CONFIGURATION CHANGE
Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] New Configuration:
Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] r(0) ip(10.3.0.11)
Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] Members Left:
Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] Members Joined:
Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] r(0) ip(10.3.0.11)
Dec 27 02:55:26 pcn1 openais[4143]: [SYNC ] This node is within the primary component and will provide se
rvice.
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] entering OPERATIONAL state.
Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] got nodejoin message 10.3.0.11
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] entering GATHER state from 11.
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] Saving state aru 9 high seq received 9
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] entering COMMIT state.
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] entering RECOVERY state.
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] position [0] member 10.3.0.10:
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] previous ring seq 332 rep 10.3.0.10
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] aru c8 high delivered c8 received flag 0
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] position [1] member 10.3.0.11:
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] previous ring seq 288 rep 10.3.0.11
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] aru 9 high delivered 9 received flag 0
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] position [2] member 10.3.0.12:
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] previous ring seq 332 rep 10.3.0.10
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] aru c8 high delivered c8 received flag 0
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] position [3] member 10.3.0.13:
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] previous ring seq 332 rep 10.3.0.10
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] aru c8 high delivered c8 received flag 0
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] position [4] member 10.3.0.14:
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] previous ring seq 332 rep 10.3.0.10
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] aru c8 high delivered c8 received flag 0
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] position [5] member 10.3.0.15:
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] previous ring seq 332 rep 10.3.0.10
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] aru c8 high delivered c8 received flag 1
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] position [6] member 10.3.0.16:
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] previous ring seq 332 rep 10.3.0.10
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] aru c8 high delivered c8 received flag 1
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] Did not need to originate any messages in recovery.
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] Storing new sequence id for ring 150
Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] CLM CONFIGURATION CHANGE
Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] New Configuration:
Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] r(0) ip(10.3.0.11)
Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] Members Left:
Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] Members Joined:
Dec 27 02:55:26 pcn1 openais[4143]: [SYNC ] This node is within the primary component and will provide se
rvice.
Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] CLM CONFIGURATION CHANGE
Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] New Configuration:
Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] r(0) ip(10.3.0.10)
Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] r(0) ip(10.3.0.11)
Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] r(0) ip(10.3.0.12)
Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] r(0) ip(10.3.0.13)
Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] r(0) ip(10.3.0.14)
Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] r(0) ip(10.3.0.15)
Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] r(0) ip(10.3.0.16)
Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] Members Left:
Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] Members Joined:
Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] r(0) ip(10.3.0.10)
Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] r(0) ip(10.3.0.12)
Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] r(0) ip(10.3.0.13)
Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] r(0) ip(10.3.0.14)
Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] r(0) ip(10.3.0.15)
Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] r(0) ip(10.3.0.16)
Dec 27 02:55:26 pcn1 openais[4143]: [SYNC ] This node is within the primary component and will provide se
rvice.
Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] entering OPERATIONAL state.
Dec 27 02:55:26 pcn1 openais[4143]: [CMAN ] quorum regained, resuming activity
Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] got nodejoin message 10.3.0.10
Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] got nodejoin message 10.3.0.11
Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] got nodejoin message 10.3.0.12
Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] got nodejoin message 10.3.0.13
Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] got nodejoin message 10.3.0.14
Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] got nodejoin message 10.3.0.15
Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] got nodejoin message 10.3.0.16
Dec 27 02:55:26 pcn1 openais[4143]: [CPG ] got joinlist message from node 100
Dec 27 02:55:26 pcn1 openais[4143]: [CPG ] got joinlist message from node 2
Dec 27 02:55:26 pcn1 openais[4143]: [CPG ] got joinlist message from node 3
Dec 27 02:55:26 pcn1 openais[4143]: [CPG ] got joinlist message from node 4
Dec 27 02:55:26 pcn1 openais[4143]: [CPG ] got joinlist message from node 5
Dec 27 02:55:26 pcn1 openais[4143]: [CPG ] got joinlist message from node 6
Dec 27 02:55:26 pcn1 ccsd[4132]: Initial status:: Quorate
Dec 27 02:55:35 pcn1 kernel: dlm: got connection from 100
Dec 27 02:55:35 pcn1 kernel: dlm: got connection from 2
Dec 27 02:55:35 pcn1 kernel: dlm: got connection from 3
Dec 27 02:55:35 pcn1 kernel: dlm: got connection from 5
Dec 27 02:55:35 pcn1 kernel: dlm: got connection from 6
Dec 27 02:55:35 pcn1 kernel: dlm: got connection from 4
Dec 27 02:55:35 pcn1 clvmd: Cluster LVM daemon started - connected to CMAN
Dec 27 03:01:04 pcn1 clurgmgrd[5515]: <notice> Starting stopped service service:i03
Dec 27 03:01:04 pcn1 clurgmgrd[5515]: <notice> Starting stopped service service:i15
[etc]
--clip--
Now I tried googling around for the mysterious error message #48, and couldn't
find any info. What might've been up?
--Janne
--
Janne Peltonen <janne.peltonen at helsinki.fi>
More information about the Linux-cluster
mailing list