[Linux-cluster] Cluster behavior

Tue Aug 18 16:22:34 UTC 2009

On Tue, Aug 18, 2009 at 8:26 AM, Moralejo,
Alfredo<alfredo.moralejo at roche.com> wrote:
> Could you send logs in messages file including the part where the cluster dissolved messages is logged?

Ok.. here are the logs . I shutdown node 3 which have 2 votes.  Below
are the logs from node1 and node 2.

X.X.X.165 is node 1
X.X.X.172 is node 2
X.X.X.173 is node 3

Log from node1:

ug 18 10:35:28 cvtst1 ntpd[3862]: synchronized to X.X.X.103, stratum 2
Aug 18 11:01:44 cvtst1 clurgmgrd[4309]: <notice> Member 3 shutting down
Aug 18 11:01:57 cvtst1 kernel: peth0: received packet with  own
address as source address
Aug 18 11:01:57 cvtst1 qdiskd[3361]: <info> Node 3 shutdown
Aug 18 11:02:02 cvtst1 kernel: peth0: received packet with  own
address as source address
Aug 18 11:02:07 cvtst1 openais[3318]: [TOTEM] The token was lost in
the OPERATIONAL state.
Aug 18 11:02:07 cvtst1 openais[3318]: [TOTEM] Receive multicast socket
recv buffer size (288000 bytes).
Aug 18 11:02:07 cvtst1 openais[3318]: [TOTEM] Transmit multicast
socket send buffer size (262142 bytes).
Aug 18 11:02:07 cvtst1 openais[3318]: [TOTEM] entering GATHER state from 2.
Aug 18 11:02:12 cvtst1 openais[3318]: [TOTEM] entering GATHER state from 11.
Aug 18 11:02:12 cvtst1 openais[3318]: [TOTEM] Creating commit token
because I am the rep.
Aug 18 11:02:12 cvtst1 openais[3318]: [TOTEM] Saving state aru 9f high
seq received 9f
Aug 18 11:02:12 cvtst1 openais[3318]: [TOTEM] Storing new sequence id
for ring 758
Aug 18 11:02:12 cvtst1 openais[3318]: [TOTEM] entering COMMIT state.
Aug 18 11:02:12 cvtst1 openais[3318]: [TOTEM] entering RECOVERY state.
Aug 18 11:02:12 cvtst1 openais[3318]: [TOTEM] position [0] member X.X.X.165:
Aug 18 11:02:12 cvtst1 openais[3318]: [TOTEM] previous ring seq 1876
rep X.X.X.165
Aug 18 11:02:12 cvtst1 openais[3318]: [TOTEM] aru 9f high delivered 9f
received flag 1
Aug 18 11:02:12 cvtst1 openais[3318]: [TOTEM] position [1] member X.X.X.172:
Aug 18 11:02:12 cvtst1 openais[3318]: [TOTEM] previous ring seq 1876
rep X.X.X.165
Aug 18 11:02:12 cvtst1 openais[3318]: [TOTEM] aru 9f high delivered 9f
received flag 1
Aug 18 11:02:12 cvtst1 openais[3318]: [TOTEM] Did not need to
originate any messages in recovery.
Aug 18 11:02:12 cvtst1 openais[3318]: [TOTEM] Sending initial ORF token
Aug 18 11:02:12 cvtst1 kernel: dlm: closing connection to node 3
Aug 18 11:02:12 cvtst1 openais[3318]: [CLM  ] CLM CONFIGURATION CHANGE
Aug 18 11:02:12 cvtst1 openais[3318]: [CLM  ] New Configuration:
Aug 18 11:02:12 cvtst1 openais[3318]: [CLM  ] 	r(0) ip(X.X.X.165)
Aug 18 11:02:12 cvtst1 openais[3318]: [CLM  ] 	r(0) ip(X.X.X.172)
Aug 18 11:02:12 cvtst1 openais[3318]: [CLM  ] Members Left:
Aug 18 11:02:12 cvtst1 openais[3318]: [CLM  ] 	r(0) ip(X.X.X.173)
Aug 18 11:02:12 cvtst1 clurgmgrd[4309]: <emerg> #1: Quorum Dissolved
Aug 18 11:02:12 cvtst1 openais[3318]: [CLM  ] Members Joined:
Aug 18 11:02:12 cvtst1 openais[3318]: [CLM  ] CLM CONFIGURATION CHANGE
Aug 18 11:02:12 cvtst1 openais[3318]: [CLM  ] New Configuration:
Aug 18 11:02:12 cvtst1 openais[3318]: [CLM  ] 	r(0) ip(X.X.X.165)
Aug 18 11:02:12 cvtst1 openais[3318]: [CLM  ] 	r(0) ip(X.X.X.172)
Aug 18 11:02:12 cvtst1 openais[3318]: [CLM  ] Members Left:
Aug 18 11:02:12 cvtst1 openais[3318]: [CLM  ] Members Joined:
Aug 18 11:02:12 cvtst1 openais[3318]: [SYNC ] This node is within the
primary component and will provide service.
Aug 18 11:02:12 cvtst1 openais[3318]: [TOTEM] entering OPERATIONAL state.
Aug 18 11:02:12 cvtst1 openais[3318]: [CLM  ] got nodejoin message X.X.X.165
Aug 18 11:02:12 cvtst1 openais[3318]: [CLM  ] got nodejoin message X.X.X.172
Aug 18 11:02:13 cvtst1 openais[3318]: [CPG  ] got joinlist message from node 2
Aug 18 11:02:13 cvtst1 openais[3318]: [CPG  ] got joinlist message from node 1
Aug 18 11:02:13 cvtst1 openais[3318]: [CMAN ] lost contact with quorum device
Aug 18 11:02:13 cvtst1 openais[3318]: [CMAN ] quorum lost, blocking activity
Aug 18 11:02:13 cvtst1 ccsd[3274]: Cluster is not quorate.  Refusing
connection.
Aug 18 11:02:13 cvtst1 ccsd[3274]: Error while processing connect:
Connection refused
Aug 18 11:02:22 cvtst1 ccsd[3274]: Cluster is not quorate.  Refusing
connection.
Aug 18 11:02:22 cvtst1 ccsd[3274]: Error while processing connect:
Connection refused
Aug 18 11:02:27 cvtst1 qdiskd[3361]: <info> Node 1 is the master
Aug 18 11:02:30 cvtst1 openais[3318]: [CMAN ] quorum regained,
resuming activity
Aug 18 11:02:36 cvtst1 kernel: xenbr0: port 3(vif1.0) entering disabled state
Aug 18 11:02:36 cvtst1 kernel: device vif1.0 left promiscuous mode
Aug 18 11:02:36 cvtst1 kernel: xenbr0: port 3(vif1.0) entering disabled state
Aug 18 11:02:39 cvtst1 clurgmgrd[4309]: <notice> Quorum Regained
Aug 18 11:02:41 cvtst1 clurgmgrd[4309]: <notice> Starting stopped
service vm:guest1
Aug 18 11:02:42 cvtst1 kernel: tap tap-2-51712: 2 getting info
Aug 18 11:02:43 cvtst1 kernel: device vif2.0 entered promiscuous mode
Aug 18 11:02:43 cvtst1 kernel: ADDRCONF(NETDEV_UP): vif2.0: link is not ready
Aug 18 11:02:43 cvtst1 clurgmgrd[4309]: <notice> Service vm:guest1 started
Aug 18 11:02:47 cvtst1 kernel: blktap: ring-ref 8, event-channel 6,
protocol 1 (x86_64-abi)
Aug 18 11:02:56 cvtst1 kernel: xenbr0: topology change detected, propagating
Aug 18 11:02:56 cvtst1 kernel: xenbr0: port 3(vif2.0) entering forwarding state
Aug 18 11:02:56 cvtst1 kernel: ADDRCONF(NETDEV_CHANGE): vif2.0: link
becomes ready

>From node2:

Aug 18 11:01:44 cvtst2 clurgmgrd[4365]: <notice> Member 3 shutting down
Aug 18 11:01:55 cvtst2 qdiskd[3403]: <info> Node 3 shutdown
Aug 18 11:01:57 cvtst2 kernel: peth0: received packet with  own
address as source address
Aug 18 11:02:02 cvtst2 kernel: peth0: received packet with  own
address as source address
Aug 18 11:02:07 cvtst2 openais[3359]: [TOTEM] entering GATHER state from 12.
Aug 18 11:02:09 cvtst2 openais[3359]: [CMAN ] lost contact with quorum device
Aug 18 11:02:09 cvtst2 openais[3359]: [CMAN ] quorum lost, blocking activity
Aug 18 11:02:09 cvtst2 clurgmgrd[4365]: <emerg> #1: Quorum Dissolved
Aug 18 11:02:09 cvtst2 kernel: dlm: closing connection to node 3
Aug 18 11:02:10 cvtst2 ccsd[3316]: Cluster is not quorate.  Refusing
connection.
Aug 18 11:02:10 cvtst2 ccsd[3316]: Error while processing connect:
Connection refused
Aug 18 11:02:12 cvtst2 openais[3359]: [TOTEM] entering GATHER state from 0.
Aug 18 11:02:12 cvtst2 openais[3359]: [TOTEM] Saving state aru 9f high
seq received 9f
Aug 18 11:02:12 cvtst2 openais[3359]: [TOTEM] Storing new sequence id
for ring 758
Aug 18 11:02:12 cvtst2 openais[3359]: [TOTEM] entering COMMIT state.
Aug 18 11:02:12 cvtst2 openais[3359]: [TOTEM] entering RECOVERY state.
Aug 18 11:02:12 cvtst2 openais[3359]: [TOTEM] position [0] member X.X.X.165:
Aug 18 11:02:12 cvtst2 openais[3359]: [TOTEM] previous ring seq 1876
rep X.X.X.165
Aug 18 11:02:12 cvtst2 openais[3359]: [TOTEM] aru 9f high delivered 9f
received flag 1
Aug 18 11:02:12 cvtst2 openais[3359]: [TOTEM] position [1] member X.X.X.172:
Aug 18 11:02:12 cvtst2 openais[3359]: [TOTEM] previous ring seq 1876
rep X.X.X.165
Aug 18 11:02:12 cvtst2 openais[3359]: [TOTEM] aru 9f high delivered 9f
received flag 1
Aug 18 11:02:12 cvtst2 openais[3359]: [TOTEM] Did not need to
originate any messages in recovery.
Aug 18 11:02:12 cvtst2 openais[3359]: [CLM  ] CLM CONFIGURATION CHANGE
Aug 18 11:02:12 cvtst2 openais[3359]: [CLM  ] New Configuration:
Aug 18 11:02:12 cvtst2 openais[3359]: [CLM  ] 	r(0) ip(X.X.X.165)
Aug 18 11:02:12 cvtst2 openais[3359]: [CLM  ] 	r(0) ip(X.X.X.172)
Aug 18 11:02:12 cvtst2 openais[3359]: [CLM  ] Members Left:
Aug 18 11:02:12 cvtst2 openais[3359]: [CLM  ] 	r(0) ip(X.X.X.173)
Aug 18 11:02:12 cvtst2 openais[3359]: [CLM  ] Members Joined:
Aug 18 11:02:12 cvtst2 openais[3359]: [CMAN ] quorum regained,
resuming activity
Aug 18 11:02:12 cvtst2 openais[3359]: [CLM  ] CLM CONFIGURATION CHANGE
Aug 18 11:02:12 cvtst2 openais[3359]: [CLM  ] New Configuration:
Aug 18 11:02:12 cvtst2 openais[3359]: [CLM  ] 	r(0) ip(X.X.X.165)
Aug 18 11:02:12 cvtst2 openais[3359]: [CLM  ] 	r(0) ip(X.X.X.172)
Aug 18 11:02:12 cvtst2 openais[3359]: [CLM  ] Members Left:
Aug 18 11:02:12 cvtst2 openais[3359]: [CLM  ] Members Joined:
Aug 18 11:02:12 cvtst2 openais[3359]: [SYNC ] This node is within the
primary component and will provide service.
Aug 18 11:02:12 cvtst2 openais[3359]: [TOTEM] entering OPERATIONAL state.
Aug 18 11:02:12 cvtst2 openais[3359]: [CLM  ] got nodejoin message X.X.X.165
Aug 18 11:02:12 cvtst2 openais[3359]: [CLM  ] got nodejoin message X.X.X.172
Aug 18 11:02:12 cvtst2 openais[3359]: [CPG  ] got joinlist message from node 2
Aug 18 11:02:12 cvtst2 openais[3359]: [CPG  ] got joinlist message from node 1
Aug 18 11:02:25 cvtst2 qdiskd[3403]: <info> Assuming master role
Aug 18 11:02:33 cvtst2 kernel: xenbr0: port 3(vif1.0) entering disabled state
Aug 18 11:02:33 cvtst2 kernel: device vif1.0 left promiscuous mode
Aug 18 11:02:33 cvtst2 kernel: xenbr0: port 3(vif1.0) entering disabled state
Aug 18 11:02:36 cvtst2 clurgmgrd[4365]: <notice> Quorum Regained
Aug 18 11:02:39 cvtst2 clurgmgrd[4365]: <notice> Starting stopped
service vm:guest2
Aug 18 11:02:41 cvtst2 kernel: tap tap-2-51712: 2 getting info
Aug 18 11:02:41 cvtst2 kernel: device vif2.0 entered promiscuous mode
Aug 18 11:02:41 cvtst2 kernel: ADDRCONF(NETDEV_UP): vif2.0: link is not ready
Aug 18 11:02:41 cvtst2 clurgmgrd[4365]: <notice> Service vm:guest2 started
Aug 18 11:02:45 cvtst2 kernel: blktap: ring-ref 8, event-channel 6,
protocol 1 (x86_64-abi)
Aug 18 11:02:53 cvtst2 kernel: xenbr0: topology change detected, propagating
Aug 18 11:02:53 cvtst2 kernel: xenbr0: port 3(vif2.0) entering forwarding state
Aug 18 11:02:53 cvtst2 kernel: ADDRCONF(NETDEV_CHANGE): vif2.0: link
becomes read

>
> Regards,
>
> Alfredo
>
> -----Original Message-----
> From: linux-cluster-bounces at redhat.com [mailto:linux-cluster-bounces at redhat.com] On Behalf Of Paras pradhan
> Sent: Monday, August 17, 2009 11:59 PM
> To: linux clustering
> Subject: [Linux-cluster] Cluster behavior
>
> I have a 3 nodes cluster.
>
> Node A - Vote 1
>
> Node B - Vote 1
>
> Node C - Votes 2
>
> Qdisk - Votes 3
>
>
> Altogether the cluster has 7 votes. The required min quorum to run the
> cluster would be in this case 4. Now if I poweroff node 1, I can see
> Quorum Dissolved in the terminals of Node 2 and Node3 . This cluster
> has xen virtual machines.
>
> Whats wrong and how to do debug the problem?
>
> Thanks
> Paras.
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>

Thanks
Paras.