[Linux-cluster] quorum device not getting a vote causes 2-node cluster to be inquorate

[Fabio M. Di Nitto]

On 03/15/2011 05:11 AM, bergman at merctech.com wrote:
> I have been using a 2-node cluster with a quorum disk successfully for
> about 2 years. Beginning today, the cluster will not boot correctly.
> 
> The RHCS services start, but fencing fails with:
> 	
> 	dlm: no local IP address has been set
> 	dlm: cannot start dlm lowcomms -107
> 
> This seems to be a symtpom of the fact that the cluster votes do not include votes from the quorum
> device:
> 
> 	# clustat
> 	Cluster Status for example-infra @ Tue Mar 15 00:02:35 2011
> 	Member Status: Inquorate
> 
> 	Member Name                                              ID   Status
> 	------ ----                                              ---- ------
> 	example-infr2-admin.domain.com                              1 Online, Local
> 	example-infr1-admin.domain.com                              2 Offline
>         /dev/mpath/quorum                                           0 Offline
> 
> 	[root at example-infr2 ~]# cman_tool status
> 	Version: 6.2.0
> 	Config Version: 239
> 	Cluster Name: example-infra
> 	Cluster Id: 42813
> 	Cluster Member: Yes
> 	Cluster Generation: 676844
> 	Membership state: Cluster-Member
> 	Nodes: 1
> 	Expected votes: 2
> 	Total votes: 1
> 	Quorum: 2 Activity blocked
> 	Active subsystems: 7
> 	Flags: 
> 	Ports Bound: 0  
> 	Node name: example-infr2-admin.domain.com
> 	Node ID: 1
> 	Multicast addresses: 239.192.167.228 
> 	Node addresses: 192.168.110.3 

You should check the output from cman_tool nodes. It appears that the
nodes are not seeing each other at all.

The first things I would check are iptables, node names resolves to the
correct ip addresses, selinux and eventually if the switch in between
the nodes support multicast.

Fabio