[Linux-cluster] Additional node "Cluster membership rejected"

[Thomas Kofler]

Hi,

we are running a 4 node cluster successfully. Now we try to join an additional 
node - but it fails.

We upgraded the cluster.conf file to reflect the new node.

[root at www5 ~]# ccs_tool update /etc/cluster/cluster.conf
Config file updated from version 10 to 11
Update complete.

cluster.conf was checked and is synchron on all nodes, hosts files are also 
fine.

When we try to join the new node gfsserver2
[root at gfsserver2 cluster]# cman_tool join

we get 

[root at gfsserver2 cluster]# CMAN: Cluster membership rejected

And the interesting part is:


Oct  7 13:51:59 gfsserver ccsd[415]: Update of cluster.conf complete (version 
10 -> 11).
Oct  7 13:52:00 gfsserver kernel: CMAN: Join request from gfsserver2.devcon.cc 
rejected, config version local 10 remote 11

Why do the 4 existing nodes not check, that they also have version 11 in use ?

Or do we have to "reload" anything additionally to the ccs_tool update 
command ?

Thanks in advance,
Regards,
Thomas

Oct  7 13:51:34 gfsserver2 kernel: GFS 2.6.11.8-20050601.152643.FC4.9 (built 
Jul 18 2005 10:42:24) installed
Oct  7 13:51:39 gfsserver2 kernel: CMAN 2.6.11.5-20050601.152643.FC4.9 (built 
Jul 18 2005 10:27:35) installed
Oct  7 13:51:39 gfsserver2 kernel: NET: Registered protocol family 30
Oct  7 13:51:39 gfsserver2 kernel: DLM 2.6.11.5-20050601.152643.FC4.10 (built 
Jul 18 2005 10:34:42) installed
Oct  7 13:51:39 gfsserver2 kernel: Lock_DLM (built Jul 18 2005 10:42:18) 
installed
Oct  7 13:51:49 gfsserver2 ccsd[839]: Starting ccsd 1.0.0:
Oct  7 13:51:49 gfsserver2 ccsd[839]:  Built: Jun 16 2005 10:45:39
Oct  7 13:51:49 gfsserver2 ccsd[839]:  Copyright (C) Red Hat, Inc.  2004  All 
rights reserved.
Oct  7 13:51:49 gfsserver2 ccsd[839]:   IP Protocol:: IPv4 only
Oct  7 13:51:59 gfsserver2 ccsd[839]: cluster.conf (cluster name = 
devconcluster, version = 11) found.
Oct  7 13:51:59 gfsserver2 ccsd[839]: Remote copy of cluster.conf is from 
quorate node.
Oct  7 13:51:59 gfsserver2 ccsd[839]:  Local version # : 11
Oct  7 13:51:59 gfsserver2 ccsd[839]:  Remote version #: 11
Oct  7 13:51:59 gfsserver2 kernel: CMAN: Waiting to join or form a Linux-
cluster
Oct  7 13:51:59 gfsserver2 ccsd[839]: Connected to cluster infrastruture via: 
CMAN/SM Plugin v1.1.2
Oct  7 13:51:59 gfsserver2 ccsd[839]: Initial status:: Inquorate
Oct  7 13:52:00 gfsserver2 kernel: CMAN: sending membership request
Oct  7 13:52:00 gfsserver2 kernel: CMAN: Cluster membership rejected
Oct  7 13:52:00 gfsserver2 ccsd[839]: Cluster manager shutdown.  Attemping to 
reconnect...
Oct  7 13:52:20 gfsserver2 ccsd[839]: Unable to connect to cluster 
infrastructure after 30 seconds.
Oct  7 13:52:50 gfsserver2 ccsd[839]: Unable to connect to cluster 
infrastructure after 60 seconds.
Oct  7 13:53:20 gfsserver2 ccsd[839]: Unable to connect to cluster 
infrastructure after 90 seconds.
Oct  7 13:53:51 gfsserver2 ccsd[839]: Unable to connect to cluster 
infrastructure after 120 seconds.
Oct  7 13:53:58 gfsserver2 ccsd[839]: Remote copy of cluster.conf is from 
quorate node.

<?xml version="1.0"?>
<cluster name="cluster" config_version="11">
<cman expected_votes="3">
 </cman>
    <clusternodes>
    <clusternode name="www5.domainxy.cc" domain="fedora0" votes="1">
     <fence>
      <method name="single">
       <device name="human" ipaddr="192.xxx.2.25"/>
     </method>
    </fence>
   </clusternode>
   <clusternode name="www3.domainxy.cc" domain="fedora1" votes="1">
    <fence>
     <method name="single">
       <device name="human" ipaddr="192.xxx.2.23"/>
     </method>
    </fence>
    </clusternode>
   <clusternode name="www4.domainxy.cc" domain="fedora2" votes="1">
    <fence>
     <method name="single">
       <device name="human" ipaddr="192.xxx.2.24"/>
     </method>
    </fence>
  </clusternode>
  <clusternode name="gfsserver.domainxy.cc" domain="fedora3" votes="1">
    <fence>
     <method name="single">
       <device name="human" ipaddr="192.xxx.2.33"/>
     </method>
    </fence>
  </clusternode>
 <clusternode name="gfsserver2.domainxy.cc" domain="fedora4" votes="1">
    <fence>
     <method name="single">
       <device name="human" ipaddr="192.xxx.2.34"/>
     </method>
    </fence>
  </clusternode>
 </clusternodes>
<fence_devices>
 <fence_device name="human" agent="fence_manual"/>
</fence_devices>
</cluster>