[Linux-cluster] Removing a node from a running cluster

[Jim Parsons]

Patrick Caulfield wrote:

>Pena, Francisco Javier wrote:
>
>>Hello,
>>
>>I am finding a strange cman behavior when removing a node from a running cluster. The starting point is:
>>
>>- 3 nodes running RHEL 4 U4, GFS 6.1	(1 vote per node)
>>- Quorum disk					(4 votes)
>>
>>I stop all cluster services on node 3, then modify the cluster.conf file to remove the node (and adjust the quorum disk votes to 3), and then "ccs_tool update" and "cman_tool version -r <new_version>". The cluster services keep running, however it looks like cman is not completely in sync with ccsd:
>>
>># ccs_tool lsnode
>>
>>Cluster name: TestCluster, config_version: 9
>>
>>Nodename                        Votes Nodeid Iface Fencetype
>>gfsnode1                           1    1          iLO_NODE1
>>gfsnode2                           1    2          iLO_NODE2
>>
>>
>># cman_tool nodes
>>
>>Node  Votes Exp Sts  Name
>>   0    4    0   M   /dev/emcpowera1
>>   1    1    3   M   gfsnode1
>>   2    1    3   M   gfsnode2
>>   3    1    3   X   gfsnode3
>>
>># cman_tool status
>>
>>Protocol version: 5.0.1
>>Config version: 9
>>Cluster name: TestCluster
>>Cluster ID: 62260
>>Cluster Member: Yes
>>Membership state: Cluster-Member
>>Nodes: 2
>>Expected_votes: 3
>>Total_votes: 6
>>Quorum: 4
>>Active subsystems: 9
>>Node name: gfsnode1
>>Node ID: 1
>>Node addresses: A.B.C.D
>>
>>CMAN still thinks the third node is part of the cluster, but has just stopped working. In addition to that, it is not updating the number of votes for the quorum disk. If I completely restart the cluster services on all nodes, I get the right information:
>>
>>- Correct votes for the quorum disk
>>- Third node dissappears
>>- The Expected_votes value is now 2
>>
>
>I can't comment on the behaviour of the quorum disk, but cman is behaving as expected. A node is NEVER removed from the internal
>lists of cman while any node of the cluster is till active. It is completely harmless in that state, the node simply remains
>permanently dead and expected votes is adjusted accordingly.
>
>
Patrick - isn't it also necessary to set a cman attribute for 
two-node='1' in the conf file? In order for cman to see this attribute, 
the entire cluster would need to be restarted.

Regards,

-Jim