[Linux-cluster] qdiskd: Updated votes configuration not used even after restart

[Chrissie Caulfield]

Lon Hohberger wrote:
> Mrugesh Karnik wrote:
>> Hi,
>>
>> I have a three node cluster, of which, one node is perpetually
>> offline. I have the cluster configured with qdisk. Since the third
>> node is always offline, I configured the quorum settings for a two
>> node cluster:
>>
>> * 1 vote assigned to each node
>> * 1 vote assigned to qdisk
>> * expected votes set to 3
>>
>> The configuration worked perfectly well.
>>
>> I then needed to add a fourth node to the cluster, so I updated the
>> configuration as follows:
>>
>> * 1 vote assigned to each node
>> * 3 votes assigned to qdisk
>> * expected votes set to 4
>>
>> I then did a ccs_tool update.
>>
>> To have the value of qdisk votes updated, I then did the following:
>>
>> * cman_tool expected -e 3
>>
>> so as not to lose quorum while the qdisk was offline.
>>
>> I restarted qdiskd on each node in turn, yet the value of votes was
>> not updated.
>>
>> I then stopped qdiskd on _all_ nodes and then started it up on each
>> node one-
>> by-one. Yet, again, the value of votes assigned to the qdisk was not
>> updated.
>>
>> Here's the debug output from when qdiskd starts up:
>> May 19 13:36:52 eru qdiskd[18691]: <debug> 0 heuristics loaded
>> May 19 13:36:52 eru qdiskd[18691]: <debug> Quorum Daemon: 0
>> heuristics, 5 interval, 5 tko, 3 votes
>> May 19 13:36:52 eru qdiskd[18691]: <debug> Run Flags: 00000031
>> May 19 13:36:52 eru qdiskd[18692]: <info> Quorum Daemon Initializing
>> May 19 13:36:52 eru qdiskd[18692]: <debug> I/O Size: 512  Page Size: 4096
>> May 19 13:36:52 eru qdiskd[18692]: <debug> Permanently setting score
>> to 1/1
>> May 19 13:37:02 eru qdiskd[18692]: <debug> Node 2 is UP
>> May 19 13:37:13 eru qdiskd[18692]: <debug> Node 4 is UP
>> May 19 13:37:18 eru qdiskd[18692]: <info> Initial score 1/1
>> May 19 13:37:18 eru qdiskd[18692]: <info> Initialization complete
>> May 19 13:37:18 eru qdiskd[18692]: <notice> Score sufficient for
>> master operation (1/1; required=1); upgrading
>> May 19 13:37:28 eru qdiskd[18692]: <debug> Making bid for master
>> May 19 13:37:38 eru qdiskd[18692]: <info> Assuming master role
>>
>> Here's the output of cman_tool status, on the same node:
>> Version: 6.1.0
>> Config Version: 35
>> Cluster Name: dom0cluster
>> Cluster Id: 10036
>> Cluster Member: Yes
>> Cluster Generation: 784
>> Membership state: Cluster-Member
>> Nodes: 3
>> Expected votes: 4
>> Quorum device votes: 1
>> Total votes: 4
>> Quorum: 3
>> Active subsystems: 11
>> Flags: Dirty
>> Ports Bound: 0 11 177
>> Node name: eru.san.arda.geodesic.net
>> Node ID: 1
>> Multicast addresses: 229.192.0.1
>> Node addresses: 192.168.10.2
>>
>> The same problem is seen on all the nodes. The updated configuration
>> is read correctly by qdiskd, but once started, the updated votes value
>> isn't used.
>>
> 
> So, once registered, cman doesn't accept new registrations or changes to
> the registration - meaning that votes can't currently be changed.  This
> obviously would explain both behaviors you saw.
> 
> It looks like we could safely add a block to
> do_cmd_register_quorum_dev() function that does something like:
> 
>  - if a quorum device exists and it is being reregistered with the same
>    name, just change the votes and recalculate quorum

cman doesn't allow the votes to be changed without deregistering and
reregistering the quorum device.

I have checked the code and I can't see any reason why doing it this way
would fail, if register succeeds then it allocates a new node structure
for the qdisk and populates it from the parameters given.

Is it possible that qdisk might not unregister the qdisk  when it is
stopped under some circumstances ?


Chrissie