[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [Linux-cluster] two node cluster with IP tiebreaker failed.



ext Kein He wrote:
> I think there is a problem, from "cman_tool status" shows:
>
> Nodes: 2
> Expected votes: 3
> Total votes: 2
>
>
> according to your cluster.conf , if all nodes and qdisk are online,
> the "Total votes" must be "3".  Probably "qdiskd" is not running, you
> can use " cman_tool nodes" to check if qdisk is working.
>
Yes, here is "cman_tool nodes" output:
Node  Sts   Inc   Joined               Name
   1   M    112   2009-02-25 03:05:19  as-1.localdomain
   2   M    104   2009-02-25 03:05:19  as-2.localdomain

A question is how to check whether qdisk is running ? and how to run it ?

Thanks.
>
>
>
> Mockey Chen wrote:
>> ext Mockey Chen wrote:
>>  
>>> ext Kein He wrote:
>>>      
>>>> Hi Mockey,
>>>>
>>>> Could you please attach the output from " cman_tool status " and "
>>>> cman_tool nodes -f" ?
>>>>
>>>>           
>>> Thanks your response.
>>>
>>> I try to run cman_tool status on as-2, but it hang, without output, and
>>> even Ctrl+C also no effect.
>>>       
>> I manually reboot as-1, and the problem solved.
>>
>> There is the output of cman_tool
>>
>> [root as-1 ~]# cman_tool status
>> Version: 6.1.0
>> Config Version: 19
>> Cluster Name: azerothcluster
>> Cluster Id: 20148
>> Cluster Member: Yes
>> Cluster Generation: 76
>> Membership state: Cluster-Member
>> Nodes: 2
>> Expected votes: 3
>> Total votes: 2
>> Quorum: 2 Active subsystems: 8
>> Flags: Dirty
>> Ports Bound: 0 177 Node name: as-1.localdomain
>> Node ID: 1
>> Multicast addresses: 239.192.78.3
>> Node addresses: 10.56.150.3
>> [root as-1 ~]# cman_tool status -f
>> Version: 6.1.0
>> Config Version: 19
>> Cluster Name: azerothcluster
>> Cluster Id: 20148
>> Cluster Member: Yes
>> Cluster Generation: 76
>> Membership state: Cluster-Member
>> Nodes: 2
>> Expected votes: 3
>> Total votes: 2
>> Quorum: 2 Active subsystems: 8
>> Flags: Dirty
>> Ports Bound: 0 177 Node name: as-1.localdomain
>> Node ID: 1
>> Multicast addresses: 239.192.78.3
>> Node addresses: 10.56.150.3
>>
>>
>> It seems cluster can not fence one of the node. How to solve it ?
>>
>>  
>>> I open a new window and can using ssh to as-2, but  after login,  I can
>>> not do anything, even a
>>> simple 'ls' command is hung.
>>>
>>> It seem the system keep alive but do not provide any service. Really
>>> bad.
>>>
>>> Any way to debug this issue ?
>>>      
>>>> Mockey Chen wrote:
>>>>          
>>>>> Hi,
>>>>>
>>>>> I have a two-nodes cluster, to avoid split-brain. I use ilo as fence
>>>>> device, IP tiebreaker. here is my /etc/cluster/cluster.conf
>>>>> <?xml version="1.0"?>
>>>>> <cluster alias="azerothcluster" config_version="19"
>>>>> name="azerothcluster">
>>>>>     <cman expected_votes="3" two_node="0"/>
>>>>>     <clusternodes>
>>>>>         <clusternode name="as-1.localdomain" nodeid="1" votes="1">
>>>>>             <fence>
>>>>>                 <method name="1">
>>>>>                     <device name="ilo1"/>
>>>>>                 </method>
>>>>>             </fence>
>>>>>         </clusternode>
>>>>>         <clusternode name="as-2.localdomain" nodeid="2" votes="1">
>>>>>             <fence>
>>>>>                 <method name="1">
>>>>>                     <device name="ilo2"/>
>>>>>                 </method>
>>>>>             </fence>
>>>>>         </clusternode>
>>>>>     </clusternodes>
>>>>>         <quorumd interval="1" tko="10" votes="1" label="pingtest">
>>>>>                 <heuristic program="ping 10.56.150.1 -c1 -t1"
>>>>> score="1"
>>>>> interval="2" tko="3"/>
>>>>>         </quorumd>
>>>>>     <fence_daemon post_fail_delay="0" post_join_delay="3"/>
>>>>>     <fencedevices>
>>>>>         <fencedevice agent="fence_ilo" hostname="10.56.154.18"
>>>>> login="power" name="ilo1" passwd="pass"/>
>>>>>         <fencedevice agent="fence_ilo" hostname="10.56.154.19"
>>>>> login="power" name="ilo2" passwd="pass"/>
>>>>>     </fencedevices>
>>>>> ...
>>>>> ...
>>>>>
>>>>> To test one node lost heartbeat case, I disable ethereal card
>>>>> (eth0) on
>>>>> as-1, I expect as-2 takeover services on as-1 and as-1 node reboot.
>>>>> The actual is as-1 lost connection to as-2.  as-2 detected it and
>>>>> try to
>>>>> re-construct cluster, but failed, here is the syslog form as-2
>>>>>
>>>>> Feb 24 21:25:35 as-2 openais[4139]: [TOTEM] The token was lost in the
>>>>> OPERATIONAL state.
>>>>> Feb 24 21:25:35 as-2 openais[4139]: [TOTEM] Receive multicast socket
>>>>> recv buffer size (288000 bytes).
>>>>> Feb 24 21:25:35 as-2 openais[4139]: [TOTEM] Transmit multicast socket
>>>>> send buffer size (262142 bytes).
>>>>> Feb 24 21:25:35 as-2 openais[4139]: [TOTEM] entering GATHER state
>>>>> from 2.
>>>>> Feb 24 21:25:40 as-2 openais[4139]: [TOTEM] entering GATHER state
>>>>> from 0.
>>>>> Feb 24 21:25:40 as-2 openais[4139]: [TOTEM] Creating commit token
>>>>> because I am the rep.
>>>>> Feb 24 21:25:40 as-2 openais[4139]: [TOTEM] Saving state aru 1f4 high
>>>>> seq received 1f4
>>>>> Feb 24 21:25:40 as-2 openais[4139]: [TOTEM] Storing new sequence
>>>>> id for
>>>>> ring 2c
>>>>> Feb 24 21:25:40 as-2 openais[4139]: [TOTEM] entering COMMIT state.
>>>>> Feb 24 21:25:40 as-2 openais[4139]: [TOTEM] entering RECOVERY state.
>>>>> Feb 24 21:25:40 as-2 openais[4139]: [TOTEM] position [0] member
>>>>> 10.56.150.4:
>>>>> Feb 24 21:25:40 as-2 openais[4139]: [TOTEM] previous ring seq 40 rep
>>>>> 10.56.150.3
>>>>> Feb 24 21:25:40 as-2 openais[4139]: [TOTEM] aru 1f4 high delivered
>>>>> 1f4
>>>>> received flag 1
>>>>>
>>>>> Message from syslogd@ at Tue Feb 24 21:25:40 2009 ...
>>>>> as-2 clurgmgrd[4194]: <emerg> #1: Quorum Dissolved Feb 24 21:25:40
>>>>> as-2
>>>>> openais[4139]: [TOTEM] Did not need to originate any messages in
>>>>> recovery.
>>>>> Feb 24 21:25:40 as-2 openais[4139]: [TOTEM] Sending initial ORF token
>>>>> Feb 24 21:25:40 as-2 openais[4139]: [CLM  ] CLM CONFIGURATION CHANGE
>>>>> Feb 24 21:25:40 as-2 openais[4139]: [CLM  ] New Configuration:
>>>>> Feb 24 21:25:40 as-2 clurgmgrd[4194]: <emerg> #1: Quorum Dissolved
>>>>> Feb 24 21:25:40 as-2 kernel: dlm: closing connection to node 1
>>>>> Feb 24 21:25:40 as-2 openais[4139]: [CLM  ]     r(0) ip(10.56.150.4)
>>>>> Feb 24 21:25:40 as-2 openais[4139]: [CLM  ] Members Left:
>>>>> Feb 24 21:25:40 as-2 openais[4139]: [CLM  ]     r(0) ip(10.56.150.3)
>>>>> Feb 24 21:25:40 as-2 openais[4139]: [CLM  ] Members Joined:
>>>>> Feb 24 21:25:40 as-2 openais[4139]: [CMAN ] quorum lost, blocking
>>>>> activity
>>>>> Feb 24 21:25:40 as-2 openais[4139]: [CLM  ] CLM CONFIGURATION CHANGE
>>>>> Feb 24 21:25:40 as-2 openais[4139]: [CLM  ] New Configuration:
>>>>> Feb 24 21:25:40 as-2 openais[4139]: [CLM  ]     r(0) ip(10.56.150.4)
>>>>> Feb 24 21:25:40 as-2 openais[4139]: [CLM  ] Members Left:
>>>>> Feb 24 21:25:40 as-2 openais[4139]: [CLM  ] Members Joined:
>>>>> Feb 24 21:25:40 as-2 openais[4139]: [SYNC ] This node is within the
>>>>> primary component and will provide service.
>>>>> Feb 24 21:25:40 as-2 ccsd[4130]: Cluster is not quorate.  Refusing
>>>>> connection.
>>>>> Feb 24 21:25:40 as-2 openais[4139]: [TOTEM] entering OPERATIONAL
>>>>> state.
>>>>> Feb 24 21:25:40 as-2 ccsd[4130]: Error while processing connect:
>>>>> Connection refused
>>>>> Feb 24 21:25:40 as-2 openais[4139]: [CLM  ] got nodejoin message
>>>>> 10.56.150.4
>>>>> Feb 24 21:25:40 as-2 ccsd[4130]: Invalid descriptor specified (-111).
>>>>> Feb 24 21:25:40 as-2 openais[4139]: [CPG  ] got joinlist message from
>>>>> node 2
>>>>> Feb 24 21:25:40 as-2 ccsd[4130]: Someone may be attempting something
>>>>> evil.
>>>>> Feb 24 21:25:40 as-2 ccsd[4130]: Error while processing get: Invalid
>>>>> request descriptor
>>>>> Feb 24 21:25:40 as-2 ccsd[4130]: Invalid descriptor specified (-111).
>>>>> Feb 24 21:25:41 as-2 ccsd[4130]: Someone may be attempting something
>>>>> evil.
>>>>> Feb 24 21:25:41 as-2 ccsd[4130]: Error while processing get: Invalid
>>>>> request descriptor
>>>>> Feb 24 21:25:41 as-2 ccsd[4130]: Invalid descriptor specified (-21).
>>>>> Feb 24 21:25:41 as-2 ccsd[4130]: Someone may be attempting something
>>>>> evil.
>>>>> Feb 24 21:25:41 as-2 ccsd[4130]: Error while processing disconnect:
>>>>> Invalid request descriptor
>>>>> Feb 24 21:25:41 as-2 avahi-daemon[3756]: Withdrawing address
>>>>> record for
>>>>> 10.56.150.144 on eth0.
>>>>> Feb 24 21:25:41 as-2 in.rdiscd[8641]: setsockopt (IP_ADD_MEMBERSHIP):
>>>>> Address already in use
>>>>> Feb 24 21:25:41 as-2 in.rdiscd[8641]: Failed joining addresse
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> I also found there are some errors in as-1's syslog
>>>>> Feb 25 11:27:09 as-1 clurgmgrd[4332]: <err> #52: Failed changing RG
>>>>> status
>>>>> Feb 25 11:27:09 as-1 clurgmgrd: [4332]: <warning> Link for eth0: Not
>>>>> detected
>>>>> Feb 25 11:27:09 as-1 clurgmgrd: [4332]: <warning> No link on eth0...
>>>>> ...
>>>>> Feb 25 11:27:36 as-1 ccsd[4268]: Unable to connect to cluster
>>>>> infrastructure after 30 seconds.
>>>>> ...
>>>>> Feb 25 11:28:06 as-1 ccsd[4268]: Unable to connect to cluster
>>>>> infrastructure after 60 seconds.
>>>>> ...
>>>>> Feb 25 11:28:06 as-1 ccsd[4268]: Unable to connect to cluster
>>>>> infrastructure after 90 seconds.
>>>>>
>>>>>
>>>>> any comment is appreciated!
>>>>>
>>>>> -- 
>>>>> Linux-cluster mailing list
>>>>> Linux-cluster redhat com
>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>                 
>>>> -- 
>>>> Linux-cluster mailing list
>>>> Linux-cluster redhat com
>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>
>>>>           
>>> -- 
>>> Linux-cluster mailing list
>>> Linux-cluster redhat com
>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>
>>>       
>>
>> -- 
>> Linux-cluster mailing list
>> Linux-cluster redhat com
>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>   
>
> -- 
> Linux-cluster mailing list
> Linux-cluster redhat com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]