[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [Linux-cluster] two node cluster with IP tiebreaker failed.



Unfortunately , you need a shared disk to run qdisk, it can not work in "diskless" mode right now.


ext Brett Cave wrote:
On Wed, Feb 25, 2009 at 11:45 AM, Mockey Chen <mockey chen nsn com> wrote:
ext Kein He wrote:
I think there is a problem, from "cman_tool status" shows:

Nodes: 2
Expected votes: 3
Total votes: 2


according to your cluster.conf , if all nodes and qdisk are online,
the "Total votes" must be "3".  Probably "qdiskd" is not running, you
can use " cman_tool nodes" to check if qdisk is working.

Yes, here is "cman_tool nodes" output:
Node  Sts   Inc   Joined               Name
  1   M    112   2009-02-25 03:05:19  as-1.localdomain
  2   M    104   2009-02-25 03:05:19  as-2.localdomain

A question is how to check whether qdisk is running ? and how to run it ?
[root blade3 ~]# service qdiskd status
qdiskd (pid 2832) is running...
[root blade3 ~]# pgrep qdisk -l
2832 qdiskd
[root blade3 ~]# cman_tool nodes
Node  Sts   Inc   Joined               Name
   0   M      0   2009-02-19 16:11:55  /dev/sda5     ## This is qdisk.
   1   M   1524   2009-02-20 22:27:32  blade1
   2   M   1552   2009-02-24 04:39:24  blade2
   3   M   1500   2009-02-19 16:11:03  blade3
   4   M   1516   2009-02-19 16:11:22  blade4

You can use "service qdisk start" to start it, or run it with
/usr/sbin/qdisk -Q if you dont have the init script. If you installed
from rpm on a rh type distro, then the script should be there.

REgards,
brett
I try to use "service qdiskd start", but it failed:
[root as-2 ~]# service qdiskd start
Starting the Quorum Disk Daemon:                           [FAILED]
[root as-2 ~]# tail /var/log/messages
...
Feb 26 09:19:40 as-2 qdiskd[14707]: <crit> Unable to match label
'testing' to any device
Feb 26 09:19:46 as-2 clurgmgrd[4032]: <notice> Reconfiguring

Here is my qdisk configuration, I copy it from "man qdisk":
        <quorumd interval="1" tko="10" votes="1" label="testing">
                <heuristic program="ping 10.56.150.1 -c1 -t1" score="1"
interval="2" tko="3"/>
        </quorumd>

How to map label to device. Note: I did not have any shared storage.

Thanks.
Mockey Chen wrote:
ext Mockey Chen wrote:

ext Kein He wrote:

Hi Mockey,

Could you please attach the output from " cman_tool status " and "
cman_tool nodes -f" ?


Thanks your response.

I try to run cman_tool status on as-2, but it hang, without output, and
even Ctrl+C also no effect.

I manually reboot as-1, and the problem solved.

There is the output of cman_tool

[root as-1 ~]# cman_tool status
Version: 6.1.0
Config Version: 19
Cluster Name: azerothcluster
Cluster Id: 20148
Cluster Member: Yes
Cluster Generation: 76
Membership state: Cluster-Member
Nodes: 2
Expected votes: 3
Total votes: 2
Quorum: 2 Active subsystems: 8
Flags: Dirty
Ports Bound: 0 177 Node name: as-1.localdomain
Node ID: 1
Multicast addresses: 239.192.78.3
Node addresses: 10.56.150.3
[root as-1 ~]# cman_tool status -f
Version: 6.1.0
Config Version: 19
Cluster Name: azerothcluster
Cluster Id: 20148
Cluster Member: Yes
Cluster Generation: 76
Membership state: Cluster-Member
Nodes: 2
Expected votes: 3
Total votes: 2
Quorum: 2 Active subsystems: 8
Flags: Dirty
Ports Bound: 0 177 Node name: as-1.localdomain
Node ID: 1
Multicast addresses: 239.192.78.3
Node addresses: 10.56.150.3


It seems cluster can not fence one of the node. How to solve it ?


I open a new window and can using ssh to as-2, but  after login,  I can
not do anything, even a
simple 'ls' command is hung.

It seem the system keep alive but do not provide any service. Really
bad.

Any way to debug this issue ?

Mockey Chen wrote:

Hi,

I have a two-nodes cluster, to avoid split-brain. I use ilo as fence
device, IP tiebreaker. here is my /etc/cluster/cluster.conf
<?xml version="1.0"?>
<cluster alias="azerothcluster" config_version="19"
name="azerothcluster">
    <cman expected_votes="3" two_node="0"/>
    <clusternodes>
        <clusternode name="as-1.localdomain" nodeid="1" votes="1">
            <fence>
                <method name="1">
                    <device name="ilo1"/>
                </method>
            </fence>
        </clusternode>
        <clusternode name="as-2.localdomain" nodeid="2" votes="1">
            <fence>
                <method name="1">
                    <device name="ilo2"/>
                </method>
            </fence>
        </clusternode>
    </clusternodes>
        <quorumd interval="1" tko="10" votes="1" label="pingtest">
                <heuristic program="ping 10.56.150.1 -c1 -t1"
score="1"
interval="2" tko="3"/>
        </quorumd>
    <fence_daemon post_fail_delay="0" post_join_delay="3"/>
    <fencedevices>
        <fencedevice agent="fence_ilo" hostname="10.56.154.18"
login="power" name="ilo1" passwd="pass"/>
        <fencedevice agent="fence_ilo" hostname="10.56.154.19"
login="power" name="ilo2" passwd="pass"/>
    </fencedevices>
...
...

To test one node lost heartbeat case, I disable ethereal card
(eth0) on
as-1, I expect as-2 takeover services on as-1 and as-1 node reboot.
The actual is as-1 lost connection to as-2.  as-2 detected it and
try to
re-construct cluster, but failed, here is the syslog form as-2

Feb 24 21:25:35 as-2 openais[4139]: [TOTEM] The token was lost in the
OPERATIONAL state.
Feb 24 21:25:35 as-2 openais[4139]: [TOTEM] Receive multicast socket
recv buffer size (288000 bytes).
Feb 24 21:25:35 as-2 openais[4139]: [TOTEM] Transmit multicast socket
send buffer size (262142 bytes).
Feb 24 21:25:35 as-2 openais[4139]: [TOTEM] entering GATHER state
from 2.
Feb 24 21:25:40 as-2 openais[4139]: [TOTEM] entering GATHER state
from 0.
Feb 24 21:25:40 as-2 openais[4139]: [TOTEM] Creating commit token
because I am the rep.
Feb 24 21:25:40 as-2 openais[4139]: [TOTEM] Saving state aru 1f4 high
seq received 1f4
Feb 24 21:25:40 as-2 openais[4139]: [TOTEM] Storing new sequence
id for
ring 2c
Feb 24 21:25:40 as-2 openais[4139]: [TOTEM] entering COMMIT state.
Feb 24 21:25:40 as-2 openais[4139]: [TOTEM] entering RECOVERY state.
Feb 24 21:25:40 as-2 openais[4139]: [TOTEM] position [0] member
10.56.150.4:
Feb 24 21:25:40 as-2 openais[4139]: [TOTEM] previous ring seq 40 rep
10.56.150.3
Feb 24 21:25:40 as-2 openais[4139]: [TOTEM] aru 1f4 high delivered
1f4
received flag 1

Message from syslogd@ at Tue Feb 24 21:25:40 2009 ...
as-2 clurgmgrd[4194]: <emerg> #1: Quorum Dissolved Feb 24 21:25:40
as-2
openais[4139]: [TOTEM] Did not need to originate any messages in
recovery.
Feb 24 21:25:40 as-2 openais[4139]: [TOTEM] Sending initial ORF token
Feb 24 21:25:40 as-2 openais[4139]: [CLM  ] CLM CONFIGURATION CHANGE
Feb 24 21:25:40 as-2 openais[4139]: [CLM  ] New Configuration:
Feb 24 21:25:40 as-2 clurgmgrd[4194]: <emerg> #1: Quorum Dissolved
Feb 24 21:25:40 as-2 kernel: dlm: closing connection to node 1
Feb 24 21:25:40 as-2 openais[4139]: [CLM  ]     r(0) ip(10.56.150.4)
Feb 24 21:25:40 as-2 openais[4139]: [CLM  ] Members Left:
Feb 24 21:25:40 as-2 openais[4139]: [CLM  ]     r(0) ip(10.56.150.3)
Feb 24 21:25:40 as-2 openais[4139]: [CLM  ] Members Joined:
Feb 24 21:25:40 as-2 openais[4139]: [CMAN ] quorum lost, blocking
activity
Feb 24 21:25:40 as-2 openais[4139]: [CLM  ] CLM CONFIGURATION CHANGE
Feb 24 21:25:40 as-2 openais[4139]: [CLM  ] New Configuration:
Feb 24 21:25:40 as-2 openais[4139]: [CLM  ]     r(0) ip(10.56.150.4)
Feb 24 21:25:40 as-2 openais[4139]: [CLM  ] Members Left:
Feb 24 21:25:40 as-2 openais[4139]: [CLM  ] Members Joined:
Feb 24 21:25:40 as-2 openais[4139]: [SYNC ] This node is within the
primary component and will provide service.
Feb 24 21:25:40 as-2 ccsd[4130]: Cluster is not quorate.  Refusing
connection.
Feb 24 21:25:40 as-2 openais[4139]: [TOTEM] entering OPERATIONAL
state.
Feb 24 21:25:40 as-2 ccsd[4130]: Error while processing connect:
Connection refused
Feb 24 21:25:40 as-2 openais[4139]: [CLM  ] got nodejoin message
10.56.150.4
Feb 24 21:25:40 as-2 ccsd[4130]: Invalid descriptor specified (-111).
Feb 24 21:25:40 as-2 openais[4139]: [CPG  ] got joinlist message from
node 2
Feb 24 21:25:40 as-2 ccsd[4130]: Someone may be attempting something
evil.
Feb 24 21:25:40 as-2 ccsd[4130]: Error while processing get: Invalid
request descriptor
Feb 24 21:25:40 as-2 ccsd[4130]: Invalid descriptor specified (-111).
Feb 24 21:25:41 as-2 ccsd[4130]: Someone may be attempting something
evil.
Feb 24 21:25:41 as-2 ccsd[4130]: Error while processing get: Invalid
request descriptor
Feb 24 21:25:41 as-2 ccsd[4130]: Invalid descriptor specified (-21).
Feb 24 21:25:41 as-2 ccsd[4130]: Someone may be attempting something
evil.
Feb 24 21:25:41 as-2 ccsd[4130]: Error while processing disconnect:
Invalid request descriptor
Feb 24 21:25:41 as-2 avahi-daemon[3756]: Withdrawing address
record for
10.56.150.144 on eth0.
Feb 24 21:25:41 as-2 in.rdiscd[8641]: setsockopt (IP_ADD_MEMBERSHIP):
Address already in use
Feb 24 21:25:41 as-2 in.rdiscd[8641]: Failed joining addresse




I also found there are some errors in as-1's syslog
Feb 25 11:27:09 as-1 clurgmgrd[4332]: <err> #52: Failed changing RG
status
Feb 25 11:27:09 as-1 clurgmgrd: [4332]: <warning> Link for eth0: Not
detected
Feb 25 11:27:09 as-1 clurgmgrd: [4332]: <warning> No link on eth0...
...
Feb 25 11:27:36 as-1 ccsd[4268]: Unable to connect to cluster
infrastructure after 30 seconds.
...
Feb 25 11:28:06 as-1 ccsd[4268]: Unable to connect to cluster
infrastructure after 60 seconds.
...
Feb 25 11:28:06 as-1 ccsd[4268]: Unable to connect to cluster
infrastructure after 90 seconds.


any comment is appreciated!

--
Linux-cluster mailing list
Linux-cluster redhat com
https://www.redhat.com/mailman/listinfo/linux-cluster

--
Linux-cluster mailing list
Linux-cluster redhat com
https://www.redhat.com/mailman/listinfo/linux-cluster


--
Linux-cluster mailing list
Linux-cluster redhat com
https://www.redhat.com/mailman/listinfo/linux-cluster


--
Linux-cluster mailing list
Linux-cluster redhat com
https://www.redhat.com/mailman/listinfo/linux-cluster

--
Linux-cluster mailing list
Linux-cluster redhat com
https://www.redhat.com/mailman/listinfo/linux-cluster

--
Linux-cluster mailing list
Linux-cluster redhat com
https://www.redhat.com/mailman/listinfo/linux-cluster

--
Linux-cluster mailing list
Linux-cluster redhat com
https://www.redhat.com/mailman/listinfo/linux-cluster


--
Linux-cluster mailing list
Linux-cluster redhat com
https://www.redhat.com/mailman/listinfo/linux-cluster


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]