[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

RE: [Linux-cluster] 3 node cluster problems



Title: RE: [Linux-cluster] 3 node cluster problems

If you are configuring your cluster by system-config-cluster then no need to run ricci/luci. Ricci/luci needed for configuring the cluster using conga. You can configure in either ways.

On seeing your clustat command outputs, it seems cluster is partitioned (spilt brain) into 2 sub clusters [Sub1- (csarcsys1-eth0, csarcsys2-eth0) 2- csarcsys3-eth0]. Without a quorum device you can more often face this situation. To avoid this you can configure a quorum device with a heuristic like ping message. Use the link (http://www.redhatmagazine.com/2007/12/19/enhancing-cluster-quorum-with-qdisk/) for configuring a quorum disk in RHCS.

Thanks,

S

-----Original Message-----
From: linux-cluster-bounces redhat com [mailto:linux-cluster-bounces redhat com] On Behalf Of Dalton, Maurice
Sent: Tuesday, March 25, 2008 5:18 PM
To: linux clustering
Subject: RE: [Linux-cluster] 3 node cluster problems

Still no change. Same as below.

I completely rebuilt the cluster using system-config-cluster

The Cluster software was installed from rhn, luci and ricci are running.

This is the new config file and it has been copied to the 2 other

systems



[root csarcsys1-eth0 cluster]# more cluster.conf

<?xml version="1.0"?>

<cluster config_version="5" name="csarcsys5">

        <fence_daemon post_fail_delay="0" post_join_delay="3"/>

        <clusternodes>

                <clusternode name="csarcsys1-eth0" nodeid="1" votes="1">

                        <fence/>

                </clusternode>

                <clusternode name="csarcsys2-eth0" nodeid="2" votes="1">

                        <fence/>

                </clusternode>

                <clusternode name="csarcsys3-eth0" nodeid="3" votes="1">

                        <fence/>

                </clusternode>

        </clusternodes>

        <cman/>

        <fencedevices/>

        <rm>

                <failoverdomains>

                        <failoverdomain name="csarcsysfo" ordered="0"

restricted="1">

                                <failoverdomainnode

name="csarcsys1-eth0" priority="1"/>

                                <failoverdomainnode

name="csarcsys2-eth0" priority="1"/>

                                <failoverdomainnode

name="csarcsys3-eth0" priority="1"/>

                        </failoverdomain>

                </failoverdomains>

                <resources>

                        <ip address="172.xx.xx.xxx" monitor_link="1"/>

                        <fs device="/dev/sdc1" force_fsck="0"

force_unmount="1" fsid="57739" fstype="ext3" mountpo

int="/csarc-test" name="csarcsys-fs" options="rw" self_fence="0"/>

                </resources>

        </rm>

</cluster>

-----Original Message-----

From: linux-cluster-bounces redhat com

[mailto:linux-cluster-bounces redhat com] On Behalf Of Bennie Thomas

Sent: Monday, March 24, 2008 4:17 PM

To: linux clustering

Subject: Re: [Linux-cluster] 3 node cluster problems

Did you load the Cluster software via Conga or manually ? You would have

had to load

luci on one node and ricci on all three.

Try copying the modified /etc/cluster/cluster.conf from csarcsys1 to the

other two nodes.

Make sure you can ping the private interface to/from all nodes and

reboot. If this does not work

post your /etc/cluster/cluster.conf file again.


Dalton, Maurice wrote:

> Yes

> I also rebooted again just now to be sure.

>

>

> -----Original Message-----

> From: linux-cluster-bounces redhat com

> [mailto:linux-cluster-bounces redhat com] On Behalf Of Bennie Thomas

> Sent: Monday, March 24, 2008 3:33 PM

> To: linux clustering

> Subject: Re: [Linux-cluster] 3 node cluster problems

>

> When you changed the nodenames in the /etc/lcuster/cluster.conf and

made

>

> sure the /etc/hosts

> file had the correct nodenames (Ie. 10.0.0.100  csarcsys1-eth0  

> csarcsys1-eth0.xxxx.xxxx.xxx.)

> Did you reboot all the nodes at the sametime ?

>

> Dalton, Maurice wrote:

>  

>> No luck. It seems as if csarcsys3 thinks its in his own cluster

>> I renamed all config files and rebuilt from system-config-cluster

>>

>> Clustat command from csarcsys3

>>

>>

>> [root csarcsys3-eth0 cluster]# clustat

>> msg_open: No such file or directory

>> Member Status: Inquorate

>>

>>   Member Name                        ID   Status

>>   ------ ----                        ---- ------

>>   csarcsys1-eth0                        1 Offline

>>   csarcsys2-eth0                        2 Offline

>>   csarcsys3-eth0                        3 Online, Local

>>

>> clustat command from csarcsys2

>>

>> [root csarcsys2-eth0 cluster]# clustat

>> msg_open: No such file or directory

>> Member Status: Quorate

>>

>>   Member Name                        ID   Status

>>   ------ ----                        ---- ------

>>   csarcsys1-eth0                        1 Online

>>   csarcsys2-eth0                        2 Online, Local

>>   csarcsys3-eth0                        3 Offline

>>

>>

>> -----Original Message-----

>> From: linux-cluster-bounces redhat com

>> [mailto:linux-cluster-bounces redhat com] On Behalf Of Bennie Thomas

>> Sent: Monday, March 24, 2008 2:25 PM

>> To: linux clustering

>> Subject: Re: [Linux-cluster] 3 node cluster problems

>>

>> You will also, need to make sure the clustered nodenames are in your

>> /etc/hosts file.

>> Also, make sure your cluster network interface is up on all nodes and

>> that the

>> /etc/cluster/cluster.conf are the same on all nodes.

>>

>>

>>

>> Dalton, Maurice wrote:

>>  

>>    

>>> The last post is incorrect.

>>>

>>> Fence is still hanging at start up.

>>>

>>> Here's another log message.

>>>

>>> Mar 24 19:03:14 csarcsys3-eth0 ccsd[6425]: Error while processing

>>> connect: Connection refused

>>>

>>> Mar 24 19:03:15 csarcsys3-eth0 dlm_controld[6453]: connect to ccs

>>> error -111, check ccsd or cluster status

>>>

>>> *From:* linux-cluster-bounces redhat com

>>> [mailto:linux-cluster-bounces redhat com] *On Behalf Of *Bennie

>>>      

> Thomas

>  

>>> *Sent:* Monday, March 24, 2008 11:22 AM

>>> *To:* linux clustering

>>> *Subject:* Re: [Linux-cluster] 3 node cluster problems

>>>

>>> try removing the fully qualified hostname from the cluster.conf

file.

>>>

>>>

>>> Dalton, Maurice wrote:

>>>

>>> I have NO fencing equipment

>>>

>>> I have been task to setup a 3 node cluster

>>>

>>> Currently I have having problems getting cman(fence) to start

>>>

>>> Fence will try to start up during cman start up but will fail

>>>

>>> I tried to run /sbin/fenced -D - I get the following

>>>

>>> 1206373475 cman_init error 0 111

>>>

>>> Here's my cluster.conf file

>>>

>>> <?xml version="1.0"?>

>>>

>>> <cluster alias="csarcsys51" config_version="26" name="csarcsys51">

>>>

>>> <fence_daemon clean_start="0" post_fail_delay="0"

>>>    

>>>      

>> post_join_delay="3"/>

>>  

>>    

>>> <clusternodes>

>>>

>>> <clusternode name="csarcsys1-eth0.xxx.xxxx.nasa.gov" nodeid="1"

>>>    

>>>      

>> votes="1">

>>  

>>    

>>> <fence/>

>>>

>>> </clusternode>

>>>

>>> <clusternode name="csarcsys2-eth0.xxx.xxxx.nasa.gov" nodeid="2"

>>>    

>>>      

>> votes="1">

>>  

>>    

>>> <fence/>

>>>

>>> </clusternode>

>>>

>>> <clusternode name="csarcsys3-eth0.xxx.xxxxnasa.gov" nodeid="3"

>>>    

>>>      

>> votes="1">

>>  

>>    

>>> <fence/>

>>>

>>> </clusternode>

>>>

>>> </clusternodes>

>>>

>>> <cman/>

>>>

>>> <fencedevices/>

>>>

>>> <rm>

>>>

>>> <failoverdomains>

>>>

>>> <failoverdomain name="csarcsys-fo" ordered="1" restricted="0">

>>>

>>> <failoverdomainnode name="csarcsys1-eth0.xxx.xxxx.nasa.gov"

>>>    

>>>      

>> priority="1"/>

>>  

>>    

>>> <failoverdomainnode name="csarcsys2-eth0.xxx.xxxx.nasa.gov"

>>>    

>>>      

>> priority="1"/>

>>  

>>    

>>> <failoverdomainnode name="csarcsys2-eth0.xxx.xxxx.nasa.gov"

>>>    

>>>      

>> priority="1"/>

>>  

>>    

>>> </failoverdomain>

>>>

>>> </failoverdomains>

>>>

>>> <resources>

>>>

>>> <ip address="xxx.xxx.xxx.xxx" monitor_link="1"/>

>>>

>>> <fs device="/dev/sdc1" force_fsck="0" force_unmount="1" fsid="57739"

>>> fstype="ext3" mountpo

>>>

>>> int="/csarc-test" name="csarcsys-fs" options="rw" self_fence="0"/>

>>>

>>> <nfsexport name="csarcsys-export"/>

>>>

>>> <nfsclient name="csarcsys-nfs-client" options="no_root_squash,rw"

>>> path="/csarc-test" targe

>>>

>>> t="xxx.xxx.xxx.*"/>

>>>

>>> </resources>

>>>

>>> </rm>

>>>

>>> </cluster>

>>>

>>> Messages from the logs

>>>

>>> ar 24 13:24:19 csarcsys2-eth0 ccsd[24888]: Cluster is not quorate.

>>> Refusing connection.

>>>

>>> Mar 24 13:24:19 csarcsys2-eth0 ccsd[24888]: Error while processing

>>> connect: Connection refused

>>>

>>> Mar 24 13:24:20 csarcsys2-eth0 ccsd[24888]: Cluster is not quorate.

>>> Refusing connection.

>>>

>>> Mar 24 13:24:20 csarcsys2-eth0 ccsd[24888]: Error while processing

>>> connect: Connection refused

>>>

>>> Mar 24 13:24:21 csarcsys2-eth0 ccsd[24888]: Cluster is not quorate.

>>> Refusing connection.

>>>

>>> Mar 24 13:24:21 csarcsys2-eth0 ccsd[24888]: Error while processing

>>> connect: Connection refused

>>>

>>> Mar 24 13:24:22 csarcsys2-eth0 ccsd[24888]: Cluster is not quorate.

>>> Refusing connection.

>>>

>>> Mar 24 13:24:22 csarcsys2-eth0 ccsd[24888]: Error while processing

>>> connect: Connection refused

>>>

>>> Mar 24 13:24:23 csarcsys2-eth0 ccsd[24888]: Cluster is not quorate.

>>> Refusing connection.

>>>

>>> Mar 24 13:24:23 csarcsys2-eth0 ccsd[24888]: Error while processing

>>> connect: Connection refused

>>>

>>> 

>>>

>>>    

>>>      

>

------------------------------------------------------------------------

>  

>>  

>>    

>>>  

>>> 

>>> --

>>> Linux-cluster mailing list

>>> Linux-cluster redhat com <mailto:Linux-cluster redhat com>

>>> https://www.redhat.com/mailman/listinfo/linux-cluster

>>>

>>>

>>>    

>>>      

>

------------------------------------------------------------------------

>  

>>  

>>    

>>> --

>>> Linux-cluster mailing list

>>> Linux-cluster redhat com

>>> https://www.redhat.com/mailman/listinfo/linux-cluster

>>>    

>>>      

>>

>> --

>> Linux-cluster mailing list

>> Linux-cluster redhat com

>> https://www.redhat.com/mailman/listinfo/linux-cluster

>>

>> --

>> Linux-cluster mailing list

>> Linux-cluster redhat com

>> https://www.redhat.com/mailman/listinfo/linux-cluster

>>  

>>    

>

>

> --

> Linux-cluster mailing list

> Linux-cluster redhat com

> https://www.redhat.com/mailman/listinfo/linux-cluster

>

> --

> Linux-cluster mailing list

> Linux-cluster redhat com

> https://www.redhat.com/mailman/listinfo/linux-cluster

>  


--

Linux-cluster mailing list

Linux-cluster redhat com

https://www.redhat.com/mailman/listinfo/linux-cluster

--

Linux-cluster mailing list

Linux-cluster redhat com

https://www.redhat.com/mailman/listinfo/linux-cluster


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]