[Linux-cluster] Starting two-node cluster with only one node

Fri Jul 17 06:41:41 UTC 2009

Thanks for the tip. It helped by stopping each node kicking each other, as per the logs, but still I have a split brain status. 

On node01

# /usr/sbin/cman_tool nodes
Node  Sts   Inc   Joined               Name
   1   M    680   2009-07-17 00:30:42  node01.company.com
   2   X      0                        node02.company.com

# /usr/sbin/clustat 
Cluster Status for GFSCluster @ Fri Jul 17 01:01:09 2009
Member Status: Quorate

 Member Name                             ID   Status
 ------ ----                             ---- ------
 node01.company.com                         1 Online, Local
 node02.company.com                         2 Offline

On node02

# /usr/sbin/cman_tool nodes
Node  Sts   Inc   Joined               Name
   1   X      0                        node01.company.com
   2   M    676   2009-07-17 00:30:43  node02.company.com

# /usr/sbin/clustat
Cluster Status for GFSCluster @ Fri Jul 17 01:01:22 2009
Member Status: Quorate

 Member Name                             ID   Status
 ------ ----                             ---- ------
 node01.company.com                         1 Offline
 node02.company.com                         2 Online, Local

Another thing that I have noticed,

1. Start node01 with only itself as the member of the cluster
2. Update cluster.conf to have node02 as an additional member
3. Start node02

Yields both nodes being quorate (split brain) but only node02 tries to fence out node01. After some time, clustat will yield both of them being in the same cluster. Then I will be starting clvmd on node02 but will not be successful. After trying to start the clvmd service, clustat will yield split brain again. 

Are there some troubleshootings that I should be doing?

--- On Thu, 7/16/09, Aaron Benner <tfrumbacher at gmail.com> wrote:

> From: Aaron Benner <tfrumbacher at gmail.com>
> Subject: Re: [Linux-cluster] Starting two-node cluster with only one node
> To: "linux clustering" <linux-cluster at redhat.com>
> Date: Thursday, 16 July, 2009, 10:04 PM
> Have you tried setting the
> "post_join_delay" value in the <fence_daemon ...>
> declaration to -1?
> 
> <fence_daemon clean_start="0" post_fail_delay="0"
> post_join_delay="-1" />
> 
> This is a hint I picked up from the fenced man page section
> on avoiding boot time fencing.  It tells fenced to wait
> until all of the nodes have joined the cluster before
> starting up.  We use this on a couple of 2 node
> clusters (with qdisk) to allow them to start up without the
> first node to grab the quorum disk fencing the other node.
> 
> --Aaron
> 
> On Jul 16, 2009, at 12:16 AM, Abed-nego G. Escobal, Jr.
> wrote:
> 
> > 
> > 
> > Tried it and now the two node cluster is running with
> only one node. My problem right now is how to force the
> second node to join the first node's cluster. Right now it
> is creating its own cluster and trying to fence the first
> node. I tried cman_tool leave on the second node but I got
> > 
> > cman_tool: Error leaving cluster: Device or resource
> busy
> > 
> > clvmd and gfs are not running on the second node. What
> is running on the second node is cman. When I did
> > 
> > service cman start
> > 
> > It took 5 approximately 5 minutes before I got the
> [ok] meassage. Am I missing something here? Not doing right?
> Should be doing something?
> > 
> > 
> > --- On Thu, 7/16/09, Abed-nego G. Escobal, Jr. <abednegoyulo at yahoo.com>
> wrote:
> > 
> >> From: Abed-nego G. Escobal, Jr. <abednegoyulo at yahoo.com>
> >> Subject: [Linux-cluster] Starting two-node cluster
> with only one node
> >> To: "linux clustering" <linux-cluster at redhat.com>
> >> Date: Thursday, 16 July, 2009, 10:46 AM
> >> 
> >> Using the config file below
> >> 
> >> <?xml version="1.0"?>
> >> <cluster name="GFSCluster"
> config_version="5">
> >> <cman expected_votes="1" two_node="1"/>
> >>   <clusternodes><clusternode
> >> name="node01.company.com" votes="1"
> >> nodeid="1"><fence><method
> >> name="single"><device
> >>
> name="node01_ipmi"/></method></fence></clusternode><clusternode
> >> name="node02.company.com" votes="1"
> >> nodeid="2"><fence><method
> >> name="single"><device
> >>
> name="node02_ipmi"/></method></fence></clusternode></clusternodes>
> >>   <fencedevices><fencedevice
> >> name="node01_ipmi" agent="fence_ipmilan"
> ipaddr="10.1.0.5"
> >> login="root"
> passwd="********"/><fencedevice
> >> name="node02_ipmi" agent="fence_ipmilan"
> ipaddr="10.1.0.7"
> >> login="root"
> passwd="********"/></fencedevices>
> >>   <rm>
> >>     <failoverdomains/>
> >>     <resources/>
> >>   </rm>
> >> </cluster>
> >> 
> >> Is it possible to start the cluster by only
> bringing up one
> >> node? The reason why I asked is because currently
> bringing
> >> them up together produces a split brain, each of
> them member
> >> of the cluster GFSCluster of their own fencing
> each other.
> >> My plan is to bring up only one node to create a
> quorum then
> >> bring the other one up and manually join it to the
> existing
> >> cluster.
> >> 
> >> I have already don the start_clean approach but it
> seems it
> >> does not work.
> >> 
> >> 
> >> 
> >> 
> >> --
> >> Linux-cluster mailing list
> >> Linux-cluster at redhat.com
> >> https://www.redhat.com/mailman/listinfo/linux-cluster
> >> 
> > 
> > 
> >      Try the new Yahoo! Messenger. Now
> with all you love about messenger and more! http://ph.messenger.yahoo.com
> > 
> > --
> > Linux-cluster mailing list
> > Linux-cluster at redhat.com
> > https://www.redhat.com/mailman/listinfo/linux-cluster
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
> 

      Get connected with chat on network profile, blog, or any personal website! Yahoo! allows you to IM with Pingbox. Check it out! http://ph.messenger.yahoo.com/pingbox