[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [Linux-cluster] CLVM & CMAN live adding nodes



I still think you should check the multicast setup or maybe use UDPU or broadcast, if only to eliminate the possibility. I've seen this sort of thing happen when snooping is switched off for example. Multicast packets do 'flow', but the switch doesn't allow new nodes to join the multicast group - or at least not quickly enough for the cluster protocol. It's a classic symptom - join 3 nodes at the same time, then add another later and it can't. I can easily reproduce it in fact.

It might not be that, but it's best to check, because it's the most common cause of this symptom IME.

The (highly) recommended thing to do with expected_votes is to leave it out of cluster.conf altogether and let cman calculate it. That avoids any nasty accidents like the last one ;-)

Chrissie

On 24/02/14 19:45, Bjoern Teipel wrote:
Thanks Chrissie,

that was an old artifact from testing with two nodes.
I set the expected votes now to 4 (3 existing nodes in the cluster and
one new) but I still have the same issue.
It seems like the new node can't gain quorum over corosync, I see
multicast packets flowing over the wire but quorum membership seems to
be static:

Feb 24 11:29:09 corosync [QUORUM] Members[3]: 1 2 3


Version: 6.2.0

Config Version: 4

Cluster Name: hv-1618-106-1

Cluster Id: 11612

Cluster Member: Yes

Cluster Generation: 244

Membership state: Cluster-Member

Nodes: 3

Expected votes: 4

Total votes: 3

Node votes: 1

Quorum: 3

Active subsystems: 8

Flags:

Ports Bound: 0 11

Node name: node01

Node ID: 1

Multicast addresses: 239.192.45.137

Node addresses: 10.14.10.6

On Node04:

Starting cluster:

    Checking if cluster has been disabled at boot...        [  OK  ]

    Checking Network Manager...                             [  OK  ]

    Global setup...                                         [  OK  ]

    Loading kernel modules...                               [  OK  ]

    Mounting configfs...                                    [  OK  ]

    Starting cman...                                        [  OK  ]

    Waiting for quorum... Timed-out waiting for cluster

                                                            [FAILED]

Stopping cluster:

    Leaving fence domain...                                 [  OK  ]

    Stopping gfs_controld...                                [  OK  ]

    Stopping dlm_controld...                                [  OK  ]

    Stopping fenced...                                      [  OK  ]

    Stopping cman...                                        [  OK  ]

    Waiting for corosync to shutdown:                       [  OK  ]

    Unloading kernel modules...                             [  OK  ]

    Unmounting configfs...                                  [  OK  ]


Node status:
Node  Sts   Inc   Joined               Name

    1   M    236   2014-02-24 00:22:32  node01

    2   M    240   2014-02-24 00:22:34  node02

    3   M    244   2014-02-24 00:22:38  node03

    4   X      0                        node04


On Mon, Feb 24, 2014 at 2:25 AM, Christine Caulfield
<ccaulfie redhat com <mailto:ccaulfie redhat com>> wrote:

    On 24/02/14 08:39, Bjoern Teipel wrote:

        Hi Fabio,

        removing UDPU does not change the behavior, the new node still
        doesn't
        join the cluster and still wants to fence node 01
        It still feels like a split brain or so.
        How do you join a new node, using the /etc/init.d/cman start or
        using
           cman_tool / dlm_tool  join ?

        Bjoern


        On Sat, Feb 22, 2014 at 10:16 PM, Fabio M. Di Nitto
        <fdinitto redhat com <mailto:fdinitto redhat com>
        <mailto:fdinitto redhat com <mailto:fdinitto redhat com>>> wrote:

             On 02/22/2014 08:05 PM, Bjoern Teipel wrote:
              > Thanks Fabio for replying may request.
              >
              > I'm using stock CentOS 6.4 versions and no rm, just
        clvmd and dlm.
              >
              > Name        : cman                         Relocations: (not
             relocatable)
              > Version     : 3.0.12.1                          Vendor:
        CentOS
              > Release     : 49.el6_4.2                    Build Date:
        Tue 03
             Sep 2013
              > 02:18:10 AM PDT
              >
              > Name        : lvm2-cluster                 Relocations: (not
             relocatable)
              > Version     : 2.02.98                           Vendor:
        CentOS
              > Release     : 9.el6_4.3                     Build Date:
        Tue 05
             Nov 2013
              > 07:36:18 AM PST
              >
              > Name        : corosync                     Relocations: (not
             relocatable)
              > Version     : 1.4.1                             Vendor:
        CentOS
              > Release     : 15.el6_4.1                    Build Date:
        Tue 14
             May 2013
              > 02:09:27 PM PDT
              >
              >
              > My question is based off this problem I have till January:
              >
              >
              > When ever I add a new node (I put into the cluster.conf
        and reloaded
              > with cman_tool version -r -S)  I end up with situations
        like the new
              > node wants to gain the quorum and starts to fence the
        existing pool
              > master and appears to generate some sort of split
        cluster. Does
             it work
              > at all, corosync and dlm do not know about the recently
        added node ?

             I can see you are using UDPU and that could be the culprit.
        Can you drop
             UDPU and work with multicast?

             Jan/Chrissie: do you remember if we support adding nodes at
        runtime with
             UDPU?

             The standalone node should not have quorum at all and
        should not be able
             to fence anybody to start with.

              >
              > New Node
              > ==========
              >
              > Node  Sts   Inc   Joined               Name
              >    1   X      0                        hv-1
              >    2   X      0                        hv-2
              >    3   X      0                        hv-3
              >    4   X      0                        hv-4
              >    5   X      0                        hv-5
              >    6   M     80   2014-01-07 21:37:42  hv-6<--- host added
              >
              >
              > Jan  7 21:37:42 hv-1 corosync[12564]:   [TOTEM ] The network
             interface
              > [10.14.18.77] is now up.
              > Jan  7 21:37:42 hv-1 corosync[12564]:   [QUORUM] Using
        quorum
             provider
              > quorum_cman
              > Jan  7 21:37:42 hv-1 corosync[12564]:   [SERV  ] Service
        engine
             loaded:
              > corosync cluster quorum service v0.1
              > Jan  7 21:37:42 hv-1 corosync[12564]:   [CMAN  ] CMAN
        3.0.12.1 (built
              > Sep  3 2013 09:17:34) started
              > Jan  7 21:37:42 hv-1 corosync[12564]:   [SERV  ] Service
        engine
             loaded:
              > corosync CMAN membership service 2.90
              > Jan  7 21:37:42 hv-1 corosync[12564]:   [SERV  ] Service
        engine
             loaded:
              > openais checkpoint service B.01.01
              > Jan  7 21:37:42 hv-1 corosync[12564]:   [SERV  ] Service
        engine
             loaded:
              > corosync extended virtual synchrony service
              > Jan  7 21:37:42 hv-1 corosync[12564]:   [SERV  ] Service
        engine
             loaded:
              > corosync configuration service
              > Jan  7 21:37:42 hv-1 corosync[12564]:   [SERV  ] Service
        engine
             loaded:
              > corosync cluster closed process group service v1.01
              > Jan  7 21:37:42 hv-1 corosync[12564]:   [SERV  ] Service
        engine
             loaded:
              > corosync cluster config database access v1.01
              > Jan  7 21:37:42 hv-1 corosync[12564]:   [SERV  ] Service
        engine
             loaded:
              > corosync profile loading service
              > Jan  7 21:37:42 hv-1 corosync[12564]:   [QUORUM] Using
        quorum
             provider
              > quorum_cman
              > Jan  7 21:37:42 hv-1 corosync[12564]:   [SERV  ] Service
        engine
             loaded:
              > corosync cluster quorum service v0.1
              > Jan  7 21:37:42 hv-1 corosync[12564]:   [MAIN  ]
        Compatibility
             mode set
              > to whitetank.  Using V1 and V2 of the synchronization
        engine.
              > Jan  7 21:37:42 hv-1 corosync[12564]:   [TOTEM ] adding
        new UDPU
             member
              > {10.14.18.65}
              > Jan  7 21:37:42 hv-1 corosync[12564]:   [TOTEM ] adding
        new UDPU
             member
              > {10.14.18.67}
              > Jan  7 21:37:42 hv-1 corosync[12564]:   [TOTEM ] adding
        new UDPU
             member
              > {10.14.18.68}
              > Jan  7 21:37:42 hv-1 corosync[12564]:   [TOTEM ] adding
        new UDPU
             member
              > {10.14.18.70}
              > Jan  7 21:37:42 hv-1 corosync[12564]:   [TOTEM ] adding
        new UDPU
             member
              > {10.14.18.66}
              > Jan  7 21:37:42 hv-1 corosync[12564]:   [TOTEM ] adding
        new UDPU
             member
              > {10.14.18.77}
              > Jan  7 21:37:42 hv-1  corosync[12564]:   [TOTEM ] A
        processor
             joined or
              > left the membership and a new membership was formed.
              > Jan  7 21:37:42 hv-1 corosync[12564]:   [CMAN  ] quorum
        regained,
              > resuming activity
              > Jan  7 21:37:42 hv-1 corosync[12564]:   [QUORUM] This
        node is
             within the
              > primary component and will provide service.
              > Jan  7 21:37:42 hv-1 corosync[12564]:   [QUORUM]
        Members[1]: 6
              > Jan  7 21:37:42 hv-1 corosync[12564]:   [QUORUM]
        Members[1]: 6
              > Jan  7 21:37:42 hv-1 corosync[12564]:   [CPG   ] chosen
        downlist:
             sender
              > r(0) ip(10.14.18.77) ; members(old:0 left:0)
              > Jan  7 21:37:42 hv-1 corosync[12564]:   [MAIN  ]
        Completed service
              > synchronization, ready to provide service.
              > Jan  7 21:37:46 hv-1 fenced[12620]: fenced 3.0.12.1 started
              > Jan  7 21:37:46 hv-1 dlm_controld[12643]: dlm_controld
        3.0.12.1
             started
              > Jan  7 21:37:47 hv-1 gfs_controld[12695]: gfs_controld
        3.0.12.1
             started
              > Jan  7 21:37:54 hv-1 fenced[12620]: fencing node hv-b1clcy1
              >
              > sudo -i corosync-objctl  |grep member
              >
              > totem.interface.member.__memberaddr=hv-1
              > totem.interface.member.__memberaddr=hv-2
              > totem.interface.member.__memberaddr=hv-3
              > totem.interface.member.__memberaddr=hv-4
              > totem.interface.member.__memberaddr=hv-5
              > totem.interface.member.__memberaddr=hv-6
              > runtime.totem.pg.mrp.srp.__members.6.ip=r(0) ip(10.14.18.77)
              > runtime.totem.pg.mrp.srp.__members.6.join_count=1
              > runtime.totem.pg.mrp.srp.__members.6.status=joined
              >
              >
              > Existing Node
              > =============
              >
              > member 6 has not been added to the quorum list :
              >
              > Jan  7 21:36:28 hv-1 corosync[7769]:   [QUORUM]
        Members[4]: 1 2 3 5
              > Jan  7 21:37:54 hv-1 corosync[7769]:   [TOTEM ] A
        processor joined or
              > left the membership and a new membership was formed.
              > Jan  7 21:37:54 hv-1 corosync[7769]:   [CPG   ] chosen
        downlist:
             sender
              > r(0) ip(10.14.18.65) ; members(old:4 left:0)
              >
              >
              > Node  Sts   Inc   Joined               Name
              >    1   M   4468   2013-12-10 14:33:27  hv-1
              >    2   M   4468   2013-12-10 14:33:27  hv-2
              >    3   M   5036   2014-01-07 17:51:26  hv-3
              >    4   X   4468                        hv-4(dead at the
        moment)
              >    5   M   4468   2013-12-10 14:33:27  hv-5
              >    6   X      0                        hv-6<--- added
              >
              >
              > Jan  7 21:36:28 hv-1 corosync[7769]:   [QUORUM]
        Members[4]: 1 2 3 5
              > Jan  7 21:37:54 hv-1 corosync[7769]:   [TOTEM ] A
        processor joined or
              > left the membership and a new membership was formed.
              > Jan  7 21:37:54 hv-1 corosync[7769]:   [CPG   ] chosen
        downlist:
             sender
              > r(0) ip(10.14.18.65) ; members(old:4 left:0)
              > Jan  7 21:37:54 hv-1 corosync[7769]:   [MAIN  ]
        Completed service
              > synchronization, ready to provide service.
              >
              >
              > totem.interface.member.__memberaddr=hv-1
              > totem.interface.member.__memberaddr=hv-2
              > totem.interface.member.__memberaddr=hv-3
              > totem.interface.member.__memberaddr=hv-4
              > totem.interface.member.__memberaddr=hv-5.
              > runtime.totem.pg.mrp.srp.__members.1.ip=r(0) ip(10.14.18.65)
              > runtime.totem.pg.mrp.srp.__members.1.join_count=1
              > runtime.totem.pg.mrp.srp.__members.1.status=joined
              > runtime.totem.pg.mrp.srp.__members.2.ip=r(0) ip(10.14.18.66)
              > runtime.totem.pg.mrp.srp.__members.2.join_count=1
              > runtime.totem.pg.mrp.srp.__members.2.status=joined
              > runtime.totem.pg.mrp.srp.__members.4.ip=r(0) ip(10.14.18.68)
              > runtime.totem.pg.mrp.srp.__members.4.join_count=1
              > runtime.totem.pg.mrp.srp.__members.4.status=left
              > runtime.totem.pg.mrp.srp.__members.5.ip=r(0) ip(10.14.18.70)
              > runtime.totem.pg.mrp.srp.__members.5.join_count=1
              > runtime.totem.pg.mrp.srp.__members.5.status=joined
              > runtime.totem.pg.mrp.srp.__members.3.ip=r(0) ip(10.14.18.67)
              > runtime.totem.pg.mrp.srp.__members.3.join_count=3
              > runtime.totem.pg.mrp.srp.__members.3.status=joined
              >
              >
              > cluster.conf:
              >
              > <?xml version="1.0"?>
              > <cluster config_version="32" name="hv-1618-110-1">
              >   <fence_daemon clean_start="0"/>
              >   <cman transport="udpu" expected_votes="1"/>



    Setting expected_votes to 1 in a six node cluster is a serious
    configuration error and needs to be changed. That is what is causing
    the new node to fence the rest of the cluster.

    Check that all of the nodes have the same cluster.conf file, any
    difference between that on the exiting nodes and the new one will
    prevent the new node from joining too.

    Chrissie





[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]