[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

[Linux-cluster] Cluster failover troubleshooting



Please help me figure out why this cluster failed over. This has happened several times in the past month, while previously it had been quite stable. What can trigger the corosync " [TOTEM ] A processor failed, forming new configuration. " message? By all appearances the primary server was functioning properly until it was fenced by the secondary.

I've got cluster3 running on debian lenny 2.6.30-1-amd64
ii  openais        1.0.0-3local1  Standards-based cluster framework (daemon an
ii  corosync       1.0.0-4        Standards-based cluster framework (daemon an
ii  rgmanager      3.0.0-1~agx0lo clustered resource group manager
ii  cman           3.0.0-1~agx0lo cluster manager


A bunch of successful status checks on the active server, nicks, leading up to:

Jan 21 04:28:08 wonder corosync[2856]: [TOTEM ] A processor failed, forming new configuration.
Jan 21 04:28:09 wonder qdiskd[2873]: Writing eviction notice for node 2
Jan 21 04:28:10 wonder qdiskd[2873]: Node 2 evicted
Jan 21 04:28:11 nicks corosync[2991]: [CMAN ] lost contact with quorum device
Jan 21 04:28:12 wonder corosync[2856]: [CLM ] CLM CONFIGURATION CHANGE
Jan 21 04:28:12 wonder corosync[2856]: [CLM ] New Configuration:
Jan 21 04:28:12 wonder corosync[2856]: [CLM ] r(0) ip(192.168.255.20)
Jan 21 04:28:12 wonder corosync[2856]: [CLM ] Members Left:
Jan 21 04:28:12 wonder corosync[2856]: [CLM ] r(0) ip(192.168.255.21)
Jan 21 04:28:12 wonder corosync[2856]: [CLM ] Members Joined:
Jan 21 04:28:12 wonder corosync[2856]: [QUORUM] This node is within the primary component and will provide service.
Jan 21 04:28:12 wonder corosync[2856]: [QUORUM] Members[1]:
Jan 21 04:28:12 wonder corosync[2856]: [QUORUM] 1
Jan 21 04:28:12 wonder corosync[2856]: [CLM ] CLM CONFIGURATION CHANGE
Jan 21 04:28:12 wonder corosync[2856]: [CLM ] New Configuration:
Jan 21 04:28:12 wonder corosync[2856]: [CLM ] r(0) ip(192.168.255.20)
Jan 21 04:28:12 wonder corosync[2856]: [CLM ] Members Left:
Jan 21 04:28:12 wonder corosync[2856]: [CLM ] Members Joined:
Jan 21 04:28:12 wonder corosync[2856]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
Jan 21 04:28:12 wonder corosync[2856]: [MAIN ] Completed service synchronization, ready to provide service.
Jan 21 04:28:12 wonder rgmanager[3578]: State change: nicks-p DOWN
Jan 21 04:28:13 wonder kernel: [1298595.738213] dlm: closing connection to node 2
Jan 21 04:28:13 wonder fenced[3206]: fencing node nicks-p
Jan 21 04:28:12 nicks corosync[2991]: [QUORUM] This node is within the primary component and will provide service.
Jan 21 04:28:13 nicks corosync[2991]: [QUORUM] Members[2]:
Jan 21 04:28:13 nicks corosync[2991]: [QUORUM] 1
Jan 21 04:28:13 nicks corosync[2991]: [QUORUM] 2
Jan 21 04:28:16 wonder fenced[3206]: fence nicks-p success
Jan 21 04:28:16 wonder fenced[3206]: fence nicks-p success
Jan 21 04:28:17 wonder rgmanager[3578]: Taking over service service:MailHost from down member nicks-p
Jan 21 04:28:17 wonder bash[13236]: Unknown file system type 'ext4' for device /dev/dm-0. Assuming fsck is required.
Jan 21 04:28:17 wonder bash[13259]: Running fsck on /dev/dm-0
Jan 21 04:28:18 wonder bash[13284]: mounting /dev/dm-0 on /home
Jan 21 04:28:18 wonder bash[13306]: mount -t ext4 -o defaults,noatime,nodiratime /dev/dm-0 /home
Jan 21 04:28:19 wonder bash[13335]: quotaon not found in /bin:/sbin:/usr/bin:/usr/sbin
Jan 21 04:28:19 wonder bash[13335]: quotaon not found in /bin:/sbin:/usr/bin:/usr/sbin
Jan 21 04:28:19 wonder bash[13368]: mounting /dev/dm-1 on /var/cluster
Jan 21 04:28:19 wonder bash[13390]: mount -t ext3 -o defaults /dev/dm-1 /var/cluster
Jan 21 04:28:19 wonder bash[13415]: quotaon not found in /bin:/sbin:/usr/bin:/usr/sbin
Jan 21 04:28:19 wonder bash[13415]: quotaon not found in /bin:/sbin:/usr/bin:/usr/sbin
Jan 21 04:28:20 wonder bash[13467]: Link for eth0: Detected
Jan 21 04:28:20 wonder bash[13489]: Adding IPv4 address 172.25.16.58/22 to eth0
Jan 21 04:28:20 wonder bash[13513]: Sending gratuitous ARP: 172.25.16.58 00:30:48:c6:df:ce brd ff:ff:ff:ff:ff:ff
Jan 21 04:28:21 wonder bash[13551]: Executing /etc/cluster/MailHost-misc-early start
Jan 21 04:28:21 wonder bash[13606]: Executing /etc/cluster/saslauthd-cluster start
Jan 21 04:28:21 wonder bash[13679]: Executing /etc/cluster/postfix-cluster start
Jan 21 04:28:22 wonder bash[13788]: Executing /etc/cluster/dovecot-wrapper start
Jan 21 04:28:22 wonder bash[13850]: Executing /etc/cluster/mailman-wrapper start
Jan 21 04:28:23 wonder bash[13901]: Executing /etc/cluster/apache2-mailhost start


<?xml version="1.0"?>
<cluster name="alpha" config_version="44">

<cman two_node="0" expected_votes="3">
</cman>

<clusternodes>
<clusternode name="wonder-p" votes="1" nodeid="1">
        <fence>
                <method name="single">
                        <device name="pwr01" option="off"/>
                        <device name="pwr02" option="off"/>
                        <device name="pwr01" option="on"/>
                        <device name="pwr02" option="on"/>
                </method>
        </fence>
</clusternode>
<clusternode name="nicks-p" votes="1" nodeid="2">
        <fence>
                <method name="single">
                        <device name="pwr03" option="off"/>
                        <device name="pwr04" option="off"/>
                        <device name="pwr03" option="on"/>
                        <device name="pwr04" option="on"/>
                </method>
        </fence>
</clusternode>
</clusternodes>

<quorumd interval="1" tko="10" votes="1" label="quorumdisk">
        <heuristic program="ping 172.25.19.254 -c1 -t1" score="1" interval="2" tko="3"/>
</quorumd>

<fence_daemon post_join_delay="20">
</fence_daemon>

<fencedevices>
        <fencedevice agent="fence_apc_snmp" ipaddr="pdu-paul-2-2" port="4" name="pwr01" udpport="161" />
        <fencedevice agent="fence_apc_snmp" ipaddr="pdu-paul-2-3" port="4" name="pwr02" udpport="161" />
        <fencedevice agent="fence_apc_snmp" ipaddr="pdu-paul-2-2" port="3" name="pwr03" udpport="161" />
        <fencedevice agent="fence_apc_snmp" ipaddr="pdu-paul-2-3" port="3" name="pwr04" udpport="161" />
</fencedevices>

<rm>

   <failoverdomains>
           <failoverdomain name="mailcluster" restricted="1" ordered="0" >
                <failoverdomainnode name="wonder-p" priority="1"/>
                <failoverdomainnode name="nicks-p" priority="1"/>
           </failoverdomain>
   </failoverdomains>

   <service name="MailHost" autostart="1" domain="mailcluster" >
           <script name="MailHost-early" file="/etc/cluster/MailHost-misc-early" />
           <fs name="mailhome" mountpoint="/home" device="/dev/dm-0" fstype="ext4" force_unmount="1" active_monitor="1" options="defaults,noatime,nodiratime" />
           <fs name="mailcluster" mountpoint="/var/cluster" device="/dev/dm-1" fstype="ext3" force_unmount="1" active_monitor="1" options="defaults" />
           <ip address="172.25.16.58" monitor_link="1" />
           <script name="saslauthd" file="/etc/cluster/saslauthd-cluster" />
           <script name="postfix" file="/etc/cluster/postfix-cluster" />
           <script name="dovecot" file="/etc/cluster/dovecot-wrapper" __independent_subtree="1" />
           <script name="mailman" file="/etc/cluster/mailman-wrapper" __independent_subtree="1" />
           <script name="apache2-mailhost" file="/etc/cluster/apache2-mailhost" __independent_subtree="1" />
           <script name="usermin" file="/etc/init.d/usermin-sb" __independent_subtree="1" />
           <script name="MailHost-late" file="/etc/cluster/MailHost-misc-late" />
   </service>

</rm>
</cluster>


Thanks

Chris

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]