[Linux-cluster] fencing single server results in two servers fenced

[Mark Chaney]

I have a 3 node cluster. When I fence a single server (skydive.local) from
wheeljack.local, I get the following results (2 servers clustered). I can
duplicate this with any of my 3 servers. Why is the second server being
fenced?

##############################################

[root at wheeljack ~]# cman_tool nodes
Node  Sts   Inc   Joined               Name
   1   M   9580   2008-10-03 16:35:10  ratchet.local
   2   M   9580   2008-10-03 16:35:10  skydive.local
   3   M   9568   2008-10-03 16:35:10  wheeljack.local
[root at wheeljack ~]# tail -f /var/log/messages
Oct  3 16:35:15 wheeljack kernel: dlm: Using TCP for communications
Oct  3 16:35:15 wheeljack kernel: dlm: got connection from 1
Oct  3 16:35:15 wheeljack kernel: dlm: got connection from 2
Oct  3 16:35:16 wheeljack clvmd: Cluster LVM daemon started - connected to
CMAN
Oct  3 16:35:16 wheeljack multipathd: dm-3: add map (uevent)
Oct  3 16:35:16 wheeljack multipathd: dm-4: add map (uevent)
Oct  3 16:35:16 wheeljack multipathd: dm-5: add map (uevent)
Oct  3 16:35:16 wheeljack multipathd: dm-6: add map (uevent)
Oct  3 16:35:16 wheeljack multipathd: dm-7: add map (uevent)
Oct  3 16:35:17 wheeljack clurgmgrd[5407]: <notice> Resource Group Manager
Starting

Oct  3 16:37:15 wheeljack ntpd[3756]: synchronized to LOCAL(0), stratum 10
Oct  3 16:37:15 wheeljack ntpd[3756]: kernel time sync enabled 0001
Oct  3 16:38:19 wheeljack ntpd[3756]: synchronized to 66.79.149.35, stratum
2
Oct  3 16:42:42 wheeljack ntpd[3756]: synchronized to 64.202.112.65, stratum
2

Oct  3 16:44:05 wheeljack openais[5217]: [TOTEM] entering GATHER state from
12.
Oct  3 16:44:10 wheeljack openais[5217]: [TOTEM] entering GATHER state from
11.
Oct  3 16:44:10 wheeljack openais[5217]: [TOTEM] Saving state aru 52 high
seq received 52
Oct  3 16:44:10 wheeljack openais[5217]: [TOTEM] Storing new sequence id for
ring 2570
Oct  3 16:44:10 wheeljack openais[5217]: [TOTEM] entering COMMIT state.
Oct  3 16:44:10 wheeljack openais[5217]: [TOTEM] entering RECOVERY state.
Oct  3 16:44:10 wheeljack openais[5217]: [TOTEM] position [0] member
192.168.1.10:
Oct  3 16:44:10 wheeljack openais[5217]: [TOTEM] previous ring seq 9580 rep
192.168.1.10
Oct  3 16:44:10 wheeljack openais[5217]: [TOTEM] aru 52 high delivered 52
received flag 1
Oct  3 16:44:10 wheeljack openais[5217]: [TOTEM] position [1] member
192.168.1.11:
Oct  3 16:44:10 wheeljack openais[5217]: [TOTEM] previous ring seq 9580 rep
192.168.1.10
Oct  3 16:44:10 wheeljack openais[5217]: [TOTEM] aru 52 high delivered 52
received flag 1
Oct  3 16:44:10 wheeljack openais[5217]: [TOTEM] Did not need to originate
any messages in recovery.
Oct  3 16:44:10 wheeljack openais[5217]: [CLM  ] CLM CONFIGURATION CHANGE
Oct  3 16:44:10 wheeljack openais[5217]: [CLM  ] New Configuration:
Oct  3 16:44:10 wheeljack openais[5217]: [CLM  ]        r(0)
ip(192.168.1.10)
Oct  3 16:44:10 wheeljack kernel: dlm: closing connection to node 2
Oct  3 16:44:10 wheeljack openais[5217]: [CLM  ]        r(0)
ip(192.168.1.11)
Oct  3 16:44:10 wheeljack openais[5217]: [CLM  ] Members Left:
Oct  3 16:44:10 wheeljack openais[5217]: [CLM  ]        r(0)
ip(192.168.1.12)
Oct  3 16:44:10 wheeljack openais[5217]: [CLM  ] Members Joined:
Oct  3 16:44:10 wheeljack openais[5217]: [CLM  ] CLM CONFIGURATION CHANGE
Oct  3 16:44:10 wheeljack openais[5217]: [CLM  ] New Configuration:
Oct  3 16:44:10 wheeljack openais[5217]: [CLM  ]        r(0)
ip(192.168.1.10)
Oct  3 16:44:10 wheeljack openais[5217]: [CLM  ]        r(0)
ip(192.168.1.11)
Oct  3 16:44:10 wheeljack openais[5217]: [CLM  ] Members Left:
Oct  3 16:44:10 wheeljack openais[5217]: [CLM  ] Members Joined:
Oct  3 16:44:10 wheeljack openais[5217]: [SYNC ] This node is within the
primary component and will provide service.
Oct  3 16:44:10 wheeljack openais[5217]: [TOTEM] entering OPERATIONAL state.
Oct  3 16:44:10 wheeljack openais[5217]: [CLM  ] got nodejoin message
192.168.1.10
Oct  3 16:44:10 wheeljack openais[5217]: [CLM  ] got nodejoin message
192.168.1.11
Oct  3 16:44:10 wheeljack openais[5217]: [CPG  ] got joinlist message from
node 3
Oct  3 16:44:10 wheeljack openais[5217]: [CPG  ] got joinlist message from
node 1
Oct  3 16:44:10 wheeljack fenced[5236]: fencing deferred to ratchet.local

[root at wheeljack ~]#
Message from syslogd@ at Fri Oct  3 16:48:12 2008 ...
wheeljack clurgmgrd[5407]: <emerg> #1: Quorum Dissolved
Message from syslogd@ at Fri Oct  3 16:52:55 2008 ...
wheeljack clurgmgrd[5407]: <emerg> #1: Quorum Dissolved