[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

[Linux-cluster] One node rebooting quite often



I am attaching the relevant log entries from a node that is rebooting about twice per day on average. I can't seem to pinpoint why? I was hoping someone could Identify something I'm overlooking. I will gladly provide more information as needed.
I'm running a two-node nfs cluster on Centos 5 with the following packages:
rgmanager-2.0.31-1.el5.centos
gfs2-utils-0.1.38-1.el5
gfs-utils-0.1.12-1.el5
system-config-lvm-1.0.22-1.0.el5
cman-2.0.73-1.el5_1.1
lvm2-cluster-2.02.26-1.el5
kmod-gfs-0.1.16-6.2.6.18_8.1.15.el5
lvm2-2.02.26-3.el5
kmod-gfs-0.1.19-7.el5_1.1
kernel-2.6.18-53.1.6.el5

Thanks!

Randy
Feb 12 00:01:16 nfs1-cluster openais[2605]: [TOTEM] The token was lost in the OPERATIONAL state. 
Feb 12 00:01:16 nfs1-cluster openais[2605]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). 
Feb 12 00:01:16 nfs1-cluster openais[2605]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
Feb 12 00:01:16 nfs1-cluster openais[2605]: [TOTEM] entering GATHER state from 2. 
Feb 12 00:01:16 nfs1-cluster openais[2605]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 00:01:16 nfs1-cluster openais[2605]: [TOTEM] Saving state aru 1e high seq received 1e 
Feb 12 00:01:16 nfs1-cluster openais[2605]: [TOTEM] Storing new sequence id for ring 1054 
Feb 12 00:01:16 nfs1-cluster openais[2605]: [TOTEM] entering COMMIT state. 
Feb 12 00:01:19 nfs1-cluster openais[2605]: [TOTEM] entering RECOVERY state. 
Feb 12 00:01:19 nfs1-cluster openais[2605]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 00:01:19 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4176 rep 140.90.91.240 
Feb 12 00:01:19 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 00:01:19 nfs1-cluster openais[2605]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 00:01:19 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4176 rep 140.90.91.240 
Feb 12 00:01:19 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 00:01:19 nfs1-cluster openais[2605]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 00:01:19 nfs1-cluster openais[2605]: [TOTEM] Sending initial ORF token 
Feb 12 00:01:19 nfs1-cluster openais[2605]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 00:01:19 nfs1-cluster openais[2605]: [CLM  ] New Configuration: 
Feb 12 00:01:19 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 00:01:19 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 00:01:19 nfs1-cluster openais[2605]: [CLM  ] Members Left: 
Feb 12 00:01:19 nfs1-cluster openais[2605]: [CLM  ] Members Joined: 
Feb 12 00:01:19 nfs1-cluster openais[2605]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 00:01:19 nfs1-cluster openais[2605]: [CLM  ] New Configuration: 
Feb 12 00:01:19 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 00:01:19 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 00:01:19 nfs1-cluster openais[2605]: [CLM  ] Members Left: 
Feb 12 00:01:19 nfs1-cluster openais[2605]: [CLM  ] Members Joined: 
Feb 12 00:01:19 nfs1-cluster openais[2605]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 00:01:19 nfs1-cluster openais[2605]: [TOTEM] entering OPERATIONAL state. 
Feb 12 00:01:19 nfs1-cluster openais[2605]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 00:01:19 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2  
Feb 12 00:01:19 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3  
Feb 12 00:01:19 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3 4  
Feb 12 00:01:19 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3 4  
Feb 12 00:01:19 nfs1-cluster openais[2605]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 00:01:19 nfs1-cluster openais[2605]: [CPG  ] got joinlist message from node 1 
Feb 12 00:01:19 nfs1-cluster openais[2605]: [CPG  ] got joinlist message from node 2 
Feb 12 00:28:23 nfs1-cluster openais[2605]: [TOTEM] The token was lost in the OPERATIONAL state. 
Feb 12 00:28:23 nfs1-cluster openais[2605]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). 
Feb 12 00:28:23 nfs1-cluster openais[2605]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
Feb 12 00:28:23 nfs1-cluster openais[2605]: [TOTEM] entering GATHER state from 2. 
Feb 12 00:28:23 nfs1-cluster openais[2605]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 00:28:23 nfs1-cluster openais[2605]: [TOTEM] Saving state aru 1e high seq received 1e 
Feb 12 00:28:23 nfs1-cluster openais[2605]: [TOTEM] Storing new sequence id for ring 1058 
Feb 12 00:28:23 nfs1-cluster openais[2605]: [TOTEM] entering COMMIT state. 
Feb 12 00:28:26 nfs1-cluster openais[2605]: [TOTEM] entering RECOVERY state. 
Feb 12 00:28:26 nfs1-cluster openais[2605]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 00:28:26 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4180 rep 140.90.91.240 
Feb 12 00:28:26 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 00:28:26 nfs1-cluster openais[2605]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 00:28:26 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4180 rep 140.90.91.240 
Feb 12 00:28:26 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 00:28:26 nfs1-cluster openais[2605]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 00:28:26 nfs1-cluster openais[2605]: [TOTEM] Sending initial ORF token 
Feb 12 00:28:26 nfs1-cluster openais[2605]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 00:28:26 nfs1-cluster openais[2605]: [CLM  ] New Configuration: 
Feb 12 00:28:26 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 00:28:26 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 00:28:26 nfs1-cluster openais[2605]: [CLM  ] Members Left: 
Feb 12 00:28:26 nfs1-cluster openais[2605]: [CLM  ] Members Joined: 
Feb 12 00:28:26 nfs1-cluster openais[2605]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 00:28:26 nfs1-cluster openais[2605]: [CLM  ] New Configuration: 
Feb 12 00:28:26 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 00:28:26 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 00:28:27 nfs1-cluster openais[2605]: [CLM  ] Members Left: 
Feb 12 00:28:27 nfs1-cluster openais[2605]: [CLM  ] Members Joined: 
Feb 12 00:28:27 nfs1-cluster openais[2605]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 00:28:27 nfs1-cluster openais[2605]: [TOTEM] entering OPERATIONAL state. 
Feb 12 00:28:27 nfs1-cluster openais[2605]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 00:28:27 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2  
Feb 12 00:28:27 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3  
Feb 12 00:28:27 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3 4  
Feb 12 00:28:27 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3 4  
Feb 12 00:28:27 nfs1-cluster openais[2605]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 00:28:27 nfs1-cluster openais[2605]: [CPG  ] got joinlist message from node 1 
Feb 12 00:28:27 nfs1-cluster openais[2605]: [CPG  ] got joinlist message from node 2 
Feb 12 00:44:09 nfs1-cluster openais[2605]: [TOTEM] The token was lost in the OPERATIONAL state. 
Feb 12 00:44:09 nfs1-cluster openais[2605]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). 
Feb 12 00:44:09 nfs1-cluster openais[2605]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
Feb 12 00:44:09 nfs1-cluster openais[2605]: [TOTEM] entering GATHER state from 2. 
Feb 12 00:44:09 nfs1-cluster openais[2605]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 00:44:09 nfs1-cluster openais[2605]: [TOTEM] Saving state aru 1e high seq received 1e 
Feb 12 00:44:09 nfs1-cluster openais[2605]: [TOTEM] Storing new sequence id for ring 105c 
Feb 12 00:44:09 nfs1-cluster openais[2605]: [TOTEM] entering COMMIT state. 
Feb 12 00:44:10 nfs1-cluster openais[2605]: [TOTEM] entering RECOVERY state. 
Feb 12 00:44:10 nfs1-cluster openais[2605]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 00:44:10 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4184 rep 140.90.91.240 
Feb 12 00:44:10 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 00:44:10 nfs1-cluster openais[2605]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 00:44:10 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4184 rep 140.90.91.240 
Feb 12 00:44:10 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 00:44:10 nfs1-cluster openais[2605]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 00:44:10 nfs1-cluster openais[2605]: [TOTEM] Sending initial ORF token 
Feb 12 00:44:10 nfs1-cluster openais[2605]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 00:44:10 nfs1-cluster openais[2605]: [CLM  ] New Configuration: 
Feb 12 00:44:10 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 00:44:10 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 00:44:10 nfs1-cluster openais[2605]: [CLM  ] Members Left: 
Feb 12 00:44:10 nfs1-cluster openais[2605]: [CLM  ] Members Joined: 
Feb 12 00:44:10 nfs1-cluster openais[2605]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 00:44:10 nfs1-cluster openais[2605]: [CLM  ] New Configuration: 
Feb 12 00:44:10 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 00:44:10 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 00:44:10 nfs1-cluster openais[2605]: [CLM  ] Members Left: 
Feb 12 00:44:10 nfs1-cluster openais[2605]: [CLM  ] Members Joined: 
Feb 12 00:44:10 nfs1-cluster openais[2605]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 00:44:10 nfs1-cluster openais[2605]: [TOTEM] entering OPERATIONAL state. 
Feb 12 00:44:10 nfs1-cluster openais[2605]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 00:44:10 nfs1-cluster openais[2605]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 00:44:10 nfs1-cluster openais[2605]: [CPG  ] got joinlist message from node 1 
Feb 12 00:44:10 nfs1-cluster openais[2605]: [CPG  ] got joinlist message from node 2 
Feb 12 00:50:27 nfs1-cluster openais[2605]: [TOTEM] The token was lost in the OPERATIONAL state. 
Feb 12 00:50:27 nfs1-cluster openais[2605]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). 
Feb 12 00:50:27 nfs1-cluster openais[2605]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
Feb 12 00:50:27 nfs1-cluster openais[2605]: [TOTEM] entering GATHER state from 2. 
Feb 12 00:50:27 nfs1-cluster openais[2605]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 00:50:27 nfs1-cluster openais[2605]: [TOTEM] Saving state aru 1e high seq received 1e 
Feb 12 00:50:27 nfs1-cluster openais[2605]: [TOTEM] Storing new sequence id for ring 1060 
Feb 12 00:50:27 nfs1-cluster openais[2605]: [TOTEM] entering COMMIT state. 
Feb 12 00:50:29 nfs1-cluster openais[2605]: [TOTEM] entering RECOVERY state. 
Feb 12 00:50:29 nfs1-cluster openais[2605]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 00:50:29 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4188 rep 140.90.91.240 
Feb 12 00:50:29 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 00:50:29 nfs1-cluster openais[2605]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 00:50:29 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4188 rep 140.90.91.240 
Feb 12 00:50:29 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 00:50:29 nfs1-cluster openais[2605]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 00:50:29 nfs1-cluster openais[2605]: [TOTEM] Sending initial ORF token 
Feb 12 00:50:29 nfs1-cluster openais[2605]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 00:50:29 nfs1-cluster openais[2605]: [CLM  ] New Configuration: 
Feb 12 00:50:29 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 00:50:29 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 00:50:29 nfs1-cluster openais[2605]: [CLM  ] Members Left: 
Feb 12 00:50:29 nfs1-cluster openais[2605]: [CLM  ] Members Joined: 
Feb 12 00:50:29 nfs1-cluster openais[2605]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 00:50:29 nfs1-cluster openais[2605]: [CLM  ] New Configuration: 
Feb 12 00:50:29 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 00:50:29 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 00:50:29 nfs1-cluster openais[2605]: [CLM  ] Members Left: 
Feb 12 00:50:29 nfs1-cluster openais[2605]: [CLM  ] Members Joined: 
Feb 12 00:50:29 nfs1-cluster openais[2605]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 00:50:29 nfs1-cluster openais[2605]: [TOTEM] entering OPERATIONAL state. 
Feb 12 00:50:29 nfs1-cluster openais[2605]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 00:50:29 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2  
Feb 12 00:50:29 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3  
Feb 12 00:50:29 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3 4  
Feb 12 00:50:29 nfs1-cluster openais[2605]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 00:50:29 nfs1-cluster openais[2605]: [CPG  ] got joinlist message from node 1 
Feb 12 00:50:29 nfs1-cluster openais[2605]: [CPG  ] got joinlist message from node 2 
Feb 12 00:53:31 nfs1-cluster openais[2605]: [TOTEM] The token was lost in the OPERATIONAL state. 
Feb 12 00:53:31 nfs1-cluster openais[2605]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). 
Feb 12 00:53:31 nfs1-cluster openais[2605]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
Feb 12 00:53:31 nfs1-cluster openais[2605]: [TOTEM] entering GATHER state from 2. 
Feb 12 00:53:31 nfs1-cluster openais[2605]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 00:53:31 nfs1-cluster openais[2605]: [TOTEM] Saving state aru 1e high seq received 1e 
Feb 12 00:53:31 nfs1-cluster openais[2605]: [TOTEM] Storing new sequence id for ring 1064 
Feb 12 00:53:31 nfs1-cluster openais[2605]: [TOTEM] entering COMMIT state. 
Feb 12 00:53:32 nfs1-cluster openais[2605]: [TOTEM] entering RECOVERY state. 
Feb 12 00:53:32 nfs1-cluster openais[2605]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 00:53:32 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4192 rep 140.90.91.240 
Feb 12 00:53:32 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 00:53:32 nfs1-cluster openais[2605]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 00:53:32 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4192 rep 140.90.91.240 
Feb 12 00:53:32 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 00:53:32 nfs1-cluster openais[2605]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 00:53:32 nfs1-cluster openais[2605]: [TOTEM] Sending initial ORF token 
Feb 12 00:53:32 nfs1-cluster openais[2605]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 00:53:32 nfs1-cluster openais[2605]: [CLM  ] New Configuration: 
Feb 12 00:53:32 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 00:53:32 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 00:53:32 nfs1-cluster openais[2605]: [CLM  ] Members Left: 
Feb 12 00:53:32 nfs1-cluster openais[2605]: [CLM  ] Members Joined: 
Feb 12 00:53:32 nfs1-cluster openais[2605]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 00:53:32 nfs1-cluster openais[2605]: [CLM  ] New Configuration: 
Feb 12 00:53:32 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 00:53:32 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 00:53:32 nfs1-cluster openais[2605]: [CLM  ] Members Left: 
Feb 12 00:53:32 nfs1-cluster openais[2605]: [CLM  ] Members Joined: 
Feb 12 00:53:32 nfs1-cluster openais[2605]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 00:53:32 nfs1-cluster openais[2605]: [TOTEM] entering OPERATIONAL state. 
Feb 12 00:53:32 nfs1-cluster openais[2605]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 00:53:32 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2  
Feb 12 00:53:32 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3  
Feb 12 00:53:32 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3 4  
Feb 12 00:53:32 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3 4  
Feb 12 00:53:32 nfs1-cluster openais[2605]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 00:53:32 nfs1-cluster openais[2605]: [CPG  ] got joinlist message from node 1 
Feb 12 00:53:32 nfs1-cluster openais[2605]: [CPG  ] got joinlist message from node 2 
Feb 12 01:06:55 nfs1-cluster openais[2605]: [TOTEM] entering GATHER state from 12. 
Feb 12 01:06:55 nfs1-cluster openais[2605]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 01:06:55 nfs1-cluster openais[2605]: [TOTEM] Saving state aru 1e high seq received 1e 
Feb 12 01:06:55 nfs1-cluster openais[2605]: [TOTEM] Storing new sequence id for ring 1068 
Feb 12 01:06:55 nfs1-cluster openais[2605]: [TOTEM] entering COMMIT state. 
Feb 12 01:06:57 nfs1-cluster openais[2605]: [TOTEM] entering RECOVERY state. 
Feb 12 01:06:57 nfs1-cluster openais[2605]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 01:06:57 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4196 rep 140.90.91.240 
Feb 12 01:06:57 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 01:06:57 nfs1-cluster openais[2605]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 01:06:57 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4196 rep 140.90.91.240 
Feb 12 01:06:57 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 01:06:57 nfs1-cluster openais[2605]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 01:06:57 nfs1-cluster openais[2605]: [TOTEM] Sending initial ORF token 
Feb 12 01:06:57 nfs1-cluster openais[2605]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 01:06:57 nfs1-cluster openais[2605]: [CLM  ] New Configuration: 
Feb 12 01:06:57 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 01:06:57 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 01:06:57 nfs1-cluster openais[2605]: [CLM  ] Members Left: 
Feb 12 01:06:57 nfs1-cluster openais[2605]: [CLM  ] Members Joined: 
Feb 12 01:06:57 nfs1-cluster openais[2605]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 01:06:57 nfs1-cluster openais[2605]: [CLM  ] New Configuration: 
Feb 12 01:06:57 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 01:06:57 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 01:06:57 nfs1-cluster openais[2605]: [CLM  ] Members Left: 
Feb 12 01:06:57 nfs1-cluster openais[2605]: [CLM  ] Members Joined: 
Feb 12 01:06:57 nfs1-cluster openais[2605]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 01:06:57 nfs1-cluster openais[2605]: [TOTEM] entering OPERATIONAL state. 
Feb 12 01:06:57 nfs1-cluster openais[2605]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 01:06:57 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2  
Feb 12 01:06:57 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3  
Feb 12 01:06:57 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3 4  
Feb 12 01:06:57 nfs1-cluster openais[2605]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 01:06:57 nfs1-cluster openais[2605]: [CPG  ] got joinlist message from node 1 
Feb 12 01:06:57 nfs1-cluster openais[2605]: [CPG  ] got joinlist message from node 2 
Feb 12 01:46:13 nfs1-cluster openais[2605]: [TOTEM] The token was lost in the OPERATIONAL state. 
Feb 12 01:46:14 nfs1-cluster openais[2605]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). 
Feb 12 01:46:14 nfs1-cluster openais[2605]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
Feb 12 01:46:14 nfs1-cluster openais[2605]: [TOTEM] entering GATHER state from 2. 
Feb 12 01:46:14 nfs1-cluster openais[2605]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 01:46:14 nfs1-cluster openais[2605]: [TOTEM] Saving state aru 1e high seq received 1e 
Feb 12 01:46:14 nfs1-cluster openais[2605]: [TOTEM] Storing new sequence id for ring 106c 
Feb 12 01:46:14 nfs1-cluster openais[2605]: [TOTEM] entering COMMIT state. 
Feb 12 01:46:14 nfs1-cluster openais[2605]: [TOTEM] entering RECOVERY state. 
Feb 12 01:46:14 nfs1-cluster openais[2605]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 01:46:14 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4200 rep 140.90.91.240 
Feb 12 01:46:14 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 01:46:14 nfs1-cluster openais[2605]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 01:46:14 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4200 rep 140.90.91.240 
Feb 12 01:46:14 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 01:46:14 nfs1-cluster openais[2605]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 01:46:14 nfs1-cluster openais[2605]: [TOTEM] Sending initial ORF token 
Feb 12 01:46:14 nfs1-cluster openais[2605]: [TOTEM] Sending initial ORF token 
Feb 12 01:46:14 nfs1-cluster openais[2605]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 01:46:14 nfs1-cluster openais[2605]: [CLM  ] New Configuration: 
Feb 12 01:46:14 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 01:46:14 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 01:46:14 nfs1-cluster openais[2605]: [CLM  ] Members Left: 
Feb 12 01:46:14 nfs1-cluster openais[2605]: [CLM  ] Members Joined: 
Feb 12 01:46:14 nfs1-cluster openais[2605]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 01:46:14 nfs1-cluster openais[2605]: [CLM  ] New Configuration: 
Feb 12 01:46:14 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 01:46:14 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 01:46:14 nfs1-cluster openais[2605]: [CLM  ] Members Left: 
Feb 12 01:46:14 nfs1-cluster openais[2605]: [CLM  ] Members Joined: 
Feb 12 01:46:14 nfs1-cluster openais[2605]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 01:46:14 nfs1-cluster openais[2605]: [TOTEM] entering OPERATIONAL state. 
Feb 12 01:46:14 nfs1-cluster openais[2605]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 01:46:14 nfs1-cluster openais[2605]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 01:46:14 nfs1-cluster openais[2605]: [CPG  ] got joinlist message from node 1 
Feb 12 01:46:14 nfs1-cluster openais[2605]: [CPG  ] got joinlist message from node 2 
Feb 12 02:25:14 nfs1-cluster openais[2605]: [TOTEM] The token was lost in the OPERATIONAL state. 
Feb 12 02:25:14 nfs1-cluster openais[2605]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). 
Feb 12 02:25:14 nfs1-cluster openais[2605]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
Feb 12 02:25:14 nfs1-cluster openais[2605]: [TOTEM] entering GATHER state from 2. 
Feb 12 02:25:14 nfs1-cluster openais[2605]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 02:25:14 nfs1-cluster openais[2605]: [TOTEM] Saving state aru 1e high seq received 1e 
Feb 12 02:25:14 nfs1-cluster openais[2605]: [TOTEM] Storing new sequence id for ring 1070 
Feb 12 02:25:14 nfs1-cluster openais[2605]: [TOTEM] entering COMMIT state. 
Feb 12 02:25:15 nfs1-cluster openais[2605]: [TOTEM] entering RECOVERY state. 
Feb 12 02:25:15 nfs1-cluster openais[2605]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 02:25:15 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4204 rep 140.90.91.240 
Feb 12 02:25:15 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 02:25:15 nfs1-cluster openais[2605]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 02:25:15 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4204 rep 140.90.91.240 
Feb 12 02:25:15 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 02:25:15 nfs1-cluster openais[2605]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 02:25:15 nfs1-cluster openais[2605]: [TOTEM] Sending initial ORF token 
Feb 12 02:25:15 nfs1-cluster openais[2605]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 02:25:15 nfs1-cluster openais[2605]: [CLM  ] New Configuration: 
Feb 12 02:25:15 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 02:25:15 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 02:25:15 nfs1-cluster openais[2605]: [CLM  ] Members Left: 
Feb 12 02:25:15 nfs1-cluster openais[2605]: [CLM  ] Members Joined: 
Feb 12 02:25:15 nfs1-cluster openais[2605]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 02:25:15 nfs1-cluster openais[2605]: [CLM  ] New Configuration: 
Feb 12 02:25:15 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 02:25:15 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 02:25:15 nfs1-cluster openais[2605]: [CLM  ] Members Left: 
Feb 12 02:25:15 nfs1-cluster openais[2605]: [CLM  ] Members Joined: 
Feb 12 02:25:15 nfs1-cluster openais[2605]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 02:25:15 nfs1-cluster openais[2605]: [TOTEM] entering OPERATIONAL state. 
Feb 12 02:25:15 nfs1-cluster openais[2605]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 02:25:15 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2  
Feb 12 02:25:15 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3  
Feb 12 02:25:15 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3 4  
Feb 12 02:25:15 nfs1-cluster openais[2605]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 02:25:15 nfs1-cluster openais[2605]: [CPG  ] got joinlist message from node 1 
Feb 12 02:25:15 nfs1-cluster openais[2605]: [CPG  ] got joinlist message from node 2 
Feb 12 02:36:52 nfs1-cluster openais[2605]: [TOTEM] The token was lost in the OPERATIONAL state. 
Feb 12 02:36:52 nfs1-cluster openais[2605]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). 
Feb 12 02:36:52 nfs1-cluster openais[2605]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
Feb 12 02:36:52 nfs1-cluster openais[2605]: [TOTEM] entering GATHER state from 2. 
Feb 12 02:36:52 nfs1-cluster openais[2605]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 02:36:52 nfs1-cluster openais[2605]: [TOTEM] Saving state aru 1e high seq received 1e 
Feb 12 02:36:52 nfs1-cluster openais[2605]: [TOTEM] Storing new sequence id for ring 1074 
Feb 12 02:36:52 nfs1-cluster openais[2605]: [TOTEM] entering COMMIT state. 
Feb 12 02:36:54 nfs1-cluster openais[2605]: [TOTEM] entering RECOVERY state. 
Feb 12 02:36:54 nfs1-cluster openais[2605]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 02:36:54 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4208 rep 140.90.91.240 
Feb 12 02:36:54 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 02:36:54 nfs1-cluster openais[2605]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 02:36:54 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4208 rep 140.90.91.240 
Feb 12 02:36:54 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 02:36:54 nfs1-cluster openais[2605]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 02:36:54 nfs1-cluster openais[2605]: [TOTEM] Sending initial ORF token 
Feb 12 02:36:54 nfs1-cluster openais[2605]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 02:36:54 nfs1-cluster openais[2605]: [CLM  ] New Configuration: 
Feb 12 02:36:54 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 02:36:54 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 02:36:54 nfs1-cluster openais[2605]: [CLM  ] Members Left: 
Feb 12 02:36:54 nfs1-cluster openais[2605]: [CLM  ] Members Joined: 
Feb 12 02:36:54 nfs1-cluster openais[2605]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 02:36:54 nfs1-cluster openais[2605]: [CLM  ] New Configuration: 
Feb 12 02:36:54 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 02:36:54 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 02:36:54 nfs1-cluster openais[2605]: [CLM  ] Members Left: 
Feb 12 02:36:54 nfs1-cluster openais[2605]: [CLM  ] Members Joined: 
Feb 12 02:36:54 nfs1-cluster openais[2605]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 02:36:54 nfs1-cluster openais[2605]: [TOTEM] entering OPERATIONAL state. 
Feb 12 02:36:54 nfs1-cluster openais[2605]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 02:36:54 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2  
Feb 12 02:36:54 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3  
Feb 12 02:36:54 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3 4  
Feb 12 02:36:54 nfs1-cluster openais[2605]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 02:36:54 nfs1-cluster openais[2605]: [CPG  ] got joinlist message from node 1 
Feb 12 02:36:54 nfs1-cluster openais[2605]: [CPG  ] got joinlist message from node 2 
Feb 12 02:41:27 nfs1-cluster openais[2605]: [TOTEM] The token was lost in the OPERATIONAL state. 
Feb 12 02:41:27 nfs1-cluster openais[2605]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). 
Feb 12 02:41:27 nfs1-cluster openais[2605]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
Feb 12 02:41:27 nfs1-cluster openais[2605]: [TOTEM] entering GATHER state from 2. 
Feb 12 02:41:27 nfs1-cluster openais[2605]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 02:41:27 nfs1-cluster openais[2605]: [TOTEM] Saving state aru 1e high seq received 1e 
Feb 12 02:41:27 nfs1-cluster openais[2605]: [TOTEM] Storing new sequence id for ring 1078 
Feb 12 02:41:27 nfs1-cluster openais[2605]: [TOTEM] entering COMMIT state. 
Feb 12 02:41:27 nfs1-cluster openais[2605]: [TOTEM] entering RECOVERY state. 
Feb 12 02:41:27 nfs1-cluster openais[2605]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 02:41:27 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4212 rep 140.90.91.240 
Feb 12 02:41:27 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 02:41:27 nfs1-cluster openais[2605]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 02:41:27 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4212 rep 140.90.91.240 
Feb 12 02:41:27 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 02:41:27 nfs1-cluster openais[2605]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 02:41:27 nfs1-cluster openais[2605]: [TOTEM] Sending initial ORF token 
Feb 12 02:41:27 nfs1-cluster openais[2605]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 02:41:27 nfs1-cluster openais[2605]: [CLM  ] New Configuration: 
Feb 12 02:41:27 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 02:41:27 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 02:41:27 nfs1-cluster openais[2605]: [CLM  ] Members Left: 
Feb 12 02:41:27 nfs1-cluster openais[2605]: [CLM  ] Members Joined: 
Feb 12 02:41:27 nfs1-cluster openais[2605]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 02:41:27 nfs1-cluster openais[2605]: [CLM  ] New Configuration: 
Feb 12 02:41:27 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 02:41:27 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 02:41:27 nfs1-cluster openais[2605]: [CLM  ] Members Left: 
Feb 12 02:41:27 nfs1-cluster openais[2605]: [CLM  ] Members Joined: 
Feb 12 02:41:27 nfs1-cluster openais[2605]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 02:41:27 nfs1-cluster openais[2605]: [TOTEM] entering OPERATIONAL state. 
Feb 12 02:41:27 nfs1-cluster openais[2605]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 02:41:27 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2  
Feb 12 02:41:27 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3  
Feb 12 02:41:27 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3 4  
Feb 12 02:41:27 nfs1-cluster openais[2605]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 02:41:27 nfs1-cluster openais[2605]: [CPG  ] got joinlist message from node 1 
Feb 12 02:41:27 nfs1-cluster openais[2605]: [CPG  ] got joinlist message from node 2 
Feb 12 02:44:56 nfs1-cluster openais[2605]: [TOTEM] entering GATHER state from 12. 
Feb 12 02:44:56 nfs1-cluster openais[2605]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 02:44:56 nfs1-cluster openais[2605]: [TOTEM] Saving state aru 1e high seq received 1e 
Feb 12 02:44:56 nfs1-cluster openais[2605]: [TOTEM] Storing new sequence id for ring 107c 
Feb 12 02:44:56 nfs1-cluster openais[2605]: [TOTEM] entering COMMIT state. 
Feb 12 02:45:01 nfs1-cluster openais[2605]: [TOTEM] entering RECOVERY state. 
Feb 12 02:45:01 nfs1-cluster openais[2605]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 02:45:01 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4216 rep 140.90.91.240 
Feb 12 02:45:01 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 02:45:01 nfs1-cluster openais[2605]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 02:45:01 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4216 rep 140.90.91.240 
Feb 12 02:45:01 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 02:45:01 nfs1-cluster openais[2605]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 02:45:01 nfs1-cluster openais[2605]: [TOTEM] Sending initial ORF token 
Feb 12 02:45:01 nfs1-cluster openais[2605]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 02:45:01 nfs1-cluster openais[2605]: [CLM  ] New Configuration: 
Feb 12 02:45:01 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 02:45:01 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 02:45:01 nfs1-cluster openais[2605]: [CLM  ] Members Left: 
Feb 12 02:45:01 nfs1-cluster openais[2605]: [CLM  ] Members Joined: 
Feb 12 02:45:01 nfs1-cluster openais[2605]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 02:45:01 nfs1-cluster openais[2605]: [CLM  ] New Configuration: 
Feb 12 02:45:01 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 02:45:01 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 02:45:01 nfs1-cluster openais[2605]: [CLM  ] Members Left: 
Feb 12 02:45:01 nfs1-cluster openais[2605]: [CLM  ] Members Joined: 
Feb 12 02:45:01 nfs1-cluster openais[2605]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 02:45:01 nfs1-cluster openais[2605]: [TOTEM] entering OPERATIONAL state. 
Feb 12 02:45:01 nfs1-cluster openais[2605]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 02:45:01 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2  
Feb 12 02:45:01 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3  
Feb 12 02:45:01 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3 4  
Feb 12 02:45:01 nfs1-cluster openais[2605]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 02:45:01 nfs1-cluster openais[2605]: [CPG  ] got joinlist message from node 1 
Feb 12 02:45:01 nfs1-cluster openais[2605]: [CPG  ] got joinlist message from node 2 
Feb 12 03:12:47 nfs1-cluster openais[2605]: [TOTEM] entering GATHER state from 12. 
Feb 12 03:12:47 nfs1-cluster openais[2605]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 03:12:47 nfs1-cluster openais[2605]: [TOTEM] Saving state aru 1e high seq received 1e 
Feb 12 03:12:47 nfs1-cluster openais[2605]: [TOTEM] Storing new sequence id for ring 1080 
Feb 12 03:12:47 nfs1-cluster openais[2605]: [TOTEM] entering COMMIT state. 
Feb 12 03:12:57 nfs1-cluster openais[2605]: [TOTEM] The token was lost in the COMMIT state. 
Feb 12 03:12:57 nfs1-cluster openais[2605]: [TOTEM] entering GATHER state from 4. 
Feb 12 03:12:57 nfs1-cluster openais[2605]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 03:12:57 nfs1-cluster openais[2605]: [TOTEM] Storing new sequence id for ring 1084 
Feb 12 03:12:57 nfs1-cluster openais[2605]: [TOTEM] entering COMMIT state. 
Feb 12 03:12:57 nfs1-cluster openais[2605]: [TOTEM] entering RECOVERY state. 
Feb 12 03:12:57 nfs1-cluster openais[2605]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 03:12:57 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4220 rep 140.90.91.240 
Feb 12 03:12:57 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 03:12:57 nfs1-cluster openais[2605]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 03:12:57 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4224 rep 140.90.91.242 
Feb 12 03:12:57 nfs1-cluster openais[2605]: [TOTEM] aru 32 high delivered 32 received flag 1 
Feb 12 03:12:57 nfs1-cluster openais[2605]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 03:12:57 nfs1-cluster openais[2605]: [TOTEM] Sending initial ORF token 
Feb 12 03:12:57 nfs1-cluster openais[2605]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 03:12:57 nfs1-cluster openais[2605]: [CLM  ] New Configuration: 
Feb 12 03:12:57 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 03:12:57 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 03:12:57 nfs1-cluster openais[2605]: [CLM  ] Members Left: 
Feb 12 03:12:57 nfs1-cluster openais[2605]: [CLM  ] Members Joined: 
Feb 12 03:12:57 nfs1-cluster openais[2605]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 03:12:57 nfs1-cluster openais[2605]: [CLM  ] New Configuration: 
Feb 12 03:12:57 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 03:12:57 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 03:12:57 nfs1-cluster openais[2605]: [CLM  ] Members Left: 
Feb 12 03:12:57 nfs1-cluster openais[2605]: [CLM  ] Members Joined: 
Feb 12 03:12:57 nfs1-cluster openais[2605]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 03:12:57 nfs1-cluster openais[2605]: [TOTEM] entering OPERATIONAL state. 
Feb 12 03:12:57 nfs1-cluster openais[2605]: [CMAN ] cman killed by node 2 because we rejoined the cluster without a full restart 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] AIS Executive Service RELEASE 'subrev 1358 version 0.80.3' 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] Copyright (C) 2002-2006 MontaVista Software, Inc and contributors. 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] Copyright (C) 2006 Red Hat, Inc. 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] AIS Executive Service: started and ready to provide service. 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] Using default multicast address of 239.192.83.8 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] openais component openais_cpg loaded. 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] Registering service handler 'openais cluster closed process group service v1.01' 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] openais component openais_cfg loaded. 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] Registering service handler 'openais configuration service' 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] openais component openais_msg loaded. 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] Registering service handler 'openais message service B.01.01' 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] openais component openais_lck loaded. 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] Registering service handler 'openais distributed locking service B.01.01' 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] openais component openais_evt loaded. 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] Registering service handler 'openais event service B.01.01' 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] openais component openais_ckpt loaded. 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] Registering service handler 'openais checkpoint service B.01.01' 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] openais component openais_amf loaded. 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] Registering service handler 'openais availability management framework B.01.01' 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] openais component openais_clm loaded. 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] Registering service handler 'openais cluster membership service B.01.01' 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] openais component openais_evs loaded. 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] Registering service handler 'openais extended virtual synchrony service' 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] openais component openais_cman loaded. 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] Registering service handler 'openais CMAN membership service 2.01' 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] Token Timeout (10000 ms) retransmit timeout (495 ms) 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] token hold (386 ms) retransmits before loss (20 retrans) 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] join (60 ms) send_join (0 ms) consensus (4800 ms) merge (200 ms) 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] downcheck (1000 ms) fail to recv const (50 msgs) 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] seqno unchanged const (30 rotations) Maximum network MTU 1500 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] window size per rotation (50 messages) maximum messages per rotation (17 messages) 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] send threads (0 threads) 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] RRP token expired timeout (495 ms) 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] RRP token problem counter (2000 ms) 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] RRP threshold (10 problem count) 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] RRP mode set to none. 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] heartbeat_failures_allowed (0) 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] max_network_delay (50 ms) 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] HeartBeat is Disabled. To enable set heartbeat_failures_allowed > 0 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] Receive multicast socket recv buffer size (262142 bytes). 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] The network interface [140.90.91.240] is now up. 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] Created or loaded sequence id 4228.140.90.91.240 for this ring. 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] entering GATHER state from 15. 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [SERV ] Initialising service handler 'openais extended virtual synchrony service' 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [SERV ] Initialising service handler 'openais cluster membership service B.01.01' 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [SERV ] Initialising service handler 'openais availability management framework B.01.01' 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [SERV ] Initialising service handler 'openais checkpoint service B.01.01' 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [SERV ] Initialising service handler 'openais event service B.01.01' 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [SERV ] Initialising service handler 'openais distributed locking service B.01.01' 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [SERV ] Initialising service handler 'openais message service B.01.01' 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [SERV ] Initialising service handler 'openais configuration service' 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [SERV ] Initialising service handler 'openais cluster closed process group service v1.01' 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [SERV ] Initialising service handler 'openais CMAN membership service 2.01' 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [CMAN ] CMAN 2.0.73 (built Nov 29 2007 18:41:45) started 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [SYNC ] Not using a virtual synchrony filter. 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] Saving state aru 0 high seq received 0 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] Storing new sequence id for ring 1088 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] entering COMMIT state. 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] entering RECOVERY state. 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4228 rep 140.90.91.240 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] aru 0 high delivered 0 received flag 1 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] Sending initial ORF token 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [CLM  ] New Configuration: 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [CLM  ] Members Left: 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [CLM  ] Members Joined: 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [CLM  ] New Configuration: 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 03:16:20 nfs1-cluster openais[2605]: [CLM  ] Members Left: 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [CLM  ] Members Joined: 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 03:16:20 nfs1-cluster openais[2605]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] entering OPERATIONAL state. 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [CMAN ] quorum regained, resuming activity 
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] entering GATHER state from 11. 
Feb 12 03:16:21 nfs1-cluster openais[2605]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 03:16:21 nfs1-cluster openais[2605]: [TOTEM] Saving state aru 3 high seq received 3 
Feb 12 03:16:21 nfs1-cluster openais[2605]: [TOTEM] Storing new sequence id for ring 108c 
Feb 12 03:16:21 nfs1-cluster openais[2605]: [TOTEM] entering COMMIT state. 
Feb 12 03:16:21 nfs1-cluster openais[2605]: [TOTEM] entering RECOVERY state. 
Feb 12 03:16:21 nfs1-cluster openais[2605]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 03:16:21 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4232 rep 140.90.91.240 
Feb 12 03:16:21 nfs1-cluster openais[2605]: [TOTEM] aru 3 high delivered 1 received flag 1 
Feb 12 03:16:21 nfs1-cluster openais[2605]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 03:16:21 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4232 rep 140.90.91.242 
Feb 12 03:16:21 nfs1-cluster openais[2605]: [TOTEM] aru 3c high delivered 3c received flag 1 
Feb 12 03:16:21 nfs1-cluster openais[2605]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 03:16:21 nfs1-cluster openais[2605]: [TOTEM] Sending initial ORF token 
Feb 12 03:16:21 nfs1-cluster openais[2605]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 03:16:21 nfs1-cluster openais[2605]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 03:16:21 nfs1-cluster openais[2605]: [CLM  ] New Configuration: 
Feb 12 03:16:21 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 03:16:21 nfs1-cluster openais[2605]: [CLM  ] Members Left: 
Feb 12 03:16:21 nfs1-cluster openais[2605]: [CLM  ] Members Joined: 
Feb 12 03:16:21 nfs1-cluster openais[2605]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 03:16:21 nfs1-cluster openais[2605]: [CLM  ] New Configuration: 
Feb 12 03:16:21 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 03:16:21 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 03:16:21 nfs1-cluster openais[2605]: [CLM  ] Members Left: 
Feb 12 03:16:21 nfs1-cluster openais[2605]: [CLM  ] Members Joined: 
Feb 12 03:16:21 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 03:16:21 nfs1-cluster openais[2605]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 03:16:21 nfs1-cluster openais[2605]: [TOTEM] entering OPERATIONAL state. 
Feb 12 03:16:21 nfs1-cluster openais[2605]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 03:16:21 nfs1-cluster openais[2605]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 03:16:21 nfs1-cluster openais[2605]: [CPG  ] got joinlist message from node 2 
Feb 12 03:33:14 nfs1-cluster openais[2605]: [TOTEM] entering GATHER state from 12. 
Feb 12 03:33:14 nfs1-cluster openais[2605]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 03:33:14 nfs1-cluster openais[2605]: [TOTEM] Saving state aru fd high seq received fd 
Feb 12 03:33:14 nfs1-cluster openais[2605]: [TOTEM] Storing new sequence id for ring 1090 
Feb 12 03:33:14 nfs1-cluster openais[2605]: [TOTEM] entering COMMIT state. 
Feb 12 03:33:17 nfs1-cluster openais[2605]: [TOTEM] entering RECOVERY state. 
Feb 12 03:33:17 nfs1-cluster openais[2605]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 03:33:17 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4236 rep 140.90.91.240 
Feb 12 03:33:17 nfs1-cluster openais[2605]: [TOTEM] aru fd high delivered fd received flag 1 
Feb 12 03:33:17 nfs1-cluster openais[2605]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 03:33:17 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4236 rep 140.90.91.240 
Feb 12 03:33:17 nfs1-cluster openais[2605]: [TOTEM] aru fd high delivered fd received flag 1 
Feb 12 03:33:17 nfs1-cluster openais[2605]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 03:33:17 nfs1-cluster openais[2605]: [TOTEM] Sending initial ORF token 
Feb 12 03:33:17 nfs1-cluster openais[2605]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 03:33:17 nfs1-cluster openais[2605]: [CLM  ] New Configuration: 
Feb 12 03:33:17 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 03:33:17 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 03:33:17 nfs1-cluster openais[2605]: [CLM  ] Members Left: 
Feb 12 03:33:17 nfs1-cluster openais[2605]: [CLM  ] Members Joined: 
Feb 12 03:33:17 nfs1-cluster openais[2605]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 03:33:17 nfs1-cluster openais[2605]: [CLM  ] New Configuration: 
Feb 12 03:33:17 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 03:33:17 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 03:33:17 nfs1-cluster openais[2605]: [CLM  ] Members Left: 
Feb 12 03:33:17 nfs1-cluster openais[2605]: [CLM  ] Members Joined: 
Feb 12 03:33:17 nfs1-cluster openais[2605]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 03:33:17 nfs1-cluster openais[2605]: [TOTEM] entering OPERATIONAL state. 
Feb 12 03:33:17 nfs1-cluster openais[2605]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 03:33:17 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2  
Feb 12 03:33:17 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3  
Feb 12 03:33:17 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3 4  
Feb 12 03:33:17 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3 4  
Feb 12 03:33:17 nfs1-cluster openais[2605]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 03:33:17 nfs1-cluster openais[2605]: [CPG  ] got joinlist message from node 1 
Feb 12 03:33:17 nfs1-cluster openais[2605]: [CPG  ] got joinlist message from node 2 
Feb 12 03:35:15 nfs1-cluster openais[2605]: [TOTEM] The token was lost in the OPERATIONAL state. 
Feb 12 03:35:15 nfs1-cluster openais[2605]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). 
Feb 12 03:35:15 nfs1-cluster openais[2605]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
Feb 12 03:35:15 nfs1-cluster openais[2605]: [TOTEM] entering GATHER state from 2. 
Feb 12 03:35:15 nfs1-cluster openais[2605]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 03:35:15 nfs1-cluster openais[2605]: [TOTEM] Saving state aru 1e high seq received 1e 
Feb 12 03:35:15 nfs1-cluster openais[2605]: [TOTEM] Storing new sequence id for ring 1094 
Feb 12 03:35:15 nfs1-cluster openais[2605]: [TOTEM] entering COMMIT state. 
Feb 12 03:35:18 nfs1-cluster openais[2605]: [TOTEM] entering RECOVERY state. 
Feb 12 03:35:18 nfs1-cluster openais[2605]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 03:35:18 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4240 rep 140.90.91.240 
Feb 12 03:35:18 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 03:35:18 nfs1-cluster openais[2605]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 03:35:18 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4240 rep 140.90.91.240 
Feb 12 03:35:18 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 03:35:18 nfs1-cluster openais[2605]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 03:35:18 nfs1-cluster openais[2605]: [TOTEM] Sending initial ORF token 
Feb 12 03:35:18 nfs1-cluster openais[2605]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 03:35:18 nfs1-cluster openais[2605]: [CLM  ] New Configuration: 
Feb 12 03:35:18 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 03:35:18 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 03:35:18 nfs1-cluster openais[2605]: [CLM  ] Members Left: 
Feb 12 03:35:18 nfs1-cluster openais[2605]: [CLM  ] Members Joined: 
Feb 12 03:35:18 nfs1-cluster openais[2605]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 03:35:18 nfs1-cluster openais[2605]: [CLM  ] New Configuration: 
Feb 12 03:35:18 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 03:35:18 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 03:35:18 nfs1-cluster openais[2605]: [CLM  ] Members Left: 
Feb 12 03:35:18 nfs1-cluster openais[2605]: [CLM  ] Members Joined: 
Feb 12 03:35:18 nfs1-cluster openais[2605]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 03:35:18 nfs1-cluster openais[2605]: [TOTEM] entering OPERATIONAL state. 
Feb 12 03:35:18 nfs1-cluster openais[2605]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 03:35:18 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2  
Feb 12 03:35:18 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3  
Feb 12 03:35:18 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3 4  
Feb 12 03:35:18 nfs1-cluster openais[2605]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 03:35:18 nfs1-cluster openais[2605]: [CPG  ] got joinlist message from node 1 
Feb 12 03:35:18 nfs1-cluster openais[2605]: [CPG  ] got joinlist message from node 2 
Feb 12 03:36:14 nfs1-cluster openais[2605]: [TOTEM] entering GATHER state from 12. 
Feb 12 03:36:14 nfs1-cluster openais[2605]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 03:36:14 nfs1-cluster openais[2605]: [TOTEM] Saving state aru 1e high seq received 1e 
Feb 12 03:36:14 nfs1-cluster openais[2605]: [TOTEM] Storing new sequence id for ring 1098 
Feb 12 03:36:14 nfs1-cluster openais[2605]: [TOTEM] entering COMMIT state. 
Feb 12 03:36:24 nfs1-cluster openais[2605]: [TOTEM] The token was lost in the COMMIT state. 
Feb 12 03:36:24 nfs1-cluster openais[2605]: [TOTEM] entering GATHER state from 4. 
Feb 12 03:36:24 nfs1-cluster openais[2605]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 03:36:24 nfs1-cluster openais[2605]: [TOTEM] Storing new sequence id for ring 109c 
Feb 12 03:36:24 nfs1-cluster openais[2605]: [TOTEM] entering COMMIT state. 
Feb 12 03:36:24 nfs1-cluster openais[2605]: [TOTEM] entering RECOVERY state. 
Feb 12 03:36:24 nfs1-cluster openais[2605]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 03:36:24 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4244 rep 140.90.91.240 
Feb 12 03:36:24 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 03:36:24 nfs1-cluster openais[2605]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 03:36:24 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4248 rep 140.90.91.242 
Feb 12 03:36:24 nfs1-cluster openais[2605]: [TOTEM] aru 32 high delivered 32 received flag 1 
Feb 12 03:36:24 nfs1-cluster openais[2605]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 03:36:24 nfs1-cluster openais[2605]: [TOTEM] Sending initial ORF token 
Feb 12 03:36:24 nfs1-cluster openais[2605]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 03:36:24 nfs1-cluster openais[2605]: [CLM  ] New Configuration: 
Feb 12 03:36:24 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 03:36:24 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 03:36:24 nfs1-cluster openais[2605]: [CLM  ] Members Left: 
Feb 12 03:36:24 nfs1-cluster openais[2605]: [CLM  ] Members Joined: 
Feb 12 03:36:24 nfs1-cluster openais[2605]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 03:36:24 nfs1-cluster openais[2605]: [CLM  ] New Configuration: 
Feb 12 03:36:24 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 03:36:24 nfs1-cluster openais[2605]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 03:36:24 nfs1-cluster openais[2605]: [CLM  ] Members Left: 
Feb 12 03:36:24 nfs1-cluster openais[2605]: [CLM  ] Members Joined: 
Feb 12 03:36:24 nfs1-cluster openais[2605]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 03:36:24 nfs1-cluster openais[2605]: [TOTEM] entering OPERATIONAL state. 
Feb 12 03:36:24 nfs1-cluster openais[2605]: [CMAN ] cman killed by node 2 because we rejoined the cluster without a full restart 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] AIS Executive Service RELEASE 'subrev 1358 version 0.80.3' 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] Copyright (C) 2002-2006 MontaVista Software, Inc and contributors. 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] Copyright (C) 2006 Red Hat, Inc. 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] AIS Executive Service: started and ready to provide service. 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] Using default multicast address of 239.192.83.8 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] openais component openais_cpg loaded. 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] Registering service handler 'openais cluster closed process group service v1.01' 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] openais component openais_cfg loaded. 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] Registering service handler 'openais configuration service' 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] openais component openais_msg loaded. 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] Registering service handler 'openais message service B.01.01' 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] openais component openais_lck loaded. 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] Registering service handler 'openais distributed locking service B.01.01' 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] openais component openais_evt loaded. 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] Registering service handler 'openais event service B.01.01' 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] openais component openais_ckpt loaded. 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] Registering service handler 'openais checkpoint service B.01.01' 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] openais component openais_amf loaded. 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] Registering service handler 'openais availability management framework B.01.01' 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] openais component openais_clm loaded. 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] Registering service handler 'openais cluster membership service B.01.01' 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] openais component openais_evs loaded. 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] Registering service handler 'openais extended virtual synchrony service' 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] openais component openais_cman loaded. 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] Registering service handler 'openais CMAN membership service 2.01' 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] Token Timeout (10000 ms) retransmit timeout (495 ms) 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] token hold (386 ms) retransmits before loss (20 retrans) 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] join (60 ms) send_join (0 ms) consensus (4800 ms) merge (200 ms) 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] downcheck (1000 ms) fail to recv const (50 msgs) 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] seqno unchanged const (30 rotations) Maximum network MTU 1500 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] window size per rotation (50 messages) maximum messages per rotation (17 messages) 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] send threads (0 threads) 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] RRP token expired timeout (495 ms) 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] RRP token problem counter (2000 ms) 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] RRP threshold (10 problem count) 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] RRP mode set to none. 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] heartbeat_failures_allowed (0) 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] max_network_delay (50 ms) 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] HeartBeat is Disabled. To enable set heartbeat_failures_allowed > 0 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (262142 bytes). 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] The network interface [140.90.91.240] is now up. 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] Created or loaded sequence id 4252.140.90.91.240 for this ring. 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 15. 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [SERV ] Initialising service handler 'openais extended virtual synchrony service' 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [SERV ] Initialising service handler 'openais cluster membership service B.01.01' 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [SERV ] Initialising service handler 'openais availability management framework B.01.01' 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [SERV ] Initialising service handler 'openais checkpoint service B.01.01' 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [SERV ] Initialising service handler 'openais event service B.01.01' 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [SERV ] Initialising service handler 'openais distributed locking service B.01.01' 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [SERV ] Initialising service handler 'openais message service B.01.01' 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [SERV ] Initialising service handler 'openais configuration service' 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [SERV ] Initialising service handler 'openais cluster closed process group service v1.01' 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [SERV ] Initialising service handler 'openais CMAN membership service 2.01' 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [CMAN ] CMAN 2.0.73 (built Nov 29 2007 18:41:45) started 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [SYNC ] Not using a virtual synchrony filter. 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 0 high seq received 0 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10a0 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state. 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state. 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4252 rep 140.90.91.240 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] aru 0 high delivered 0 received flag 1 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 03:39:48 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 03:39:49 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 03:39:49 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 03:39:49 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 03:39:49 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 03:39:49 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 03:39:49 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 03:39:49 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 03:39:49 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state. 
Feb 12 03:39:49 nfs1-cluster openais[2604]: [CMAN ] quorum regained, resuming activity 
Feb 12 03:39:49 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 11. 
Feb 12 03:39:49 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 03:39:49 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 3 high seq received 3 
Feb 12 03:39:49 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10a4 
Feb 12 03:39:49 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state. 
Feb 12 03:39:49 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state. 
Feb 12 03:39:49 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 03:39:49 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4256 rep 140.90.91.240 
Feb 12 03:39:49 nfs1-cluster openais[2604]: [TOTEM] aru 3 high delivered 1 received flag 1 
Feb 12 03:39:49 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 03:39:49 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4256 rep 140.90.91.242 
Feb 12 03:39:49 nfs1-cluster openais[2604]: [TOTEM] aru 3d high delivered 3d received flag 1 
Feb 12 03:39:49 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 03:39:49 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token 
Feb 12 03:39:49 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 03:39:49 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 03:39:49 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 03:39:49 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 03:39:49 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 03:39:49 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 03:39:49 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 03:39:49 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 03:39:49 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 03:39:49 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 03:39:49 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 03:39:49 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 03:39:49 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 03:39:49 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 03:39:49 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state. 
Feb 12 03:39:49 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 03:39:49 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 03:39:49 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 2 
Feb 12 03:40:01 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: bd  
Feb 12 03:40:44 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state. 
Feb 12 03:40:44 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). 
Feb 12 03:40:44 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
Feb 12 03:40:44 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2. 
Feb 12 03:40:44 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 03:40:44 nfs1-cluster openais[2604]: [TOTEM] Saving state aru fa high seq received fa 
Feb 12 03:40:44 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10a8 
Feb 12 03:40:44 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state. 
Feb 12 03:40:47 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state. 
Feb 12 03:40:47 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 03:40:47 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4260 rep 140.90.91.240 
Feb 12 03:40:47 nfs1-cluster openais[2604]: [TOTEM] aru fa high delivered fa received flag 1 
Feb 12 03:40:47 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 03:40:47 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4260 rep 140.90.91.240 
Feb 12 03:40:47 nfs1-cluster openais[2604]: [TOTEM] aru fa high delivered fa received flag 1 
Feb 12 03:40:47 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 03:40:47 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token 
Feb 12 03:40:47 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 03:40:47 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 03:40:47 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 03:40:47 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 03:40:47 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 03:40:47 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 03:40:47 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 03:40:47 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 03:40:47 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 03:40:47 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 03:40:47 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 03:40:47 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 03:40:47 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 03:40:47 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state. 
Feb 12 03:40:47 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2  
Feb 12 03:40:47 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 03:40:47 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2  
Feb 12 03:40:47 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 3  
Feb 12 03:40:47 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 03:40:47 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 1 
Feb 12 03:40:47 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 2 
Feb 12 03:51:37 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state. 
Feb 12 03:51:37 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). 
Feb 12 03:51:37 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
Feb 12 03:51:37 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2. 
Feb 12 03:51:37 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 03:51:37 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e 
Feb 12 03:51:37 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10ac 
Feb 12 03:51:37 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state. 
Feb 12 03:51:41 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state. 
Feb 12 03:51:41 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 03:51:41 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4264 rep 140.90.91.240 
Feb 12 03:51:41 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 03:51:41 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 03:51:41 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4264 rep 140.90.91.240 
Feb 12 03:51:41 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 03:51:41 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 03:51:41 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token 
Feb 12 03:51:41 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 03:51:41 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 03:51:41 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 03:51:41 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 03:51:41 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 03:51:41 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 03:51:41 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 03:51:41 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 03:51:41 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 03:51:41 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 03:51:41 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 03:51:41 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 03:51:41 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 03:51:41 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state. 
Feb 12 03:51:41 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 03:51:41 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2  
Feb 12 03:51:41 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3  
Feb 12 03:51:41 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3 4  
Feb 12 03:51:41 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3 4  
Feb 12 03:51:41 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 03:51:41 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 1 
Feb 12 03:51:41 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 2 
Feb 12 03:53:41 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state. 
Feb 12 03:53:41 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). 
Feb 12 03:53:41 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
Feb 12 03:53:41 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2. 
Feb 12 03:53:41 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 03:53:41 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e 
Feb 12 03:53:41 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10b0 
Feb 12 03:53:41 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state. 
Feb 12 03:53:42 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state. 
Feb 12 03:53:42 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 03:53:42 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4268 rep 140.90.91.240 
Feb 12 03:53:42 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 03:53:42 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 03:53:42 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4268 rep 140.90.91.240 
Feb 12 03:53:42 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 03:53:42 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 03:53:42 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token 
Feb 12 03:53:42 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 03:53:42 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 03:53:42 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 03:53:42 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 03:53:42 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 03:53:42 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 03:53:42 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 03:53:42 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 03:53:42 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 03:53:42 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 03:53:42 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 03:53:42 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 03:53:42 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 03:53:42 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state. 
Feb 12 03:53:42 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 03:53:42 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2  
Feb 12 03:53:42 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3  
Feb 12 03:53:42 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3 4  
Feb 12 03:53:42 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 03:53:42 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 1 
Feb 12 03:53:42 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 2 
Feb 12 04:10:54 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state. 
Feb 12 04:10:54 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). 
Feb 12 04:10:54 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
Feb 12 04:10:54 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2. 
Feb 12 04:10:54 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 04:10:54 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e 
Feb 12 04:10:54 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10b4 
Feb 12 04:10:54 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state. 
Feb 12 04:10:56 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state. 
Feb 12 04:10:56 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 04:10:56 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4272 rep 140.90.91.240 
Feb 12 04:10:56 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 04:10:56 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 04:10:56 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4272 rep 140.90.91.240 
Feb 12 04:10:56 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 04:10:56 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 04:10:56 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token 
Feb 12 04:10:56 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 04:10:56 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 04:10:56 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 04:10:56 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 04:10:56 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 04:10:56 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 04:10:56 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 04:10:56 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 04:10:56 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 04:10:56 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 04:10:56 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 04:10:56 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 04:10:56 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 04:10:56 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state. 
Feb 12 04:10:56 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 04:10:56 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2  
Feb 12 04:10:56 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3  
Feb 12 04:10:56 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3 4  
Feb 12 04:10:57 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 04:10:57 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 1 
Feb 12 04:10:57 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 2 
Feb 12 04:11:08 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state. 
Feb 12 04:11:08 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). 
Feb 12 04:11:08 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
Feb 12 04:11:08 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2. 
Feb 12 04:11:08 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 04:11:08 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e 
Feb 12 04:11:08 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10b8 
Feb 12 04:11:08 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state. 
Feb 12 04:11:11 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state. 
Feb 12 04:11:11 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 04:11:11 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4276 rep 140.90.91.240 
Feb 12 04:11:11 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 04:11:11 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 04:11:11 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4276 rep 140.90.91.240 
Feb 12 04:11:11 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 04:11:11 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 04:11:11 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token 
Feb 12 04:11:11 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 04:11:11 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 04:11:11 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 04:11:11 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 04:11:11 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 04:11:11 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 04:11:11 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 04:11:11 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 04:11:11 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 04:11:11 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 04:11:11 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 04:11:11 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 04:11:11 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 04:11:11 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state. 
Feb 12 04:11:11 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 04:11:11 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2  
Feb 12 04:11:11 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3  
Feb 12 04:11:11 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3 4  
Feb 12 04:11:11 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3 4  
Feb 12 04:11:11 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 04:11:11 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 1 
Feb 12 04:11:11 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 2 
Feb 12 04:16:58 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state. 
Feb 12 04:16:58 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). 
Feb 12 04:16:58 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
Feb 12 04:16:58 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2. 
Feb 12 04:16:58 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 04:16:58 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e 
Feb 12 04:16:58 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10bc 
Feb 12 04:16:58 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state. 
Feb 12 04:17:01 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state. 
Feb 12 04:17:01 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 04:17:01 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4280 rep 140.90.91.240 
Feb 12 04:17:01 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 04:17:01 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 04:17:01 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4280 rep 140.90.91.240 
Feb 12 04:17:01 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 04:17:01 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 04:17:01 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token 
Feb 12 04:17:01 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 04:17:01 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 04:17:01 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 04:17:01 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 04:17:01 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 04:17:01 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 04:17:01 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 04:17:01 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 04:17:01 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 04:17:01 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 04:17:01 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 04:17:01 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 04:17:01 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 04:17:01 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state. 
Feb 12 04:17:01 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 04:17:01 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2  
Feb 12 04:17:01 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3  
Feb 12 04:17:01 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3 4  
Feb 12 04:17:01 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3 4  
Feb 12 04:17:01 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 04:17:01 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 1 
Feb 12 04:17:01 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 2 
Feb 12 04:28:54 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state. 
Feb 12 04:28:54 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). 
Feb 12 04:28:54 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
Feb 12 04:28:54 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2. 
Feb 12 04:28:54 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 04:28:54 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e 
Feb 12 04:28:54 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10c0 
Feb 12 04:28:54 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state. 
Feb 12 04:28:56 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state. 
Feb 12 04:28:56 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 04:28:56 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4284 rep 140.90.91.240 
Feb 12 04:28:56 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 04:28:56 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 04:28:56 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4284 rep 140.90.91.240 
Feb 12 04:28:56 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 04:28:56 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 04:28:56 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token 
Feb 12 04:28:56 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 04:28:56 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 04:28:56 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 04:28:56 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 04:28:56 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 04:28:56 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 04:28:56 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 04:28:56 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 04:28:56 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 04:28:56 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 04:28:56 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 04:28:56 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 04:28:56 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 04:28:56 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state. 
Feb 12 04:28:56 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 04:28:56 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2  
Feb 12 04:28:56 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3  
Feb 12 04:28:56 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 4  
Feb 12 04:28:56 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 04:28:56 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 1 
Feb 12 04:28:56 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 2 
Feb 12 04:53:45 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state. 
Feb 12 04:53:45 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). 
Feb 12 04:53:45 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
Feb 12 04:53:45 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2. 
Feb 12 04:53:45 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 04:53:45 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e 
Feb 12 04:53:45 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10c4 
Feb 12 04:53:45 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state. 
Feb 12 04:53:45 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state. 
Feb 12 04:53:45 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 04:53:45 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4288 rep 140.90.91.240 
Feb 12 04:53:45 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 04:53:45 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 04:53:45 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4288 rep 140.90.91.240 
Feb 12 04:53:45 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 04:53:45 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 04:53:45 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token 
Feb 12 04:53:45 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 04:53:45 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 04:53:45 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 04:53:45 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 04:53:45 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 04:53:45 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 04:53:45 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 04:53:45 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 04:53:45 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 04:53:45 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 04:53:45 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 04:53:45 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 04:53:45 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 04:53:45 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state. 
Feb 12 04:53:45 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 04:53:45 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 04:53:45 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 1 
Feb 12 04:53:45 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 2 
Feb 12 05:05:39 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state. 
Feb 12 05:05:39 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). 
Feb 12 05:05:39 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
Feb 12 05:05:39 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2. 
Feb 12 05:05:39 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 05:05:39 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e 
Feb 12 05:05:39 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10c8 
Feb 12 05:05:39 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state. 
Feb 12 05:05:39 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state. 
Feb 12 05:05:39 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 05:05:39 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4292 rep 140.90.91.240 
Feb 12 05:05:39 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 05:05:39 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 05:05:39 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4292 rep 140.90.91.240 
Feb 12 05:05:39 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 05:05:39 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 05:05:39 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token 
Feb 12 05:05:39 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 05:05:39 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 05:05:39 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 05:05:39 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 05:05:39 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 05:05:39 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 05:05:39 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 05:05:39 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 05:05:39 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 05:05:39 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 05:05:39 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 05:05:39 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 05:05:39 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 05:05:39 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state. 
Feb 12 05:05:39 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 05:05:39 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2  
Feb 12 05:05:39 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3  
Feb 12 05:05:39 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3 4  
Feb 12 05:05:39 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 05:05:39 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 1 
Feb 12 05:05:39 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 2 
Feb 12 05:09:55 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state. 
Feb 12 05:09:55 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). 
Feb 12 05:09:55 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
Feb 12 05:09:55 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2. 
Feb 12 05:09:55 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 05:09:55 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e 
Feb 12 05:09:55 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10cc 
Feb 12 05:09:55 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state. 
Feb 12 05:09:58 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state. 
Feb 12 05:09:58 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 05:09:58 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4296 rep 140.90.91.240 
Feb 12 05:09:58 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 05:09:58 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 05:09:58 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4296 rep 140.90.91.240 
Feb 12 05:09:58 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 05:09:58 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 05:09:58 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token 
Feb 12 05:09:58 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 05:09:58 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 05:09:58 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 05:09:58 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 05:09:58 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 05:09:58 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 05:09:58 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 05:09:58 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 05:09:58 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 05:09:58 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 05:09:58 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 05:09:58 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 05:09:58 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 05:09:58 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state. 
Feb 12 05:09:58 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 05:09:58 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2  
Feb 12 05:09:58 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3  
Feb 12 05:09:58 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3 4  
Feb 12 05:09:58 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3 4  
Feb 12 05:09:58 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 05:09:58 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 1 
Feb 12 05:09:58 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 2 
Feb 12 05:48:14 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state. 
Feb 12 05:48:14 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). 
Feb 12 05:48:14 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
Feb 12 05:48:14 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2. 
Feb 12 05:48:14 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 05:48:14 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e 
Feb 12 05:48:14 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10d0 
Feb 12 05:48:14 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state. 
Feb 12 05:48:14 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state. 
Feb 12 05:48:14 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 05:48:14 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4300 rep 140.90.91.240 
Feb 12 05:48:14 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 05:48:14 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 05:48:14 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4300 rep 140.90.91.240 
Feb 12 05:48:14 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 05:48:14 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 05:48:14 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token 
Feb 12 05:48:14 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 05:48:14 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 05:48:14 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 05:48:14 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 05:48:14 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 05:48:14 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 05:48:14 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 05:48:14 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 05:48:14 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 05:48:14 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 05:48:14 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 05:48:14 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 05:48:14 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 05:48:14 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state. 
Feb 12 05:48:14 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 05:48:14 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 05:48:14 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 1 
Feb 12 05:48:14 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 2 
Feb 12 05:55:34 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state. 
Feb 12 05:55:34 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). 
Feb 12 05:55:34 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
Feb 12 05:55:34 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2. 
Feb 12 05:55:34 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 05:55:34 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e 
Feb 12 05:55:34 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10d4 
Feb 12 05:55:34 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state. 
Feb 12 05:55:34 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state. 
Feb 12 05:55:34 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 05:55:34 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4304 rep 140.90.91.240 
Feb 12 05:55:34 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 05:55:34 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 05:55:34 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4304 rep 140.90.91.240 
Feb 12 05:55:34 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 05:55:34 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 05:55:34 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token 
Feb 12 05:55:34 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 05:55:34 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 05:55:34 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 05:55:34 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 05:55:34 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 05:55:34 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 05:55:34 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 05:55:34 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 05:55:34 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 05:55:34 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 05:55:34 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 05:55:34 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 05:55:34 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 05:55:34 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state. 
Feb 12 05:55:34 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 05:55:34 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2  
Feb 12 05:55:34 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3  
Feb 12 05:55:34 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3 4  
Feb 12 05:55:35 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 05:55:35 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 1 
Feb 12 05:55:35 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 2 
Feb 12 05:58:37 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state. 
Feb 12 05:58:37 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). 
Feb 12 05:58:37 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
Feb 12 05:58:37 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2. 
Feb 12 05:58:37 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 05:58:37 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e 
Feb 12 05:58:37 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10d8 
Feb 12 05:58:37 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state. 
Feb 12 05:58:37 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state. 
Feb 12 05:58:37 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 05:58:37 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4308 rep 140.90.91.240 
Feb 12 05:58:37 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 05:58:37 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 05:58:37 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4308 rep 140.90.91.240 
Feb 12 05:58:37 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 05:58:37 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 05:58:37 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token 
Feb 12 05:58:37 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 05:58:37 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 05:58:37 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 05:58:37 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 05:58:37 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 05:58:37 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 05:58:37 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 05:58:37 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 05:58:37 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 05:58:37 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 05:58:37 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 05:58:37 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 05:58:37 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 05:58:37 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state. 
Feb 12 05:58:37 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 05:58:37 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 05:58:37 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 1 
Feb 12 05:58:37 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 2 
Feb 12 06:28:45 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state. 
Feb 12 06:28:45 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). 
Feb 12 06:28:45 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
Feb 12 06:28:45 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2. 
Feb 12 06:28:45 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 06:28:45 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e 
Feb 12 06:28:45 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10dc 
Feb 12 06:28:45 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state. 
Feb 12 06:28:46 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state. 
Feb 12 06:28:46 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 06:28:46 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4312 rep 140.90.91.240 
Feb 12 06:28:46 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 06:28:46 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 06:28:46 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4312 rep 140.90.91.240 
Feb 12 06:28:46 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 06:28:46 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 06:28:46 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token 
Feb 12 06:28:46 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 06:28:46 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 06:28:46 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 06:28:46 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 06:28:46 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 06:28:46 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 06:28:46 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 06:28:46 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 06:28:46 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 06:28:46 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 06:28:46 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 06:28:46 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 06:28:46 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 06:28:46 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state. 
Feb 12 06:28:46 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 06:28:46 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2  
Feb 12 06:28:46 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3  
Feb 12 06:28:46 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3 4  
Feb 12 06:28:46 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3 4  
Feb 12 06:28:46 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 06:28:46 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 1 
Feb 12 06:28:46 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 2 
Feb 12 06:42:10 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state. 
Feb 12 06:42:10 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). 
Feb 12 06:42:10 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
Feb 12 06:42:10 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2. 
Feb 12 06:42:10 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 06:42:10 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e 
Feb 12 06:42:10 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10e0 
Feb 12 06:42:10 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state. 
Feb 12 06:42:12 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state. 
Feb 12 06:42:12 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 06:42:12 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4316 rep 140.90.91.240 
Feb 12 06:42:12 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 06:42:12 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 06:42:12 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4316 rep 140.90.91.240 
Feb 12 06:42:12 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 06:42:12 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 06:42:12 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token 
Feb 12 06:42:12 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 06:42:12 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 06:42:12 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 06:42:12 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 06:42:12 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 06:42:12 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 06:42:12 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 06:42:12 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 06:42:12 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 06:42:12 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 06:42:12 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 06:42:12 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 06:42:12 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 06:42:12 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state. 
Feb 12 06:42:12 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 06:42:12 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2  
Feb 12 06:42:12 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3  
Feb 12 06:42:12 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3 4  
Feb 12 06:42:12 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3 4  
Feb 12 06:42:12 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 06:42:12 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 1 
Feb 12 06:42:12 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 2 
Feb 12 07:38:26 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state. 
Feb 12 07:38:26 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). 
Feb 12 07:38:26 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
Feb 12 07:38:26 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2. 
Feb 12 07:38:26 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 07:38:26 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e 
Feb 12 07:38:26 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10e4 
Feb 12 07:38:26 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state. 
Feb 12 07:38:27 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state. 
Feb 12 07:38:27 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 07:38:27 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4320 rep 140.90.91.240 
Feb 12 07:38:27 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 07:38:27 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 07:38:27 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4320 rep 140.90.91.240 
Feb 12 07:38:27 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 07:38:27 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 07:38:27 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token 
Feb 12 07:38:27 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 07:38:27 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 07:38:27 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 07:38:27 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 07:38:27 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 07:38:27 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 07:38:27 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 07:38:27 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 07:38:27 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 07:38:27 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 07:38:27 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 07:38:27 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 07:38:27 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 07:38:27 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state. 
Feb 12 07:38:27 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 07:38:27 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2  
Feb 12 07:38:27 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3  
Feb 12 07:38:27 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3 4  
Feb 12 07:38:27 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 07:38:27 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 1 
Feb 12 07:38:27 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 2 
Feb 12 07:43:29 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state. 
Feb 12 07:43:29 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). 
Feb 12 07:43:29 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
Feb 12 07:43:29 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2. 
Feb 12 07:43:29 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 07:43:29 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e 
Feb 12 07:43:29 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10e8 
Feb 12 07:43:29 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state. 
Feb 12 07:43:31 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state. 
Feb 12 07:43:31 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 07:43:31 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4324 rep 140.90.91.240 
Feb 12 07:43:31 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 07:43:31 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 07:43:31 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4324 rep 140.90.91.240 
Feb 12 07:43:31 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 07:43:31 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 07:43:31 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token 
Feb 12 07:43:31 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 07:43:31 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 07:43:31 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 07:43:31 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 07:43:31 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 07:43:31 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 07:43:31 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 07:43:31 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 07:43:31 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 07:43:31 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 07:43:31 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 07:43:31 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 07:43:31 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 07:43:31 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state. 
Feb 12 07:43:31 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 07:43:31 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2  
Feb 12 07:43:31 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3  
Feb 12 07:43:31 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3 4  
Feb 12 07:43:31 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 07:43:31 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 1 
Feb 12 07:43:31 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 2 
Feb 12 07:43:46 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state. 
Feb 12 07:43:46 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). 
Feb 12 07:43:46 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
Feb 12 07:43:46 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2. 
Feb 12 07:43:46 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 07:43:46 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e 
Feb 12 07:43:46 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10ec 
Feb 12 07:43:46 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state. 
Feb 12 07:43:46 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state. 
Feb 12 07:43:46 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 07:43:46 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4328 rep 140.90.91.240 
Feb 12 07:43:46 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 07:43:46 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 07:43:46 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4328 rep 140.90.91.240 
Feb 12 07:43:46 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 07:43:46 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 07:43:46 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token 
Feb 12 07:43:46 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 07:43:46 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 07:43:46 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 07:43:46 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 07:43:46 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 07:43:46 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 07:43:46 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 07:43:46 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 07:43:46 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 07:43:46 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 07:43:46 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 07:43:46 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 07:43:46 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 07:43:46 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state. 
Feb 12 07:43:46 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 07:43:46 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 07:43:46 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 1 
Feb 12 07:43:46 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 2 
Feb 12 08:43:03 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state. 
Feb 12 08:43:03 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). 
Feb 12 08:43:03 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
Feb 12 08:43:03 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2. 
Feb 12 08:43:03 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 08:43:03 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e 
Feb 12 08:43:03 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10f0 
Feb 12 08:43:03 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state. 
Feb 12 08:43:03 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state. 
Feb 12 08:43:03 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 08:43:03 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4332 rep 140.90.91.240 
Feb 12 08:43:03 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 08:43:03 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 08:43:03 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4332 rep 140.90.91.240 
Feb 12 08:43:03 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 08:43:03 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 08:43:03 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token 
Feb 12 08:43:03 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 08:43:03 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 08:43:03 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 08:43:03 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 08:43:03 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 08:43:03 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 08:43:03 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 08:43:03 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 08:43:03 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 08:43:03 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 08:43:03 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 08:43:03 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 08:43:03 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 08:43:03 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state. 
Feb 12 08:43:03 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 08:43:03 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 08:43:03 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 1 
Feb 12 08:43:03 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 2 
Feb 12 08:52:26 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state. 
Feb 12 08:52:26 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). 
Feb 12 08:52:26 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
Feb 12 08:52:26 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2. 
Feb 12 08:52:26 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 08:52:26 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e 
Feb 12 08:52:26 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10f4 
Feb 12 08:52:26 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state. 
Feb 12 08:52:26 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state. 
Feb 12 08:52:26 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 08:52:26 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4336 rep 140.90.91.240 
Feb 12 08:52:26 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 08:52:26 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 08:52:26 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4336 rep 140.90.91.240 
Feb 12 08:52:26 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 08:52:26 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 08:52:26 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token 
Feb 12 08:52:26 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 08:52:26 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 08:52:26 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 08:52:26 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 08:52:26 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 08:52:26 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 08:52:26 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 08:52:26 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 08:52:26 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 08:52:26 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 08:52:26 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 08:52:26 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 08:52:26 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 08:52:26 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state. 
Feb 12 08:52:26 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 08:52:26 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 08:52:26 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 1 
Feb 12 08:52:26 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 2 
Feb 12 13:00:28 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state. 
Feb 12 13:00:28 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). 
Feb 12 13:00:28 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
Feb 12 13:00:28 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2. 
Feb 12 13:00:28 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 13:00:28 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e 
Feb 12 13:00:28 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10f8 
Feb 12 13:00:28 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state. 
Feb 12 13:00:29 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state. 
Feb 12 13:00:29 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 13:00:29 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4340 rep 140.90.91.240 
Feb 12 13:00:29 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 13:00:29 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 13:00:29 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4340 rep 140.90.91.240 
Feb 12 13:00:29 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 13:00:29 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 13:00:29 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token 
Feb 12 13:00:29 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 13:00:29 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 13:00:29 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 13:00:29 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 13:00:29 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 13:00:29 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 13:00:29 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 13:00:29 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 13:00:29 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 13:00:29 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 13:00:29 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 13:00:29 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 13:00:29 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 13:00:29 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state. 
Feb 12 13:00:29 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 13:00:29 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 13:00:29 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 1 
Feb 12 13:00:29 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 2 
Feb 12 13:24:02 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state. 
Feb 12 13:24:02 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). 
Feb 12 13:24:02 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
Feb 12 13:24:02 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2. 
Feb 12 13:24:02 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 13:24:02 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e 
Feb 12 13:24:02 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10fc 
Feb 12 13:24:02 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state. 
Feb 12 13:24:02 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state. 
Feb 12 13:24:02 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 13:24:02 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4344 rep 140.90.91.240 
Feb 12 13:24:02 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 13:24:02 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 13:24:02 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4344 rep 140.90.91.240 
Feb 12 13:24:02 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 13:24:02 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 13:24:02 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token 
Feb 12 13:24:02 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 13:24:02 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 13:24:02 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 13:24:02 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 13:24:02 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 13:24:02 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 13:24:02 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 13:24:02 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 13:24:02 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 13:24:02 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 13:24:02 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 13:24:02 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 13:24:02 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 13:24:02 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state. 
Feb 12 13:24:02 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 13:24:02 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 13:24:02 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 1 
Feb 12 13:24:02 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 2 
Feb 12 13:31:38 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state. 
Feb 12 13:31:38 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). 
Feb 12 13:31:38 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
Feb 12 13:31:38 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2. 
Feb 12 13:31:38 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 13:31:38 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e 
Feb 12 13:31:38 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 1100 
Feb 12 13:31:38 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state. 
Feb 12 13:31:39 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state. 
Feb 12 13:31:39 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 13:31:39 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4348 rep 140.90.91.240 
Feb 12 13:31:39 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 13:31:39 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 13:31:39 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4348 rep 140.90.91.240 
Feb 12 13:31:39 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 13:31:39 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 13:31:39 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token 
Feb 12 13:31:39 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 13:31:39 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 13:31:39 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 13:31:39 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 13:31:39 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 13:31:39 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 13:31:39 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 13:31:39 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 13:31:39 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 13:31:39 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 13:31:39 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 13:31:39 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 13:31:39 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 13:31:39 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state. 
Feb 12 13:31:39 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 13:31:39 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2  
Feb 12 13:31:39 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3  
Feb 12 13:31:39 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3 4  
Feb 12 13:31:39 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3 4  
Feb 12 13:31:39 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 13:31:39 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 1 
Feb 12 13:31:39 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 2 
Feb 12 13:36:58 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state. 
Feb 12 13:36:58 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). 
Feb 12 13:36:58 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
Feb 12 13:36:58 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2. 
Feb 12 13:36:58 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 13:36:58 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e 
Feb 12 13:36:58 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 1104 
Feb 12 13:36:58 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state. 
Feb 12 13:36:58 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state. 
Feb 12 13:36:58 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 13:36:58 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4352 rep 140.90.91.240 
Feb 12 13:36:58 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 13:36:58 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 13:36:58 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4352 rep 140.90.91.240 
Feb 12 13:36:58 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 13:36:58 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 13:36:58 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token 
Feb 12 13:36:58 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 13:36:58 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 13:36:58 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 13:36:58 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 13:36:58 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 13:36:58 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 13:36:58 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 13:36:58 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 13:36:58 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 13:36:58 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 13:36:58 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 13:36:58 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 13:36:58 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 13:36:58 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state. 
Feb 12 13:36:58 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 13:36:58 nfs1-cluster openais[2604]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 13:36:58 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 1 
Feb 12 13:36:58 nfs1-cluster openais[2604]: [CPG  ] got joinlist message from node 2 
Feb 12 14:53:54 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 12. 
Feb 12 14:53:54 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 14:53:54 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e 
Feb 12 14:53:54 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 1108 
Feb 12 14:53:54 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state. 
Feb 12 14:54:04 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the COMMIT state. 
Feb 12 14:54:04 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 4. 
Feb 12 14:54:04 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 14:54:04 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 110c 
Feb 12 14:54:04 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state. 
Feb 12 14:54:04 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state. 
Feb 12 14:54:04 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 14:54:04 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4356 rep 140.90.91.240 
Feb 12 14:54:04 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 14:54:04 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 14:54:04 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4360 rep 140.90.91.242 
Feb 12 14:54:04 nfs1-cluster openais[2604]: [TOTEM] aru 32 high delivered 32 received flag 1 
Feb 12 14:54:04 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 14:54:04 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token 
Feb 12 14:54:04 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 14:54:04 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 14:54:04 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 14:54:04 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 14:54:04 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 14:54:04 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 14:54:04 nfs1-cluster openais[2604]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 14:54:04 nfs1-cluster openais[2604]: [CLM  ] New Configuration: 
Feb 12 14:54:04 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 14:54:04 nfs1-cluster openais[2604]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 14:54:04 nfs1-cluster openais[2604]: [CLM  ] Members Left: 
Feb 12 14:54:04 nfs1-cluster openais[2604]: [CLM  ] Members Joined: 
Feb 12 14:54:04 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 14:54:04 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state. 
Feb 12 14:54:04 nfs1-cluster openais[2604]: [CMAN ] cman killed by node 2 because we rejoined the cluster without a full restart 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] AIS Executive Service RELEASE 'subrev 1358 version 0.80.3' 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] Copyright (C) 2002-2006 MontaVista Software, Inc and contributors. 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] Copyright (C) 2006 Red Hat, Inc. 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] AIS Executive Service: started and ready to provide service. 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] Using default multicast address of 239.192.83.8 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] openais component openais_cpg loaded. 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] Registering service handler 'openais cluster closed process group service v1.01' 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] openais component openais_cfg loaded. 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] Registering service handler 'openais configuration service' 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] openais component openais_msg loaded. 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] Registering service handler 'openais message service B.01.01' 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] openais component openais_lck loaded. 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] Registering service handler 'openais distributed locking service B.01.01' 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] openais component openais_evt loaded. 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] Registering service handler 'openais event service B.01.01' 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] openais component openais_ckpt loaded. 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] Registering service handler 'openais checkpoint service B.01.01' 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] openais component openais_amf loaded. 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] Registering service handler 'openais availability management framework B.01.01' 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] openais component openais_clm loaded. 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] Registering service handler 'openais cluster membership service B.01.01' 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] openais component openais_evs loaded. 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] Registering service handler 'openais extended virtual synchrony service' 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] openais component openais_cman loaded. 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] Registering service handler 'openais CMAN membership service 2.01' 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] Token Timeout (10000 ms) retransmit timeout (495 ms) 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] token hold (386 ms) retransmits before loss (20 retrans) 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] join (60 ms) send_join (0 ms) consensus (4800 ms) merge (200 ms) 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] downcheck (1000 ms) fail to recv const (50 msgs) 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] seqno unchanged const (30 rotations) Maximum network MTU 1500 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] window size per rotation (50 messages) maximum messages per rotation (17 messages) 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] send threads (0 threads) 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] RRP token expired timeout (495 ms) 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] RRP token problem counter (2000 ms) 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] RRP threshold (10 problem count) 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] RRP mode set to none. 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] heartbeat_failures_allowed (0) 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] max_network_delay (50 ms) 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] HeartBeat is Disabled. To enable set heartbeat_failures_allowed > 0 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] Receive multicast socket recv buffer size (262142 bytes). 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] The network interface [140.90.91.240] is now up. 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] Created or loaded sequence id 4364.140.90.91.240 for this ring. 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] entering GATHER state from 15. 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [SERV ] Initialising service handler 'openais extended virtual synchrony service' 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [SERV ] Initialising service handler 'openais cluster membership service B.01.01' 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [SERV ] Initialising service handler 'openais availability management framework B.01.01' 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [SERV ] Initialising service handler 'openais checkpoint service B.01.01' 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [SERV ] Initialising service handler 'openais event service B.01.01' 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [SERV ] Initialising service handler 'openais distributed locking service B.01.01' 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [SERV ] Initialising service handler 'openais message service B.01.01' 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [SERV ] Initialising service handler 'openais configuration service' 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [SERV ] Initialising service handler 'openais cluster closed process group service v1.01' 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [SERV ] Initialising service handler 'openais CMAN membership service 2.01' 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CMAN ] CMAN 2.0.73 (built Nov 29 2007 18:41:45) started 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [SYNC ] Not using a virtual synchrony filter. 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] Saving state aru 0 high seq received 0 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] Storing new sequence id for ring 1110 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] entering COMMIT state. 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] entering RECOVERY state. 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] previous ring seq 4364 rep 140.90.91.240 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] aru 0 high delivered 0 received flag 1 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] Sending initial ORF token 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM  ] New Configuration: 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM  ] Members Left: 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM  ] Members Joined: 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM  ] New Configuration: 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM  ] Members Left: 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM  ] Members Joined: 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 14:57:31 nfs1-cluster openais[2603]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] entering OPERATIONAL state. 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CMAN ] quorum regained, resuming activity 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] entering GATHER state from 11. 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] Saving state aru 3 high seq received 3 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] Storing new sequence id for ring 1114 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] entering COMMIT state. 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] entering RECOVERY state. 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] previous ring seq 4368 rep 140.90.91.240 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] aru 3 high delivered 1 received flag 1 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] previous ring seq 4368 rep 140.90.91.242 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] aru 42 high delivered 42 received flag 1 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] Sending initial ORF token 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM  ] New Configuration: 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM  ] Members Left: 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM  ] Members Joined: 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM  ] New Configuration: 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM  ] Members Left: 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM  ] Members Joined: 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 14:57:31 nfs1-cluster openais[2603]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] entering OPERATIONAL state. 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CPG  ] got joinlist message from node 2 
Feb 12 14:57:32 nfs1-cluster openais[2603]: [TOTEM] Retransmit List: 1a  
Feb 12 15:58:00 nfs1-cluster openais[2603]: [TOTEM] The token was lost in the OPERATIONAL state. 
Feb 12 15:58:00 nfs1-cluster openais[2603]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). 
Feb 12 15:58:00 nfs1-cluster openais[2603]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
Feb 12 15:58:00 nfs1-cluster openais[2603]: [TOTEM] entering GATHER state from 2. 
Feb 12 15:58:00 nfs1-cluster openais[2603]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 15:58:00 nfs1-cluster openais[2603]: [TOTEM] Saving state aru fd high seq received fd 
Feb 12 15:58:00 nfs1-cluster openais[2603]: [TOTEM] Storing new sequence id for ring 1118 
Feb 12 15:58:00 nfs1-cluster openais[2603]: [TOTEM] entering COMMIT state. 
Feb 12 15:58:03 nfs1-cluster openais[2603]: [TOTEM] entering RECOVERY state. 
Feb 12 15:58:03 nfs1-cluster openais[2603]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 15:58:03 nfs1-cluster openais[2603]: [TOTEM] previous ring seq 4372 rep 140.90.91.240 
Feb 12 15:58:03 nfs1-cluster openais[2603]: [TOTEM] aru fd high delivered fd received flag 1 
Feb 12 15:58:03 nfs1-cluster openais[2603]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 15:58:03 nfs1-cluster openais[2603]: [TOTEM] previous ring seq 4372 rep 140.90.91.240 
Feb 12 15:58:03 nfs1-cluster openais[2603]: [TOTEM] aru fd high delivered fd received flag 1 
Feb 12 15:58:03 nfs1-cluster openais[2603]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 15:58:03 nfs1-cluster openais[2603]: [TOTEM] Sending initial ORF token 
Feb 12 15:58:03 nfs1-cluster openais[2603]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 15:58:03 nfs1-cluster openais[2603]: [CLM  ] New Configuration: 
Feb 12 15:58:03 nfs1-cluster openais[2603]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 15:58:03 nfs1-cluster openais[2603]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 15:58:03 nfs1-cluster openais[2603]: [CLM  ] Members Left: 
Feb 12 15:58:03 nfs1-cluster openais[2603]: [CLM  ] Members Joined: 
Feb 12 15:58:03 nfs1-cluster openais[2603]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 15:58:03 nfs1-cluster openais[2603]: [CLM  ] New Configuration: 
Feb 12 15:58:03 nfs1-cluster openais[2603]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 15:58:03 nfs1-cluster openais[2603]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 15:58:03 nfs1-cluster openais[2603]: [CLM  ] Members Left: 
Feb 12 15:58:03 nfs1-cluster openais[2603]: [CLM  ] Members Joined: 
Feb 12 15:58:03 nfs1-cluster openais[2603]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 15:58:03 nfs1-cluster openais[2603]: [TOTEM] entering OPERATIONAL state. 
Feb 12 15:58:03 nfs1-cluster openais[2603]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 15:58:03 nfs1-cluster openais[2603]: [TOTEM] Retransmit List: 2  
Feb 12 15:58:03 nfs1-cluster openais[2603]: [TOTEM] Retransmit List: 2 3  
Feb 12 15:58:03 nfs1-cluster openais[2603]: [TOTEM] Retransmit List: 2 3 4  
Feb 12 15:58:03 nfs1-cluster openais[2603]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 15:58:03 nfs1-cluster openais[2603]: [CPG  ] got joinlist message from node 1 
Feb 12 15:58:03 nfs1-cluster openais[2603]: [CPG  ] got joinlist message from node 2 
Feb 12 16:01:20 nfs1-cluster openais[2603]: [TOTEM] entering GATHER state from 12. 
Feb 12 16:01:20 nfs1-cluster openais[2603]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 16:01:20 nfs1-cluster openais[2603]: [TOTEM] Saving state aru 1e high seq received 1e 
Feb 12 16:01:20 nfs1-cluster openais[2603]: [TOTEM] Storing new sequence id for ring 111c 
Feb 12 16:01:20 nfs1-cluster openais[2603]: [TOTEM] entering COMMIT state. 
Feb 12 16:01:20 nfs1-cluster openais[2603]: [TOTEM] entering RECOVERY state. 
Feb 12 16:01:20 nfs1-cluster openais[2603]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 16:01:20 nfs1-cluster openais[2603]: [TOTEM] previous ring seq 4376 rep 140.90.91.240 
Feb 12 16:01:20 nfs1-cluster openais[2603]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 16:01:20 nfs1-cluster openais[2603]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 16:01:20 nfs1-cluster openais[2603]: [TOTEM] previous ring seq 4376 rep 140.90.91.240 
Feb 12 16:01:20 nfs1-cluster openais[2603]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 16:01:20 nfs1-cluster openais[2603]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 16:01:20 nfs1-cluster openais[2603]: [TOTEM] Sending initial ORF token 
Feb 12 16:01:20 nfs1-cluster openais[2603]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 16:01:20 nfs1-cluster openais[2603]: [CLM  ] New Configuration: 
Feb 12 16:01:20 nfs1-cluster openais[2603]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 16:01:20 nfs1-cluster openais[2603]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 16:01:20 nfs1-cluster openais[2603]: [CLM  ] Members Left: 
Feb 12 16:01:20 nfs1-cluster openais[2603]: [CLM  ] Members Joined: 
Feb 12 16:01:20 nfs1-cluster openais[2603]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 16:01:20 nfs1-cluster openais[2603]: [CLM  ] New Configuration: 
Feb 12 16:01:20 nfs1-cluster openais[2603]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 16:01:20 nfs1-cluster openais[2603]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 16:01:20 nfs1-cluster openais[2603]: [CLM  ] Members Left: 
Feb 12 16:01:20 nfs1-cluster openais[2603]: [CLM  ] Members Joined: 
Feb 12 16:01:20 nfs1-cluster openais[2603]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 16:01:20 nfs1-cluster openais[2603]: [TOTEM] entering OPERATIONAL state. 
Feb 12 16:01:20 nfs1-cluster openais[2603]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 16:01:20 nfs1-cluster openais[2603]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 16:01:20 nfs1-cluster openais[2603]: [CPG  ] got joinlist message from node 1 
Feb 12 16:01:20 nfs1-cluster openais[2603]: [CPG  ] got joinlist message from node 2 
Feb 12 16:12:26 nfs1-cluster openais[2603]: [TOTEM] The token was lost in the OPERATIONAL state. 
Feb 12 16:12:26 nfs1-cluster openais[2603]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). 
Feb 12 16:12:26 nfs1-cluster openais[2603]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
Feb 12 16:12:26 nfs1-cluster openais[2603]: [TOTEM] entering GATHER state from 2. 
Feb 12 16:12:26 nfs1-cluster openais[2603]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 16:12:26 nfs1-cluster openais[2603]: [TOTEM] Saving state aru 1e high seq received 1e 
Feb 12 16:12:26 nfs1-cluster openais[2603]: [TOTEM] Storing new sequence id for ring 1120 
Feb 12 16:12:26 nfs1-cluster openais[2603]: [TOTEM] entering COMMIT state. 
Feb 12 16:12:29 nfs1-cluster openais[2603]: [TOTEM] entering RECOVERY state. 
Feb 12 16:12:29 nfs1-cluster openais[2603]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 16:12:29 nfs1-cluster openais[2603]: [TOTEM] previous ring seq 4380 rep 140.90.91.240 
Feb 12 16:12:29 nfs1-cluster openais[2603]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 16:12:29 nfs1-cluster openais[2603]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 16:12:29 nfs1-cluster openais[2603]: [TOTEM] previous ring seq 4380 rep 140.90.91.240 
Feb 12 16:12:29 nfs1-cluster openais[2603]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 16:12:29 nfs1-cluster openais[2603]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 16:12:29 nfs1-cluster openais[2603]: [TOTEM] Sending initial ORF token 
Feb 12 16:12:29 nfs1-cluster openais[2603]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 16:12:29 nfs1-cluster openais[2603]: [CLM  ] New Configuration: 
Feb 12 16:12:29 nfs1-cluster openais[2603]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 16:12:29 nfs1-cluster openais[2603]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 16:12:29 nfs1-cluster openais[2603]: [CLM  ] Members Left: 
Feb 12 16:12:29 nfs1-cluster openais[2603]: [CLM  ] Members Joined: 
Feb 12 16:12:29 nfs1-cluster openais[2603]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 16:12:29 nfs1-cluster openais[2603]: [CLM  ] New Configuration: 
Feb 12 16:12:29 nfs1-cluster openais[2603]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 16:12:29 nfs1-cluster openais[2603]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 16:12:29 nfs1-cluster openais[2603]: [CLM  ] Members Left: 
Feb 12 16:12:29 nfs1-cluster openais[2603]: [CLM  ] Members Joined: 
Feb 12 16:12:29 nfs1-cluster openais[2603]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 16:12:29 nfs1-cluster openais[2603]: [TOTEM] entering OPERATIONAL state. 
Feb 12 16:12:29 nfs1-cluster openais[2603]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 16:12:29 nfs1-cluster openais[2603]: [TOTEM] Retransmit List: 2  
Feb 12 16:12:29 nfs1-cluster openais[2603]: [TOTEM] Retransmit List: 2 3  
Feb 12 16:12:29 nfs1-cluster openais[2603]: [TOTEM] Retransmit List: 2 3 4  
Feb 12 16:12:29 nfs1-cluster openais[2603]: [TOTEM] Retransmit List: 2 3 4  
Feb 12 16:12:29 nfs1-cluster openais[2603]: [TOTEM] Retransmit List: 2 3  
Feb 12 16:12:29 nfs1-cluster openais[2603]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 16:12:29 nfs1-cluster openais[2603]: [CPG  ] got joinlist message from node 1 
Feb 12 16:12:29 nfs1-cluster openais[2603]: [CPG  ] got joinlist message from node 2 
Feb 12 16:12:56 nfs1-cluster openais[2603]: [TOTEM] The token was lost in the OPERATIONAL state. 
Feb 12 16:12:56 nfs1-cluster openais[2603]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). 
Feb 12 16:12:56 nfs1-cluster openais[2603]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
Feb 12 16:12:56 nfs1-cluster openais[2603]: [TOTEM] entering GATHER state from 2. 
Feb 12 16:12:56 nfs1-cluster openais[2603]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 16:12:56 nfs1-cluster openais[2603]: [TOTEM] Saving state aru 1e high seq received 1e 
Feb 12 16:12:56 nfs1-cluster openais[2603]: [TOTEM] Storing new sequence id for ring 1124 
Feb 12 16:12:56 nfs1-cluster openais[2603]: [TOTEM] entering COMMIT state. 
Feb 12 16:12:59 nfs1-cluster openais[2603]: [TOTEM] entering RECOVERY state. 
Feb 12 16:12:59 nfs1-cluster openais[2603]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 16:12:59 nfs1-cluster openais[2603]: [TOTEM] previous ring seq 4384 rep 140.90.91.240 
Feb 12 16:12:59 nfs1-cluster openais[2603]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 16:12:59 nfs1-cluster openais[2603]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 16:12:59 nfs1-cluster openais[2603]: [TOTEM] previous ring seq 4384 rep 140.90.91.240 
Feb 12 16:12:59 nfs1-cluster openais[2603]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 16:12:59 nfs1-cluster openais[2603]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 16:12:59 nfs1-cluster openais[2603]: [TOTEM] Sending initial ORF token 
Feb 12 16:12:59 nfs1-cluster openais[2603]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 16:12:59 nfs1-cluster openais[2603]: [CLM  ] New Configuration: 
Feb 12 16:12:59 nfs1-cluster openais[2603]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 16:12:59 nfs1-cluster openais[2603]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 16:12:59 nfs1-cluster openais[2603]: [CLM  ] Members Left: 
Feb 12 16:12:59 nfs1-cluster openais[2603]: [CLM  ] Members Joined: 
Feb 12 16:12:59 nfs1-cluster openais[2603]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 16:12:59 nfs1-cluster openais[2603]: [CLM  ] New Configuration: 
Feb 12 16:12:59 nfs1-cluster openais[2603]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 16:12:59 nfs1-cluster openais[2603]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 16:12:59 nfs1-cluster openais[2603]: [CLM  ] Members Left: 
Feb 12 16:12:59 nfs1-cluster openais[2603]: [CLM  ] Members Joined: 
Feb 12 16:12:59 nfs1-cluster openais[2603]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 16:12:59 nfs1-cluster openais[2603]: [TOTEM] entering OPERATIONAL state. 
Feb 12 16:12:59 nfs1-cluster openais[2603]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 16:12:59 nfs1-cluster openais[2603]: [TOTEM] Retransmit List: 2  
Feb 12 16:12:59 nfs1-cluster openais[2603]: [TOTEM] Retransmit List: 2 3  
Feb 12 16:12:59 nfs1-cluster openais[2603]: [TOTEM] Retransmit List: 2 3 4  
Feb 12 16:12:59 nfs1-cluster openais[2603]: [TOTEM] Retransmit List: 3  
Feb 12 16:12:59 nfs1-cluster openais[2603]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 16:12:59 nfs1-cluster openais[2603]: [CPG  ] got joinlist message from node 1 
Feb 12 16:12:59 nfs1-cluster openais[2603]: [CPG  ] got joinlist message from node 2 
Feb 12 16:29:22 nfs1-cluster openais[2603]: [TOTEM] entering GATHER state from 12. 
Feb 12 16:29:22 nfs1-cluster openais[2603]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 16:29:22 nfs1-cluster openais[2603]: [TOTEM] Saving state aru 1e high seq received 1e 
Feb 12 16:29:22 nfs1-cluster openais[2603]: [TOTEM] Storing new sequence id for ring 1128 
Feb 12 16:29:22 nfs1-cluster openais[2603]: [TOTEM] entering COMMIT state. 
Feb 12 16:29:32 nfs1-cluster openais[2603]: [TOTEM] The token was lost in the COMMIT state. 
Feb 12 16:29:32 nfs1-cluster openais[2603]: [TOTEM] entering GATHER state from 4. 
Feb 12 16:29:32 nfs1-cluster openais[2603]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 16:29:32 nfs1-cluster openais[2603]: [TOTEM] Storing new sequence id for ring 112c 
Feb 12 16:29:32 nfs1-cluster openais[2603]: [TOTEM] entering COMMIT state. 
Feb 12 16:29:32 nfs1-cluster openais[2603]: [TOTEM] entering RECOVERY state. 
Feb 12 16:29:32 nfs1-cluster openais[2603]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 16:29:32 nfs1-cluster openais[2603]: [TOTEM] previous ring seq 4388 rep 140.90.91.240 
Feb 12 16:29:32 nfs1-cluster openais[2603]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 16:29:32 nfs1-cluster openais[2603]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 16:29:32 nfs1-cluster openais[2603]: [TOTEM] previous ring seq 4392 rep 140.90.91.242 
Feb 12 16:29:32 nfs1-cluster openais[2603]: [TOTEM] aru 1c high delivered 1c received flag 1 
Feb 12 16:29:32 nfs1-cluster openais[2603]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 16:29:32 nfs1-cluster openais[2603]: [TOTEM] Sending initial ORF token 
Feb 12 16:29:32 nfs1-cluster openais[2603]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 16:29:32 nfs1-cluster openais[2603]: [CLM  ] New Configuration: 
Feb 12 16:29:32 nfs1-cluster openais[2603]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 16:29:32 nfs1-cluster openais[2603]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 16:29:32 nfs1-cluster openais[2603]: [CLM  ] Members Left: 
Feb 12 16:29:32 nfs1-cluster openais[2603]: [CLM  ] Members Joined: 
Feb 12 16:29:32 nfs1-cluster openais[2603]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 16:29:32 nfs1-cluster openais[2603]: [CLM  ] New Configuration: 
Feb 12 16:29:32 nfs1-cluster openais[2603]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 16:29:32 nfs1-cluster openais[2603]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 16:29:32 nfs1-cluster openais[2603]: [CLM  ] Members Left: 
Feb 12 16:29:32 nfs1-cluster openais[2603]: [CLM  ] Members Joined: 
Feb 12 16:29:32 nfs1-cluster openais[2603]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 16:29:32 nfs1-cluster openais[2603]: [TOTEM] entering OPERATIONAL state. 
Feb 12 16:29:32 nfs1-cluster openais[2603]: [CMAN ] cman killed by node 2 because we rejoined the cluster without a full restart 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] AIS Executive Service RELEASE 'subrev 1358 version 0.80.3' 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] Copyright (C) 2002-2006 MontaVista Software, Inc and contributors. 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] Copyright (C) 2006 Red Hat, Inc. 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] AIS Executive Service: started and ready to provide service. 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] Using default multicast address of 239.192.83.8 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] openais component openais_cpg loaded. 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] Registering service handler 'openais cluster closed process group service v1.01' 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] openais component openais_cfg loaded. 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] Registering service handler 'openais configuration service' 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] openais component openais_msg loaded. 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] Registering service handler 'openais message service B.01.01' 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] openais component openais_lck loaded. 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] Registering service handler 'openais distributed locking service B.01.01' 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] openais component openais_evt loaded. 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] Registering service handler 'openais event service B.01.01' 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] openais component openais_ckpt loaded. 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] Registering service handler 'openais checkpoint service B.01.01' 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] openais component openais_amf loaded. 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] Registering service handler 'openais availability management framework B.01.01' 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] openais component openais_clm loaded. 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] Registering service handler 'openais cluster membership service B.01.01' 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] openais component openais_evs loaded. 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] Registering service handler 'openais extended virtual synchrony service' 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] openais component openais_cman loaded. 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] Registering service handler 'openais CMAN membership service 2.01' 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] Token Timeout (10000 ms) retransmit timeout (495 ms) 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] token hold (386 ms) retransmits before loss (20 retrans) 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] join (60 ms) send_join (0 ms) consensus (4800 ms) merge (200 ms) 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] downcheck (1000 ms) fail to recv const (50 msgs) 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] seqno unchanged const (30 rotations) Maximum network MTU 1500 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] window size per rotation (50 messages) maximum messages per rotation (17 messages) 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] send threads (0 threads) 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] RRP token expired timeout (495 ms) 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] RRP token problem counter (2000 ms) 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] RRP threshold (10 problem count) 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] RRP mode set to none. 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] heartbeat_failures_allowed (0) 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] max_network_delay (50 ms) 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] HeartBeat is Disabled. To enable set heartbeat_failures_allowed > 0 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] Receive multicast socket recv buffer size (262142 bytes). 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] The network interface [140.90.91.240] is now up. 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] Created or loaded sequence id 4396.140.90.91.240 for this ring. 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] entering GATHER state from 15. 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [SERV ] Initialising service handler 'openais extended virtual synchrony service' 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [SERV ] Initialising service handler 'openais cluster membership service B.01.01' 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [SERV ] Initialising service handler 'openais availability management framework B.01.01' 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [SERV ] Initialising service handler 'openais checkpoint service B.01.01' 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [SERV ] Initialising service handler 'openais event service B.01.01' 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [SERV ] Initialising service handler 'openais distributed locking service B.01.01' 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [SERV ] Initialising service handler 'openais message service B.01.01' 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [SERV ] Initialising service handler 'openais configuration service' 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [SERV ] Initialising service handler 'openais cluster closed process group service v1.01' 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [SERV ] Initialising service handler 'openais CMAN membership service 2.01' 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CMAN ] CMAN 2.0.73 (built Nov 29 2007 18:41:45) started 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [SYNC ] Not using a virtual synchrony filter. 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] Saving state aru 0 high seq received 0 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] Storing new sequence id for ring 1130 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] entering COMMIT state. 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] entering RECOVERY state. 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] previous ring seq 4396 rep 140.90.91.240 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] aru 0 high delivered 0 received flag 1 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] Sending initial ORF token 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM  ] New Configuration: 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM  ] Members Left: 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM  ] Members Joined: 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM  ] New Configuration: 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM  ] Members Left: 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM  ] Members Joined: 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 16:32:57 nfs1-cluster openais[2603]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] entering OPERATIONAL state. 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CMAN ] quorum regained, resuming activity 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] entering GATHER state from 11. 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] Saving state aru 3 high seq received 3 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] Storing new sequence id for ring 1134 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] entering COMMIT state. 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] entering RECOVERY state. 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] previous ring seq 4400 rep 140.90.91.240 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] aru 3 high delivered 1 received flag 1 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] previous ring seq 4400 rep 140.90.91.242 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] aru 40 high delivered 40 received flag 1 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] Sending initial ORF token 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM  ] New Configuration: 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM  ] Members Left: 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM  ] Members Joined: 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM  ] New Configuration: 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM  ] Members Left: 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM  ] Members Joined: 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 16:32:57 nfs1-cluster openais[2603]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] entering OPERATIONAL state. 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CPG  ] got joinlist message from node 2 
Feb 12 16:34:40 nfs1-cluster openais[2603]: [TOTEM] The token was lost in the OPERATIONAL state. 
Feb 12 16:34:40 nfs1-cluster openais[2603]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). 
Feb 12 16:34:40 nfs1-cluster openais[2603]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
Feb 12 16:34:40 nfs1-cluster openais[2603]: [TOTEM] entering GATHER state from 2. 
Feb 12 16:34:40 nfs1-cluster openais[2603]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 16:34:40 nfs1-cluster openais[2603]: [TOTEM] Saving state aru fe high seq received fe 
Feb 12 16:34:40 nfs1-cluster openais[2603]: [TOTEM] Storing new sequence id for ring 1138 
Feb 12 16:34:40 nfs1-cluster openais[2603]: [TOTEM] entering COMMIT state. 
Feb 12 16:34:42 nfs1-cluster openais[2603]: [TOTEM] entering RECOVERY state. 
Feb 12 16:34:42 nfs1-cluster openais[2603]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 16:34:42 nfs1-cluster openais[2603]: [TOTEM] previous ring seq 4404 rep 140.90.91.240 
Feb 12 16:34:42 nfs1-cluster openais[2603]: [TOTEM] aru fe high delivered fe received flag 1 
Feb 12 16:34:42 nfs1-cluster openais[2603]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 16:34:42 nfs1-cluster openais[2603]: [TOTEM] previous ring seq 4404 rep 140.90.91.240 
Feb 12 16:34:42 nfs1-cluster openais[2603]: [TOTEM] aru fe high delivered fe received flag 1 
Feb 12 16:34:42 nfs1-cluster openais[2603]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 16:34:42 nfs1-cluster openais[2603]: [TOTEM] Sending initial ORF token 
Feb 12 16:34:42 nfs1-cluster openais[2603]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 16:34:42 nfs1-cluster openais[2603]: [CLM  ] New Configuration: 
Feb 12 16:34:42 nfs1-cluster openais[2603]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 16:34:42 nfs1-cluster openais[2603]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 16:34:42 nfs1-cluster openais[2603]: [CLM  ] Members Left: 
Feb 12 16:34:42 nfs1-cluster openais[2603]: [CLM  ] Members Joined: 
Feb 12 16:34:42 nfs1-cluster openais[2603]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 16:34:42 nfs1-cluster openais[2603]: [CLM  ] New Configuration: 
Feb 12 16:34:42 nfs1-cluster openais[2603]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 16:34:42 nfs1-cluster openais[2603]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 16:34:42 nfs1-cluster openais[2603]: [CLM  ] Members Left: 
Feb 12 16:34:42 nfs1-cluster openais[2603]: [CLM  ] Members Joined: 
Feb 12 16:34:42 nfs1-cluster openais[2603]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 16:34:42 nfs1-cluster openais[2603]: [TOTEM] entering OPERATIONAL state. 
Feb 12 16:34:42 nfs1-cluster openais[2603]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 16:34:42 nfs1-cluster openais[2603]: [TOTEM] Retransmit List: 2  
Feb 12 16:34:42 nfs1-cluster openais[2603]: [TOTEM] Retransmit List: 2 3  
Feb 12 16:34:42 nfs1-cluster openais[2603]: [TOTEM] Retransmit List: 2 3 4  
Feb 12 16:34:42 nfs1-cluster openais[2603]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 16:34:42 nfs1-cluster openais[2603]: [CPG  ] got joinlist message from node 1 
Feb 12 16:34:42 nfs1-cluster openais[2603]: [CPG  ] got joinlist message from node 2 
Feb 12 16:40:30 nfs1-cluster openais[2603]: [TOTEM] The token was lost in the OPERATIONAL state. 
Feb 12 16:40:30 nfs1-cluster openais[2603]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). 
Feb 12 16:40:30 nfs1-cluster openais[2603]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). 
Feb 12 16:40:30 nfs1-cluster openais[2603]: [TOTEM] entering GATHER state from 2. 
Feb 12 16:40:30 nfs1-cluster openais[2603]: [TOTEM] Creating commit token because I am the rep. 
Feb 12 16:40:30 nfs1-cluster openais[2603]: [TOTEM] Saving state aru 1e high seq received 1e 
Feb 12 16:40:30 nfs1-cluster openais[2603]: [TOTEM] Storing new sequence id for ring 113c 
Feb 12 16:40:30 nfs1-cluster openais[2603]: [TOTEM] entering COMMIT state. 
Feb 12 16:40:31 nfs1-cluster openais[2603]: [TOTEM] entering RECOVERY state. 
Feb 12 16:40:31 nfs1-cluster openais[2603]: [TOTEM] position [0] member 140.90.91.240: 
Feb 12 16:40:31 nfs1-cluster openais[2603]: [TOTEM] previous ring seq 4408 rep 140.90.91.240 
Feb 12 16:40:31 nfs1-cluster openais[2603]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 16:40:31 nfs1-cluster openais[2603]: [TOTEM] position [1] member 140.90.91.242: 
Feb 12 16:40:31 nfs1-cluster openais[2603]: [TOTEM] previous ring seq 4408 rep 140.90.91.240 
Feb 12 16:40:31 nfs1-cluster openais[2603]: [TOTEM] aru 1e high delivered 1e received flag 1 
Feb 12 16:40:31 nfs1-cluster openais[2603]: [TOTEM] Did not need to originate any messages in recovery. 
Feb 12 16:40:31 nfs1-cluster openais[2603]: [TOTEM] Sending initial ORF token 
Feb 12 16:40:31 nfs1-cluster openais[2603]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 16:40:31 nfs1-cluster openais[2603]: [CLM  ] New Configuration: 
Feb 12 16:40:31 nfs1-cluster openais[2603]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 16:40:31 nfs1-cluster openais[2603]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 16:40:31 nfs1-cluster openais[2603]: [CLM  ] Members Left: 
Feb 12 16:40:31 nfs1-cluster openais[2603]: [CLM  ] Members Joined: 
Feb 12 16:40:31 nfs1-cluster openais[2603]: [CLM  ] CLM CONFIGURATION CHANGE 
Feb 12 16:40:31 nfs1-cluster openais[2603]: [CLM  ] New Configuration: 
Feb 12 16:40:31 nfs1-cluster openais[2603]: [CLM  ] 	r(0) ip(140.90.91.240)  
Feb 12 16:40:31 nfs1-cluster openais[2603]: [CLM  ] 	r(0) ip(140.90.91.242)  
Feb 12 16:40:31 nfs1-cluster openais[2603]: [CLM  ] Members Left: 
Feb 12 16:40:31 nfs1-cluster openais[2603]: [CLM  ] Members Joined: 
Feb 12 16:40:31 nfs1-cluster openais[2603]: [SYNC ] This node is within the primary component and will provide service. 
Feb 12 16:40:31 nfs1-cluster openais[2603]: [TOTEM] entering OPERATIONAL state. 
Feb 12 16:40:31 nfs1-cluster openais[2603]: [CLM  ] got nodejoin message 140.90.91.240 
Feb 12 16:40:31 nfs1-cluster openais[2603]: [TOTEM] Retransmit List: 2  
Feb 12 16:40:31 nfs1-cluster openais[2603]: [TOTEM] Retransmit List: 2 3  
Feb 12 16:40:31 nfs1-cluster openais[2603]: [TOTEM] Retransmit List: 2 3 4  
Feb 12 16:40:31 nfs1-cluster openais[2603]: [CLM  ] got nodejoin message 140.90.91.242 
Feb 12 16:40:31 nfs1-cluster openais[2603]: [CPG  ] got joinlist message from node 1 
Feb 12 16:40:31 nfs1-cluster openais[2603]: [CPG  ] got joinlist message from node 2 
<?xml version="1.0"?>
<cluster alias="ohd_cluster" config_version="154" name="ohd_cluster">
	<fence_daemon post_fail_delay="0" post_join_delay="60"/>
	<clusternodes>
		<clusternode name="nfs1-cluster.nws.noaa.gov" nodeid="1" votes="1">
			<fence>
				<method name="1">
					<device name="nfspower" port="8" switch="1"/>
				</method>
			</fence>
		</clusternode>
		<clusternode name="nfs2-cluster.nws.noaa.gov" nodeid="2" votes="1">
			<fence>
				<method name="1">
					<device name="nfspower" port="7" switch="1"/>
				</method>
			</fence>
		</clusternode>
	</clusternodes>
	<cman expected_votes="1" two_node="1"/>
	<rm>
		<failoverdomains>
			<failoverdomain name="nfs-failover" ordered="0" restricted="1">
				<failoverdomainnode name="nfs1-cluster.nws.noaa.gov" priority="1"/>
				<failoverdomainnode name="nfs2-cluster.nws.noaa.gov" priority="1"/>
			</failoverdomain>
		</failoverdomains>
		<resources>
			<ip address="140.90.91.244" monitor_link="1"/>
			<clusterfs device="/dev/VolGroupFS/LogVol-shared" force_unmount="0" fsid="30647" fstype="gfs" mountpoint="/fs/shared" name="fs-shared" options="rw"/>
			<nfsexport name="fs-shared-exp"/>
			<nfsclient name="fs-shared-client" options="no_root_squash,rw" path="/fs/shared" target="140.90.91.0/24"/>
			<ip address="140.90.91.245" monitor_link="1"/>
			<clusterfs device="/dev/VolGroupFS/LogVol-brianj" force_unmount="0" fsid="39025" fstype="gfs" mountpoint="/fs/rfcdata" name="fs-rfcdata" options="rw"/>
			<nfsexport name="fs-rfcdata-exp"/>
			<nfsclient name="fs-rfcdata-client" options="no_root_squash,rw" path="/fs/rfcdata" target="140.90.91.0/24"/>
			<clusterfs device="/dev/VolGroupFS/LogVol-hsmb2" force_unmount="0" fsid="39203" fstype="gfs" mountpoint="/fs/hsmb2" name="fs-hsmb2" options="rw"/>
			<clusterfs device="/dev/VolGroupFS/LogVol-hsmb5" force_unmount="0" fsid="3200" fstype="gfs" mountpoint="/fs/hsmb5" name="fs-hsmb5" options="rw"/>
			<clusterfs device="/dev/VolGroupFS/LogVol-nobk-hsmb3" force_unmount="0" fsid="47183" fstype="gfs" mountpoint="/nobk/hsmb3" name="nobk-hsmb3" options="rw"/>
			<clusterfs device="/dev/VolGroupFS/LogVol-nobk-hsmb4" force_unmount="0" fsid="11756" fstype="gfs" mountpoint="/nobk/hsmb4" name="nobk-hsmb4" options="rw"/>
			<nfsexport name="fs-hsmb2-exp"/>
			<nfsexport name="fs-hsmb5-exp"/>
			<nfsexport name="nobk-hsmb3-exp"/>
			<nfsexport name="nobk-hsmb4-exp"/>
			<nfsclient name="fs-hsmb2-client" options="no_root_squash,rw" path="/fs/hsmb2" target="140.90.91.0/24"/>
			<nfsclient name="fs-hsmb5-client" options="no_root_squash,rw" path="/fs/hsmb5" target="140.90.91.0/24"/>
			<nfsclient name="nobk-hsmb3-client" options="no_root_squash,rw" path="/nobk/hsmb3" target="140.90.91.0/24"/>
			<nfsclient name="nobk-hsmb4-client" options="no_root_squash,rw" path="/nobk/hsmb4" target="140.90.91.0/24"/>
			<clusterfs device="/dev/VolGroupFS/LogVol-ensembles" force_unmount="0" fsid="36988" fstype="gfs" mountpoint="/fs/ensembles" name="fs-ensembles" options="rw"/>
			<nfsexport name="fs-ensembles-exp"/>
			<nfsclient name="fs-ensembles-client" options="no_root_squash,rw" path="/fs/ensembles" target="140.90.91.0/24"/>
			<clusterfs device="/dev/VolGroupFS/LogVol-hseb" force_unmount="0" fsid="41992" fstype="gfs" mountpoint="/fs/hseb" name="fs-hseb" options="rw"/>
			<nfsexport name="fs-hseb-exp"/>
			<nfsclient name="fs-hseb-client" options="no_root_squash,rw" path="/fs/hseb" target="140.90.91.0/24"/>
		</resources>
		<service autostart="1" domain="nfs-failover" name="nfs">
			<clusterfs ref="fs-shared">
				<nfsexport ref="fs-shared-exp">
					<nfsclient ref="fs-shared-client"/>
				</nfsexport>
			</clusterfs>
			<ip ref="140.90.91.244"/>
			<clusterfs ref="fs-rfcdata">
				<nfsexport ref="fs-rfcdata-exp">
					<nfsclient ref="fs-rfcdata-client"/>
				</nfsexport>
				<ip ref="140.90.91.244"/>
			</clusterfs>
			<clusterfs ref="fs-hsmb2">
				<nfsexport ref="fs-hsmb2-exp">
					<nfsclient ref="fs-hsmb2-client"/>
				</nfsexport>
				<ip ref="140.90.91.244"/>
			</clusterfs>
			<clusterfs ref="fs-hsmb5">
				<nfsexport ref="fs-hsmb5-exp">
					<nfsclient ref="fs-hsmb5-client"/>
				</nfsexport>
				<ip ref="140.90.91.244"/>
			</clusterfs>
			<clusterfs ref="nobk-hsmb3">
				<nfsexport ref="nobk-hsmb3-exp">
					<nfsclient ref="nobk-hsmb3-client"/>
				</nfsexport>
				<ip ref="140.90.91.244"/>
			</clusterfs>
			<clusterfs ref="nobk-hsmb4">
				<nfsexport ref="nobk-hsmb4-exp">
					<nfsclient ref="nobk-hsmb4-client"/>
				</nfsexport>
				<ip ref="140.90.91.244"/>
			</clusterfs>
			<clusterfs ref="fs-ensembles">
				<nfsexport ref="fs-ensembles-exp">
					<nfsclient ref="fs-ensembles-client"/>
				</nfsexport>
			</clusterfs>
			<ip ref="140.90.91.244"/>
			<clusterfs ref="fs-hseb">
				<nfsexport ref="fs-hseb-exp">
					<nfsclient ref="fs-hseb-client"/>
				</nfsexport>
				<ip ref="140.90.91.244"/>
			</clusterfs>
		</service>
	</rm>
	<fencedevices>
		<fencedevice agent="fence_apc" ipaddr="192.168.42.30" login="cluster" name="nfspower" passwd="Tele4m32"/>
	</fencedevices>
</cluster>
begin:vcard
fn:Randy Brown
n:Brown;Randy
org:National Weather Service;Office of Hydrologic Development
adr:;;1325 East West Highway;Silver Spring;MD;20910;USA
email;internet:randy brown noaa gov
title:Senior Systems Administrator
tel;work:301-713-1669 x110
url:http://www.nws.noaa.gov/ohd/
version:2.1
end:vcard


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]