[Linux-cluster] cman_init error (nil) 111

[Chen Ming]

I install a new two nodes cluster, one node can start successfully. 
while the other node can not start cman.
There is the logs. Any comment is appreciated.

Apr 23 12:37:22 notice AS-2 openais[8307]: [SERV ] Initialising service 
handler \'openais message service B.01.01\'
Apr 23 12:37:22 notice AS-2 openais[8307]: [SERV ] Initialising service 
handler \'openais configuration service\'
Apr 23 12:37:22 notice AS-2 openais[8307]: [SERV ] Initialising service 
handler \'openais cluster closed process group service
 v1.01\'
Apr 23 12:37:22 notice AS-2 openais[8307]: [SERV ] Initialising service 
handler \'openais CMAN membership service 2.01\'
Apr 23 12:37:22 info AS-2 openais[8307]: [CMAN ] CMAN 2.0.98 (built Dec  
3 2008 16:32:34) started
Apr 23 12:37:22 notice AS-2 openais[8307]: [SYNC ] Not using a virtual 
synchrony filter.
Apr 23 12:37:22 notice AS-2 openais[8307]: [TOTEM] Creating commit token 
because I am the rep.
Apr 23 12:37:22 notice AS-2 openais[8307]: [TOTEM] Saving state aru 0 
high seq received 0
Apr 23 12:37:22 notice AS-2 openais[8307]: [TOTEM] Storing new sequence 
id for ring 38
Apr 23 12:37:22 notice AS-2 openais[8307]: [TOTEM] entering COMMIT state.
Apr 23 12:37:22 notice AS-2 openais[8307]: [TOTEM] entering RECOVERY state.
Apr 23 12:37:22 notice AS-2 openais[8307]: [TOTEM] position [0] member 
10.56.150.139:
Apr 23 12:37:22 notice AS-2 openais[8307]: [TOTEM] previous ring seq 52 
rep 10.56.150.139
Apr 23 12:37:22 notice AS-2 openais[8307]: [TOTEM] aru 0 high delivered 
0 received flag 1
Apr 23 12:37:22 notice AS-2 openais[8307]: [TOTEM] Did not need to 
originate any messages in recovery.
Apr 23 12:37:22 notice AS-2 openais[8307]: [TOTEM] Sending initial ORF token
Apr 23 12:37:22 notice AS-2 openais[8307]: [CLM  ] CLM CONFIGURATION CHANGE
Apr 23 12:37:22 notice AS-2 openais[8307]: [CLM  ] New Configuration:
Apr 23 12:37:22 notice AS-2 openais[8307]: [CLM  ] Members Left:
Apr 23 12:37:22 notice AS-2 openais[8307]: [CLM  ] Members Joined:
Apr 23 12:37:22 notice AS-2 openais[8307]: [CLM  ] CLM CONFIGURATION CHANGE
Apr 23 12:37:22 notice AS-2 openais[8307]: [CLM  ] New Configuration:
Apr 23 12:37:22 notice AS-2 openais[8307]: [CLM  ]      r(0) 
ip(10.56.150.139)
Apr 23 12:37:22 notice AS-2 openais[8307]: [CLM  ] Members Left:
Apr 23 12:37:22 notice AS-2 openais[8307]: [CLM  ] Members Joined:
Apr 23 12:37:22 notice AS-2 openais[8307]: [CLM  ]      r(0) 
ip(10.56.150.139)
Apr 23 12:37:22 notice AS-2 openais[8307]: [SYNC ] This node is within 
the primary component and will provide service.
Apr 23 12:37:22 notice AS-2 openais[8307]: [TOTEM] entering OPERATIONAL 
state.
Apr 23 12:37:22 info AS-2 openais[8307]: [CMAN ] quorum regained, 
resuming activity
Apr 23 12:37:22 notice AS-2 openais[8307]: [CLM  ] got nodejoin message 
10.56.150.139
Apr 23 12:37:22 err AS-2 syslog-ng[7516]: Error connecting to remote 
host AF_INET(logserver:601), reattempting in 5 seconds
Apr 23 12:37:22 err AS-2 groupd[8319]: found uncontrolled kernel object 
rgmanager in /sys/kernel/dlm
Apr 23 12:37:22 err AS-2 groupd[8319]: local node must be reset to clear 
1 uncontrolled instances of gfs and/or dlm
Apr 23 12:37:22 info AS-2 openais[8307]: [CMAN ] cman killed by node 2 
because we were killed by cman_tool or other applicatio
n
Apr 23 12:37:22 err AS-2 fenced[8327]: cman_init error (nil) 111
Apr 23 12:37:22 err AS-2 fence_node[8320]: Fence of \"AS-2\" was 
unsuccessful
Apr 23 12:37:22 err AS-2 gfs_controld[8339]: cman_init error 111
Apr 23 12:37:30 err AS-2 syslog-ng[7516]: Error connecting to remote 
host AF_INET(logserver:601), reattempting in 5 seconds
Apr 23 12:37:32 err AS-2 dlm_controld[8333]: group_init error (nil) 111