[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

[Linux-cluster] 3rd node won't rejoin cluster



I’ve just run into a odd problem on my production cluster.  One of the nodes got fenced (still digging through logs to find out why) and on it’s way back up, it appears to join the cluster find but the node that fenced it starts spewing out tons of these in /var/log/messages:

 

Sep 10 14:25:34 redwing gfs_controld[6119]: cpg_mcast_joined retry 176200 unknown

Sep 10 14:25:35 redwing gfs_controld[6119]: cpg_mcast_joined retry 176300 unknown

Sep 10 14:25:35 redwing gfs_controld[6119]: cpg_mcast_joined retry 176400 unknown

Sep 10 14:25:35 redwing gfs_controld[6119]: cpg_mcast_joined retry 176500 unknown

Sep 10 14:25:35 redwing gfs_controld[6119]: cpg_mcast_joined retry 176600 unknown

Sep 10 14:25:35 redwing gfs_controld[6119]: cpg_mcast_joined retry 176700 unknown

….

 

The node that got fenced just hangs at the “Starting Fencing…” part of cman, while redwing (the node that fenced it) starts to climb in load slowly but surely.  I ended up bringing down the fenced node and I’m running fine off the 2 remaining nodes.  Has anyone ran into this problem.

 

I’m running RHEL5.3 with these packages:

 

[a_arwin redwing ~]$ rpm -qa | egrep 'cman|rgman|gfs|lvm'

lvm2-2.02.40-6.el5

kmod-gfs-0.1.31-3.el5

cman-2.0.98-1.el5_3.1

gfs-utils-0.1.18-1.el5

rgmanager-2.0.46-1.el5_3.3

gfs2-utils-0.1.53-1.el5_3.2

lvm2-cluster-2.02.40-7.el5

[a_arwin redwing ~]$ uname -a

Linux redwing.csun.edu 2.6.18-128.1.6.el5 #1 SMP Tue Mar 24 12:05:57 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux

 

Thanks ahead of time,

Arwin


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]