[Linux-cluster] GFS mounting fails when rejoining the cluster

[gordan at bobich.net]

Hi,

I seem to have a weird problem. I have a 5 node (actually 4 with 
room to put another node in) cluster, if all the nodes 
come up at the same time, everything works fine. If I reboot one of the 
nodes, it doesn't re-connect the GFS share. It says:

Trying to join cluster "lock_dlm" "mycluster:myshare"

dlm: connecting to 1
dlm: connecting to 2
dlm: connection to 3

Joined cluster. Now mounting FS...

dlm: Got connection from 3

and then it just sits there.

The other nodes also lose access to the shared file system. How do I 
troubleshoot this? Everything works OK when the nodes all come up at the 
same time, but the re-joining seems to break the whole cluster.

I'm using CentOS 5 with the latest updates.

Gordan