[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

[Linux-cluster] shutdown (OS) of all GFS nodes



hi,

I have a GFS cluster (4 node + qdisk on SAN), and have problems shutting down cman service / unmount gfs mountpoints - it causes the shutdown to hang.  I am running GFS & CLVM (lv's are xen guest drives). If i try and shut down cman service manually, i get an error that resources are still in use.   1 gfs directory is exported via NFS.

I think it may be because of service stop order, specifically openais stopping before cman - could this be a valid reason?

Init6 levels are:
K00xendomains
K01xend
K03libvirtd
K20nfs
K20openais
K74gfs
K74gfs2
K76clvmd
K78qdiskd
K79cman
K86nfslock


If I manually run through the stopping of these services, gfs service hangs. This is the log:
Feb 13 10:35:25 vmhost-01 gfs_controld[3227]: cluster is down, exiting
Feb 13 10:35:25 vmhost-01 dlm_controld[3221]: cluster is down, exiting
Feb 13 10:35:25 vmhost-01 fenced[3215]: cluster is down, exiting
Feb 13 10:35:25 vmhost-01 kernel: dlm: closing connection to node 4
Feb 13 10:35:25 vmhost-01 kernel: dlm: closing connection to node 3
Feb 13 10:35:25 vmhost-01 kernel: dlm: closing connection to node 2
Feb 13 10:35:25 vmhost-01 kernel: dlm: closing connection to node 1
Feb 13 10:35:27 vmhost-01 qdiskd[3201]: <err> cman_dispatch: Host is down
Feb 13 10:35:27 vmhost-01 qdiskd[3201]: <err> Halting qdisk operations
Feb 13 10:35:51 vmhost-01 ccsd[3165]: Unable to connect to cluster infrastructure after 30 seconds.
Feb 13 10:36:13 vmhost-01 mountd[3927]: Caught signal 15, un-registering and exiting.
Feb 13 10:36:13 vmhost-01 kernel: nfsd: last server has exited
Feb 13 10:36:13 vmhost-01 kernel: nfsd: unexporting all filesystems
Feb 13 10:36:21 vmhost-01 ccsd[3165]: Unable to connect to cluster infrastructure after 60 seconds.

ccsd continues to repeat the last message, increasing time: 60s, 90s, 120s, 180s, 210s, etc

dmesg shows:
dlm: closing connection to node 4
dlm: closing connection to node 3
dlm: closing connection to node 2
dlm: closing connection to node 1

There are no open  files on GFS (from lsof)

I am using gfs (1).

The only workaround I have now is to reset the nodes via ILO once the shutdown process starts (and hangs on either gfs or cman service stop).



[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]