[Linux-cluster] GFS Mount Problem

[Leo Pleiman]

> On Thu, Feb 05, 2009 at 09:29:51AM -0500, Leo Pleiman wrote:
> > I have a 7 node virtualized RHEL5.3 cluster. All nodes mount
> mulitple gfs
> > filesystems. One node only only of the file systems won't mount. I
> know
> > (from a previous incident) that rebooting the node will clear the
> problem.
> > Below is some information, I'd appreciate any help to resolve this
> without
> > rebooting the node.
> > 
> > [root at rdccluster2-5 ~]# umount /decennial/
> > umount: /decennial/: not mounted
> > [root at rdccluster2-5 ~]# mount /decennial
> > /sbin/mount.gfs: fs is being unmounted
> > /sbin/mount.gfs: error mounting lockproto lock_dlm
> 
> Seems something went wrong with an unmount; gfs_controld thinks the
> unmount is
> still in progress.  Does 'ps' show any unmount still running?  Does
> /proc/mounts show the fs still mounted in the kernel?  What does
> group_tool -v
> show about the fs's?  If the kernel portion of the unmount is done,
> and you can
> unmount any other gfs fs's, then you should be able to kill
> gfs_controld, stop
> the cluster on this node and restart it.
> 
> Dave

The other gfs fs are in use by the customer, the idea is to restore the mount without interruption, hence I didn't want to reboot the node.

/proc/mounts shows the fs is unmounted
group_tool -v shows
type             level name              id       state node id local_done
fence            0     default           00010007 none        
[1 2 3 4 5 6 7]
dlm              1     clvmd             00010006 none        
[1 2 3 4 5 6 7]
dlm              1     apps1             00100005 none        
[1 2 3 4 5 6 7]
dlm              1     Home              00120005 none        
[1 2 3 4 5 6 7]
dlm              1     Demographic       00140005 none        
[1 2 3 4 5 6 7]
dlm              1     Economic          00160005 none        
[1 2 3 4 5 6 7]
dlm              1     mixed             00180005 none        
[1 2 3 4 5 6 7]
dlm              1     Rdcstaff          00190005 none        
[1 2 3 4 5 6 7]
dlm              1     Lbddmz            001b0005 none        
[1 2 3 4 5 6 7]
dlm              1     washdc_projects1  001c0005 none        
[1 2 3 4 5 6 7]
dlm              1     Baruch            001e0005 none        
[1 2 3 4 5 6 7]
dlm              1     Berkley           00200005 none        
[1 2 3 4 5 6 7]
dlm              1     Boston            00220005 none        
[1 2 3 4 5 6 7]
dlm              1     Chicago           00230005 none        
[1 3 4 5 6 7]
dlm              1     Cornell           00130002 none        
[1 2 3 4 5 6 7]
dlm              1     mich_projects     00150002 none        
[1 2 3 4 5 6 7]
dlm              1     tri_projects      00170002 none        
[1 2 3 4 5 6 7]
dlm              1     rdcfed1           00240005 none        
[1 2 3 4 5 7]
dlm              1     rgmanager         00190002 none        
[1 2 3 4 5 6 7]
gfs              2     Decennial         00020005 none        
[1 2 3 4 5 6 7]
gfs              2     public            00060005 none        
[1 2 3 4 5 6 7]
gfs              2     ucla_projects1    000e0005 none        
[1 2 3 4 5 6 7]
gfs              2     apps1             000f0005 none        
[1 2 3 4 5 6 7]
gfs              2     Home              00110005 none        
[1 2 3 4 5 6 7]
gfs              2     Demographic       00130005 none        
[1 2 3 4 5 6 7]
gfs              2     Economic          00150005 none        
[1 2 3 4 5 6 7]
gfs              2     mixed             00170005 none        
[1 2 3 4 5 6 7]
gfs              2     Rdcstaff          00070001 none        
[1 2 3 4 5 6 7]
gfs              2     Lbddmz            001a0005 none        
[1 2 3 4 5 6 7]
gfs              2     washdc_projects1  00090001 none        
[1 2 3 4 5 6 7]
gfs              2     Baruch            001d0005 none        
[1 2 3 4 5 6 7]
gfs              2     Berkley           001f0005 none        
[1 2 3 4 5 6 7]
gfs              2     Boston            00210005 none        
[1 2 3 4 5 6 7]
gfs              2     Chicago           000d0002 none        
[1 2 3 4 5 6 7]
gfs              2     Cornell           00120002 none        
[1 2 3 4 5 6 7]
gfs              2     mich_projects     00140002 none        
[1 2 3 4 5 6 7]
gfs              2     tri_projects      00160002 none        
[1 2 3 4 5 6 7]
gfs              2     rdcfed1           00140006 none        
[1 2 3 4 5 6 7]

--Leo