[Linux-cluster] GFS2 subdirectory hang

[Steven Whitehouse]

Hi,

On Thu, 2009-08-27 at 09:25 -0500, Johnson, Eric wrote:
> I have a 32-bit RHEL 5.3 Cluster Suite setup of two nodes with GFS2 file
> systems on FC attached SAN. I have run into this issue twice now, where
> attempts to access a certain directory within one of the GFS2 file
> systems never return. Other directories and paths within that file
> system work just fine.
> 
> The first time it happened, I had to crash the node to get it to release
> the FS, then unmount it on both nodes, fsck it, remount it, and it was
> fine. It has happened again (different path, different file system). A
> simple "ls" in the directory (which has maybe 20 files in it) leaves the
> process in an uninterruptible sleep state. I left it all night and it
> never returned.
> 
> I'm not sure what other info would be useful on this, but this is what I
> see from a gfs2_tool lockdump output for ls PID on that node:
> 
> G:  s:UN n:2/bf1df f:l t:SH d:EX/0 l:0 a:0 r:4
>  H: s:SH f:aW e:0 p:9938 [ls] gfs2_lookup+0x44/0x90 [gfs2]
              ^ The W flag indicates that this is waiting for a glock

Currently the glock is in the UN (unlocked) state, and its trying to get
a SH (shared) lock. The next step in the investigation is to look for
the same glock number 2/bf1df on the other nodes, and see what is
holding that lock. This particular node will hang until the lock is
released on whichever other node is holding it.

If there is nothing on any other node apparently holding that lock in
the glock dumps, then looking at dlm lock dumps would be the next step,

Steve.


> G:  s:SH n:5/bf9ce f: t:SH d:EX/0 l:0 a:0 r:3
>  H: s:SH f:EH e:0 p:9938 [ls] gfs2_inode_lookup+0x12d/0x1f0 [gfs2]
> G:  s:SH n:5/bf1df f: t:SH d:EX/0 l:0 a:0 r:3
>  H: s:SH f:EH e:0 p:9938 [ls] gfs2_inode_lookup+0x12d/0x1f0 [gfs2]
> G:  s:SH n:5/20c9b f: t:SH d:EX/0 l:0 a:0 r:3
>  H: s:SH f:EH e:0 p:9938 [ls] gfs2_inode_lookup+0x12d/0x1f0 [gfs2]
> 
> I know there have been a few updates that have been released that
> address a couple of GFS2 issues, but I don't recall seeing one for this
> type of problem. I've got kernel 2.6.18-128.1.14el5PAE and
> gfs2-utils-0.1.53-1.el5_3.3.
> 
> Thanks,
> Eric
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster