[Linux-cluster] GFS Performance Problems (RHEL5)

[Paul Risenhoover]

Hi All,

I am experiencing some substantial performance problems on my RHEL 5 
server running GFS.  The specific symptom that I'm seeing is that the 
file system will hang for anywhere from 5 to 45 seconds on occasion.  
When this happens it stalls all processes that are attempting to access 
the file system (ie, "ls -l") such that even a ctrl-break can't stop it.

It also appears that gfs_scand is working extremely hard.  It runs at 
7-10% CPU almost constantly.  I did some research on this and discovered 
a discussion about cluster locking in relation to directories with large 
numbers of files, and believe it might be related.  I've got some 
directories with 5000+ files.  However, I get the stalling behavior even 
when nothing is accessing those particular directories.

I also tried some tuning some of the parameters:

gfs_tool settune /mnt/promise demote_secs 10
gfs_tool settune /mnt/promise scand_secs 2
gfs_tool settune /mnt/promise/ reclaim_limit 1000

But this doesn't appear to have done much.    Does anybody have some 
thoughts on how I might resolve this?

Paul