[Linux-cluster] Hight I/O Wait Rates - RHEL 6.1 + GFS2 + NFS

[Fabio M. Di Nitto]

On 6/28/2011 7:55 AM, anderson souza wrote:
> Hi everyone,
>  
> I have an Active/Passive RHCS 6.1 runing with 8TB of GFS2 with NFS on
> top and exporting 26 mouting points to 250 NFS clients. The GFS2
> mounting points are mounted with noatime, nodiratime, data=writeback and
> localflocks options, and also the SAN and servers are fast (4Gbps and
> 8Gb, dual controllers working in LB, H.A... QuadCore, 48GB of
> memory...). The cluster has been doing its work (failover working
> fine...), however and unfortunately I have seen hight I/Owait rates,
> sometimes around 60-70% (on which is very bad), and a couple
> of glock_workqueue jobs, so I get a bunch of gfs2_quotad, nfsd errors
> and qdisk latency. The debugfs didn't show me "W", only "G" and "H".
>  
> Have you guys seen it before?
> Looks like some glock's contention?
> How could I get it fixed and what does it mean?

Please contact GSS and file a ticket.

You are probably experiencing this:
https://bugzilla.redhat.com/show_bug.cgi?id=717010

(you might not be able to see the whole content directly, but try
downgrading the kernel to 6.0 should make things better)

Also, given the nature of your setup, I would recommend to request a
cluster architecture review to GSS for GFS2 usage in such environment.

Fabio