head node has an extremely high load average.

[Jonathan Billings]

I saw a 'gmetad' running on your head node, which means you're running
ganglia.  You can probably monitor your NFS traffic client-side on each
node and have ganglia collect that information.

Just keep in mind, your CPU load doesn't look that unusual for a NFS server
with many nodes running jobs writing to it.  You said you had 10 nfsd
threads, and your top output above said there were 3 tasks running on a
2-core system, with a load average between 13 and 14.  Sounds about right.
Don't focus on the load average, you'd be better off trying to improve
storage and network speed.

For example, I suggest installing the 'tuned' package and enable the
'enterprise-storage' profile (with tuned-adm).  That will change the
scheduler elevator algorithm as well as a bunch of sysctl items to
something optimized for storage.


On Thu, Jun 27, 2013 at 9:48 AM, Doll, Margaret Ann <margaret_doll at brown.edu
> wrote:

> Thanks.  This sounds like what I need.
>
> On Thu, Jun 27, 2013 at 9:44 AM, Miner, Jonathan W (US SSA) <
> jonathan.w.miner at baesystems.com> wrote:
>
> >
> > > How does one determine from which node the massive io requests are
> > coming?
> >
> > This falls under network monitoring.  You could use a program like
> > "iptraf"; if you run it on the server you should be able to get it to
> > display bytes per client, etc.
> >
> > Or if your switches have snmp monitoring, you could use mrtg to graph the
> > per port usage.  Assuming that each port maps to a single device, then
> > you'd be able to map high usage back to the client machine.  You could
> also
> > use a snmp monitoring program (Nagios is one...)  to actively alert you
> if
> > usage was above a certain threshold.
> >
> > https://fedorahosted.org/iptraf-ng/
> > http://oss.oetiker.ch/mrtg/
> > http://www.nagios.org/
> >
> > - Jon
> >
> > --
> > redhat-list mailing list
> > unsubscribe mailto:redhat-list-request at redhat.com?subject=unsubscribe
> > https://www.redhat.com/mailman/listinfo/redhat-list
> >
> --
> redhat-list mailing list
> unsubscribe mailto:redhat-list-request at redhat.com?subject=unsubscribe
> https://www.redhat.com/mailman/listinfo/redhat-list
>



-- 
Jonathan Billings <jsbillin at umich.edu>
College of Engineering - CAEN - Unix and Linux Support