Re: Very high load spikes

[Dag Wieers <dag wieers com>]

On Fri, 3 Dec 2004, Mimmus wrote:

> I'm experiencing very high (even 40-50!) load peaks, during only a few of
> seconds.
> During these peaks, user cpu is usually low but system cpu is ~80-99%.
> System is a DL740 with 8 CPUs and 32 GB of RAM with Linux Red Hat Advanced
> Server 3.0 (kernel 2.4.21-9.0.3.ELhugemem), connected by a 2GBps SAN to an
> EVA3000 disk-array.
> It is an Oracle-only machine (but 9 istances are active), no other
> significative processes are running.
> I tuned it using usual parameters from Oracle (eventually I can attach a
> kernel parameters list).
> It seems that there is no exceptional I/O or paging/swap activity during peaks
> but I'm not sure (I'm using dstat - http://dag.wieers.com/home-made/dstat/ -
> to monitor several metrics at same time).
> 
> How can I identify what system does when load is high?

I just recently added softirq and hardirq values, so you may want to take 
the latest dstat from subversion to see what happens.

Also I would enable proc stats (-p) to see if any new processes are 
started at the same time, maybe you have something running from cron 
(sysstat ?) and that's causing a sudden surge of processes.

If the hardirq and softirq values peak during this load peak, you could 
enable the int stats (-i) together with -f to see if any hardware could be 
causing these spikes. I'm filtering out some interrupts when using -f, but 
you can override that behaviour using -I.

Let me know if you find what is causing it and how you were able to track 
it. It would be useful to know if we can improve something.

Kind regards,
--   dag wieers,  dag wieers com,  http://dag.wieers.com/   --
[all I want is a warm bed and a kind word and unlimited power]