Re: "Out of Memory: Killed process" errors on server running Oracle or VMware

[Eric Sisler <esisler westminster lib co us>]

Hi Tom,

> Also, don't ignore the lower_zone_protection settings.  We've found
> those to help tremendously in OOM situations involving normal zone
> starvation.

Wow!  Setting lower_zone_protection = 250 made a *HUGE* difference, even
with the regular 32-bit kernel.  Do you have a good source of
documentation on /proc settings?  I can always Google, but if you have a
good one could you pass it along?  Thanks!

Some preliminary test results, I'm still planning on trying the hugemem
& 64-bit kernels:

Test environment:
  Physical server w/8b RAM, 32-bit RHEL 4.5 (RAM LowTotal = 837724Kb)
  5 virtual machines configured w/2Gb RAM each
  1 virtual machine configured w/512Mb

This potentially overloads the server's physical RAM by ~2.5Gb,
especially when starting all VMs simultaneously.

lower_zone_protection = 0  : lowest recorded LowFree was  13,480Kb
lower_zone_protection = 250: lowest recorded LowFree was 268,408Kb

With the setting at 0, there were times when the load average was high
enough to impact the server's responsiveness and drop VM heartbeats
below 100%.  Generally at least one VM was oom-killed.  I also had
intermittent problems with VMs hanging during startup and/or the
physical server grinding to a halt.  I was able to get all the VMs
running, but it was a near thing & I had to stagger the startup.

With the setting at 250, results were *much* better.  The load average
was much lower, server responsiveness was good, no VMs hung on startup
and the server didn't halt.  I was also able to start all VMs
simultaneously with no problems, even one that's the most likely to get
oom-killed.

I *was* able to get HighFree as low as 1408Kb, but there was still
enough low memory available.  Obviously some tweaking is in order,
probably based on the amount of physical RAM.

Gene: If you haven't tried adjusting lower_zone_protection, I would
recommend doing so.

-Eric

-- 
Eric Sisler <esisler westminster lib co us>
Library Network Specialist
Westminster Public Library
Westminster, CO USA

Linux - Don't fear the Penguin.
Want to know what we use Linux for?
Visit http://wallace.westminster.lib.co.us/linux