Re: "Out of Memory: Killed process" errors on server running Oracle or VMware

[Tom Sightler <ttsig tuxyturvy com>]

On Wed, 2007-07-25 at 15:37 -0600, Eric Sisler wrote:
> Hi Tom,
> 
> > Also, don't ignore the lower_zone_protection settings.  We've found
> > those to help tremendously in OOM situations involving normal zone
> > starvation.
> 
> Wow!  Setting lower_zone_protection = 250 made a *HUGE* difference, even
> with the regular 32-bit kernel.  Do you have a good source of
> documentation on /proc settings?  I can always Google, but if you have a
> good one could you pass it along?  Thanks!

Google is certainly your friend here, but you can install the kernel-doc
package and the find the information
in /usr/share/doc/kernel-doc-2.6.9/Documentation/sysctl 

> Some preliminary test results, I'm still planning on trying the hugemem
> & 64-bit kernels:
> 
> Test environment:
>   Physical server w/8b RAM, 32-bit RHEL 4.5 (RAM LowTotal = 837724Kb)
>   5 virtual machines configured w/2Gb RAM each
>   1 virtual machine configured w/512Mb

> This potentially overloads the server's physical RAM by ~2.5Gb,
> especially when starting all VMs simultaneously.
...<snip>...
> I *was* able to get HighFree as low as 1408Kb, but there was still
> enough low memory available.  Obviously some tweaking is in order,
> probably based on the amount of physical RAM.

I've seen people use amounts from 50-500 for lower_zone_protection.  My
cases seem to work fine with values in the 100 range, but some people
with large systems and lot's of memory say larger numbers, like 500, are
safer.  All your really doing is reserving memory that cannot be used by
applications or the pagecache.

BTW, it's seems likely to me that your configuration is suspect,
especially if you are using the default VMware server configuration.  By
default VMware Server locks most of the memory for the virtual machines
so that the bulk of the VM will stay in memory.  It does this for
performance since locked memory cannot be swapped.

So let's take a look at your config, you have an 8GB machine and 10.5GB
of VM's to start.  Now you state that this overloads the physical system
by 2.5GB, but actually, you seem to forget that the OS needs some and
VMware application needs some memory too, so it's far more likely that
your overloading the system by 3GB or more.

Now you start all of your VM's, and VMware grabs all the memory it can
(upper and lower zones) and locks it in place so it can't be swapped.
The kernel gladly allocates this memory until such point where it needs
some for itself.  Normally in this case it would simply swap out an
application page and take it for itself except it finds that all of this
memory is locked, so it can't swap it.  Now what does it do?  It has
only one choice really, kill the application with the locked memory.
The system really is OOM.

Setting lower_zone_protection simply gives the kernel a little breathing
room by reserving a section of memory that VMware can't have and thus
can't lock.

You can change the memory locking behavior of VMware Server slightly by
setting the memory host parameters from "Allow some virtual memory to be
swapped" to "Allow most virtual memory to be swapped".  Oversubscribing
memory and trying to combine that with locking the memory in place is a
very dangerous combo and may never be as stable as you would like.

> Gene: If you haven't tried adjusting lower_zone_protection, I would
> recommend doing so.

I think Gene's problem was on RHEL3.  I don't think RHEL3 even has this
tunable, and it's likely that his problem is actually different,
although perhaps similar.  I had meant to suggest that we move your
thread to the RHEL4 list earlier.

Later,
Tom