Re: "Out of Memory: Killed process" errors on server running Oracle

[Tom Sightler <ttsig tuxyturvy com>]

On Sun, 2007-07-22 at 12:52 +0200, wolf2k5 wrote:
> On 7/17/07, Eric Sisler <esisler westminster lib co us> wrote:
> > "...This problem results from a change in the way the Linux version 2.6
> > kernel's OOM process killer works. In the version 2.4 kernel, the OOM
> > process killer killed the process that requested more memory than could
> > be provided. In version 2.6, this behavior changed so that the OOM
> > process killer kills the largest active process, regardless of the
> > process whose request exceeded available memory."
> 
> We're seeing the same issue on a RHEL4 host running VMware Server with
> several VMs.
> 
> Do you have the link to the VMware KB? What do they suggest to do?

I think the KB article in question references Fedora Core 3 and
basically says FC3 is not supported so your on your own.

If you really think the kernel is not out of memory you could simply
turn the OOM killer off (sysctl vm.oom-kill=0).  I think this needs a
fairly recent for RHEL3 (I think around U6).  Of course, if the system
is actually out of memory then it will simply hang or panic rather than
kill a process and survive.

You could also try playing with overcommit_memory and overcommit_ratio
settings.  By default the Linux kernel will allow memory overcommit
which will actually allow you to start more VM's than you could ever
really support.  If you've started more VM's than your memory will
support they will still start up because of overcommit, but as the
individual VM's actually use their pages they will hit the real limit
and be killed.

Using the overcommit_memory and overcommit_ratio settings you can
control how much memory the kernel is willing to overcommit, this might
keep you VM's from even starting if you have allocated more memory than
the system has.

Also, keep in mind that the OOM killer has to monitor memory in multiple
zones and just because you have 4GB of free memory on your system does
not mean that there is free memory in the low memory zone (this is
normally less of on issue on 64-bit).

If you are running 32-bit and seeing low-memory starvation (which can
lead to OOM killer killing processes even if there appears to be
"plenty" of free memory) then there are several options, like tweaking
the lower_zone_protection settings (RHEL4 and above only I think) or
running a hugemem kernel which, although officially only required for
systems with >16GB of RAM, can help if you system is starved for lower
zone memory.

Perhaps if you post some of the log output from the OOM killer we can
offer more specific suggestions.

Later,
Tom