Re: "Out of Memory: Killed process" errors on server running Oracle

[Eric Sisler <esisler westminster lib co us>]

Hi Gene,

> We're still weighing the options on our end.  We plan on upgrading to
> the update 9 kernel and seeing if that resolves anything.  Also we're
> adding another server to the cluster to see if that helps alleviate
> the problem. 

Interesting that you're having this problem with RHEL 3.  Our older RHEL
3 GSX servers have *never* experienced this type of problem.  It's only
as we've been upgrading to RHEL 4 & VMware server that this problem
began to appear.  I initially thought the first server I upgraded was
having issues with physical RAM.

A couple of options, which may or may not alleviate your problem.  Tried
on RHEL 4 / 2.6.x kernel, so the 2.4.x kernels may be a little
different:

Check status of oom-killer:
# more /proc/sys/vm/oom-kill

Turn oom-killer off/on:
echo "0" > /proc/sys/vm/oom-kill
echo "1" > /proc/sys/vm/oom-kill

To make this change take effect at boot time, add the following
to /etc/sysctl.conf:
vm.oom-kill = 0

For processes that would have been killed, but weren't because the oom-
killer is disabled, you'll see the following message
in /var/log/messages:

"Would have oom-killed but /proc/sys/vm/oom-kill is disabled"

This seems to be a bit of a two-edged sword with VMware.  Processes that
would normally trigger the oom-killer don't get killed.  Sometimes
things return to normal after the VM starts up and everything's ok.
Other times the server goes into a downward spiral of heavy RAM
utilization and high CPU load and the only "fix" is a hard reboot.

I'm not entirely sure what triggers the oom-killer.  I've noticed it
seems to be a combination of free RAM getting low and a load of 20.00 or
higher.  If free RAM gets low but load stays under ~15.00, generally I
have no problems.  Is Oracle kicking off a process that causes a similar
situation on your end?

I *thought* I had "solved" the problem by redistributing my virtual
machines to different servers.  I started a VM configured with 1.5Gb RAM
on a lightly-loaded, dual quad-core / 16Gb RAM server and the VM got
oom-killed.  At this point our old servers running RHEL 3 & VMware GSX
perform better than our new servers running RHEL 4 & VMware Server.  Go
figure.

Unless a better solution appears, I'm considering rolling back to RHEL 3
(keeping VMware Server) on our servers.

Hope you have better luck on your end.

-Eric

-- 
Eric Sisler <esisler westminster lib co us>
Library Network Specialist
Westminster Public Library
Westminster, CO USA

Linux - Don't fear the Penguin.
Want to know what we use Linux for?
Visit http://wallace.westminster.lib.co.us/linux