Re: Strange RHEL4 U3 Behavior

[Tom Sightler <ttsig tuxyturvy com>]

On Mon, 2006-03-27 at 20:07 -0800, Jesse Weisner wrote:
> I'm going to increase a particularly "crashy" slice to 2048 each and see
> how it goes. Have you isolated anything that can make it "crash early,
> crash often"?

We've cleaned off one of our IBM x336 servers running ESX 2.5.2 Patch 4
and are building a test case so that we can open a support case.  We
cloned a copy of the server and downgraded all of the packages so that
everything is exactly like RHEL4 U2.   With 1GB of RAM allocated we can
basically just power the system up and wait 10 minutes or so and things
start crashing.  For some reason Sendmail is one of the first things to
go down, then usually Samba, but eventually everything starts going
crazy.

If we allocate 2GB or more the very same VM it seems solid.

> Out of curiosity, have you made any adjustments to the memory resource
> settings in the VMs? I notice that our VMware admins set the machines
> with 2GB RAM to have a 2GB "minimum" and a higher than normal "shares"
> setting. The 1GB RAM machines all have "normal" shares and no minimum.

We haven't made any adjustments, but in the past we have increased the
number of "shares" to try to compensate for the relatively poor
performance of RHEL4 in a VM.

> Do you suppose VMware could be stealing too much RAM back through
> vmmemctl ballooning? I think you'd start seeing the dreaded OOM killer
> if that was the case.

I don't think that's the case as we've tried running completely without
the VMware tools and the problem still seems to occur, although perhaps
with slightly different behavior.  Our host has 8GB of RAM and we have
removed all other VM's from it so there shouldn't be any memory pressure
anyway and thus the ballon driver shouldn't be trying to reclaim memory.
That's not to say that we can rule it out, but it doesn't appear to be
involved.

Thanks for you feedback, this problem is getting even more interesting
but definitely appears to be related to VMware rather than RHEL4 itself.
I probably should move the conversation over to that forum.

Later,
Tom