Re: Strange RHEL4 U3 Behavior

[Tom Sightler <ttsig tuxyturvy com>]

On Fri, 2006-03-24 at 18:40 -0800, Jesse Weisner wrote:
> I run 8 RHEL AS4 virtual machines on a pair of ESX 2.5.2 hosts, and I 
> have had the exact problem you are describing, but only on certain VMs. 
> All of my VMs are kickstart installed, so I doubt that it could be 
> caused by different software versions. Someone suggested this might be a 
> symptom of a "sendmail hack", but all of our machines have sendmail 
> bound only to 127.0.0.1.

I'm about 99.999% sure this isn't a "sendmail hack" in our case, but I
guess it's hard to rule it out completely.  This system is not directly
connected to the Internet but rather hidden behind both an application
firewall and another store and forward spam/virus filter.  Also, based
on the logs it doesn't appear that there were any active connections
when the sendmail daemon stopped responding.

> The two obvious symptoms of the problem are that Sendmail and Cron seem 
> to die, sometimes sshd as well. If left long enough, it seems that init 
> will just "fall off" as well. Rebooting these zombie VMs requires 
> logging-in to Virtual Center and resetting them because init no longer 
> responds to any shutdown attempts.

Wow, that sounds just like our problem.  Have you ever found anything
else in the log that might give any clues?  I guess maybe this might
have something to do with VMware after all.

> I have triedrunning without the vmmemctl module, without vmware-tools, 
> without the vmxnet driver, all had no effect... pretty frustrating since 
> it's an intermittent problem that can take a week to re-appear.

In our case the system has been running on ESX for about 9 months and
we've never seen this problem before, but we've only been running on U3
for about 8 days before this failure so now we're concerned that it will
happen more often, maybe we were just lucky before.

Thanks,
Tom