Re: Strange RHEL4 U3 Behavior

["Wayne G" <wgoodric gmail com>]

I notice this  in /var/log/cron directly after update to U3, during
which time no system admin mail was going out.

Mar 18 11:40:01 cfms14 crond[4137]: (root) CMD (/opt/sarcheck/bin/ps1)
Mar 18 11:40:01 cfms14 crond[4138]: (root) CMD (/opt/sarcheck/bin/prst1)
Mar 18 11:43:01 cfms14 crond[2635]: (*system*) RELOAD (/etc/cron.d/sysstat)
Mar 18 11:50:01 cfms14 crond[5546]: PAM unable to
dlopen(/lib/security/$ISA/pam_tally.so)
Mar 18 11:50:01 cfms14 crond[5546]: PAM [dlerror:
/lib/security/../../lib/security/pam_tally.so: undefined symbol:
audit_log_user_message]
Mar 18 11:50:01 cfms14 crond[5546]: PAM adding faulty module:
/lib/security/$ISA/pam_tally.so
Mar 18 11:50:01 cfms14 crond[5546]: Module is unknown
.
.
.
.
Mar 21 08:00:01 cfms14 crond[7490]: PAM unable to
dlopen(/lib/security/$ISA/pam_tally.so)
Mar 21 08:00:01 cfms14 crond[7489]: Module is unknown
Mar 21 08:00:01 cfms14 crond[7490]: PAM [dlerror:
/lib/security/../../lib/security/pam_tally.so: undefined symbol:
audit_log_user_message]
Mar 21 08:00:01 cfms14 crond[7490]: PAM adding faulty module:
/lib/security/$ISA/pam_tally.so
Mar 21 08:00:01 cfms14 crond[7490]: Module is unknown
Mar 21 08:00:59 cfms14 crond[7506]: (CRON) STARTUP (V5.0)
Mar 21 08:01:01 cfms14 crond[7510]: (root) CMD (run-parts /etc/cron.hourly)
Mar 21 08:10:01 cfms14 crond[7517]: (root) CMD (/usr/lib/sa/sa1 1 1)
Mar 21 08:10:01 cfms14 crond[7518]: (root) CMD (/opt/sarcheck/bin/prst1)
Mar 21 08:20:01 cfms14 crond[7532]: (root) CMD (/usr/lib/sa/sa1 1 1)
Mar 21 08:20:01 cfms14 crond[7535]: (root) CMD (/opt/sarcheck/bin/ps1)

And you see, once cron was restarted, everything was back to normal.
Same messages occured on many newly updated machines, but not all
stopped mailing out.

I also now notice that there is no /var/log/sa/sa19 or sa20

[wayne cfms14 sa]$ ls
sa16  sa17  sa18  sa21  sa22  sa23  sa24  sar15  sar16  sar17  sar21 
sar22  sar23

Wayne


On 3/24/06, Ed Brown <ebrown lanl gov> wrote:
> Have you updated sendmail for the recent remote vulnerability?  Perhaps
> someone was trying to (did?) exploit it.  Or possibly the update has
> introduced a problem...
>
> -Ed
>
>
> On Fri, 2006-03-24 at 16:19 -0500, Tom Sightler wrote:
> > We recently had a strange event occur on one of our RHEL4 U3 systems and
> > I'm wondering if anyone out there has any suggestions on what might have
> > happened.  Basically, the system is a RHEL4 U3 system.  It is running on
> > an VMware ESX 2.5.2 Patch 4 system which officially only supports RHEL4
> > U2, however, we don't think this issue was related to VMware although we
> > can't rule that out.
> >
> > Anyway, this system run Squid, Apache, DHCP, DNS, Sendmail, MySQL,
> > Samba, and a few other services.  The Sendmail is configured as an email
> > gateway for inbound mail and performs virus and anti-spam filtering
> > using MailScanner.  We have used this system in this basic configuration
> > for several years, starting with RHEL3 and moving to RHEL4 around the
> > time U1 was released.
> >
> > Now, on to the actually issue.  Yesterday, at almost exactly 10:30AM,
> > the system quit accepting inbound mail.  The sendmail service appeared
> > to be running, but an attempt to telnet to port 25 was greeted only in
> > "connection refused".  We checked logs and could find nothing of
> > interest.  We eventually restarted sendmail and everything was fine.
> > This in itself was unusual, I can't ever remember sendmail stopping in
> > this way previously.
> >
> > At first it looked like that was the only service that was affected,
> > however, upon deeper investigation, we found at least one additional
> > unusual issue.  The system has the sysstat package installed and we
> > noticed that the last stats gathered were at 10:20AM.  Normally, cron
> > would run the sa1 process every 10 minutes, however, this wasn't
> > happening and actually, no cron jobs were running at all, however, the
> > crond service appeared to be still running (ps ax showed it), it was
> > just no longer processing tasks.  We eventually restarted the cron
> > service and things when back to normal.  We've found no other affected
> > services from the event.
> >
> > We spent significant time looking through logs, both on the system
> > itself, the ESX host, and other virtual machines running on that system,
> > and nothing unusual seems to show up in that time frame.  Has anyone
> > seen processes simply "stop" running even though they continue to appear
> > in the process list to look normal?  What other information should I
> > look for if the problem should happen again?  After the fact I realized
> > that I should have probably at least attempted to strace the hung
> > processes.  Any other ideas or suggestions or any similar experiences
> > would be appreciated.
> >
> > Later,
> > Tom
> >
> >
> > --
> > nahant-list mailing list
> > nahant-list redhat com
> > https://www.redhat.com/mailman/listinfo/nahant-list
> --
> Ed Brown <ebrown lanl gov>
>
> --
> nahant-list mailing list
> nahant-list redhat com
> https://www.redhat.com/mailman/listinfo/nahant-list
>