auditd and redhat cluster

Tue Mar 1 21:25:29 UTC 2016

Philippe,

What does a perf top show?

Do you see get_task_cred or audit_filter_rules as high consumers? If
they are high, then try turning off the monitoring of the /tmp, /dev/shm
and /var/lock/lvm trees or if appropriate, switch to monitoring via a
path directive if you don't need to monitor the entire tree.

Steve, Paul,

I have yet to put together a bug report, or researched to see if the
problem exists upstream, but have discovered recursive directory rules
can be expensive on the kernel. The rules below on a system running
rabbitmq can see get_task_cred and audit_filter_rules above 10% each.

-w /etc/pam.d -p wa -k PAM_Mods
-w /boot -k BOOT_Mods
-w /boot/grub/grub.conf -p war -k BOOT_Mods
-w /etc/security -p wa -k Security_Mods
-w /etc/sysconfig -p wa -k Sysconfig_Mods
-w /etc/ld.so.conf.d -p wa -k Library_Mods
-w /etc/inittab -p wa -k StartUp_Mods
-w /etc/rc.d -p wa -k StartUp_Mods

Regards

On Tue, 2016-03-01 at 09:14 -0500, Steve Grubb wrote:
> On Tuesday, March 01, 2016 02:57:45 PM Maupertuis Philippe wrote:
> > The kernel is  : 2.6.32-573.12.1.el6.x86_64
> > And the whole audit.rules file is  :
> 
> <snip>
> 
> > During the hour preceding the fence we got  these events from the passive
> > node Key Summary Report
> > ===========================
> > total  key
> > ===========================
> > 891  system_commands (ping)
> > 
> > And on the active node  :
> > 
> > Key Summary Report
> > ===========================
> > total  key
> > ===========================
> > 1330  system_commands
> > 286  deletion
> > 
> > I am going to follow your advice and to open a call with redhat.
> > Anyway, I am interested in knowing if auditd has been reported to cause
> > trouble without generating many events.
> 
> Those numbers work out to 27 events per minute. That's not really a lot of 
> events. To see if its the rules or auditd causing the iowait, you might set 
> the logging format to NOLOG. This will discard events rather than log them. If 
> you still have iowait, its something to do with the rules. If that cleared it 
> up, then auditd might be the source. Either way, put the format back to raw. 
> 
> I did some benchmarking of auditd over the holidays and posted some results 
> here:
> 
> https://www.redhat.com/archives/linux-audit/2015-December/msg00061.html
> 
> I'd recommend:
> 
> flush = incremental
> freq = 100
> 
> for a modest performance improvement.
> 
> -Steve
> 
> 
> > -----Message d'origine-----
> > De : Paul Moore [mailto:paul at paul-moore.com]
> > Envoyé : mardi 1 mars 2016 14:25
> > À : Maupertuis Philippe
> > Cc : linux-audit at redhat.com
> > Objet : Re: auditd and redhat cluster
> > 
> > On Mon, Feb 29, 2016 at 7:45 AM, Maupertuis Philippe 
> <philippe.maupertuis at worldline.com> wrote:
> > > Hi list,
> > > 
> > > One clusters fenced the passive node around two hours  after auditd
> > > was started.
> > > 
> > > We have found that iowait has increased since auditd was started and
> > > was unusually high.
> > > 
> > > Auditd wasn’t generating many messages and there were no noticeable
> > > added activity on the disk were the audit and syslog files were written.
> > > 
> > > Besides watches, the only general rules were :
> > > 
> > > # creation
> > > -a exit,always -F arch=b32 -S creat -S mkdir -S mknod -S link -S
> > > symlink -S mkdirat -S mknodat -S linkat -S symlinkat -F uid=root -F
> > > success=1 -k creation -a exit,always -F arch=b64 -S creat -S mkdir -S
> > > mknod -S link -S symlink -S mkdirat -S mknodat -S linkat -S symlinkat
> > > -F uid=root -F success=1 -k creation
> > > 
> > > # deletion
> > > -a exit,always -F arch=b32 -S rmdir -S unlink -S unlinkat -F uid=root
> > > -F
> > > success=1 -k deletion
> > > -a exit,always -F arch=b64 -S rmdir -S unlink -S unlinkat -F uid=root
> > > -F
> > > success=1 -k deletion
> > > 
> > > After the rebot we deleted all rules and didn’t notice extra iowait
> > > anymore.
> > > 
> > > Could these rules be the cause of additional iowait even if not
> > > generating many events (around 20 in two hours) ?
> > > 
> > > Is there any other auditd mechanism  that could explain this phenomenon ?
> > > 
> > > I would appreciate any hints.
> > 
> > Hi Philippe,
> > 
> > First, as this is a RH cluster product, I would suggest contacting RH
> > support with your question if you haven't already; this list is primarily
> > for upstream development and support.
> > 
> > If you are able to experiment with the system, or have a test environment, I
> > would suggest trying to narrow down the list of audit rules/watches to see
> > which rules/watches have the most affect on the iowait times.  You've
> > listed four rules, but you didn't list the watches you have configured. 
> > Also, what kernel version are you using?
> > 
> > --
> > paul moore
> > www.paul-moore.com
> > 
> > !!!*************************************************************************
> > ************ "Ce message et les pièces jointes sont confidentiels et
> > réservés à l'usage exclusif de ses destinataires. Il peut également être
> > protégé par le secret professionnel. Si vous recevez ce message par erreur,
> > merci d'en avertir immédiatement l'expéditeur et de le détruire.
> > L'intégrité du message ne pouvant être assurée sur Internet, la
> > responsabilité de Worldline ne pourra être recherchée quant au contenu de
> > ce message. Bien que les meilleurs efforts soient faits pour maintenir
> > cette transmission exempte de tout virus, l'expéditeur ne donne aucune
> > garantie à cet égard et sa responsabilité ne saurait être recherchée pour
> > tout dommage résultant d'un virus transmis.
> > 
> > This e-mail and the documents attached are confidential and intended solely
> > for the addressee; it may also be privileged. If you receive this e-mail in
> > error, please notify the sender immediately and destroy it. As its
> > integrity cannot be secured on the Internet, the Worldline liability cannot
> > be triggered for the message content. Although the sender endeavours to
> > maintain a computer virus-free network, the sender does not warrant that
> > this transmission is virus-free and will not be liable for any damages
> > resulting from any virus transmitted.!!!"
> > 
> > --
> > Linux-audit mailing list
> > Linux-audit at redhat.com
> > https://www.redhat.com/mailman/listinfo/linux-audit
> 
> 
> --
> Linux-audit mailing list
> Linux-audit at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-audit