auditd and redhat cluster

Wed Mar 9 09:44:53 UTC 2016

Sorry for the delayed answer.
We restarted auditd on both node of the cluster after taking snapshot with perf top several time before and after.
The auditd processes were higher on the passive node (15% together) but it's probably a sampling effect since the server was mostly idle.
On the active node , audit_filter_rules was around 1% and get_task_cred around 0.8%.
I will try to replicate the settings in a test environment to have more leeway in playing with the rules.
Regards
Philippe

-----Message d'origine-----
De : Burn Alting [mailto:burn at swtf.dyndns.org]
Envoyé : mardi 1 mars 2016 22:25
À : Steve Grubb
Cc : linux-audit at redhat.com; Maupertuis Philippe
Objet : Re: auditd and redhat cluster

Philippe,

What does a perf top show?

Do you see get_task_cred or audit_filter_rules as high consumers? If they are high, then try turning off the monitoring of the /tmp, /dev/shm and /var/lock/lvm trees or if appropriate, switch to monitoring via a path directive if you don't need to monitor the entire tree.

Steve, Paul,

I have yet to put together a bug report, or researched to see if the problem exists upstream, but have discovered recursive directory rules can be expensive on the kernel. The rules below on a system running rabbitmq can see get_task_cred and audit_filter_rules above 10% each.

-w /etc/pam.d -p wa -k PAM_Mods
-w /boot -k BOOT_Mods
-w /boot/grub/grub.conf -p war -k BOOT_Mods -w /etc/security -p wa -k Security_Mods -w /etc/sysconfig -p wa -k Sysconfig_Mods -w /etc/ld.so.conf.d -p wa -k Library_Mods -w /etc/inittab -p wa -k StartUp_Mods -w /etc/rc.d -p wa -k StartUp_Mods

Regards

On Tue, 2016-03-01 at 09:14 -0500, Steve Grubb wrote:
> On Tuesday, March 01, 2016 02:57:45 PM Maupertuis Philippe wrote:
> > The kernel is  : 2.6.32-573.12.1.el6.x86_64 And the whole
> > audit.rules file is  :
>
> <snip>
>
> > During the hour preceding the fence we got  these events from the
> > passive node Key Summary Report =========================== total
> > key ===========================
> > 891  system_commands (ping)
> >
> > And on the active node  :
> >
> > Key Summary Report
> > ===========================
> > total  key
> > ===========================
> > 1330  system_commands
> > 286  deletion
> >
> > I am going to follow your advice and to open a call with redhat.
> > Anyway, I am interested in knowing if auditd has been reported to
> > cause trouble without generating many events.
>
> Those numbers work out to 27 events per minute. That's not really a
> lot of events. To see if its the rules or auditd causing the iowait,
> you might set the logging format to NOLOG. This will discard events
> rather than log them. If you still have iowait, its something to do
> with the rules. If that cleared it up, then auditd might be the source. Either way, put the format back to raw.
>
> I did some benchmarking of auditd over the holidays and posted some
> results
> here:
>
> https://www.redhat.com/archives/linux-audit/2015-December/msg00061.htm
> l
>
> I'd recommend:
>
> flush = incremental
> freq = 100
>
> for a modest performance improvement.
>
> -Steve
>
>
> > -----Message d'origine-----
> > De : Paul Moore [mailto:paul at paul-moore.com] Envoyé : mardi 1 mars
> > 2016 14:25 À : Maupertuis Philippe Cc : linux-audit at redhat.com Objet
> > : Re: auditd and redhat cluster
> >
> > On Mon, Feb 29, 2016 at 7:45 AM, Maupertuis Philippe
> <philippe.maupertuis at worldline.com> wrote:
> > > Hi list,
> > >
> > > One clusters fenced the passive node around two hours  after
> > > auditd was started.
> > >
> > > We have found that iowait has increased since auditd was started
> > > and was unusually high.
> > >
> > > Auditd wasn’t generating many messages and there were no
> > > noticeable added activity on the disk were the audit and syslog files were written.
> > >
> > > Besides watches, the only general rules were :
> > >
> > > # creation
> > > -a exit,always -F arch=b32 -S creat -S mkdir -S mknod -S link -S
> > > symlink -S mkdirat -S mknodat -S linkat -S symlinkat -F uid=root
> > > -F
> > > success=1 -k creation -a exit,always -F arch=b64 -S creat -S mkdir
> > > -S mknod -S link -S symlink -S mkdirat -S mknodat -S linkat -S
> > > symlinkat -F uid=root -F success=1 -k creation
> > >
> > > # deletion
> > > -a exit,always -F arch=b32 -S rmdir -S unlink -S unlinkat -F
> > > uid=root -F
> > > success=1 -k deletion
> > > -a exit,always -F arch=b64 -S rmdir -S unlink -S unlinkat -F
> > > uid=root -F
> > > success=1 -k deletion
> > >
> > > After the rebot we deleted all rules and didn’t notice extra
> > > iowait anymore.
> > >
> > > Could these rules be the cause of additional iowait even if not
> > > generating many events (around 20 in two hours) ?
> > >
> > > Is there any other auditd mechanism  that could explain this phenomenon ?
> > >
> > > I would appreciate any hints.
> >
> > Hi Philippe,
> >
> > First, as this is a RH cluster product, I would suggest contacting
> > RH support with your question if you haven't already; this list is
> > primarily for upstream development and support.
> >
> > If you are able to experiment with the system, or have a test
> > environment, I would suggest trying to narrow down the list of audit
> > rules/watches to see which rules/watches have the most affect on the
> > iowait times.  You've listed four rules, but you didn't list the watches you have configured.
> > Also, what kernel version are you using?
> >
> > --
> > paul moore
> > www.paul-moore.com
> >
> > !!!*****************************************************************
> > ********
> > ************ "Ce message et les pièces jointes sont confidentiels et
> > réservés à l'usage exclusif de ses destinataires. Il peut également
> > être protégé par le secret professionnel. Si vous recevez ce message
> > par erreur, merci d'en avertir immédiatement l'expéditeur et de le détruire.
> > L'intégrité du message ne pouvant être assurée sur Internet, la
> > responsabilité de Worldline ne pourra être recherchée quant au
> > contenu de ce message. Bien que les meilleurs efforts soient faits
> > pour maintenir cette transmission exempte de tout virus,
> > l'expéditeur ne donne aucune garantie à cet égard et sa
> > responsabilité ne saurait être recherchée pour tout dommage résultant d'un virus transmis.
> >
> > This e-mail and the documents attached are confidential and intended
> > solely for the addressee; it may also be privileged. If you receive
> > this e-mail in error, please notify the sender immediately and
> > destroy it. As its integrity cannot be secured on the Internet, the
> > Worldline liability cannot be triggered for the message content.
> > Although the sender endeavours to maintain a computer virus-free
> > network, the sender does not warrant that this transmission is
> > virus-free and will not be liable for any damages resulting from any virus transmitted.!!!"
> >
> > --
> > Linux-audit mailing list
> > Linux-audit at redhat.com
> > https://www.redhat.com/mailman/listinfo/linux-audit
>
>
> --
> Linux-audit mailing list
> Linux-audit at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-audit

!!!*************************************************************************************
"Ce message et les pièces jointes sont confidentiels et réservés à l'usage exclusif de ses destinataires. Il peut également être protégé par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de le détruire. L'intégrité du message ne pouvant être assurée sur Internet, la responsabilité de Worldline ne pourra être recherchée quant au contenu de ce message. Bien que les meilleurs efforts soient faits pour maintenir cette transmission exempte de tout virus, l'expéditeur ne donne aucune garantie à cet égard et sa responsabilité ne saurait être recherchée pour tout dommage résultant d'un virus transmis.

This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Worldline liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.!!!"