BIG performance hit with auditd on large systems (>64 CPUs)

[Klaus Lichtenwalder]

Hi,

we have a few SAP systems on RHEV (so virtualized on KVM) with >= 74
CPUs and >= 400G RAM.
When the system is busy with large SAP jobs, it goes onto its knees with
cpu %system up to 80%, thus making the SAP jobs run twice as long. As
soon as you stop auditd everything returns to normal...

Facts:
RHEL6 instances on RHEL7 hosts.
the rule set (see below) runs fine on any other system with less cpus
(<64, maybe this is the cut off?). We have smaller systems with this
rule set that rotate the audit file nearly every minute without any
noticable performance hit, these SAP systems rotate once every
20-24hours....

Anyone has an idea?

Here's an excerpt from "perf top":
with auditd running:

> Samples: 28M of event 'cpu-clock', Event count (approx.): 236747914918
> Overhead Shared Object Symbol
> 23.13% [kernel] [k] get_task_cred
> 10.05% [kernel] [k] audit_filter_rules
> 4.21% [kernel] [k] _spin_unlock_irqrestore
> 3.30% libdb2e.so.1 [.] sqlbfix
> 2.92% [kernel] [k] finish_task_switch
> 1.69% disp+work [.] rrol_in
> 1.69% disp+work [.] rrol_out
> 0.98% [kernel] [k] run_timer_softirq
> 0.96% [kernel] [k] rcu_process_gp_end
> 

auditd stopped:

> Samples: 3M of event 'cpu-clock', Event count (approx.): 526535382557
> Overhead Shared Object Symbol
> 2.41% disp+work [.] memcmpU16
> 2.32% disp+work [.] MmxMalloc2
> 2.25% disp+work [.] ab_Rudi
> 2.07% disp+work [.] rrol_out
> 1.98% disp+work [.] rrol_in
> 1.95% disp+work [.] ab_CompByCmpCntx
> 1.88% libdb2e.so.1 [.] sqlbfix
> 1.73% disp+work [.] MmxFree2
> 1.62% [kernel] [k] run_timer_softirq
> 1.56% [kernel] [k] __do_softirq
> 1.39% disp+work [.] ab_InitRcDecompress
> 
> These are the audit rules:
> auditctl -l
> -a always,exit -S all -F path=/etc/environment -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F path=/etc/login.defs -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F path=/etc/rsyslog.conf -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F path=/etc/ssh/sshd_config -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F path=/etc/cron.allow -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F path=/etc/cron.deny -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F dir=/etc/cron.d -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F dir=/etc/cron.daily -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F dir=/etc/cron.hourly -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F dir=/etc/cron.monthly -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F dir=/etc/cron.weekly -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F path=/etc/aliases -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F dir=/etc/alternatives -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F path=/etc/at.allow -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F path=/etc/at.deny -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F path=/etc/audisp/plugins.d/syslog.conf -F perm=wa -F auid>=400 -F key=CRIT_AUDIT
> -a always,exit -S all -F path=/etc/audisp/audispd.conf -F perm=wa -F auid>=400 -F key=CRIT_AUDIT
> -a always,exit -S all -F path=/etc/audit/auditd.conf -F perm=wa -F auid>=400 -F key=CRIT_AUDIT
> -a always,exit -S all -F path=/etc/bashrc -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F path=/etc/crontab -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F path=/etc/shells -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F dir=/etc/default -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F path=/etc/depmod.conf -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F dir=/etc/depmod.d -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F path=/etc/exports -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F path=/etc/group -F perm=wa -F auid>=400 -F key=USER_MGMT
> -a always,exit -S all -F path=/etc/passwd -F perm=wa -F auid>=400 -F key=USER_MGMT
> -a always,exit -S all -F path=/etc/shadow -F perm=wa -F auid>=400 -F key=USER_MGMT
> -a always,exit -S all -F path=/etc/inittab -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F dir=/bin -F perm=wa -F auid>=400 -F key=CRIT_PROG
> -a always,exit -S all -F dir=/sbin -F perm=wa -F auid>=400 -F key=CRIT_PROG
> -a always,exit -S all -F dir=/usr/bin -F perm=wa -F auid>=400 -F key=CRIT_PROG
> -a always,exit -S all -F dir=/usr/sbin -F perm=wa -F auid>=400 -F key=CRIT_PROG
> -a always,exit -S all -F dir=/etc/init.d -F perm=wa -F auid>=400 -F key=CRIT_PROG
> -a always,exit -S all -F path=/etc/nsswitch.conf -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F path=/etc/ldap.conf -F perm=wa -F auid>=400 -F key=USER_MGMT
> -a always,exit -S all -F path=/etc/sssd/sssd.conf -F perm=wa -F auid>=400 -F key=USER_MGMT
> -a always,exit -S all -F dir=/var/spool/cron -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F path=/var/spool/atjobs -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F path=/usr/bin/sudo -F perm=x -F auid>=400 -F key=USER_MGMT
> -a always,exit -S all -F path=/etc/sudoers -F perm=wa -F auid>=400 -F key=USER_MGMT
> -a always,exit -S all -F dir=/etc/sudoers.d -F perm=wa -F auid>=400 -F key=USER_MGMT
> -a always,exit -F arch=b64 -S execve -F auid>=500 -F auid<10000 -F key=USER_EXEC
> -a always,exit -F arch=b64 -S execve -F auid>=5000000 -F auid!=-1 -F key=USER_EXEC
> -a always,exit -S all -F dir=/etc/pam.d -F perm=wa -F auid>=400 -F key=CRIT_PAM
> -a always,exit -S all -F dir=/etc/security -F perm=wa -F auid>=400 -F key=CRIT_CONF
> -a always,exit -S all -F path=/etc/libaudit.conf -F perm=wa -F auid>=400 -F key=CRIT_AUDIT
> -a always,exit -S all -F path=/etc/init.d/auditd -F perm=wa -F auid>=400 -F key=CRIT_AUDIT
> -a always,exit -S all -F dir=/appdata/daten/S3_audit -F perm=rwa -F auid>=400 -F auid<10000 -F auid!=-1 -F key=S3DATA
> -a always,exit -S all -F dir=/appdata/daten/S3_audit -F perm=rwa -F auid>=5000000 -F auid!=-1 -F key=S3DATA
> 

-- 
------------------------------------------------------------------------
 Klaus Lichtenwalder, Dipl. Inform.,  http://www.lichtenwalder.name/
 PGP Key fingerprint: 3AE6 044D 1161 1ABF AC2D 23B3 4C15 7232 FDCA 0980