Systemd Journald and audit logging causing journal issues

[Steve Grubb]

On Thursday, October 19, 2017 1:08:22 PM EDT Brad Zynda wrote:
> >> grep perm_mod /etc/audit/audit.rules
> >> -a always,exit -F arch=b64 -S chmod -S fchmod -S fchmodat -F auid>=1000
> >> -F auid!=4294967295 -k perm_mod
> >> -a always,exit -F arch=b32 -S chmod -S fchmod -S fchmodat -F auid>=1000
> >> -F auid!=4294967295 -k perm_mod
> >> -a always,exit -F arch=b64 -S chown -S fchown -S fchownat -S lchown -F
> >> auid>=1000 -F auid!=4294967295 -k perm_mod
> >> -a always,exit -F arch=b32 -S chown -S fchown -S fchownat -S lchown -F
> >> auid>=1000 -F auid!=4294967295 -k perm_mod
> >> -a always,exit -F arch=b64 -S setxattr -S lsetxattr -S fsetxattr -S
> >> removexattr -S lremovexattr -S fremovexattr -F auid>=1000 -F
> >> auid!=4294967295 -k perm_mod
> >> -a always,exit -F arch=b32 -S setxattr -S lsetxattr -S fsetxattr -S
> >> removexattr -S lremovexattr -S fremovexattr -F auid>=1000 -F
> >> auid!=4294967295 -k perm_mod
> >> 
> >> grep delete /etc/audit/audit.rules
> >> -a always,exit -F arch=b64 -S unlink -S unlinkat -S rename -S renameat
> >> -F auid>=1000 -F auid!=4294967295 -k delete
> >> -a always,exit -F arch=b32 -S unlink -S unlinkat -S rename -S renameat
> >> -F auid>=1000 -F auid!=4294967295 -k delete
> >> -a always,exit -F arch=b64 -S init_module -S delete_module -k modules
> >> -a always,exit -F arch=b32 -S init_module -S delete_module -k modules
> > 
> > These rules are well formed. Meaning no obvious holes that would cause
> > unintended events. The other ausearch/aureport commands I gave you should
> > show you what is causing the events and to which files. This information
> > may be used to create some kind of "never" rule that limits what gets
> > audited. If you do create some exclusion rule, it needs to be above the
> > perm_mod rules because audit is a "first match wins" system.
> > 
> > -Steve
> > 
> > -Steve
> 
> Here is a peak report (at least in the last 24 hours) didnt include the
> file summaries as that would make this email HUGE:

Well, the idea is to look for something that's getting hit a lot. What it 
sounds like is that things are getting deleted and modified quite a bit all 
over the place. Does the executable report show a pattern such as one 
application doing a lot? For example, with the rule you have, doing a yum 
update will delete a whole lot of stuff and modify a whole lot of stuff.

Its possible to put exceptions in the rules so that one program does not flood 
the logs. But looking at the quantities below, the audit system can easily 
handle that.

Its also possible to exclude directories from auditing if the pattern is that 
you have a daemon receiving and modifying files and then deleting them. You 
should look at the executables & files and see if you can do with auditing 
what they are doing because its not interesting.

If this is causing you problems on the journald side where its causing other 
tasks to fail. Then I'd break the link between auditd and journald. The amount 
of event you show is highish but well within range of what auditd can do. Just 
make sure flush is set to incremental_async and freq is 100 or 200. You should 
be OK.

-Steve

> Key Summary Report
> ===========================
> total  key
> ===========================
> 8170  perm_mod
> 5390  delete
> 1073  access
> 56  time-change
> 26  session
> 12  privileged
> 7  logins
> 
> 
> Syscall Summary Report
> ==========================
> total  syscall
> ==========================
> 4250  fchmodat
> 1613  chmod
> 831  fchmod
> 521  fchown
> 97  chown
> 12  setxattr
> 
> 
> Syscall Summary Report
> ==========================
> total  syscall
> ==========================
> 2887  unlink
> 2189  rename
> 186  unlinkat
> 
> 
> so from the list the top 2 in each perm_mod and delete from the above
> list seem to be the culprits..