Linux audit performance impact

Wed Feb 4 16:15:57 UTC 2015

On 15/02/04, Viswanath, Logeswari P (MCOU OSTL) wrote:
> The intent is to calculate the performance impact by the auditing components such as 
> 
> 1) impact because of kauditd without auditd - but kauditd writes to syslog, so we are unable to determine the impact just because of kauditd - It is fine even if the audit record is dropped by kauditd. Is there any way to do this?

Not yet.  That is a mode that has not been useful to anyone yet.  You
are welcome to hack a custom kernel to disable klog for doing testing
instrumentation.

> 2) impact because of running auditd - log format NOLOG
> 3) impact because of running audispd - small plugin is written which will just read the audit records and doesn't processes it.
> 
> -----Original Message-----
> From: Richard Guy Briggs [mailto:rgb at redhat.com] 
> Sent: Tuesday, February 03, 2015 10:33 PM
> To: Satish Chandra Kilaru
> Cc: Viswanath, Logeswari P (MCOU OSTL); Steve Grubb; linux-audit at redhat.com
> Subject: Re: Linux audit performance impact
> 
> On 15/02/03, Satish Chandra Kilaru wrote:
> > Thanks for The info. But my question was rhetorical... I meant to say 
> > that it would not be much... She is trying to bombard the system with 
> > open calls ... So lots and lots of events will be generated and kernel 
> > has to write down the events some where or discard them...
> 
> Exactly.  It is of little practical use.  You have to do I/O at some point, either to the same disk or another, or to a network interface or serial port, otherwise, just chuck it out.  You could do a performance measurement on a short burst, then drain the queue, but what will that actually tell us?
> 
> > On Tuesday, February 3, 2015, Richard Guy Briggs <rgb at redhat.com> wrote:
> > 
> > > On 15/02/03, Satish Chandra Kilaru wrote:
> > > > How many events can kernel accumulate without I/o ?
> > >
> > > The kernel default is 64 *buffers*, but I think Fedora and RHEL set 
> > > it to 320.  It is now possible to set it to "0" which means limited 
> > > only by system resources.  See "man auditctl", "-b" option.  An 
> > > event can be made up of several buffers.
> > >
> > > Of course, how long a system lasts before the queue blows up depends 
> > > on your rule set...
> > >
> > > However, at the moment, it will still write out to klog if auditd 
> > > isn't running.
> > >
> > > > On Tuesday, February 3, 2015, Viswanath, Logeswari P (MCOU OSTL) < 
> > > > logeswari.pv at hp.com <javascript:;>> wrote:
> > > >
> > > > > I don't want to disable auditing (i.e. disable audit record
> > > collection),
> > > > > but just do not want the records to delivered to user space 
> > > > > since I
> > > want to
> > > > > remove the I/O overhead while running the performance test.
> > > > > Is there any option for this?
> > > > >
> > > > > -----Original Message-----
> > > > > From: Richard Guy Briggs [mailto:rgb at redhat.com <javascript:;>
> > > <javascript:;>]
> > > > > Sent: Thursday, January 29, 2015 10:23 PM
> > > > > To: Viswanath, Logeswari P (MCOU OSTL)
> > > > > Cc: Satish Chandra Kilaru; Steve Grubb; linux-audit at redhat.com
> > > <javascript:;>
> > > > > <javascript:;>
> > > > > Subject: Re: Linux audit performance impact
> > > > >
> > > > > On 15/01/29, Viswanath, Logeswari P (MCOU OSTL) wrote:
> > > > > > Please read my question as “Is there any option to configure 
> > > > > > kaudit not to log audit records to syslog? when auditd not running.”
> > > > >
> > > > > Yeah, remove audit=1 from the kernel command line, or set 
> > > > > audit=0 in
> > > its
> > > > > place.  This will stop all but AVCs and if auditd has ever run 
> > > > > since
> > > boot.
> > > > > If audit=0 is on the kernel boot line, it will be impossible to 
> > > > > run
> > > auditd.
> > > > >
> > > > > There is a feature request that is likely coming soon that could 
> > > > > be
> > > > > useful:
> > > > >
> > > > > https://bugzilla.redhat.com/show_bug.cgi?id=1160046
> > > > > "If no audit daemon is running, but an audit multicast 
> > > > > subscriber is around, then the kernel shouldn't forward audit data to kmsg"
> > > > >
> > > > > > From: Viswanath, Logeswari P (MCOU OSTL)
> > > > > > Sent: Thursday, January 29, 2015 11:49 AM
> > > > > > To: 'Satish Chandra Kilaru'; Steve Grubb
> > > > > > Cc: linux-audit at redhat.com <javascript:;> <javascript:;>
> > > > > > Subject: RE: Linux audit performance impact
> > > > > >
> > > > > > Is there any option to configure kaudit not to log audit 
> > > > > > records to
> > > > > syslog when auditd is running?
> > > > > > This way we can assess the impact of enabling audit without 
> > > > > > involving
> > > > > disk I/o overhead.
> > > > > >
> > > > > > From: Satish Chandra Kilaru [mailto:iam.kilaru at gmail.com
> > > <javascript:;> <javascript:;>]
> > > > > > Sent: Thursday, January 29, 2015 9:12 AM
> > > > > > To: Steve Grubb
> > > > > > Cc: linux-audit at redhat.com <javascript:;> <javascript:;><mailto:
> > > linux-audit at redhat.com <javascript:;>
> > > > > <javascript:;>>; Viswanath,
> > > > > > Logeswari P (MCOU OSTL)
> > > > > > Subject: Re: Linux audit performance impact
> > > > > >
> > > > > > I agree with you... but writing to disk can trigger further 
> > > > > > events
> > > > > leading spiralling of events...
> > > > > > I brought down my server few times with stupid rules...
> > > > > >
> > > > > > On Wed, Jan 28, 2015 at 10:39 PM, Steve Grubb 
> > > > > > <sgrubb at redhat.com
> > > <javascript:;>
> > > > > <javascript:;><mailto:sgrubb at redhat.com <javascript:;>
> > > <javascript:;>>> wrote:
> > > > > > On Wednesday, January 28, 2015 10:18:47 AM Satish Chandra 
> > > > > > Kilaru
> > > wrote:
> > > > > > > Write your own program to receive audit events directly 
> > > > > > > without using auditd...
> > > > > > > That should be faster ....
> > > > > > > Auditd will log the events to disk causing more I/o than u need...
> > > > > >
> > > > > > But even that is configurable in many ways. You can decide if 
> > > > > > you
> > > want
> > > > > > logging to disk or not and what kind of assurance that it made 
> > > > > > it to disk and the priority of that audit daemon. Then you 
> > > > > > also have all
> > > the
> > > > > > normal tuning knobs for disk throughput that you would use for 
> > > > > > any disk performance critical system.
> > > > > >
> > > > > > -Steve
> > > > > >
> > > > > > > On Wednesday, January 28, 2015, Viswanath, Logeswari P (MCOU 
> > > > > > > OSTL)
> > > <
> > > > > > >
> > > > > > > logeswari.pv at hp.com <javascript:;> <javascript:;><mailto:
> > > logeswari.pv at hp.com <javascript:;>
> > > > > <javascript:;>>> wrote:
> > > > > > > >  Hi Steve,
> > > > > > > >
> > > > > > > > I am Logeswari working for HP.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > We want to know audit performance impact on RHEL and Suse 
> > > > > > > > linux
> > > to
> > > > > > > > help us evaluate linux audit as data source for our host 
> > > > > > > > based
> > > IDS.
> > > > > > > >
> > > > > > > > When we ran our own performance test with a test audispd 
> > > > > > > > plugin, we found if a system can perform 200000 open/close 
> > > > > > > > system calls per second without auditing, system can 
> > > > > > > > perform only 3000 open/close system calls auditing is 
> > > > > > > > enabled for open/close system call which is a HUGE impact 
> > > > > > > > on the system performance. It would
> > > be
> > > > > > > > great if anyone can help us answering the following questions.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > 1)      Is this performance impact expected? If yes, what is the
> > > > > reason
> > > > > > > > behind it and can we fix it?
> > > > > > > >
> > > > > > > > 2)      Have anyone done any benchmarking for performance
> > > impact? If
> > > > > yes,
> > > > > > > > can you please share the numbers and also the 
> > > > > > > > steps/programs used the run the same.
> > > > > > > >
> > > > > > > > 3)      Help us validating the performance test we have done in
> > > our
> > > > > test
> > > > > > > > setup using the steps mentioned along with the results attached.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Attached test program (loader.c) to invoke open and close 
> > > > > > > > system
> > > > > calls.
> > > > > > > >
> > > > > > > > Attached idskerndsp is the audispd plugin program.
> > > > > > > >
> > > > > > > > We used time command to determine how much time the system 
> > > > > > > > took
> > > to
> > > > > > > > complete 50000 open/close system calls without (results 
> > > > > > > > attached
> > > > > > > > Without-auditing) and with auditing enabled on the system 
> > > > > > > > (With-auditing-NOLOG-audispd-plugin and With-auditing-RAW)
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > System details:
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > 1 CPU machine
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > *OS Version*
> > > > > > > >
> > > > > > > > RHEL 6.5
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > *Kernel Version*
> > > > > > > >
> > > > > > > > uname –r
> > > > > > > >
> > > > > > > > 2.6.32-431.el6.x86_64
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Note: auditd was occupying 35% of CPU and was sleeping for 
> > > > > > > > most
> > > of
> > > > > > > > the time whereas kauditd was occupying 20% of the CPU.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Thanks & Regards,
> > > > > > > >
> > > > > > > > Logeswari.
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Please Donate to www.wikipedia.org<http://www.wikipedia.org>
> > > > >
> > > > > > --
> > > > > > Linux-audit mailing list
> > > > > > Linux-audit at redhat.com <javascript:;> <javascript:;> 
> > > > > > https://www.redhat.com/mailman/listinfo/linux-audit
> > > > >
> > > > >
> > > > > - RGB
> > > > >
> > > > > --
> > > > > Richard Guy Briggs <rbriggs at redhat.com <javascript:;> 
> > > > > <javascript:;>> Senior Software Engineer, Kernel Security, AMER 
> > > > > ENG Base Operating Systems, Red Hat Remote, Ottawa, Canada
> > > > > Voice: +1.647.777.2635, Internal: (81) 32635, Alt: 
> > > > > +1.613.693.0684x3545
> > > > >
> > > >
> > > >
> > > > --
> > > > Please Donate to www.wikipedia.org
> > >
> > > - RGB
> > >
> > > --
> > > Richard Guy Briggs <rbriggs at redhat.com <javascript:;>> Senior 
> > > Software Engineer, Kernel Security, AMER ENG Base Operating Systems, 
> > > Red Hat Remote, Ottawa, Canada
> > > Voice: +1.647.777.2635, Internal: (81) 32635, Alt: 
> > > +1.613.693.0684x3545
> > >
> > 
> > 
> > --
> > Please Donate to www.wikipedia.org
> 
> - RGB
> 
> --
> Richard Guy Briggs <rbriggs at redhat.com>
> Senior Software Engineer, Kernel Security, AMER ENG Base Operating Systems, Red Hat Remote, Ottawa, Canada
> Voice: +1.647.777.2635, Internal: (81) 32635, Alt: +1.613.693.0684x3545

- RGB

--
Richard Guy Briggs <rbriggs at redhat.com>
Senior Software Engineer, Kernel Security, AMER ENG Base Operating Systems, Red Hat
Remote, Ottawa, Canada
Voice: +1.647.777.2635, Internal: (81) 32635, Alt: +1.613.693.0684x3545