[PATCH][RFC] audit: set wait time to zero when audit failed

[Paul Moore]

On Mon, Sep 16, 2019 at 9:08 PM Li,Rongqing <lirongqing at baidu.com> wrote:
> > -----邮件原件-----
> > 发件人: Paul Moore [mailto:paul at paul-moore.com]
> > 发送时间: 2019年9月17日 6:52
> > 收件人: Li,Rongqing <lirongqing at baidu.com>
> > 抄送: Eric Paris <eparis at redhat.com>; linux-audit at redhat.com
> > 主题: Re: [PATCH][RFC] audit: set wait time to zero when audit failed
> >
> > On Sun, Sep 15, 2019 at 10:55 PM Li,Rongqing <lirongqing at baidu.com> wrote:
> > > > > if audit_log_start failed because queue is full, kauditd is
> > > > > waiting the receiving queue empty, but no receiver, a task will be
> > > > > forced to wait 60 seconds for each audited syscall, and it will be
> > > > > hang for a very long time
> > > > >
> > > > > so at this condition, set the wait time to zero to reduce wait,
> > > > > and restore wait time when audit works again
> > > > >
> > > > > it partially restore the commit 3197542482df ("audit: rework
> > > > > audit_log_start()")
> > > > >
> > > > > Signed-off-by: Li RongQing <lirongqing at baidu.com>
> > > > > Signed-off-by: Liang ZhiCheng <liangzhicheng at baidu.com>
> > > > > ---
> > > > > reboot is taking a very long time on my machine(centos 6u4 +kernel
> > > > > 5.3) since TIF_SYSCALL_AUDIT is set by default, and when reboot,
> > > > > userspace process which receiver audit message , will be killed,
> > > > > and lead to that no user drain the audit queue
> > > > >
> > > > > git bitsect show it is caused by 3197542482df ("audit: rework
> > > > > audit_log_start()")
> > > > >
> > > > >  kernel/audit.c | 9 +++++++--
> > > > >  1 file changed, 7 insertions(+), 2 deletions(-)
> > > >
> > > > This is typically solved by increasing the backlog using the
> > "audit_backlog_limit"
> > > > kernel parameter (link to the docs below).
> > >
> > > It should be able to avoid my issue, but the default behaviors does not
> > working for me; And not all have enough knowledge about audit, who maybe
> > spend lots of effort to find the root cause, and estimate how large should be
> > "audit_backlog_limit"
> >
> > The pause/sleep behavior is desired behavior and is intended to help
> > kauditd/auditd process the audit backlog on a busy system.  If we didn't sleep
> > the current process and give kauditd/auditd a chance to flush the backlog when
> > it was full, a lot of bad things could happen with respect to audit.  We
> > generally select the backlog limit so that this is not a problem for most systems,
> > although there will always be edge cases where the default does not work well;
> > it is impossible to pick defaults that work well for every case.
> >
>
> I just want to it as before 3197542482df ("audit: rework audit_log_start()"),
> wait 60 seconds once if auditd/readaheaad-collector have some problem to
> drain the audit backlog.

The patch you mention fixed what was deemed to be buggy behavior; as
mentioned previously in this thread I see no good reason to go back to
the old behavior.

> > If you are not using audit, you can always disable it via the kernel command line,
> > or at runtime (look at what Fedora does).
> >
> > > > You might also want to investigate
> > > > what is generating some many audit records prior to starting the
> > > > audit daemon.
> > >
> > > It is /sbin/readahead-collector, in fact, we stop the auditd; We are doing a
> > reboot test, which rebooting machine continue to test hardware/software.
> > >
> > > it is same as below:
> > > auditctl -a always,exit -S all -F pid='xxx'
> > > kill -s 19 `pidof auditd`
> > >
> > > then the audited task will be hung
> >
> > So you are seeing this problem only when you run a test, or did you provide this
> > as a reproducer?
>
> auditctl -a always,exit -S all -F ppid=`pidof sshd`
> kill -s 19 `pidof auditd`
> ssh root at 127.0.0.1
>
> then ssh will be hung forever

That is expected behavior.  You are putting a massive audit load on
the system by telling the kernel to audit every syscall that sshd
makes, then you are intentionally killing the audit daemon and
attempting to ssh into the system.  The proper fix(es) here would be
to 1) set reasonable audit rules and/or 2) use an init system that
monitors and restarts auditd when it fails (systemd has this
capability, I believe some others do as well).

-- 
paul moore
www.paul-moore.com