problem while restarting auditd

[Steve Grubb]

On Thursday, September 15, 2011 02:32:59 AM Vipin Rathor wrote:
> One strange thing I'm seeing in /var/log/messages w.r.t. auditd restart.
> 
> 2011-09-14T11:49:14.541661-07:00 audisp-remote: audisp-remote is
> exiting on stop request
> 2011-09-14T11:49:18.741166-07:00 kernel: audit: *NO* daemon at
> audit_pid=1652525 2011-09-14T11:49:18.741190-07:00 kernel: __ratelimit:
> 366 callbacks suppressed 2011-09-14T11:49:18.745558-07:00 auditd[1654362]:
> Started dispatcher: /sbin/audispd pid: 1654364
> 2011-09-14T11:49:18.746081-07:00 audispd: max_restarts_parser called with:
> 10 2011-09-14T11:49:18.746099-07:00 audispd: priority_boost_parser called
> with: 10 2011-09-14T11:49:18.746666-07:00 audispd: audispd initialized
> with q_depth=90000 and 1 active plugins
> 2011-09-14T11:49:18.747047-07:00 audisp-remote: Connected to
> <remote_audit_logging_server_IP>
> 2011-09-14T11:49:18.750761-07:00 kernel: audit: audit_lost=3823
> audit_rate_limit=0 audit_backlog_limit=20480
> 2011-09-14T11:49:18.750773-07:00 kernel: audit: auditd dissapeared
>                       <========= why this message?
> 2011-09-14T11:49:18.750777-07:00 kernel:

This comes from the following code:
http://lxr.linux.no/#linux+v3.0.4/kernel/audit.c#L401

It sort of follows this:

 446                        if (audit_pid)
 447                                kauditd_send_skb(skb);

Then 
 401        err = netlink_unicast(audit_sock, skb, audit_nlk_pid, 0);
 402        if (err < 0) {
 404                printk(KERN_ERR "audit: *NO* daemon at audit_pid=%d\n", 
audit_pid);
 405                audit_log_lost("auditd disappeared\n");

So, what looks like happened is you have a busy system and an event was queued to be 
sent to user space, the audit_pid exited so it started the call, but by the time the 
call was made, the netlink layer couldn't find the pid and then failed.

Eric, is there anything that can be done about this race?

 
> Whenever I'm restarting the auditd using 'service auditd restart'
> command, the auditd gets restarted. But the very next moment, I get
> "kernel: audit: auditd dissapeared " message & auditing stops
> (actually it falls back to syslog). I've to again run 'service auditd
> restart' to get the auditing back. So it is taking two restart
> operation to do the job. This behavior is consistent & I can recreate
> at will.

This is something strange too. But sounds like perhaps another race of some kind.

-Steve