Auditd reconfigure using SIGHUP

[Steve Grubb]

On Thursday, January 5, 2017 11:04:55 AM EST Bhagwat, Shriniketan Manjunath 
wrote:
> Greetings,
> 
> I was trying to reconfigure the Audit using SIGHUP and came across with
> below behavior of Audit. The audit version I am using is 2.3.6.

I have been able to reproduce the issue. I am still tracing through the 
problem. As best I can tell, there are at least 3 problems. 1) libev eating 
the signal - this will be fixed by defining EV_CHILD_ENABLE to 0. This will make 
all child interference go away; 2) on startup audispd detects no plugins and 
exits before auditd can register a handler in scenario 2 - this might be fixed 
with a sigaction handler until libev starts processing events; 3) pipe 
descriptors being invalid in audispd sometimes. This one is a mystery.

Still investigating. Thanks for reporting this.

-Steve

> Scenario 1: Starting auditd with active audispd plugin.
> 1. Activate the audispd plugin by setting active = yes in configuration file
> (example: /etc/audisp/plugins.d/syslog.conf) 2. Start the Auditd. Starting
> auditd will create the both Auditd and Audispd process. 3. Deactivate the
> audispd plugin by setting active = no in configuration file. 4. Reconfigure
> the Auditd by sending SIGHUP to Auditd (>> kill -SIGHUP <auditd_pid>). This
> reconfiguration stops the audispd process. 5. Activate the audispd plugin
> by setting active = yes in configuration file. 6. Reconfigure the Auditd by
> sending SIGHUP to Auditd. This should start the audispd process, however
> audispd process will not be started in this reconfiguration.
> 
> Scenario2: Starting auditd with no active audispd plugin.
> 1. Start the Auditd. Starting auditd will create the only Auditd process.
> 2. Activate the audispd plugin by setting active = yes in configuration
> file. 3. Reconfigure the Auditd by sending SIGHUP to Auditd. This should
> start the audispd process, however in few cases audispd process will not be
> started in this reconfiguration. 4. Deactivate the audispd plugin by
> setting active = no in configuration file. 5. Reconfigure the Auditd by
> sending SIGHUP to Auditd. This reconfiguration stops the audispd process.
> 6. Activate the audispd plugin by setting active = yes in configuration
> file. 7. Reconfigure the Auditd by sending SIGHUP to Auditd. This should
> start the audispd process, however audispd process will not be started in
> this reconfiguration.
> 
> 
> As per the change log of audit version  2.3.2, below fix was made.
> 
> Fix: In auditd, restart dispatcher on SIGHUP if it had previously exited
> 
> 
> 
> I have analyzed Auditd code of version 2.3.6 and below is my observation.
> 
> When the Auditd is started, it calls init_dispatcher() to start the Audispd.
> init_dispatcher() starts the Audispd and maintains its pid value in a
> global variable (auditd-dispatch.c). When the audispd is terminated,
> SIGCHLD handler of Auditd i.e child_handler() does the waitpid() to remove
> the child process from zombie state and calls dispatcher_reaped(), where
> this pid is set to zero. During reconfigure using SIGHUP,
> reconfigure_dispatcher() checks for this pid value. If pid is valid then,
> SIGHUP is sent to audispd otherwise init_dispatcher() is called to start
> the audispd.
> 
> 
> Auditd uses event loop ev_signal to track the child process (SIGCHLD). The
> event loop from libev also has child handler childcb() (in ev.c). This
> handler childcb() also does the waitpid(). When audispd terminates, libev's
> child handler is getting called first, waitpid() removes the audispd from
> zombie state. Then the control is passed to Auditd's child handler
> child_handler(). Since the audispd has been already removed from process
> stable, waitpid() call done in child_handler() will return ECHILD. Hence
> the dispatcher_reaped() is not getting called to make the internally
> maintained pid variable to zero. This results in subsequent SIGHUP set to
> Auditd end up in sending SIGHUP to non-existing audispd process.
> 
> In case of step 3 of scenario 2: When the auditd is started without any
> active audispd plugin(step1), Auditd starts the audispd. However since
> there are no active plugins Audispd will get terminated. In this case
> shutdown_dispatcher() was called from dispatch_event() where pid value was
> set to zero. Hence the reconfiguration in step 3 started audispd process in
> my case.
> 
> I have put traces in auditd code to validate the above behavior. This
> behavior is applicable for Audit version 2.6.4 as well.  Is this behavior a
> known issue?
> 
> Regards,
> Ketan