[dm-devel] [PATCH 09/13] multipathd: Implement systemd watchdog integration

[Hannes Reinecke]

On 11/22/2013 11:17 PM, Benjamin Marzinski wrote:
> On Fri, Nov 15, 2013 at 11:29:40AM +0100, Hannes Reinecke wrote:
>> In the past there have been several instances where multipathd
>> would hang with the checkerloop as some path checker might not
>> be able to return in time.
>> This patch now activates the watchdog feature from systemd
>> to shutdown (and possibly restart) multipathd in these
>> situations.
> 
> This might need more of a systemd fix that a multipathd one, but once
> multipathd times out the watchdog timer, even if it starts sending
> notifications at an acceptable rate again, the service is still listed
> as failed.
> 
> # service multipathd status
> Redirecting to # /bin/systemctl status multipathd.service
> multipathd.service - Device-Mapper Multipath Device Controller
>    Loaded: loaded (/usr/lib/systemd/system/multipathd.service; enabled)
>    Active: failed (Result: watchdog) since Fri 2013-11-22 09:43:01 CST; 9min ago
>  Main PID: 6321
>    Status: "running"
>    CGroup: name=systemd:/system/multipathd.service
>            └─6321 /sbin/multipathd -d -s
> 
> More annoying, the logs fills up with messages like
> 
> Nov 22 09:46:28 ask-08 systemd[1]: multipathd.service: Got notification
> message from PID 6321, but reception only permitted for PID 0
> Nov 22 09:46:29 ask-08 systemd[1]: multipathd.service: Got notification
> message from PID 6321, but reception only permitted for PID 0
> Nov 22 09:46:30 ask-08 systemd[1]: multipathd.service: Got notification
> message from PID 6321, but reception only permitted for PID 0
> Nov 22 09:46:31 ask-08 systemd[1]: multipathd.service: Got notification
> message from PID 6321, but reception only permitted for PID 0
> 
> Also
> 
> # service multipathd stop
> 
> won't kill it. Even worse
> 
> # service multipathd start
> 
> WILL kill it without successfully restarting another version. A second
> 
> # service multipathd start
> 
> is necessary to get things back to a functional state again.
> 
Actually, upstream systemd (>= v207) now has a new flag

restart=on-watchdog

With that systemd should be restarting multipathd after a watchdog
timeout. That should solve you immediate problem here.

> I'm not asking for systemd to actually shut down multipathd.  In a
> production setup, killing multipathd because it had a temporary stall
> seems like bad default behavior.  I haven't looked at the systemd
> watchdog code to know if this is possible, but ideally, multipathd would
> be able to just start sending watchdog notifications again, and be able
> to continue on with just a message in the logs recording the timeout.
> 
Not stopping. Restarting.
The whole point of the watchdog code is to take some action if the
watchdog messages fail.
We should aim for
a) make the watchdog interval the longest interval we're prepared to
   checkerloop to complete (hence the patch to measure the elapsed
   time per loop iteration)
b) have systemd restart multipathd whenever the watchdog triggers,
   as then we're sure we can't recover from this.

That should cover your sentiment, right?

> I realize that there is a benefit to letting people know that there was
> a problem, but the way it's appearing now, it will be pretty confusing to
> the sysadmin who sees that, and filling up the logs with notification
> rejections is pretty annoying.
> 
Yeah, correct. We should be using the 'restart' flag in the service
file. I did not do this as the patch went into systemd only
recently, and one would need to figure out how to treat
installations where an older systemd version is running.

> And as long as I'm asking for systemd things, the ability to add a rule
> to the unit file that kills the service and forces a core dump when
> watchdog timer was tripped would help tracking down what's stalling the
> checker loop.  Like I said before, I don't think this should be
> happening by default, but putting it in there commented out might not be
> a bad idea.
> 
Yeah, that would be preferable. Sadly there is no 'force coredump'
option. What I would like to have is a
'on-watchdog'
option in systemd, where one can configure the action which needs to
be taken when the watchdog triggers.
Only adding a new option is touching systemd in tons of various
places, so my initial attempt here failed.
So I went for the easier option to just add a new flag to an
existing setting.

Cheers,

Hannes
P.S.: But hey, at least someone is actually testing this stuff.
Cool.
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare at suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)