[dm-devel] LSF: Multipathing and path checking question

[Hannes Reinecke]

Hi Mike,

Mike Christie wrote:
> Oops, I mashed two topics together. See below.
> 
> Mike Christie wrote:
>> Hannes Reinecke wrote:
>>>
>>> FC Transport already maintains an attribute for the path state, and even
>>> sends netlink events if and when this attribute changes. For iSCSI I
>>> have
>>
>> Are you referring to fc_host_post_event? Is the same thing we talked
>> about last year, where you wanted events? Is this in multipath tools
>> now or just in the SLES ones?
>>
>> For something like FCH_EVT_LINKDOWN, are you going to fail the path at
>> that time or when would the multipath path be marked failed?
>>
> 
> I was asking this because it seems we have people always making
> bugzillas saying they did not want the path to be marked failed for
> short problems.
> 
> There was the problem where we might get DID_ERROR for temporary dropped
> frame. That would be fixed by just listening to transport events like
> you explained.
> 
> But then I thought there was the case where if we get a linkdown then
> linkup within a couple seconds, we would not want to transition the
> multipath path state.
> 
> So below while you were talking about when to remove the device, I was
> talking about when to mark the path failed.
> 
> 
I have the same bugzillas, too :-)

My proposal is to handle this in several stages:

- path fails
-> Send out netlink event
-> start dev_loss_tmo and fast_fail_io timer
-> fast_fail_io timer triggers: Abort all oustanding I/O with
   DID_TRANSPORT_DISRUPTED, return DID_TRANSPORT_FAILFAST for
   any future I/O, and send out netlink event.
-> dev_loss_tmo timer triggers: Remove sdev and cleanup rport.
   netlink event is sent implicitely by removing the sdev.

Multipath would then interact with this sequence by:

- Upon receiving 'path failed' event: mark path as 'ghost' or 'blocked',
  ie no I/O is currently possible and will be queued (no path switch yet).
- Upon receiving 'fast_fail_io' event: switch paths and resubmit queued I/Os
- Upon receiving 'path removed' event: remove path from internal structures,
  update multipath maps etc.

The time between 'path failed' and 'fast_fail_io triggers' would then be
able to capture any jitter / intermittent failures. Between 
'fast_fail_io triggers' and 'path removed' the path would be held in some
sort of 'limbo' in case it comes back again, eg for maintenance/SP update
etc. And we can even increase this one to rather long timespans (eg hours)
to give the admin enough time for a manual intervention.

I still like this proposal as it makes multipath interaction far cleaner.
And we can do away with path checkers completely here.

> 
>>
>> You got my hopes up for a solution in the the long explanation, then
>> you destroyed them :)
>>
>>
>> Was the reason people did not like this because of the scsi device
>> lifetime issue?
>>
>>
>> I think we still want someone to set the fast io fail tmo for users
>> when multipath is being used, because we want IO out of the queues and
>> drivers and sent to the multipath layer before dev_loss_tmo if
>> dev_loss_tmo is still going to be a lot longer. fast io fail tmo is
>> usually less than 10 or 5 and for dev_loss_tmo seems like we still
>> have user setting that to minutes.
>>
>>
>> Can't the transport layers just send two events?
>> 1. On the initial link down when the port/session is blocked.
>> 2. When there fast io fail tmos fire.
> 
> 
> So for #2, I just want a way to figure out when the transport is giving
> up on executing IO and is going to fail everything. At that time, I was
> thinking we want to mark the path failed.
> 
See above. Exactly my proposal.

> I guess if multipiath tools is going to set fast io fail, it could also
> use that as its down timer to decide when to fail the path and not have
> to send SG IO or a bsg transport command.
> 
But that's a bit of out-guessing the midlayer, no?
We're instructing the midlayer to fail all I/O at one point; so it makes
far more sense to me to have the midlayer telling us when this is going
to happen instead of trying to figure this one out ourselves.

For starters we just should send a netlink event when fast_fail_io has
fired. We could easily integrate that one in multipathd and would gain
an instant benefit from that as we can switch paths in advance.
Next step would be to implement an additional sdev state which would
return 'DID_TRANSPORT_FASTFAIL' for any 'normal' I/O; it would be
inserted between 'RUNNING' and 'CANCEL'.
Transition would be possible between 'RUNNING' and 'FASTFAIL', but
it would only be possible to transition into 'CANCEL' from 'FASTFAIL'.

Oh, and of course we have to persuade Eric Moore et al to implement
fast_fail_io into mptfc ...

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare at suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)