[dm-devel] failed multipaths are never re-validated

[Mike Christie]

Guy Coates wrote:

> Hi,
> 
> I have a dual fabric / dual contoller setup which I've got partially to
> work with the multipath device mapper personality. I can create a
> multipath device and I can get multipathd to remove failed paths to the
> storage.  However, once failed paths become active again, multipathd never
> adds the paths back into the devicemap.
> 
> 
> 
> The multipath tools correctly identify the four paths to the storage and
> created the appropraite map.
> 
> 
> ia64g:~# multipath -v -d
> # all paths :
> 600508b40001045d0001c00006de0000 (2 0 0 1) /dev/sdc [HSV110 (C)COMPAQ]
> 600508b40001045d0001c00006de0000 (2 0 1 1) /dev/sdd [HSV110 (C)COMPAQ]
> 600508b40001045d0001c00006de0000 (3 0 0 1) /dev/sde [HSV110 (C)COMPAQ]
> 600508b40001045d0001c00006de0000 (3 0 1 1) /dev/sdf [HSV110 (C)COMPAQ]
> # all multipaths :
> 600508b40001045d0001c00006de0000
>  \_(2 0 0 1) /dev/sdc [HSV110 (C)COMPAQ]
>  \_(2 0 1 1) /dev/sdd [HSV110 (C)COMPAQ]
>  \_(3 0 0 1) /dev/sde [HSV110 (C)COMPAQ]
>  \_(3 0 1 1) /dev/sdf [HSV110 (C)COMPAQ]
> 
> ia64g:~# multipath
> U:600508b40001045d0001c00006de0000:0 209715200 multipath 2 round-robin 2 0
> /dev/sdc /dev/sde round-robin 2 0 /dev/sdd /dev/sdf
> 
> ia64g:~# dmsetup  status
> 600508b40001045d0001c00006de0000: 0 209715200 multipath 2 2 0 8:32 A 1 8:64 A 1 2 0 8:48 A 1 8:80 A 1
> 
> 
> I can format the multipath device and IO  does indeed go down both SAN
> fabrics.
> 
> If I pull one of the fabrics, multipathd correctly reconfigures the
> device, and IO continues down a single fabric.
> 
> <syslog>
> May 28 15:16:46 ia64g kernel: qla2300 0000:40:01.1: LOOP DOWN detected.
> May 28 15:16:54 ia64g kernel: SCSI error : <3 0 0 1> return code = 0x10000
> May 28 15:16:54 ia64g kernel: end_request: I/O error, dev sde, sector 33967656
> 
> May 28 15:15:33 ia64g /usr/bin/multipathd: [event thread] refresh failpaths list
> May 28 15:15:33 ia64g /usr/bin/multipathd: [event thread] start up event loops
> May 28 15:16:59 ia64g /usr/bin/multipathd: [checker thread] reconfigure 600508b40001045d0001c00006de
> 
> 
> ia64g:~# dmsetup  status
> 600508b40001045d0001c00006de0000: 0 209715200 multipath 2 2 0 8:32 A 1 8:64 F 0 2 0 8:48 A 1 8:80 A 1
> 
> ia64g:~# multipath -d -v
> # all paths :
> 600508b40001045d0001c00006de0000 (2 0 0 1) /dev/sdc [HSV110 (C)COMPAQ]
> 600508b40001045d0001c00006de0000 (2 0 1 1) /dev/sdd [HSV110 (C)COMPAQ]
> 00000000000000000000000000000000 (3 0 0 1) /dev/sde [HSV110 (C)COMPAQ]
> 00000000000000000000000000000000 (3 0 1 1) /dev/sdf [HSV110 (C)COMPAQ]
> # all multipaths :
> 600508b40001045d0001c00006de0000
>  \_(2 0 0 1) /dev/sdc [HSV110 (C)COMPAQ]
>  \_(2 0 1 1) /dev/sdd [HSV110 (C)COMPAQ]
> 00000000000000000000000000000000
>  \_(3 0 0 1) /dev/sde [HSV110 (C)COMPAQ]
>  \_(3 0 1 1) /dev/sdf [HSV110 (C)COMPAQ]
> 
> 
> When I reconnect the seconds fabric, the newly active path does not get
> automatically added back into the device map.
> 
> May 28 15:19:02 ia64g kernel: qla2300 0000:40:01.1: LIP reset occured (f7f7).
> May 28 15:19:02 ia64g kernel: qla2300 0000:40:01.1: LOOP UP detected (2 Gbps).
> 
> ia64g:~# dmsetup  status
> 600508b40001045d0001c00006de0000: 0 209715200 multipath 2 2 0 8:32 A 1 8:64 F 0 2 0 8:48 A 1 8:80 A 1

At this point, has multipathd retested the paths, or are you hoping that 
the loop up event will trigger the multipathd? I do not think the latter 
is possible today (see below), but once the multipathd tests the paths 
they should readded to the multipath device. Does the retesting part not 
  work too?

> 
> I can force a reconfiguration manually by issuing the "multipath" command
> again.
> 
> ia64g:~# multipath
> U:600508b40001045d0001c00006de0000:0 209715200 multipath 2 round-robin 2 0
> /dev/sdc /dev/sde round-robin 2 0 /dev/sdd /dev/sdf
> 
> ia64g:~# dmsetup status
> 600508b40001045d0001c00006de0000: 0 209715200 multipath 2 2 0 8:32 A 1
> 8:64 A 1 2 0 8:48 A 1 8:80 A 1
> 
> Does anyone have any suggestions as to how to get multipathd to
> automatically act on the LOOP UP event?

Hey they are talking about that on linux-scsi right now. 
http://marc.theaimsgroup.com/?l=linux-scsi&m=108564273619016&w=2
http://marc.theaimsgroup.com/?t=108573124200003&r=1&w=2

> 
> I running on kernel 2.6.6 on IA64 with 2.6.6-udm1 device mapper patchset
> with v0.2.0 of the multipath tools. The HBA is using QLogic Fibre Channel
> HBA Driver: 8.00.00b11-k and the storage targets are dual Compaq HSV110s.
> 
> 
> Cheers,
> 
> Guy Coates
>