[dm-devel] Bug in multipath detecting devices

[Jeff Gibson]

I believe I have found a bug in multipath when it detects devices.  But
first off here is my config:

Redhat 4 update 3 x86-64
1 internal LSI Logic RAID controller (mirrored disks on device sda)
2 dual-port Qlogic 2342 Fibre-channel HBA.  Port 1 grabs disks sdb-sds,
Port 2 grabs sdt-sdak.
2 McData 4500 Fibre switches
1 Hitachi DF600F SAN.  Dual RAID controller with 2 Fibre ports per
controller.

I have each port from the qlogic card going to a separate switch.  Each
switch then has 2 links back to the SAN, 1 going to each port on each
raid controller.  A rough text drawing would look like this:

			_______________
		     |	HBA	   |
			---1--------2--
			   |        |
		___________      ___________
	     |   SW-1    |    |    SW-2   |
		-|-------|-      -|-------|-
		 \       |  ______|_______/
		  \______|_/____  |      
			   | |    \ |          
			---|-|SAN-|-|---
		     |        |       |
		     | ctlr1  | ctlr2 |
		     |        |       |
			----------------

The result is that each disk/lun on the SAN shows up to the OS 4 times.
Using pp_hds_modular with multipath I can load balance to both ports on
the same controller on the SAN (there is a performance loss on Hitachi
systems of you load balance between controllers).  The downside to this
is that your end up using a lot of disk devices in linux.  For example,
I have 9 luns shared out on the SAN, but the OS sees 36 disks.  Using
multipath I can bring that back down to 9, but here's where we run into
the bug.		

It seems that if you have more than 26 disks/luns it will detect sdaa
before it detects sdb (sda is the onboard raid & is blacklisted).  The
result is that the major/minor numbers in /dev/mapper get out of order
with the disks presented to the system.  For example:

# multipath -v2
create: 1HITACHI_D60085320025
[size=50 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [prio=2]
 \_ 2:0:1:7 sdaj 66:48  [ready]
 \_ 1:0:0:7 sdi  8:128  [ready]
\_ round-robin 0
 \_ 2:0:0:7 sdaa 65:160 [ready]
 \_ 1:0:1:7 sdr  65:16  [ready]

create: 1HITACHI_D60085320018
[size=200 MB][features="0"][hwhandler="0"]
\_ round-robin 0 [prio=2]
 \_ 1:0:1:0 sdk  8:160  [ready]
 \_ 2:0:0:0 sdt  65:48  [ready]
\_ round-robin 0
 \_ 2:0:1:0 sdac 65:192 [ready]
 \_ 1:0:0:0 sdb  8:16   [ready]


# ls -l /dev/mapper
total 0
brw-rw----  1 root disk 253,  2 Jul 31 16:29 1HITACHI_D60085320018
brw-rw----  1 root disk 253,  3 Jul 31 16:29 1HITACHI_D60085320019
brw-rw----  1 root disk 253,  4 Jul 31 16:29 1HITACHI_D60085320020
brw-rw----  1 root disk 253,  5 Jul 31 16:29 1HITACHI_D60085320021
brw-rw----  1 root disk 253,  6 Jul 31 16:29 1HITACHI_D60085320022
brw-rw----  1 root disk 253,  7 Jul 31 16:29 1HITACHI_D60085320023
brw-rw----  1 root disk 253,  8 Jul 31 16:29 1HITACHI_D60085320024
brw-rw----  1 root disk 253,  0 Jul 31 16:29 1HITACHI_D60085320025
brw-rw----  1 root disk 253,  1 Jul 31 16:29 1HITACHI_D60085320026
crw-------  1 root root  10, 63 Jul 31 16:01 control

In the above 1HITACHI_D60085320018 (the container of sdb) *should* be at
253,0.  Instead sdaa is.

The major problem with this is that if I add another disk/lun in the
future sdaa will change to another disk.  This is because sdaa is on the
second port of the qlogic card and gets detected as a later device than
before.  When multipath comes along it creates the device in /dev/mapper
that sdaa is a member of before sdb's device is created.  As a result
the multipath devices aren't in the same sequnce as the devices that the
OS detects (sdb, sdc, etc).  It would be better if sdb was always
detected before sdaa.  However, I don't see a way to do that in
/etc/multipath.conf without blacklisting all of the sda* drives.
Another option would be to have a user-configurable scan order, which I
don't think exists (correct me if I'm wrong).  

Anyway, I hope this information helps.  Let me know if you have any
questions of if I'm missing something.

Thanks,
JG