[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [dm-devel] Problems with multipathd



On mer, 2005-08-31 at 17:29 +0200, Simon wrote:

> 
> testhalde2 tmp # ls -lF /dev/mapper/
> total 0
> brw-------  1 root root 254,  0 Aug 31 12:20 150gb
> crw-rw----  1 root root  10, 63 Aug 31  2005 control
> 
No /dev/150gb node :) ?
/etc/udev/rules.d/20-multipath.rules should create it, see below.

> 
> testhalde2 ~ # fdisk -l /dev/mapper/150gb 
> Disk /dev/mapper/150gb: 161.0 GB, 161061273600 bytes
> 255 heads, 63 sectors/track, 19581 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Disk /dev/mapper/150gb doesn't contain a valid partition table
> 
> 
> ===> Is it possible to _use_ partitions on this device? I know that it is
>      possible to create them, but what is the device-name (/dev/...) from
>      partition 1?
> 
A little bit harder, but I guess so :
- remove the multipath map
- partition a path (/dev/sda for example)
- re-create the multipath map through '/sbin/multipath /dev/sda'

> testhalde2 rules.d # udevtest /sys/block/dm-0 block
> udevtest.c: looking at device '/block/dm-0' from subsystem 'block'
> udevtest.c: opened class_dev->name='dm-0'
> udev_rules.c: configured rule in '/etc/udev/rules.d/20-multipath.rules[3]' applie
> d, added symlink '%c'
> udev_rules.c: add symlink '150gb'
> udev_rules.c: configured rule in '/etc/udev/rules.d/20-multipath.rules[3]' applie
> d, 'dm-0' becomes '%k'
> udev_rules.c: configured rule in '/etc/udev/rules.d/50-udev.rules[63]' applied, '
> dm-0' is ignored
> 
Default udev.rules file has a directive to ignore dm-*
Something like :
KERNEL=="dm-[0-9]*", OPTIONS+="ignore_device"

/etc/udev/rules.d/20-multipath.rules is useless unless you you comment
out this rule.

> 
> testhalde2 tmp # ls -lF /dev/1*
> ls: /dev/1*: No such file or directory
> 
> 
> ===> udevtest shows that udev reads the 20-multipath.rules rule. Why doesn't
>      udev creates /dev/150gb?
> 
See above
> 
> testhalde2 tmp # multipath -l   
> 150gb (3600508b40010079d0001900000460000)
> [size=150 GB][features="0"][hwhandler="0"]
> \_ round-robin 0 [enabled][first]
>   \_ 0:0:0:1 sda  8:0     [ready ][active]
> \_ round-robin 0 [enabled]
>   \_ 1:0:0:1 sdb  8:16    [ready ][active]
> 
> 
> testhalde2 tmp # dmsetup table
> 150gb: 0 314572800 multipath 0 0 2 1 round-robin 0 1 1 8:32 1000 round-robin 0 1 
> 1 8:16 1000 
> 
> 
> testhalde2 tmp # cat devd_multipath    # multipath.dev debugging output
> ...
> 142037 - ENV_DEVPATH: ram
> 142037 - ENV_DEVNAME: /dev/rd/9
> 142046 - ENV_ACTION:  add
> 142046 - ENV_DEVPATH: sda
> 142046 - ENV_DEVNAME: /dev/sda
> 122045 - ENV_ACTION:  add
> 122045 - ENV_DEVPATH: sdb
> 122045 - ENV_DEVNAME: /dev/sdb
> 
> 
> testhalde2 tmp # fgrep dm devd_multipath 
> testhalde2 tmp # 
> 
> 
> ===> My idea from the sysfs/device-mapper/udev/multipath cooperation is the
> following: After loading the hba module qla2300 the kernel creates
> /sys/block/sda and /sys/block/sdb und executes udevsend - udevd - udev. udev
> invokes /etc/dev.d/block/multipath.dev (ADD, sda/sdb). multipath.dev executes
> multipath that creates the device-mapper table and the device-mapper device
> /sys/block/dm-0. Ok, now we have a new device (dm-0). And again: udevsend -
> udevd - udev and multipath.dev (ADD, dm-0). multipath.dev should start kpartx,
> but the debug file /tmp/devd_multipath shows nothing! So, I think kpartx will
> never started. Is this behavior ok? It seems to work without kpartx, so I don't
> understand why I need this tool.
>      
> 
kpartx is triggered for dm-* "adds" only by multipath.dev hotplug
script. *and* the script expects the node to be in /dev/
(not /dev/mapper/). 
This problem is linked to the previous one.


> 
> testhalde2 ~ # ps ax | fgrep multipathd
> 10870 pts/0    SL     0:00 multipathd
> 10871 pts/0    SL     0:00 multipathd
> 10872 pts/0    SL     0:00 multipathd
> 10875 pts/0    S+     0:00 fgrep multipathd
> 
> testhalde2 ~ # ls /var/run/multipathd.pid
> ls: /var/run/multipathd.pid: No such file or directory
> 
> ===> Does the system really need _three_ multipathd daemons and why is
>      there no pid file?
> 
I don't know default ps/nptl Gentoo choice, but it might well be the
different threads you see there. Consecutive PID numbers are a sign.

> 
> testhalde2 ~ # echo 10870 > /var/run/multipathd.pid
> testhalde2 ~ # strace -f -p 10870 >/tmp/strace_multipatd 2>&1 &
> [1] 11192
> 
Don't debug this way.
Use 'multipathd -v4' and see the log or 'strace -f multipathd'
> 
> 
> 
> Now, I disable HBA-fabric-B port on the san-switch...
> 
> testhalde2 ~ # multipath -l
> [ sleeping 35 seconds ]
> open class /sys/block/sdc failed: No such file or directory
> error calling out /sbin/scsi_id -g -u -s /block/sdc
> 150gb (3600508b40010079d0001900000460000)
> [size=150 GB][features="0"][hwhandler="0"]
> \_ round-robin 0 [enabled][first]
>   \_ 1:0:0:1 sdb  8:16    [ready ][active]
> \_ round-robin 0 [enabled]
>   \_ 0:0:0:1 sdc  8:32    [ready ][active]
> 
> testhalde2 ~ # multipath -l       # again
> 150gb (3600508b40010079d0001900000460000)
> [size=150 GB][features="0"][hwhandler="0"]
> \_ round-robin 0 [enabled][first]
>   \_ 1:0:0:1 sdb  8:16    [ready ][active]
> \_ round-robin 0 [enabled]
>   \_ 0:0:0:0      8:32    [undef ][active]
> 
Lower the timeouts in your Qlogic driver.

> testhalde2 tmp # touch /mnt/test/test    # ok
> testhalde2 tmp # rm /mnt/test/test       # ok
> 
> testhalde2 tmp # ps ax | fgrep multipathd
> 10871 pts/0    SL     0:00 multipathd
> 10872 pts/0    SL     0:00 multipathd
> 10870 pts/0    SL     0:00 multipathd
> 11534 pts/0    S+     0:00 fgrep multipathd
> 
> testhalde2 tmp # cat strace_multipatd 
> Process 10870 attached - interrupt to quit
> testhalde2 tmp # 
> 
> ===> No output in the strace-debug file from multipathd. It seems that
>      multipathd don't recognize the changes.
>  
Do the log agree with that ?

> 
> Enabling HBA-fabric-B port on the san-switch...
> 
> testhalde2 tmp # multipath -l
> 150gb (3600508b40010079d0001900000460000)
> [size=150 GB][features="0"][hwhandler="0"]
> \_ round-robin 0 [enabled][first]
>   \_ 1:0:0:1 sdb  8:16    [ready ][active]
> \_ round-robin 0 [enabled]
>   \_ 0:0:0:1 sdc  8:32    [ready ][active]
> 
> testhalde2 tmp # touch /mnt/test/test    # ok
> testhalde2 tmp # rm /mnt/test/test       # ok
> 
> 
> 
> 
> Disabling HBA-fabric-A port on the other san-switch...
> 
> testhalde2 ~ # multipath -l
> [ sleeping 35 seconds ]
> 1:0:0:1: cannot open /tmp/scsi-maj8-min16-11665: No such device or address
> error calling out /sbin/scsi_id -g -u -s /block/sdb
> error calling out /sbin/pp_balance_units 8:32
> 150gb (3600508b40010079d0001900000460000)
> [size=150 GB][features="0"][hwhandler="0"]
> \_ round-robin 0 [active][first]
>   \_ 1:0:0:1 sdb  8:16    [faulty][active]
> \_ round-robin 0 [enabled]
>   \_ 0:0:0:1 sdc  8:32    [ready ][active]
> 
> testhalde2 tmp # multipath -l             # again
> error calling out /sbin/pp_balance_units 8:32
> 150gb (3600508b40010079d0001900000460000)
> [size=150 GB][features="0"][hwhandler="0"]
> \_ round-robin 0 [active][first]
>   \_ 0:0:0:0      8:16    [undef ][active]
> \_ round-robin 0 [enabled]
>   \_ 0:0:0:1 sdc  8:32    [ready ][active]
> 
> testhalde2 tmp # touch /mnt/test/test    # ok
> testhalde2 tmp # rm /mnt/test/test       # ok
> 
> 
> ===> Why do I get the "error calling out..." error only when I disable the
>      HBA-port from _fabric-A_?
> 
Your log shows this message when disabling B too.
These are scsi_id error messages.
> 
> Enabling HBA-fabric-A port...
> 
> testhalde2 tmp # multipath -l
> 150gb (3600508b40010079d0001900000460000)
> [size=150 GB][features="0"][hwhandler="0"]
> \_ round-robin 0 [enabled][first]
>   \_ 0:0:0:1 sdc  8:32    [ready ][active]
> \_ round-robin 0 [enabled]
>   \_ 1:0:0:1 sda  8:0     [ready ][active]
> 
> testhalde2 tmp # touch /mnt/test/test    # ok
> testhalde2 tmp # rm /mnt/test/test       # ok
> 
> 

> SUMMARY:
> ========
> 
> The failover mechanism seems to work, but it's very very slow (>= 35 sec).
> I am sure that the host will die when I have a lot of I/O's in this moment.
> The documentation says that multipathd "is in charge of checking the paths
> in case they come up or down" and multipathd seems to do nothing... I think
> that is the problem... What do you mean?
> 
Hope the previous comments clarifies a bit.
Also know the 0.4.5 snapshots are largely better suited to the task.
Consider upgrading.
And consider updating the wiki FAQ with the response you found to be
enlightening :/

Regards,
-- 
christophe varoqui <christophe varoqui free fr>



[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]