[dm-devel] udevsettle command takes long time to settle in RHEL 5.9 & 5.10

[Benjamin Marzinski]

On Wed, Mar 12, 2014 at 04:56:11AM +0530, Adarsh wrote:
>    Hi All,
>    Can you please help me with an issue which I am facing in RHEL 5.9 & 5.10
>    setups.
>    udevsettle command takes quite long time (60-150 seconds) to return
>    sometimes.
>    This is mostly after creating a LUN & issuing a "rescan-scsi-bus.sh"
>    Please note that only around 10-20 LUNs are present while this issue is
>    hit.
>    Looked into the udev logs & looks like the culprit is the following line
>    in "/etc/udev/rules.d/40-multipath.rules":
>    � � �RUN+="/bin/bash -c '/sbin/mpath_wait /dev/mapper/%c; /sbin/kpartx -a
>    -p p /dev/mapper/%c'"
> 
>    The rule is getting called multiple times for the same device & hence
>    udevsettle keeps waiting for all these to finish
> 
>    Mar 10 13:47:00 x336-207-55 udevd-event[8013]: run_program: '/bin/bash -c
>    '/sbin/mpath_wait /dev/mapper/360a98000316b61396a2b3946424b6f2d;
>    /sbin/kpartx -a -p p /dev/mapper/360a98000316b61396a2b3946424b6f2d'
> 
>    Mar 10 13:47:04 x336-207-55 udevd-event[8033]: run_program: '/bin/bash -c
>    '/sbin/mpath_wait /dev/mapper/360a98000316b61396a2b3946424b6f2d;
>    /sbin/kpartx -a -p p /dev/mapper/360a98000316b61396a2b3946424b6f2d'
> 
>    Mar 10 13:47:04 x336-207-55 udevd-event[8107]: run_program: '/bin/bash -c
>    '/sbin/mpath_wait /dev/mapper/360a98000316b61396a2b3946424b6f2d;
>    /sbin/kpartx -a -p p /dev/mapper/360a98000316b61396a2b3946424b6f2d'
> 
>    Mar 10 13:47:10 x336-207-55 udevd-event[8212]: run_program: '/bin/bash -c
>    '/sbin/mpath_wait /dev/mapper/360a98000316b61396a2b3946424b6f2d;
>    /sbin/kpartx -a -p p /dev/mapper/360a98000316b61396a2b3946424b6f2d'
> 
>    Hence, I commented out the line from "multipath.rules" & it started
>    working fine:
>    � ��RUN+="/bin/bash -c '/sbin/mpath_wait /dev/mapper/%c; /sbin/kpartx -a
>    -p p /dev/mapper/%c'"
>    Couple of queries:
>    1. Please let me know if there are any side effects for commenting out the
>    above mentioned line.

This is a RHEL5 only issue.  In RHEL5, udev doesn't handle device
creation, so there is no guarantee that you will be able to operate on
the device when you get the uevent.  mpath_wait waits until the device
is actually available before continuing with the multipath.rules.
Removing this may cause devices to not be correctly symlinked or have
missing partition devices.

>    � � Ben mentions that block device addition is now taken care of by
>    NETLINK events as long as multipath is running & no need for udev to fire
>    off multipath:�[1]https://bugzilla.redhat.com/show_bug.cgi?id=460301
> 
>  2. Please let me know if this is a known issue. I observe it only in RHEL 5.9 and 5.10 setups.

RHEL6 relies on udev to do the device creation, and so doesn't have this
issue at all.

>  Any pointers is highly appreciated.
> 
>    Setup:
>    ======
>    Red Hat Enterprise Linux Server release 5.10 (Tikanga)
>    Kernel \r on an \m
>    Multipath.conf:
>    ===========
>    defaults {
>    � � � � user_friendly_names � � � � � � no
>    � � � � queue_without_daemon � � � � � �no
>    � � � � flush_on_last_del � � � � � � � yes
>    � � � � max_fds � � � � � � � � � � � � max
>    � � � � pg_prio_calc � � � � � � � � � �avg
>    }
>    blacklist {
>    � � � � wwid � � � � � � � � � � � � � �SIBM-ESXSMAW3073NC_FDAR9P6402NE0
>    � � � � wwid � � � � � � � � � � � � � �SIBM-ESXSMAW3073NC_FDAR9P6402PP2
>    � � � � devnode � � � � � � � � � � � � "^cciss.*"
>    }
>    devices {
>    � � � � device {
>    � � � � � � � � vendor � � � � � � � � � �"NETAPP"
>    � � � � � � � � product � � � � � � � � � "LUN"
>    � � � � � � � � features � � � � � � � � �"3 queue_if_no_path
>    pg_init_retries 50"
>    � � � � � � � � path_grouping_policy � �group_by_prio
>    � � � � � � � � prio_callout � � � � � � "/sbin/mpath_prio_alua /dev/%n"
>    � � � � � � � � path_checker � � � � � �tur
>    � � � � � � � � failback � � � � � � � � � � �immediate
>    � � � � � � � � hardware_handler � � "1 alua"
>    � � � � � � � � rr_weight � � � � � � � � � �uniform
>    � � � � � � � � rr_min_io � � � � � � � � � �128
>    � � � � � � � � getuid_callout � � � � �"/sbin/scsi_id -g -u -s /block/%n"
>    � � � �}
>    }
>    regards,
>    Adarsh.
> 
> References
> 
>    Visible links
>    1. https://bugzilla.redhat.com/show_bug.cgi?id=460301