[dm-devel] multipathd segfault and error calling out
John A. Sullivan III
jsullivan at opensourcedevel.com
Thu Feb 26 02:07:44 UTC 2009
Hello, all. I am running on kernel 2.6.27 on CentOS 5.2 with VServer
and device-mapper-multipath-0.4.7-17.el5. I have a custom
mpath_prio_ssi script which takes the device name (e.g., sdaa), pulls
out the path from /etc/disk/by-path and then echos a priority based upon
a lookup table. It works perfectly fine from the command line.
multipath -ll shows the priorities assigned perfectly and exactly the
right paths are active.
However, when I start multipathd, it all goes down the tubes. The paths
disappear and /var/log/messages is filled with:
Feb 25 20:50:17 vd01 multipathd: error calling out /usr/local/sbin/mpath_prio_ssi sdh
Feb 25 20:50:17 vd01 multipathd: error calling out /usr/local/sbin/mpath_prio_ssi sdi
Feb 25 20:50:17 vd01 multipathd: error calling out /usr/local/sbin/mpath_prio_ssi sdj
Feb 25 20:50:17 vd01 multipathd: error calling out /usr/local/sbin/mpath_prio_ssi sdc
Feb 25 20:50:17 vd01 multipathd: error calling out /usr/local/sbin/mpath_prio_ssi sdd
Feb 25 20:50:17 vd01 multipathd: error calling out /usr/local/sbin/mpath_prio_ssi sde
Feb 25 20:50:17 vd01 multipathd: error calling out /usr/local/sbin/mpath_prio_ssi sdf
Feb 25 20:50:17 vd01 multipathd: error calling out /usr/local/sbin/mpath_prio_ssi sdg
Feb 25 20:50:17 vd01 multipathd: error calling out /usr/local/sbin/mpath_prio_ssi sdh
Feb 25 20:50:17 vd01 multipathd: error calling out /usr/local/sbin/mpath_prio_ssi sdi
Feb 25 20:50:17 vd01 multipathd: error calling out /usr/local/sbin/mpath_prio_ssi sdj
Feb 25 20:50:17 vd01 multipathd: error calling out /usr/local/sbin/mpath_prio_ssi sdc
Feb 25 20:50:17 vd01 multipathd: error calling out /usr/local/sbin/mpath_prio_ssi sdd
Feb 25 20:50:17 vd01 multipathd: error calling out /usr/local/sbin/mpath_prio_ssi sde
Feb 25 20:50:17 vd01 multipathd: error calling out /usr/local/sbin/mpath_prio_ssi sdf
Feb 25 20:50:17 vd01 multipathd: error calling out /usr/local/sbin/mpath_prio_ssi sdg
Feb 25 20:50:17 vd01 multipathd: error calling out /usr/local/sbin/mpath_prio_ssi sdh
Feb 25 20:50:17 vd01 multipathd: error calling out /usr/local/sbin/mpath_prio_ssi sdi
Feb 25 20:50:17 vd01 multipathd: error calling out /usr/local/sbin/mpath_prio_ssi sdj
Feb 25 20:50:17 vd01 multipathd: error calling out /usr/local/sbin/mpath_prio_ssi sdk
Feb 25 20:50:17 vd01 multipathd: error calling out /usr/local/sbin/mpath_prio_ssi sdl
Feb 25 20:50:17 vd01 multipathd: error calling out /usr/local/sbin/mpath_prio_ssi sdm
Feb 25 20:50:17 vd01 multipathd: error calling out /usr/local/sbin/mpath_prio_ssi sdn
Feb 25 20:50:17 vd01 multipathd: error calling out /usr/local/sbin/mpath_prio_ssi sdo
Feb 25 20:50:17 vd01 multipathd: error calling out /usr/local/sbin/mpath_prio_ssi sdp
Feb 25 20:50:17 vd01 multipathd: error calling out /usr/local/sbin/mpath_prio_ssi sdq
Feb 25 20:50:17 vd01 multipathd: error calling out /usr/local/sbin/mpath_prio_ssi sdr
Feb 25 20:50:18 vd01 multipathd: error calling out /usr/local/sbin/mpath_prio_ssi sdk
Feb 25 20:50:18 vd01 multipathd: error calling out /usr/local/sbin/mpath_prio_ssi sdl
Feb 25 20:50:18 vd01 multipathd: error calling out /usr/local/sbin/mpath_prio_ssi sdm
The callout does exist, is executable, and works with multipath. Based
upon some email threads, I tried moving it to / and /sbin. No change.
When multipathd starts I get:
Feb 25 20:50:17 vd01 multipathd: cannot open /sbin/dasd_id : No such file or directory
Feb 25 20:50:17 vd01 multipathd: [copy.c] cannot open /sbin/dasd_id
Feb 25 20:50:17 vd01 multipathd: cannot copy /sbin/dasd_id in ramfs : No such file or directory
Feb 25 20:50:17 vd01 multipathd: error calling out /usr/local/sbin/mpath_prio_ssi sda
and when it shuts stops I get:
Feb 25 20:43:17 vd01 multipathd: error calling out /usr/local/bin/mpath_prio_ssi sdj
Feb 25 20:43:17 vd01 multipathd: isdb: stop event checker thread
Feb 25 20:43:17 vd01 multipathd: isdc: stop event checker thread
Feb 25 20:43:17 vd01 multipathd: isda: stop event checker thread
Feb 25 20:43:17 vd01 multipathd: isdd: stop event checker thread
Feb 25 20:43:17 vd01 kernel: multipathd[28942]: segfault at a ip 0000003e8a26fa3d sp 00007fff47e946d0 error 4 in libc-2.5.so[3e8a200000+14a000]
Here are the uncommented portions of my /etc/multipath.conf:
blacklist {
# devnode "*"
# sdb
wwid SATA_ST3250310NS_9SF0L234
#sda
wwid SATA_ST3250310NS_9SF0LVSR
}
defaults {
failback 5
path_grouping_policy failover
path_checker tur
prio_callout "/usr/local/sbin/mpath_prio_ssi %n"
}
multipaths {
multipath {
wwid 3600144f049a4acec00003048c6912c00
alias isda
}
multipath {
wwid 3600144f049a4acf500003048c6912c00
alias isdb
}
multipath {
wwid 3600144f049a4ace600003048c6912c00
alias isdc
}
multipath {
wwid 3600144f049a448a700003048c6912c00
alias isdd
}
}
devices {
device {
vendor "SUN"
product "SOLARIS"
getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
features "0"
hardware_handler "0"
path_grouping_policy failover
rr_weight uniform
rr_min_io 1000
path_checker readsector0
}
}
Here is the callout script:
#!/bin/bash
# if not passed any device name, return a priority of 0
if [ -z "${1}" ];then
echo 0
exit
fi
ENTRY="$(ls -l1 /dev/disk/by-path | grep /${1}'$')"
DEV="${ENTRY##* ip-}"
if [ "$DEV" = "${ENTRY}" ];then #This is not an iSCSI device
echo 0
exit
fi
DEV="${DEV%% ->*}"
echo $(grep ${DEV} /usr/local/etc/iscsi.list | cut -f2)
and here is iscsi.list:
172.x.x.158:3260-iscsi-iqn.1986-03.com.sun:02:04e2454b-e60c-cfaf-d8b9-923b9fed0020 5
172.x.x.158:3260-iscsi-iqn.1986-03.com.sun:02:8af2554f-13f6-c0e1-8343-c457e4802cb9 99
172.x.x.158:3260-iscsi-iqn.1986-03.com.sun:02:b8a8e5cd-4501-c8f7-be51-ad7fcb31f782 5
172.30.13.158:3260-iscsi-iqn.1986-03.com.sun:02:c3420146-eecc-c364-dc6e-a3914db5a639 24
172.x.x.174:3260-iscsi-iqn.1986-03.com.sun:02:04e2454b-e60c-cfaf-d8b9-923b9fed0020 1
172.x.x.174:3260-iscsi-iqn.1986-03.com.sun:02:8af2554f-13f6-c0e1-8343-c457e4802cb9 2
172.x.x.174:3260-iscsi-iqn.1986-03.com.sun:02:b8a8e5cd-4501-c8f7-be51-ad7fcb31f782 1
172.x.x.174:3260-iscsi-iqn.1986-03.com.sun:02:c3420146-eecc-c364-dc6e-a3914db5a639 49
172.x.x.190:3260-iscsi-iqn.1986-03.com.sun:02:04e2454b-e60c-cfaf-d8b9-923b9fed0020 0
172.x.x.190:3260-iscsi-iqn.1986-03.com.sun:02:8af2554f-13f6-c0e1-8343-c457e4802cb9 49
172.x.x.190:3260-iscsi-iqn.1986-03.com.sun:02:b8a8e5cd-4501-c8f7-be51-ad7fcb31f782 0
172.x.x.190:3260-iscsi-iqn.1986-03.com.sun:02:c3420146-eecc-c364-dc6e-a3914db5a639 2
172.x.x.206:3260-iscsi-iqn.1986-03.com.sun:02:04e2454b-e60c-cfaf-d8b9-923b9fed0020 11
172.x.x.206:3260-iscsi-iqn.1986-03.com.sun:02:8af2554f-13f6-c0e1-8343-c457e4802cb9 24
172.x.x.206:3260-iscsi-iqn.1986-03.com.sun:02:b8a8e5cd-4501-c8f7-be51-ad7fcb31f782 11
172.x.x.206:3260-iscsi-iqn.1986-03.com.sun:02:c3420146-eecc-c364-dc6e-a3914db5a639 99
172.x.x.30:3260-iscsi-iqn.1986-03.com.sun:02:04e2454b-e60c-cfaf-d8b9-923b9fed0020 24
172.x.x.30:3260-iscsi-iqn.1986-03.com.sun:02:8af2554f-13f6-c0e1-8343-c457e4802cb9 11
172.x.x.30:3260-iscsi-iqn.1986-03.com.sun:02:b8a8e5cd-4501-c8f7-be51-ad7fcb31f782 99
172.x.x.30:3260-iscsi-iqn.1986-03.com.sun:02:c3420146-eecc-c364-dc6e-a3914db5a639 5
172.x.x.46:3260-iscsi-iqn.1986-03.com.sun:02:04e2454b-e60c-cfaf-d8b9-923b9fed0020 49
172.x.x.46:3260-iscsi-iqn.1986-03.com.sun:02:8af2554f-13f6-c0e1-8343-c457e4802cb9 0
172.x.x.46:3260-iscsi-iqn.1986-03.com.sun:02:b8a8e5cd-4501-c8f7-be51-ad7fcb31f782 2
172.x.x.46:3260-iscsi-iqn.1986-03.com.sun:02:c3420146-eecc-c364-dc6e-a3914db5a639 1
172.x.x.62:3260-iscsi-iqn.1986-03.com.sun:02:04e2454b-e60c-cfaf-d8b9-923b9fed0020 2
172.x.x.62:3260-iscsi-iqn.1986-03.com.sun:02:8af2554f-13f6-c0e1-8343-c457e4802cb9 1
172.x.x.62:3260-iscsi-iqn.1986-03.com.sun:02:b8a8e5cd-4501-c8f7-be51-ad7fcb31f782 49
172.x.x.62:3260-iscsi-iqn.1986-03.com.sun:02:c3420146-eecc-c364-dc6e-a3914db5a639 0
172.x.x.78:3260-iscsi-iqn.1986-03.com.sun:02:04e2454b-e60c-cfaf-d8b9-923b9fed0020 99
172.x.x.78:3260-iscsi-iqn.1986-03.com.sun:02:8af2554f-13f6-c0e1-8343-c457e4802cb9 5
172.x.x.78:3260-iscsi-iqn.1986-03.com.sun:02:b8a8e5cd-4501-c8f7-be51-ad7fcb31f782 24
172.x.x.78:3260-iscsi-iqn.1986-03.com.sun:02:c3420146-eecc-c364-dc6e-a3914db5a639 11
What have I misconfigured? How do I make this work properly? Thanks -
John
--
John A. Sullivan III
Open Source Development Corporation
+1 207-985-7880
jsullivan at opensourcedevel.com
http://www.spiritualoutreach.com
Making Christianity intelligible to secular society
More information about the dm-devel
mailing list