[dm-devel] Multipath and HSG80 phase 2

Tue Dec 7 17:34:54 UTC 2004

I'm here again, i did not resolve the problems about multipath setupping.
I tried again from scratch, here was my road map:

* fresh slackware 10.0 installation
* downloaded kernel 2.6.10-rc1 and patched with 2.6.10-rc1-udm2
* removed nostart from kernel source in order to spin up disk of hsg80
* compiled with hotplug, generic scsi, dm and multipath support
* deleted standard udev 0.26 slackware packages and compiled udev 0.46
(adjusted some script in orther to have back /dev/pts etc., and to create
/dev/mapper/control)
* deleted standard slackware hotplug and installed hotplug-2004_09_23
* installed sysfsutils 1.1.0 and scsi_id 0.7
* downloaded multipath-tools 0.3.9  and make BUILD=klibc,  while compiling
main.c in multipathd directory i get a livdevmapper.h no such file or
directory, is it no using dropped lib mapper?
* i undefined CLONE in order to use fork (?)
* compiled device-mapper.1.00.19 and now multipathd is compiled fine, the
tools are all installed.

reboot the system now the logs:

Dec  7 17:35:58 m3 kernel: device-mapper: 4.3.0-ioctl (2004-09-30)
initialised: dm-devel at redhat.com
Dec  7 17:35:58 m3 kernel: device-mapper: dm-multipath version 1.0.3 loaded
Dec  7 17:35:58 m3 kernel: device-mapper: dm-round-robin version 0.2.1
loaded

after qla2300 loading:

sda: Spinning up disk.....ready
Dec  7 17:36:02 m3 kernel: SCSI device sda: 71114623 512-byte hdwr sectors
(36411 MB)
Dec  7 17:36:02 m3 kernel: SCSI device sda: drive cache: write through
Dec  7 17:36:02 m3 kernel:  sda: sda1

Dec  7 17:36:04 m3 kernel: sdb: Spinning up disk.....ready
Dec  7 17:36:04 m3 kernel: SCSI device sdb: 71114623 512-byte hdwr sectors
(36411 MB)
Dec  7 17:36:04 m3 kernel: SCSI device sdb: drive cache: write through
Dec  7 17:36:04 m3 kernel:  sdb: sdb1

Dec  7 17:36:05 m3 udev[1399]: creating device node '/dev/sg0'
Dec  7 17:36:05 m3 udev[1404]: creating device node '/dev/sda1'
Dec  7 17:36:05 m3 udev[1403]: creating device node '/dev/sda'
Dec  7 17:36:05 m3 udev[1407]: creating device node '/dev/sg1'
Dec  7 17:36:07 m3 udev[1441]: creating device node '/dev/sdb1'
Dec  7 17:36:07 m3 udev[1438]: creating device node '/dev/sg2'
Dec  7 17:36:07 m3 udev[1440]: creating device node '/dev/sdb'
Dec  7 17:36:07 m3 udev[1443]: creating device node '/dev/sg3'
Dec  7 17:36:07 m3 udev[1444]: configured rule in
'/etc/udev/rules.d/50-udev.rules' at line 12 applied, added symlink '%c'
Dec  7 17:36:07 m3 udev[1444]: configured rule in
'/etc/udev/rules.d/50-udev.rules' at line 12 applied, 'dm-0' becomes '%k'
Dec  7 17:36:07 m3 udev[1444]: creating device node '/dev/dm-0'
Dec  7 17:36:08 m3 udev[1475]: configured rule in
'/etc/udev/rules.d/50-udev.rules' at line 12 applied, added symlink '%c'
Dec  7 17:36:08 m3 udev[1475]: configured rule in
'/etc/udev/rules.d/50-udev.rules' at line 12 applied, 'dm-1' becomes '%k'
Dec  7 17:36:08 m3 udev[1475]: creating device node '/dev/dm-1'

and after starting multipathd
Dec  7 18:04:59 m3 multipathd: --------start up--------
Dec  7 18:04:59 m3 multipathd: read /etc/multipath.conf
Dec  7 18:04:59 m3 multipathd: add /sbin/multipath to binvec
Dec  7 18:04:59 m3 multipathd: add /sbin/scsi_id to binvec
Dec  7 18:04:59 m3 multipathd: add /bin/false to binvec
Dec  7 18:04:59 m3 multipathd: --------start up--------
Dec  7 18:04:59 m3 multipathd: already running : out
Dec  7 18:04:59 m3 multipathd: start DM events thread
Dec  7 18:04:59 m3 multipathd: initial reconfigure multipath maps
Dec  7 18:04:59 m3 multipathd: path checkers start up
Dec  7 18:04:59 m3 multipathd: checking paths
Dec  7 18:04:59 m3 multipathd: refresh devmaps list
Dec  7 18:04:59 m3 multipathd: devmap disk1p1 :
Dec  7 18:04:59 m3 multipathd: \_ 0 1975932 linear
Dec  7 18:04:59 m3 multipathd:    skip non multipath target
Dec  7 18:04:59 m3 multipathd: devmap disk1 :
Dec  7 18:04:59 m3 multipathd: \_ 0 71114623 multipath
Dec  7 18:04:59 m3 multipathd: refresh failpaths list
Dec  7 18:04:59 m3 multipathd: cciss!c0d0 blacklisted
Dec  7 18:04:59 m3 multipathd: dm-0 blacklisted
Dec  7 18:04:59 m3 multipathd: dm-1 blacklisted
Dec  7 18:04:59 m3 multipathd: fd0 blacklisted
Dec  7 18:04:59 m3 multipathd: hda blacklisted
Dec  7 18:04:59 m3 multipathd: loop0 blacklisted
Dec  7 18:04:59 m3 multipathd: loop1 blacklisted
Dec  7 18:04:59 m3 multipathd: loop2 blacklisted
Dec  7 18:04:59 m3 multipathd: loop3 blacklisted
Dec  7 18:04:59 m3 multipathd: loop4 blacklisted
Dec  7 18:04:59 m3 multipathd: loop5 blacklisted
Dec  7 18:04:59 m3 multipathd: loop6 blacklisted
Dec  7 18:04:59 m3 multipathd: loop7 blacklisted
Dec  7 18:04:59 m3 multipathd: set tur path checker for sda
Dec  7 18:04:59 m3 multipathd: path checker startup : 8:0
Dec  7 18:04:59 m3 multipathd: set tur path checker for sdb
Dec  7 18:04:59 m3 multipathd: path checker startup : 8:16
Dec  7 18:04:59 m3 multipathd: start up event loops
Dec  7 18:04:59 m3 multipathd: event checker startup : disk1
Dec  7 18:04:59 m3 multipathd: waiter->event_nr = 0
Dec  7 18:05:09 m3 multipathd: checking paths

now in /dev i have:

0 lrwxrwxrwx  1 root root      4 2004-12-07 18:03 disk1 -> dm-0
0 lrwxrwxrwx  1 root root      4 2004-12-07 18:03 disk1p1 -> dm-1
0 brw-------  1 root root 254, 0 2004-12-07 18:03 dm-0
0 brw-------  1 root root 254, 1 2004-12-07 18:03 dm-1

so i mount /dev/disk1p1 /mnt

every few seconds in the log:

Dec  7 17:49:13 m3 multipathd: checking paths

cp /dev/urandom /mnt/foo &

on the hsg80 console i do a "restart other" to simulate a path down so:
cp: writing `pippo': Input/output error
Dec  7 18:06:59 m3 multipathd: checking paths
Dec  7 18:07:19 m3 last message repeated 2 times
Dec  7 18:07:24 m3 kernel: Device sdb not ready.
Dec  7 18:07:24 m3 kernel: end_request: I/O error, dev sdb, sector 5807
Dec  7 18:07:24 m3 multipathd: /sbin/multipath -v 0 -S disk1
Dec  7 18:07:24 m3 multipathd: devmap event on disk1
Dec  7 18:07:24 m3 multipathd: disk1 : reconfigure multipath map
Dec  7 18:07:24 m3 kernel: Device sda not ready.
Dec  7 18:07:24 m3 kernel: end_request: I/O error, dev sda, sector 5807
Dec  7 18:07:24 m3 kernel: Buffer I/O error on device dm-1, logical block
719
Dec  7 18:07:24 m3 kernel: lost page write due to I/O error on dm-1
Dec  7 18:07:24 m3 kernel: Buffer I/O error on device dm-1, logical block
720
Dec  7 18:07:24 m3 kernel: lost page write due to I/O error on dm-1
Dec  7 18:07:24 m3 kernel: Buffer I/O error on device dm-1, logical block
721
Dec  7 18:07:24 m3 kernel: lost page write due to I/O error on dm-1
Dec  7 18:07:24 m3 multipathd: refresh devmaps list
Dec  7 18:07:24 m3 kernel: Buffer I/O error on device dm-1, logical block
722
Dec  7 18:07:24 m3 multipathd: devmap disk1p1 :
Dec  7 18:07:24 m3 kernel: lost page write due to I/O error on dm-1
Dec  7 18:07:25 m3 multipathd: \_ 0 1975932 linear
Dec  7 18:07:25 m3 kernel: Buffer I/O error on device dm-1, logical block
723
Dec  7 18:07:25 m3 multipathd:    skip non multipath target
Dec  7 18:07:25 m3 kernel: lost page write due to I/O error on dm-1
Dec  7 18:07:25 m3 multipathd: devmap disk1 :
Dec  7 18:07:25 m3 kernel: Buffer I/O error on device dm-1, logical block
724
Dec  7 18:07:25 m3 multipathd: \_ 0 71114623 multipath
Dec  7 18:07:25 m3 kernel: lost page write due to I/O error on dm-1
Dec  7 18:07:25 m3 multipathd: refresh failpaths list
Dec  7 18:07:25 m3 kernel: Buffer I/O error on device dm-1, logical block
725
Dec  7 18:07:25 m3 multipathd: cciss!c0d0 blacklisted
Dec  7 18:07:25 m3 kernel: lost page write due to I/O error on dm-1
Dec  7 18:07:25 m3 multipathd: dm-0 blacklisted
Dec  7 18:07:25 m3 kernel: Buffer I/O error on device dm-1, logical block
718
Dec  7 18:07:25 m3 multipathd: dm-1 blacklisted
Dec  7 18:07:25 m3 kernel: lost page write due to I/O error on dm-1
Dec  7 18:07:25 m3 multipathd: fd0 blacklisted
Dec  7 18:07:25 m3 kernel: Buffer I/O error on device dm-1, logical block
726
Dec  7 18:07:25 m3 multipathd: hda blacklisted
Dec  7 18:07:25 m3 kernel: lost page write due to I/O error on dm-1
Dec  7 18:07:25 m3 multipathd: loop0 blacklisted
Dec  7 18:07:25 m3 kernel: REISERFS: abort (device dm-1): Journal write
error in flush_commit_list
Dec  7 18:07:25 m3 multipathd: loop1 blacklisted
Dec  7 18:07:25 m3 kernel: REISERFS: Aborting journal for filesystem on dm-1
Dec  7 18:07:25 m3 multipathd: loop2 blacklisted
Dec  7 18:07:25 m3 kernel: Device sda not ready.
Dec  7 18:07:25 m3 multipathd: loop3 blacklisted
Dec  7 18:07:25 m3 kernel: end_request: I/O error, dev sda, sector 506431
Dec  7 18:07:25 m3 multipathd: loop4 blacklisted
Dec  7 18:07:25 m3 kernel: Device sda not ready.
Dec  7 18:07:25 m3 multipathd: loop5 blacklisted
Dec  7 18:07:25 m3 kernel: end_request: I/O error, dev sda, sector 507455
Dec  7 18:07:25 m3 multipathd: loop6 blacklisted
Dec  7 18:07:25 m3 kernel: Device sda not ready.
Dec  7 18:07:25 m3 multipathd: loop7 blacklisted
Dec  7 18:07:25 m3 kernel: end_request: I/O error, dev sda, sector 508479
Dec  7 18:07:25 m3 multipathd: path checker already active : 8:0
Dec  7 18:07:25 m3 kernel: Device sda not ready.
Dec  7 18:07:25 m3 multipathd: path checker already active : 8:16
Dec  7 18:07:25 m3 kernel: end_request: I/O error, dev sda, sector 509503
Dec  7 18:07:25 m3 multipathd: start up event loops
Dec  7 18:07:25 m3 kernel: Device sda not ready.
Dec  7 18:07:25 m3 multipathd: event checker startup : disk1
Dec  7 18:07:25 m3 kernel: end_request: I/O error, dev sda, sector 509511
Dec  7 18:07:25 m3 multipathd: waiter->event_nr = 6
Dec  7 18:07:25 m3 kernel: Device sdb not ready.
Dec  7 18:07:25 m3 kernel: end_request: I/O error, dev sdb, sector 506431
Dec  7 18:07:25 m3 kernel: Buffer I/O error on device dm-1, logical block
63297
Dec  7 18:07:25 m3 kernel: lost page write due to I/O error on dm-1
Dec  7 18:07:25 m3 kernel: Device sdb not ready.
Dec  7 18:07:25 m3 kernel: end_request: I/O error, dev sdb, sector 507455
Dec  7 18:07:25 m3 kernel: Device sdb not ready.
Dec  7 18:07:25 m3 kernel: end_request: I/O error, dev sdb, sector 508479
Dec  7 18:07:25 m3 kernel: Device sdb not ready.
Dec  7 18:07:25 m3 kernel: end_request: I/O error, dev sdb, sector 509503
Dec  7 18:07:25 m3 kernel: Device sdb not ready.
Dec  7 18:07:25 m3 kernel: end_request: I/O error, dev sdb, sector 509511
Dec  7 18:07:29 m3 multipathd: checking paths
Dec  7 18:07:59 m3 multipathd: 8:16 : tur checker reports path is down
Dec  7 18:08:09 m3 multipathd: checking paths
Dec  7 18:08:40 m3 last message repeated 3 times
Dec  7 18:08:40 m3 multipathd: 8:0 : tur checker reports path is up
Dec  7 18:08:40 m3 multipathd: /sbin/multipath -v 0 -S -D 8:0
Dec  7 18:08:40 m3 multipathd: reconfigure multipath map
Dec  7 18:08:40 m3 multipathd: /sbin/multipath -v 0 -S disk1
Dec  7 18:08:40 m3 multipathd: devmap event on disk1
Dec  7 18:08:40 m3 multipathd: disk1 : reconfigure multipath map
Dec  7 18:08:40 m3 multipathd: refresh devmaps list
Dec  7 18:08:40 m3 multipathd: devmap disk1p1 :
Dec  7 18:08:40 m3 multipathd: \_ 0 1975932 linear
Dec  7 18:08:40 m3 multipathd:    skip non multipath target
Dec  7 18:08:40 m3 multipathd: devmap disk1 :
Dec  7 18:08:40 m3 multipathd: \_ 0 71114623 multipath
Dec  7 18:08:40 m3 multipathd: refresh failpaths list
Dec  7 18:08:40 m3 multipathd: cciss!c0d0 blacklisted
Dec  7 18:08:40 m3 multipathd: dm-0 blacklisted
Dec  7 18:08:40 m3 multipathd: dm-1 blacklisted
Dec  7 18:08:40 m3 multipathd: fd0 blacklisted
Dec  7 18:08:40 m3 multipathd: hda blacklisted
Dec  7 18:08:40 m3 multipathd: loop0 blacklisted
Dec  7 18:08:40 m3 multipathd: loop1 blacklisted
Dec  7 18:08:40 m3 multipathd: loop2 blacklisted
Dec  7 18:08:40 m3 multipathd: loop3 blacklisted
Dec  7 18:08:40 m3 multipathd: loop4 blacklisted
Dec  7 18:08:40 m3 multipathd: loop5 blacklisted
Dec  7 18:08:40 m3 multipathd: loop6 blacklisted
Dec  7 18:08:40 m3 multipathd: loop7 blacklisted
Dec  7 18:08:40 m3 multipathd: path checker already active : 8:0
Dec  7 18:08:40 m3 multipathd: path checker already active : 8:16
Dec  7 18:08:40 m3 multipathd: start up event loops
Dec  7 18:08:40 m3 multipathd: event checker startup : disk1
Dec  7 18:08:40 m3 multipathd: waiter->event_nr = 8
Dec  7 18:08:50 m3 multipathd: checking paths

the cp failed and now the disk is mounted read-only :(

this is my multipath.conf

defaults {
        multipath_tool  "/sbin/multipath -v 0 -S"
        udev_dir        /dev
        polling_interval 10
        default_selector        "round-robin 0"
        default_path_grouping_policy    failover
        default_getuid_callout  "/sbin/scsi_id -g -u -s /block/%n"
        default_prio_callout    "/bin/false"
}

devnode_blacklist {
        devnode cciss
        devnode fd
        devnode hd
        devnode md
        devnode dm
        devnode sr
        devnode scd
        devnode st
        devnode ram
        devnode raw
        devnode loop
}

multipaths {

        multipath {
                wwid    360001fe1001613800009205005470164
                alias   disk1
                path_checker    readsector0
        }

}

the same if use tur or readsector...

can you help me?

    Nicola Ranaldo