[Linux-cluster] AW: GNBD multipath with devicemapper? -- possible solution
Benjamin Marzinski
bmarzins at redhat.com
Tue Apr 19 16:19:31 UTC 2005
On Tue, Apr 19, 2005 at 08:37:54AM +0200, Hansjoerg Maurer wrote:
>
> The most anoying point is for me at the moment the differnence between
> gnbd read and write performance.
> Therefore I am glad, that you as a gnbd-developer answered...
> In my tests, gnbd write is about two to three times faster the gnbd reads.
> I tried a lot of things (exporting cached, changing readahead with
> blockdev command (on the underlying device), changing TCP-IP buffersizes)
> but I had nor improvement.
>
> In the upper example, I get a write speed of about 85MB/s over gnbd and
> a read speed of about 26 MB/s .
> (the underlying device's sda and sdb manages about 50MB/s (read and write).
> Therefore read speed is very good....
Um. On my machines I get 41.6707 MB/sec for reads and 41.6805 MB/sec for
writes using lmdd, so I don't see your difference. If I just write to the
gnbd device, and don't sync the device after the writes I get a write speed of
54.3400 MB/sec. This only takes into account how fast gnbd can write out all
the requests, not how long it takes for them to actually get written to disk
on the server. I would be suspicious of a gnbd write speed that is higher than
the served device's speed.
-Ben
> First I thougt, it might be related to the strange dm-setup I was
> running, and therefore I
> tried it with gnbd-exporting and importing just a single block device
> (without lvm and dm)
> but the problem remains...
>
> Do I have misconfiguerd something completly (I am using GBEth bonding
> devices) or can you or anybody else confirm the
> behavior of much better write than read performance?
> I was testing with RHEL4 2.6.9-6.38.EL
>
> Thank you for your help and your great work...
>
> Greetings from a rainy morning in Munich
>
>
> Hansjörg
>
>
>
>
>
>
>
>
>
>
>
>
>
> Benjamin Marzinski wrote:
>
> >On Fri, Apr 15, 2005 at 04:24:21PM +0200, Hansjoerg.Maurer at dlr.de wrote:
> >
> >
> >>Hi
> >>
> >>I found a solution for the problem descriped below,
> >>but I am not sure if it is the right way.
> >>
> >>- importing the two gnbd's (wich point to the same device) from two
> >>servers
> >>-> /dev/gnbd0 and /dev/gnbd1 on the client
> >>
> >>- creating a multipath device with something like this:
> >>echo "0 167772160 multipath 0 0 1 1 round-robin 0 2 1 251:0 1000 251:1
> >>1000 " | dmsetup create dm0
> >>(251:0 ist the major:minor id of /dev/gnbd0)
> >>
> >>- mounting the created device
> >>eg:
> >>mount -t gfs /dev/mapper/dm0 /mnt/lvol0
> >>
> >>If I do a write on /mnt/lvol0 the gnbd_server task on both gnbd_servers
> >>start (with a noticeable speedup)
> >>
> >>If one gnbd_server fails dm removes that path with the following log
> >>kernel: device-mapper: dm-multipath: Failing path 251:0.
> >>
> >>I was able to add it again with
> >>
> >>dmsetup message dm0 0 reinstate_path 251:0
> >>
> >>
> >>I was able to deactivate a path manually with
> >>
> >>dmsetup message dm0 0 fail_path 251:0
> >>
> >>But I can not unimport the underlying gnbd
> >>
> >>gnbd_import: ERROR cannot disconnect device #1 : Device or resource busy
> >>
> >>
> >>Is there a way to remove a gnbd, which is bunndled in a dm-multipath
> >>device?
> >>(might be necessary, if one gnbd server must be rebooted)
> >>
> >>How can I reimport an gnbd on the client in state disconnected?
> >>(I had to manually start
> >>gnbd_recvd -d 0 to do so)
> >>
> >>Is the descriped solution for gnbd multipath the right one?
> >>
> >>
> >
> >Um... It's a really ugly one. Unfortunately it's the only one that works,
> >since
> >multipath-tools do not currently support non-scsi devices.
> >
> >There are also some bugs in gnbd that make multipathing even more annoying.
> >
> >But to answer your question, in order to remove gnbd, you must first get it
> >out of the multipath table, otherwise dm-multipath will still have it open.
> >
> >To do this, after dmsetup status shows that the path is failed, you run:
> >
> ># echo "0 167772160 multipath 0 0 1 1 round-robin 0 1 1 251:1 1000 " |
> >dmsetup reload dm0
> ># dmsetup resume dm0
> >
> >This removes the gnbd from the path.
> >
> >However, if you use the gnbd code from the cvs head, it is no longer
> >necessary
> >to do this to reimport the device. In the stable branch, gnbd_monitor
> >waits
> >until all users close the device before setting it to restartable. In the
> >head
> >code, this happens as soon as the device is successfully fenced. So, if you
> >loose a gnbd server, reboot it, and reexport the device, gnbd_monitor
> >should
> >automatically reimport the device, and you can simply run
> >
> ># dmsetup message dm0 0 reinstate_path 251:0
> >
> >and you should never need to remove the gnbd device with the method I
> >described
> >above.
> >
> >
> >-Ben
> >
> >
> >
> >>Thank you very much
> >>
> >>Greetings from munich
> >>
> >>Hansjörg
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>>Hi
> >>>
> >>>I am trying to set up gnbd with multipath.
> >>>Accoding to the gnbd_usage.txt file, I understand, that this should work
> >>>with
> >>>dm-multipath.
> >>>But unfortunatly only the gfs part of the setup is descriped there.
> >>>
> >>>Has anybody experiance with this setup, especially how to set up
> >>>multipath with multiple /dev/gnbd* and how to setup the multipath.conf
> >>>file
> >>>
> >>>
> >>>Thank you very much
> >>>
> >>>Hansjörg Maurer
> >>>
> >>>
> >>--
> >>_________________________________________________________________
> >>
> >>Dr. Hansjoerg Maurer | LAN- & System-Manager
> >> |
> >>Deutsches Zentrum | DLR Oberpfaffenhofen
> >> f. Luft- und Raumfahrt e.V. |
> >>Institut f. Robotik |
> >>Postfach 1116 | Muenchner Strasse 20
> >>82230 Wessling | 82234 Wessling
> >>Germany |
> >> |
> >>Tel: 08153/28-2431 | E-mail: Hansjoerg.Maurer at dlr.de
> >>Fax: 08153/28-1134 | WWW: http://www.robotic.dlr.de/
> >>__________________________________________________________________
> >>
> >>
> >>There are 10 types of people in this world,
> >>those who understand binary and those who don't.
> >>
> >>
> >>
> >>
> >>--
> >>Linux-cluster mailing list
> >>Linux-cluster at redhat.com
> >>http://www.redhat.com/mailman/listinfo/linux-cluster
> >>
> >>
> >
> >--
> >Linux-cluster mailing list
> >Linux-cluster at redhat.com
> >http://www.redhat.com/mailman/listinfo/linux-cluster
> >
> >
> >
> >
>
>
> --
> _________________________________________________________________
>
> Dr. Hansjoerg Maurer | LAN- & System-Manager
> |
> Deutsches Zentrum | DLR Oberpfaffenhofen
> f. Luft- und Raumfahrt e.V. |
> Institut f. Robotik |
> Postfach 1116 | Muenchner Strasse 20
> 82230 Wessling | 82234 Wessling
> Germany |
> |
> Tel: 08153/28-2431 | E-mail: Hansjoerg.Maurer at dlr.de
> Fax: 08153/28-1134 | WWW: http://www.robotic.dlr.de/
> __________________________________________________________________
>
>
> There are 10 types of people in this world,
> those who understand binary and those who don't.
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> http://www.redhat.com/mailman/listinfo/linux-cluster
More information about the Linux-cluster
mailing list