[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [Linux-cluster] AW: GNBD multipath with devicemapper? -- possible solution



On Tue, Apr 19, 2005 at 08:37:54AM +0200, Hansjoerg Maurer wrote:
> 
> The most anoying point is for me at the moment the differnence between 
> gnbd read and write performance.
> Therefore I am glad, that you as a gnbd-developer answered...
> In my tests, gnbd write is about two to three times faster the gnbd reads.
> I tried a lot of things (exporting cached, changing readahead with 
> blockdev command (on the underlying device), changing TCP-IP buffersizes)
> but I had nor improvement.
> 
> In the upper example, I get a write speed of about 85MB/s over gnbd and 
> a read speed of about 26 MB/s .
> (the underlying device's sda and sdb manages about 50MB/s (read and write).
> Therefore read speed is very good....

Um. On my machines I get 41.6707 MB/sec for reads and 41.6805 MB/sec for
writes using lmdd, so I don't see your difference.  If I just write to the
gnbd device, and don't sync the device after the writes I get a write speed of
54.3400 MB/sec.  This only takes into account how fast gnbd can write out all
the requests, not how long it takes for them to actually get written to disk
on the server. I would be suspicious of a gnbd write speed that is higher than
the served device's speed.

-Ben 

> First I thougt, it might be related to the strange dm-setup I was 
> running, and therefore I
> tried it with gnbd-exporting and importing just a single block device 
> (without lvm and dm)
> but the problem remains...
> 
> Do I have misconfiguerd something completly (I am using GBEth bonding 
> devices) or can you or anybody else confirm the
> behavior of much better write than read performance?
> I was testing with RHEL4 2.6.9-6.38.EL
> 
> Thank you for your help and your great work...
> 
> Greetings from a rainy morning  in Munich
> 
> 
> Hansjörg
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Benjamin Marzinski wrote:
> 
> >On Fri, Apr 15, 2005 at 04:24:21PM +0200, Hansjoerg Maurer dlr de wrote:
> > 
> >
> >>Hi
> >>
> >>I found a solution for the problem descriped below,
> >>but I am not sure if it is the right way.
> >>
> >>- importing the two gnbd's (wich point to the same device) from two 
> >>servers
> >>-> /dev/gnbd0 and /dev/gnbd1 on the client
> >>
> >>- creating a multipath device with something like this:
> >>echo "0 167772160  multipath 0 0 1 1 round-robin 0 2 1 251:0 1000 251:1 
> >>1000 " | dmsetup create dm0
> >>(251:0 ist the major:minor id of /dev/gnbd0)
> >>
> >>- mounting the created device
> >>eg:
> >>mount -t gfs /dev/mapper/dm0 /mnt/lvol0
> >>
> >>If I do a write on /mnt/lvol0 the gnbd_server task on both gnbd_servers 
> >>start (with a noticeable speedup)
> >>
> >>If one gnbd_server fails dm removes that path with the following log
> >>kernel: device-mapper: dm-multipath: Failing path 251:0.
> >>
> >>I was able to add it again with
> >>
> >>dmsetup  message dm0 0  reinstate_path 251:0
> >>
> >>
> >>I was able to deactivate a path manually with
> >>
> >>dmsetup  message dm0 0  fail_path 251:0 
> >>
> >>But I can not unimport the underlying gnbd
> >>
> >>gnbd_import: ERROR cannot disconnect device #1 : Device or resource busy
> >>
> >>
> >>Is there a way to remove a gnbd, which is bunndled in a dm-multipath 
> >>device?
> >>(might be necessary, if one gnbd server must be rebooted)
> >>
> >>How can I reimport an gnbd on the client in state disconnected?
> >>(I had to manually start 
> >>gnbd_recvd -d 0  to do so)
> >>
> >>Is the descriped solution for gnbd multipath the right one?
> >>   
> >>
> >
> >Um... It's a really ugly one. Unfortunately it's the only one that works, 
> >since
> >multipath-tools do not currently support non-scsi devices.
> >
> >There are also some bugs in gnbd that make multipathing even more annoying.
> >
> >But to answer your question, in order to remove gnbd, you must first get it
> >out of the multipath table, otherwise dm-multipath will still have it open.
> >
> >To do this, after dmsetup status shows that the path is failed, you run:
> >
> ># echo "0 167772160  multipath 0 0 1 1 round-robin 0 1 1 251:1 1000 " | 
> >dmsetup reload dm0
> ># dmsetup resume dm0
> >
> >This removes the gnbd from the path.
> >
> >However, if you use the gnbd code from the cvs head, it is no longer 
> >necessary
> >to do this to reimport the device.  In the stable branch, gnbd_monitor 
> >waits
> >until all users close the device before setting it to restartable. In the 
> >head
> >code, this happens as soon as the device is successfully fenced. So, if you
> >loose a gnbd server, reboot it, and reexport the device, gnbd_monitor 
> >should
> >automatically reimport the device, and you can simply run 
> >
> ># dmsetup  message dm0 0  reinstate_path 251:0
> >
> >and you should never need to remove the gnbd device with the method I 
> >described
> >above.
> >
> >
> >-Ben
> >
> > 
> >
> >>Thank you very much
> >>
> >>Greetings from munich
> >>
> >>Hansjörg
> >>
> >>
> >>
> >>
> >>
> >>
> >>   
> >>
> >>>Hi
> >>>
> >>>I am trying to set up gnbd with multipath.
> >>>Accoding to the gnbd_usage.txt file, I understand, that this should work 
> >>>with
> >>>dm-multipath.
> >>>But unfortunatly only the gfs part of the setup is descriped there.
> >>>
> >>>Has anybody experiance with this setup, especially how to set up
> >>>multipath with multiple /dev/gnbd* and how to setup the multipath.conf 
> >>>file
> >>>
> >>>
> >>>Thank you very much
> >>>
> >>>Hansjörg Maurer
> >>>     
> >>>
> >>-- 
> >>_________________________________________________________________
> >>
> >>Dr.  Hansjoerg Maurer           | LAN- & System-Manager
> >>                               |
> >>Deutsches Zentrum               | DLR Oberpfaffenhofen
> >> f. Luft- und Raumfahrt e.V.   |
> >>Institut f. Robotik             |
> >>Postfach 1116                   | Muenchner Strasse 20
> >>82230 Wessling                  | 82234 Wessling
> >>Germany                         |
> >>                               |
> >>Tel: 08153/28-2431              | E-mail: Hansjoerg Maurer dlr de
> >>Fax: 08153/28-1134              | WWW: http://www.robotic.dlr.de/
> >>__________________________________________________________________
> >>
> >>
> >>There are 10 types of people in this world, 
> >>those who understand binary and those who don't.
> >>
> >>
> >>
> >>
> >>--
> >>Linux-cluster mailing list
> >>Linux-cluster redhat com
> >>http://www.redhat.com/mailman/listinfo/linux-cluster
> >>   
> >>
> >
> >--
> >Linux-cluster mailing list
> >Linux-cluster redhat com
> >http://www.redhat.com/mailman/listinfo/linux-cluster
> >
> >
> > 
> >
> 
> 
> -- 
> _________________________________________________________________
> 
> Dr.  Hansjoerg Maurer           | LAN- & System-Manager
>                                |
> Deutsches Zentrum               | DLR Oberpfaffenhofen
>  f. Luft- und Raumfahrt e.V.   |
> Institut f. Robotik             |
> Postfach 1116                   | Muenchner Strasse 20
> 82230 Wessling                  | 82234 Wessling
> Germany                         |
>                                |
> Tel: 08153/28-2431              | E-mail: Hansjoerg Maurer dlr de
> Fax: 08153/28-1134              | WWW: http://www.robotic.dlr.de/
> __________________________________________________________________
> 
> 
> There are 10 types of people in this world, 
> those who understand binary and those who don't.
> 
> --
> Linux-cluster mailing list
> Linux-cluster redhat com
> http://www.redhat.com/mailman/listinfo/linux-cluster


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]