[Linux-cluster] GNBD speed

[Chris Picton <lists tangent co za>]

Hi All

I am testing the following scenario:

A DRBD mirror between two servers, which heartbeat failover the drbd 
primary, gnbd export and ip address.

I am trying to find potential bottlenecks, and have done the following 
tests.

Network speed between the DRBD servers (A and B)
---------------------------------------------------------
(A) dd if=/dev/zero bs=1G count=1 | nc 10.100.1.2 5001 
(B) nc k -l 5001 | dd of=/dev/null
(A) reports:
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 7.6384 seconds, 141 MB/s

DRBD sync speed:
----------------------------------------------------------
dd if=/dev/zero bs=1G count=1 of=/dev/drbd0 oflag=sync
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 10.7832 seconds, 99.6 MB/s

Network speed between GNBD export (A) and import (C)
-----------------------------------------------------------
(C) dd if=/dev/zero bs=1G count=1 | nc nfs1 5001
(A) nc -k -l 5001 | dd of=/dev/null
(C) reports:
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 10.4001 seconds, 103 MB/s

Network speed between GNBD import (C) and export (A)
-----------------------------------------------------------
(A) dd if=/dev/zero bs=1G count=1 | nc 10.200.3.10 5001
(C) nc -k -l 5001 | dd of=/dev/null
(A) reports:
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 10.4001 seconds, 93 MB/s


So I have established that writing to drbd directly is fast, and network 
speed is fast

However, using gnbd as follows:
on the drbd server:
gnbd_serv -n
/sbin/gnbd_export -c -e r0 -d /dev/drbd0

On the client:
gnbd_import -i 10.200.3.3

I try the speed tests over the gnbd devices:

Reading from GNBD:
------------------------------------------------------------
dd if=/dev/gnbd0 of=/dev/null bs=1G count=1
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 17.0842 seconds, 62.8 MB/s

Writing to GNBD (no sync flag)
------------------------------------------------------------
dd if=/dev/zero of=/dev/gnbd0 bs=1G count=1
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 54.4142 seconds, 19.7 MB/s

Writing to GNBD (sync flag)
------------------------------------------------------------
dd if=/dev/zero of=/dev/gnbd0 bs=1G count=1 oflag=sync
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 53.3085 seconds, 20.1 MB/s



I am almost happy with the 62 Mb/s read speed, but the 20 MB/sec write 
speed seems a bit low, compared to the write rate to drbd, and the 
network speed.

Can anyone give any hints for optimising the gnbd write speed (and read 
speed)


Chris