[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

[Linux-cluster] GFS is *very* slow when NFS exported



We are looking at using GFS for load balanced NFS. Going with GNBD exported GFS isn't really an option since we're talking about over 1000 machines needing to access the storage. For this reason we were looking at providing a relatively small cluster of GFS machines serving the storage via NFS. The clients would then use round-robin DNS to load balance across these servers.

The results we've got back from our tests are shown below:

-= Setup ========================================-

Two machines with GFS filesystem mounted from a
dual-port RAID, connected to ethernet via GigE and
serving NFS as follows:

/mnt/gfs *(no_root_squash,rw,insecure,async)

-= End of Setup =================================-


-= Local GFS Filesystem Access ==================-


The tests show Mb/s computed by using 10Gb dd

1 machine, write (gfstest1):
 166
1 machine, read (gfstest2 reading file gfstest1 created):
 139.5
2 machines, sim writes (different files):
 113.7 (gfstest1) 108 (gfstest2) - 221.7aggr
2 machines, sim reads (different files):
 101.5 (gfstest1) 101.6 (gfstest2) - 203aggr
2 machines, sim reads (same file):
 130 (gfstest1) 134 (gfstest2) - 264aggr

-= End of Local GFS Filesystem Access ===========-


-= NFS/GFS Access ===============================-


1 write: (client1 to gfstest1)
 42.3
1 read: (client1 from gfstest1)
 Varies enormously between 10-35Mb/s)

Simultaneous writes and reads done as follows:
 client1 NFS mounting gfstest1
 clients2 NFS mounting gfstest2

2 sim writes (different files):
 39.6 (client1) 42.4 (client2) - 82aggr
2 sim reads (different files):
 11.3 (client1) 11.2 (client2) - 22.5aggr

-= End of NFS/GFS Access ========================-


-= NFS/XFS Access ===============================-


Done for comparison of XFS & GFS export speeds

1 write: 65.2
1 read: 73.5

-= End of NFS/XFS Access ========================-


We know NFS isn't the highest performing thing in the world, but it's a concern that the NFS performance of a GFS mounted filesystem is so much lower than that of an XFS system. There are of course, the clustering overheads, but this would affect local performance as well. Anyone got any ideas as to why this might be and how to get more performance?


Thanks,

Stephen


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]