[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

[Linux-cluster] GFS2 metadata performance



Hi,

Has anyone done any GFS2 metadata performance benchmarks? If so, what have you found? Also, what performance tuning would be recommended to increase the metadata performance of a GFS2 filesystem?

I've recently ran 'fdtree' [1] against a GFS2 filesystem as well as an ext3 filesystem. Here's what I've found:

(fdtree on a GFS2 filesystem)
# ./fdtree.bash -l 10 -d 3 -f 4 -s 1 -o /gfs2bench/fdtree
fdtree-1.0.2: starting at /gfs2bench/fdtree//LEVEL0.vm1.23787/
        creating/deleting 10 directory levels with 3 directories at each level
        for a total of 88573 directories
        with 4 files of size 4KiB per directory
        for a total of 354292 files and 1417168KiB
Sun Mar 13 00:45:31 EST 2011
Sun Mar 13 00:58:46 EST 2011
DIRECTORY CREATE TIME IN, OUT, TOTAL = 0, 795, 795
        Directory creates per second =  111
Sun Mar 13 00:58:46 EST 2011
Sun Mar 13 03:00:44 EDT 2011
FILE CREATE TIME IN, OUT, TOTAL      = 795, 4513, 3718
        File creates per second      =  95
        KiB per second               =  381
Sun Mar 13 03:00:44 EDT 2011
Sun Mar 13 04:49:08 EDT 2011
FILE REMOVE TIME IN, OUT, TOTAL      = 4513, 11017, 6504
        File removals per second     =  54
Sun Mar 13 04:49:08 EDT 2011
Sun Mar 13 05:02:58 EDT 2011
DIRECTORY REMOVE TIME IN, OUT, TOTAL = 11017, 11847, 830
        Directory removals per second =  106

(fdtree on an ext3 filesystem)
# ./fdtree.bash -l 10 -d 3 -f 4 -s 1 -o /ext3bench/fdtree
fdtree-1.0.2: starting at /ext3bench/fdtree//LEVEL0.vm1.25896/
        creating/deleting 10 directory levels with 3 directories at each level
        for a total of 88573 directories
        with 4 files of size 4KiB per directory
        for a total of 354292 files and 1417168KiB
Sun Mar 13 18:41:11 EDT 2011
Sun Mar 13 18:45:48 EDT 2011
DIRECTORY CREATE TIME IN, OUT, TOTAL = 0, 277, 277
        Directory creates per second =  319
Sun Mar 13 18:45:49 EDT 2011
Sun Mar 13 19:04:33 EDT 2011
FILE CREATE TIME IN, OUT, TOTAL      = 278, 1402, 1124
        File creates per second      =  315
        KiB per second               =  1260
Sun Mar 13 19:04:33 EDT 2011
Sun Mar 13 19:09:15 EDT 2011
FILE REMOVE TIME IN, OUT, TOTAL      = 1402, 1684, 282
        File removals per second     =  1256
Sun Mar 13 19:09:15 EDT 2011
Sun Mar 13 19:10:42 EDT 2011
DIRECTORY REMOVE TIME IN, OUT, TOTAL = 1684, 1771, 87
        Directory removals per second =  1018

In other words, ext3 is about 3 times faster at creating files/dirs, about 20 times faster at removing existing files, and about 10 times faster at removing existing directories. I've added the following lines to /etc/cluster/cluster.conf to remove the plock rate limit:
 <dlm plock_ownership="1" plock_rate_limit="0"/>
 <gfs_controld plock_rate_limit="0"/>
but this didn't help increase the GFS2 metadata performance.

Note that I've used the same setup for the GFS2 and ext3 tests: same machine, same networking config, same storage array (which is not used by anything else).
I also confirmed using "pingpong" [2] that I get a rate of about 4K locks/sec on this particular node against GFS2.

Does anyone have any hints/ideas as what might help increase the metadata performance of a GFS2 filesystem?

[1] https://computing.llnl.gov/?set=code&page=sio_downloads
[2] http://wiki.samba.org/index.php/Ping_pong

Best,
-- 
Valeriu Mutu


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]