[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [Linux-cluster] clvmd without GFS?



David Teigland wrote:
Lots of small files can certainly expose some of the performance
limitations of gfs.  "Hours" sounds very odd, though, so I ran a couple
sanity tests on my own test hardware.

One node mounted with lock_dlm, the directory has 100,000 4k files,
running "time ls -l | wc -l".

- dual P3 700 MHz, 256 MB, some old FC disks in a JBOD
  5 min 30 sec

- P4 2.4 GHz, 512 MB, iscsi to a netapp
  2 min 30 sec

Having more nodes mounted didn't change this.  (Four nodes of the first
kind all running this at the same time averaged about 17 minutes each.)

Here are some more data points from the latest test I have tried. I was feeling emboldened by the speed of the writing I was seeing, so I tried loading up a few more files.


The setup: 2 nodes, lock_dlm. Both are P3/866s with 1 GB of RAM apiece. One of the nodes (hudson) has two CPUs.

The process:
1) Mount disks.
2) Copy image files into subdirectory.  Other node is idle.
3)
hudson:/mnt/xs_media# time sh -c 'ls 100032/mls/fmls_stills | wc -l'
298407

real    7m40.726s
user    0m5.541s
sys     1m58.229s

While that's not stunningly great I consider it acceptable performance. It's a lot of dentries to crawl through.

4) Feeling frisky, I decided to do a "real" test by unmounting and remounting the FS in order to clear caches, etc. The other host did not get touched in this interval.

5)
hudson:/mnt/xs_media# time sh -c 'ls 100032/mls/fmls_stills | wc -l'
298407

real    74m43.284s
user    0m5.533s
sys     0m40.146s

Order of magnitude slower.

6) OK, I think, that might be the time to acquire locks the first time. Doing it again should be fast:

hudson:/mnt/xs_media# time sh -c 'ls 100032/mls/fmls_stills | wc -l'
298407

real    75m29.150s
user    0m5.528s
sys     0m40.724s

7) Ugh. OK, let's try it on the other node:

greenville:/mnt/xs_media# time sh -c 'ls 100032/mls/fmls_stills | wc -l'
298407

real    77m38.569s
user    0m8.850s
sys     0m35.006s


Both systems are sitting there idle now. What did I do by unmounting and remounting the GFS partition?


For the record that is just over 12GB of data in those 298407 files. Partition is 3% full (as reported by df).

Help...?

-m


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]