[Linux-cluster] clvmd without GFS?

[Matt Mitchell]

David Teigland wrote:
> Lots of small files can certainly expose some of the performance
> limitations of gfs.  "Hours" sounds very odd, though, so I ran a couple
> sanity tests on my own test hardware.
> 
> One node mounted with lock_dlm, the directory has 100,000 4k files,
> running "time ls -l | wc -l".
> 
> - dual P3 700 MHz, 256 MB, some old FC disks in a JBOD
>   5 min 30 sec
> 
> - P4 2.4 GHz, 512 MB, iscsi to a netapp
>   2 min 30 sec
> 
> Having more nodes mounted didn't change this.  (Four nodes of the first
> kind all running this at the same time averaged about 17 minutes each.)

Here are some more data points from the latest test I have tried.  I was 
feeling emboldened by the speed of the writing I was seeing, so I tried 
loading up a few more files.

The setup: 2 nodes, lock_dlm.  Both are P3/866s with 1 GB of RAM apiece. 
  One of the nodes (hudson) has two CPUs.

The process:
1) Mount disks.
2) Copy image files into subdirectory.  Other node is idle.
3)
hudson:/mnt/xs_media# time sh -c 'ls 100032/mls/fmls_stills | wc -l'
298407

real    7m40.726s
user    0m5.541s
sys     1m58.229s

While that's not stunningly great I consider it acceptable performance. 
  It's a lot of dentries to crawl through.

4) Feeling frisky, I decided to do a "real" test by unmounting and 
remounting the FS in order to clear caches, etc.  The other host did not 
get touched in this interval.

5)
hudson:/mnt/xs_media# time sh -c 'ls 100032/mls/fmls_stills | wc -l'
298407

real    74m43.284s
user    0m5.533s
sys     0m40.146s

Order of magnitude slower.

6) OK, I think, that might be the time to acquire locks the first time. 
  Doing it again should be fast:

hudson:/mnt/xs_media# time sh -c 'ls 100032/mls/fmls_stills | wc -l'
298407

real    75m29.150s
user    0m5.528s
sys     0m40.724s

7) Ugh.  OK, let's try it on the other node:

greenville:/mnt/xs_media# time sh -c 'ls 100032/mls/fmls_stills | wc -l'
298407

real    77m38.569s
user    0m8.850s
sys     0m35.006s


Both systems are sitting there idle now.  What did I do by unmounting 
and remounting the GFS partition?

For the record that is just over 12GB of data in those 298407 files. 
Partition is 3% full (as reported by df).

Help...?

-m