Re: Poor Performance WhenNumber of Files > 1M

[Bruno Wolff III <bruno wolff to>]

On Wed, Aug 01, 2007 at 18:55:53 -0700,
  Sean McCauliff <smccauliff mail arc nasa gov> wrote:
> Hi all,
> 
> I plan on having about 100M files totaling about 8.5TiBytes.   To see 
> how ext3 would perform with large numbers of files I've written a test 
> program which creates a configurable number of files into a configurable 
> number of directories, reads from those files, lists them and then 
> deletes them.  Even up to 1M files ext3 seems to perform well and scale 
> linearly; the time to execute the program on 1M files is about double 
> the time it takes it to execute on .5M files.  But past 1M files it 
> seems to have n^2 scalability.  Test details appear below.
> 
> Looking at the various options for ext3 nothing jumps out as the obvious 
> one to use to improve performance.
> 
> Any recommendations?

Did you make sure directory indexing is available? I think that is the
default now for ext3, but maybe it wasn't turned on for your test.