[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: How many files can I have safely in a subdirectory?



On Sep 01, 2009  13:20 -0400, Charles Riley wrote:
> I'm not sure if it's still the case, but there used to be a limit
> to how many subdirectories a directory can have.  32k, to be exact.
> We ended up creating our own (application level) directory hashing
> algorithm to work around it several years ago.  This might only be a
> kernel 2.4 thing though.

This is true for ext3 (max 32000 subdirectories), but in ext4 there
is no specific limit on the number of subdirectories.  The subdirectory
limit is the same as the number of entries in the directory.

> I'm unaware of any limit to number of files.  However once the number
> of files in a directory gets above about 64k, filesystem performance will
> significantly decrease unless the filesystem has the dir_index option.
> dir_index can be specified at filesystem creation or added later using
> tune2fs (an fsck is required).

If formatted with dir_index (which is the default for newer mke2fs for
some time now) we tested up to 10M files in a single directory on a
regular basis.  The maximum limit depends on the filename length, but
is somewhere around 15-20M for "short" filenames (e.g. 32 characters
or less).


> ----- Original Message -----
> From: "z0diac" <web2009 zeroreality com>
> To: ext3-users redhat com
> Sent: Sunday, August 30, 2009 12:00:12 PM GMT -05:00 US/Canada Eastern
> Subject: How many files can I have safely in a subdirectory?
> 
> 
> Ok, I'm running a vBulletin forum (3.8.4) and found that all user attachments
> go into 1 single directory for each user. For each attached file in the
> forum, there's 2 files on disk (*.attach and *.thumb), for pictures that is.
> 
> One user already has over 100,000 attachements, thus, over 200,000 files in
> his attach directory.
> 
> Someone recently told me to 'keep an eye on it' because certain setups can't
> hold more than X number of files in a single directory. Yet someone else
> said I could have over 1 trillion files in a single directory if the HDD was
> large enough...
> 
> Here's my setup:
> 
> linux version: 2.6.18-92.1.10.el5
> php: 5.1.6
> mySQL: 5.0.45
> File system: ext3
> 
> vB support has told me any limitation there might be, will not be the result
> of vB, so now I'm looking at either Linux and the way it handles files, or
> the ext3 file system.
> 
> Does anyone know if I can just keep going with putting files into one
> directory? (there will be over 1million probably by year's end. Hopefully
> not more than 5 million ever).
> 
> And, will having so many files in a single directory cause any performance
> problems? (ie: slowdowns)
> 
> My only option is to hire a coder to somehow have it split the 1M+ files
> into several subdirs, say 50,000 per subdir. But even though it's messy, if
> it really doesn't make a difference in the end whether they're in 50
> subdirs, or just 1 dir, then I won't bother (and can sigh a breath of
> relief)
> 
> 
> Thanks in advance!!
> z0diac is offline   	
> Looking for Linux Hosting? Click Here.
> 
> -- 
> View this message in context: http://www.nabble.com/How-many-files-can-I-have-safely-in-a-subdirectory--tp25212801p25212801.html
> Sent from the Ext3 - User mailing list archive at Nabble.com.
> 
> _______________________________________________
> Ext3-users mailing list
> Ext3-users redhat com
> https://www.redhat.com/mailman/listinfo/ext3-users
> 
> _______________________________________________
> Ext3-users mailing list
> Ext3-users redhat com
> https://www.redhat.com/mailman/listinfo/ext3-users

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]