[Linux-cluster] Linux-cluster Digest, Vol 91, Issue 7

[Randy Zagar]

I'm betting, Alan, that you're used to being able to tell your users how 
to do their work... :-)  Some of us don't live in that world, and our 
users are allowed to do some incredibly stupid things.

I'm curious what filesystem you would recommend for his use-case.

-RZ

p.s.  Nicolas' users are only slightly "angry" compared to mine... My 
users are well and truly MAD, however.  Think 16-TB ext4 filesystems 
with 400,000,000 files smaller than 64K.

On 11/07/2011, Alan Brown <ajb2 at mssl.ucl.ac.uk> wrote:
> Nicolas Ross wrote:
>> On some services, there are document directories that are huge, not that
>> much in size (about 35 gigs), but in number of files, around one
>> million. One service even has 3 data directories with that many files each.
> You are utterly mad.
>
> Apart from the human readability aspects if someone attempts a directory
> listing, you're putting a substantial load on your system each time you
> attempt to go into those directories, even with dentry/inode caching
> tweaked out to maximums.
>
> Directory inode hashing helps, but not for filesystem abuse on this scale.
>
> Be glad you're using ext3/4 and not GFS, the problems are several orders
> of magnitude worse there (it can take 10 minutes to list a directory
> with 10,000 files in it, let alone 1,000,000)
>
>    >  It works pretty well for now, but when it comes to data update (via
>> rsync) and backup (also via rsync), the node doing the rsync crawls to a
>> stop, all 16 logical cores are used at 100% system, and it sometimes
>> freezes the file system for other services on other nodes.
> That's not particularly surprising - and a fairly solid hint you should
> be revisiting the way you lay out your files.
>
> If you go for a hierarchical layout you'll see several orders of
> magnitude speedup in access time without any real effort at all.
>
> If you absolutely must put that many files in a directory, then use a
> filesystem able to cope with such activities. Ext3/4 aren't it.


-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5434 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20111107/3f4ba2c0/attachment.p7s>