EXT3 and large directories

[Christian]

(please refrain from sendin HTML mails)

On Wed, 4 Oct 2006, Matt Dodson wrote:
> I have an ext3 filesystem that has several directories and each
> directory gets a large number of files inserted and then deleted over
> time.

Can you specify "large number"? What do "ls large-directory | wc -l" 
say?

> The filesystem is basically used as a temp store before files are
> processed. The issue is over time the directory scans get extremely slow
> even if the directories are empty. I have noticed the directories can
> range in size from 4k - 100M even when they are empty.

proably deleted-but-still-open files. When lsof(8) is installed, you can 
find out with: "lsof -ln | grep large-but-empty-directory"

Can you specify "slow" as well? You also might want strace(1) an "ls" on 
your large directory to see what is taking so long.

> Is there a way to fix this without recreating the directories or
> bringing the filesystem offline?

You have enabled htree (dir_index) already:

> Filesystem features:      has_journal resize_inode dir_index filetype
> needs_recovery sparse_super large_file

If you've enabled dir_index after the directories have been created, you 
might want to "e2fsck -D" (see the manpage for details) the filesystem. 
For partitions with temprary files you could play with "noatime","async" 
and "data" mount-options (please read the manpage, really!).
Which kernel do you use? Which arch?

C.
-- 
BOFH excuse #83:

Support staff hung over, send aspirin and come back LATER.