ext4 and extremely slow filesystem traversal

Andreas Dilger adilger at dilger.ca
Fri Mar 15 07:14:17 UTC 2013


On 2013-03-14, at 16:07, Vincent Caron <vcaron at bearstech.com> wrote:

> On 14/03/2013 02:05, Theodore Ts'o wrote:
>> Just as a note, e2fsck -v can sometimes get this information much more
>> quickly than other alternatives, since it can scan the file system in
>> inode order, instead of the essentially random order.
>> 
>> Just as a side, if you just want to get a rough count of the number of
>> directories, you can get that by grabbing the information out of
>> dumpe2fs.
> 
>  Very useful. Global stats without having to scan the whole filesystems
> are very precious...
> 
>  I was wondering : couldn't we use dumpe2fs or something based on
> libext2fs to quickly extract a snapshot of all inodes from a given
> filesystem ? For incremental backups, simply checking the mtime on
> millions of inodes and discovering that only a handful of them were
> updated since the previous pass looks very inefficient with
> readdir()+lstat().

That's exactly what e2scan does. I'm pretty sure that is in upstream e2fsprogs now (not just our Lustre version), but I'm on a plane and cannot check.

It will scan the inode table directly and can generate the pathnames of files efficiently. It can filter on timestamps. 

Cheers, Andreas

> So mnay syscalls, so man spoonfed bits of
> information. When I had a peek, I tought I'd got a list of inodes but
> would not be able to link them back to their name(s) without inducing
> the same cost as a regular find-like filesystem traversal. Does it make
> sense ?
> 
>  AFAIK I would be better served with block-level snapshot solutions,
> but LVM snapshots are supposed to double your writes if I got it right,
> and I'm not sure there's something else in the Linux and free software
> world. Plus I'd love to not migrate away from my ext3/4's without a
> compelling reason. Btrfs is not (yet) and option and ZFS doesn't fit
> legally with Linux.
> 
> _______________________________________________
> Ext3-users mailing list
> Ext3-users at redhat.com
> https://www.redhat.com/mailman/listinfo/ext3-users




More information about the Ext3-users mailing list