ext4 and extremely slow filesystem traversal

[Bodo Thiesen]

* Vincent Caron <vcaron at bearstech.com> hat geschrieben:

> AFAIK I would be better served with block-level snapshot solutions,
> but LVM snapshots are supposed to double your writes if I got it right,
> and I'm not sure there's something else in the Linux and free software
> world.

There is a simple and reliable solution for block level backups: dd

umount && dd if="our raid partition" of="some new big enough disc" &&
mount

and then wait for the data to go at 100MB/s or so to the new disc.

Using snapshots is not a reliable way to do backups, since you would
still have to trust the LVM code to be totally error free and protect
your data under any circumstances (including hardware failures in your
raid array etc).

For your actual problem: Ask your developers to use some mapping
system.

When they want to access a file "filename" then calculate md5sum of
"filename", take the first 6 characters of the ascii representation
(here it would be 435ed7) and create a file called "43/5e/d7-X".
This way you would end up with at most 65792 directories. The X is needed
to distinquish between files with same 6 first letters md5sum. So, first
file gets name "43/5e/d7-1", second file gets name "43/5e/d7-2" and so on.

Somewhere else, they would then store the mapping table, mapping file
"filename" to "43/5e/d7-2". All accesses go through this mapping table.

Regards, Bodo