files missing

[Jeff Dinisco]

Thanks for the reply.  I'm already in the process of copying the
filesystem to an image using "cp /dev/sdf /another_fs/sdf.image" and I
plan to fsck -y that image.  Is there another method I should use?  I'm
not sure how long it was mounted r/w on the other host but it's pretty
clear that that's the cause.  Is there particular service you can
recommmend for this type of data recovery (does that answer your backup
question)?  Thanks.  

-----Original Message-----
From: Theodore Ts'o [mailto:tytso at thunk.org] On Behalf Of Theodore Ts'o
Sent: Tuesday, November 23, 2004 12:48 PM
To: Jeff Dinisco
Cc: ext3-users at redhat.com
Subject: Re: files missing

On Mon, Nov 22, 2004 at 06:05:56PM -0800, Jeff Dinisco wrote:
> I have a 1 TB ext3 filesystem mounted via iscsi on a redhat 9 system 
> w/ kernel version - 2.4.20-30.9.  I'm not sure when it happened, but 
> today there appears to be about 7,000 files (600GB) missing.  The 
> output from df implies that the files are still there.  It shows 861
GB utilized.
> But du shows only 300 GB of data.  I'm sure that there are no 
> processes holding onto deleted files because I have unmounted/mounted 
> the filesystem several times, synced, etc.  Here's an excerpt from 
> e2fsck -nf /dev/sdf ...

The output from df is calculated by using the free blocks field in the
superblock, and is only as accurate as the superblock statistics will
be.

It does look like garbage has been written into your filesystem, and the
situation is probably pretty grim; there will almost certainly be some
data loss.  It may not be as bad as you think, however, since there may
be disconnected inodes (and possibly entire directory
hierarchies) which e2fsck can recover and link into the lost+found
directory. 

> Things look pretty grim right now.  As soon as I run e2fsck -p, these 
> inodes will be deleted and I will lose data, correct?

Actually, e2fsck -y will be needed to fix the data.  E2fsck -p will make
"safe" fixes, but anything that might require human judgement will cause
e2fsck -p to abort (since it is intended to be used in unattended boot
scripts).  You will need to answer each of e2fsck's questions manually,
or use e2fsck -y to cause e2fsck to automatically assume an answer of
"yes" for each question.  I'd strongly suggest making a bit-for-bit
image backup of the filesystem before proceeding, since it might be
possible for an expert to try other means of recoverying data after
getting back what data you can with e2fsck -y.

> I suspect the
> cause of this issue was multiple hosts mounting this filesystem r/w.

Ah, yup, that would do it.  How long was the filesystem mounted by
multiple hosts?

> Does anyone know of any method that could be used to recover this
data?
> Any help would be greatly appreciated.

You can try e2fsck -y and hope for the best.  Aside from that, you can
grep the disk looking for a specific text pattern if there is a few
extremely valuable files that have to be recovered at all costs.
Failing that, it will be restore from backup times.....  (you did keep
regular backups, right?  :-)

						- Ted