[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

SV: Ext3 destroying ownerships and permissions



> > A few weeks ago we upgraded 9 large webservers from ext2 to
> ext3. Since then we've seen very strange behavior on several
> of the machines. Permissions of files are repeatedly changed
> at random occasions. Several  times, ownership of files have
> been totally mangled. Several users have logged in to
> discover that all their files suddenly are owned by another
> user! At two of these occasions the machine froze and
> rebooted when we started to chown the files back to their real owner.
>
> OK, a few questions:
>
> Did you see any _other_ fs effects?  Contents of the files changing,
> for example?  I can't imagine any possible fs corruption which would
> touch permissions but leave the rest of the inode data intact.

No, actually not. It seems that the only effect is changes of ownership
and permissions.

Corrupted permissions have been reported mostly with files needing
global write permissions (www-counters writable for nobody, etc) which
have lost their special permissions. Usually they've been "reset"
according to the normal umask (ie. getting permissions 644). On the
other hand there might be a lot of other files whose permissions have
changed but where it's not noticed. When a counter or a cgi-script stops
to function it's noticed pretty quickly.

Changed ownerships seem to happen in larger groups or batches. The most
frequent occurence is that "user A" finds that his homedir and *all*
files and dirs under it are suddenly owned by "user B". Also files under
"user A"'s homedir which where previously owned by root (some
auto-generated statistics are owned by root) get their ownership changed
to "user B". Files belonging to "user A" in other places in the
filesystem (mailboxes, crontabs, temporary files, etc) are not affected.

We have our homedirs arranged according to the scheme: /home/a/a*,
/home/b/b*, so that all usernames begining with "a" are found in
/home/a, those begining with "b" are in /home/b, and so forth. Yesterday
there was an unpleasant surprise in /home/f: about 10 of 60 "f*"
homedirs had their ownerships recursively changed to another "f*" user,
everything under the 10 homedirs was now owned by this user. Nothing
else in the filesystem seemed to be changed or corrupted.

> Did you have any unplanned recoveries?

On some of the servers, yes, on others, no. It doesn't seem to be
directly related to the permission/ownership corruptions.

As mentioned earlier - on at least two occasions the kernel freaked out
and rebooted *as soon as we started to "chown" files back* to their
rightful owner.

> > There are no fs related kernel error messages, nothing to
> indicate that the kernel knows that something is wrong. I'd
> be happy to provide you with more information if needed.
>
> If you have any more, then yes, please.  I assume you're using
> hardware raid only, no soft raid?

Yes, only hardware raid. I can't think of any more information that
might be relevant. I guess it might be important that we made the change
from ext2 to ext3 on nine servers simultaneously, and that all of them
got the same kind of permission/ownership corruption.

Regards,
/Johan Ekenberg





[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]