Recent unexplained quota problems

[Chris St. Pierre]

On Sun, 9 Sep 2007, Ryan Golhar wrote:

> I'm running a RHEL v3 server, completely up to date...
>
> I tried to edit a user's quota (as root) using the command
> '/usr/sbin/edquota someuser' and I got the error:
>
> edquota: Can't open quotafile /home/aquota.user: Read-only file system
> No filesystems with quota detected.
>
> Doing a listing of aquota.user reports:
>
> [root at server log]# ll /home/aquota.user
> -rw-------    1 root     root        15360 Sep  9 04:22 /home/aquota.user
>
> /etc/fstab has:
> LABEL=/home            /home          ext3    defaults,usrquota 1 2
>
> A listing of /home shows:
> drwxr-xr-x  133 root     root         4096 Sep  7 12:49 home
>
> If I try to 'touch test' in /home I get:
> [root at server home]# touch test
> touch: creating `test': Read-only file system
>
> I rebooted the server and everything seems to be okay.  I'm a little concerned 
> about this though because I can't explain it.

You should be concerned about this.  The kernel will change a
filesystem to read-only when it detects an IO error against that FS.
This can happen for a number of reasons:

   - Your connection to your SAN dropped;
   - Your hard drive(s) are dying;
   - You have significant data corruption;
   - and on and on...

Except for the first reason I listed, all of the other reasons I know
of are Real Bad.

If you're lucky, you've got some minor data corruption that caused the
kernel to try to write beyond the end of the drive or something like
that; you should try running fsck on the filesystem first.  Be warned,
though, that if you have significant data corruption, fsck may
completely hose the filesystem, so get as good a backup as you can first.

You should check /var/log/messages for kernel messages about this.  If
it happens again, dmesg will also have useful information (at least,
it will until you reboot).

If the problem is transient, a simple userspace mount call will fix
it:

mount -o remount,rw,usrquota /home

But that's a gamble.

Despite what other posters have said, when the kernel changes the
status of the volume, it does so using kernel-level tools, _not_
userspace mount calls, so the arguments show in the mount(1) command
will _not_ reflect the read-only status of the drive.  If mount(1)
shows that the drive is 'ro', then a person or a program, not the
kernel, has mounted it read-only.

Chris St. Pierre
Unix Systems Administrator
Nebraska Wesleyan University