authentication hangs when disk > 80% full

[Stephen Jascourt]

I discovered that PAM authentication hangs in at least 4 different 
kernels of Enterprise Linux, including one installed last week, when the 
partition containing /var is more than 80% full (e.g. less than 20% 
available).

The following appears in the /var/log/messages file:
audbin[2673]: saving binary audit log /var/log/audit.d/bin.2
audbin[2673]: threshold 20.00 exceeded for filesystem /var/log/audit.d/. 
- free blocks down to 16.24%
auditd[2130]: Notify command /usr/sbin/audbin -S 
/var/log/audit.d/save.%u -C -T 20% exited with status 1
auditd[2130]: output error
auditd[2130]: output error
auditd[2130]: output error; suspending execution

[numbers in brackets are probably process id numbers or something, they 
change each time; all other parts of the message repeat on each boot 
attempt]

I don't know what auditd does or what it's connection to authentication 
is. What I do know is that when this happens at a time when you have 
open xterm windows, you can still execute commands and use the system, 
but you cannot ssh or sftp or otherwise enter from outside - you can 
connect, but upon entering your password, the connection will hang. 
Similarly, trying to change user from an open window or go to root via 
"su" also hangs upon entering password. Other open windows and processes 
will be unaffected. I don't have any cron scripts, but if they involve 
authentication, they probably would also hang. However, once you end 
your session and have no open windows into the machine, you are locked 
out entirely. A regular boot proceeds normally aside from the above 
messages in the message log, but it does not startx, the screen with the 
color login box (control-alt-F7) does not appear, and logging in from 
the text login screen (such as the screen on control-alt-F1) hangs upon 
entering password, as do attempts to login from outside. Same result for 
logging in as root.

The problem is resolved by reducing disk usage under 80% for the 
partition containing /var. You need to find a bootable disk, mount your 
hard drives, set permissions to write, and delete some files. After 
getting under 80%, everything is fine, shut down and do a normal boot 
and you're back in business.

I am not a system manager or IT person, nor am I a linux expert, just a 
linux user who needed to access his work PC to do work when our local 
linux experts are on their holiday vacation. I could not find any web 
messages or other documentation on this problem, so apparently it has 
not been documented before or it is documented in a place that does not 
show up in web searches.

The problem had been happening intermittently for a while and would 
clear up upon rebooting. I guess I may have been near 80% and booting 
cleaned up all the temporary files bringing me under 80%. I didn't 
troubleshoot the problem, didn't have any clues, so I didn't know about 
the magic 80% threshold. Then I downloaded some large data files and got 
locked out until someone suggested I use a bootable CD, mount my hard 
drives, and someone else suggested looking in /var/log/messages, where I 
found the above lines and tried deleting files. Once I got under 80%, it 
was fine.

There may be some parameter in some configuration file that sets the 20% 
threshold for available disk space required, but I don't know where that 
would be. If so, I could lower it to say 1% or something small. Another 
solution is when configuring a new system, you could partition /var by 
itself with only the amount of space it will need and run a daily script 
to monitor space in /var and send a warning message when space available 
starts getting close to 20%. But it would be much better if folks who 
work on linux source code could just fix the problem.

Stephen Jascourt
UCAR/COMET meteorologist at National Weather Service Headquarters