RHEL connundrum with df and du

[Simon Alman]

Hi All

I am having an issue with disk space and since it is happening using an
ext3 formatted partition I felt that this would be the most appropriate
list to post to.

The problem is this; I have access to two RHEL systems (one running RHEL3
and one running
RHEL4). Both are in the same company and both exhibit the same problem
that I have not seen anywhere else before.

Both have full / partitions. df shows this to be the case so it must be
true ... right ?

Well du disagrees, on one system with a full 20GB / parition that df shows
to be full, df can only find 3.6GB of files.

So off to the redhat FAQ I went and found this:
http://kbase.redhat.com/faq/FAQ_35_5209.shtm

Great that explains the problem precisely ... except it didn't help. No
processes were holding large deleted files in a locked state.

So I looked at inodes thinking that they may have all been used up ...
they haven't df shows only 5% inode usage.

I forced an fsck run on the partition on reboot and neither this nor the
reboot helped, fsck shows clean and after the reboot things were still
broken.

So I'm sat here trying to explain to the client why their 5k worth of dell
server is currently no much use to them (I can't install anything on it
due to the space issue ...).

Has anyone come across anything similar before ? I've worked with linux
for six years and this is a new one on me ... and twice in the same
company. I suspect major kernel b0rkage since both systems use Dell's RHEL
build for the specific model of server but proving it is beyond me right
now.

Any help/advice would be very gratefully appreciated.

Kind regards

Simon Alman