[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [Linux-cluster] GFS volume filled to the brim - "No space left on device" although still data blocks free



I just wanted to confirm that I saw just this exact same issue today on a stock RHEL 4 Update 5.  Running gfs_tool reclaim made my file system "work" again.

# rpm -qa | grep -e 'GFS\|cman\|magma\|ccs'|sort
ccs-1.0.10-0
ccs-devel-1.0.10-0
cman-1.0.17-0
cman-devel-1.0.17-0
cman-kernel-xenU-2.6.9-50.2
cman-kernheaders-2.6.9-50.2
GFS-6.1.14-0
GFS-kernel-xenU-2.6.9-72.2
magma-1.0.7-1
magma-plugins-1.0.12-0

# uname -rm
2.6.9-55.ELxenU x86_64




2008/7/15 Reiner Rottmann <rottmann atix de>
Hello everyone,

I've experienced strange behavior on a 20 GB GFS formatted volume (although
same behaviour applies to smaller and larger sizes) when reaching the max
available disk space by writing lots of 256 byte files in a nested directory
structure (~15k files in one dir).

The expected behaviour would be that all free data blocks are transformed to
inodes and metadata as required but although there are still plenty
datablocks free, new 256 byte files cannot be created due to "No space left
on device".

After that, when creating sequential files via touch, it is expected that they
are created till all data blocks are transformed in inodes representing the
files. When all data blocks are used, "No space left on device" is expected.
But in this strange scenario, files are created at random!?

Also when executing gfs_tool reclaim, new files are createable again. But
gfs_tool reclaim only should increase the number of already available free
data blocks by cleaning unused metadata blocks.

In my understanding, it should not be necessary to reclaim blocks, if there
are still free data blocks left.

Has anyone an explanation for this?

Best regards,

Reiner Rottmann

--%<---------------------------------------------------------------------------
(Filesystem filled with 256 byte files.)

# for i in $(seq 1 1000); do touch waste.$i; done
touch: cannot touch `waste.3': No space left on device
touch: cannot touch `waste.6': No space left on device
touch: cannot touch `waste.12': No space left on device
touch: cannot touch `waste.13': No space left on device
touch: cannot touch `waste.15': No space left on device
touch: cannot touch `waste.16': No space left on device
touch: cannot touch `waste.20': No space left on device
touch: cannot touch `waste.25': No space left on device
touch: cannot touch `waste.28': No space left on device
touch: cannot touch `waste.29': No space left on device
touch: cannot touch `waste.32': No space left on device
touch: cannot touch `waste.37': No space left on device
touch: cannot touch `waste.38': No space left on device
touch: cannot touch `waste.39': No space left on device
touch: cannot touch `waste.48': No space left on device
touch: cannot touch `waste.55': No space left on device
touch: cannot touch `waste.56': No space left on device
touch: cannot touch `waste.59': No space left on device
touch: cannot touch `waste.60': No space left on device
touch: cannot touch `waste.63': No space left on device
^C

# for i in $(seq 1 1000); do touch waste2.$i; done
touch: cannot touch `waste2.1': No space left on device
touch: cannot touch `waste2.8': No space left on device
touch: cannot touch `waste2.10': No space left on device
touch: cannot touch `waste2.11': No space left on device
touch: cannot touch `waste2.12': No space left on device
touch: cannot touch `waste2.14': No space left on device
touch: cannot touch `waste2.17': No space left on device
touch: cannot touch `waste2.19': No space left on device
touch: cannot touch `waste2.21': No space left on device
touch: cannot touch `waste2.24': No space left on device
touch: cannot touch `waste2.28': No space left on device
touch: cannot touch `waste2.31': No space left on device
touch: cannot touch `waste2.32': No space left on device
touch: cannot touch `waste2.33': No space left on device
touch: cannot touch `waste2.40': No space left on device
touch: cannot touch `waste2.43': No space left on device
touch: cannot touch `waste2.44': No space left on device
touch: cannot touch `waste2.49': No space left on device
touch: cannot touch `waste2.54': No space left on device
touch: cannot touch `waste2.55': No space left on device
touch: cannot touch `waste2.57': No space left on device
touch: cannot touch `waste2.58': No space left on device
touch: cannot touch `waste2.61': No space left on device
^C

# gfs_tool df .
/mnt/gfstest:
 SB lock proto = "lock_dlm"
 SB lock table = "axqa01:gfstest"
 SB ondisk format = 1309
 SB multihost format = 1401
 Block size = 1024
 Journals = 3
 Resource Groups = 78
 Mounted lock proto = "lock_dlm"
 Mounted lock table = "axqa01:gfstest"
 Mounted host data = "">  Journal number = 0
 Lock module flags =
 Local flocks = FALSE
 Local caching = FALSE
 Oopses OK = FALSE

 Type           Total          Used           Free           use%
 ------------------------------------------------------------------------
 inodes         18343309       18343309       0              100%
 metadata       1690156        1687524        2632           100%
 data           43931          0              43931          0%

# rpm -qa | grep -e 'GFS\|cman\|magma\|ccs'|sort
GFS-6.1.15-1
GFS-kernel-2.6.9-60.9
GFS-kernel-2.6.9-75.11
GFS-kernel-smp-2.6.9-60.9
GFS-kernel-smp-2.6.9-75.11
ccs-1.0.11-1
cman-1.0.17-0.el4_6.3
cman-kernel-smp-2.6.9-45.15
cman-kernel-smp-2.6.9-53.8
magma-1.0.8-1
magma-devel-1.0.8-1
magma-plugins-1.0.12-0

# cat /etc/redhat-release
Red Hat Enterprise Linux AS release 4 (Nahant Update 6)

# uname -a
Linux realserver10 2.6.9-67.0.4.ELsmp #1 SMP Fri Jan 18 05:00:00 EST 2008
x86_64 x86_64 x86_64 GNU/Linux
--%<---------------------------------------------------------------------------

--
Gruss / Regards,

Dipl.-Ing. (FH) Reiner Rottmann

Phone: +49-89 452 3538-12

http://www.atix.de/
http://open-sharedroot.org/

PGP Key ID: 0xCA67C5A6
PGP Key Fingerprint = BF59FF006360B6E8D48F26B10D9F5A84CA67C5A6

**
ATIX Informationstechnologie und Consulting AG
Einsteinstr. 10
85716 Unterschleissheim
Deutschland/Germany

Phone: +49-89 452 3538-0
Fax:   +49-89 990 1766-0

Registergericht: Amtsgericht Muenchen
Registernummer: HRB 168930
USt.-Id.: DE209485962

Vorstand:
Marc Grimme, Mark Hlawatschek, Thomas Merz (Vors.)

Vorsitzender des Aufsichtsrats:
Dr. Martin Buss

--
Linux-cluster mailing list
Linux-cluster redhat com
https://www.redhat.com/mailman/listinfo/linux-cluster



--
Dave Costakos
mailto:david costakos gmail com

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]