[Linux-cluster] GFS volume filled to the brim - "No space left on device" although still data blocks free
Dave Costakos
david.costakos at gmail.com
Sat Jul 26 03:05:43 UTC 2008
I just wanted to confirm that I saw just this exact same issue today on a
stock RHEL 4 Update 5. Running gfs_tool reclaim made my file system "work"
again.
# rpm -qa | grep -e 'GFS\|cman\|magma\|ccs'|sort
ccs-1.0.10-0
ccs-devel-1.0.10-0
cman-1.0.17-0
cman-devel-1.0.17-0
cman-kernel-xenU-2.6.9-50.2
cman-kernheaders-2.6.9-50.2
GFS-6.1.14-0
GFS-kernel-xenU-2.6.9-72.2
magma-1.0.7-1
magma-plugins-1.0.12-0
# uname -rm
2.6.9-55.ELxenU x86_64
2008/7/15 Reiner Rottmann <rottmann at atix.de>
> Hello everyone,
>
> I've experienced strange behavior on a 20 GB GFS formatted volume (although
> same behaviour applies to smaller and larger sizes) when reaching the max
> available disk space by writing lots of 256 byte files in a nested
> directory
> structure (~15k files in one dir).
>
> The expected behaviour would be that all free data blocks are transformed
> to
> inodes and metadata as required but although there are still plenty
> datablocks free, new 256 byte files cannot be created due to "No space left
> on device".
>
> After that, when creating sequential files via touch, it is expected that
> they
> are created till all data blocks are transformed in inodes representing the
> files. When all data blocks are used, "No space left on device" is
> expected.
> But in this strange scenario, files are created at random!?
>
> Also when executing gfs_tool reclaim, new files are createable again. But
> gfs_tool reclaim only should increase the number of already available free
> data blocks by cleaning unused metadata blocks.
>
> In my understanding, it should not be necessary to reclaim blocks, if there
> are still free data blocks left.
>
> Has anyone an explanation for this?
>
> Best regards,
>
> Reiner Rottmann
>
>
> --%<---------------------------------------------------------------------------
> (Filesystem filled with 256 byte files.)
>
> # for i in $(seq 1 1000); do touch waste.$i; done
> touch: cannot touch `waste.3': No space left on device
> touch: cannot touch `waste.6': No space left on device
> touch: cannot touch `waste.12': No space left on device
> touch: cannot touch `waste.13': No space left on device
> touch: cannot touch `waste.15': No space left on device
> touch: cannot touch `waste.16': No space left on device
> touch: cannot touch `waste.20': No space left on device
> touch: cannot touch `waste.25': No space left on device
> touch: cannot touch `waste.28': No space left on device
> touch: cannot touch `waste.29': No space left on device
> touch: cannot touch `waste.32': No space left on device
> touch: cannot touch `waste.37': No space left on device
> touch: cannot touch `waste.38': No space left on device
> touch: cannot touch `waste.39': No space left on device
> touch: cannot touch `waste.48': No space left on device
> touch: cannot touch `waste.55': No space left on device
> touch: cannot touch `waste.56': No space left on device
> touch: cannot touch `waste.59': No space left on device
> touch: cannot touch `waste.60': No space left on device
> touch: cannot touch `waste.63': No space left on device
> ^C
>
> # for i in $(seq 1 1000); do touch waste2.$i; done
> touch: cannot touch `waste2.1': No space left on device
> touch: cannot touch `waste2.8': No space left on device
> touch: cannot touch `waste2.10': No space left on device
> touch: cannot touch `waste2.11': No space left on device
> touch: cannot touch `waste2.12': No space left on device
> touch: cannot touch `waste2.14': No space left on device
> touch: cannot touch `waste2.17': No space left on device
> touch: cannot touch `waste2.19': No space left on device
> touch: cannot touch `waste2.21': No space left on device
> touch: cannot touch `waste2.24': No space left on device
> touch: cannot touch `waste2.28': No space left on device
> touch: cannot touch `waste2.31': No space left on device
> touch: cannot touch `waste2.32': No space left on device
> touch: cannot touch `waste2.33': No space left on device
> touch: cannot touch `waste2.40': No space left on device
> touch: cannot touch `waste2.43': No space left on device
> touch: cannot touch `waste2.44': No space left on device
> touch: cannot touch `waste2.49': No space left on device
> touch: cannot touch `waste2.54': No space left on device
> touch: cannot touch `waste2.55': No space left on device
> touch: cannot touch `waste2.57': No space left on device
> touch: cannot touch `waste2.58': No space left on device
> touch: cannot touch `waste2.61': No space left on device
> ^C
>
> # gfs_tool df .
> /mnt/gfstest:
> SB lock proto = "lock_dlm"
> SB lock table = "axqa01:gfstest"
> SB ondisk format = 1309
> SB multihost format = 1401
> Block size = 1024
> Journals = 3
> Resource Groups = 78
> Mounted lock proto = "lock_dlm"
> Mounted lock table = "axqa01:gfstest"
> Mounted host data = ""
> Journal number = 0
> Lock module flags =
> Local flocks = FALSE
> Local caching = FALSE
> Oopses OK = FALSE
>
> Type Total Used Free use%
> ------------------------------------------------------------------------
> inodes 18343309 18343309 0 100%
> metadata 1690156 1687524 2632 100%
> data 43931 0 43931 0%
>
> # rpm -qa | grep -e 'GFS\|cman\|magma\|ccs'|sort
> GFS-6.1.15-1
> GFS-kernel-2.6.9-60.9
> GFS-kernel-2.6.9-75.11
> GFS-kernel-smp-2.6.9-60.9
> GFS-kernel-smp-2.6.9-75.11
> ccs-1.0.11-1
> cman-1.0.17-0.el4_6.3
> cman-kernel-smp-2.6.9-45.15
> cman-kernel-smp-2.6.9-53.8
> magma-1.0.8-1
> magma-devel-1.0.8-1
> magma-plugins-1.0.12-0
>
> # cat /etc/redhat-release
> Red Hat Enterprise Linux AS release 4 (Nahant Update 6)
>
> # uname -a
> Linux realserver10 2.6.9-67.0.4.ELsmp #1 SMP Fri Jan 18 05:00:00 EST 2008
> x86_64 x86_64 x86_64 GNU/Linux
>
> --%<---------------------------------------------------------------------------
>
> --
> Gruss / Regards,
>
> Dipl.-Ing. (FH) Reiner Rottmann
>
> Phone: +49-89 452 3538-12
>
> http://www.atix.de/
> http://open-sharedroot.org/
>
> PGP Key ID: 0xCA67C5A6
> PGP Key Fingerprint = BF59FF006360B6E8D48F26B10D9F5A84CA67C5A6
>
> **
> ATIX Informationstechnologie und Consulting AG
> Einsteinstr. 10
> 85716 Unterschleissheim
> Deutschland/Germany
>
> Phone: +49-89 452 3538-0
> Fax: +49-89 990 1766-0
>
> Registergericht: Amtsgericht Muenchen
> Registernummer: HRB 168930
> USt.-Id.: DE209485962
>
> Vorstand:
> Marc Grimme, Mark Hlawatschek, Thomas Merz (Vors.)
>
> Vorsitzender des Aufsichtsrats:
> Dr. Martin Buss
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
--
Dave Costakos
mailto:david.costakos at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20080725/179aebe5/attachment.htm>
More information about the Linux-cluster
mailing list