[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

[Linux-cluster] GFS volume filled to the brim - "No space left on device" although still data blocks free



Hello everyone,

I've experienced strange behavior on a 20 GB GFS formatted volume (although 
same behaviour applies to smaller and larger sizes) when reaching the max 
available disk space by writing lots of 256 byte files in a nested directory 
structure (~15k files in one dir).

The expected behaviour would be that all free data blocks are transformed to 
inodes and metadata as required but although there are still plenty 
datablocks free, new 256 byte files cannot be created due to "No space left 
on device". 

After that, when creating sequential files via touch, it is expected that they 
are created till all data blocks are transformed in inodes representing the 
files. When all data blocks are used, "No space left on device" is expected. 
But in this strange scenario, files are created at random!?

Also when executing gfs_tool reclaim, new files are createable again. But 
gfs_tool reclaim only should increase the number of already available free 
data blocks by cleaning unused metadata blocks.

In my understanding, it should not be necessary to reclaim blocks, if there 
are still free data blocks left.

Has anyone an explanation for this?

Best regards,

Reiner Rottmann

--%<---------------------------------------------------------------------------
(Filesystem filled with 256 byte files.)

# for i in $(seq 1 1000); do touch waste.$i; done
touch: cannot touch `waste.3': No space left on device
touch: cannot touch `waste.6': No space left on device
touch: cannot touch `waste.12': No space left on device
touch: cannot touch `waste.13': No space left on device
touch: cannot touch `waste.15': No space left on device
touch: cannot touch `waste.16': No space left on device
touch: cannot touch `waste.20': No space left on device
touch: cannot touch `waste.25': No space left on device
touch: cannot touch `waste.28': No space left on device
touch: cannot touch `waste.29': No space left on device
touch: cannot touch `waste.32': No space left on device
touch: cannot touch `waste.37': No space left on device
touch: cannot touch `waste.38': No space left on device
touch: cannot touch `waste.39': No space left on device
touch: cannot touch `waste.48': No space left on device
touch: cannot touch `waste.55': No space left on device
touch: cannot touch `waste.56': No space left on device
touch: cannot touch `waste.59': No space left on device
touch: cannot touch `waste.60': No space left on device
touch: cannot touch `waste.63': No space left on device
^C

# for i in $(seq 1 1000); do touch waste2.$i; done
touch: cannot touch `waste2.1': No space left on device
touch: cannot touch `waste2.8': No space left on device
touch: cannot touch `waste2.10': No space left on device
touch: cannot touch `waste2.11': No space left on device
touch: cannot touch `waste2.12': No space left on device
touch: cannot touch `waste2.14': No space left on device
touch: cannot touch `waste2.17': No space left on device
touch: cannot touch `waste2.19': No space left on device
touch: cannot touch `waste2.21': No space left on device
touch: cannot touch `waste2.24': No space left on device
touch: cannot touch `waste2.28': No space left on device
touch: cannot touch `waste2.31': No space left on device
touch: cannot touch `waste2.32': No space left on device
touch: cannot touch `waste2.33': No space left on device
touch: cannot touch `waste2.40': No space left on device
touch: cannot touch `waste2.43': No space left on device
touch: cannot touch `waste2.44': No space left on device
touch: cannot touch `waste2.49': No space left on device
touch: cannot touch `waste2.54': No space left on device
touch: cannot touch `waste2.55': No space left on device
touch: cannot touch `waste2.57': No space left on device
touch: cannot touch `waste2.58': No space left on device
touch: cannot touch `waste2.61': No space left on device
^C

# gfs_tool df .
/mnt/gfstest:
  SB lock proto = "lock_dlm"
  SB lock table = "axqa01:gfstest"
  SB ondisk format = 1309
  SB multihost format = 1401
  Block size = 1024
  Journals = 3
  Resource Groups = 78
  Mounted lock proto = "lock_dlm"
  Mounted lock table = "axqa01:gfstest"
  Mounted host data = ""
  Journal number = 0
  Lock module flags =
  Local flocks = FALSE
  Local caching = FALSE
  Oopses OK = FALSE

  Type           Total          Used           Free           use%
  ------------------------------------------------------------------------
  inodes         18343309       18343309       0              100%
  metadata       1690156        1687524        2632           100%
  data           43931          0              43931          0%
  
# rpm -qa | grep -e 'GFS\|cman\|magma\|ccs'|sort
GFS-6.1.15-1
GFS-kernel-2.6.9-60.9
GFS-kernel-2.6.9-75.11
GFS-kernel-smp-2.6.9-60.9
GFS-kernel-smp-2.6.9-75.11
ccs-1.0.11-1
cman-1.0.17-0.el4_6.3
cman-kernel-smp-2.6.9-45.15
cman-kernel-smp-2.6.9-53.8
magma-1.0.8-1
magma-devel-1.0.8-1
magma-plugins-1.0.12-0

# cat /etc/redhat-release
Red Hat Enterprise Linux AS release 4 (Nahant Update 6)

# uname -a
Linux realserver10 2.6.9-67.0.4.ELsmp #1 SMP Fri Jan 18 05:00:00 EST 2008 
x86_64 x86_64 x86_64 GNU/Linux
--%<---------------------------------------------------------------------------

-- 
Gruss / Regards,

Dipl.-Ing. (FH) Reiner Rottmann

Phone: +49-89 452 3538-12

http://www.atix.de/
http://open-sharedroot.org/

PGP Key ID: 0xCA67C5A6
PGP Key Fingerprint = BF59FF006360B6E8D48F26B10D9F5A84CA67C5A6

**
ATIX Informationstechnologie und Consulting AG
Einsteinstr. 10 
85716 Unterschleissheim
Deutschland/Germany

Phone: +49-89 452 3538-0
Fax:   +49-89 990 1766-0

Registergericht: Amtsgericht Muenchen
Registernummer: HRB 168930
USt.-Id.: DE209485962

Vorstand: 
Marc Grimme, Mark Hlawatschek, Thomas Merz (Vors.)

Vorsitzender des Aufsichtsrats:
Dr. Martin Buss

Attachment: signature.asc
Description: This is a digitally signed message part.


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]