[Linux-cluster] GFS reserved blocks?

[Kevin Anderson]

For gfs, the recommended solution is to periodically run gfs_tool
reclaim on your filesystems at a time of your choosing.  Depending on
the frequency of your deletes, this might be once a day or once a week.
The only downside is the during the reclaim operation, the filesystem is
locked from other activities.  As the reclaim is relatively fast, this
doesn't really cause a problem.  But scheduling the command to be run
during "idle" times of the day will mitigate the impact.

We attempted to come up with a method of doing this automatically, but
there are deadlock lock issues between gfs and the vfs layer that
prevent it from being implemented.  In addition, there is still the
issue of when is the right time to do the reclaim, and this would be
application specific.

So, just run gfs_tool reclaim if your storage is getting consumed by
metadata storage.

Kevin

On Mon, 2008-10-13 at 15:18 -0500, Jason Huddleston wrote:
> I've been watching mine do this for about two months now. I think it
> started when I upgraded from RHEL 4.5 to 4.6. The app team only has
> about 18 gig used on that 1.7TB drive but they create and delete allot
> of files because that is the loading area they used when new data
> comes in. In the last month I have seen it go up to 70 to 85% used but
> it usually comes back down to about 50% within about 24 hours.
> Hopefully they will find a fix for this soon.
> 
> ---
> Jay
> 
> Shawn Hood wrote: 
> > I actually just ran the reclaim on a live filesystem and it seems to
> > be working okay now.  Hopefully this isn't problematic, as a large
> > number of operations in the GFS tool suite operate on mounted
> > filesystems.
> > 
> > Shawn
> > 
> > On Mon, Oct 13, 2008 at 4:00 PM, Jason Huddleston
> > <jason.huddleston at verizon.com> wrote:
> >   
> > > Shawn,
> > >   I have been seeing the same thing on one of my clusters (shown below)
> > > under Red Hat 4.6. I found some details on this under an article on the
> > > open-shared root web site
> > > (http://www.open-sharedroot.org/faq/troubleshooting-guide/file-systems/gfs/file-system-full)
> > > and an article in Red Hat's knowledge base
> > > (http://kbase.redhat.com/faq/FAQ_78_10697.shtm). It seems to be a bug in the
> > > reclaim of metadata blocks when an inode is released. I saw a patch
> > > (bz298931) released for this in the 2.99.10 cluster release notes but it was
> > > reverted (bz298931) a few days after it was submitted. The only suggestion
> > > that I have gotten back from Red Hat is to shutdown the app so the GFS
> > > drives are not being accessed and then run the "gfs_tool reclaim <mount
> > > point>" command.
> > > 
> > > [root at omzdwcdrp003 ~]# gfs_tool df /l1load1
> > > /l1load1:
> > > SB lock proto = "lock_dlm"
> > > SB lock table = "DWCDR_prod:l1load1"
> > > SB ondisk format = 1309
> > > SB multihost format = 1401
> > > Block size = 4096
> > > Journals = 20
> > > Resource Groups = 6936
> > > Mounted lock proto = "lock_dlm"
> > > Mounted lock table = "DWCDR_prod:l1load1"
> > > Mounted host data = ""
> > > Journal number = 13
> > > Lock module flags =
> > > Local flocks = FALSE
> > > Local caching = FALSE
> > > Oopses OK = FALSE
> > > 
> > > Type           Total          Used           Free           use%
> > > ------------------------------------------------------------------------
> > > inodes         155300         155300         0              100%
> > > metadata       2016995        675430         1341565        33%
> > > data           452302809      331558847      120743962      73%
> > > [root at omzdwcdrp003 ~]# df -h /l1load1
> > > Filesystem            Size  Used Avail Use% Mounted on
> > > /dev/mapper/l1load1--vg-l1load1--lv
> > >                    1.7T  1.3T  468G  74% /l1load1
> > > [root at omzdwcdrp003 ~]# du -sh /l1load1
> > > 18G     /l1load1
> > > 
> > > ----
> > > Jason Huddleston, RHCE
> > > ----
> > > PS-USE-Linux
> > > Partner Support - Unix Support and Engineering
> > > Verizon Information Processing Services
> > > 
> > > 
> > > 
> > > Shawn Hood wrote:
> > >     
> > > > Does GFS reserve blocks for the superuser, a la ext3's "Reserved block
> > > > count"?  I've had a ~1.1TB FS report that it's full with df reporting
> > > > ~100GB remaining.
> > > > 
> > > > 
> > > >       
> > > --
> > > Linux-cluster mailing list
> > > Linux-cluster at redhat.com
> > > https://www.redhat.com/mailman/listinfo/linux-cluster
> > > 
> > >     
> > 
> > 
> > 
> >   
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster