[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [Linux-cluster] GFS reserved blocks?



Sweet. Maybe your notes will save someone else some time. I know it was a great resource for me when I set up my first GFS cluster.

---
Jay

Shawn Hood wrote:
Someone give me write access to the FAQ!  I've been compiling these
undocumented (or hard to find) bits of knowledge for some time now.


Shawn

On Mon, Oct 13, 2008 at 4:29 PM, Kevin Anderson <kanderso redhat com> wrote:
  
For gfs, the recommended solution is to periodically run gfs_tool
reclaim on your filesystems at a time of your choosing.  Depending on
the frequency of your deletes, this might be once a day or once a week.
The only downside is the during the reclaim operation, the filesystem is
locked from other activities.  As the reclaim is relatively fast, this
doesn't really cause a problem.  But scheduling the command to be run
during "idle" times of the day will mitigate the impact.

We attempted to come up with a method of doing this automatically, but
there are deadlock lock issues between gfs and the vfs layer that
prevent it from being implemented.  In addition, there is still the
issue of when is the right time to do the reclaim, and this would be
application specific.

So, just run gfs_tool reclaim if your storage is getting consumed by
metadata storage.

Kevin

On Mon, 2008-10-13 at 15:18 -0500, Jason Huddleston wrote:
    
I've been watching mine do this for about two months now. I think it
started when I upgraded from RHEL 4.5 to 4.6. The app team only has
about 18 gig used on that 1.7TB drive but they create and delete allot
of files because that is the loading area they used when new data
comes in. In the last month I have seen it go up to 70 to 85% used but
it usually comes back down to about 50% within about 24 hours.
Hopefully they will find a fix for this soon.

---
Jay

Shawn Hood wrote:
      
I actually just ran the reclaim on a live filesystem and it seems to
be working okay now.  Hopefully this isn't problematic, as a large
number of operations in the GFS tool suite operate on mounted
filesystems.

Shawn

On Mon, Oct 13, 2008 at 4:00 PM, Jason Huddleston
<jason huddleston verizon com> wrote:

        
Shawn,
  I have been seeing the same thing on one of my clusters (shown below)
under Red Hat 4.6. I found some details on this under an article on the
open-shared root web site
(http://www.open-sharedroot.org/faq/troubleshooting-guide/file-systems/gfs/file-system-full)
and an article in Red Hat's knowledge base
(http://kbase.redhat.com/faq/FAQ_78_10697.shtm). It seems to be a bug in the
reclaim of metadata blocks when an inode is released. I saw a patch
(bz298931) released for this in the 2.99.10 cluster release notes but it was
reverted (bz298931) a few days after it was submitted. The only suggestion
that I have gotten back from Red Hat is to shutdown the app so the GFS
drives are not being accessed and then run the "gfs_tool reclaim <mount
point>" command.

[root omzdwcdrp003 ~]# gfs_tool df /l1load1
/l1load1:
SB lock proto = "lock_dlm"
SB lock table = "DWCDR_prod:l1load1"
SB ondisk format = 1309
SB multihost format = 1401
Block size = 4096
Journals = 20
Resource Groups = 6936
Mounted lock proto = "lock_dlm"
Mounted lock table = "DWCDR_prod:l1load1"
Mounted host data = ""
Journal number = 13
Lock module flags =
Local flocks = FALSE
Local caching = FALSE
Oopses OK = FALSE

Type           Total          Used           Free           use%
------------------------------------------------------------------------
inodes         155300         155300         0              100%
metadata       2016995        675430         1341565        33%
data           452302809      331558847      120743962      73%
[root omzdwcdrp003 ~]# df -h /l1load1
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/l1load1--vg-l1load1--lv
                   1.7T  1.3T  468G  74% /l1load1
[root omzdwcdrp003 ~]# du -sh /l1load1
18G     /l1load1

----
Jason Huddleston, RHCE
----
PS-USE-Linux
Partner Support - Unix Support and Engineering
Verizon Information Processing Services



Shawn Hood wrote:

          
Does GFS reserve blocks for the superuser, a la ext3's "Reserved block
count"?  I've had a ~1.1TB FS report that it's full with df reporting
~100GB remaining.



            
--
Linux-cluster mailing list
Linux-cluster redhat com
https://www.redhat.com/mailman/listinfo/linux-cluster


          


        
--
Linux-cluster mailing list
Linux-cluster redhat com
https://www.redhat.com/mailman/listinfo/linux-cluster
      
--
Linux-cluster mailing list
Linux-cluster redhat com
https://www.redhat.com/mailman/listinfo/linux-cluster

    



  


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]