[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [Linux-cluster] Cluster Project FAQ - GFS tuning section


What version of the gfs_mkfs code were you running to get this?
gfs_mkfs -V produced the following results:

gfs_mkfs 6.1.6 (built May 9 2006 17:48:45)
Copyright (C) Red Hat, Inc.  2004-2005 All rights reserved


On 1/11/07, Robert Peterson <rpeterso redhat com> wrote:
Jon Erickson wrote:
> I have a couple of question regarding the Cluster Project FAQ – GFS
> tuning section (http://sources.redhat.com/cluster/faq.html#gfs_tuning).
> First:
> -    Use –r 2048 on gfs_mkfs and mkfs.gfs2 for large file systems.
> I noticed that when I used the –r 2048 switch while creating my file
> system it ended up creating the file system with the 256MB resource
> group size.  When I omitted the –r flag the file system was created
> with 2048Mb resource group size.  Is there a problem with the –r flag,
> and does gfs_mkfs dynamically come up with the best resource group
> size based on your file system size?  Another thing I did which ended
> up in a problem was executing the gfs_mkfs command while my current
> GFS file system was mounted.  The command completed successfully but
> when I went into the mount point all the old files and directories
> still showed up.  When I attempted to remove files bad things
> happened…I believe I received invalid metadata blocks error and the
> cluster went into an infinite loop trying to restart the service.  I
> ended up fixing this problem by un-mounting my file system re-creating
> the GFS file system and re-mounting.  This problem was caused by my
> user error, but maybe there should be some sort of check that
> determines whether the file system is currently mounted.
> Second:
> -    Break file systems up when huge numbers of file are involved.
> This FAQ states that there is an amount of overhead when dealing with
> lots (millions) of files.  What is a recommended limit of files in a
> file system?  The theoretical limit of 8 exabytes for a file system
> does not seem at all realistic if you can't have (millions) of files
> in a file system.
> I just curious to see what everyone thinks about this.  Thanks
Hi Jon,

The newer gfs_mkfs (gfs1) and mkfs.gfs2 (gfs2) in the CVS HEAD will
create the RG size based on the size of the file system if "-r" is not
so that would explain why it used 2048 in the case where you didn't
specify -r.
The previous versions just always assumed 256MB unless -r was specified.

If you specified -r 2048 and it used 256 for its rg size, that would be
a bug.
What version of the gfs_mkfs code were you running to get this?

I agree that it would be very nice if all the userspace GFS-related
tools could
make sure the file system is not mounted anywhere first before running.
We even have a bugzilla from long ago about this regarding gfs_fsck:


It's easy enough to check if the local node (the one running mkfs or fsck)
has it mounted, but it's harder to figure out whether other nodes do because
the userland tools can't assume access to the cluster infrastructure
like the
kernel code can.  So I guess we haven't thought of an elegant solution to
this yet; we almost need to query every node and check its cman_tool
services output to see if it is using resources pertaining to the file
but that would require some kind of socket or connection,
(e.g. ssh) but what should it do when it can't contact a node that's powered
off, etc?

Regarding the number of files in a GFS file system:  I don't have any kind
of recommendations because I haven't studied the exact performance impact
based on the number of inodes.  It would be cool if someone could do some
tests and see where the performance starts to degrade.

The cluster team at Red Hat can work toward improving the performance
of GFS (in fact, we are; hence the change to gfs_mkfs for the rg size),
but many of the performance issues are already addressed with GFS2,
and since GFS2 was accepted by the upstream linux kernel, in a way
I think it makes more sense to focus more of our efforts there.

One thing I thought about doing was trying to use btrees instead of
linked lists for some of our more critical resources, like the RGs and
the glocks.  We'd have to figure out the impact of doing that; the overhead
to do that might also impact performance.  Just my $0.02.


Bob Peterson
Red Hat Cluster Suite

Linux-cluster mailing list
Linux-cluster redhat com


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]