[Linux-cluster] rm -r on gfs2 filesystem is very slow

[yu song]

HI,

I have been seeing a similar GFS2 performance issue.

when I did a file copy (4.3G) from a node to another node's gfs2 filesystem,
it shows me *793.4KB/s* . however, same file copy to the same node but non
gfs2 file system, it gives me *53.6MB/s*.

not too sure why? it seems like gfs2 has a performance issue here.

appreciate if anyone can tell me what I can tune to get it better.


Yu

On Sat, Jul 11, 2009 at 1:56 AM, Steven Whitehouse <swhiteho at redhat.com>wrote:

> Hi,
>
> On Fri, 2009-07-10 at 08:49 -0700, Peter Schobel wrote:
> > The initial writing is done via the network by checking out source
> > trees from a Perforce repository. Beyond that, source trees are
> > compiled causing the creation of many object files.
> >
> > Multiple source trees will be compiled from the same node or from
> > multiple nodes.
> >
> > This performance problem exhibits itself even when using a single
> > node. Writing to the filesystem seems to work fine. The time to do a
> > cp -r dir /gfs/dir is very comparable to writing to local disk
> > however, rm -r /gfs/dir takes considerably longer than it does on
> > local disk. I am guessing this is a feature of dlm checking for a lock
> > on each individual file but I'm not sure.
> >
> > Peter
>
> Partly that is the case. There are some things which can be done to
> improve performance in the deallocation area, and so that is likely to
> improve in future. The main issue is to ensure that we continue to
> maintain the correct locking order in that code. It can be complex since
> it involves the inode lock, transaction lock, and (maybe) multiple
> resource group locks,
>
> Steve.
>
> > ~
> >
> > On Fri, Jul 10, 2009 at 8:27 AM, Steven Whitehouse<swhiteho at redhat.com>
> wrote:
> > > Hi,
> > >
> > > On Fri, 2009-07-10 at 07:42 -0700, Peter Schobel wrote:
> > >> When we did our initial proof of concept, we did not notice any
> > >> performance problem of this magnitude. We were using OS release 2. Our
> > >> QA engineers passed approval on the performance stats of the gfs2
> > >> filesystem and now that we are in deployment phase they are calling it
> > >> unusable.
> > >>
> > >> Have there been any recent software changes that could have caused
> > >> degraded performance or something I may have missed in configuration?
> > >> Are there any tunable parameters in gfs2 that may increase our
> > >> performance?
> > >>
> > > Not that I'm aware of. There are no tunable parameters which might
> > > affect this particular aspect of performance, but to be clear exactly
> > > what the issue is, let me ask a few questions...
> > >
> > >> Our application is very write intensive. Basically we are compiling a
> > >> source tree and running a make clean between builds.
> > >>
> > >> Thanks in advance,
> > >>
> > >> Peter
> > >> ~
> > >>
> > > What is the nature of the writes? Are the different nodes writing into
> > > different directories in the main?
> > >
> > > GFS2 is pretty good at large directories, given certain conditions.
> Look
> > > ups should be pretty fast. Once there is a writer into a particular
> > > directory, then ideally one would take care not to read or write that
> > > directory from other nodes until the writer is finished.
> > >
> > > Directory listing of large directories can be slow, and counts as
> > > reading the directory from a caching point of view. Look ups of
> > > individual files should be fast though,
> > >
> > > Steve.
> > >
> > >
> > >> On Wed, Jul 08, 2009 at 01:58:30PM -0700, Peter Schobel wrote:
> > >>
> > >> >> I am trying to set up a four node cluster but am getting very poor
> > >> >> performance when removing large directories. A directory
> approximately
> > >> >> 1.6G  in size takes around 5 mins to remove from the gfs2
> filesystem
> > >> >> but removes in around 10 seconds from the local disk.
> > >> >>
> > >> >> I am using CentOS 5.3 with kernel 2.6.18-128.1.16.el5PAE.
> > >> >>
> > >> >> The filesystem was formatted in the following manner: mkfs.gfs2 -t
> > >> >> wtl_build:dev_home00 -p lock_dlm -j 10
> > >> >> /dev/mapper/VolGroupGFS-LogVolDevHome00 and is being mounted with
> the
> > >> >> following options: _netdev,noatime,defaults.
> > >> >
> > >> > This is something you have to live with.  GFS(2) works great, but
> with
> > >> > large(r) directories performance is extremely bad and for many
> > >> > applications a real show-stopper.
> > >> >
> > >> > There have been many discussions on this list, with GFS parameter
> tuning
> > >> > suggestions that at least for me didn't result in any improvements,
> with
> > >> > promises that the problems would be solved in GFS2 (I see no
> significant
> > >> > performance improvements between GFS and GFS2), etc.
> > >>
> > >> > --
> > >> > --    Jos Vos <jos at xos.nl>
> > >> > --    X/OS Experts in Open Systems BV   |   Phone: +31 20 6938364
> > >> > --    Amsterdam, The Netherlands        |     Fax: +31 20 6948204
> > >>
> > >
> > > --
> > > Linux-cluster mailing list
> > > Linux-cluster at redhat.com
> > > https://www.redhat.com/mailman/listinfo/linux-cluster
> > >
> >
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20090924/0528bd8a/attachment.htm>