[Linux-cluster] Slowness above 500 RRDs

[David Teigland]

On Tue, Jun 12, 2007 at 06:39:34PM +0200, Ferenc Wagner wrote:
> David Teigland <teigland at redhat.com> writes:
> 
> > On Tue, Jun 12, 2007 at 05:06:56PM +0200, Ferenc Wagner wrote:
> > 
> >> Here is the old mail I haven't sent before.  Meanwhile, I'm switching
> >> in other nodes to continue the tests in my previous mail.
> 
> [...]
> 
> >> But looks like nodeA feels obliged to communicate its locking
> >> process around the cluster.
> >
> > I'm not sure what you mean here.  To see the amount of dlm locking traffic
> > on the network, look at port 21064.  There should be very little in the
> > test above... and the dlm locking that you do see should mostly be related
> > to file i/o, not flocks.
> 
> There was much traffic on port 21064.  Possibly related to file I/O
> and not flocks, I can't tell.  But that's agrees with my speculation,
> that it's not the explicit [pf]locks that take much time, but
> something else.

Could you comment the fcntl/flock calls out of the application entirely
and try it?

> >> # cman_tool services
> >> type             level name     id       state       
> >> fence            0     default  00010001 none        
> >> [1 2 3]
> >> dlm              1     clvmd    00020001 none        
> >> [1 2 3]
> >> dlm              1     test     000a0001 none        
> >> [1 2]
> >> gfs              2     test     00090001 none        
> >> [1 2]
> >
> > !?!? but now you're using the old RHEL4 generation stuff -- gfs_controld
> > is completely irrelevant there.  The analysis completely changes between
> > the RHEL4/RHEL5 (old/new) generations of infrastructure.
> 
> To my best knowledge, I'm using the new infrastructure.  There's no
> cman kernel module loaded, there's no cman process running, there's an
> aisexec process running, syslog contains messages like

Sorry, I was wrong.  I saw 'cman_tool services' which is the old method of
doing group_tool, and forgot that we made cman_tool services call
group_tool.

> How could I be running the old stuff?  Am I totally confused?

You're running the new stuff, it's me who was confused.
Dave