[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [Linux-cluster] Slowness above 500 RRDs



On Tue, Jun 12, 2007 at 04:01:04PM +0200, Ferenc Wagner wrote:
> Hi David,
> 
> Sorry if all what follows is misguided nonsense.  I'm eager to learn...
> 
> David Teigland <teigland redhat com> writes:
> 
> > The new code has much better caching in the dlm which will benefit flocks,
> > look at these flock numbers I sent before: [...]
> >
> > This is testing raw flock performance.  The dlm locks for normal file
> > operations should be cached and locally mastered also, so I'm not sure
> > what's causing the long times.  Make sure that drop_count is zero again,
> > now it's in sysfs:
> >   echo 0 > /sys/fs/gfs/<foo>:<bar>/lock_module/drop_count
> >
> > Also, mount debugfs so we can check some stuff later:
> >   mount -t debugfs none /sys/kernel/debug
> >
> > Then run some tests:
> > - mount on nodeA
> > - run the test on nodeA
> > - count locks on nodeA
> >   (cat /sys/kernel/debug/dlm/<bar> | grep Master | wc -l)
> > - mount on nodeB (don't do anything on this node)
> > - run the test again on nodeA
> > - count locks on nodeA and nodeB (see above)
> > - mount on nodeC (don't do anything on nodes B or C)
> > - run the test again on nodeA
> > - count locks on nodes A, B and C (see above)
> >
> > We're basically trying to produce the best-case performance from one node,
> > nodeA.  That means making sure that nodeA is mastering all locks and doing
> > maximum caching.  That's why it's important that we not do anything at all
> > that accesses the fs on nodes B or C, or do any extra mounts/unmounts.
> 
> I made all the above tests and composed the reply a long time ago, but
> now, getting back to it after that long time, I decided to satisfy your
> curiosity, behold...
> 
> > Plocks will be much slower and are probably not interesting to test, but
> > I'm curious if you added the "-l0" option to gfs_controld?  That option
> > turns off the code that intentionally limits the rate of plocks.  See the
> > old results again: [...]
> 
> Now, that switch makes ALL the difference.  With a single node
> switched on, I get results like this (with abbreviated strace -c
> output appended):
> 
> without -l0:
> 
> filecount=500
>   iteration=0 elapsed time=10.444446 s
>   iteration=1 elapsed time=9.693618 s
>   iteration=2 elapsed time=10.520073 s
>   iteration=3 elapsed time=10.521504 s
>   iteration=4 elapsed time=10.520183 s
> total elapsed time=51.699824 s
> Process 5265 detached
> % time     seconds  usecs/call     calls    errors syscall
> ------ ----------- ----------- --------- --------- ----------------
>  83.27    0.048525           6      7551           read
>   6.73    0.003923           2      2502           fcntl64
>   4.47    0.002606           1      2528           close
>   3.09    0.001801           1      2551        23 open
>   0.74    0.000432           0      2507           write
>   0.71    0.000415           0      5033           mmap2
>   0.41    0.000237           0     12528         3 _llseek
>   0.31    0.000178           0      5001           munmap
>   0.18    0.000107           0      5015           fstat64
>   0.08    0.000049           0      2506           gettimeofday
>   0.00    0.000000           0        16        14 ioctl
>   0.00    0.000000           0       202       182 stat64
> ------ ----------- ----------- --------- --------- ----------------
> 100.00    0.058273                 47974       229 total
> 
> with -l0:
> 
> filecount=500
>   iteration=0 elapsed time=5.966146 s
>   iteration=1 elapsed time=0.582058 s
>   iteration=2 elapsed time=0.528272 s
>   iteration=3 elapsed time=0.936438 s
>   iteration=4 elapsed time=0.528147 s
> total elapsed time=8.541061 s
> Process 10030 detached
> % time     seconds  usecs/call     calls    errors syscall
> ------ ----------- ----------- --------- --------- ----------------
>  57.17    0.016527           2      7551           read
>  21.49    0.006213           2      2528           close
>   8.16    0.002358           1      2502           fcntl64
>   6.59    0.001904           1      2551        23 open
>   2.21    0.000638           0      2507           write
>   1.46    0.000421           0      5033           mmap2
>   0.86    0.000249         249         1           execve
>   0.73    0.000212           0      5001           munmap
>   0.65    0.000187           0     12528         3 _llseek
>   0.57    0.000165           0      5015           fstat64
>   0.12    0.000034           0      2506           gettimeofday
>   0.00    0.000000           0        16        14 ioctl
>   0.00    0.000000           0       202       182 stat64
> ------ ----------- ----------- --------- --------- ----------------
> 100.00    0.028908                 47974       229 total
> 
> Looks like the bottleneck isn't the explicit locking (be it plock or
> flock), but something else, like the built-in GFS locking.

I'm guessing that these were run with a single node in the cluster?  The
second set of numbers (with -l0) wouldn't make much sense otherwise.  I
think if you add nodes to the cluster, the -l0 numbers will go up quite a
bit.  In the end I expect that flocks are still going to be the fastest
for you.

> Similar dramatic speedup can be achieved (with a single node switched
> on, again), by the lockproto=lock_nolock mount option, even if used
> together with ignore_local_fs.  It I understand it right, this
> combination leaves the cluster-wide [pf]locks alone, just eliminates
> the GFS internal locking, which guards the internal consistency of the
> file system (please correct me if I'm wrong).

With nolock there is no cluster (lock_nolock just returns 0 for
everything), so the cluster-wide [pf]locks have zero cost.  So this test
doesn't tell you anything.

> What's strange, is that gfs_controld -l0 seems like a perfectly safe
> invocation (what's the catch, ie. why was the artifical limit
> introduced?), 

The rate limit was introduced to prevent bad programs from flooding the
network with plock operations.  It may not be a very real problem, though,
so we might eventually disable it (-l0) by default.

> still it achieves almost the same speedup like using
> lock_nolock, which would be a disaster with more than one node
> mounting the fs.  (Also this trick scales pretty well to 4000 files.)

No, -l0 is not going to give you the performance of nolock.  I think you
must have been running with a single node in the cluster.  In that case
there are no other nodes to send/recv messages to/from, so the plock
messages are very fast.

> Again, the above tests were done with a single node switched on, and
> I'm not sure whether the results carry over to the real cluster setup,
> will test is soon.  

Ah, yep.  When you add nodes the plocks will become much slower.  Again, I
think you'll have better luck with flocks.

> I didn't touch drop_count either, everything was
> left as default, except for the mount options and the -l option.
> 
> Also, I can send the results of the scenario suggested by you, if it's
> still relevant.  In short: the locks are always mastered on node A
> only, but the performance is poor nevertheless.

Poor even in the first step when you're just mounting on nodeA?

Dave


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]