[Fedora-directory-users] FDS SSL performance tuning query

[Jonathan Barber]

Hello all, currently we have a FDS instance running on RHEL4 with a
small number of entries (6,000), we also have a linux compute cluster of
100 nodes which uses LDAP for user account data (via libnss_ldap).

nss_ldap on the cluster is configured to use SSL, and everything is fine
most of the time. However, occasionally, when a large job is started on
the cluster, the number of connections increases from 100/minute to
1600/minute (26/sec).

This causes the server to become generally unresponsive, and FDS
especially so (as judged by the time required to retrieve the DSE via
TLS). Which is a right pain as it causes our samba PDC to timeout and
everything goes wrong very quickly.

I can reproducably, impact on FDS performance by running:
$ getent passwd | cut -d: -f 1 | while read i; do id $i; done

across the cluster. When SSL is off, the command to run fine and doesn't
impact on other searches.

As a short term measure, we've disabled LDAPS on the cluster nodes,
which is fine as users don't log into them, but we had planned to expand
the use of LDAP to cover more hosts (Macs and Linux) that require a 
confidential channal for authentication. So this experience is giving us
some trepidation about moving forward with that plan.

Our system is configured following the guidance of the wiki [0], with a
maximum of 16834 available file descriptors and 50M of cache (more than
enough to hold the DB) - and the ratio of cache hits/misses look good
with little paging out. Running logconv.pl on the access logs doesn't
show any unindexed searches, so that isn't an issue.

Our server CPU is a 3Ghz Xeon with 1G of RAM, and looking at the
performance of NSS 3.2 [1], I would expect the machine to be able to
setup and tear down many more connections than we are currently seeing.
Indeed, running the test described in [1] with the nss-3.11.4 binaries,
I get over 1200 connections per second [2], so it certainly doesn't seem
to be a problem with NSS.

This suggests to me that the problem lies in FDS somewhere. So, does
anyone have any suggestions as to how to improve the SSL/TLS performance
of FDS, or point me at tuning docs for the SSL side of FDS?

Cheers.

[0] http://directory.fedoraproject.org/wiki/Performance_Tuning
[1] http://www.mozilla.org/projects/security/pki/nss/nss-3.2-performance-results
[2] server$ ./selfserv -n "Server-Cert" -p 6000
    client$ time ./strsclnt -p 6000 server -c 1000
    strsclnt: -- SSL: Server Certificate Validated.
    strsclnt: 0 cache hits; 1 cache misses, 0 cache not reusable
    strsclnt: 999 cache hits; 1 cache misses, 0 cache not reusable

    real    0m0.605s
    user    0m0.795s
    sys     0m0.226s
-- 
Jonathan Barber
High Performance Computing Analyst
Tel. +44 (0) 1382 386389