[Fedora-directory-users] FDS SSL performance tuning query

[Satish Chetty]

Jonathan Barber wrote:
> Hello all, currently we have a FDS instance running on RHEL4 with a
> small number of entries (6,000), we also have a linux compute cluster of
> 100 nodes which uses LDAP for user account data (via libnss_ldap).
> 
> nss_ldap on the cluster is configured to use SSL, and everything is fine
> most of the time. However, occasionally, when a large job is started on
> the cluster, the number of connections increases from 100/minute to
> 1600/minute (26/sec).
> 
> This causes the server to become generally unresponsive, and FDS
> especially so (as judged by the time required to retrieve the DSE via
> TLS). Which is a right pain as it causes our samba PDC to timeout and
> everything goes wrong very quickly.
> 
> I can reproducably, impact on FDS performance by running:
> $ getent passwd | cut -d: -f 1 | while read i; do id $i; done

	When you are seeing slow performance on the SSL port, are you also 
seeing slow performance on 389 (non SSL port) too at the same time? 
(ldapsearch for objectclass=posixaccount) Also, is it possible that the 
handshakes and subsequent data transfer is taking time that the OS 
queues up the requests? How about the performance when you do getent 
from localhost vs getent from the client machine?

-Satish.

> 
> across the cluster. When SSL is off, the command to run fine and doesn't
> impact on other searches.
> 
> As a short term measure, we've disabled LDAPS on the cluster nodes,
> which is fine as users don't log into them, but we had planned to expand
> the use of LDAP to cover more hosts (Macs and Linux) that require a 
> confidential channal for authentication. So this experience is giving us
> some trepidation about moving forward with that plan.
> 
> Our system is configured following the guidance of the wiki [0], with a
> maximum of 16834 available file descriptors and 50M of cache (more than
> enough to hold the DB) - and the ratio of cache hits/misses look good
> with little paging out. Running logconv.pl on the access logs doesn't
> show any unindexed searches, so that isn't an issue.
> 
> Our server CPU is a 3Ghz Xeon with 1G of RAM, and looking at the
> performance of NSS 3.2 [1], I would expect the machine to be able to
> setup and tear down many more connections than we are currently seeing.
> Indeed, running the test described in [1] with the nss-3.11.4 binaries,
> I get over 1200 connections per second [2], so it certainly doesn't seem
> to be a problem with NSS.
> 
> This suggests to me that the problem lies in FDS somewhere. So, does
> anyone have any suggestions as to how to improve the SSL/TLS performance
> of FDS, or point me at tuning docs for the SSL side of FDS?
> 
> Cheers.
> 
> [0] http://directory.fedoraproject.org/wiki/Performance_Tuning
> [1] http://www.mozilla.org/projects/security/pki/nss/nss-3.2-performance-results
> [2] server$ ./selfserv -n "Server-Cert" -p 6000
>     client$ time ./strsclnt -p 6000 server -c 1000
>     strsclnt: -- SSL: Server Certificate Validated.
>     strsclnt: 0 cache hits; 1 cache misses, 0 cache not reusable
>     strsclnt: 999 cache hits; 1 cache misses, 0 cache not reusable
> 
>     real    0m0.605s
>     user    0m0.795s
>     sys     0m0.226s