[Freeipa-users] Very slow kerberos performance after upgrade to IPA 2.2

[Sigbjorn Lie]

On 07/31/2012 01:50 PM, Simo Sorce wrote:
> On Tue, 2012-07-31 at 10:50 +0200, Sigbjorn Lie wrote:
>> On Tue, July 31, 2012 10:20, Petr Spacek wrote:
>>> On 07/30/2012 10:37 PM, Sigbjorn Lie wrote:
>>>
>>>> Hi,
>>>>
>>>>
>>>> I've been having performance issues after I upgraded to RHEL 6.3 / IPA 2.2. I
>>>> still have a LDAP server having unusual high cpu usage even after it's been removed from the SRV
>>>> records and is serving almost no clients anymore, but it would seem as my main issues is with
>>>> the kerberos server.
>>>>
>>>> All kerberos services are performing very slowly, and the IPA servers has much
>>>> higher CPU load now then what they had with IPA 2.1. Some services are timing out, like
>>>> kerberized web servers, other kerberized services perform authentication very slowly. I had to
>>>> switch our automounter away from kerberos authentication as it is no longer usable.
>>>>
>>>> Using SSH to log on to SSSD enabled hosts are also very slow, a login takes
>>>> anything from 5 seconds up to 20 seconds. Noticably longer than pre IPA 2.2.
>>>>
>>>> The IPA web admin interface is definitely not faster than in IPA 2.1.
>>>>
>>>>
>>>> For a comparison, listing out all the folders in an automount map, causing
>>>> them to be looked up from LDAP and mounted takes over 5 minutes with IPA 2.2 when using kerberos
>>>> authentication for the automounter. There are approx 130 folders in that automount map.
>>>>
>>>> After unmounting all the mounted folders, and changing to using a username and
>>>> password authentication with a TLS connection, attempting the same operating again, and it now
>>>> finishes in about 14 seconds for both the lookup from LDAP and the mount operation.
>>>>
>>>> After unmounting all the mounted folders again, changing to username and
>>>> password authentication with a simple unencrypted bind, and then attempting the same operation
>>>> and it now finishes both lookup and mount in just over 5 seconds!
>>>>
>>>> I don't have any timing for kerberized automount pre IPA-2.2, but we we're not
>>>> talking about several minutes to mount all the folders in this automount map. Unfortunately
>>>> mounting all the folders is what happens when the users use konqueror to browse the automount
>>>> maps, so this is a very noticable issue.
>>>>
>>>> Even loading a new gnome-terminal or konsole terminal which causes an
>>>> automount folder to be mounted takes anything between 5 - 15 seconds after the upgrade. There
>>>> we're no notiable delay when opening a new terminal window pre IPA-2.2.
>>>>
>>>>
>>>> I am not using SSSD for the automounter.
>>>>
>>>>
>>>> I do notice that the dbmodule for the kerberos server has changed from "kldap"
>>>> to "ipadb.so" Perhaps there is some issues with the new library?
>>>>
>>>>
>>>>
>>>>
>>>> Regards,
>>>> Siggi
>>>>
>>>
>>> Hello,
>>>
>>>
>>> I'm not a Kerberos guy, so I can give only general advice:
>>> "Overloaded-CPU-problems" can be troubleshooted with OProfile.
>>>
>>>
>>> Oprofile is lightweight statistic profiler (AFAIK it was designed for
>>> production environment).
>>>
>>> Step-by-step documentation for RHEL 6 is available from:
>>> http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html-single/Deployment_Guide/index.ht
>>> ml#ch-OProfile
>>>
>>> As you can see in section 22.5.1., it allows to break whole CPU usage between
>>> processes, libraries and even individual symbols (if proper debuginfos are installed).
>>>
>>> I recommend to run OProfile on problematic system - results from opreport can
>>> provide missing clue to us.
>>>
>>> OProfile gives best results on bare-metal machines. On virtual machines you
>>> has to use timer mode in place of hardware performance counters, please see the documentation.
>>>
>>>
>>> Short getting started guide:
>>> http://oprofile.sourceforge.net/doc/overview.html#getting-started
>>>
>>>
>>> Nice article with theory&&  examples:
>>> http://people.redhat.com/wcohen/Oprofile.pdf
>>>
>>>
>>> Homepage with a lot of useful information:
>>> http://oprofile.sourceforge.net/
>>>
>>>
>>>
>> Thank you.
>>
>> All 3 IPA servers are close to idle now after switching from kerberos to user/pwd bind for the
>> Linux automounter.
>>
>> Still there is an issue with kerberos failing to issue a ticket every now and then and it's
>> responding very slowly.
>>
>> There seem to be low activity on this list just now. Is the kerberos people away on vacation?
> Hi Siggi,
> some people are on vacation, some are busy covering others :-)
>
> Would you be able to take a wireshark trace of an automount going on ?
> I would like to see precise timing of packets on the wire to make a
> first assesment of where is the bottleneck.
>
> We did change from ldap.so to ipadb.so, but the structure of the drivers
> is not much different, so I am surprised it would be much slower,
> however it is possible, I would like to find out what is going on with
> your help.
>

OK, I will get that done when I'm back in the office tomorrow. I suspect 
it will be somewhat better than my first results as the load on the IPA 
servers are now much lower when the linux automounters are no longer 
using kerberos for authentication.

It seem like there is a race condition going on as the shit didn't hit 
the fan until the week after the upgrade to IPA 2.2 when people returned 
to work. The slowness issues then gradually became worse and worse.

I will send you the captures in a private email. Do you need anything 
besides TCP 389, 636 and TCP/UDP 88 ?



Rgds,
Siggi