[Freeipa-users] Very slow kerberos performance after upgrade to IPA 2.2

Simo Sorce simo at redhat.com
Tue Jul 31 21:25:58 UTC 2012


On Tue, 2012-07-31 at 21:08 +0200, Sigbjorn Lie wrote:
> On 07/31/2012 01:50 PM, Simo Sorce wrote:
> > On Tue, 2012-07-31 at 10:50 +0200, Sigbjorn Lie wrote:
> >> On Tue, July 31, 2012 10:20, Petr Spacek wrote:
> >>> On 07/30/2012 10:37 PM, Sigbjorn Lie wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>>
> >>>> I've been having performance issues after I upgraded to RHEL 6.3 / IPA 2.2. I
> >>>> still have a LDAP server having unusual high cpu usage even after it's been removed from the SRV
> >>>> records and is serving almost no clients anymore, but it would seem as my main issues is with
> >>>> the kerberos server.
> >>>>
> >>>> All kerberos services are performing very slowly, and the IPA servers has much
> >>>> higher CPU load now then what they had with IPA 2.1. Some services are timing out, like
> >>>> kerberized web servers, other kerberized services perform authentication very slowly. I had to
> >>>> switch our automounter away from kerberos authentication as it is no longer usable.
> >>>>
> >>>> Using SSH to log on to SSSD enabled hosts are also very slow, a login takes
> >>>> anything from 5 seconds up to 20 seconds. Noticably longer than pre IPA 2.2.
> >>>>
> >>>> The IPA web admin interface is definitely not faster than in IPA 2.1.
> >>>>
> >>>>
> >>>> For a comparison, listing out all the folders in an automount map, causing
> >>>> them to be looked up from LDAP and mounted takes over 5 minutes with IPA 2.2 when using kerberos
> >>>> authentication for the automounter. There are approx 130 folders in that automount map.
> >>>>
> >>>> After unmounting all the mounted folders, and changing to using a username and
> >>>> password authentication with a TLS connection, attempting the same operating again, and it now
> >>>> finishes in about 14 seconds for both the lookup from LDAP and the mount operation.
> >>>>
> >>>> After unmounting all the mounted folders again, changing to username and
> >>>> password authentication with a simple unencrypted bind, and then attempting the same operation
> >>>> and it now finishes both lookup and mount in just over 5 seconds!
> >>>>
> >>>> I don't have any timing for kerberized automount pre IPA-2.2, but we we're not
> >>>> talking about several minutes to mount all the folders in this automount map. Unfortunately
> >>>> mounting all the folders is what happens when the users use konqueror to browse the automount
> >>>> maps, so this is a very noticable issue.
> >>>>
> >>>> Even loading a new gnome-terminal or konsole terminal which causes an
> >>>> automount folder to be mounted takes anything between 5 - 15 seconds after the upgrade. There
> >>>> we're no notiable delay when opening a new terminal window pre IPA-2.2.
> >>>>
> >>>>
> >>>> I am not using SSSD for the automounter.
> >>>>
> >>>>
> >>>> I do notice that the dbmodule for the kerberos server has changed from "kldap"
> >>>> to "ipadb.so" Perhaps there is some issues with the new library?
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> Regards,
> >>>> Siggi
> >>>>
> >>>
> >>> Hello,
> >>>
> >>>
> >>> I'm not a Kerberos guy, so I can give only general advice:
> >>> "Overloaded-CPU-problems" can be troubleshooted with OProfile.
> >>>
> >>>
> >>> Oprofile is lightweight statistic profiler (AFAIK it was designed for
> >>> production environment).
> >>>
> >>> Step-by-step documentation for RHEL 6 is available from:
> >>> http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html-single/Deployment_Guide/index.ht
> >>> ml#ch-OProfile
> >>>
> >>> As you can see in section 22.5.1., it allows to break whole CPU usage between
> >>> processes, libraries and even individual symbols (if proper debuginfos are installed).
> >>>
> >>> I recommend to run OProfile on problematic system - results from opreport can
> >>> provide missing clue to us.
> >>>
> >>> OProfile gives best results on bare-metal machines. On virtual machines you
> >>> has to use timer mode in place of hardware performance counters, please see the documentation.
> >>>
> >>>
> >>> Short getting started guide:
> >>> http://oprofile.sourceforge.net/doc/overview.html#getting-started
> >>>
> >>>
> >>> Nice article with theory&&  examples:
> >>> http://people.redhat.com/wcohen/Oprofile.pdf
> >>>
> >>>
> >>> Homepage with a lot of useful information:
> >>> http://oprofile.sourceforge.net/
> >>>
> >>>
> >>>
> >> Thank you.
> >>
> >> All 3 IPA servers are close to idle now after switching from kerberos to user/pwd bind for the
> >> Linux automounter.
> >>
> >> Still there is an issue with kerberos failing to issue a ticket every now and then and it's
> >> responding very slowly.
> >>
> >> There seem to be low activity on this list just now. Is the kerberos people away on vacation?
> > Hi Siggi,
> > some people are on vacation, some are busy covering others :-)
> >
> > Would you be able to take a wireshark trace of an automount going on ?
> > I would like to see precise timing of packets on the wire to make a
> > first assesment of where is the bottleneck.
> >
> > We did change from ldap.so to ipadb.so, but the structure of the drivers
> > is not much different, so I am surprised it would be much slower,
> > however it is possible, I would like to find out what is going on with
> > your help.
> >
> 
> OK, I will get that done when I'm back in the office tomorrow. I suspect 
> it will be somewhat better than my first results as the load on the IPA 
> servers are now much lower when the linux automounters are no longer 
> using kerberos for authentication.
> 
> It seem like there is a race condition going on as the shit didn't hit 
> the fan until the week after the upgrade to IPA 2.2 when people returned 
> to work. The slowness issues then gradually became worse and worse.
> 
> I will send you the captures in a private email. Do you need anything 
> besides TCP 389, 636 and TCP/UDP 88 ?

no need for TCP 636, but it may be intresting to see DNS queries, do you
use the IPA integrated DNS or do you use your own infra ?

Simo.

-- 
Simo Sorce * Red Hat, Inc * New York




More information about the Freeipa-users mailing list