[Freeipa-users] Slow ipa performance -- why so many ldap lookups ?

[Jakub Hrozek]

On Tue, Mar 19, 2013 at 11:05:14PM +0100, Jan-Frode Myklebust wrote:
> On Tue, Mar 19, 2013 at 10:01:16PM +0100, Jakub Hrozek wrote:
> > Hello Jan,
> > I'm sorry you're seeing performance problems. 
> 
> We have been struggeling with performance and crashes for a while now.
> Have had one crash were a replica dropped it's whole database, and a
> couple of hangs probably caused by clients flooding 389ds. 
> (ref ticket 00799719 and 00800931)..
> 
> > You should really use the ipa backend for better performance as it uses
> > the memberof attribute (and a couple of other shortcuts to be able to
> > tell if a missing member is a user or a group based on the format of the
> > DN for example).
> 
> Sure, we very much want to use the IPA backend, but the LDAP backend seems
> to have been working better for us. More robust. It might have been caused
> by one of the ipa-servers running with too high error-log-level, but we've
> too often seen users not getting their groups populated with the IPA
> backend -- while this has never happened with LDAP backend. We fixed the
> error-log-level today, and have moved our lab-servers over to
> ipa-backend. Will see in a few days if the problem is fixed now.
> 

This really sounds like a bug. If you encounter a situation like this,
where a group does not show all its members, feel free to open a bug.

> > 
> > > What we find very strange in the trace is:
> > > 
> > > 	- how many ldap searches are done (144!)
> > 
> > The number really depends on the group structure and nesting levels.
> 
> We have a few nested groups.. but sgallagh explained to me that this
> large number of lookups was caused by me testing using "id", which
> "calls 'initgroups()' followed by a loop of 'getgrgid()' for every group
> you are a member of" and the getgrgid() needs to fetch all members of
> each group.

Yes, on the other hand, with a cold cache, the SSSD should be able to
just download all group members in one go using the dereference control.
I'm not completely sure, but I would presume even this dereference
lookup would show up as a single search in wireshark, not look them up
individually.

> 
> > 
> > > 	- that nesting is handled by the client, instead of using
> > > 	  "memberOf".
> > 
> > I'm sorry, I don't quite understand the problem here. If ipa backend was
> > used, then all groups would be resolved in a single search by fetching
> > the objects the memberof attribute points at.
> 
> I was expecting the same to be done with RFC2307bis, but found no way of
> telling it to use memberof instead of un-nesting all groups by itself.
> 
> > 
> > On the other hand, the RFC2307bis schema does not guarantee there is a
> > memberof attribute at all, so the client has to perform multiple queries
> > based on the member attribute. This is one of the prime reasons to stick to
> > the ipa backend as opposed to the LDAP back end with the RFC2307bis schema.
> 
> Ok.
> 
> > 
> > > 	- that all group members are searched individually, and multiple
> > > 	  times if they're members of multiple groups
> > > 
> > 
> > They shouldn't be fetched multiple times, sounds like a bug to me. How
> > did you measure this metric? Wireshark lookups? 
> 
> Wireshark lookups. Ref. trace attached to my previous message.
> 

Yes, sorry, I saw the attachment after I sent my initial reply :-)

> > 
> > Can you tell us a little bit about your nesting structure? How many
> > users, how many groups, how deep is the nesting?
> 
> 305 users.
> 58 groups.
> I belive there are max 2 levels of nesting group1(group2(group3)).
> 
> 
> > By the way, the "id" command is not really a fair benchmark as, contrary
> > to the initgroups() operation that happens during a login, also fetches
> > all the group members. If you are seeing slow logins, then the best way
> > to benchmark the initgroups is "id -G", not "id".
> 
> "sudo" will typically hang for many seconds before giving the password-prompt,
> and this delay seems to have been approximately the same as the delay we
> see with "id". Guess that's why I found it to be a good benchmark for
> the performance problems we see.

OK, sudo might need the group members, fair point. I was thinking
logins previously.

Bottom line, if you are seeing inconsistent results with ipa backend,
please open a bug. This is something that would need fixing right away.