[Freeipa-users] 2.20 dirsrv memory usage

[Stephen Ingram]

On Mon, Jul 16, 2012 at 12:23 PM, Rob Crittenden <rcritten at redhat.com> wrote:
> Stephen Ingram wrote:
>>
>> On Mon, Jul 16, 2012 at 11:34 AM, Rich Megginson <rmeggins at redhat.com>
>> wrote:
>>>
>>> On 07/16/2012 11:48 AM, Stephen Ingram wrote:
>>>>
>>>>
>>>> On Mon, Jul 16, 2012 at 9:35 AM, Rich Megginson<rmeggins at redhat.com>
>>>> wrote:
>>>>>
>>>>>
>>>>> On 07/16/2012 10:19 AM, Stephen Ingram wrote:
>>>>>>
>>>>>>
>>>>>> On Fri, Jul 13, 2012 at 6:14 AM, Rob Crittenden<rcritten at redhat.com>
>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>> Stephen Ingram wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Jul 12, 2012 at 2:59 PM, Steven
>>>>>>>> Jones<Steven.Jones at vuw.ac.nz>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I had huge memory issues pre 6.3, now its low and flat....Sounds
>>>>>>>>> like
>>>>>>>>> you
>>>>>>>>> have an issue somewhere. My normal cpu use is a few hundred
>>>>>>>>> mhz....but
>>>>>>>>> when
>>>>>>>>> "something" goes wrong such as replication failing that
>>>>>>>>> climbs...ditto
>>>>>>>>> memory use....
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Yes, I saw your conversation with Rich on this list about that. And,
>>>>>>>> yes, 6.2 (2.1.3) was bad for me too. I'm not sure why 2.2.0 is still
>>>>>>>> having issues. It was an upgrade from 2.1.3, but the upgrade seemed
>>>>>>>> to
>>>>>>>> complete without issue. I'm also not even doing replication yet so
>>>>>>>> I'm
>>>>>>>> not sure why memory is so high. Web interface is much slower too so
>>>>>>>> perhaps something else is wrong.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Can you tell where it is being slow? Does it seem related to
>>>>>>> retrieving
>>>>>>> data
>>>>>>> from LDAP?
>>>>>>
>>>>>>
>>>>>> I'm not really sure yet what is causing the slowness. I have the same
>>>>>> number of directory entries as before the upgrade. It was very quick
>>>>>> with 2.1.3, but once I upgraded, I felt like I was back to the pre-2.0
>>>>>> days--without a doubt much, much slower.
>>>>>>
>>>>>>> You might check your 389-ds access logs and look for searches with
>>>>>>> notes=U.
>>>>>>> Perhaps you are missing an index.
>>>>>>
>>>>>>
>>>>>> Yes there are lots of notes=U. What does this mean? Was something
>>>>>> missed in the upgrade script?
>>>>>
>>>>>
>>>>> Try running logconv.pl
>>>>
>>>>
>>>> Nice! I'm guessing that notes=U are unindexed searches then. I have 34
>>>> over the last 24 hours so I'm not sure this would be causing the issue
>>>> as the slowness persists through every click.
>>>
>>>
>>> Yeah, I would expect to see a lot more than 34 if that were the cause.
>>>
>>> Can you post the search filters that are unindexed?
>>
>>
>> Sure, here's a partial list (sanitized):
>>
>>
>> filter="(managedBy=fqdn=ec2-x.x.x.us-west-2.compute.amazonaws.com,cn=computers,cn=accounts,dc=example,dc=com)
>> attrs="fqdn"
>>
>> filter="(managedBy=fqdn=imap.example.com,cn=computers,cn=accounts,dc=example,dc=com)"
>> attrs="fqdn"
>>
>> filter="(managedBy=fqdn=imap1.example.com,cn=computers,cn=accounts,dc=example,dc=com)"
>> attrs="fqdn"
>>
>> filter="(managedBy=fqdn=imap2.example.com,cn=computers,cn=accounts,dc=example,dc=com)"
>> attrs="fqdn"
>>
>> filter="(managedBy=fqdn=ipa1.example.com,cn=computers,cn=accounts,dc=example,dc=com)"
>> attrs="fqdn"
>>
>> All the rest are the same, just with different hosts.
>>
>>>> I've traced the
>>>> unindexed searches back to the time of Web UI access and they don't
>>>> match. I also don't see any other obvious errors when running
>>>> logconv.pl.
>>>>
>>>> One strange thing I have noticed is that the 389 server logs seem to
>>>> update in "spurts". If I'm tailing the logs while I access a Web UI
>>>> page, there is nothing, then a couple of seconds later, I see the logs
>>>> quickly scroll with new entires. Has this always been the case? I
>>>> don't seem to remember this before.
>>>
>>>
>>> Yes.  The 389 access log is buffered, for performance reasons.
>>
>>
>> Just thought it might be relevant. I'm not sure what is causing the
>> extreme slowness. I've also shut off memcached and tried without it
>> with no discernible difference. The directory seems to be handling the
>> load of external queries just fine, although I'm not sure I've solved
>> the memory issue--I'm still testing with the compat plugin disabled to
>> see if I can stop the memory creep. Maybe it's something in the code
>> of the Web UI itself as its even slow when changing from page to page
>> of users and hosts.
>
>
> Shutting of memcached is just going to make things even slower.

I really didn't see much difference so I turned it back on right away.

> Things you can try on a quiet system:
>
> 1. Create /etc/ipa/server.conf:
>
> [global]
> debug=True
>
> Restart Apache
>
> Use the UI to do a few things. Verify in the logs that the session cache is
> being used.

Yes, it is. It's interesting, 2.2 is slower such that you can see the
frame load, and then the loading symbol spins below in the display
area for a few seconds while that loads up. Before, with 2.1.3, you
really couldn't distinguish the two as they loaded so quickly.

> 2. Check your browser configuration. You don't need delegation-uris set any
> more. Having it set might mask other problems (you still need
> negotiation-auth.trusted-uris set).

I forgot about this. I changed it, completely cleared the browser
cache and accessed without any noticeable difference.

> 3. Watch what URI is being used in the Apache access log. It should be
> /ipa/session/json.

Check. this is where it lands.

I'm beginning to think this is just the Web UI itself instead of 389
although it is really difficult to tell. I've poured over the debug
logs and didn't see anything that caused me concern.

It's certainly usable, but I just got really spoiled by the
unbelievable quickness of 2.1.3. When your release notes indicate it
should be faster, what are you comparing it to? Maybe this only
happens with upgraded instances and not fresh installs.

Steve