[Freeipa-users] 'Request is a replay'

[Sigbjorn Lie]

On Wed, July 25, 2012 09:54, Sigbjorn Lie wrote:
> On Tue, July 24, 2012 20:29, Simo Sorce wrote:
>
>> On Tue, 2012-07-24 at 10:22 +0200, Sigbjorn Lie wrote:
>>
>>
>>> Hi,
>>>
>>>
>>>
>>> I keep seing this error message in our production environment "Request is a replay" in
>>> variuos services using kerberos like ssh, sssd, automounter, squid +++ after the upgrade to
>>> RHEL 6.3 /
>>> IPA
>>> 2.2.
>>>
>>>
>>>
>>>
>>> Jul 24 10:16:11 server027 sssd_be: GSSAPI Error: Unspecified GSS failure.  Minor code may
>>> provide more information (Request is a replay)
>>>
>>> Seaching google seem to suggest that this is an error with time. However we have NTP
>>> configured (IPA servers as NTP servers) which is synchronized to external NTP servers. There
>>> has been no issue before, and I cannot find issue with the time being out of sync on the
>>> machines where this is happening.
>>
>> This error usually appears only when a same request is found in the
>> replay cache. It shouldn't be related to time issues, in that case you usually get clock-skew.
>>
>> Can you tell me what operation was being performed by sssd when you
>> caught that error ? Can you check if immediately before another identical operation had been
>> performed ?
>>
>
> That being said, I do have 1 IPA server (out of 3) that has significantly higher CPU usage than
> the other 2, the 15-minute load average is sitting at between 0.85 and 0.95 the entire day, where
> ns-slapd 389-ds process is running at 100% most of the time.
>
> Load: 1.02, 0.94, 0.87
>
>
> In comparison the other two IPA servers has a 15-minute average between 0.10 - 0.30 throughout
> the day, and the ns-slapd process is far from being such a cpu hog.
>
> On the server having high load, running even a command such as "ipactl status" can take up to 20
> seconds to complete, where "Directory Service: RUNNING" returns after a second or so, and to list
> the rest of the services takes the remainding 19 seconds.
>
> Also the web interface on this particular IPA server is rendered unusable, returning "Limits
> exceeded for the query" for almost any action.
>
> Restarting all the IPA servies (ipactl restart) on the problematic host soemwhat improves the
> situation, however that particular server returns to having heavy load quickly.
>
> Using logconv.pl to analyze the dirsrv access log file displays that the server in question has
> the lowest search queries per min with 106 queries/min. The other servers have 710 search
> queries/sec and 168 queries/sec.
>
> For modifications all the IPA servers has about 5-6 queries/sec. For unindexed searches the
> problematic server is the server with the lowest number. It does however have more than twice the
> amount of GSSAPI binds than the other servers with over 61000 GSSAPI binds over a 17 hour period.
>
>
> The problematic server is a physical server with 2 x AMD 2.4GHz Quad core CPU and 8GB of RAM.
>
>
> This issue is also impacting all the clients, where I see random hangs with anything involving a
> ldap or kerberos query to the IPA servers.
>
> Any suggestions?
>
>

Anyone ?

I am starting to see the Replay error when using the "ipa" CLI tool as well, causing the request
to drop out in an error.

ipa dnsrecord-show example.com hostname
ipa: ERROR: Local error: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor
code may provide more information (Request is a replay)


Rgds,
Siggi