[Freeipa-users] IPA Startup issues

[Sigbjorn Lie]

On 05/16/2011 04:56 PM, Rich Megginson wrote:
> On 05/16/2011 08:43 AM, Sigbjorn Lie wrote:
>> On 05/16/2011 03:52 PM, Simo Sorce wrote:
>>> On Sat, 2011-05-14 at 16:46 +0200, Sigbjorn Lie wrote:
>>>> I've noticed that if the machine running IPA is very busy at startup,
>>>> the IPA services will not be online when the machine is started.
>>>>
>>>> I noticed this is as my test virtualization host has had it's power 
>>>> cord
>>>> knocked out a few times. When I restart the host machine, all the
>>>> virtual machines is started at the same time, causing (a lot) higher
>>>> than normal latency for each virtual machine.
>>>>
>>>> This causes the IPA daemons to start, while during the startup one or
>>>> several IPA daemons fails due to dependencies of other daemons 
>>>> which is
>>>> not started yet, and all the IPA daemons is stopped as not all the IPA
>>>> daemons started successfully. I've noticed that the default 
>>>> behavior of
>>>> the ipactl command is to shut down all the IPA daemons, if any of the
>>>> IPA daemons should fail during startup.
>>>>
>>>> This can be seen in the logs of the individual services, as some is
>>>> started successfully, just to receive a shutdown signal shortly after.
>>>> It seem to be the pki-ca which shut down my IPA services this morning.
>>>>
>>>> When rebooting the virtual machine running the IPA daemons during 
>>>> normal
>>>> load of the host machine, all the IPA daemons start successfully.
>>>> Logging on to the IPA server and manually starting the IPA daemons 
>>>> after
>>>> the load of the host machine has decreased also works.
>>>>
>>>> I suggest changing the startup scripts to allow (a lot) longer startup
>>>> times for the IPA daemons prior to failing them.
>>> At the moment we just run service<name>  start and wait until it is
>>> done. If the pki-cad service timeouts and returns an error I think we
>>> need to open a bug against the dogtag component as that is the cause.
>>>
>>> Can you open a bug in the freeipa trac with logs showing that 
>>> service is
>>> responsible for the failure ?
>>
>> I haven't been able to figure out which service that failed IPA yet. 
>> A lot of log files scattered around. As you can see from the slapd 
>> errors file, the slapd daemon was available for almost 3 minutes 
>> before receiving the shutdown signal. I notice now that the PKI 
>> daemon failed 8 seconds after slapd had shut down, so I was wrong in 
>> blaming the PKI daemon.
>>
>> See below for a list of log files I've been trough. They all have on 
>> thing in common, the daemons starts when the host machine is started, 
>> at approx 06:34, then receives a shutdown signal around 06:37. Some 
>> time later when the host has calmed down, I'm logging in and manually 
>> starting IPA using "ipactl start", and all the daemons start without 
>> any problem. And they keep running after my manual intervention.
>>
>> I wish I could be more specific, but I'm unsure where else to look. 
>> Suggestions?
>>
>>
>> /var/log/krb5kdc.log
>> /var/log/pki-ca/catalina.out
>> /var/log/dirsrv/slapd-IX-TEST-COM/errors
>> /var/log/dirsrv/slapd-PKI-IPA/errors
>> /var/log/httpd/error_log
>> /var/log/messages (named log)
>>
>> slapd errors:
>>
>> [14/May/2011:06:33:52 +0200] - 389-Directory/1.2.8.rc1 B2011.062.1416 
>> starting up
>> [14/May/2011:06:33:54 +0200] - Detected Disorderly Shutdown last time 
>> Directory Server was running, recovering database.
> 1) Disorderly Shutdown means a) crash b) kill -9 or similar - neither 
> of which should be happening - is this the replica install or the 
> first master install?



First master install.