[Freeipa-devel] [PATCH] Patch to allow IPA to work with dogtag 10 on f18
Rob Crittenden
rcritten at redhat.com
Tue Sep 11 19:38:27 UTC 2012
Rob Crittenden wrote:
> Rob Crittenden wrote:
>> Petr Viktorin wrote:
>>> On 09/11/2012 04:38 PM, Rob Crittenden wrote:
>>>> Ade Lee wrote:
>>>>> On Tue, 2012-09-11 at 08:59 -0400, Rob Crittenden wrote:
>>>>>> Petr Viktorin wrote:
>>>>>>> On 09/11/2012 04:04 AM, Ade Lee wrote:
>>>>>>>> On Mon, 2012-09-10 at 16:58 -0400, Rob Crittenden wrote:
>>>>>>>>> Petr Viktorin wrote:
>>>>>>>>>> Attaching rebased and squashed patches. I've done some testing
>>>>>>>>>> with
>>>>>>>>>> them
>>>>>>>>>> but please test some more.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Most of these aren't IPA issues, but dogtag issues. I'll try to
>>>>>>>>> split
>>>>>>>>> them out.
>>>>>>>>>
>>>>>>>>> IPA:
>>>>>>>>>
>>>>>>>>> For the configuration files in install/conf to be updated at rpm
>>>>>>>>> update
>>>>>>>>> time the VERSION needs to be incremented.
>>>>>>>
>>>>>>> These files should stay the same since on upgrade we're still
>>>>>>> using a
>>>>>>> Dogtag 9 style instance. The Dogtag 10 ports are only used in new
>>>>>>> installs.
>>>>>>>
>>>>>>>>> The ipa package lacks any updated dogtag dependencies, so I abused
>>>>>>>>> it.
>>>>>>>
>>>>>>> What should the updated dependencies be? Since it should work with
>>>>>>> both
>>>>>>> dogtag 9 and 10, I don't see how they should change.
>>>>>>
>>>>>> I don't know either, but we need to prevent people from installing
>>>>>> incompatible package combinations.
>>>>>>
>>>>> Would'nt the Conflicts: ipa < 3.0 in pki-ca mentioned below satisfy
>>>>> this
>>>>> requirement? The main concern is that you must have ipa 3.0 if you
>>>>> have
>>>>> dogtag 10.
>>>>>
>>>>> Given that dogtag is consumed by IPA though, it makes more sense to
>>>>> put
>>>>> the relevant conflicts in IPA rather than in dogtag. So in this case,
>>>>> that would mean putting Conflicts: pki-ca >= 10.0 in IPA 2.x.
>>>>> Recall that dogtag 10 will only be officially available in f18+.
>>>>
>>>> That isn't enough. If a F-17 user with IPA 2.2 installed upgrades to
>>>> F-18 they would be able to install dogtag 10 and blow up their IPA
>>>> server.
>>>>
>>>>>
>>>>>>>>> I installed IPA with dogtag 9 and created a replica.
>>>>>>>>>
>>>>>>>>> I updated the IPA bits, that worked fine.
>>>>>>>>>
>>>>>>>>> I updated to dogtag 10 and now the CA doesn't work on the master,
>>>>>>>>> including starting the dogtag instance. Note that the rpm update
>>>>>>>>> process
>>>>>>>>> worked, no notice that the CA service didn't restart.
>>>>>>>>>
>>>>>>>> Did you try to restart the CA with selinux in permissive mode?
>>>>>>>> This is
>>>>>>>> still required right now until I get the selinux policy all
>>>>>>>> straightened
>>>>>>>> out.
>>>>>>>>
>>>>>>>> There is also a separate dogtag ticket (which is currently being
>>>>>>>> worked
>>>>>>>> on) to restart existing dogtag instances when dogtag is upgraded
>>>>>>>> from
>>>>>>>> 9->10.
>>>>>>>
>>>>>>> In permissive mode, this upgrade works for me.
>>>>>>
>>>>>> I was in enforcing mode but saw no AVCs. What is the ETA on fixing
>>>>>> this?
>>>>>>
>>>>>
>>>>> Within the next week or two, I need to finish the IPA merge database
>>>>> patch first, and then co-ordinate changes with the selinux guys.
>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Sometimes I do get this error intermittently:
>>>>>>>
>>>>>>> ipa: ERROR: Certificate operation cannot be completed: Unable to
>>>>>>> communicate with CMS (Service Temporarily Unavailable)
>>>>>>>
>>>>>>> Usually, waiting a couple of minutes clears this up. Perhaps we are
>>>>>>> not
>>>>>>> doing startup detection correctly. Ade mentioned that waiting for
>>>>>>> ports
>>>>>>> may not be ideal. How do we know if Dogtag is initialized?
>>>>>>
>>>>>> Years ago we had discussed with Andrew and Matt creating a URI that
>>>>>> can
>>>>>> be queried to determine dogtag status. I don't think that ever went
>>>>>> anywhere.
>>>>>>
>>>>> Petr, this happens only on reboot, right? And not on regular "service
>>>>> ipa restart"?
>>>
>>> I've now seen it happen right after a 9 → 10 upgrade.
>>>
>>>>> Yeah, I remember this conversation - and even created a bug for it at
>>>>> some point. This went away because the mechanism you were using
>>>>> seemed
>>>>> to be working. The timing may be very different now with tomcat 7 and
>>>>> systemd. I'll open a dogtag trac ticket for this.
>>>>
>>>> Ok.
>>>>
>>>>>
>>>>>>>
>>>>>>>>> Uninstalling failed because it tried to run pkidestroy and not
>>>>>>>>> pkiremove.
>>>>>>>
>>>>>>> I was under the impression that pkidestroy was the correct
>>>>>>> command to
>>>>>>> remove an upgraded instance. I'll check with Ade.
>>>>>>>
>>>>>>>> I'll test this too.
>>>>>>>>
>>>>>>>>> The contents of the file passed to pkispawn should be logged so we
>>>>>>>>> can
>>>>>>>>> see exactly what was passed in.
>>>>>>>>>
>>>>>>>> Its a pretty big file. You might want to only log the
>>>>>>>> modifications.
>>>>>>>> Or save the file somewhere.
>>>>>>>
>>>>>>> Our logs are pretty verbose, so that shouldn't be a problem. I'll
>>>>>>> put it
>>>>>>> in the next version of the patch.
>>>>>>
>>>>>> The question to ask is: would you need the contents of this file if
>>>>>> all
>>>>>> you got were logs and needed to evaluate why installation failed? In
>>>>>> most cases this is yes.
>>>>>>
>>>>> Up to you guys. There is a patch I am working on in which I will be
>>>>> logging the object that is being passed to the server from pkispawn.
>>>>> That - and the diffs to the standard config file as I mentioned
>>>>> above -
>>>>> will likely be sufficient to debug almost all cases.
>>>>>
>>>>> Make sure not to log any passwords.
>>>>>
>>>
>>> Thanks for the catch. Attaching updated patch that sanitizes the
>>> passwords.
>>>
>>>>>>>>> DOGTAG:
>>>>>>>>>
>>>>>>>>> When upgrading using the dogtag-devel repo I had to specify
>>>>>>>>> pki-tools.x86_64 otherwise it tried to install both 32 and 64-bit
>>>>>>>>> versions (and failed).
>>>>>>>>>
>>>>>>>>> I ended up running: yum update pki-ca tomcatjss pki-tools.x86_64
>>>>>>>>> --enablerepo=dogtag-devel --enablerepo=updates-testing
>>>>>>>>>
>>>>>>>> We'll look into this. I think I know why this happens.
>>>>>>>>
>>>>>>>>> What happens if someone manually upgrades pki-ca without first
>>>>>>>>> updating
>>>>>>>>> ipa? I think that pki-ca is going to need a Conflicts ipa < 3.0 in
>>>>>>>>> it.
>>>>>>>>
>>>>>>>> We can add that.
>>>>>>>>
>>>>>>>>> certificate renewal failed. I spent far too long trying to figure
>>>>>>>>> out
>>>>>>>>> why tomcat wasn't listening on port 9180 but failed. I think
>>>>>>>>> 9180 is
>>>>>>>>> actually the old server, right? So another missing dependency on a
>>>>>>>>> fixed
>>>>>>>>> certmonger?
>>>>>>>>>
>>>>>>>>> The best I could find was the certmonger error:
>>>>>>>>>
>>>>>>>>> ca-error: Error 7 connecting to
>>>>>>>>> http://edsel.example.com:9180/ca/ee/ca/profileSubmit: Couldn't
>>>>>>>>> connect
>>>>>>>>> to server.
>>>>>>>>>
>>>>>>>
>>>>>>> I'll set the certmonger min version to 0.60 as per Nalin's mail.
>>>>>>
>>>>>> Ok.
>>>>>>
>>>>>>>> Is this cert renewal on a dogtag 10 instance? Or the upgraded
>>>>>>>> dogtag 9?
>>>>>>>> If its dogtag 10, perhaps you do not have the certmonger version
>>>>>>>> that
>>>>>>>> has the relevant fix? If its dogtag 9, then we need to figure out
>>>>>>>> whats
>>>>>>>> going on. That reminds me - I need to file a bug to allow
>>>>>>>> certmonger to
>>>>>>>> talk to the newly defined dogtag ports. Do you have selinux
>>>>>>>> permissive?
>>>>>>>>
>>>>>>>>> There is no man page for pkispawn/pkidestroy :-( According to the
>>>>>>>>> FHS
>>>>>>>>> these should not be in /bin but in /usr/sbin (not end-user
>>>>>>>>> commands).
>>>>>>>>>
>>>>>>>> There is a trac ticket open for man pages for pkispawn and
>>>>>>>> pkidestroy.
>>>>>>>> We plan to complete this ticket by the time f18 is released.
>>>>>>>>
>>>>>>>> We'll look into the location of pkispawn/pkicreate.
>>>>>>>>
>>>>>>>>> The output of pkicreate/pkisilent was really terrible and not
>>>>>>>>> usable at
>>>>>>>>> all so we didn't display it when failures occurred. It looks like
>>>>>>>>> that
>>>>>>>>> has been addressed, at least for the case where a CA is already
>>>>>>>>> configured and you try to install IPA. Perhaps we should capture
>>>>>>>>> stderr
>>>>>>>>> and display that instead of the command-line of pkispawn? Again, a
>>>>>>>>> man
>>>>>>>>> page would help with the integration.
>>>>>>>>>
>>>>>>>>> 2012-09-10T20:51:45Z DEBUG [2/18]: configuring certificate
>>>>>>>>> server
>>>>>>>>> instance
>>>>>>>>> 2012-09-10T20:51:45Z DEBUG args=/bin/pkispawn -s CA -f
>>>>>>>>> /tmp/tmp_Urraq
>>>>>>>>> 2012-09-10T20:51:45Z DEBUG stdout=
>>>>>>>>> 2012-09-10T20:51:45Z DEBUG stderr=pkispawn : ERROR .......
>>>>>>>>> PKI
>>>>>>>>> subsystem 'CA' for instance 'pki-tomcat' already exists!
>>>>>>>>>
>>>>>>>>> 2012-09-10T20:51:45Z CRITICAL failed to configure ca instance
>>>>>>>>> Command
>>>>>>>>> '/bin/pkispawn -s CA -f /tmp/tmp_Urraq' returned non-zero exit
>>>>>>>>> status 1
>>>>>>>>>
>>>>>>>> That may be a good idea. Of course. thats an IPA thing, right?
>>>>>>
>>>>>> Right, not a show-stopper for this but a new ticket should be
>>>>>> opened to
>>>>>> track it.
>>>
>>> https://fedorahosted.org/freeipa/ticket/3072
>>
>> Thanks.
>>
>> I tested a 2.2 -> 3.0 upgrade with dogtag 9 and it seems to have failed
>> to restart something:
>>
>> [ post rpm -Uvh dist/rpms/freeipa*.rpm ]
>>
>> [root at edsel freeipa]# ipa cert-show 1
>> ipa: ERROR: Certificate operation cannot be completed: Unable to
>> communicate with CMS (Service Temporarily Unavailable)
>> [root at edsel freeipa]# ipactl restart
>> Restarting Directory Service
>> Restarting KDC Service
>> Restarting KPASSWD Service
>> Restarting MEMCACHE Service
>> Restarting HTTP Service
>> Restarting CA Service
>> [root at edsel freeipa]# ipa cert-show 1
>> Certificate: MIIDjTCCAnWgAwIBAgIBATANBgk
>> ...
>>
>> The apache log had this:
>>
>> [Tue Sep 11 14:35:42 2012] [error] (111)Connection refused: proxy: AJP:
>> attempt to connect to 127.0.0.1:9447 (localhost) failed
>> [Tue Sep 11 14:35:42 2012] [error] ap_proxy_connect_backend disabling
>> worker for (localhost)
>> [Tue Sep 11 14:35:42 2012] [error] proxy: AJP: failed to make connection
>> to backend: localhost
>>
>> So I'm gathering that dogtag didn't restart properly, but I'm just
>> guessing. It could have been the PKI-IPA ds instance too, I'm not sure
>> where to check.
>>
>> I also noticed this in /var/log/ipaupgrade.log:
>>
>> 2012-09-11T18:28:22Z DEBUG args=/bin/systemctl start certmonger.service
>> 2012-09-11T18:28:22Z DEBUG stdout=
>> 2012-09-11T18:28:22Z DEBUG stderr=
>> 2012-09-11T18:28:22Z DEBUG args=/bin/systemctl is-active
>> certmonger.service
>> 2012-09-11T18:28:22Z DEBUG stdout=active
>> ...
>> ... [ snip certutil output ]
>> ...
>> 2012-09-11T18:28:52Z DEBUG args=/usr/bin/getcert start-tracking -d
>> /var/lib/pki-ca/alias -n auditSigningCert cert-pki-ca -c
>> dogtag-ipa-renew-agent -C /usr/lib64/ipa/certmonger/renew_ca_cert
>> "auditSigningCert cert-pki-ca" -P XXXXXXXX
>> 2012-09-11T18:28:52Z DEBUG stdout=Please verify that the certmonger
>> service is still running.
>>
>> 2012-09-11T18:28:52Z DEBUG stderr=
>> 2012-09-11T18:28:52Z INFO File
>> "/usr/lib/python2.7/site-packages/ipaserver/install/installutils.py",
>> line 614, in run_script
>> return_value = main_function()
>>
>> File "/usr/sbin/ipa-upgradeconfig", line 540, in main
>> enable_certificate_renewal(api.env.realm)
>>
>> File "/usr/sbin/ipa-upgradeconfig", line 455, in
>> enable_certificate_renewal
>> ca.configure_renewal()
>>
>> File
>> "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line
>> 1298, in configure_renewal
>> self.dogtag_constants.ALIAS_DIR, 'renew_ca_cert "%s"' % nickname)
>>
>> File "/usr/lib/python2.7/site-packages/ipapython/certmonger.py", line
>> 394, in dogtag_start_tracking
>> (stdout, stderr, returncode) = ipautil.run(args, nolog=[pin])
>>
>> File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line
>> 309, in run
>> raise CalledProcessError(p.returncode, args)
>>
>> 2012-09-11T18:28:52Z INFO The ipa-upgradeconfig command failed,
>> exception: CalledProcessError: Command '/usr/bin/getcert start-tracking
>> -d /var/lib/pki-ca/alias -n auditSigningCert cert-pki-ca -c
>> dogtag-ipa-renew-agent -C /usr/lib64/ipa/certmonger/renew_ca_cert
>> "auditSigningCert cert-pki-ca" -P XXXXXXXX' returned non-zero exit
>> status 1
>>
>> I'm not sure if this is related to this patch or not. If it isn't can
>> you file a new ticket on it, or add it the 2.2 -> 3.0 upgrade ticket?
>>
>> rob
>
> And to reply to myself, how do we imagine that upgrades will work?
>
> Is it legal for someone to upgrade to IPA 3.0 and dogtag 10 separately,
> or do we expect/require it be done at the same time, or one first?
>
Sorry to break this into a ton of e-mails. I updated pki-ca and getting
a selinux error in permissive:
# /bin/systemctl -a status pki-cad at pki-ca.service
pki-cad at pki-ca.service - PKI Certificate Authority Server pki-ca
Loaded: loaded (/usr/lib/systemd/system/pki-cad at .service;
enabled)
Active: failed (Result: exit-code) since Tue, 11 Sep 2012
15:36:37 -0400; 3s ago
Process: 7067 ExecStart=/usr/bin/pkicontrol start ca %i
(code=exited, status=1/FAILURE)
Main PID: 5830 (code=exited, status=0/SUCCESS)
CGroup: name=systemd:/system/pki-cad at .service/pki-ca
Sep 11 15:36:37 edsel.greyoak.com pkicontrol[7067]: /usr/bin/runcon:
invalid context: system_u:system_r:pki_ca_script_t:s0: Invalid argument
I assume this means I need to re-install pki-selinux? It should be
possible to tweak the dependencies such that this package is installed
first.
rob
More information about the Freeipa-devel
mailing list