[Freeipa-devel] [PATCH] Patch to allow IPA to work with dogtag 10 on f18
Petr Viktorin
pviktori at redhat.com
Wed Sep 12 16:43:55 UTC 2012
On 09/11/2012 09:38 PM, Rob Crittenden wrote:
> Rob Crittenden wrote:
>> Rob Crittenden wrote:
>>> Petr Viktorin wrote:
>>>> On 09/11/2012 04:38 PM, Rob Crittenden wrote:
>>>>> Ade Lee wrote:
>>>>>> On Tue, 2012-09-11 at 08:59 -0400, Rob Crittenden wrote:
>>>>>>> Petr Viktorin wrote:
>>>>>>>> On 09/11/2012 04:04 AM, Ade Lee wrote:
>>>>>>>>> On Mon, 2012-09-10 at 16:58 -0400, Rob Crittenden wrote:
>>>>>>>>>> Petr Viktorin wrote:
>>>>>>>>>>> Attaching rebased and squashed patches. I've done some testing
>>>>>>>>>>> with
>>>>>>>>>>> them
>>>>>>>>>>> but please test some more.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Most of these aren't IPA issues, but dogtag issues. I'll try to
>>>>>>>>>> split
>>>>>>>>>> them out.
>>>>>>>>>>
>>>>>>>>>> IPA:
>>>>>>>>>>
>>>>>>>>>> For the configuration files in install/conf to be updated at rpm
>>>>>>>>>> update
>>>>>>>>>> time the VERSION needs to be incremented.
>>>>>>>>
>>>>>>>> These files should stay the same since on upgrade we're still
>>>>>>>> using a
>>>>>>>> Dogtag 9 style instance. The Dogtag 10 ports are only used in new
>>>>>>>> installs.
>>>>>>>>
>>>>>>>>>> The ipa package lacks any updated dogtag dependencies, so I
>>>>>>>>>> abused
>>>>>>>>>> it.
>>>>>>>>
>>>>>>>> What should the updated dependencies be? Since it should work with
>>>>>>>> both
>>>>>>>> dogtag 9 and 10, I don't see how they should change.
>>>>>>>
>>>>>>> I don't know either, but we need to prevent people from installing
>>>>>>> incompatible package combinations.
>>>>>>>
>>>>>> Would'nt the Conflicts: ipa < 3.0 in pki-ca mentioned below satisfy
>>>>>> this
>>>>>> requirement? The main concern is that you must have ipa 3.0 if you
>>>>>> have
>>>>>> dogtag 10.
>>>>>>
>>>>>> Given that dogtag is consumed by IPA though, it makes more sense to
>>>>>> put
>>>>>> the relevant conflicts in IPA rather than in dogtag. So in this
>>>>>> case,
>>>>>> that would mean putting Conflicts: pki-ca >= 10.0 in IPA 2.x.
>>>>>> Recall that dogtag 10 will only be officially available in f18+.
>>>>>
>>>>> That isn't enough. If a F-17 user with IPA 2.2 installed upgrades to
>>>>> F-18 they would be able to install dogtag 10 and blow up their IPA
>>>>> server.
>>>>>
>>>>>>
>>>>>>>>>> I installed IPA with dogtag 9 and created a replica.
>>>>>>>>>>
>>>>>>>>>> I updated the IPA bits, that worked fine.
>>>>>>>>>>
>>>>>>>>>> I updated to dogtag 10 and now the CA doesn't work on the master,
>>>>>>>>>> including starting the dogtag instance. Note that the rpm update
>>>>>>>>>> process
>>>>>>>>>> worked, no notice that the CA service didn't restart.
>>>>>>>>>>
>>>>>>>>> Did you try to restart the CA with selinux in permissive mode?
>>>>>>>>> This is
>>>>>>>>> still required right now until I get the selinux policy all
>>>>>>>>> straightened
>>>>>>>>> out.
>>>>>>>>>
>>>>>>>>> There is also a separate dogtag ticket (which is currently being
>>>>>>>>> worked
>>>>>>>>> on) to restart existing dogtag instances when dogtag is upgraded
>>>>>>>>> from
>>>>>>>>> 9->10.
>>>>>>>>
>>>>>>>> In permissive mode, this upgrade works for me.
>>>>>>>
>>>>>>> I was in enforcing mode but saw no AVCs. What is the ETA on fixing
>>>>>>> this?
>>>>>>>
>>>>>>
>>>>>> Within the next week or two, I need to finish the IPA merge database
>>>>>> patch first, and then co-ordinate changes with the selinux guys.
>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Sometimes I do get this error intermittently:
>>>>>>>>
>>>>>>>> ipa: ERROR: Certificate operation cannot be completed: Unable to
>>>>>>>> communicate with CMS (Service Temporarily Unavailable)
>>>>>>>>
>>>>>>>> Usually, waiting a couple of minutes clears this up. Perhaps we are
>>>>>>>> not
>>>>>>>> doing startup detection correctly. Ade mentioned that waiting for
>>>>>>>> ports
>>>>>>>> may not be ideal. How do we know if Dogtag is initialized?
>>>>>>>
>>>>>>> Years ago we had discussed with Andrew and Matt creating a URI that
>>>>>>> can
>>>>>>> be queried to determine dogtag status. I don't think that ever went
>>>>>>> anywhere.
>>>>>>>
>>>>>> Petr, this happens only on reboot, right? And not on regular
>>>>>> "service
>>>>>> ipa restart"?
>>>>
>>>> I've now seen it happen right after a 9 → 10 upgrade.
>>>>
>>>>>> Yeah, I remember this conversation - and even created a bug for it at
>>>>>> some point. This went away because the mechanism you were using
>>>>>> seemed
>>>>>> to be working. The timing may be very different now with tomcat 7
>>>>>> and
>>>>>> systemd. I'll open a dogtag trac ticket for this.
>>>>>
>>>>> Ok.
>>>>>
>>>>>>
>>>>>>>>
>>>>>>>>>> Uninstalling failed because it tried to run pkidestroy and not
>>>>>>>>>> pkiremove.
>>>>>>>>
>>>>>>>> I was under the impression that pkidestroy was the correct
>>>>>>>> command to
>>>>>>>> remove an upgraded instance. I'll check with Ade.
>>>>>>>>
>>>>>>>>> I'll test this too.
>>>>>>>>>
>>>>>>>>>> The contents of the file passed to pkispawn should be logged
>>>>>>>>>> so we
>>>>>>>>>> can
>>>>>>>>>> see exactly what was passed in.
>>>>>>>>>>
>>>>>>>>> Its a pretty big file. You might want to only log the
>>>>>>>>> modifications.
>>>>>>>>> Or save the file somewhere.
>>>>>>>>
>>>>>>>> Our logs are pretty verbose, so that shouldn't be a problem. I'll
>>>>>>>> put it
>>>>>>>> in the next version of the patch.
>>>>>>>
>>>>>>> The question to ask is: would you need the contents of this file if
>>>>>>> all
>>>>>>> you got were logs and needed to evaluate why installation failed? In
>>>>>>> most cases this is yes.
>>>>>>>
>>>>>> Up to you guys. There is a patch I am working on in which I will be
>>>>>> logging the object that is being passed to the server from pkispawn.
>>>>>> That - and the diffs to the standard config file as I mentioned
>>>>>> above -
>>>>>> will likely be sufficient to debug almost all cases.
>>>>>>
>>>>>> Make sure not to log any passwords.
>>>>>>
>>>>
>>>> Thanks for the catch. Attaching updated patch that sanitizes the
>>>> passwords.
>>>>
>>>>>>>>>> DOGTAG:
>>>>>>>>>>
>>>>>>>>>> When upgrading using the dogtag-devel repo I had to specify
>>>>>>>>>> pki-tools.x86_64 otherwise it tried to install both 32 and 64-bit
>>>>>>>>>> versions (and failed).
>>>>>>>>>>
>>>>>>>>>> I ended up running: yum update pki-ca tomcatjss pki-tools.x86_64
>>>>>>>>>> --enablerepo=dogtag-devel --enablerepo=updates-testing
>>>>>>>>>>
>>>>>>>>> We'll look into this. I think I know why this happens.
>>>>>>>>>
>>>>>>>>>> What happens if someone manually upgrades pki-ca without first
>>>>>>>>>> updating
>>>>>>>>>> ipa? I think that pki-ca is going to need a Conflicts ipa <
>>>>>>>>>> 3.0 in
>>>>>>>>>> it.
>>>>>>>>>
>>>>>>>>> We can add that.
>>>>>>>>>
>>>>>>>>>> certificate renewal failed. I spent far too long trying to figure
>>>>>>>>>> out
>>>>>>>>>> why tomcat wasn't listening on port 9180 but failed. I think
>>>>>>>>>> 9180 is
>>>>>>>>>> actually the old server, right? So another missing dependency
>>>>>>>>>> on a
>>>>>>>>>> fixed
>>>>>>>>>> certmonger?
>>>>>>>>>>
>>>>>>>>>> The best I could find was the certmonger error:
>>>>>>>>>>
>>>>>>>>>> ca-error: Error 7 connecting to
>>>>>>>>>> http://edsel.example.com:9180/ca/ee/ca/profileSubmit: Couldn't
>>>>>>>>>> connect
>>>>>>>>>> to server.
>>>>>>>>>>
>>>>>>>>
>>>>>>>> I'll set the certmonger min version to 0.60 as per Nalin's mail.
>>>>>>>
>>>>>>> Ok.
>>>>>>>
>>>>>>>>> Is this cert renewal on a dogtag 10 instance? Or the upgraded
>>>>>>>>> dogtag 9?
>>>>>>>>> If its dogtag 10, perhaps you do not have the certmonger version
>>>>>>>>> that
>>>>>>>>> has the relevant fix? If its dogtag 9, then we need to figure out
>>>>>>>>> whats
>>>>>>>>> going on. That reminds me - I need to file a bug to allow
>>>>>>>>> certmonger to
>>>>>>>>> talk to the newly defined dogtag ports. Do you have selinux
>>>>>>>>> permissive?
>>>>>>>>>
>>>>>>>>>> There is no man page for pkispawn/pkidestroy :-( According to the
>>>>>>>>>> FHS
>>>>>>>>>> these should not be in /bin but in /usr/sbin (not end-user
>>>>>>>>>> commands).
>>>>>>>>>>
>>>>>>>>> There is a trac ticket open for man pages for pkispawn and
>>>>>>>>> pkidestroy.
>>>>>>>>> We plan to complete this ticket by the time f18 is released.
>>>>>>>>>
>>>>>>>>> We'll look into the location of pkispawn/pkicreate.
>>>>>>>>>
>>>>>>>>>> The output of pkicreate/pkisilent was really terrible and not
>>>>>>>>>> usable at
>>>>>>>>>> all so we didn't display it when failures occurred. It looks like
>>>>>>>>>> that
>>>>>>>>>> has been addressed, at least for the case where a CA is already
>>>>>>>>>> configured and you try to install IPA. Perhaps we should capture
>>>>>>>>>> stderr
>>>>>>>>>> and display that instead of the command-line of pkispawn?
>>>>>>>>>> Again, a
>>>>>>>>>> man
>>>>>>>>>> page would help with the integration.
>>>>>>>>>>
>>>>>>>>>> 2012-09-10T20:51:45Z DEBUG [2/18]: configuring certificate
>>>>>>>>>> server
>>>>>>>>>> instance
>>>>>>>>>> 2012-09-10T20:51:45Z DEBUG args=/bin/pkispawn -s CA -f
>>>>>>>>>> /tmp/tmp_Urraq
>>>>>>>>>> 2012-09-10T20:51:45Z DEBUG stdout=
>>>>>>>>>> 2012-09-10T20:51:45Z DEBUG stderr=pkispawn : ERROR .......
>>>>>>>>>> PKI
>>>>>>>>>> subsystem 'CA' for instance 'pki-tomcat' already exists!
>>>>>>>>>>
>>>>>>>>>> 2012-09-10T20:51:45Z CRITICAL failed to configure ca instance
>>>>>>>>>> Command
>>>>>>>>>> '/bin/pkispawn -s CA -f /tmp/tmp_Urraq' returned non-zero exit
>>>>>>>>>> status 1
>>>>>>>>>>
>>>>>>>>> That may be a good idea. Of course. thats an IPA thing, right?
>>>>>>>
>>>>>>> Right, not a show-stopper for this but a new ticket should be
>>>>>>> opened to
>>>>>>> track it.
>>>>
>>>> https://fedorahosted.org/freeipa/ticket/3072
>>>
>>> Thanks.
>>>
>>> I tested a 2.2 -> 3.0 upgrade with dogtag 9 and it seems to have failed
>>> to restart something:
>>>
>>> [ post rpm -Uvh dist/rpms/freeipa*.rpm ]
>>>
>>> [root at edsel freeipa]# ipa cert-show 1
>>> ipa: ERROR: Certificate operation cannot be completed: Unable to
>>> communicate with CMS (Service Temporarily Unavailable)
>>> [root at edsel freeipa]# ipactl restart
>>> Restarting Directory Service
>>> Restarting KDC Service
>>> Restarting KPASSWD Service
>>> Restarting MEMCACHE Service
>>> Restarting HTTP Service
>>> Restarting CA Service
>>> [root at edsel freeipa]# ipa cert-show 1
>>> Certificate: MIIDjTCCAnWgAwIBAgIBATANBgk
>>> ...
>>>
>>> The apache log had this:
>>>
>>> [Tue Sep 11 14:35:42 2012] [error] (111)Connection refused: proxy: AJP:
>>> attempt to connect to 127.0.0.1:9447 (localhost) failed
>>> [Tue Sep 11 14:35:42 2012] [error] ap_proxy_connect_backend disabling
>>> worker for (localhost)
>>> [Tue Sep 11 14:35:42 2012] [error] proxy: AJP: failed to make connection
>>> to backend: localhost
>>>
>>> So I'm gathering that dogtag didn't restart properly, but I'm just
>>> guessing. It could have been the PKI-IPA ds instance too, I'm not sure
>>> where to check.
>>>
>>> I also noticed this in /var/log/ipaupgrade.log:
>>>
>>> 2012-09-11T18:28:22Z DEBUG args=/bin/systemctl start certmonger.service
>>> 2012-09-11T18:28:22Z DEBUG stdout=
>>> 2012-09-11T18:28:22Z DEBUG stderr=
>>> 2012-09-11T18:28:22Z DEBUG args=/bin/systemctl is-active
>>> certmonger.service
>>> 2012-09-11T18:28:22Z DEBUG stdout=active
>>> ...
>>> ... [ snip certutil output ]
>>> ...
>>> 2012-09-11T18:28:52Z DEBUG args=/usr/bin/getcert start-tracking -d
>>> /var/lib/pki-ca/alias -n auditSigningCert cert-pki-ca -c
>>> dogtag-ipa-renew-agent -C /usr/lib64/ipa/certmonger/renew_ca_cert
>>> "auditSigningCert cert-pki-ca" -P XXXXXXXX
>>> 2012-09-11T18:28:52Z DEBUG stdout=Please verify that the certmonger
>>> service is still running.
>>>
>>> 2012-09-11T18:28:52Z DEBUG stderr=
>>> 2012-09-11T18:28:52Z INFO File
>>> "/usr/lib/python2.7/site-packages/ipaserver/install/installutils.py",
>>> line 614, in run_script
>>> return_value = main_function()
>>>
>>> File "/usr/sbin/ipa-upgradeconfig", line 540, in main
>>> enable_certificate_renewal(api.env.realm)
>>>
>>> File "/usr/sbin/ipa-upgradeconfig", line 455, in
>>> enable_certificate_renewal
>>> ca.configure_renewal()
>>>
>>> File
>>> "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line
>>> 1298, in configure_renewal
>>> self.dogtag_constants.ALIAS_DIR, 'renew_ca_cert "%s"' % nickname)
>>>
>>> File "/usr/lib/python2.7/site-packages/ipapython/certmonger.py", line
>>> 394, in dogtag_start_tracking
>>> (stdout, stderr, returncode) = ipautil.run(args, nolog=[pin])
>>>
>>> File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line
>>> 309, in run
>>> raise CalledProcessError(p.returncode, args)
>>>
>>> 2012-09-11T18:28:52Z INFO The ipa-upgradeconfig command failed,
>>> exception: CalledProcessError: Command '/usr/bin/getcert start-tracking
>>> -d /var/lib/pki-ca/alias -n auditSigningCert cert-pki-ca -c
>>> dogtag-ipa-renew-agent -C /usr/lib64/ipa/certmonger/renew_ca_cert
>>> "auditSigningCert cert-pki-ca" -P XXXXXXXX' returned non-zero exit
>>> status 1
>>>
>>> I'm not sure if this is related to this patch or not. If it isn't can
>>> you file a new ticket on it, or add it the 2.2 -> 3.0 upgrade ticket?
>>>
>>> rob
>>
>> And to reply to myself, how do we imagine that upgrades will work?
>>
>> Is it legal for someone to upgrade to IPA 3.0 and dogtag 10 separately,
>> or do we expect/require it be done at the same time, or one first?
It's legal to upgrade separately, since we'll support the IPA 3.0 +
Dogtag 9 combination. But to minimize the number of ways things can go
wrong, we could require that IPA is upgraded before Dogtag.
> Sorry to break this into a ton of e-mails. I updated pki-ca and getting
> a selinux error in permissive:
>
> # /bin/systemctl -a status pki-cad at pki-ca.service
> pki-cad at pki-ca.service - PKI Certificate Authority Server pki-ca
> Loaded: loaded (/usr/lib/systemd/system/pki-cad at .service;
> enabled)
> Active: failed (Result: exit-code) since Tue, 11 Sep 2012
> 15:36:37 -0400; 3s ago
> Process: 7067 ExecStart=/usr/bin/pkicontrol start ca %i
> (code=exited, status=1/FAILURE)
> Main PID: 5830 (code=exited, status=0/SUCCESS)
> CGroup: name=systemd:/system/pki-cad at .service/pki-ca
>
> Sep 11 15:36:37 edsel.greyoak.com pkicontrol[7067]: /usr/bin/runcon:
> invalid context: system_u:system_r:pki_ca_script_t:s0: Invalid argument
>
> I assume this means I need to re-install pki-selinux? It should be
> possible to tweak the dependencies such that this package is installed
> first.
I'll look into it, but it would be nice to get some guidance from
someone who knows more about RPMs.
--
Petr³
More information about the Freeipa-devel
mailing list