[Freeipa-devel] [PATCH] Patch to allow IPA to work with dogtag 10 on f18

[Rob Crittenden]

Ade Lee wrote:
> On Wed, 2012-09-05 at 16:20 -0400, Rob Crittenden wrote:
>> Martin Kosek wrote:
>>> On 08/31/2012 04:53 PM, Petr Viktorin wrote:
>>>> On 08/28/2012 03:40 PM, Petr Viktorin wrote:
>>>>> On 08/17/2012 06:04 PM, Ade Lee wrote:
>>>>>> On Fri, 2012-08-17 at 09:34 -0400, Ade Lee wrote:
>>>>>>> On Thu, 2012-08-16 at 18:45 +0200, Martin Kosek wrote:
>>>>>>>> On 08/16/2012 01:28 PM, Ade Lee wrote:
>>>>>>>>> Patch attached this time.  I should know better than to do this in the
>>>>>>>>> middle of the night ..
>>>>>>>>>
>>>>>>>>> On Thu, 2012-08-16 at 09:12 +0200, Martin Kosek wrote:
>>>>>>>>>> On 08/16/2012 07:53 AM, Ade Lee wrote:
>>>>>>>>>>> On Wed, 2012-08-15 at 23:41 -0400, Ade Lee wrote:
>>>>>>>>>>>> On Wed, 2012-08-15 at 16:34 +0200, Martin Kosek wrote:
>>>>>>>>>>>>> On 08/15/2012 03:54 PM, Ade Lee wrote:
>>>>>>>>>>>>>> On Wed, 2012-08-15 at 13:24 +0200, Martin Kosek wrote:
>>>>>>>>>>>>>>> On 08/08/2012 10:05 PM, Ade Lee wrote:
>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Dogtag 10 is being released on f18, and has a number of
>>>>>>>>>>>>>>>> changes that
>>>>>>>>>>>>>>>> will affect IPA.  In particular, the following changes will
>>>>>>>>>>>>>>>> affect
>>>>>>>>>>>>>>>> current IPA code.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> * The directory layout of the dogtag instance has changed.
>>>>>>>>>>>>>>>> Instead of
>>>>>>>>>>>>>>>> using separate tomcat instances to host different
>>>>>>>>>>>>>>>> subsystems, the
>>>>>>>>>>>>>>>> standard dogtag installation will allow one to install a CA.
>>>>>>>>>>>>>>>> KRA, OCSP
>>>>>>>>>>>>>>>> and TKS within the same instance.  There have been
>>>>>>>>>>>>>>>> corresponding changes
>>>>>>>>>>>>>>>> in the directory layout, as well as the default instance name
>>>>>>>>>>>>>>>> (pki-tomcat instead of pki-ca), and startup daemon
>>>>>>>>>>>>>>>> (pki-tomcatd, instead
>>>>>>>>>>>>>>>> of pki-cad, pki-krad etc.)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> * The default instance will use only four ports (HTTPS,
>>>>>>>>>>>>>>>> HTTP, AJP and
>>>>>>>>>>>>>>>> tomcat shutdown port) rather than the 6 previously used.
>>>>>>>>>>>>>>>> The default
>>>>>>>>>>>>>>>> ports will be changed to the standard tomcat ports.  As
>>>>>>>>>>>>>>>> these ports are
>>>>>>>>>>>>>>>> local to the ipa server machine, this should not cause too much
>>>>>>>>>>>>>>>> disruption.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> * There is a new single step installer written in python.
>>>>>>>>>>>>>>>> (pkispawn/destroy) vs. pkicreate/pkisilent/pkiremove.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> * Dogtag 10 runs on tomcat7 - with a new corresponding
>>>>>>>>>>>>>>>> version of
>>>>>>>>>>>>>>>> tomcatjss.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The attached patch integrates all the above changes in IPA
>>>>>>>>>>>>>>>> installation
>>>>>>>>>>>>>>>> and maintenance code.  Once the patch is applied, users will
>>>>>>>>>>>>>>>> be able to:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 1. run ipa-server-install to completion on f18 with dogtag 10.
>>>>>>>>>>>>>>>> 2. install a new replica on f18 on dogtag 10.
>>>>>>>>>>>>>>>> 3. upgrade an f17 machine with an existing IPA instance to
>>>>>>>>>>>>>>>> f18/ dogtag
>>>>>>>>>>>>>>>> 10 - and have that old-style dogtag instance continue to run
>>>>>>>>>>>>>>>> correctly.
>>>>>>>>>>>>>>>> This will require the installation of the latest version of
>>>>>>>>>>>>>>>> tomcatjss as
>>>>>>>>>>>>>>>> well as the installation of tomcat6.  The old-style instance
>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>> continue to use tomcat6.
>>>>>>>>>>>>>>>> 4. in addition, the new cert renewal code has been patched
>>>>>>>>>>>>>>>> and should
>>>>>>>>>>>>>>>> continue to work.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> What is not yet completed / supported:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 1. Installation with an external CA is not yet completed in
>>>>>>>>>>>>>>>> the new
>>>>>>>>>>>>>>>> installer.  We plan to complete this soon.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2. There is some IPA upgrade code that has not yet been touched
>>>>>>>>>>>>>>>> (install/tools/ipa-upgradeconfig).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 3. A script needs to be written to allow admins to convert
>>>>>>>>>>>>>>>> their
>>>>>>>>>>>>>>>> old-style dogtag instances to new style instances, as well
>>>>>>>>>>>>>>>> as code to
>>>>>>>>>>>>>>>> periodically prompt admins to do this.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 4. Installation of old-style instances using
>>>>>>>>>>>>>>>> pkicreate/pkisilent on
>>>>>>>>>>>>>>>> dogtag 10 will no longer be supported, and will be disabled
>>>>>>>>>>>>>>>> soon.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 5.  The pki-selinux policy has been updated to reflect these
>>>>>>>>>>>>>>>> changes,
>>>>>>>>>>>>>>>> but is still in flux.  In fact, it is our intention to place
>>>>>>>>>>>>>>>> the dogtag
>>>>>>>>>>>>>>>> selinux policy in the base selinux policy for f18.  In the
>>>>>>>>>>>>>>>> meantime, it
>>>>>>>>>>>>>>>> may be necessary to run installs in permissive mode.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The dogtag 10 code will be released shortly into f18.  Prior
>>>>>>>>>>>>>>>> to that
>>>>>>>>>>>>>>>> though, we have placed the new dogtag 10 and tomcatjss code
>>>>>>>>>>>>>>>> in a
>>>>>>>>>>>>>>>> developer repo that is located at
>>>>>>>>>>>>>>>> http://nkinder.fedorapeople.org/dogtag-devel/
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Testing can be done on both f18 and f17 - although the
>>>>>>>>>>>>>>>> target platform -
>>>>>>>>>>>>>>>> and the only platform for which official builds will be
>>>>>>>>>>>>>>>> created is f18.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> Ade
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Ade,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks for the patch, I started with review and integration
>>>>>>>>>>>>>>> tests (currently
>>>>>>>>>>>>>>> running on Fedora 17 with Nathan's repo).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Installation on single master was smooth, it worked just
>>>>>>>>>>>>>>> fine, even with
>>>>>>>>>>>>>>> enforced SELinux, without any error - kudos to you and the
>>>>>>>>>>>>>>> whole dogtag team.
>>>>>>>>>>>>>>> The resulting logs and the structure of your log directory
>>>>>>>>>>>>>>> seems improved. I
>>>>>>>>>>>>>>> believe that the brand new Python installers will make it
>>>>>>>>>>>>>>> easier to debug
>>>>>>>>>>>>>>> issues with dogtag installation when they come.  When I tried
>>>>>>>>>>>>>>> our unit tests or
>>>>>>>>>>>>>>> some simple cert operation, it worked fine as well.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Now the bad news, or rather few issues and suggestions I found:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 1) As we already discussed on IRC, tomcat 7 was not pulled
>>>>>>>>>>>>>>> automatically on
>>>>>>>>>>>>>>> Fedora 17 when I updated pki-ca, you somewhere miss a Requires.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We have a dogtag patch that is currently in review that will
>>>>>>>>>>>>>> address
>>>>>>>>>>>>>> this.  Once this is in, tomcatjss >=7.0.0 will be required for
>>>>>>>>>>>>>> f17+,
>>>>>>>>>>>>>> rather than f18+
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2) I had installed IPA with dogtag10 on master. However, CA
>>>>>>>>>>>>>>> installation on a
>>>>>>>>>>>>>>> replica (ipa-ca-install) with dogtag9 failed - is this
>>>>>>>>>>>>>>> expectable?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yes.  The current IPA patch is designed to work with dogtag 10
>>>>>>>>>>>>>> only,
>>>>>>>>>>>>>> which will be officially available on f18+.  So if you update
>>>>>>>>>>>>>> to dogtag
>>>>>>>>>>>>>> 10, you must have this patch and visa versa.  We probably need
>>>>>>>>>>>>>> to add
>>>>>>>>>>>>>> the relevant requires to IPA to enforce this.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If you have an existing dogtag 9 instance, and you upgrade to
>>>>>>>>>>>>>> the new
>>>>>>>>>>>>>> dogtag 10 and patched IPA, then that instance will continue to
>>>>>>>>>>>>>> work.
>>>>>>>>>>>>>> But any new instances would be created using dogtag 10.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 3) I had installed IPA with dogtag10 on master. Replica had
>>>>>>>>>>>>>>> dogtag10 as well
>>>>>>>>>>>>>>> and I got the following error:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> # ipa-ca-install
>>>>>>>>>>>>>>> /home/mkosek/replica-info-vm-114.idm.lab.bos.redhat.com.gpg
>>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>>> Configuring certificate server: Estimated time 3 minutes 30
>>>>>>>>>>>>>>> seconds
>>>>>>>>>>>>>>>      [1/14]: creating certificate server user
>>>>>>>>>>>>>>>      [2/14]: configuring certificate server instance
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Your system may be partly configured.
>>>>>>>>>>>>>>> Run /usr/sbin/ipa-server-install --uninstall to clean up.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Unexpected error - see /var/log/ipareplica-ca-install.log for
>>>>>>>>>>>>>>> details:
>>>>>>>>>>>>>>> IOError: [Errno 2] No such file or directory:
>>>>>>>>>>>>>>> '/var/lib/pki/pki-tomcat/alias/ca_backup_keys.p12'
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Root cause:
>>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>>>      File
>>>>>>>>>>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py",
>>>>>>>>>>>>>>> line
>>>>>>>>>>>>>>> 625, in __spawn_instance
>>>>>>>>>>>>>>>        "/root/cacert.p12")
>>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I need to look into this.  I had fixed ipa-replica-install,
>>>>>>>>>>>>>> rather than
>>>>>>>>>>>>>> ipa-ca-install to create replicas.  I didn't know ipa-ca-install
>>>>>>>>>>>>>> existed!  Should not be too bad to fix though - most likely
>>>>>>>>>>>>>> just need to
>>>>>>>>>>>>>> move files to the right place.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Ok, thanks! Btw. CA on replica can be either installed during
>>>>>>>>>>>>> ipa-replica-install time (when --setup-ca option is passed, you
>>>>>>>>>>>>> probably used
>>>>>>>>>>>>> that one) and the aforementioned ipa-ca-install run after
>>>>>>>>>>>>> ipa-replica-install.
>>>>>>>>>>>>>
>>>>>>>>>>>> I will be testing this out again.  As ipa-ca-install uses the
>>>>>>>>>>>> same code
>>>>>>>>>>>> as ipa-replica-install, I would have expected it to work.  Did
>>>>>>>>>>>> you try
>>>>>>>>>>>> it with selinux in permissive mode?
>>>>>>>>>>>>
>>>>>>>>>>> OK -- I tested this out and realized that between the time I
>>>>>>>>>>> initially
>>>>>>>>>>> tested this and your tests, we had changed a URL
>>>>>>>>>>> from /ca/pki/installer/installToken to
>>>>>>>>>>> /ca/rest/installer/installToken,
>>>>>>>>>>> but I forgot to update ipa-pki-proxy.conf.
>>>>>>>>>>>
>>>>>>>>>>> The new patch has been rebased and changes the relevant patch.  With
>>>>>>>>>>> this, the CA replica installs correctly.
>>>>>>>>>>
>>>>>>>>>> Umh, where can I find the new rebased patch? :-)
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> There was one issue that I encountered.  I have seen this once
>>>>>>>>>>> before
>>>>>>>>>>> and will need to investigate further.  IPA sometimes restarts the
>>>>>>>>>>> dogtag
>>>>>>>>>>> instance by doing systemctl stop pki-tomcatd.target.  and then
>>>>>>>>>>> systemctl
>>>>>>>>>>> start pki-tomcatd.target.   I have noticed that occasionally the
>>>>>>>>>>> start
>>>>>>>>>>> invocation fails, while the stop and restart invocations succeed
>>>>>>>>>>> just
>>>>>>>>>>> fine.  This makes absolutely no sense because restart is essentially
>>>>>>>>>>> just stop and then start.
>>>>>>>>>>>
>>>>>>>>>>> I think this is actually a bug in systemd.  If I reboot the
>>>>>>>>>>> machine, it
>>>>>>>>>>> all works perfectly.  If we can't figure out whats going on with
>>>>>>>>>>> systemd, we can always replace this start/stop with
>>>>>>>>>>> starting/stopping
>>>>>>>>>>> the service directly.  (pki-tomcatd at pki-tomcat.service).  That
>>>>>>>>>>> seems to
>>>>>>>>>>> work all the time with no problems.
>>>>>>>>>>>
>>>>>>>>>>> I suggest we leave this fix (if needed) for a separate patch.
>>>>>>>>>>
>>>>>>>>>> Ok, I will see if I hit this issue again. Btw. is it recommended
>>>>>>>>>> in systemd to
>>>>>>>>>> use target units this way? I thought that only service files are
>>>>>>>>>> meant to be
>>>>>>>>>> used for manipulation with a service. As you said yourself, using
>>>>>>>>>> service unit
>>>>>>>>>> file for restart worked every time.
>>>>>>>>>>
>>>>>>>>>> Martin
>>>>>>>>>
>>>>>>>>
>>>>>>>> Now, I tested the rebased patch with VMs with permissive SELinux.
>>>>>>>> IPA server
>>>>>>>> install was OK, as always, but replica installation ended up with
>>>>>>>> error.
>>>>>>>> Although it got much further than before:
>>>>>>>>
>>>>>>>> # ipa-ca-install
>>>>>>>> /home/mkosek/replica-info-vm-114.idm.lab.bos.redhat.com.gpg
>>>>>>>> ...
>>>>>>>>      [3/3]: restarting directory server
>>>>>>>> done configuring pkids.
>>>>>>>> Configuring certificate server: Estimated time 3 minutes 30 seconds
>>>>>>>>      [1/14]: creating certificate server user
>>>>>>>>      [2/14]: configuring certificate server instance
>>>>>>>>      [3/14]: disabling nonces
>>>>>>>>      [4/14]: importing CA chain to RA certificate database
>>>>>>>>      [5/14]: fixing RA database permissions
>>>>>>>>      [6/14]: setting up signing cert profile
>>>>>>>>      [7/14]: set up CRL publishing
>>>>>>>>      [8/14]: set certificate subject base
>>>>>>>>      [9/14]: enabling Subject Key Identifier
>>>>>>>>      [10/14]: configuring certificate server to start on boot
>>>>>>>>      [11/14]: configure certmonger for renewals
>>>>>>>>      [12/14]: configure clone certificate renewals
>>>>>>>>      [13/14]: configure Server-Cert certificate renewal
>>>>>>>>      [14/14]: Configure HTTP to proxy connections
>>>>>>>> done configuring pki-tomcatd.
>>>>>>>> Restarting the directory and certificate servers
>>>>>>>>
>>>>>>>> Your system may be partly configured.
>>>>>>>> Run /usr/sbin/ipa-server-install --uninstall to clean up.
>>>>>>>>
>>>>>>>> It seems like that CA on a replica did not start and so a check for
>>>>>>>> port 8080
>>>>>>>> failed. Attached logs (pki-ca-logs.tgz) may provide more information
>>>>>>>> to this issue.
>>>>>>>>
>>>>>>> This is the systemd issue I mentioned before.  I will submit a patch
>>>>>>> which will change the restart mechanism to restart the service and not
>>>>>>> the target.
>>>>>>>>
>>>>>>
>>>>>> Patch attached.  This patch should be applied on top of the large patch
>>>>>> (being reviewed).
>>>>>>
>>>>>>>> Now I am also testing upgrade from IPA+dogtag9 to PatchedIPA+dogtag10
>>>>>>>> (permissive SELinux). Now, the service was upgraded smoothly, CA
>>>>>>>> service
>>>>>>>> could be restarted without failure. However, certificate operations now
>>>>>>>> did not work:
>>>>>>>>
>>>>>>>> # ipactl restart
>>>>>>>> Restarting Directory Service
>>>>>>>> Restarting KDC Service
>>>>>>>> Restarting KPASSWD Service
>>>>>>>> Restarting MEMCACHE Service
>>>>>>>> Restarting HTTP Service
>>>>>>>> Restarting CA Service
>>>>>>>> # ipactl status
>>>>>>>> Directory Service: RUNNING
>>>>>>>> KDC Service: RUNNING
>>>>>>>> KPASSWD Service: RUNNING
>>>>>>>> MEMCACHE Service: RUNNING
>>>>>>>> HTTP Service: RUNNING
>>>>>>>> CA Service: RUNNING
>>>>>>>> # ipa cert-show 1
>>>>>>>> ipa: ERROR: Certificate operation cannot be completed: Unable to
>>>>>>>> communicate with CMS (Internal Server Error)
>>>>>>>>
>>>>>>>> Attaching logs for this case as well (ipa-ca-logs-upgrade.tgz).
>>>>>>>>
>>>>>>> The reason for this is that the old dogtag instances are missing:
>>>>>>> 1. a new parameter in CS.cfg
>>>>>>> 2. a couple of symlinks
>>>>>>>
>>>>>>> I plan to fix this in the dogtag code.  Specifically,
>>>>>>> 1. I will modify the dogtag code to provide a default value in the case
>>>>>>> where the parameter does not exist.
>>>>>>> 2. I plan to add a function to the dogtag startup scripts to check and
>>>>>>> remake any required symlinks.
>>>>>>>
>>>>>>> Both of these changes are in the dogtag code, and do not affect this
>>>>>>> patch.  As a workaround, to confirm that the upgrade actually works, do
>>>>>>> the following manual steps on your upgraded instance:
>>>>>>>
>>>>>>> 1. Add the following parameter to CS.cfg
>>>>>>>       pidDir=/var/run
>>>>>>>
>>>>>>> 2. Add the following links;
>>>>>>>       cd /var/lib/pki-ca/common/lib
>>>>>>>       ln -s /usr/share/java/apache-commons-codec.jar
>>>>>>> apache-commons-codec.jar
>>>>>>>       ln -s /usr/share/java/log4j.jar log4j.jar
>>>>>>>
>>>>>>> 3 Restart the dogtag instance
>>>>>>>
>>>>>>>> HTH,
>>>>>>>> Martin
>>>>>>>
>>>>>>>
>>>>>
>>>>> The attached patch conditionalizes the changes, so that IPA supports
>>>>> both Dogtag versions.
>>>>>
>>>>> I didn't put it in platform code for two reasons
>>>>> - One platform (f17) can have either Dogtag version installed
>>>>> - I didn't want to conflict with Timo Aaltonen's "platformizing" work.
>>>>> According to his WIP patches, he plans to move the whole dogtag module
>>>>> into platform code.
>>>>>
>>>>> I used the existence of /usr/bin/pkispawn as a test for Dogtag 10. If
>>>>> you know a better way, please comment.
>>>>>
>>>>> The dogtag_version option is now always added to
>>>>>
>>>>> I've reverted the changes to install/ui/test/data/ipa_init.json, so
>>>>> you'll have to change the ports manually when testing the UI with Dogtag
>>>>> 10.
>>>>>
>>>>>
>>>>>
>>>>
>>>> Attaching a patch with fixed ipa-pki-proxy.conf, as discussed on IRC.
>>>>
>>>
>>> This approach looks good. I did not hit any regression on F17 with dogtag9, or
>>> clean installs of IPA+dogtag10. I think we are getting close to pushing this code.
>>>
>>> The only issues I hit so far are for upgraded dogtag9 instances (on F17):
>>>
>>> 1) The service did not start for me. There were some SELinux AVCs and even when
>>> I disabled SELinux the instance still did not start (logs attached).
>>>
>>> 2) Uninstall was also not clean, we leave some dogtag installation states for
>>> upgraded dogtag9 instance:
>>>
>>> # ipa-server-install --uninstall --unattended
>>> Shutting down all IPA services
>>> Removing IPA client configuration
>>> Unconfiguring ntpd
>>> Unconfiguring CA directory server
>>> Unconfiguring web server
>>> Unconfiguring krb5kdc
>>> Unconfiguring kadmin
>>> Unconfiguring directory server
>>> Unconfiguring ipa_memcached
>>> ipa         : ERROR    Some installation state for pki-cad has not been
>>> restored, see /var/lib/ipa/sysrestore/sysrestore.state
>>>
>>> /var/lib/ipa/sysrestore/sysrestore.state:
>>> [pki-cad]
>>> enabled = False
>>
>> Ade is working on a new build to address the dogtag upgrade issues.
>>
>> The left over state should be removed when we drop the separate instance
>> in the dogtag 9 -> 10 upgrade. I'm not completely sure reading the patch
>> who actually does this part, is this handled by dogtag?
>>
>> rob
>>
>
> The build is available on the developer repo.  Just do a yum update.
>
> As for this leftover state, this is an ipa file and such is never
> handled by dogtag.  It must be handled somewhere within the ipa code.

Well, the question is, what handles the dogtag 9 -> dogtag 10 upgrade?

If IPA isn't involved then it has no chance to remove this state.

rob