[Freeipa-devel] [PATCH] Patch to allow IPA to work with dogtag 10 on f18

[Ade Lee]

On Thu, 2012-08-16 at 18:45 +0200, Martin Kosek wrote:
> On 08/16/2012 01:28 PM, Ade Lee wrote:
> > Patch attached this time.  I should know better than to do this in the
> > middle of the night ..
> > 
> > On Thu, 2012-08-16 at 09:12 +0200, Martin Kosek wrote:
> >> On 08/16/2012 07:53 AM, Ade Lee wrote:
> >>> On Wed, 2012-08-15 at 23:41 -0400, Ade Lee wrote:
> >>>> On Wed, 2012-08-15 at 16:34 +0200, Martin Kosek wrote:
> >>>>> On 08/15/2012 03:54 PM, Ade Lee wrote:
> >>>>>> On Wed, 2012-08-15 at 13:24 +0200, Martin Kosek wrote:
> >>>>>>> On 08/08/2012 10:05 PM, Ade Lee wrote:
> >>>>>>>> Hi, 
> >>>>>>>>
> >>>>>>>> Dogtag 10 is being released on f18, and has a number of changes that
> >>>>>>>> will affect IPA.  In particular, the following changes will affect
> >>>>>>>> current IPA code. 
> >>>>>>>>
> >>>>>>>> * The directory layout of the dogtag instance has changed.  Instead of
> >>>>>>>> using separate tomcat instances to host different subsystems, the
> >>>>>>>> standard dogtag installation will allow one to install a CA. KRA, OCSP
> >>>>>>>> and TKS within the same instance.  There have been corresponding changes
> >>>>>>>> in the directory layout, as well as the default instance name
> >>>>>>>> (pki-tomcat instead of pki-ca), and startup daemon (pki-tomcatd, instead
> >>>>>>>> of pki-cad, pki-krad etc.) 
> >>>>>>>>
> >>>>>>>> * The default instance will use only four ports (HTTPS, HTTP, AJP and
> >>>>>>>> tomcat shutdown port) rather than the 6 previously used.  The default
> >>>>>>>> ports will be changed to the standard tomcat ports.  As these ports are
> >>>>>>>> local to the ipa server machine, this should not cause too much
> >>>>>>>> disruption. 
> >>>>>>>>
> >>>>>>>> * There is a new single step installer written in python.
> >>>>>>>> (pkispawn/destroy) vs. pkicreate/pkisilent/pkiremove.
> >>>>>>>>
> >>>>>>>> * Dogtag 10 runs on tomcat7 - with a new corresponding version of
> >>>>>>>> tomcatjss.
> >>>>>>>>
> >>>>>>>> The attached patch integrates all the above changes in IPA installation
> >>>>>>>> and maintenance code.  Once the patch is applied, users will be able to:
> >>>>>>>>
> >>>>>>>> 1. run ipa-server-install to completion on f18 with dogtag 10.
> >>>>>>>> 2. install a new replica on f18 on dogtag 10.
> >>>>>>>> 3. upgrade an f17 machine with an existing IPA instance to f18/ dogtag
> >>>>>>>> 10 - and have that old-style dogtag instance continue to run correctly.
> >>>>>>>> This will require the installation of the latest version of tomcatjss as
> >>>>>>>> well as the installation of tomcat6.  The old-style instance will
> >>>>>>>> continue to use tomcat6.
> >>>>>>>> 4. in addition, the new cert renewal code has been patched and should
> >>>>>>>> continue to work.
> >>>>>>>>
> >>>>>>>> What is not yet completed / supported:
> >>>>>>>>
> >>>>>>>> 1. Installation with an external CA is not yet completed in the new
> >>>>>>>> installer.  We plan to complete this soon.
> >>>>>>>>
> >>>>>>>> 2. There is some IPA upgrade code that has not yet been touched
> >>>>>>>> (install/tools/ipa-upgradeconfig).
> >>>>>>>>
> >>>>>>>> 3. A script needs to be written to allow admins to convert their
> >>>>>>>> old-style dogtag instances to new style instances, as well as code to
> >>>>>>>> periodically prompt admins to do this.
> >>>>>>>>
> >>>>>>>> 4. Installation of old-style instances using pkicreate/pkisilent on
> >>>>>>>> dogtag 10 will no longer be supported, and will be disabled soon.
> >>>>>>>>
> >>>>>>>> 5.  The pki-selinux policy has been updated to reflect these changes,
> >>>>>>>> but is still in flux.  In fact, it is our intention to place the dogtag
> >>>>>>>> selinux policy in the base selinux policy for f18.  In the meantime, it
> >>>>>>>> may be necessary to run installs in permissive mode.
> >>>>>>>>
> >>>>>>>> The dogtag 10 code will be released shortly into f18.  Prior to that
> >>>>>>>> though, we have placed the new dogtag 10 and tomcatjss code in a
> >>>>>>>> developer repo that is located at 
> >>>>>>>> http://nkinder.fedorapeople.org/dogtag-devel/
> >>>>>>>>
> >>>>>>>> Testing can be done on both f18 and f17 - although the target platform -
> >>>>>>>> and the only platform for which official builds will be created is f18.
> >>>>>>>>
> >>>>>>>> Thanks, 
> >>>>>>>> Ade
> >>>>>>>>
> >>>>>>>
> >>>>>>> Hi Ade,
> >>>>>>>
> >>>>>>> Thanks for the patch, I started with review and integration tests (currently
> >>>>>>> running on Fedora 17 with Nathan's repo).
> >>>>>>>
> >>>>>>> Installation on single master was smooth, it worked just fine, even with
> >>>>>>> enforced SELinux, without any error - kudos to you and the whole dogtag team.
> >>>>>>> The resulting logs and the structure of your log directory seems improved. I
> >>>>>>> believe that the brand new Python installers will make it easier to debug
> >>>>>>> issues with dogtag installation when they come.  When I tried our unit tests or
> >>>>>>> some simple cert operation, it worked fine as well.
> >>>>>>>
> >>>>>>> Now the bad news, or rather few issues and suggestions I found:
> >>>>>>>
> >>>>>>> 1) As we already discussed on IRC, tomcat 7 was not pulled automatically on
> >>>>>>> Fedora 17 when I updated pki-ca, you somewhere miss a Requires.
> >>>>>>>
> >>>>>> We have a dogtag patch that is currently in review that will address
> >>>>>> this.  Once this is in, tomcatjss >=7.0.0 will be required for f17+,
> >>>>>> rather than f18+
> >>>>>>
> >>>>>>> 2) I had installed IPA with dogtag10 on master. However, CA installation on a
> >>>>>>> replica (ipa-ca-install) with dogtag9 failed - is this expectable?
> >>>>>>>
> >>>>>> Yes.  The current IPA patch is designed to work with dogtag 10 only,
> >>>>>> which will be officially available on f18+.  So if you update to dogtag
> >>>>>> 10, you must have this patch and visa versa.  We probably need to add
> >>>>>> the relevant requires to IPA to enforce this.
> >>>>>>
> >>>>>> If you have an existing dogtag 9 instance, and you upgrade to the new
> >>>>>> dogtag 10 and patched IPA, then that instance will continue to work.
> >>>>>> But any new instances would be created using dogtag 10.
> >>>>>>
> >>>>>>> 3) I had installed IPA with dogtag10 on master. Replica had dogtag10 as well
> >>>>>>> and I got the following error:
> >>>>>>>
> >>>>>>> # ipa-ca-install /home/mkosek/replica-info-vm-114.idm.lab.bos.redhat.com.gpg
> >>>>>>> ...
> >>>>>>> Configuring certificate server: Estimated time 3 minutes 30 seconds
> >>>>>>>   [1/14]: creating certificate server user
> >>>>>>>   [2/14]: configuring certificate server instance
> >>>>>>>
> >>>>>>> Your system may be partly configured.
> >>>>>>> Run /usr/sbin/ipa-server-install --uninstall to clean up.
> >>>>>>>
> >>>>>>> Unexpected error - see /var/log/ipareplica-ca-install.log for details:
> >>>>>>> IOError: [Errno 2] No such file or directory:
> >>>>>>> '/var/lib/pki/pki-tomcat/alias/ca_backup_keys.p12'
> >>>>>>>
> >>>>>>> Root cause:
> >>>>>>> ...
> >>>>>>>   File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line
> >>>>>>> 625, in __spawn_instance
> >>>>>>>     "/root/cacert.p12")
> >>>>>>> ...
> >>>>>>>
> >>>>>> I need to look into this.  I had fixed ipa-replica-install, rather than
> >>>>>> ipa-ca-install to create replicas.  I didn't know ipa-ca-install
> >>>>>> existed!  Should not be too bad to fix though - most likely just need to
> >>>>>> move files to the right place.
> >>>>>
> >>>>> Ok, thanks! Btw. CA on replica can be either installed during
> >>>>> ipa-replica-install time (when --setup-ca option is passed, you probably used
> >>>>> that one) and the aforementioned ipa-ca-install run after ipa-replica-install.
> >>>>>
> >>>> I will be testing this out again.  As ipa-ca-install uses the same code
> >>>> as ipa-replica-install, I would have expected it to work.  Did you try
> >>>> it with selinux in permissive mode?
> >>>>
> >>> OK -- I tested this out and realized that between the time I initially
> >>> tested this and your tests, we had changed a URL
> >>> from /ca/pki/installer/installToken to /ca/rest/installer/installToken,
> >>> but I forgot to update ipa-pki-proxy.conf. 
> >>>
> >>> The new patch has been rebased and changes the relevant patch.  With
> >>> this, the CA replica installs correctly.
> >>
> >> Umh, where can I find the new rebased patch? :-)
> >>
> >>>
> >>> There was one issue that I encountered.  I have seen this once before
> >>> and will need to investigate further.  IPA sometimes restarts the dogtag
> >>> instance by doing systemctl stop pki-tomcatd.target.  and then systemctl
> >>> start pki-tomcatd.target.   I have noticed that occasionally the start
> >>> invocation fails, while the stop and restart invocations succeed just
> >>> fine.  This makes absolutely no sense because restart is essentially
> >>> just stop and then start.
> >>>
> >>> I think this is actually a bug in systemd.  If I reboot the machine, it
> >>> all works perfectly.  If we can't figure out whats going on with
> >>> systemd, we can always replace this start/stop with starting/stopping
> >>> the service directly.  (pki-tomcatd at pki-tomcat.service).  That seems to
> >>> work all the time with no problems.
> >>>
> >>> I suggest we leave this fix (if needed) for a separate patch.
> >>
> >> Ok, I will see if I hit this issue again. Btw. is it recommended in systemd to
> >> use target units this way? I thought that only service files are meant to be
> >> used for manipulation with a service. As you said yourself, using service unit
> >> file for restart worked every time.
> >>
> >> Martin
> > 
> 
> Now, I tested the rebased patch with VMs with permissive SELinux. IPA server
> install was OK, as always, but replica installation ended up with error.
> Although it got much further than before:
> 
> # ipa-ca-install /home/mkosek/replica-info-vm-114.idm.lab.bos.redhat.com.gpg
> ...
>   [3/3]: restarting directory server
> done configuring pkids.
> Configuring certificate server: Estimated time 3 minutes 30 seconds
>   [1/14]: creating certificate server user
>   [2/14]: configuring certificate server instance
>   [3/14]: disabling nonces
>   [4/14]: importing CA chain to RA certificate database
>   [5/14]: fixing RA database permissions
>   [6/14]: setting up signing cert profile
>   [7/14]: set up CRL publishing
>   [8/14]: set certificate subject base
>   [9/14]: enabling Subject Key Identifier
>   [10/14]: configuring certificate server to start on boot
>   [11/14]: configure certmonger for renewals
>   [12/14]: configure clone certificate renewals
>   [13/14]: configure Server-Cert certificate renewal
>   [14/14]: Configure HTTP to proxy connections
> done configuring pki-tomcatd.
> Restarting the directory and certificate servers
> 
> Your system may be partly configured.
> Run /usr/sbin/ipa-server-install --uninstall to clean up.
> 
> It seems like that CA on a replica did not start and so a check for port 8080
> failed. Attached logs (pki-ca-logs.tgz) may provide more information to this issue.

This is the issue I mentioned before about systemctl not starting the
target.  If you try a systemctl restart pki-tomcatd.target, it works.
I will change the code to start the service rather than the target to
avoid this whole problem.

> 
> Now I am also testing upgrade from IPA+dogtag9 to PatchedIPA+dogtag10
> (permissive SELinux). Now, the service was upgraded smoothly, CA service
> could be restarted without failure. However, certificate operations now
> did not work:
> 
> # ipactl restart
> Restarting Directory Service
> Restarting KDC Service
> Restarting KPASSWD Service
> Restarting MEMCACHE Service
> Restarting HTTP Service
> Restarting CA Service
> # ipactl status
> Directory Service: RUNNING
> KDC Service: RUNNING
> KPASSWD Service: RUNNING
> MEMCACHE Service: RUNNING
> HTTP Service: RUNNING
> CA Service: RUNNING
> # ipa cert-show 1
> ipa: ERROR: Certificate operation cannot be completed: Unable to
> communicate with CMS (Internal Server Error)
> 
> Attaching logs for this case as well (ipa-ca-logs-upgrade.tgz).
> 

Will look into this.

> HTH,
> Martin