[Freeipa-devel] [PATCHES] Re: Changes to use a single database for dogtag and IPA

Fri Nov 23 08:02:28 UTC 2012

On 11/22/2012 05:54 PM, Petr Viktorin wrote:
> On 11/21/2012 01:46 PM, Martin Kosek wrote:
>> On 11/21/2012 10:54 AM, Petr Viktorin wrote:
>>> On 11/21/2012 10:48 AM, Martin Kosek wrote:
>>>> On 11/21/2012 10:46 AM, Martin Kosek wrote:
>>>>> On 11/20/2012 02:59 PM, Petr Viktorin wrote:
>>>>>> On 11/19/2012 03:14 PM, Petr Viktorin wrote:
>>>>>>> On 11/19/2012 02:09 PM, Martin Kosek wrote:
>>>>>>>> On 11/16/2012 02:25 PM, Martin Kosek wrote:
>>>>>>>>> On 11/16/2012 11:23 AM, Martin Kosek wrote:
>>>>>>>>>> On 11/15/2012 07:17 PM, Petr Viktorin wrote:
>>>>>>>>>>> On 11/15/2012 05:09 PM, Martin Kosek wrote:
>>>>>>>>>>>> On 11/15/2012 03:19 PM, Petr Viktorin wrote:
>>>>>>>>>>>>> Recently, the specfile changed (dce53e4) and the patch for
>>>>>>>>>>>>> changed Dogtag
>>>>>>>>>>>>> defaults made it to master independently (91e477b). Attaching
>>>>>>>>>>>>> rebased patch.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Note that to continue development on f17, you will need to use the
>>>>>>>>>>>>> dogtag-devel
>>>>>>>>>>>>> repo:
>>>>>>>>>>>>>       sudo yum-config-manager
>>>>>>>>>>>>> --add-repo=http://nkinder.fedorapeople.org/dogtag-devel/dogtag-devel-fedora.repo
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 11/13/2012 03:57 PM, Petr Viktorin wrote:
>>>>>>>>>>>>> [...]
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> For convenience, I've also pushed the changes to a personal
>>>>>>>>>>>>>> repository.
>>>>>>>>>>>>>> To fetch to branch "pviktori-dogtag-10" you can do:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>          git fetch -f git://github.com/encukou/freeipa.git
>>>>>>>>>>>>>> dogtag-10:pviktori-dogtag-10
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I started reviewing the patches, and found the first thing that looks
>>>>>>>>>>>> suspicious. I had IPA with 2 databases, then upgraded it to
>>>>>>>>>>>> single-database
>>>>>>>>>>>> IPA, the upgrade was OK.
>>>>>>>>>>>>
>>>>>>>>>>>> But when I uninstalled the IPA, PKI-IPA dirsrv instance was not
>>>>>>>>>>>> removed
>>>>>>>>>>>> because
>>>>>>>>>>>> when I installed single-db IPA afterwards, I had 2 dirsrv
>>>>>>>>>>>> instances running.
>>>>>>>>>>>
>>>>>>>>>>> You're right. This is an uninstaller error already present in 2.2:
>>>>>>>>>>> https://fedorahosted.org/freeipa/ticket/3258
>>>>>>>>>>>
>>>>>>>>>>> I'll start looking into it tomorrow, if nothing more important
>>>>>>>>>>> shows up.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks for the pointer. But this is definitely not a show stopper,
>>>>>>>>>> running
>>>>>>>>>> additional DS instance seems more or less benign and as you pointed
>>>>>>>>>> out, it is
>>>>>>>>>> rather an old bug.
>>>>>>>>>>
>>>>>>>>>> There are bigger issues. Now I focused on ipa-replica-manage and
>>>>>>>>>> ipa-csreplica-manage tools. ipa-replica-manage gets confused with the
>>>>>>>>>> additional replication agreements in IPA dirsrv instance (although
>>>>>>>>>> targeted to
>>>>>>>>>> nsDS5ReplicaRoot: o=ipaca).
>>>>>>>>>>
>>>>>>>>>> First scenario: 3 IPA servers with CA in this topology:
>>>>>>>>>>
>>>>>>>>>> B - A - C
>>>>>>>>>>
>>>>>>>>>> On A:
>>>>>>>>>> # ipa-replica-manage list `hostname`
>>>>>>>>>> vm-055.idm.lab.bos.redhat.com: replica
>>>>>>>>>> vm-070.idm.lab.bos.redhat.com: replica
>>>>>>>>>> vm-055.idm.lab.bos.redhat.com: replica
>>>>>>>>>> vm-070.idm.lab.bos.redhat.com: replica
>>>>>>>>>>
>>>>>>>>>> it should not display agreements that are for IPA only, not IPA CA
>>>>>>>>>> ones.
>>>>>>>>>>
>>>>>>>>>> Now, when I try to connect B to C, ipa-replica-manage succeeded:
>>>>>>>>>> [B] # ipa-replica-manage connect C
>>>>>>>>>> Connected 'B' to 'C'
>>>>>>>>>>
>>>>>>>>>> This changed the topology to:
>>>>>>>>>>         A
>>>>>>>>>>       /   \
>>>>>>>>>> B   -  C
>>>>>>>>>>
>>>>>>>>>> But ipa-csreplica-manage connect did not succeed then:
>>>>>>>>>> [B] # ipa-csreplica-manage connect C
>>>>>>>>>> Directory Manager password:
>>>>>>>>>>
>>>>>>>>>> This replication agreement already exists.
>>>>>>>>>>
>>>>>>>>>> Del command also failed for me:
>>>>>>>>>> [A] ipa-replica-manage del [C]
>>>>>>>>>>
>>>>>>>>>> Still trying to investigate why. If I manage to get some workable
>>>>>>>>>> fix during my
>>>>>>>>>> investigations, I will attach it later.
>>>>>>>>>>
>>>>>>>>>> Martin
>>>>>>>>>
>>>>>>>>> The fix for that for easier than expected. Attached patch restored
>>>>>>>>> the previous
>>>>>>>>> functionality for ipa-(cs)replica-manage. I tried that with all basic
>>>>>>>>> commands
>>>>>>>>> - add, del, connect, disconnect and it worked fine so far.
>>>>>>>>>
>>>>>>>>> But this was a case with all D10 masters, I will need to try if that
>>>>>>>>> flies with
>>>>>>>>> D9->D10 replicas or upgraded D9 masters.
>>>>>>>>>
>>>>>>>>> Martin
>>>>>>>>>
>>>>>>>>
>>>>>>>> I managed to create a 2.2 (F17) -> 3.1 (F18) replica, everything seem
>>>>>>>> to work
>>>>>>>> well. I just think we will need to also backport the previous patch at
>>>>>>>> least to
>>>>>>>> 3.0 and 2.2 versions to fix errors with ipa-replica-manage replication
>>>>>>>> management tool. I created a ticket for this purpose:
>>>>>>>>
>>>>>>>> https://fedorahosted.org/freeipa/ticket/3262
>>>>>>>>
>>>>>>>> Attaching a patch for IPA 2.2 branch in case somebody is also testing
>>>>>>>> it. With
>>>>>>>> this patch, I was able to list, force-sync, re-initialize, connect,
>>>>>>>> disconnect
>>>>>>>> from 2.2 to 3.1 replica without any errors.
>>>>>>>>
>>>>>>>> Martin
>>>>>>>>
>>>>>>>
>>>>>>> I checked ipa-csreplica-install, and found some more problems.
>>>>>>>
>>>>>>> In the "connect" and "disconnect" subcommands we now assume both masters
>>>>>>> use port 389 for the PKI DS. While the number is relatively easily
>>>>>>> parametrized, the assumption that both sides use the same port seems to
>>>>>>> run pretty deep.
>>>>>>> I'm working on a patch to detect if the remote server has merged DBs and
>>>>>>> use the appropriate port.
>>>>>>
>>>>>> Attaching patches to do this.
>>>>>> Patch 0100 adds an argument to IPAdmin to overrides guessing of the
>>>>>> protocol to
>>>>>> use.
>>>>>> Patch 0101 modifies ipa-csreplica-manage to figure out the ports of the DSs
>>>>>> involved and use those.
>>>>>>
>>>>>> Note that this touches code shared with ipa-replica-manage, please re-test
>>>>>> that
>>>>>> as well.
>>>>>>
>>>>>
>>>>> Works fine, I was able to create a network of few IPA 2.2 replicas and IPA
>>>>> 3.1
>>>>> replicas and use ipa-replica-manage and ipa-csreplica-manage to play with the
>>>>> agreements. No regression discovered so far.
>>>>>
>>>>> I just see that in patch 101 you touch setup_replication and force TLS as a
>>>>> default. But in this case, r_sslport parameter is never used and we can
>>>>> remove it.
>>>>>
>>>>> In 101, you also set LDAP+TLS as default connection protocol with
>>>>> +        super(CSReplicationManager, self).__init__(
>>>>> +            realm, hostname, dirman_passwd, port, starttls=True)
>>>>>                                                      ^^^^^^^^^^^^^
>>>>>
>>>>> Wouldn't we want to make LDAP+TLS as a default also in a bunch of
>>>>> ReplicationManager initializations in ipa-replica-manage? Otherwise, we use
>>>>> ldaps/SSL by default. AFAIU, LDAP+TLS is preferred over ldaps/SSL so this
>>>>> would
>>>>> be a good step to do. Adding Rob and Simo to CC to correct me if I miss
>>>>> anything and we want to keep using ldaps/SSL by default.
>>>> ... adding to CC now!
>>>>
>>>
>>> Yes, that would be good. I think it's out of scope for this patchset, though.
>>> Do we have a bug for this already?
>>> I think John or I can include it in the
>>> https://fedorahosted.org/freeipa/ticket/2660 or
>>> https://fedorahosted.org/freeipa/ticket/2652 efforts.
>>>
>>
>> Right, I think we do not have to do this now. I would not include it in 2652,
>> it is too general. I would prefer a separate ticket (if we agree with this
>> change) or inclusion in 2660.
>>
>> On a different note, I tried a scenario of 3.0 split instance IPA master and
>> 3.1 single instance replica and I got an error again:
>>
>> # ipa-replica-install replica-info-vm-055.idm.lab.bos.redhat.com.gpg --setup-ca
>> Directory Manager (existing master) password:
>> ...
>>    [28/30]: enabling compatibility plugin
>>    [29/30]: tuning directory server
>>    [30/30]: configuring directory to start on boot
>> Done configuring directory server (dirsrv).
>> Configuring certificate server (pki-tomcatd): Estimated time 3 minutes 30
>> seconds
>>    [1/16]: creating certificate server user
>>    [2/16]: configuring certificate server instance
>>
>> Your system may be partly configured.
>> Run /usr/sbin/ipa-server-install --uninstall to clean up.
>>
>> Unexpected error - see /var/log/ipareplica-install.log for details:
>> IOError: [Errno 2] No such file or directory:
>> '/var/lib/pki/pki-tomcat/alias/ca_backup_keys.p12'
>>
>> # ipa-ca-install replica-info-vm-055.idm.lab.bos.redhat.com.gpg
>> ...
>> Configuring certificate server (pki-tomcatd): Estimated time 3 minutes 30
>> seconds
>>    [1/15]: creating certificate server user
>>    [2/15]: configuring certificate server instance
>>
>> Your system may be partly configured.
>> Run /usr/sbin/ipa-server-install --uninstall to clean up.
>>
>> Unexpected error - see /var/log/ipareplica-ca-install.log for details:
>> IOError: [Errno 2] No such file or directory:
>> '/var/lib/pki/pki-tomcat/alias/ca_backup_keys.p12'
>>
>> # rpm -q pki-ca
>> pki-ca-10.0.0-0.52.b3.fc18.noarch
>>
>> # tail -1 /var/log/pki/pki-tomcat/ca/system
>> 3357.http-bio-8443-exec-1 - [21/Nov/2012:07:16:02 EST] [8] [3] In Ldap (bound)
>> connection pool to host vm-104.idm.lab.bos.redhat.com port 7389, Cannot connect
>> to LDAP server. Error: netscape.ldap.LDAPException: failed to connect to server
>> ldap://vm-104.idm.lab.bos.redhat.com:7389 (91)
>>
>> But when I try an ldapsearch to this address, it works:
>> # ldapsearch -H "ldap://vm-104.idm.lab.bos.redhat.com:7389" -s base -b
>> "o=ipaca" -D "cn=Directory Manager" -x -w kokos123
>> # extended LDIF
>>
>> # ipaca
>> dn: o=ipaca
>> objectClass: top
>> objectClass: organization
>> o: ipaca
>>
>> Attaching relevant logs and CC-ing Ade, I think we will need his help on this
>> one.
>>
>> IMO it is important to have this scenario working too, F17 IPA users who'd
>> upgrade to F18 and try to setup a replica may hit the same issue.
>>
>> Martin
>>
> 
> I have successfully installed a 3.1 replica of a 3.0 master on my VMs.
> 
> I believe the crash is due to broken setup, but I'll leave it to Ade to look at
> the Java exceptions.
> 

Hrm, I finally found the root cause. Its a good old SELinux AVC preventing
pki-ca connecting to unreserved port 7389. I assume you have simply SELinux
turned off in your setup. I have opened a SELinux policy Bug to fix that, Ade
is CCed:

https://bugzilla.redhat.com/show_bug.cgi?id=879516

With this in mind, and an easy workaround allowing me to create 3.0->3.1
replica (setenforce 0), I think we can wrap up and polish all patches in this
thread and push this beast to master.

Martin