[Freeipa-users] General status of my FreeIPA servers - is there a method for cleaning them?

[Dan Scott]

On Fri, Apr 13, 2012 at 15:24, Rich Megginson <rmeggins at redhat.com> wrote:
> On 04/13/2012 01:03 PM, Dan Scott wrote:
>>>> If I'm interpreting this correctly, it can't be deleted because it's
>>>> not a leaf node, but it doesn't have any sub-entries that I can delete
>>>> first.
>>>
>>> You are correct.  Try this:
>>>
>>> ldapsearch -D 'cn=directory manager' -W -v -b
>>>
>>> 'cn=fileserver5.ecg.mit.edu,cn=masters,cn=ipa,cn=etc,dc=ecg,dc=mit,dc=edu'
>>> '(|(objectclass=nstombstone)(objectclass=*))'
>>
>> Ahh, so there are some 'child' entries:
>>

[snip]

>> Is it safe to delete them?
>
> Yes.

I deleted them, but it's still complaining about a non-leaf:

[root at fileserver1 ~]# ldapmodify -f rmfileserver5.ldif -D
'cn=directory manager' -W
Enter LDAP Password:
deleting entry "cn=fileserver5.ecg.mit.edu,cn=masters,cn=ipa,cn=etc,dc=ecg,dc=mit,dc=edu"
ldap_delete: Operation not allowed on non-leaf (66)

[root at fileserver1 ~]# ldapsearch -D 'cn=directory manager' -W -b
'cn=fileserver5.ecg.mit.edu,cn=masters,cn=ipa,cn=etc,dc=ecg,dc=mit,dc=edu'
'(|(objectclass=nstombstone)(objectclass=*))'
Enter LDAP Password:
# extended LDIF
#
# LDAPv3
# base <cn=fileserver5.ecg.mit.edu,cn=masters,cn=ipa,cn=etc,dc=ecg,dc=mit,dc=edu>
with scope subtree
# filter: (|(objectclass=nstombstone)(objectclass=*))
# requesting: ALL
#

# fileserver5.ecg.mit.edu, masters, ipa, etc, ecg.mit.edu
dn: cn=fileserver5.ecg.mit.edu,cn=masters,cn=ipa,cn=etc,dc=ecg,dc=mit,dc=edu
cn: fileserver5.ecg.mit.edu
objectClass: top
objectClass: nsContainer

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1
[root at fileserver1 ~]#

>>>>>> I also see
>>>>>> inconsistent replication states on the servers. i.e. server1 shows
>>>>>> that it's replicating with server2 but server2 does not show that it's
>>>>>> replicating with server1.
>>>>>
>>>>>
>>>>> Do you have errors in the server2 log showing that it is attempting to
>>>>> replicate with server1 but failing with some error?
>>>>
>>>> [root at fileserver1 ~]# ipa-csreplica-manage list -v
>>>> fileserver1.ecg.mit.edu
>>>> Directory Manager password:
>>>>
>>>> fileserver2.ecg.mit.edu
>>>>   last init status: None
>>>>   last init ended: None
>>>>   last update status: 0 Replica acquired successfully: Incremental
>>>> update succeeded
>>>>   last update ended: 2012-04-13 17:57:39+00:00
>>>> [root at fileserver1 ~]# ipa-csreplica-manage list -v
>>>> fileserver2.ecg.mit.edu
>>>> Directory Manager password:
>>>>
>>>> fileserver1.ecg.mit.edu
>>>>   last init status: None
>>>>   last init ended: None
>>>>   last update status: 0 Replica acquired successfully: Incremental
>>>> update succeeded
>>>>   last update ended: 2012-04-13 17:57:41+00:00
>>>> fileserver3.ecg.mit.edu
>>>>   last init status: None
>>>>   last init ended: None
>>>>   last update status: 0 Replica acquired successfully: Incremental
>>>> update succeeded
>>>>   last update ended: 2012-04-13 17:57:41+00:00
>>>> [root at fileserver1 ~]# ipa-csreplica-manage list -v
>>>> fileserver3.ecg.mit.edu
>>>> Directory Manager password:
>>>>
>>>> fileserver2.ecg.mit.edu
>>>>   last init status: None
>>>>   last init ended: None
>>>>   last update status: 0 Replica acquired successfully: Incremental
>>>> update succeeded
>>>>   last update ended: 2012-04-13 17:57:44+00:00
>>>> fileserver1.ecg.mit.edu
>>>>   last init status: None
>>>>   last init ended: None
>>>>   last update status: 0 Replica acquired successfully: Incremental
>>>> update succeeded
>>>>   last update ended: 2012-04-13 17:57:43+00:00
>>>> [root at fileserver1 ~]#
>>>>
>>>> fileserver1's (and fileserver2s) /var/log/dirsrv/slapd-PKI-IPA/errors
>>>> contains lots of:
>>>> [13/Apr/2012:13:57:43 -0400] NSMMReplicationPlugin -
>>>> repl_set_mtn_referrals: could not set referrals for replica o=ipaca:
>>>> 20
>>>
>>>
>>> This error usually means a replica was deleted and the RUV needs to be
>>> cleaned.
>>> see http://port389.org/wiki/Howto:CLEANRUV
>>> and
>>> https://fedorahosted.org/freeipa/ticket/2303
>>> and
>>> https://fedorahosted.org/389/ticket/337
>>
>> OK, I've seen this before - is it important to remove them? I've had
>> to add and remove replicas so much that I don't really want to do it
>> unless it's necessary. I'm happy to live with them if it's not a
>> problem.
>
>
> It's not a problem until it's a problem :-)  I would go ahead and run
> CLEANRUV.

I cleaned up a load of these entries, but now I think I've broken the
replication between fileserver1 and 3:

fileserver1:/var/log/dirsrv/slapd-ECG-MIT-EDU/errors
[13/Apr/2012:15:57:56 -0400] NSMMReplicationPlugin - changelog program
- agmt="cn=meTofileserver3.ecg.mit.edu" (fileserver3:389): CSN
4f5039960000002b0000 not found, we aren't as up to date, or we purged
[13/Apr/2012:15:57:56 -0400] NSMMReplicationPlugin -
agmt="cn=meTofileserver3.ecg.mit.edu" (fileserver3:389): Data required
to update replica has been purged. The replica must be reinitialized.
[13/Apr/2012:15:57:56 -0400] NSMMReplicationPlugin -
agmt="cn=meTofileserver3.ecg.mit.edu" (fileserver3:389): Incremental
update failed and requires administrator action

fileserver3:/var/log/dirsrv/slapd-ECG-MIT-EDU/errors
[13/Apr/2012:16:19:38 -0400] NSMMReplicationPlugin - changelog program
- agmt="cn=meTofileserver1.ecg.mit.edu" (fileserver1:389): CSN
4f031e76001d000b0000 not found, we aren't as up to date, or we purged

Is it safe to run:
[root at fileserver3 ~]# ipa-replica-manage re-initialize --from
fileserver1.ecg.mit.edu

I want to make sure I get it the correct way round!

>>>> fileserver3's /var/log/dirsrv/slapd-PKI-IPA/errors contains lots of:
>>>> [13/Apr/2012:13:52:50 -0400] slapi_ldap_bind - Error: could not send
>>>> startTLS request: error -1 (Can't contact LDAP server) errno 107
>>>> (Transport endpoint is not connected)
>>>
>>>
>>> This is a real connection error - could be cert or hostname lookup
>>> related.
>>
>> How do I find out if it's cert or hostname lookup? Which hostname?
>> Fileserver3 runs DNS, and it seems to be working fine.
>
>
> Try ldapsearch - on server3
>
> LDAPTLS_CACERTDIR=/etc/dirsrv/slapd-PKI-IPA ldapsearch -x -ZZ -H
> ldap://server2.fqdn -D "cn=directory manager" -W -s base -b ""
>
> If that works, check to make sure the replication agreement has the correct
> server2.fqdn
>
> If that doesn't work, use ldapsearch -d 1 -x ..... to get further debugging
> information.

The replication agreements (according to ipa-replica-manage) all have
the correct host names - I'm not sure what ldapsearch command to run
to check the replication agreements.

>> The /var/log/dirsrv/slapd-ECG-MIT-EDU/errors is
>> now full of:
>>
>> [13/Apr/2012:14:59:19 -0400] NSMMReplicationPlugin - conn=1 op=571
>> csn=4f70a9e5000100060000: Can't created glue entry
>> cn=fileserver4.ecg.mit.edu,cn=masters,cn=ipa,cn=etc,dc=ecg,dc=mit,dc=edu
>> uniqueid=6949d104-775b11e1-abce82a1-a45dd3c3, error 68
>>
>> Should I delete the LDAP entry which is trying to replicate
>> fileserver2 with fileserver4?
>
>
> Yes.  And it may be due to the fact that the entry it is trying to delete
> has those tombstone children that have to be deleted too.

OK, I'll see how this goes, once the tombstones are gone.

Thanks,

Dan