[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [Freeipa-users] Still not working -- Re: What to do next???: IPA replica host entry is removed on web UI by mistake



Hi JR, Rob and Rich,

Thanks a lot for helping! A massage may be the choice for me now. :)

Though I still have two questions here. :)

 1, do you have an idea on how to clear the ghost RUVs thoroughly in one run? For my case today it took me quite some time to clear it again and again from across server farm -- it looks like the affected LDAP entries are overwritten from each other, like a basket of bumping balls.

 2, And, does it bring troubles if I also run:

  ipa-csreplica-manage del <failedIPAReplica> --force   ## on IPA master

and 

  clear the CA ghost RUV record from under 'cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config'? 

I thought this above could be more complete, But the link http://directory.fedoraproject.org/wiki/Howto:CLEANRUV documented only user LDAP backend and normal user LDAP replica, not including this CA replication and CA ldap backend clearance.  

So I got confused on the purposes the document link didn't mention this (CA). It is because clear CA RUV is wrong? or the author just took it for granted that all users are non-newbies, any ideas?   :)

Thanks a lot for your help today.


--David



  

--David




From: JR Aquino <JR Aquino citrix com>
To: David Copperfield <cao2dan yahoo com>
Cc: "freeipa-users redhat com" <freeipa-users redhat com>; Rob Crittenden <rcritten redhat com>
Sent: Wednesday, May 16, 2012 4:41 PM
Subject: Re: Still not working -- Re: [Freeipa-users] What to do next???: IPA replica host entry is removed on web UI by mistake

Whew, glad to hear you got through it!

The 389 ds crew is working on making the cleanruv into an internal automated process. I empathize completely.

The gssapi errors are generally benign. They come up because ldap starts before the kdc.

"Keeping your head in the cloud"
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Jr Aquino | Sr. Information Security Specialist
GIAC Certified Incident Handler | GIAC WebApp Penetration Tester
Citrix Online | 7408 Hollister Avenue | Goleta, CA 93117
jr aquino citrix com<mailto:jr aquino citrix com>
http://www.citrixonline.com

On May 16, 2012, at 4:29 PM, "David Copperfield" <cao2dan yahoo com<mailto:cao2dan yahoo com>> wrote:

Could that be because of removing ghost entries in CA database?

Another possible place could be the deleting/clearing option itself. One annoying thing that I've found is:

I cleared the RUV records from IPA servers one by one, then I restart IPA services on the servers one by one again, ldapsearch showed that the RUV ghost entries popped up again. :(

I had to kill it again and again across the IPA server farms, then restart IPA servers one by one, check again, until the ghost RUV entries disappeared from all and didn't come back -- It is very, VERY exhausting and annoying.

After that I still need to stop IPA replica first, then restart IPA master and until now it worked -- ipa commands and kinit worked.  At last I brought up the valid replica and it worked this time as well.

Now it was time to reinstall the failed IPA replica and it was installed and up and running well.

After I tested with 'ipa user-add', 'ipa-user-delete' and found that the replication did work across the IPA master and IPA replicas. I tested the last time and found the following messages in the error log file on IPA master, it maybe harmless but I am not sure:

[16/May/2012:16:18:36 -0700] - 389-Directory/1.2.9.16 B2012.023.214 starting up[16/May/2012:16:18:36 -0700] schema-compat-plugin - warning: no entries set up under ou=SUDOers, dc=jigsaw,dc=com
[16/May/2012:16:18:36 -0700] - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=jigsaw,dc=com--no CoS Templates found, which should be added before the CoS Definition.
[16/May/2012:16:18:36 -0700] - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=jigsaw,dc=com--no CoS Templates found, which should be added before the CoS Definition.
[16/May/2012:16:18:36 -0700] set_krb5_creds - Could not get initial credentials for principal [ldap/ipamaster example com EXAMPLE COM<mailto:ldap/ipamaster example com EXAMPLE COM>] in keytab [WRFILE:/etc/dirsrv/ds.keytab]: -1765328324 (Generic error (see e-text))[16/May/2012:16:18:36 -0700] - slapd started.  Listening on All Interfaces port 389 for LDAP requests
[16/May/2012:16:18:36 -0700] set_krb5_creds - Could not get initial credentials for principal [ldap/ipamaster example com EXAMPLE COM<mailto:ldap/ipamaster example com EXAMPLE COM>] in keytab [WRFILE:/etc/dirsrv/ds.keytab]: -1765328324 (Generic error (see e-text))[16/May/2012:16:18:36 -0700] - Listening on All Interfaces port 636 for LDAPS requests
[16/May/2012:16:18:36 -0700] - Listening on /var/run/slapd-EXAMPLE-COM.socket for LDAPI requests
[16/May/2012:16:18:36 -0700] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Credentials cache file '/tmp/krb5cc_496' not found))
[16/May/2012:16:18:36 -0700] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error)
[16/May/2012:16:18:36 -0700] NSMMReplicationPlugin - agmt="cn=meToipareplica02.example.com" (ipareplica02:389): Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Credentials cache file '/tmp/krb5cc_496' not found))
[16/May/2012:16:18:36 -0700] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Credentials cache file '/tmp/krb5cc_496' not found))
[16/May/2012:16:18:36 -0700] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error)
[16/May/2012:16:18:36 -0700] NSMMReplicationPlugin - agmt="cn=meToipareplica01.example.com" (ipareplica01:389): Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Credentials cache file '/tmp/krb5cc_496' not found))[16/May/2012:16:18:39 -0700] NSMMReplicationPlugin - agmt="cn=meToipareplica02.example.com" (ipareplica02:389): Replication bind with GSSAPI auth resumed
[16/May/2012:16:18:39 -0700] NSMMReplicationPlugin - agmt="cn=meToipareplica01.example.com" (ipareplica01:389): Replication bind with GSSAPI auth resumed


--David


________________________________
From: JR Aquino <JR Aquino citrix com<mailto:JR Aquino citrix com>>
To: David Copperfield <cao2dan yahoo com<mailto:cao2dan yahoo com>>
Cc: JR Aquino <JR Aquino citrix com<mailto:JR Aquino citrix com>>; Rob Crittenden <rcritten redhat com<mailto:rcritten redhat com>>; "freeipa-users redhat com<mailto:freeipa-users redhat com>" <freeipa-users redhat com<mailto:freeipa-users redhat com>>
Sent: Wednesday, May 16, 2012 4:00 PM
Subject: Re: Still not working -- Re: [Freeipa-users] What to do next???: IPA replica host entry is removed on web UI by mistake

Try: ipactl stop then ipactl start

Doesn't look like dirsrv is running on 389 and 636

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Jr Aquino | Sr. Information Security Specialist
GIAC Certified Incident Handler | GIAC WebApp Penetration Tester
Citrix Online | 7408 Hollister Avenue | Goleta, CA 93117<x-apple-data-detectors://0/0>
T:  +1 805.690.3478<tel:+1%C2%A0805.690.3478>
C: +1 805.717.0365<tel:+1%20805.717.0365>
jr aquino citrixonline com<mailto:jr aquino citrixonline com><mailto:jr aquino citrixonline com<mailto:jr aquino citrixonline com>>
http://www.citrixonline.com<http://www.citrixonline.com/>

On May 16, 2012, at 2:54 PM, David Copperfield wrote:

Sorry to declare success too quick, :( In fact, it is worse now, the IPA master fail after performing the above steps including the RUV cleaning.  I've only one working replica and I'm afraid to do anything on it.

On The IPA master, after I ran 'service ipa restart' it reported OK, but  'ipa user-find' failed. so I cleared my Kerboers TGT ticket, ran 'kinit admin' to try my luck, the IPA master  failed with the following message, it showed that 389 port listening disappeared for unknown reasons.

[root ipamaster slapd-EXAMPLE-COM]# kinit admin

kinit: Generic error (see e-text) while getting initial credentials
[root ipamaster slapd-EXAMPLE-COM]# netstat -antup | grep -i LISTEN | grep ns
tcp        0      0 :::7389                    :::*                        LISTEN      6550/ns-slapd
tcp        0      0 :::7390                    :::*                        LISTEN      6550/ns-slapd
[root ipamaster slapd-EXAMPLE-COM]#

The error logs are pasted here too.

[16/May/2012:14:41:43 -0700] set_krb5_creds - Could not get initial credentials for principal [ldap/ipamaster example com EXAMPLE COM<mailto:ipamaster example com EXAMPLE COM><mailto:ldap/ipamaster example com EXAMPLE COM<mailto:ipamaster example com EXAMPLE COM>>] in keytab [WRFILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm)
[16/May/2012:14:41:43 -0700] - slapd started.  Listening on All Interfaces port 389 for LDAP requests
[16/May/2012:14:41:43 -0700] - Listening on All Interfaces port 636 for LDAPS requests
[16/May/2012:14:41:43 -0700] - Listening on /var/run/slapd-EXAMPLE-COM.socket for LDAPI requests
[16/May/2012:14:41:43 -0700] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Credentials cache file '/tmp/krb5cc_496' not found))
[16/May/2012:14:41:43 -0700] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error)
[16/May/2012:14:41:43 -0700] NSMMReplicationPlugin - agmt="cn=meToipareplica01.example.com" (ipareplica01:389): Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Credentials cache file '/tmp/krb5cc_496' not found))
[16/May/2012:14:41:46 -0700] NSMMReplicationPlugin - agmt="cn=meToipareplica01.example.com" (ipareplica01:389): Replication bind with GSSAPI auth resumed

Thanks.

--David

________________________________
From: David Copperfield <cao2dan yahoo com<mailto:cao2dan yahoo com><mailto:cao2dan yahoo com<mailto:cao2dan yahoo com>>>
To: JR Aquino <JR Aquino citrix com<mailto:JR Aquino citrix com><mailto:JR Aquino citrix com<mailto:JR Aquino citrix com>>>
Cc: "freeipa-users redhat com<mailto:freeipa-users redhat com><mailto:freeipa-users redhat com<mailto:freeipa-users redhat com>>" <freeipa-users redhat com<mailto:freeipa-users redhat com><mailto:freeipa-users redhat com<mailto:freeipa-users redhat com>>>
Sent: Wednesday, May 16, 2012 1:23 PM
Subject: Re: [Freeipa-users] What to do next???: IPA replica host entry is removed on web UI by mistake

Hi JR,

Thanks a lot! It works perfectly.

The only extra thing probably goes with 2.1.3 only: I need to find and clear ghost RUV records for CA database, and remove it from master and all other live replicas as well.

BTW, on 2.2.0 the two database backends still are separate, or merged into one?

Thanks.

--David

________________________________
From: JR Aquino <JR Aquino citrix com<mailto:JR Aquino citrix com><mailto:JR Aquino citrix com<mailto:JR Aquino citrix com>>>
To: David Copperfield <cao2dan yahoo com<mailto:cao2dan yahoo com><mailto:cao2dan yahoo com<mailto:cao2dan yahoo com>>>
Cc: FreeIPAUsers <freeipa-users redhat com<mailto:freeipa-users redhat com><mailto:freeipa-users redhat com<mailto:freeipa-users redhat com>>>
Sent: Wednesday, May 16, 2012 12:57 PM
Subject: Re: [Freeipa-users] What to do next???: IPA replica host entry is removed on web UI by mistake

On May 16, 2012, at 12:23 PM, David Copperfield wrote:

> Hi all,
>
>  I accidentally removed one of my IPA replica host on IPA web UI by mistake, on the host list I planed to remove ipaclient02.example.com<http://ipaclient02.example.com><http://ipaclient02.example.com/>, but accidentally the mouse moved to ipareplica02.example.com<http://ipareplica02.example.com><http://ipareplica02.example.com/> and the latter got removed without a prompt.
>
> I realized the mistake and tried to recover from this disaster but it was already too late, the change propagated to all the replicas and the poor ipareplica02 now stops functioning.
>
> [root ipareplica02 slapd-EXAMPLE-COM]# ipa service-find
> ipa: ERROR: cannot connect to u'https://ipareplica02.qe9.jigsaw.com/ipa/xml': Internal Server Error
> [root ipareplica02 slapd-EXAMPLE-COM]# ipa user-find
> ipa: ERROR: cannot connect to u'https://ipareplica02.qe9.jigsaw.com/ipa/xml': Internal Server Error
> [root ipareplica02 slapd-EXAMPLE-COM]# ipa host-find
> ipa: ERROR: cannot connect to u'https://ipareplica02.qe9.jigsaw.com/ipa/xml': Internal Server Error
> [root ipareplica02 slapd-EXAMPLE-COM]#
>
> On the IPA master, It was found that ipareplica02 didn't show up in 'host-find' list or 'service-find' list. Though it still showed in the master list reported by 'ipa-replica-manage' and 'ipa-csreplica-manage', the real command 'ipa-replica-manage list ipareplica02' fails with LDAP could't reach error.
>
> What should I do now? Is there are any other ways to recover besides uninstall and reinstall of IPA replica ipareplica02?
>
>  BTW, it will be more than appreciated if the web UI could pop up a warning prompt when removing host/services entries associated with IPA masters and IPA replicas.

Been there... Done that... The bug is fixed in 2.2... It will prompt and prevent you from deleting a replica host if there is an agreement.

To clean up...

0. On the master replica: ipa-replica-manage del ipareplica02.example.com<http://ipareplica02.example.com><http://ipareplica02.example.com> --force
-This will delete the replica agreement for the host.

1. $ ldapsearch -xLLL -D "cn=directory manager" -W -b dc=example,dc=com \
'(&(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff)(objectclass=nstombstone))'

Look for your your nsds50ruv that matches your ghost replica.

2. Create an ldif following the directions here: http://directory.fedoraproject.org/wiki/Howto:CLEANRUV
Something like:

$ cat cleanup.ldif
dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config
changetype: modify
replace: nsds5task
nsds5task: CLEANRUV## <- ## == The ReplicaID number for the ghost replica.

3. Run on all of the remaining replicas: ldapmodify -x -D "cn=directory manager" -W -f fixed.ldif
- This removes the ghost entry.

4. on the broken replica: ipa-server-install --uninstall

5. Follow the normal directions for 'installing a replica'
- on master: ipa-replica-prepare ipareplica02.example.com<http://ipareplica02.example.com><http://ipareplica02.example.com>
- scp /path/to/ipareplica02.example.com.gpg  ipareplica02.example.com<http://ipareplica02.example.com><http://ipareplica02.example.com>: ipareplica02.example.com<http://ipareplica02.example.com><http://ipareplica02.example.com.gp/>.gpg
- on replica: ipa-replica-install  ipareplica02.example.com<http://ipareplica02.example.com><http://ipareplica02.example.com> --whatever_options_you_used_previously

6. Check to make sure the server was built correctly and command work as expected: kinit admin, ipa user-find, ipa host-find, id admin, etc etc

7. Sigh and drink coffee

> Thanks.
>
> --David
> From: Rich Megginson <rmeggins redhat com<mailto:rmeggins redhat com><mailto:rmeggins redhat com<mailto:rmeggins redhat com>>>
> To: Ben Ho <ben13ho hotmail com<mailto:ben13ho hotmail com><mailto:ben13ho hotmail com<mailto:ben13ho hotmail com>>>
> Cc: freeipa-users redhat com<mailto:freeipa-users redhat com><mailto:freeipa-users redhat com<mailto:freeipa-users redhat com>>
> Sent: Tuesday, May 15, 2012 5:33 PM
> Subject: Re: [Freeipa-users] Help with ipa-replica-manage
>
> On 05/15/2012 02:49 PM, Ben Ho wrote:
>> This is the information I retrieved about my server.
>>
>> ipa-server-selinux-2.1.3-9.el6.x86_64
>> ipa-client-2.1.3-9.el6.x86_64
>> ipa-server-2.1.3-9.el6.x86_64
>> CentOS release 6.2
>> 389-ds-base-1.2.9.14-1.el6_2.2.x86_64
>>
>> Thanks again.
>
> Is replication otherwise working?
>
>>
>> -Ben
>>
>> Date: Tue, 15 May 2012 13:15:46 -0600
>> From: rmeggins redhat com<mailto:rmeggins redhat com><mailto:rmeggins redhat com<mailto:rmeggins redhat com>>
>> To: ben13ho hotmail com<mailto:ben13ho hotmail com><mailto:ben13ho hotmail com<mailto:ben13ho hotmail com>>
>> CC: freeipa-users redhat com<mailto:freeipa-users redhat com><mailto:freeipa-users redhat com<mailto:freeipa-users redhat com>>
>> Subject: Re: [Freeipa-users] Help with ipa-replica-manage
>>
>> On 05/15/2012 01:00 PM, Ben Ho wrote:
>> Hello,
>>  I am pretty new to IPA.  Right now I have three servers that are running IPA.  I am trying to replicate one server to two other servers.  I use this command:
>>
>> ipa-replica-manage re-initialize --from example2.edu<http://example2.edu><http://example2.edu>
>>
>>  On the first server I need to replicate, it works fine.  However, on the second server I get this message in my log files.  The errors get printed out once every 1 to 5 minutes.
>>
>> [15/May/2012:14:22:43 -0400] NSMMReplicationPlugin - agmt="cn=meToexample1.edu" (example1:389): Schema replication update failed: Type or value exists
>> [15/May/2012:14:22:43 -0400] NSMMReplicationPlugin - agmt="cn=meToexample1.edu" (example1:389): Warning: unable to replicate schema: rc=1
>> [15/May/2012:14:22:47 -0400] NSMMReplicationPlugin - agmt="cn=meToexample2.edu" (example2:389): Schema replication update failed: Type or value exists
>> [15/May/2012:14:22:47 -0400] NSMMReplicationPlugin - agmt="cn=meToexample2.edu" (example2:389): Warning: unable to replicate schema: rc=1
>>
>>
>>  Again, I am pretty new to this, so any help or tips would be appreciated.
>>
>> What platform and what version of 389-ds-base and ipa-server for all of your servers?
>>
>>
>>  Thanks!
>>
>> -Ben
>>
>>
>>
>> _______________________________________________
>> Freeipa-users mailing list
>>
>> Freeipa-users redhat com<mailto:Freeipa-users redhat com><mailto:Freeipa-users redhat com<mailto:Freeipa-users redhat com>>
>> https://www.redhat.com/mailman/listinfo/freeipa-users
>>
>
>
> _______________________________________________
> Freeipa-users mailing list
> Freeipa-users redhat com<mailto:Freeipa-users redhat com><mailto:Freeipa-users redhat com<mailto:Freeipa-users redhat com>>
> https://www.redhat.com/mailman/listinfo/freeipa-users
>
> _______________________________________________
> Freeipa-users mailing list
> Freeipa-users redhat com<mailto:Freeipa-users redhat com><mailto:Freeipa-users redhat com<mailto:Freeipa-users redhat com>>
> https://www.redhat.com/mailman/listinfo/freeipa-users




_______________________________________________
Freeipa-users mailing list
Freeipa-users redhat com<mailto:Freeipa-users redhat com><mailto:Freeipa-users redhat com<mailto:Freeipa-users redhat com>>
https://www.redhat.com/mailman/listinfo/freeipa-users







[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]