[Freeipa-users] Please help: How to restore IPA Master/Replicas from daily IPA Replica setup???

Mon May 21 23:01:20 UTC 2012

On 05/21/2012 01:25 PM, Gelen James wrote:
> Hi Rob,
>
> Just wonder whether your guys have abandoned IPA 2.1.3 users on Redhat
> 6.2 or not. :(
>
> The IPA replication/restoration procedure/document request has been
> submitted for more than a week, but I can not see any meaningful work
> has done for customers although IPA replication and restoration is so
> vital to users' production IPA reliability! 
>
> Even when after I've done a lot of investigation work and asking for
> helps/suggestions, there is still no much attentions paid from you
> guys. Am I, or any others users here, are just non-paid Q/A IPA team
> stuff could be ignored for no reasons :)
>
>  I've mentioned this again and again, and urging IPA team to setup a
> typical user setup, because only this way you can see what the
> problems IPA administrators/users are facing and scared of.  But
> unfortunately, we don't have a feeling that you have done so. 
>   
>  Thanks.
>
> --Gelen
>

Hello Glen,

We have not done so because we are pretty busy preparing next release
and were hoping that our replies were sufficient to help you to figure
out the best procedure that works for you. JR has a running environment
so his guidance is first hand. We tried to provide as much help as we can.

We also have not been going the path of setting the environment because
we are not sure what is your typical environment and what are the main
concerns. Your input is very valuable but it is one of the first clearly
spelled data points. We need to get a bit more info to make sure that we
are addressing the right use case and problem.
We apologize for the delays but the turn around would not be fast. It
will  take us at least several weeks to come up with something we are
comfortable with upstream and downstream. I hear your frustration and
feel the urgency but we can't move faster than we can, sorry. Please do
not feel abandoned we are working hard too.

Also it seems that setting the environment and crafting the guidelines
should also be combined with attempt to automate the process. I already
contacted Foreman project developers in attempt to integrate the replica
provisioning for scalability and disaster recovery cases. We will have a
conversation with them later this week. This might help with doing
automatic provisioning of replicas rather than manually performing
couple dozen of steps. Would such integration help?

Also if you need some immediate help opening a support ticket might be a
better avenue to get the situation prioritized accordingly. 

Sorry for delays,
Thanks
Dmitri 

> ------------------------------------------------------------------------
> *From:* Gelen James <hahaha_30k at yahoo.com>
> *To:* Rob Crittenden <rcritten at redhat.com>; Dmitri Pal <dpal at redhat.com>
> *Cc:* "Freeipa-users at redhat.com" <Freeipa-users at redhat.com>
> *Sent:* Sunday, May 20, 2012 12:08 AM
> *Subject:* Re: [Freeipa-users] Please help: How to restore IPA
> Master/Replicas from daily IPA Replica setup???
>
> Hi Mmitri, Rob and all.
>
>  Thanks for your instructions. I've performed your steps on case#1:
> replacing failed IPA master.  The results, and my confusion and
> questions, are all detailed below. In general, please setup your own
> real test environment, and write down the detailed steps one by one
> clearly.
>
>  It took me more than one week and still no clues. Frankly, your steps
> in the formal email are kind of over-simplified for normal IPA users,
> and not covering how the CA LDAP backend will be handled.
>
> The problem is the CA backend. All the replicas still trying to sync
> to old failed IPA master, even after reboot.  
>
> Could be that the 'ipa-replica-manage' only manages the user data
> replication? and 'ipa-csreplica-manage' only handles CA-end
> replication? In other words, when build, or tear down, IPA replication
> between two servers, do we need to deal with both replication types
> with 'ipa-replica-mange' AND 'ipa-csreplica-manage'? If so, then why
> who should run first?
>
> The error messages in /var/log/dirsrv/slapd-PKI-IPA/errors are
> attached, same from B,C,D replicas. 
>
> [19/May/2012:19:40:48 -0700] - 389-Directory/1.2.9.16 B2012.023.214
> starting up
> [19/May/2012:19:40:48 -0700] - slapd started.  Listening on All
> Interfaces port 7389 for LDAP requests
> [19/May/2012:19:40:48 -0700] - Listening on All Interfaces port 7390
> for LDAPS requests
> [19/May/2012:19:40:50 -0700] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server)
> [19/May/2012:19:40:50 -0700] NSMMReplicationPlugin -
> agmt="cn=cloneAgreement1-B.example.com-pki-ca" (<A>:7389): Replication
> bind with SIMPLE auth failed: LDAP error -1 (Can't contact LDAP
> server) ((null))
> [19/May/2012:19:40:57 -0700] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server)
> [19/May/2012:19:41:03 -0700] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server)
> [19/May/2012:19:41:15 -0700] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server)
> [19/May/2012:19:41:39 -0700] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server)
> [19/May/2012:19:42:27 -0700] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server)
> [19/May/2012:19:44:03 -0700] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server)
> [19/May/2012:19:47:15 -0700] slapi_ldap_bind - Error: could not send
> startTLS request: error -1 (Can't contact LDAP server)
> [root@<B> ~]#  
>
> After seeing the above messages, I tried to run similar commands for
> CA replication, it shows that replication agreement (which replication
> agreement? User data, or CA data ?? ) exists already.
>
> on B,
>  
> ipa-csreplica-manage connect C
> ipa-csreplica-manage connect D
> ipa-csreplica-manage del A --force
> ipactl restart 
>
> on C, 
> ipa-csreplica-manage del A --force
> ipactl restart 
>
> on D,
> ipa-csreplica-manage del A --force
> ipactl restart 
>
>
> [root at B ~]# ipa-csreplica-manage --password=xxxxxxx connect
> C.example.com <http://C.example.com>
> This replication agreement already exists.
> [root at B ~]# 
>
> [root at B ~]# ipa-csreplica-manage --password=xxxxxxx connect
> D.example.com <http://D.example.com>
> This replication agreement already exists.
> [root at B ~]# 
>
> [root at B ~]# ipa-csreplica-manage --password=xxxxxxx del C.example.com
> --force
> Unable to connect to replica A.example.com <http://A.example.com>,
> forcing removal
> Failed to get data from 'A.example.com': Can't contact LDAP server
> Forcing removal on 'B.example.com <http://B.example.com>'
> [root at B ~]# 
>
> ....
>
> After restarting IPA services on B, C, D, and now the error messages
> finally got away from CA errors log file. 
>
> But we still can not find the CA replication setups. Please see the
> difference of output from 'ipa-replica-manage' and 'ipa-csreplica-manage':
>
> [root at B ~] ipa-replica-manage list
> B.example.com
> C.example.com
> D.example.com
>
> [root at B ~] ipa-csreplica-manage list
> B.example.com
> C.example.com
> D.example.com
>
> [root at B ~] ipa-replica-manage list B.example.com
> C.example.com
> D.example.com
>
> [root at B ~] ipa-csreplica-manage list B.example.com
> ## Nothing at all!
>
> Please have a check and give correct command and sequences for us IPA
> users. It is such a pain to spend so much time and still can not get
> restoration work as expected.  Even worse is, Have no idea how the
> 'ipa-replica-manage' and 'ipa-csreplica-manage' work together behind
> the scene.
>
> Thanks a lot.
>
> --Gelen
>
>
>
>
> ------------------------------------------------------------------------
> *From:* Rob Crittenden <rcritten at redhat.com>
> *To:* Robinson Tiemuqinke <hahaha_30k at yahoo.com>
> *Cc:* "Freeipa-users at redhat.com" <Freeipa-users at redhat.com>; Rich
> Megginson <rmeggins at redhat.com>; Dmitri Pal <dpal at redhat.com>
> *Sent:* Tuesday, May 15, 2012 9:57 AM
> *Subject:* Re: [Freeipa-users] Please help: How to restore IPA
> Master/Replicas from daily IPA Replica setup???
>
> Robinson Tiemuqinke wrote:
> > Hi Dmitri, Rich and all,
> >
> > I am a newbie to Redhat IPA, It looks like pretty cool compared with
> > other solutions I've tried before. Thanks a lot for this great
> product! :)
> >
> > But there are still some things I needs your help. My main question is:
> > How to restore the IPA setup with a daily machine-level IPA Replica
> backup?
> >
> > Please let me explain my IPA setup background and backup/restore goals
> > trying to reach:
> >
> > I'm running IPA 2.1.3 on Redhat Enterprise 6.2. The IPA master is setup
> > with Dogtag CA system. It is installed first. Then two IPA replicas are
> > installed -- with '--setup-ca' options -- for load balancing and
> > failover purposes.
> >
> > To describe my problems/objectives, I'll name the IPA Master as machine
> > A, IPA replicas as B and C. and now I've one more extra IPA replica 'D'
> > (virtual machine) setup ONLY for backup purposes.
> > The setup looks like the following, A is the configuration Hub. B,C,D
> > are siblings.
> >
> > A
> > / | \
> > B C D
> >
> > The following are the steps I backup IPA setups and LDAP backends daily
> > -- it is a whole machine-level backup (through virtual machine D).
> >
> > 1, First, IPA replica D is backed up daily. The backup happens like
> this:
> >
> > 1.1 on IP replica D, run 'service IPA stop'. Then run 'shutdown -h <D>'.
> > On the Hypervisor which holds virtual machine D, do a daily backup of
> > the whole virtual disk that D is on.
> > 1.2 turn on the IP replica D again.
> > 1.3 after virtual machine D is up, on D optionally run a
> > 'ipa-replica-manage --force-sync --from <A>' to sync the IPA databases
> > forcibly.
> >
> > Now comes to restore part, which is pretty confusing to me. I've tried
> > several times, and every times it comes this or that kinds of issues and
> > so I am wondering that correct steps/ineraction of IPA Master/replicas
> > are the king :(
> >
> > 2, case #1, A is broken, like disc failure, and then re-imaged after
> > several days.
> >
> > 2.1 How to rebuild the IPA Master/Hub A after A is re-imaged, with the
> > daily backup from IPA replica D?
>
> The first thing you'll need to do is to connect your other replias
> together, either by picking a new hub or adding links to each one. Then
> you'll need to delete the replication agreement to A. You should be left
> with a set of servers that continues to replicate.
>
> So, for arguments sake, we promote B to be the new hub:
>
> On B:
>
> # ipa-replica-manage connect C
> # ipa-replica-manage connect D
> # ipa-replica-manage del --force A
> # ipactl restart
>
> On C:
>
> # ipa-replica-manage del --force A
> # ipactl restart
>
> On D:
>
> # ipa-replica-manage del --force A
> # ipactl restart
>
> It is unclear what you mean by re-imaged. Are you restoring from backup
> or installing it fresh? I'll assume it is a new install. You'll need to
> prepare a replica file for A and install it as a replica. Then if you
> want to keep A as the primary you'll need to change the replication
> agreements back to it is the hub (using ipa-replica-manage connect and
> disconnect).
>
> When you install the new A server it should get all the changes needed,
> you should be done.
>
> You'll want to check the documentation on promoting a master to verify
> that only one server is the CRL generator (at this point there may be
> none).
>
> > 2.2 do I have to check some files on A into subversion immediately after
> > A was initially installed?
>
> The only thing you really need to save is the cacert.p12 file. This is
> your root CA.
>
> > 2.3 Please describe the steps. I'll follow exactly and report the
> results.
> >
> > 3, case #2, A is working, but either B, or C is broken.
> >
> > 3.1 It looks that I don't need the daily backup of D to kick in, is that
> > right?
>
> No, D is unrelated.
>
> > 3.2 What are the correct steps on A; and B after it is re-imaged?
>
> On A:
> # ipa-replica-manage del B
> # ipactl restart
> # ipa-replica-prepare B
>
> On B
> # ipa-replica-install B
>
> You'll probably need/want to clean RUV,
> http://directory.fedoraproject.org/wiki/Howto:CLEANRUV
>
> > 3.3 Please describe the steps. I'll follow exactly and report the
> results.
> >
> > 4, case #3, If some un-expected IPA changes happens on A -- like all
> > users are deleted by human mistakes --, and even worse, all the changes
> > are propagated to B and C in minutes.
> >
> > 4.1 How can I recover the IPA setup from daily backup from D?
>
> We have not yet documented how to recover from tombstones or an offline
> replica.
>
> > 4.2 which IPA master/replicas I should recover first? IPA master A, or
> > IPA replicas B/C? and then how to recover others left one by one?
>
> If the entries are re-added on any of the replicas it will be propogated
> out.
>
> > 4.3 Do I have to disconnect replication agreement of B,C,D from A first?
>
> Depends on how 4.1 gets answered which we are still investigating.
>
> > 4.4 Please describe the steps. I'll follow exactly and report the
> results.
> >
> > I've heard something about tombstone records too, Not sure whether the
> > problem still exists in 2.1.3, or 2.2.0(on 6.3Beta)? If so, How can I
> > avoid it with correct recovery steps/interactions.
>
> It is RUV that is the problem. This 389-ds wiki page describes how to
> clean up: http://directory.fedoraproject.org/wiki/Howto:CLEANRUV
>
> The 389-ds team is working to make this less manual.
>
> rob
>
>
>
> _______________________________________________
> Freeipa-users mailing list
> Freeipa-users at redhat.com <mailto:Freeipa-users at redhat.com>
> https://www.redhat.com/mailman/listinfo/freeipa-users
>

-- 
Thank you,
Dmitri Pal

Sr. Engineering Manager IPA project,
Red Hat Inc.

-------------------------------
Looking to carve out IT costs?
www.redhat.com/carveoutcosts/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/freeipa-users/attachments/20120521/471d71fa/attachment.htm>