[Freeipa-devel] Backup and Restore design

[Rich Megginson]

On 02/20/2013 08:38 AM, Rob Crittenden wrote:
> Simo Sorce wrote:
>> On Tue, 2013-02-19 at 22:43 -0500, Rob Crittenden wrote:
>>> I've looked into some basic backup and restore procedures for IPA. My
>>> findings are here: http://freeipa.org/page/V3/Backup_and_Restore
>>
>> Great summary!
>>
>> For the catastrofic failure scenario, should we mention how to put back
>> a full failed and restored machine online ? I am thinking the restored
>> server may be behind (even if only by a few entries) in the replication,
>> so the CSNs in the other replicas will not match.
>> I guess we should mention a full resync may be needed ? Or is there a
>> way to bring back CSNs on replicas ?
>
> Good questions. It depends on how long the machine was down and how 
> many changes have happened. It is possible that one would want to do a 
> full re-init. I'll add that to the design.

The replication protocol will detect if a replica is too out of date to 
bring up to date with an incremental update, and requires a re-init.

>
>> In the 'Returning to a good state' case, can we consider some split
>> brain approach, were we sever replication and rebuild one server at a
>> time ?
>
> Perhaps using a firewall, but then you run the risk of each of those 
> servers accepting changes during the rebuild and you could end up with 
> a lot of collisions, which sort of goes against the point of restoring 
> to a known good state.
>
> The changelog is the key here. I'll have to ponder this one a bit, I'm 
> a bit conflicted on the right approach.
>
>> Maybe we can think of a way to 'mark' all server as 'bad' so that on
>> restore the replication agreements do not need to be changed but changes
>> from 'bad' servers will not be accepted ?
>> I guess this crosses also a request by someone to be able to 'pause'
>> replication, would use the same mechanism I guess.
>
> AFAIK there is an option to pause replication now (at least in 1.3).
>
> What you can't do is drop the changelog AFAIK. That is the real 
> problem. If you want to restore to a known state you need to drop all 
> the changelog entries since that time. I'll check with the 389-ds team 
> to see if that is possible. Since we know the time of the backup, we 
> might be able to drop newer entries than that.

Not sure what you mean - what exactly do you want to do with the changelog?

>
>> Full system backup:
>> in the first part it is said the process is offline, but in the 'LDAP'
>> section you say ldapi is used, but that would mean DS is running ?
>> Also are we sure we can get all data we need via ldapi ? Are we going to
>> miss any "operational" attribute ?
>
> The full backup is offline because it is just using tar. This is sort 
> of a brute-force backup, copying bits from A to B.
>
> The data backup is online and creates a task in 389-ds to write the 
> data and changelog to a file. It should write everything completely. 
> We don't do an ldapsearch.
>
> I chose to not back up in ldif because this would back up just the 
> data and not the changelog. The other advantage is that the db2bak 
> format includes the indexes and ldif restore would require a rebuild.

It is good to have a long term backup in LDIF format - no matter what 
happens to the database, if you have an LDIF backup, you can always 
recreate your data.  So it's good to have both - db2bak format for 
shorter term/frequent backups, and LDIF for longer term/infrequent backups.

>
>> For restore are we sure we can reload data w/o alterations ? What about
>> plugins ? will we have to disable all plugins during a restore ?
>
> Yes, it should be fine. I'm hoping that the version will help us with 
> this, to prevent someone from restoring an ancient backup on a new 
> system, for example (or the reverse).
>
>> For the open questions.
>>
>> Size of backup:
>> I think we should make it easy to configure the system to use a custom
>> directory to dump the backups. This way admins can make sure it is on a
>> different partition/disk or even over NFS and that the backup will not
>> fill up the disk on which DS is running.
>
> That's a good idea. I'll have to think about where we would configure 
> that. Perhaps as an optional argument to the backup command.
You'll have to figure out a way around selinux, or add some sort of 
selinux magic that allows db2bak to write there.
>
>> We should definitely allow to encrypt backup files, a gpg public key
>> would be sufficient.
>
> Ok. I wasn't sure if there would be any corruption concerns.
>
>> For replica cases, maybe we can create a command that dumps the
>> changelog from a good replica and then allow us to reply all changes
>> that are missing from the backup to bring the server up to the last
>> minute ?
>
> This would happen when we went online anyway though, at least for the 
> entries currently in the changelog. I guess this would have the 
> advantage of doing it in bulk and not over a (potentially) slow link.

That should happen automatically with the replication protocol - it will 
attempt to bring older replicas up-to-date or, if they are too far out 
of date, will complain that they need a re-init.

>
> rob
>
> _______________________________________________
> Freeipa-devel mailing list
> Freeipa-devel at redhat.com
> https://www.redhat.com/mailman/listinfo/freeipa-devel