[Freeipa-devel] Backup and Restore design

[Rob Crittenden]

Simo Sorce wrote:
> On Tue, 2013-02-19 at 22:43 -0500, Rob Crittenden wrote:
>> I've looked into some basic backup and restore procedures for IPA. My
>> findings are here: http://freeipa.org/page/V3/Backup_and_Restore
>
> Great summary!
>
> For the catastrofic failure scenario, should we mention how to put back
> a full failed and restored machine online ? I am thinking the restored
> server may be behind (even if only by a few entries) in the replication,
> so the CSNs in the other replicas will not match.
> I guess we should mention a full resync may be needed ? Or is there a
> way to bring back CSNs on replicas ?

Good questions. It depends on how long the machine was down and how many 
changes have happened. It is possible that one would want to do a full 
re-init. I'll add that to the design.

> In the 'Returning to a good state' case, can we consider some split
> brain approach, were we sever replication and rebuild one server at a
> time ?

Perhaps using a firewall, but then you run the risk of each of those 
servers accepting changes during the rebuild and you could end up with a 
lot of collisions, which sort of goes against the point of restoring to 
a known good state.

The changelog is the key here. I'll have to ponder this one a bit, I'm a 
bit conflicted on the right approach.

> Maybe we can think of a way to 'mark' all server as 'bad' so that on
> restore the replication agreements do not need to be changed but changes
> from 'bad' servers will not be accepted ?
> I guess this crosses also a request by someone to be able to 'pause'
> replication, would use the same mechanism I guess.

AFAIK there is an option to pause replication now (at least in 1.3).

What you can't do is drop the changelog AFAIK. That is the real problem. 
If you want to restore to a known state you need to drop all the 
changelog entries since that time. I'll check with the 389-ds team to 
see if that is possible. Since we know the time of the backup, we might 
be able to drop newer entries than that.

> Full system backup:
> in the first part it is said the process is offline, but in the 'LDAP'
> section you say ldapi is used, but that would mean DS is running ?
> Also are we sure we can get all data we need via ldapi ? Are we going to
> miss any "operational" attribute ?

The full backup is offline because it is just using tar. This is sort of 
a brute-force backup, copying bits from A to B.

The data backup is online and creates a task in 389-ds to write the data 
and changelog to a file. It should write everything completely. We don't 
do an ldapsearch.

I chose to not back up in ldif because this would back up just the data 
and not the changelog. The other advantage is that the db2bak format 
includes the indexes and ldif restore would require a rebuild.

> For restore are we sure we can reload data w/o alterations ? What about
> plugins ? will we have to disable all plugins during a restore ?

Yes, it should be fine. I'm hoping that the version will help us with 
this, to prevent someone from restoring an ancient backup on a new 
system, for example (or the reverse).

> For the open questions.
>
> Size of backup:
> I think we should make it easy to configure the system to use a custom
> directory to dump the backups. This way admins can make sure it is on a
> different partition/disk or even over NFS and that the backup will not
> fill up the disk on which DS is running.

That's a good idea. I'll have to think about where we would configure 
that. Perhaps as an optional argument to the backup command.

> We should definitely allow to encrypt backup files, a gpg public key
> would be sufficient.

Ok. I wasn't sure if there would be any corruption concerns.

> For replica cases, maybe we can create a command that dumps the
> changelog from a good replica and then allow us to reply all changes
> that are missing from the backup to bring the server up to the last
> minute ?

This would happen when we went online anyway though, at least for the 
entries currently in the changelog. I guess this would have the 
advantage of doing it in bulk and not over a (potentially) slow link.

rob