[libvirt] [RFC v2] external (pull) backup API

[Nikolay Shirokovskiy]

On 13.04.2018 01:10, John Snow wrote:
> 
> On 04/12/2018 04:58 AM, Nikolay Shirokovskiy wrote:
>> On 11.04.2018 19:32, Eric Blake wrote:
>>> On 04/03/2018 07:01 AM, Nikolay Shirokovskiy wrote:
> 
> 
> [snip]
> 
> 
>>>
>>> I'm trying to figure out how BlockCheckpoint and BlockSnapshots relate.
>>> Maybe it will be more clear when I read the implementation section
>>> below.  Is the idea that I can't create a BlockSnapshot without first
>>> having a checkpoint available?  If so, where does that fit in the
>>> <domainblocksnapshot> XML?
>>
>> No, you can create snapshot without available checkpoints. Actually the first snapshot
>> is like that.
>>
>> Now if you create a snapshot with checkpoint and then delete the snapshot
>> the checkpoint remains, so we need an API to delete them if we wish.
>>
> 
> Hmm - OK, you are being careful to label three notions separately:
> 
> (1) Checkpoints (which are metadata objects in libvirt supported by
> bitmaps in QEMU, roughly)
> (2) BlockSnapshots (which expose data using checkpoints as metadata)
> (3) Backups (which are made by a 3rd party client using a snapshot)
> 
> In this case, though, if a snapshot is requested it probably ought to be
> *prepared* to create a checkpoint in case that snapshot is used by the
> third party client to make a backup, right?

Block snapshots can be used without checkpoint altogether. Imagine we always
make full backups. It looks like next. Create block snapshot without checkpoint,
export it, make a backup, delete from export and delete snapshot. No any
checkpoints appear. On next full backup steps are the same.

> 
> IOW, a snapshot -- though ignorant of how it is used -- can be and often
> will be used to accomplish an incremental backup and as such must be
> prepared to manipulate the checkpoints/bitmaps/etc in such a way to be
> able to make a new checkpoint.
> 
> Right?

I don't unrestand what these preparations are. If we want to make incremental
backups we need to create every snapshot with checkpoint so that later we
can export CBT between current snapshot and snapshot in past.

If we create snapshot with checkpoint then internally it means we create bitmap
which starts at this point in time. Later we can use this bitmap (and
probably other bitmaps corresponding to checkpoints in the middle) to calculate
the CBT we interested in.

> 
> 
> [snip]
> 
> 
>>> Earlier, you said that the new virDomainBlockSnapshotPtr are
>>> independent, with no relations between them.  But here, you are wanting
>>> to keep incremental backups related to one another.
>>
>> Yes, but backups are not snapshots. All backup relation management are on
>> client. In pull backup scheme libvirt is only here to export a snapshotted
>> disk state with optionally a CBT from some point in time. Client itself
>> makes backups and track their relationships.
>>
>> However as we use chain of disabled bitmaps with one active bitmap on tip
>> of the chain and qemu does not track their order we need to do it in
>> libvirt.
>>
> 
> Well, you seem to be tracking it in *qemu*, by using the name field.
> Should we not make a commitment to whether or not we store this lineage
> information in either qemu OR libvirt, but not distributed across both...?
> 

I don't know actual use cases to decide. A commitment that this meta is stored
in disks like proposed can be useful IMHO so that mgmt can expect that dumb reinserting
disks to a different domain (recreated for example) keep all checkpoints.