[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

[libvirt] [PATCH v3 00/20] Incremental Backup API additions



The following is the latest version of my API proposal for
incremental backups, and matches the demo I will be presenting
as part of my KVM Forum 2018 talk tomorrow afternoon:
https://kvmforum2018.sched.com/event/FzuB/facilitating-incremental-backup-eric-blake-red-hat

The patches are also available via https://repo.or.cz/libvirt/ericb.git
at the tag backup-v3.

The qemu integration is still incomplete (I don't have push
mode backups working, only pull mode; and there are still some
things to fine-tune as I figure out how to better support
external snapshots and block commit/stream), and I still haven't
gotten any opinions on whether it is better to have:

virDomainSnapshotCreateXML("<domainsnapshot>...") # existing
virDomainBackupBegin("<domainbackup>...", "<domaincheckpoint>...") # this series
virDomainSnapshotCheckpointCreateXML("<domainsnapshot>...", "<domaincheckpoint>...") # new

or to make checkpoint creation part of the snapshot and backup XML, as in:

virDomainSnapshotCreateXML("<domainsnapshot><domaincheckpoint>...</domainsnapshot>") # extension of existing
virDomainBackupBegin("<domainbackup><domaincheckpoint>...</domainbackup>") # tweak to this series

It's also obvious that we don't want to check in the qemu implementation
until after Peter's blockdev work has landed AND qemu has gotten rid
of the x- prefix in its QMP interactions, although the state of this
series SHOULD be sufficient to prove that the proposed interface is
at least flexible enough to adapt to whatever qemu tweaks land and
whatever other hypervisors may support. Whether to accept the API now
(to make backporting qemu support downstream across versions without
rebasing to an even later date for the API) is a harder question;
Jirka has already expressed distaste at the idea.

There are probably still loads of things needing fixing in this version,
I already know I did not address all of Peter's review comments on v2.
My unit test coverage is not very complete on the XML parsing, and there
are definitely some hacks in place to get things to the point of a working
demo that need to be polished for robustness.

Below is a diff since the v2 posting (patches 1-10 here, for later
patches a comparison against the state of my tree at the time I posted v2):
https://www.redhat.com/archives/libvir-list/2018-October/msg00700.html

001/20:[----] [--] 'snapshots: Avoid term 'checkpoint' for full system snapshot'
002/20:[----] [--] 'domain_conf: Expose virDomainStorageNetworkParseHost'
003/20:[down] 'qemu: Allow optional export name during NBD export'
004/20:[----] [--] 'backup: Document nuances between different state capture APIs'
005/20:[----] [--] 'backup: Introduce virDomainCheckpointPtr'
006/20:[0012] [FC] 'backup: Document new XML for backups'
007/20:[----] [--] 'backup: Introduce virDomainCheckpoint APIs'
008/20:[0024] [FC] 'backup: Introduce virDomainBackup APIs'
009/20:[----] [--] 'backup: Add new domain:checkpoint access control'
010/20:[0001] [FC] 'backup: Implement backup APIs for remote driver'
011/20:[0007] [FC] 'wip: backup: Parse and output checkpoint XML'
012/20:[0199] [FC] 'wip: backup: Parse and output backup XML'
013/20:[----] [--] 'backup: Implement virsh support for checkpoints'
014/20:[----] [--] 'wip: backup: virsh support for backup'
015/20:[0223] [FC] 'backup: Add new qemu monitor interactions'
016/20:[0123] [FC] 'backup: qemu: Implement metadata tracking for checkpoint APIs'
017/20:[down] 'wip: backup: Wire up qemu checkpoint commands over QMP'
018/20:[0072] [FC] 'wip: backup: qemu: Implement framework for backup job APIs'
019/20:[down] 'backup: Wire up qemu full pull backup commands over QMP'
020/20:[down] 'backup: implement qemu incremental pull backup'

No direct dependency on this one, but included in my git tag:
https://www.redhat.com/archives/libvir-list/2018-October/msg00899.html

Eric Blake (20):
  snapshots: Avoid term 'checkpoint' for full system snapshot
  domain_conf: Expose virDomainStorageNetworkParseHost
  qemu: Allow optional export name during NBD export
  backup: Document nuances between different state capture APIs
  backup: Introduce virDomainCheckpointPtr
  backup: Document new XML for backups
  backup: Introduce virDomainCheckpoint APIs
  backup: Introduce virDomainBackup APIs
  backup: Add new domain:checkpoint access control
  backup: Implement backup APIs for remote driver
  wip: backup: Parse and output checkpoint XML
  wip: backup: Parse and output backup XML
  backup: Implement virsh support for checkpoints
  wip: backup: virsh support for backup
  backup: Add new qemu monitor interactions
  backup: qemu: Implement metadata tracking for checkpoint APIs
  wip: backup: Wire up qemu checkpoint commands over QMP
  wip: backup: qemu: Implement framework for backup job APIs
  backup: Wire up qemu full pull backup commands over QMP
  backup: implement qemu incremental pull backup

 include/libvirt/virterror.h                  |    6 +-
 src/util/virerror.c                          |   21 +-
 include/libvirt/libvirt-domain-checkpoint.h  |  177 ++
 include/libvirt/libvirt-domain-snapshot.h    |    2 +-
 include/libvirt/libvirt-domain.h             |   17 +-
 include/libvirt/libvirt.h                    |    3 +-
 src/access/viraccessperm.h                   |    8 +-
 src/conf/checkpoint_conf.h                   |  218 +++
 src/conf/domain_conf.h                       |   16 +-
 src/datatypes.h                              |   31 +-
 src/driver-hypervisor.h                      |   74 +-
 src/qemu/qemu_conf.h                         |    2 +
 src/qemu/qemu_domain.h                       |   29 +-
 src/qemu/qemu_monitor.h                      |   17 +
 src/qemu/qemu_monitor_json.h                 |   19 +
 tools/virsh-checkpoint.h                     |   29 +
 tools/virsh-completer.h                      |    4 +
 tools/virsh-util.h                           |    3 +
 tools/virsh.h                                |    1 +
 docs/Makefile.am                             |    3 +
 docs/apibuild.py                             |    2 +
 docs/docs.html.in                            |    9 +-
 docs/domainstatecapture.html.in              |  314 ++++
 docs/format.html.in                          |    1 +
 docs/formatcheckpoint.html.in                |  291 ++++
 docs/formatsnapshot.html.in                  |   33 +-
 docs/index.html.in                           |    3 +-
 docs/schemas/domainbackup.rng                |  185 ++
 docs/schemas/domaincheckpoint.rng            |   94 ++
 examples/object-events/event-test.c          |    3 +
 libvirt.spec.in                              |    3 +
 mingw-libvirt.spec.in                        |    6 +
 po/POTFILES                                  |    3 +
 src/Makefile.am                              |    2 +
 src/access/viraccessperm.c                   |    5 +-
 src/conf/Makefile.inc.am                     |    2 +
 src/conf/checkpoint_conf.c                   | 1541 +++++++++++++++++
 src/conf/domain_conf.c                       |   52 +-
 src/conf/snapshot_conf.c                     |    4 +-
 src/datatypes.c                              |   62 +-
 src/libvirt-domain-checkpoint.c              |  925 ++++++++++
 src/libvirt-domain-snapshot.c                |    7 +-
 src/libvirt-domain.c                         |    8 +-
 src/libvirt_private.syms                     |   30 +
 src/libvirt_public.syms                      |   23 +
 src/qemu/qemu_conf.c                         |    5 +
 src/qemu/qemu_domain.c                       |  205 ++-
 src/qemu/qemu_driver.c                       | 1494 ++++++++++++++++-
 src/qemu/qemu_migration.c                    |    2 +-
 src/qemu/qemu_monitor.c                      |   73 +-
 src/qemu/qemu_monitor_json.c                 |  217 +++
 src/qemu/qemu_process.c                      |    7 +
 src/remote/remote_daemon_dispatch.c          |   15 +
 src/remote/remote_driver.c                   |   33 +-
 src/remote/remote_protocol.x                 |  239 ++-
 src/remote_protocol-structs                  |  129 ++
 src/rpc/gendispatch.pl                       |   32 +-
 tests/Makefile.am                            |   15 +-
 tests/domainbackupxml2xmlin/backup-pull.xml  |    9 +
 tests/domainbackupxml2xmlin/backup-push.xml  |    9 +
 tests/domainbackupxml2xmlin/empty.xml        |    1 +
 tests/domainbackupxml2xmlout/backup-pull.xml |    9 +
 tests/domainbackupxml2xmlout/backup-push.xml |    9 +
 tests/domainbackupxml2xmlout/empty.xml       |    7 +
 tests/domaincheckpointxml2xmlin/empty.xml    |    1 +
 tests/domaincheckpointxml2xmlin/sample.xml   |    7 +
 tests/domaincheckpointxml2xmlout/empty.xml   |   10 +
 tests/domaincheckpointxml2xmlout/sample.xml  |   16 +
 tests/domaincheckpointxml2xmltest.c          |  231 +++
 tests/qemumonitorjsontest.c                  |    2 +-
 tests/virschematest.c                        |    4 +
 tools/Makefile.am                            |    3 +-
 tools/virsh-checkpoint.c                     | 1578 ++++++++++++++++++
 tools/virsh-completer.c                      |   52 +-
 tools/virsh-domain.c                         |    6 +-
 tools/virsh-snapshot.c                       |    2 +-
 tools/virsh-util.c                           |   11 +
 tools/virsh.c                                |    2 +
 tools/virsh.pod                              |   14 +-
 79 files changed, 8619 insertions(+), 118 deletions(-)
 create mode 100644 include/libvirt/libvirt-domain-checkpoint.h
 create mode 100644 src/conf/checkpoint_conf.h
 create mode 100644 tools/virsh-checkpoint.h
 create mode 100644 docs/domainstatecapture.html.in
 create mode 100644 docs/formatcheckpoint.html.in
 create mode 100644 docs/schemas/domainbackup.rng
 create mode 100644 docs/schemas/domaincheckpoint.rng
 create mode 100644 src/conf/checkpoint_conf.c
 create mode 100644 src/libvirt-domain-checkpoint.c
 create mode 100644 tests/domainbackupxml2xmlin/backup-pull.xml
 create mode 100644 tests/domainbackupxml2xmlin/backup-push.xml
 create mode 100644 tests/domainbackupxml2xmlin/empty.xml
 create mode 100644 tests/domainbackupxml2xmlout/backup-pull.xml
 create mode 100644 tests/domainbackupxml2xmlout/backup-push.xml
 create mode 100644 tests/domainbackupxml2xmlout/empty.xml
 create mode 100644 tests/domaincheckpointxml2xmlin/empty.xml
 create mode 100644 tests/domaincheckpointxml2xmlin/sample.xml
 create mode 100644 tests/domaincheckpointxml2xmlout/empty.xml
 create mode 100644 tests/domaincheckpointxml2xmlout/sample.xml
 create mode 100644 tests/domaincheckpointxml2xmltest.c
 create mode 100644 tools/virsh-checkpoint.c

-- 
2.17.2


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]