[libvirt] [PATCH] Support a new peer-to-peer migration mode & public API

Daniel Veillard veillard at redhat.com
Tue Sep 22 12:43:20 UTC 2009


[ Sending again due to mail output problems ]

On Thu, Sep 17, 2009 at 06:25:01PM +0100, Daniel P. Berrange wrote:
> Introduces several new public API options for migration
> 
>  - VIR_MIGRATE_PEER2PEER: With this flag the client only
>    invokes the virDomainMigratePerform method, expecting
>    the source host driver to do whatever is required to
>    complete the entire migration process.
>  - VIR_MIGRATE_TUNNELLED: With this flag the actual data
>    for migration will be tunnelled over the libvirtd RPC
>    channel. This requires that VIR_MIGRATE_PEER2PEER is
>    also set.

  Hum, I would rather add that when entering libvirt.c entry point
set explicitely VIR_MIGRATE_PEER2PEER if VIR_MIGRATE_TUNNELLED is asked
for, that's an internal impl. detail.

>  - virDomainMigrateToURI: This is variant of the existing
>    virDomainMigrate method which can be used when the
>    VIR_MIGRATE_PEER2PEER flag is set. The benefit of this
>    method is that no virConnectPtr object is required for
>    the destination host, only a destination URI.

  Sounds good if we have proper error handling. I'm also wondering
how authentification and access control can be handled in that mode,
the normal method of access somehow garantee that the user credentials
can be checked in some ways, but if it's direct daemon to daemon
talking how can this be performed ?

> The URI for VIR_MIGRATE_TUNNELLED must be a valid libvirt
> URI. For non-tunnelled migration a hypervisor specific
> migration URI is used.
[...]
> diff --git a/include/libvirt/libvirt.h b/include/libvirt/libvirt.h
> index 10ec04a..1745396 100644
> --- a/include/libvirt/libvirt.h
> +++ b/include/libvirt/libvirt.h
> @@ -336,7 +336,9 @@ typedef virDomainInterfaceStatsStruct *virDomainInterfaceStatsPtr;
>  
>  /* Domain migration flags. */
>  typedef enum {
> -  VIR_MIGRATE_LIVE              = 1, /* live migration */
> +    VIR_MIGRATE_LIVE              = (1 << 0), /* live migration */
> +    VIR_MIGRATE_PEER2PEER         = (1 << 1), /* direct source -> dest host control channel */
> +    VIR_MIGRATE_TUNNELLED         = (1 << 2), /* tunnel migration data over libvirtd connection */
>  } virDomainMigrateFlags;
>  
>  /* Domain migration. */
> @@ -344,6 +346,10 @@ virDomainPtr virDomainMigrate (virDomainPtr domain, virConnectPtr dconn,
>                                 unsigned long flags, const char *dname,
>                                 const char *uri, unsigned long bandwidth);
>  
> +int virDomainMigrateToURI (virDomainPtr domain, const char *duri,
> +                           unsigned long flags, const char *dname,
> +                           unsigned long bandwidth);
> +

  s/duri/dest/ or target, the fact it's a connection URI should be made
  obvious in the function comment.

[...]
> +    } else {
> +        if (flags & VIR_MIGRATE_TUNNELLED) {
> +            virLibConnError(domain->conn, VIR_ERR_OPERATION_INVALID,
> +                            _("cannot perform tunnelled migration without using peer2peer flag"));
> +            goto error;
> +        }

  I don't really agree, we should just or the flag and hide this from
  the API user.

> +
> +/**
> + * virDomainMigrateToURI:
> + * @domain: a domain object
> + * @duri: libvirt connection URI for destination host
> + * @flags: flags
> + * @dname: (optional) rename domain to this at destination
> + * @bandwidth: (optional) specify migration bandwidth limit in Mbps
> + *
> + * Migrate the domain object from its current host to the destination
> + * host given by duri. The duri is a libvirt connection URI that the
> + * source host will use to talk to the destination.
> + *
> + * The difference from the regular virDomainMigrate method is
> + * that the calling application does not need a direct connection
> + * to the destination host. The source libvirt driver makes a
> + * direct peer-to-peer connection to the destination libvirtd
> + * without the client appliction being involved.
> + *
> + * The VIR_MIGRATE_PEER2PEER flag is mandatory for this method.
> + * If an application wishes to run without this flag, then it
> + * may use the alternative virDomainMigrate method which requires
> + * an virConnectPtr object for the destination host

  looks good except that block. Why force a flag systematically ?

> + * The following additional flags may also be set
> + *   VIR_MIGRATE_LIVE      Do not pause the VM during migration
> + *   VIR_MIGRATE_TUNNELLED Tunnel migration data over the libvirt RPC channel
> + *
> + * If a hypervisor supports renaming domains during migration,
> + * then you may set the dname parameter to the new name (otherwise
> + * it keeps the same name).  If this is not supported by the
> + * hypervisor, dname must be NULL or else you will get an error.
> + *
> + * The maximum bandwidth (in Mbps) that will be used to do migration
> + * can be specified with the bandwidth parameter.  If set to 0,
> + * libvirt will choose a suitable default.  Some hypervisors do
> + * not support this feature and will return an error if bandwidth
> + * is not 0.
> + *
> + * To see which features are supported by the current hypervisor,
> + * see virConnectGetCapabilities, /capabilities/host/migration_features.
> + *
> + * There are many limitations on migration imposed by the underlying
> + * technology - for example it may not be possible to migrate between
> + * different processors even with the same architecture, or between
> + * different types of hypervisor.
> + *
> + * Returns 0 if the migration succeeded, -1 upon error.
> + */
> +int
> +virDomainMigrateToURI (virDomainPtr domain,
> +                       const char *duri,
> +                       unsigned long flags,
> +                       const char *dname,
> +                       unsigned long bandwidth)
> +{
> +    DEBUG("domain=%p, duri=%p, flags=%lu, dname=%s, bandwidth=%lu",
> +          domain, NULLSTR(duri), flags, NULLSTR(dname), bandwidth);
> +
> +    virResetLastError();
> +
> +    /* First checkout the source */
> +    if (!VIR_IS_CONNECTED_DOMAIN (domain)) {
> +        virLibDomainError(NULL, VIR_ERR_INVALID_DOMAIN, __FUNCTION__);
> +        return -1;
> +    }
> +    if (domain->conn->flags & VIR_CONNECT_RO) {
> +        virLibDomainError(domain, VIR_ERR_OPERATION_DENIED, __FUNCTION__);
> +        goto error;
> +    }
> +
> +    if (duri == NULL) {
> +        virLibConnError (domain->conn, VIR_ERR_INVALID_ARG, __FUNCTION__);
> +        goto error;
> +    }
> +
> +    /* XXX perhaps we should just automatically set this flag instead ? */

  Ah , yes :-)

> +    if (flags & VIR_MIGRATE_PEER2PEER) {
> +        if (VIR_DRV_SUPPORTS_FEATURE (domain->conn->driver, domain->conn,
> +                                      VIR_DRV_FEATURE_MIGRATION_P2P)) {
> +            if (virDomainMigrateP2P (domain, flags, dname, duri, bandwidth) < 0)
> +                goto error;
> +        } else {
> +            virLibConnError (domain->conn, VIR_ERR_NO_SUPPORT, __FUNCTION__);
> +            goto error;
> +        }
> +    } else {
> +        virLibConnError (domain->conn, VIR_ERR_OPERATION_INVALID,
> +                         _("cannot migrate to a destination URI without peer2peer flag"));
> +        goto error;
> +    }
> +
> +    return 0;
> +
> +error:
> +    /* Copy to connection error object for back compatability */
> +    virSetConnError(domain->conn);
> +    return -1;
> +}
> +
> +
>  /*
>   * Not for public use.  This function is part of the internal
>   * implementation of migration in the remote case.
> diff --git a/src/libvirt_internal.h b/src/libvirt_internal.h
> index 5913798..5f1a7fe 100644
> --- a/src/libvirt_internal.h
> +++ b/src/libvirt_internal.h
> @@ -17,6 +17,7 @@
>   * License along with this library; if not, write to the Free Software
>   * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307  USA
>   *
> + * NB This file is (secret) ABI sensitive. Append only

   Huh ???

>   */
>  
>  #ifndef __LIBVIRT_H_
> @@ -31,6 +32,35 @@ int virStateReload(void);
>  int virStateActive(void);
>  #endif
>  
> +/* Feature detection.  This is a libvirt-private interface for determining
> + * what features are supported by the driver.
> + *
> + * The remote driver passes features through to the real driver at the
> + * remote end unmodified, except if you query a VIR_DRV_FEATURE_REMOTE*
> + * feature.
> + *
> + */
> +enum {
> +    /* Driver supports V1-style virDomainMigrate, ie. domainMigratePrepare/
> +     * domainMigratePerform/domainMigrateFinish.
> +     */
> +    VIR_DRV_FEATURE_MIGRATION_V1 = 1,
> +
> +    /* Driver is not local. */
> +    VIR_DRV_FEATURE_REMOTE = 2,
> +
> +    /* Driver supports V2-style virDomainMigrate, ie. domainMigratePrepare2/
> +     * domainMigratePerform/domainMigrateFinish2.
> +     */
> +    VIR_DRV_FEATURE_MIGRATION_V2 = 3,
> +
> +    /* Driver supports peer-2-peer virDomainMigrate ie soruce host
> +     * do all the prepare/perform/finish steps directly
> +     */
> +    VIR_DRV_FEATURE_MIGRATION_P2P = 4,
> +};
> +
> +
>  int virDrvSupportsFeature (virConnectPtr conn, int feature);
>  
>  int virDomainMigratePrepare (virConnectPtr dconn,
> diff --git a/src/libvirt_public.syms b/src/libvirt_public.syms
> index 888ea26..757e54c 100644
> --- a/src/libvirt_public.syms
> +++ b/src/libvirt_public.syms
> @@ -323,6 +323,7 @@ LIBVIRT_0.7.1 {
>  	virStreamFinish;
>  	virStreamAbort;
>  	virStreamFree;
> +	virDomainMigrateToURI;
>  } LIBVIRT_0.7.0;
>  
>  # .... define new API here using predicted next version number ....
> diff --git a/src/virsh.c b/src/virsh.c
> index 4825f1c..cabbd3d 100644
> --- a/src/virsh.c
> +++ b/src/virsh.c
> @@ -2463,6 +2463,8 @@ static const vshCmdInfo info_migrate[] = {
>  
>  static const vshCmdOptDef opts_migrate[] = {
>      {"live", VSH_OT_BOOL, 0, gettext_noop("live migration")},
> +    {"p2p", VSH_OT_BOOL, 0, gettext_noop("peer-2-peer migration")},
> +    {"tunnelled", VSH_OT_BOOL, 0, gettext_noop("tunnelled migration")},
>      {"domain", VSH_OT_DATA, VSH_OFLAG_REQ, gettext_noop("domain name, id or uuid")},
>      {"desturi", VSH_OT_DATA, VSH_OFLAG_REQ, gettext_noop("connection URI of the destination host")},
>      {"migrateuri", VSH_OT_DATA, 0, gettext_noop("migration URI, usually can be omitted")},
> @@ -2478,8 +2480,6 @@ cmdMigrate (vshControl *ctl, const vshCmd *cmd)
>      const char *migrateuri;
>      const char *dname;
>      int flags = 0, found, ret = FALSE;
> -    virConnectPtr dconn = NULL;
> -    virDomainPtr ddom = NULL;
>  
>      if (!vshConnectionUsability (ctl, ctl->conn, TRUE))
>          return FALSE;
> @@ -2500,20 +2500,40 @@ cmdMigrate (vshControl *ctl, const vshCmd *cmd)
>      if (vshCommandOptBool (cmd, "live"))
>          flags |= VIR_MIGRATE_LIVE;
>  
> -    /* Temporarily connect to the destination host. */
> -    dconn = virConnectOpenAuth (desturi, virConnectAuthPtrDefault, 0);
> -    if (!dconn) goto done;
> +    if (vshCommandOptBool (cmd, "p2p"))
> +        flags |= VIR_MIGRATE_PEER2PEER;
>  
> -    /* Migrate. */
> -    ddom = virDomainMigrate (dom, dconn, flags, dname, migrateuri, 0);
> -    if (!ddom) goto done;
> +    if (vshCommandOptBool (cmd, "tunnelled"))
> +        flags |= VIR_MIGRATE_TUNNELLED;
>  
> -    ret = TRUE;
> +    if ((flags & VIR_MIGRATE_PEER2PEER)) {
> +        /* For peer2peer migration we only expect one URI, a libvirt URI */
> +
> +        if (migrateuri != NULL) {
> +            vshError(ctl, FALSE, "%s", _("migrate: Unexpected migrateuri for peer2peer migration"));
> +            goto done;
> +        }
> +
> +        if (virDomainMigrateToURI (dom, desturi, flags, dname, 0) == 0)
> +            ret = TRUE;
> +    } else {
> +        /* For regular live migration, connect to the destination host directly. */
> +        virConnectPtr dconn = NULL;
> +        virDomainPtr ddom = NULL;
> +
> +        dconn = virConnectOpenAuth (desturi, virConnectAuthPtrDefault, 0);
> +        if (!dconn) goto done;
> +
> +        ddom = virDomainMigrate (dom, dconn, flags, dname, migrateuri, 0);
> +        if (ddom) {
> +            virDomainFree(ddom);
> +            ret = TRUE;
> +        }
> +        virConnectClose (dconn);
> +    }
>  
>   done:
>      if (dom) virDomainFree (dom);
> -    if (ddom) virDomainFree (ddom);
> -    if (dconn) virConnectClose (dconn);
>      return ret;
>  }
>  
> diff --git a/src/xen_unified.c b/src/xen_unified.c
> index dfa9ca5..954b187 100644
> --- a/src/xen_unified.c
> +++ b/src/xen_unified.c
> @@ -455,8 +455,11 @@ static int
>  xenUnifiedSupportsFeature (virConnectPtr conn ATTRIBUTE_UNUSED, int feature)
>  {
>      switch (feature) {
> -    case VIR_DRV_FEATURE_MIGRATION_V1: return 1;
> -    default: return 0;
> +    case VIR_DRV_FEATURE_MIGRATION_V1:
> +    case VIR_DRV_FEATURE_MIGRATION_P2P:
> +        return 1;
> +    default:
> +        return 0;
>      }
>  }
>  
> diff --git a/src/xend_internal.c b/src/xend_internal.c
> index 7f55116..da5c039 100644
> --- a/src/xend_internal.c
> +++ b/src/xend_internal.c
> @@ -4400,6 +4400,12 @@ xenDaemonDomainMigratePerform (virDomainPtr domain,
>          strcpy (live, "1");
>          flags &= ~VIR_MIGRATE_LIVE;
>      }
> +    /* Trivially support this in Xen, since XenD on dest is always
> +     * ready to accept incoming migration */
> +    if ((flags & VIR_MIGRATE_PEER2PEER)) {
> +        flags &= ~VIR_MIGRATE_PEER2PEER;
> +    }
> +    /* XXX we could easily do tunnelled migration too if we want to */
>      if (flags != 0) {
>          virXendError (conn, VIR_ERR_NO_SUPPORT,
>                        "%s", _("xenDaemonDomainMigrate: unsupported flag"));

  Sounds and looks good to me, with the small caveat of the required
VIR_MIGRATE_PEER2PEER flag and making sure it would work later with
QEmu/KVM before commiting this.

  I must admit peer-2-peer left me a bit puzzled at the beginning
it wasn't obvious what kind of transfer it actually meant. Only de
the documentation clarified it but I have no better suggestion :-)
"direct" maybe ...

Daniel

-- 
Daniel Veillard      | libxml Gnome XML XSLT toolkit  http://xmlsoft.org/
daniel at veillard.com  | Rpmfind RPM search engine http://rpmfind.net/
http://veillard.com/ | virtualization library  http://libvirt.org/




More information about the libvir-list mailing list