[libvirt] [RFC] libvirt support multi-thread compression for live migration

[Qiao, Liyong]

Hi Eric
Thanks for replying the mail, replied in lines.

BR, Eli(Li Yong)Qiao

-----Original Message-----
From: Eric Blake [mailto:eblake at redhat.com] 
Sent: Tuesday, June 02, 2015 8:21 PM
To: Feng, Shaohe; libvir-list at redhat.com
Cc: Qiao, Liyong; Bhandaru, Malini K; Ding, Jian-feng; Chylinski, Arek; Koniszewski, Pawel; Li, Liang Z; berrange at redhat.com; veillard at redhat.com
Subject: Re: [RFC] libvirt support multi-thread compression for live migration

[correcting list address: libvirt-list, not libver-list]

On 06/02/2015 05:58 AM, Feng, Shaohe wrote:
> Hi folks:
> I'd like to request some comments on enabling multi-thread compression in libvirt.
> 
> Currently, qemu upstream has supported multi-thread compression for live migration.
> $ git log 263170e~1..362ba4e -oneline
> This can preserve bandwidth and speed up the live migration process, 
> so this is an import feature we need to enable so for other high level such as openstack will be benefit.
> 
> We plan to add feature of multi-thread compression and actually the patch is working in process.
> 
> Now some things need for comments.
> 
> 1.      Expose new set/get multi-thread compression parameters API for live migration.
>     Now qemu only supports zlib compression. Maybe LZ4 and gzip will be supported later.
> but there are 3 parameters needed by qemu currently.
> "compress-level":  compression level, default is 1. For zlib compression, it is from 0-9, 0 means use default level, 9 means high compressed.
> "compress-threads":  compression thread count for multi-thread migration, default is 8 in qemu.
> "decompress-threads": decompression thread count for multi-thread 
> migration, default is 2 in qemu
> 
> So in libvirt we will expose the public symbols as follow, we only 3 
> parameters, missing compression method(zlib, LZ4 and gzip) parameters, for qemu do not support it at present.
> index d851225..103b3b9 100644
> --- a/include/libvirt/libvirt-domain.h
> +++ b/include/libvirt/libvirt-domain.h
> @@ -798,6 +798,17 @@ int virDomainMigrateGetMaxSpeed(virDomainPtr domain,
>                                  unsigned long *bandwidth,
>                                  unsigned int flags);
> 
> +int virdomainMigrateSetParameters(virDomainPtr domain,
> +                                  int level,
> +                                  int threads,
> +                                  int dthreads,
> +                                  int flags) int 
> +virdomainMigrateGetParameters(virDomainPtr domain,
> +                                  int *level,
> +                                  int *threads,
> +                                  int *dthreads,
> +                                  int flags)
> +

I'd rather we used virTypedParameters, to make it easier to use the same API to pass ALL future possible tuning parameters, rather than just hard-coding to only the parameters of this one feature.

Okay ,that sound good, but if virTypedParameters can be passed to qemu_driver such as qemu_monitor_json.c qemu_monitor_text.c ?

> 
> For virdomainMigrateSetParameters, if specifying a negative value to 
> level, threads, and dthreads parameters, such as -1, means do not set the parameters.
> The underlying code will not pass the corresponding parameters to qemu monitor.
> Such as threads and dthreads is -1, then pass the following json streaming to qemu.
> { "execute": "migrate-set-parameters" , "arguments":  { 
> "compress-level": 1 } }
> 
> The virsh will expose the two commands:
> # virsh migrate-setparameters  <domain> [--level level] [--threads 
> threads] [--dthreads dthread] # virsh migrate-getparameters  <domain>
> 
> 2.      How to enable multi-thread compression in application interface?
>    There is not a special interface for setting migration capabilities.
>         But we can set the capabilities when execute an virsh command as following:
>         $ migrate --live] [--offline] [--p2p] [--direct] [--tunnelled] 
> [--persistent] [--undefinesource] [--suspend] [--copy-storage-all] 
> [--copy-storage-inc] [--change-protection] [--unsafe] [--verbose] 
> [--compressed] [--abort-on-error] <domain> <desturi> [<migrateuri>] 
> [<graphicsuri>] [<listen-address>] [<dname>] [--timeout <number>] 
> [--xml <string>]
> 
> There is already a "compressed" option,  here "compressed" means "xbzrle" not "multi- compressed".
> can I rename the "compressed" to "xbzrle"?

If we think that it is worth always turning on both compression styles simultaneously, then reuse the bit.  Otherwise, we need a different bit, and users can choose which (or both) of the two compression styles as desired.

+1 for reuse compressed, and we can set compress-level, compress-threads, compress-dthreads by virdomainMigrateSetParameters(maybe some new virsh command--- migrate-setparameter)
But how can we passing these parameter when we using `virsh migrate `, is there any parameter we can use in 'virsh migrate' command ?
Can you point me out ?

> so we add another option for multi-thread compression, which is better option name? "-multi- compressed" ?
> Any way this is confuse to user. We need to distinguish<http://dict.youdao.com/w/distinguish/> these two.
> 
> 
>    So I wonder should we expose new API to enable the migrate multi-thread compress.
> +int virdomainMigrateEnableMultiThreadCompress (virDomainPtr domain,  
> +int flags)
>    It will pass the following command to qemu monitor.
>    { "execute": "migrate-set-capabilities" , "arguments": { 
> "capabilities": [ { "capability": " compress", "state": true } ] } }
> 
> NOTE:  Now, multiple thread compression can co-work with xbzrle. When xbzrle is on, multiple thread compression will only work at the first round of RAM data sync.
> Qemu, $ git show 98f1138902195bd9ab8a753d0ee2cf2a0a88b6e8
> if compress and xbzrle are both on, compress only takes effect in the ram bulk stage, after that, it will be disabled and only xbzrle takes effect, this can help to minimize migration traffic.
> 
> 3.      Migration between different livirt/qemu version
> We can support to expose 2 new API to set/get migrate capabilities.
> + virdomainMigrateSetCapabilities
> + virdomainMigrateGetCapabilities
>    And we can suggest the user application to probe the capabilities before execute live migration in our doc.
> This need discussion is it necessary support these two in libvirt?
> 
> Without the above GetCapabilities API, user application can probe multi-thread compress capabilities by virdomainMigrateGetParameters.
> 
> Or
> return error directly when user application execute live migration with multi-thread flag specifying, but do not any Capabilities probe.
> 
> 
> INFO: qmeu command
> {"execute": "query-migrate-capabilities"}
>    {"return": [{"state": false, "capability": "xbzrle"}, {"state": 
> false, "capability": "rdma-pin-all"}, {"state": false, "capability": 
> "auto-converge"}, {"state": false, "capability": "zero-blocks"}, 
> {"state": false, "capability": "compress"}]}
> 
> 
> 

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org