[libvirt] Tunnelled post-copy live migration - support of bidirectional communication over the tunnel

[Vojtech Cima]

> On 15 Jan 2015, at 14:56, Daniel P. Berrange <berrange at redhat.com> wrote:
> 
> On Thu, Jan 15, 2015 at 02:45:13PM +0100, Vojtech Cima wrote:
>> Hello everyone,
>> 
>> we are very interested about the post-copy live migration mechanism in
>> libvirt [1] using QEMU. Mentioning post-copy in this context means the setup
>> with the first full iteration of standard pre-copy and then switching to the
>> post-copy. VIR_MIGRATE_ENABLE_POSTCOPY, VIR_MIGRATE_POSTCOPY_AFTER_PRECOPY
>> together with VIR_MIGRATE_TUNNELLED flag.
>> 
>> Recently we have noticed the issue that post-copy live migration doesn't
>> work over the tunnel since since current tunnel implementation in libvirt
>> supports only uni-directional communication. I spent some time trying to
>> enable bidirectional communication over the tunnel but still facing several
>> issues and I would appreciate any hints how to move forward.
>> 
>> The idea is to add a function (probably a callback function) responsible for
>> handling the traffic in the opposite direction (dest->src) reading from the
>> virStream and writing to the QEMU [2]. To be able to do so I replaced the
>> current 'pipe' calls with the 'socketpair' calls to create bidirectional
>> paths.
>> 
>> Now, trying to use tunnelled post-copy live migration, it gets initiated,
>> there is some traffic going from source to destination (obviously the
>> pre-copy phase) but at some point it gets stuck on the 'saferead' function
>> [3] which is now blocking. Can this issue issue be linked with the default
>> socket blocking policy?
>> Trying to use standard 'read' function instead of 'saferead' results that
>> the lower data transfers and it seems it's missing significant part of
>> information.
> 
> Since the code was historically unidirectional, the qemuMigrationIOFunc()
> method used the virStream object in blocking I/O mode, and used a simple
> loop to read from the pipe / write to the stream. If you want to do
> bi-directional transfer that won't be possible - you'll need to put the
> stream into fully non-blocking mode and use event callbacks to deal with
> all I/O. This also means you can't use saferead either. The code in
> tools/virsh-console.c is a reasonably good example of how to use a
> stream in non-blocking mode doing  bi-directional I/O between the stream
> and regular file descriptors. The virsh-console uses stdin/stdout but
> you'd just use your socketpair instead.

Thanks for your hints. I have created a patch handling the bi-directional communication using callbacks in 'src/qemu/qemu_migration.c' and it seems to be working fine although it handles migration traffic only on the source host.

The migration traffic on the destination host is handled in 'daemonStreamEvent' callback in ‘daemon/stream.c’. There is a stream object representing connection to/from destination QEMU client. The problem in current implementation is that callback is setup to check for VIR_STREAM_EVENT_WRITABLE most of the time (only if the transmit parameter allowed the callback listens for VIR_STREAM_EVENT_READABLE but just once. Since immediately after it reads one msg from QEMU callback is updated back to trigger only on WRITABLE event again). 
Do you think there can be any problem if this callback is setup to be triggered on both events all the time in order to handle migration I/O equally? 

> 
> 
>> Another point is that we suppose that the virStream on its own supports
>> bidirectional communication, is that correct?
> 
> Yes, absolutely.
> 
>> 
>> [1] https://github.com/orbitfp7/qemu/tree/wp3-postcopy
>> 
>> [2] http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/qemu/qemu_migration.c;h=0acbb5795e930e27157c580a3236701acd91bbaf;hb=e3435caf6af41748204e542dee13ede8441d88c0#l3595
>> 
>> [3] http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/qemu/qemu_migration.c;h=0acbb5795e930e27157c580a3236701acd91bbaf;hb=e3435caf6af41748204e542dee13ede8441d88c0#l3644
>> 
>> Thank you very much for any ideas on this topic.
> 
> Regards,
> Daniel
> -- 
> |: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
> |: http://libvirt.org              -o-             http://virt-manager.org :|
> |: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
> |: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Regards,
Vojtech