[libvirt] [PATCH 00/11] Generic data stream handling

[Daniel P. Berrange]

The following series of patches introduce support for generic data
streams in the libvirt API, the remote protocol, client & daemon.

The public API takes the form of a new object virStreamPtr and
methods to read/write/close it

The remote protocol was the main hard bit. Since the protocol
allows multiple concurrent API calls on a single connection,
this needed  to also allow concurrent data streams. It is also
not acceptable for a large data stream to block other traffic
while it is transferring.

Thus, we introduce a new protocol message type 'REMOTE_STREAM'
to handle transfer for the stream data. A method involving a
data streams starts off in the normal way with a REMOTE_CALL
to the server, and a REMOTE_REPLY  response message. If this
was successful, there now follows the data stream traffic.

For outgoing streams (data from client to server), the client
will send zero or more REMOTE_STREAM packets containing the
data with status == REMOTE_CONTINUE. These are asynchronous
and not acknowledged by the server. At any time the server
may send an async message with a type of REMOTE_STREAM and
status of REMOTE_ERROR. This indicates to the client that the
transfer is aborting at server request. If the client wishes
to abort, it can send the server a REMOTE_STREAM+REMOTE_ERROR
message. If the client finishes its data transfer, it will
send a final REMOTE_STREAM+REMOTE_OK message, and the server
will respond with the same. This full roundtrip handshake
ensures any async error messages are guarenteed to be flushed

For incoming data streams (data from server to client), the
server sends zero or more REMOTE_STREAM packets containing the
data with status == REMOTE_CONTINUE. These are asynchronous
and not acknowledged by the client. At any time the client
may send an async message with a type of REMOTE_STREAM and
status of REMOTE_ERROR. This indicates to the server that the 
transfer is aborting at client request. If the server wishes
to abort, it can send the server a REMOTE_STREAM+REMOTE_ERROR
message. When the server finishes its data transfer, it will
send a final REMOTE_STREAM+REMOTE_CONTINUE message ewith a 
data length of zero (ie EOF). The client will then send a 
REMOTE_STREAM+REMOTE_OK packet and the server will respond
with the same. This full roundtrip handshake ensures any async
error messages are guarenteed to be flushed

This all ensures that multiple data streams can be active in
parallel, and with a maximum data packet size of 256 KB, no
single stream can cause too much latency on the connection for
other API calls/streams.

The only thing it does not allow for is one API method to use
two or more streams. These may be famous last words, but I
don't think that use case will be neccessary for any of our
APIs...

The last 5 patches with a subject of [DEMO] are *NOT* intended
to be committed to the repository. They merely demonstrate the
use of data streams for a couple of hypothetical file upload
and download APIs. Actually they were mostly to allow me to
test the code streams code without messing around with the QEMU
migration code.

The immediate use case for this data stream code is Chris' QEMU
migration patchset.

The next use case is to allow serial console access to be tunnelled
over libvirtd, eg to make  'virsh console GUEST' work remotely.
This use case is why I included the support for non-blocking data
streams and event loop integration (not required for Chris'
migration use case)

Anyway, assuming Chris confirms that I've not broken his code, then
patches 1-6 are targetted for this next release.

.x-sc_avoid_write                  |    3 
 include/libvirt/libvirt.h          |  101 ++++
 include/libvirt/libvirt.h.in       |  101 ++++
 po/POTFILES.in                     |    1 
 qemud/Makefile.am                  |    1 
 qemud/dispatch.c                   |  131 ++++++
 qemud/dispatch.h                   |   17 
 qemud/event.c                      |   54 +-
 qemud/qemud.c                      |   56 +-
 qemud/qemud.h                      |   33 +
 qemud/remote.c                     |  237 ++++++++++-
 qemud/remote.h                     |    1 
 qemud/remote_dispatch_args.h       |    2 
 qemud/remote_dispatch_prototypes.h |  153 +++++++
 qemud/remote_dispatch_table.h      |   10 
 qemud/remote_generate_stubs.pl     |    1 
 qemud/remote_protocol.c            |   18 
 qemud/remote_protocol.h            |   20 
 qemud/remote_protocol.x            |   57 ++
 qemud/stream.c                     |  605 +++++++++++++++++++++++++++++
 qemud/stream.h                     |   53 ++
 src/datatypes.c                    |   59 ++
 src/datatypes.h                    |   33 +
 src/driver.h                       |   40 +
 src/esx/esx_driver.c               |    2 
 src/libvirt.c                      |  769 +++++++++++++++++++++++++++++++++++++
 src/libvirt_private.syms           |    2 
 src/libvirt_public.syms            |   18 
 src/lxc_driver.c                   |    2 
 src/opennebula/one_driver.c        |    2 
 src/openvz_driver.c                |    2 
 src/qemu_driver.c                  |    2 
 src/remote_internal.c              |  601 ++++++++++++++++++++++++++++
 src/test.c                         |  477 ++++++++++++++++++++++
 src/uml_driver.c                   |    2 
 src/vbox/vbox_tmpl.c               |    3 
 src/virsh.c                        |  164 +++++++
 src/xen_unified.c                  |    2 
 tests/eventtest.c                  |   19 
 39 files changed, 3783 insertions(+), 71 deletions(-)

Regards,
Daniel