[libvirt] [[RFC] 0/8] Implement async QEMU event handling in libvirtd.

Wed Mar 28 09:16:37 UTC 2018

Hi Marc,
Currently the block job handling needs to be sorted out before events can
assume independent handling from RPC contexts. Sorry, I have not been able
to revisit this in the past 2 months, but this is something I would very
much like to fix. I will try looking up the exact dependency of the block
layer so that this series could make some progress.

regards,
Prerna

On Wed, Mar 21, 2018 at 1:24 PM, Marc Hartmayer <mhartmay at linux.vnet.ibm.com
> wrote:

> On Tue, Oct 24, 2017 at 07:34 PM +0200, Prerna Saxena <
> saxenap.ltc at gmail.com> wrote:
> > As noted in
> > https://www.redhat.com/archives/libvir-list/2017-May/msg00016.html
> > libvirt-QEMU driver handles all async events from the main loop.
> > Each event handling needs the per-VM lock to make forward progress. In
> > the case where an async event is received for the same VM which has an
> > RPC running, the main loop is held up contending for the same lock.
> >
> > This impacts scalability, and should be addressed on priority.
> >
> > Note that libvirt does have a 2-step deferred handling for a few event
> > categories, but (1) That is insufficient since blockign happens before
> > the handler could disambiguate which one needs to be posted to this
> > other queue.
> > (2) There needs to be homogeniety.
> >
> > The current series builds a framework for recording and handling VM
> > events.
> > It initializes per-VM event queue, and a global event queue pointing to
> > events from all the VMs. Event handling is staggered in 2 stages:
> > - When an event is received, it is enqueued in the per-VM queue as well
> >   as the global queues.
> > - The global queue is built into the QEMU Driver as a threadpool
> >   (currently with a single thread).
> > - Enqueuing of a new event triggers the global event worker thread, which
> >   then attempts to take a lock for this event's VM.
> >     - If the lock is available, the event worker runs the function
> handling
> >       this event type. Once done, it dequeues this event from the global
> >       as well as per-VM queues.
> >     - If the lock is unavailable(ie taken by RPC thread), the event
> worker
> >       thread leaves this as-is and picks up the next event.
> > - Once the RPC thread completes, it looks for events pertaining to the
> >   VM in the per-VM event queue. It then processes the events serially
> >   (holding the VM lock) until there are no more events remaining for
> >   this VM. At this point, the per-VM lock is relinquished.
> >
> > Patch Series status:
> > Strictly RFC only. No compilation issues. I have not had a chance to
> > (stress) test it after rebase to latest master.
> > Note that documentation and test coverage is TBD, since a few open
> > points remain.
> >
> > Known issues/ caveats:
> > - RPC handling time will become non-deterministic.
> > - An event will only be "notified" to a client once the RPC for same VM
> completes.
> > - Needs careful consideration in all cases where a QMP event is used to
> >   "signal" an RPC thread, else will deadlock.
> >
> > Will be happy to drive more discussion in the community and completely
> > implement it.
> >
> > Prerna Saxena (8):
> >   Introduce virObjectTrylock()
> >   QEMU Event handling: Introduce async event helpers in qemu_event.[ch]
> >   Setup global and per-VM event queues. Also initialize per-VM queues
> >     when libvirt reconnects to an existing VM.
> >   Events: Allow monitor to "enqueue" events to a queue. Also introduce a
> >     framework of handlers for each event type, that can be called when
> >     the handler is running an event.
> >   Events: Plumb event handling calls before a domain's APIs complete.
> >   Code refactor: Move helper functions of doCoreDump*, syncNicRxFilter*,
> >     and qemuOpenFile* to qemu_process.[ch]
> >   Fold back the 2-stage event implementation for a few events :
> >     Watchdog, Monitor EOF, Serial changed, Guest panic, Nic RX filter
> >     changed .. into single level.
> >   Initialize the per-VM event queues in context of domain init.
> >
> >  src/Makefile.am              |    1 +
> >  src/conf/domain_conf.h       |    3 +
> >  src/libvirt_private.syms     |    1 +
> >  src/qemu/qemu_conf.h         |    4 +
> >  src/qemu/qemu_driver.c       | 1710 +++++++----------------------------
> >  src/qemu/qemu_event.c        |  317 +++++++
> >  src/qemu/qemu_event.h        |  231 +++++
> >  src/qemu/qemu_monitor.c      |  592 ++++++++++--
> >  src/qemu/qemu_monitor.h      |   80 +-
> >  src/qemu/qemu_monitor_json.c |  291 +++---
> >  src/qemu/qemu_process.c      | 2031 ++++++++++++++++++++++++++++++
> ++++--------
> >  src/qemu/qemu_process.h      |   88 ++
> >  src/util/virobject.c         |   26 +
> >  src/util/virobject.h         |    4 +
> >  src/util/virthread.c         |    5 +
> >  src/util/virthread.h         |    1 +
> >  tests/qemumonitortestutils.c |    2 +-
> >  17 files changed, 3411 insertions(+), 1976 deletions(-)
> >  create mode 100644 src/qemu/qemu_event.c
> >  create mode 100644 src/qemu/qemu_event.h
> >
> > --
> > 2.9.5
> >
> > --
> > libvir-list mailing list
> > libvir-list at redhat.com
> > https://www.redhat.com/mailman/listinfo/libvir-list
>
> Hey Prerna,
>
> any updates so far? :) I’m just curious about the status of this series
> as it would fix a performance problem we’ve.
>
> Thank you.
>
> Beste Grüße / Kind regards
>    Marc Hartmayer
>
> IBM Deutschland Research & Development GmbH
> Vorsitzende des Aufsichtsrats: Martina Koederitz
> Geschäftsführung: Dirk Wittkopp
> Sitz der Gesellschaft: Böblingen
> Registergericht: Amtsgericht Stuttgart, HRB 243294
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20180328/ee8d94e5/attachment-0001.htm>