qemu:///embed and isolation from global components

[Daniel P. Berrangé]

On Fri, Mar 06, 2020 at 06:24:15PM +0100, Andrea Bolognani wrote:
> Recently I've been working on integrating qemu:///embed into an
> application. It's been reasonably smooth so far :)
> 
> There's one thing, however, that has caused a bit of confusion, and
> I would like to clarify whether my expectations are incorrect, there
> are genuine bugs in the implementation that need to be addressed, or
> maybe the experimental nature of embedded driver support results in
> not all scenarios having yet been considered and catered for.
> 
> Using virt-qemu-run as an example, when I run it as root I see that
> 
>   * the domain doesn't show up in the output of 'virsh list' for
>     qemu:///system;

This is good.

>   * it does, however, show up in the output of 'machinectl', with
>     class=vm and service=libvirt-qemu;

This is bad. It is one of the gaps we need to deal with.

A long while back I proposed a domain XML option for this:

  https://www.redhat.com/archives/libvir-list/2017-December/msg00729.html

    <resource register="none|direct|machined|systemd"/>

The "none" case meant inherit the cgroups placement of the parent
libvirtd (or now the app using the embedded driver), and would be
a reasonable default for the embedded case.

For the other cases, we certainly need to do something to ensure
uniqueness. This is possibly as simple as including a fixed
prefix like "qemu-$PID" where $PID is the app embedding the
driver. That said, we support closing the app, and re-starting
using the same embedded driver directory, which would change
PID.

>   * virtlogd is automatically spawned, if not already active, to
>     take care of the domain's log file.

This is trickier. The use of virtlogd was to fix a security DoS
where malicious QEMU can write to serial console, or trigger
QEMU to write to stdout/err, such that it exhausts the host
filesystem.  IIUC, virtlogd should still end up writing to
the logfile path associated with the embedded  driver root
directory prefix, so there shouldn't be a filename clash
unless I screwed up.

Since introducing virtlogd, however, I did think of a different
strategy, which would be to connect a FIFO to QEMU as the log
file FD. The QEMU driver itself can own the other end of the FIFO
and do rollover.

Of course you can turn off virtlogd via qemu.conf

> The first one is expected, but the other two were a surprise, at
> least to me. Basically, what I would expect is that qemu:///embed
> would work in a completely isolated way, only interacting with
> system-wide components when that's explicitly requested.

The goal wasn't explicitly to avoid system-wide components,
but it certainly was intended to avoid clashing resources.

IOW, machined, virtlogd are both valid to use, as long as
the resource clashes are avoided. We should definitely have
a way to disable these too.

> In other words, I would expect virtlogd not to be spawned, and the
> domain not to be registered with machinectl; at the same time, if
> the domain configuration is such that it contains for example
> 
>   <interface type='network'>
>     <source network='default'/>
>     <model type='virtio'/>
>   </interface>
> 
> then I would expect to see a failure unless a connection to
> network:///system had been explicitly and pre-emptively opened, and
> not the current behavior which apparently is to automatically open
> that connection and spawning virtnetworkd as a result.

The QEMU embedded driver is explicitly intended to be able to
interact with other global secondary drivers.

If you don't want to use virtnetworkd, then you won't be
creating such an <interface> config in the first place.
The app will have the option to open an embedded seconary
driver if desired. Some of the drivers won't really make
sense as embedded things though, at least not without
extra work. ie a embedded network or nwfilter driver has
no value unless your app has moved into a new network
namespace, as otherwise it will just fight with the
global network driver.

> As a corollary to this, I would expect that two qemu:///embed
> connections using different roots to be able to define domains with
> the same name without any problem, but as it stands this is not
> possible because machinectl will refuse to register the second one.
> There might be other components that freak out as well, that's just
> the first one that blocked me.

Yes, that is definitely needed to work for resources created by
those drivers, so we must solve the name clashes for example.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|