[libvirt] [RFC PATCH 0/7] Adding 'config' driver

Adam Walters adam at pandorasboxen.com
Wed Jan 22 13:09:10 UTC 2014


On Mon, Jan 20, 2014 at 11:27 AM, Daniel P. Berrange <berrange at redhat.com>wrote:

> On Fri, Dec 20, 2013 at 11:03:53PM -0500, Adam Walters wrote:
> > This patchset adds a driver named 'config' that allows access to
> configuration
> > data, such as secret and storage definitions. This is a pre-requisite
> for my
> > next patchset which resolves the race condition on libvirtd startup and
> the
> > circular dependencies between QEMU and the storage driver.
>
> I vaguely recall something being mentioned in the past, but not the
> details. Can you explain the details of the circular dependancy
> problem that you're currently facing ?
>
> > The basic rationale behind this idea is that there exist circumstances
> under
> > which a driver may need to access things such as secrets during a time at
> > which there is no active connection to a hypervisor. Without a
> connection,
> > the data can't be accessed currently. I felt that this was a much simpler
> > solution to the problem that building new APIs that do not require a
> connection
> > to operate.
>
> We have a handful of places in our code where we call out to public
> APIs to implement some piece of functionality. I've never been all
> that happy about these scenarios. If we call the public API directly,
> they cause havoc with error reporting because we end up dispatching
> and resetting errors in the middle of an nested API call. Calling the
> driver methods directly by doing conn->driver->invokeMethod() is
> somewhat tedious & error prone because we're then skipping various
> sanity tests that the public APIs do.  With ACL checking now implemented
> we also have the slightly odd situation that a public API check which
> is documented as requiring permission 'xxxx' may also require permissions
> 'yyyy' and 'zzzz' to deal with other public APIs we invoke secretly.
>
> I think there is a fairly strong argument that our internal implementations
> could be decoupled from the public APIs, so we can call methods internally
> without having to go via the public APIs at all.
>
> On the flip side though, there's also a long term desire to separate
> the different drivers into separate daemons, eg so the secrets driver
> might move into a  libvirt-secretsd daemon, which might explicitly
> require use to go via the public APIs.
>

Mulling this over last night, I think there may be an alternative
implementation
that would be sustainable long-term. You mentioned a desire to split the
drivers
into separate daemons eventually... It might seem silly today, but what if
I went
ahead and implemented a connection URI for each of the existing drivers?
This
would result in a 'network:///', 'secrets:///', 'storage:///', etc. Once
complete, the
existing code base could slowly be updated to utilize connections to those
new
URIs in preparation for splitting the code out into daemons. This would end
up
with the current libvirtd process eventually becoming a connection broker,
with
a compatibility shim to allow access to the API as it is currently defined
for
backwards compatibility.

In effect, today nothing would change, other than the storage driver would
use
'secrets:///' to open a connection for its backends. Eventually, though,
all drivers
would use the new URIs (the actual APIs implemented would not need to
change),
allowing an easier split of the daemons. The compatibility shim I mentioned
would
just be some code to open a connection to the proper URI, make the request,
close the connection, and return the data. The one thing that would be
needed
at some point (which I'm not sure how to really implement today), would be a
restriction on which API functions could be called. Basically, if you
opened a
connection to 'secrets:///', you should only be able to access secrets. If
you tried
to access network information, an error would need to be thrown in order to
prevent
all of the drivers having a connection to all of the other drivers, burning
up sockets
on the host system. I'm guessing that this could be implemented using the
ACL
framework that is already in place, but I haven't dug through the code to
verify this yet.

In this long-term desire to split libvirt into multiple daemons, is there a
plan to also
create a libvirt-domaind? Currently, the various hypervisors don't actually
know how
to read the domain XML fully, so a daemon like that would probably be
desired. As a
bonus, it could also allow a single connection to view all running domains,
regardless
of which hypervisor driver actually created it, though that isn't strictly
needed.

The reason I ask is that currently, it is nigh impossible to determine
driver inter-dependencies
automatically. If strict controls were in place to prevent cross-URI calls
(in effect making
libvirt as a single process act as though it were already split), it should
become
easier to determine a driver's dependencies at compile (or possibly even
run) time.
Having that data would allow for a true dependency-based driver load order.
While
I think implementation of a dependency-aware driver loader is probably
beyond my
personal programming skills, it would be the best method in the long run.
Though,
in a multi-process model, it may be simpler to define a new connection
function that
waits (optionally until a timeout expires) for the connection to succeed,
performing
automatic retries in that time. This would allow the various daemons to
start in any order,
yet automatically initialize in the proper dependency order. Any circular
dependency
would, of course, cause a long hang during startup, but a timeout could
detect that and
fail the startup of libvirt as a whole. The same timeout-based connection
function could
also be used in the single-process model, of course, removing the need for
driver load
order to be changed at all if all existing code used the proposed new URIs.
In effect, the
driver load order change would be needed today, but once all existing code
is updated,
the change could be reverted to the current free-for-all model, as there
would be implied
control over that by virtue of the fact that the drivers would wait for
their dependencies to
finish connecting.


>
> > This driver is technically what one may call a hypervisor driver, but it
> > does not implement any domain operations. It simply exists to handle
> requests
> > by drivers for access to informatino that would otherwise require a
> connection.
> > The URI used for this driver is 'config:///' and has been tested working
> on 4
> > different machines of mine, running three different distributions of
> Linux
> > (Archlinux, Gentoo, and CentOS). Being a very simple driver, I would
> expect
> > it to work pretty much anywhere.
> >
> > I would love to hear any comments and suggestions you may have about this
> > driver. At the very least this plus my next patchset resolves the startup
> > race condition on my machine. If a more robust setup (likely a new
> internal
> > API) is in the works, this driver could act as a band-aid to allow access
> > to this type of data in the interim if a better resolution is a ways off.
>
> One big concern I have about this approach is the fact that once we add
> this 'config:///' driver, it is very hard for us to ever remove it again,
> since this concept leaks out into the public API. So we're going to want
> to be very sure that this is something we wish to support long term, and
> I don't really have such confidence myself.
>
> I'd like to understand a bit more about the dependancy issue you're facing
> to see if there's something else we can do to address it.
>
> Daniel
> --
> |: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/:|
> |: http://libvirt.org              -o-             http://virt-manager.org:|
> |: http://autobuild.org       -o-         http://search.cpan.org/~danberr/:|
> |: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc:|
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20140122/01a188d5/attachment-0001.htm>


More information about the libvir-list mailing list