[libvirt] libvirt<->QEMU interfaces for CPU models

[Eduardo Habkost]

On Fri, Mar 01, 2013 at 02:12:38PM +0100, Jiri Denemark wrote:
> On Thu, Feb 21, 2013 at 11:58:18 -0300, Eduardo Habkost wrote:
> > Hi,
> > 
> > After a long time trying to figure out the proper modelling inside QEMU,
> > I believe the plans are now clearer in QEMU, so it's time to coordinate
> > more closely with libvirt to try to make use of the new stuff.
> > 
> > I tried to enumerate the libvirt requirements and current problems, and
> > how we should be able to solve those problems using the X86CPU
> > subclasses and properties, on the following wiki page:
> > 
> > http://wiki.qemu.org/Features/CPUModels#Interfaces.2Frequirements_from_libvirt
> 
> > = Ensuring predictable set of guest features =
> > 
> > Requirement: libvirt needs to ensure all features required on the command-line
> > are present and exposed to the guest.
> > 
> > Current problem: libvirt doesn't use the "enforce" flag so it can't guarantee
> > that a given feature will be actually exposed to the guest.
> > 
> > Solution: use the "enforce" flag on the "-cpu" option.
> 
> Definitely, we plan to start using "enforce" flag as soon as we have
> better CPU probing interface with QEMU. Since libvirt does not currently
> consult CPU specs with QEMU, some configurations in fact rely on QEMU
> dropping features it can't provide. Of course, that's bad for several
> reasons but we don't want such configurations to suddenly stop working.
> We want to first fix the CPU specs libvirt creates so that we know they
> will work with "enforce".

Also: more important than fixing the CPU definitions from libvirt, is to
ask QEMU for host capabilities and CPU model definitions. The whole
point of this is to solve the CPU model data duplication/synchronization
problems between libvirt and QEMU.

Once you are able to query CPU model definitions on runtime, you don't
even need to make cpu_map.xml agree with QEMU. You can simply ask QEMU
how each model looks like, and remove/add features from the command-line
as necessary, so the resulting VM matches what the user asked for.


> 
> >     Limitation: no proper machine-friendly interface to report which features
> >     are missing.
> > 
> >         Workaround: See "querying for host capabilities" below.
> 
> I doubt we will be ready to start using "enforce" before the machine
> friendly interface is available...

If you query for the "-cpu host" capabilities first and ensure all
features from a CPU model is available, enforce is supposed to not fail.

I understand that a machine-friendly error reporting for "enforce" would
be very useful, but note that if "enforce" fails, it is probably already
too late for libvirt, and that means that what libvirt thinks about host
capabilities and CPU models is already incorrect.


The main problem preventing us from making a machine-friendly interface
is that "enforce" makes QEMU abort immediately, making us lose the main
machine-friendly communication mechanism, that is QMP.

(But I had an idea to solve that, look for "removed-features" below for
a description).


> 
> 
> > = Listing CPU models =
> > 
> > Requirement: libvirt needs to know which CPU models are available to be used
> > with the "-cpu" option.
> > 
> > Current problem: libvirt relies on help output parsing for that.
> > 
> > Solution: use QMP qom-list-types command.
> > 
> >     Dependency: X86CPU subclasses.
> >     Limitation: needs a live QEMU process for the query.
> 
> No problem, we already run QEMU and use several QMP commands to probe
> its capabilities. And "qom-list-types" is actually one of them. To get
> the list of CPU models, we would just call
> 
>     {
>         "execute": "qom-list-types",
>         "arguments": {
>             "implements": "X86CPU"
>         }
>     }
> 
> right? What about other non-x86 architectures? Will we need to use
> different class name or is there a generic CPU class that could be used
> universally?

Actually I don't know much about the QMP command syntax and didn't test
it a lot. But that's basically how I think it will look like. Except
that instead of "X86CPU", the type name is "x86_64-cpu" (on
qemu-system-x86_64) and "i386-cpu" (on qemu-system-i386). Maybe it is
easier to simply use: implements: "cpu", abstract: false }.

We may also end up with different CPU model classes for KVM and TCG,
this is still under discussion.

Another caveat: the CPU model class names will be longer than the names
used on the "-cpu" command-line: something like "<model>-<arch>-cpu" or
"<model>-kvm-<arch>-cpu".


> 
> > Solution: use QMP query-cpu-definitions command.
> > 
> >     Limitation: needs a live QEMU process for the query.
> 
> IIUC, the result of this command will depend on machine type and we
> can't use -M none we currently use for probing, right?

The class data don't depend on machine-types. But the resulting CPU
objects may look different depending on machine type.

In other words: you don't need -M to list CPU models, but you need -M if
you want to know which features are going to be available on each model,
exactly.

> 
> > == Future plans ==
> > 
> > It would be interesting to get rid of the requirement for a live QEMU process
> > (with a complete machine being created) to be already running.
> 
> Hmm, so is this complete machine needed even for getting CPU models from
> qom-list-types or only for querying exact definitions using
> query-cpu-definitions command?

Maybe "complete machine" isn't the right expression, here. What I mean is:
AFAIK, it is not possible to get a QMP monitor without actually having a
machine being created by QEMU (even if it is a machine that will never run).

But answering your question: to query the CPU model list you don't need
the right machine type or anything fancy. "qemu -nodefaults" should
work. But to query the exact CPU definition (the item below), you need
to specify the right machine-type.

> 
> Actually, what is query-cpu-definitions supposed to return? Currently it
> seems it's just the CPU model names rather than details about all CPU
> models. From the command name, one would expect to get more than just
> names.

query-cpu-definitions returns only the CPU model names, and a proposal
to include more detailed CPUID information was rejected, IIRC. I believe
we should deprecate query-cpu-definitions it in favor of the QOM
commands.

> 
> 
> > = Getting information about CPU models =
> > 
> > Requirement: libvirt uses the predefined CPU models from QEMU, but it needs to
> > be able to query for CPU model details, to find out how it can create a VM that
> > matches what was requested by the user.
> > 
> > Current problem: libvirt has a copy of the CPU model definitions on its
> > cpu_map.xml file, and the copy can be out of sync in case CPU models in QEMU
> > change. libvirt also assumes that the set of features on each model is always
> > the same on all machine-types, which is not true.
> > 
> >     Challenge: the resulting CPU features depend on lots of factors, including
> >     the machine-type.
> > 
> >         Workaround: start a paused VM and query for the CPU device information
> >         after the CPU was created.

I just noticed another problem here, but this gave me an idea that would
help solve the "enforce" error reporting problem:

  Problem: "qemu -machine <M> -cpu <model>" will create CPU objects
  where the CPU features are _already_ filtered based on host
  capabilities.

    * Using "enforce" wouldn't solve it, because then QEMU would abort, and
      QMP would be unavailable.

    Solution: we could have a CPU object property like
    "removed-features" that would have the list of features that were
    disabled because they are not supported by the host (and would make
    "enforce" fail).

      * This would solve the problem above and also be a machine-friendly
        way to check for possible "enforce" errors.

      * In other words: instead of "enforce", libvirt could use "check"
        instead of "enforce", and before unpausing the VM (or even starting
        migration), it should first check if the "removed-features" property is
        empty.

Would that work for you?


> > 
> >     Solution: start a paused VM with no devices, but with the right
> >     machine-type and right CPU model. Use QMP QOM commands to query for CPU
> >     flags (especially the properties starting with the "f-" prefix).
> > 
> >         Dependency: X86CPU feature properties ("f-*" properties).
> >         Limitation: requires a live QEMU process with the right machine-type/
> >         CPU-model to be started, to make the query.
> 
> This would be very useful for ensuring the guest sees the exact same CPU
> after it's been migrated or restored from a stored state or a snapshot.
> Should we make sure the guest will always see the same CPU even after
> shutdown or is it ok if the guest CPU changes a bit on next boot, e.g.,
> in case the host kernel was upgraded and is able to provide more
> features?

It is up to you, it depends on what the libvirt API promises. But I always
assumed that the same libvirt XML config would always create an equivalent
machine (with not guest-visible differences), it didn't matter on which host it
is running. Otherwise you would have problems like:

 * A VM being migratable to another host and suddenly it stops being
   migratable just because it was restarted.
 * A VM being migratable from A to B when started on host A,
   but not migratable from B to A if exactly the same VM XML is started
   on host B.


> 
> However, probing several CPU definitions for compatibility with
> host/kernel/QEMU would be quite inefficient. Although I guess we should
> be able to limit doing so only in case a specific API (e.g.,
> virConnectCompareCPU or virConnectBaselineCPU) is called, which would be
> acceptable, I think.

I am not familiar with the libvirt code, but maybe the only cases where
you need to do the probing are:

 * virConnectCompareCPU()
 * virConnectBaselineCPU() (I don't understand completely what it does)
 * Before starting a VM, when using match={minimu,exact,strict}.
 * When "host-model"/VIR_DOMAIN_XML_UPDATE_CPU is used
 * When checking if a given VM is migratable to a specific host (before
   actually starting migration). How is that implemented today?

But I worry that at some point you may need to list detailed information
about every CPU model, and this would require running QEMU once for each
CPU model. Would that be a huge problem, or something we can work with?

I quickly tested how quick QEMU can answer QMP queries:

    $ time sh -c '(echo "{\"execute\":\"qmp_capabilities\"}";echo "{\"execute\":\"qom-list-types\", \"arguments\": {}}";echo "{\"execute\":\"quit\"}"; ) | qemu-system-x86_64 -S -nodefaults -cpu SandyBridge,check -qmp stdio -nographic'
    [...]
    real    0m0.038s
    user    0m0.024s
    sys     0m0.012s

Maybe this is reasonable enough? You probably just need to do that before
starting a VM you never started before (for the cases where you want to choose
the CPU model that would generate the shorter command-line).

Also, note that the resulting CPU features for a given machine-type + CPU pair
is supposed to never ever change on any future QEMU version, so libvirt should
be able to safely cache those values even if the QEMU binary changes.

> 
> 
> > = Querying host capabilities =
> > 
> > Requirement: libvirt needs to know which feature can really be enabled, before
> > it tries to start a VM, and before it tries to start a live-migration process.
> 
> If we have machine friendly output of -cpu enforce, we may limit this
> need to just a few configuration types (the ones effectively requesting
> some kind of "host" CPU model). For other cases, libvirt could just try
> to start QEMU with the requested CPU definition and report usable error
> back rather than trying to pre-check the CPU.

True.


PS: I will be out of office for 3 weeks, starting tomorrow, so I will probably
not be able to continue this discussion after today, until I'm back on
25/March.

-- 
Eduardo