[libvirt] Failure to migrate a guest from libvirt-0.9.10+ to libvirt-0.9.4-

[Laine Stump]

On 04/10/2012 05:36 AM, Daniel P. Berrange wrote:
> On Tue, Apr 10, 2012 at 11:58:52AM +0800, Daniel Veillard wrote:
>>   I realize that we have that behaviour for quite some times but I
>> wonder about it, basically we always dump an usb controller on the
>> XML domain format:
>>
>>   <controller type='usb' index='0'/>
>>
>> even if there is no USB device connected to the machine. The drawback
>> I see is that if we try to reuse (or migrate) that domain to an older
>> verlion of libvirt without that USB controller support, then we just
>> fail to parse the domain, even though we don't need the missing
>> capability.


The real problem is that libvirt prior to 0.9.5 does not recognize a
controller of type 'usb', and since this new element is added during any
parse of the domain (rather than just during define), it will be added
to the xml that is sent during migrate - if the destination host is <
libvirt-0.9.5, the migration will always fail. Note that, prior to
0.9.5, libvirt was automatically adding a usb controller to the qemu
commandline anyway, so the hardware being presented to the guest in both
cases is identical, and supported by the destination libvirt/qemu. The
only problem is that the XML that's now being sent (which would tell the
destination libvirt to create a device that it's going to create anyway)
is causing an error when libvirt tries to parse the domain xml.


> As with downgrading libvirt on a host, migrating a guest to an older
> version of libvirt is not something we can really support, since you
> run a high risk of breaking QEMU / guest ABI, due to the older libvirt
> missing out some hardware device that new libvirt has added.

While it's not ideal, migrating a running guest to an older host may be
something that is necessary from time to time and, while it's obviously
not possible in the case where the older host doesn't support the
machine type of the guest, we've already done lots of work to make sure
that various versions of libvirt can talk to each other in order to
perform a migrations (witness all the different migration protocols
currently supported by libvirt). There are a lot of situations where
this migration would work properly, if not for this one little problem
in the XML (which leads to absolutely *no* change in the hardware
actually presented to the guest).

Some of us have had offline discussion about this today, and a few
suggestions were made. Since all the distros could face this problem, it
seems best to try and continue this discussion on the list so that
everyone can participate.

(Unfortunately, the libvirt version doesn't appear anywhere in the XML
nor in the migration protocol, so we can't just make a decision whether
or not to include the usb controller based on a version number that's
easily available.)

Here's some of the ideas that were discussed, and at least one reason
for each not being an obvious solution:

1) Require the destination to first upgrade to a libvirt that supports
the new XML. While this is a reasonable thing for upstream to say, it
may not be so easy for a downstream (distro) version.

2) don't auto-generate <controller type='usb'> when parsing/formatting
for the purposes of migration. This is less than ideal because it
deprives migration of the advantages provided by an explicit <controller>.

3) key off of some specific migration detail of the destination host
that is only present in libvirt >= 0.9.5. This seems very tricky.

4) only auto-generate the <controller> if there are actually USB devices
in the domain that will be affected by its existence. This isn't really
useful, because in most cases there is a usb tablet device, and even if
there wasn't, it's still desirable to have a usb controller in the XML
so that virt-manager can easily change it to USB2.0 (and also for the
PCI slot of the usb controller to be explicitly reserved on the config)

(In the end it may be that the decision is just that upstream libvirt
doesn't go to any pains to support this scenario, and the the various
downstream builds need to figure out their own solutions if they want to
support it, but I thought I'd at least give it a chance here.)


>
>>   So I wonder if the correct thing isn't to add the USB controller
>> only if there is an USB device associated to the domain, then failure
>> to migrate on an older libvirt does make sense because we require the
>> feature. I would assume that application using the USB support wouldn't
>> need to have the USB hub exposed to allow adding an USB device, and once
>> the USB device is added then changing the kind of USB hub to select
>> a different version does make sense.
>>
>>   In general I'm tempted to not dump in the XML things which are there
>> by default and would be automatically added if changed or used in some
>> ways. I think there is two advantages to this:
>>   - keep the XML smaller and somehow more readable
>>   - avoid portability problems on unsupported but unused constructs
>> One drawback I could perceive though is that having defaults values not
>> explicitely dumped in the XML could lead to change of guest behaviour if
>> we changed the default meaning of such default value. But we can just
>> document this (like for reserved PCI default slots) and do our best to
>> not breakdefault behaviour.
> Over time we have been moving towards the scenario where we fully
> describe all devices in the VM. This has been particularly important
> in the area of device addressing, since applications need to know
> what PCI addresses the default devices are consuming, if they want
> to add new devices with fixed PCI addresses.
>
> If we removed the default USB controller, applications would have
> to hardcode knowledge of the fact that there is a USB controller
> consuming address X.Y.Z. This would be a step backwards IMHO, as
> we want apps to have less hardcoded knowledge of this kind
>
> Daniel