[PATCH v1 02/10] qemu_domain.c: align memory modules before calculating 'initialmem'

Wed Nov 18 11:07:18 UTC 2020

On 11/16/20 4:43 PM, Andrea Bolognani wrote:
> On Fri, 2020-11-13 at 16:23 -0300, Daniel Henrique Barboza wrote:
>> On 11/13/20 10:51 AM, Andrea Bolognani wrote:
>>> I only skimmed the remaining patches and didn't dig into the code as
>>> much, or as recently, as you have, but from a high-level perspective
>>> I don't see why you wouldn't be able to simply move the existing
>>> rounding logic from the command line generator to PostParse? It's not
>>> like the former has access to additional information that the latter
>>> can't get to, right?
>>
>> I was looking into the code and I think that might have the wrong idea here.
>> Apparently we're not aligning memory during migration or snapshot restore.
>> This specific line in qemu_command.c got my attention:
>>
>> -- qemuBuildCommandLine() --
>>
>>       if (!migrateURI && !snapshot && qemuDomainAlignMemorySizes(def) < 0)
>>           return NULL;
>>
>> ------
>>
>> I investigated the history behind this logic and found the following commit:
>>
>> commit c7d7ba85a6242d789ba3f4dae313e950fbb638c5
>> Author: Peter Krempa <pkrempa at redhat.com>
>> Date:   Thu Sep 17 08:14:05 2015 +0200
>>
>>       qemu: command: Align memory sizes only on fresh starts
>>       
>>       When we are starting a qemu process for an incomming migration or
>>       snapshot reloading we should not modify the memory sizes in the domain
>>       since we could potentially change the guest ABI that was tediously
>>       checked before. Additionally the function now updates the initial memory
>>       size according to the NUMA node size, which should not happen if we are
>>       restoring state.
>>       
>>       Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1252685
>>
>> ---------
>>
>> This means that the changes made in this series will not break migration, since
>> the alignment of 'initialmem' is not being triggered in those cases. Which
>> is good.
>>
>> However, you also brought up in an earlier reply that these changes might break
>> "the guest ABI across guest boots (if libvirt is upgraded in between them)". This
>> can't be helped I think - an older ppc64 guest with 4GiB of 'currentMemory' in the
>> XML, that is ending up having 4.256GiB (extra 256MiB) due to the way alignment was
>> being done, will have exactly 4GiB of 'initialmem' after these changes. My point is
>> that we're giving the exact memory the guest is demanding, as intended by the domain
>> XML, in a fresh guest start. This might be considered an ABI break probably, but
>> why would one complain that Libvirt is now giving precisely what was asked for?
>> Avoiding granting extra 256MBs of mem for domains seems worth it, given that we're
>> not impacting live domains or migration.
> 
> In general, changing guest ABI between boots is not something that we
> want to happen.
> 
> I have trouble keeping all the details of memory alignment inside my
> head and I can't quite spend the time necessary to swap them back in
> right now, so please apologies if I'm being vague and of course
> correct me if I'm wrong... Having Peter in the thread will also
> surely help with that :)
> 
> The aim of this series should *not* be to change the calculations
> that happen when aligning memory, but only to reflect them back to
> the domain XML where they can be queried: so for example if the input
> 
>    <memory unit='GiB'>4</memory>
>    <devices>
>      <memory model='nvdimm'>
>        <target>
>          <size unit='MiB'>500</size>
> 
> results in the command line
> 
>    -m 4352m
>    -object memory-backend-file,size=512m
> 
> (the exact numbers are not relevant), then what we want is for the
> XML to be updated at define time so that it reads

I investigate it more and I think I got a few premises wrong here:

- I pointed out that the alignment changes aren't being reflected. This
is wrong. I figured it out when launching real guests and seeing that
the live XML was being updated properly. This diminishes the urgency
of this patch series considerably, and I apologize for the confusion.

- The reason I claimed that is because I got misled by my own
misunderstanding of what qemuxml2xmltest.c does: it checks the
result XML only up to PostParse changes, and that's it. Any further
change in the XML aren't being reflected by the tests. qemuxml2argvtest.c
goes as far as building the command line, and I got the wrong idea
that both tests reflected the same stages of a VM launch. For what
I'm doing here, qemuxml2xml isn't helpful.

All this out of the way:

> 
>    <memory unit='MiB'>4864</memory>
>    <devices>
>      <memory model='nvdimm'>
>        <target>
>          <size unit='MiB'>512</size>
> 
> (again, the numbers are almost certainly wrong) and we want *that*
> XML to generate the same QEMU command line as before.
> 
> If this can't be achieved, or there are other side effects to it that
> I haven't considered, then we're better off leaving the current
> behavior alone (documenting the heck out of it if necessary) instead
> of changing it in ways that would alter guest ABI between boots.

This can be achieve, sort of. I can't just move the alignment to PostParse,
even without changing how it is calculated, because I would break the premises
it's based on today (not be executed on migration or snapshot). To do
that I would need to gate the code into a VIR_DOMAIN_DEF_PARSE_ABI_UPDATE
parse flag. But I'm not sure if this flag is activated on guest reboot
without changing the domain (I'm looking into it ATM, but I don't think
that's the case). So even if we do that, we would still need the
alignment call after PostParse time, as is today, to accommodate existing
guests. Regardless of whether this is worth doing or not (TBH, I'm not
sure) I'll end up sending a couple of patches to reposition the alignment
code out of qemu_command.c (but not up to ParseTime).

What I still want to do is to use VIR_DOMAIN_DEF_PARSE_ABI_UPDATE to
force the alignment of pseries DIMMs in PostParse time. This will not
break ABI of existing guests in any capacity and will fix the alignment
calculation for newer ones. We have precedent for such scenario in
qemuDomainControllerDefPostParse(), when we use this same flag to
define VIR_DOMAIN_CONTROLLER_MODEL_USB_QEMU_XHCI for new pSeries
guests.

I'll post patches today. Let's see how it goes.

DHB

> 