[vfio-users] [OVMF+VIFO+QEMU] Large VM RAM allocation = Long Boot Times

[A Z]

On Thu, Nov 15, 2018 at 2:47 AM Laszlo Ersek <lersek at redhat.com> wrote:
>
> If nothing at all had changed in your configuration, then the behavior
> wouldn't have changed either. You must certainly have upgraded some
> components since then, between host kernel, QEMU, and OVMF (likely all
> of them). Please state the version numbers that work and don't work, and
> try to isolate the issue to a version change that affects a *single*
> component.

I wasn't clear on this, sorry. I meant that the VM configuration --
the qemu command-line parameters, *except* for the OVMF firmware image
and the amount of memory assigned, haven't changed since I got
everything up-and-running (i.e., CPU caps, disk, network and all the
PCI-passthrough assignments, etc haven't changed.)
I added more memory to the host a few months ago, and upped the VM
allocation to match at the same time. This made the boot times even
worse.
I've also occasionally downloaded a newer version of OVMF to see if it
fixes this boot time issue.
The host OS kernel, qemu and other package versions have changed over
this period as I've installed OS updates. So has the build of Windows
10.

> I vaguely recall similar reports from the past. I've been under the
> impression that this kind of issue has been fixed. Thus, I'd guess you
> are running old(er) versions of the affected component (whichever it
> is), or else you are running bleeding edge, and the bug has crept back
> in (new regression).

It's possibly the latter.

Host kernel: 4.18.10-arch1-1-ARCH #1 SMP PREEMPT Wed Sep 26 09:48:22
UTC 2018 x86_64 GNU/Linux
qemu version: QEMU emulator version 3.0.0
OMVF build: ovmf-x64-0-20181105.757.g114a75ad86 (downloaded from www.kraxel.org)


On Thu, Nov 15, 2018 at 6:14 AM Hristo Iliev <hristo at hiliev.eu> wrote:

> This sounds quite like a problem I used to have in the past with certain versions of OVMF and the Linux kernel. If I remember correctly, some memory ranges would get marked as non-cacheable, which resulted in the terrible slowdown you describe. The resolution back then was to stick to older versions of both the firmware and the kernel.
>
> I still keep around an older version of OVMF that I used until recently - edk2.git-ovmf-x64-0-20160714.b1992.gfc3005.noarch.rpm. You could download the RPM here and try if it works for you:
>
> https://exa.flopser.de/s/z3cjqFUlZ2gBPCy

I will try this firmware build. I have read similar things about
versions from 2016 working correctly.
But TBH I'd also prefer to run something newer -- bug fixes (and
regressions!), security patches, etc.

> Recent QEMU versions started complaining about firmware incompatibilities, so I tried the latest RPM from Kraxel (https://www.kraxel.org/repos/jenkins/edk2/) and it works just fine. The host system is Arch Linux with the latest Arch kernel and QEMU.

While I'm not running today's build, I am running one downloaded from
Kraxel.org within the past week.


On Thu, Nov 15, 2018 at 7:28 AM Alex Williamson
<alex.williamson at redhat.com> wrote:

> If you can confirm that your system uses a NUMA memory configuration
> and that 24G exceeds the node local memory, then this is a known
> regression in the Linux kernel mm.  AIUI it's being worked upstream,
> workarounds are to either use hugepages or locked memory for the VM.

I'm currently using hugepages. Not sure how to lock the memory, sorry.

It is a dual CPU system. The topology:

$ sudo numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 24 25 26 27 28 29 30 31 32 33 34 35
node 0 size: 32143 MB
node 0 free: 236 MB
node 1 cpus: 12 13 14 15 16 17 18 19 20 21 22 23 36 37 38 39 40 41 42
43 44 45 46 47
node 1 size: 32232 MB
node 1 free: 95 MB
node distances:
node   0   1
  0:  10  21
  1:  21  10



For reference, the qemu command line I'm exec'ing from a script is:

/usr/sbin/qemu-system-x86_64 \
        -name win10,debug-threads=on -monitor
unix:win10.sock,server,nowait -nodefconfig -debugcon file:debugcon.log
-global isa-debugcon.iobase=0x402 \
        -nodefconfig -no-user-config -nodefaults -nographic \
        -machine q35,accel=kvm \
        -enable-kvm -cpu
host,kvm=off,hv_spinlocks=0x1fff,hv_relaxed,hv_time,hv_vendor_id=Nvidia43FIX
\
        -rtc base=localtime,clock=host,driftfix=slew \
        \
        -drive file=/vmstorage/firmware/ovmf-x64-0-20181105.757.g114a75ad86/OVMF-pure-efi.fd,if=pflash,format=raw,unit=0,readonly=on
\
        -drive file=/home/adam/win10-vars.fd,if=pflash,format=raw,readonly=off,unit=1
\
        \
        -object iothread,id=io1 \
        \
        -mem-path /dev/hugepages  \
        -m 24576 \
        -smp cores=8,threads=2,sockets=1 \
        \
        -netdev bridge,id=netdev0,br=br0 -device
virtio-net-pci,netdev=netdev0,mac=52:54:00:19:34:57 \
        -device
virtio-scsi-pci,id=scsi0,ioeventfd=on,iothread=io1,num_queues=4 \
        \
        -device vfio-pci,host=7:00.0,addr=0x6,multifunction=on \
        -device vfio-pci,host=7:00.1,addr=0x6.0x1 \
        -device vfio-pci,host=05:00.0 \
        -device vfio-pci,host=06:00.0


Regards,
Adam