[vfio-users] Awful boot times, current OVMF to blame?

Tue Jul 25 08:48:49 UTC 2017

This sounds really a lot like that old kernel MTRR problem I described in my previous post, though it resulted in boot times on the order of tens of minutes. Which kernel version do you have?

Here is the information you requested:

Yes, I'm using libvirt. The set of qemu command-line arguments it produces is as follows:

-name guest=win10,debug-threads=on
-S
-object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-3-win10/master-key.aes
-machine pc-i440fx-2.5,accel=kvm,usb=off,vmport=off,dump-guest-core=off
-cpu host,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,hv_vendor_id=KeenlyKVM,kvm=off
-drive file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd,if=pflash,format=raw,unit=0,readonly=on
-drive file=/var/lib/libvirt/qemu/nvram/win10_VARS.fd,if=pflash,format=raw,unit=1
-m 12288
-mem-prealloc
-mem-path /dev/hugepages/libvirt/qemu/3-win10
-realtime mlock=off
-smp 6,sockets=1,cores=3,threads=2
-object iothread,id=iothread1
-object iothread,id=iothread2
-uuid somelong-uuid-here
-display none
-no-user-config
-nodefaults
-chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-3-win10/monitor.sock,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control
-rtc base=localtime,driftfix=slew
-global kvm-pit.lost_tick_policy=delay
-no-hpet
-no-shutdown
-global PIIX4_PM.disable_s3=1
-global PIIX4_PM.disable_s4=1
-boot strict=on
-device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7
-device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2
-drive file=/dev/sdc,format=raw,if=none,id=drive-virtio-disk0,cache=none,aio=native
-device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-netdev tap,fd=24,id=hostnet0,vhost=on,vhostfd=26
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:11:22:33:44:55,bus=pci.0,addr=0x3
-device vfio-pci,host=05:00.0,id=hostdev0,bus=pci.0,addr=0x4,rombar=1
-device vfio-pci,host=05:00.1,id=hostdev1,bus=pci.0,addr=0x5,rombar=1
-device vfio-pci,host=00:1b.0,id=hostdev2,bus=pci.0,addr=0x2
-device usb-host,hostbus=1,hostaddr=5,id=hostdev3,bus=usb.0,port=1
-device usb-host,hostbus=1,hostaddr=6,id=hostdev4,bus=usb.0,port=2
-device usb-host,hostbus=1,hostaddr=7,id=hostdev5,bus=usb.0,port=3
-msg timestamp=on

Besides the rombar=1 for the passed-through GPU functions and mlock=off in the realtime parameter, I don't see much difference here.

The kernel boot arguments are similar to yours:

intel_iommu=on iommu=pt
hugepagesz=1G default_hugepagesz=1G hugepages=12
isolcpus=3,4,5,9,10,11 nohz_full=3,4,5,9,10,11 rcu_nocbs=3,4,5,9,10,11
pci-stub.ids=10de:13c2,10de:0fbb,8086:8d20
nvidia-drm.modeset=1

8086:8d20 is the on-board audio controller at 00:1b.0.

Hope that helps.

Cheers,
Hristo

25. Juli 2017 02:22, "John Koelndorfer"  schrieb:
An update:
 The slow boot does not occur if I remove host devices from the virtual machine (without any other configuration changes). It doesn't matter whether GPU, USB, or both are passed in. Any device being passed in at all triggers the problem, so it's something related to the physical device passthrough I think.
An older version of OVMF did not help, nor did booting a non-Windows OS. I tried two alternate versions of OVMF from here: https://www.rpmfind.net/linux/rpm2html/search.php?query=edk2-ovmf (https://www.rpmfind.net/linux/rpm2html/search.php?query=edk2-ovmf). One from 2016 November, one from 2016 April. I initially did my last VFIO setup in 2016 July.
I also tried the pc-i440fx machine, per Hristo's suggestion (though I tried 2.6, which was what my last configuration used).
Hristo, it seems your setup is very similar to mine, so I have a few questions for you:

* Are you using libvirt?
* Could you send the qemu arguments for your VM? 
* What are your kernel boot parameters? 
My suspicion now is that libvirt is doing some extra configuration that I'm not. I looked at my old libvirt XML file and nothing is jumping out at me, so maybe it's a hardcoded default behavior.
Thanks for all the suggestions so far, folks.  
On Mon, Jul 24, 2017 at 4:03 PM, Hagbard Celine  wrote:

 Hi, just registered to the list to share my experience on this;

I've been getting my OVMF builds from kraxel.org (http://kraxel.org) since
edk2.git-ovmf-x64-0-20150804.b1143.g8ca1489.noarch.rpm and was
regularly updating when new builds came.
Somwhere around edk2.git-ovmf-x64-0-20151221.b1390.g5ba9f06.noarch.rpm
boot with large amounts of memory got slower.
And around edk2.git-ovmf-x64-0-20160324.b1634.g3decba3.noarch.rpm the
EFI part of boot could last about 15min with 16GB assigned to VM out
of a total of 32GB. Lovering the assigned memory for VM to below 8GB
resulted in normal boot times.

PS. The version numbers are approximations, my recollection is not exact.
_______________________________________________
vfio-users mailing list
vfio-users at redhat.com (mailto:vfio-users at redhat.com)
https://www.redhat.com/mailman/listinfo/vfio-users (https://www.redhat.com/mailman/listinfo/vfio-users)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20170725/5ae9976c/attachment.htm>