[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [vfio-users] MSI Interrupt balance in guests



It would be great to get information on all the points your brought up. Regarding your last few points, specifically

"Also, how does x2APIC relate to MSI and/or VFIO? I’m overriding my BIOS opt-out to force x2APIC. Is there any reason to not do this? Does this interfere with MSI at all? Why would my bios want to disable a much larger APIC? Can qemu/kvm even utilize this?"

what I have found so far, a lot of people recently have been suggesting to remove the hyperv vapic enlightenment so that guests can use APICv hardware support of more recent Intel CPUs. However, it seems this actually requires x2apic support in order for guest to actually use apicv. In my case, my motherboard also requires the optout to be overridden for x2apic to be enabled. Upon booting with this override option, I have observed a significant negative performance impact in the host. Browsers lag, and even ramspeed program shows 15% or worse decrease in ram throughput, among others.

On Mar 31, 2016 10:57 AM, "Colin Godsey" <crgodsey gmail com> wrote:
I had some questions about IRQ balancing with MSI interrupts from passthrough devices using VFIO.

I have a system that is designed to serve as a gaming streaming server for 2 windows VMs each using PCIe passthrough. So far the recurring problem has been interrupt management. This system ends up being a perfect storm (no pun indented) for interrupts, having to manage: high performance timers for video and audio, high data transfer from the GPU (more than normal because we’re passing 50mbps video stream from the GPU to the system), then 50mbps per VM transfer back to the host via virtio networking. I’m giving full CPU mapping to both VMs, with staggered affinity (VM0 VCPU0=CPU0, VM1 VCPU0=CPU2). It is a non-HT intel SMP CPU.

MSI so far seems to be the key to getting this all to work right. Enabling MSI was the first big thing, guests not detecting it led to really poor interrupt management, reading more about it explained why. 

The low latency kernel seems to be the other part of this- games like to run on an accurate clock, and context switching seems minimal when the GPU can drive frame composition like normal, and in the end i think it wastes less CPU cycles due to accurate media response. Even though this system deals with large amounts of throughput, latency and device-responsiveness are still mostly preferred. 

This leads to the last piece of the puzzle. My mysterious performance drop across the board when both systems are under GPU load. Both MSI GPUs seem to be piling the majority of their interrupts onto the same CPU. The MSI based ethernet adapter seems to fall on another CPU, so that’s great. But the GPU seems to just tank overall guest performance and the GPU interrupts dont seem to want to budge from CPU1 no matter what. There is some distribution of the GPU interrupts across CPUs, but the bulk seems to always land on CPU1.

Ultimately what im wondering:
* How does MSI balance? Via VFIO, are the MSI interrupts assigned/mapped/masked on a guest level, host level, or hardware level?
* Also, how does x2APIC relate to MSI and/or VFIO? I’m overriding my BIOS opt-out to force x2APIC. Is there any reason to not do this? Does this interfere with MSI at all? Why would my bios want to disable a much larger APIC? Can qemu/kvm even utilize this?
* Could my performance issues just be related to context-switching and CPU/cache bounds? Would the low latency kernel (1000Mhz) make that worse? If so, is there any way to force MSI affinity?

Thanks tons!

-Colin G

_______________________________________________
vfio-users mailing list
vfio-users redhat com
https://www.redhat.com/mailman/listinfo/vfio-users


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]