[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [vfio-users] MSI Interrupt balance in guests

So I’ve made a little more progress here-

Ubuntu 16 - doing the full upgrade with the newest kernel seemed to give me massive performance increase with multiple different configurations. While this has fixed a lot of my issues, it raises even further questions about interrupt management. Switching to the lowlatency kernel also seems to have  made a big difference- I believe the linux kernel is more able to keep up with the timer demands of the guest. But at this point I have 0 audio stutter, and latency jitters at +/- 5ms with a frequency of about 20Hz (which is better than I’ve seen on some bare metal streaming hosts!)

The other thing I did was force x2apic mode in windows 10: https://support.microsoft.com/en-us/kb/2303458

So this is where i get a lot of confusion… recent posts are saying to turn of hyper-v APIC because APICv on newer intel is faster (and AFAIK this is done through x2apic emulation). KVM does not seem to enable APICv for me, so I get more stuttering with hyper-v apic off. Now, where the magic really seems to come in…. forcing windows to use X2APIC AND enabling hyper-v gives me the best performance. From what I’ve read in the KVM code, qemu should be giving hints to Windows on what APIC to use. From what i understand, windows should really choose one or the other, but in my case it seems to be using both X2APIC and hyper-v APIC. i think hyper-v APIC is just fast EOI emulation, I can see via kvm_stat that I get paravirtual EOIs with it on, but not with it off. 

The other magic, is that with X2APIC, I magically don’t have to force MSI interrupts anymore. I get almost identical performance either way (probably some loss for shared edge interrupts with non-msi). So at this point I’m really confused how these different types are being emulated/passed-through. Without x2apic I seem to have no ability to use edge interrupts, but MSI is fine. With x2APIC it all seems… fine, and with really low overhead even though linux tells me I dont have APICv enabled. 

I posted another thread trying to clear up the difference between the interrupt types:

On Thu, Apr 28, 2016 at 10:43 PM Muted Bytes <mutedbytes gmail com> wrote:

It would be great to get information on all the points your brought up. Regarding your last few points, specifically

"Also, how does x2APIC relate to MSI and/or VFIO? I’m overriding my BIOS opt-out to force x2APIC. Is there any reason to not do this? Does this interfere with MSI at all? Why would my bios want to disable a much larger APIC? Can qemu/kvm even utilize this?"

what I have found so far, a lot of people recently have been suggesting to remove the hyperv vapic enlightenment so that guests can use APICv hardware support of more recent Intel CPUs. However, it seems this actually requires x2apic support in order for guest to actually use apicv. In my case, my motherboard also requires the optout to be overridden for x2apic to be enabled. Upon booting with this override option, I have observed a significant negative performance impact in the host. Browsers lag, and even ramspeed program shows 15% or worse decrease in ram throughput, among others.

On Mar 31, 2016 10:57 AM, "Colin Godsey" <crgodsey gmail com> wrote:
I had some questions about IRQ balancing with MSI interrupts from passthrough devices using VFIO.

I have a system that is designed to serve as a gaming streaming server for 2 windows VMs each using PCIe passthrough. So far the recurring problem has been interrupt management. This system ends up being a perfect storm (no pun indented) for interrupts, having to manage: high performance timers for video and audio, high data transfer from the GPU (more than normal because we’re passing 50mbps video stream from the GPU to the system), then 50mbps per VM transfer back to the host via virtio networking. I’m giving full CPU mapping to both VMs, with staggered affinity (VM0 VCPU0=CPU0, VM1 VCPU0=CPU2). It is a non-HT intel SMP CPU.

MSI so far seems to be the key to getting this all to work right. Enabling MSI was the first big thing, guests not detecting it led to really poor interrupt management, reading more about it explained why. 

The low latency kernel seems to be the other part of this- games like to run on an accurate clock, and context switching seems minimal when the GPU can drive frame composition like normal, and in the end i think it wastes less CPU cycles due to accurate media response. Even though this system deals with large amounts of throughput, latency and device-responsiveness are still mostly preferred. 

This leads to the last piece of the puzzle. My mysterious performance drop across the board when both systems are under GPU load. Both MSI GPUs seem to be piling the majority of their interrupts onto the same CPU. The MSI based ethernet adapter seems to fall on another CPU, so that’s great. But the GPU seems to just tank overall guest performance and the GPU interrupts dont seem to want to budge from CPU1 no matter what. There is some distribution of the GPU interrupts across CPUs, but the bulk seems to always land on CPU1.

Ultimately what im wondering:
* How does MSI balance? Via VFIO, are the MSI interrupts assigned/mapped/masked on a guest level, host level, or hardware level?
* Also, how does x2APIC relate to MSI and/or VFIO? I’m overriding my BIOS opt-out to force x2APIC. Is there any reason to not do this? Does this interfere with MSI at all? Why would my bios want to disable a much larger APIC? Can qemu/kvm even utilize this?
* Could my performance issues just be related to context-switching and CPU/cache bounds? Would the low latency kernel (1000Mhz) make that worse? If so, is there any way to force MSI affinity?

Thanks tons!

-Colin G

vfio-users mailing list
vfio-users redhat com

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]