[vfio-users] Intel x2APIC and APIC Virtualization (APICv or vAPIC)

[Zir Blazer]

With Ivy Bridge-E, Intel introduced their APIC Virtualization extension, APICv (Rather contradictory, on other sources its called vAPIC). APICv seems to be built on top of x2APIC, so it requires it to be working on the host to be used. You can check that in the host by using:
dmesg | grep x2apic
If its returns something like "x2apic enabled", its working. This is something that users should always check, since it may be possible that the Firmware tells the OS to NOT use x2APIC.
x2APIC was introduced with Nehalem to supercede the older xAPIC, introduced with the Pentium 4. Sadly, the initial Intel Tylersburg Chipsets (X58, 5500 and 5520) for LGA 1366 Nehalems had broken Interrupt Remapping support, which x2APIC seems to rely on. With IR broken, x2APIC didn't worked properly, and there could be crashes, freezes and such. The solution was to fall back to older xAPIC.Around that timeframe, instead of blacklisting x2APIC on those Chipsets OS side with a hotfix/patch so it doesn't defaults to it when autodetecting platform features, or even better, allow the Firmware to hide the x2apic CPU Flag which is supposed to inform about x2APIC support, a genius decided to introduce to the Intel VT-d specifications the "x2APIC Opt Out Bit", which is a new Bit to tell the OS if the Firmware wants x2APIC to be enabled or not. The weird part of it is that this Bit resides in the DMAR ACPI Table, so it seems to require VT-d enabled, since it generates the DMAR ACPI Table. This sounds like a bad idea, since you STILL have to add OS support for detecting the Bit instead of merely blacklisting the x2APIC on those Chipsets, along with a Firmware upgrade for adding it. Not only that, VT-d must be enabled to create the ACPI DMAR Table, so that Bit is unavailable if the IOMMU (Which also relies on IR, and I think than that is why the enable_unsafe_interrupts exist in VFIO, a workaround specifically for that. Or it was another one?) is off. Basically, the whole thing is a hack and looks like an epic mess.Point is, in my Haswell Motherboard (A Supermicro X10SAT), a Firmware upgrade introduced an X2APIC Opt Out Bit option that comes enabled by default. Linux says "DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit" if VT-d is enabled along with this Bit. So, x2APIC may not work out of the box even in non-broken platforms because the Firmware may still suggest the OS to not use x2APIC (As a last resort, it can also be overrided by a Kernel Parameter to force x2APIC if you fail to figure out what to do Firmware side).
VMs also supports an emulated form of x2APIC, which performs better than the previous alternative (Very old APIC emulation?? Not sure). Actually, you do not need host or Processor support for x2APIC, and using it in VMs gives pretty much free performance - so much that a QEMU dev decided to include the x2APIC CPU Flag in nearly ALL Processors models (-cpu whatever), so you can use in QEMU -cpu core2duo and it will by default expose the emulated x2APIC via the CPU Flag even through that platform didn't had it. You can check in a Linux VM if x2APIC exist as CPU Flag using...
grep x2apic /proc/cpuinfo

Back to APICv. Some comments I saw in recent months states that the HyperV hv-vapic extension should NOT be enabled if APICv is available since it has less overhead, so it performs better. I'm fine with that. The issue is that there is scarse info regarding what platforms actually supports APICv. Many people said that Haswell supports it, but I failed to enable it even with x2APIC working properly (Host side, the emulated x2APIC for VMs is another thing), and found no solid evidence that Haswell supports APICv at all.
The kvm_intel Kernel Module has the enable_apicv parameter. You can check its status by using...
cat /sys/module/kvm_intel/parameters/enable_apicv
It should return either a N or Y. In my platform (With a Xeon E3-1245V3 Haswell), by default its N.
The kvm_intel Kernel Module has another rather know parameter which is nested. You can check its status by using...
cat /sys/module/kvm_intel/parameters/nested
It should return either a N or Y. In my platform, by default its also N.
I can unload kvm_intel, then load it again with appropiated parameters...
modprobe -r kvm_intelmodprobe kvm_intel nested=y
This works as expected, checking nested returns a Y. This is well know, and a requeriment to enable Nested Virtualization on Intel platforms. But, if in the same way, I do this...
modprobe -r kvm_intelmodprobe kvm_intel enable_apicv=y
...It has no effect, as checking that parameter still returns a N. Basically, it seems to point that APICv refuses to enable (Didn't tried as a Kernel Parameter in Boot Loader, just to make sure it doesn't need to be enabled at boot time). It may not work probabily because there is no Hardware support for it...
After a lot of googling, I found absolutely NO info about consumer Haswell supporting APICv, which is something that other people stated (I think that on Reddit). What I did found, besides the Ivy Bridge-E announcement, are reviews mentioning that Haswell-E and Broadwell-E also supports APICv. So, it seems that APICv is a feature that its exclusive to the enterprise platform (A la PCIe ACS), since I didn't saw it mentioned for any consumer platform (LGA 1150 Haswell, Broadwell, and LGA 1151 Skylake) at all.

Basically, if you have a Haswell, Broadwell or Skylake consumer platform, you may want to check for x2APIC status by using...
dmesg | grep x2apic
And if its enabled, you can try checking APICv status by using...
cat /sys/module/kvm_intel/parameters/enable_apicv
Should its possible to determine that consumer platforms doesn't supports APICv at all, I suppose that you may want to add again hv-vapic for your Windows VMs if your removed it thinking that your platform supports APICv. I suppose that the performance scale for VMs goes like this: Emulated x2APIC -> HyperV hv-vapic (For Windows VMs) -> Intel APICv (On Intel enterprise platforms). 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20160614/4b124c67/attachment.htm>