[vfio-users] Lots of info for AMD Ryzen early adopters

Sun Mar 5 22:54:55 UTC 2017

I spend the last days trying to get as much info as I could about Ryzen from authoritative sources. Reelevant for us (Virtualization and Passthrough) are the followings:

1) GENERAL RYZEN SUPPORT IN LINUX

You need a really recent Linux Kernel to get Ryzen to work at all. Popular mainstream Linux distributions are plagued with problems because they aren't using the very latest. As things looks, I suppose that many distributions could end up mid term releases specifically for Ryzen support:

https://www.servethehome.com/amd-ryzen-with-ubuntu-here-is-what-you-have-to-do-to-fix-constant-crashes/
https://www.servethehome.com/stop-constant-centos-7-3-crashes-amd-ryzen-using-kernel-4-10/

2) AMD AVIC (AMD Virtual Interrupt Controller)

According to The Stilt (A guy extremely know in enthusiasts circles, with a lot of in-depth knowledge about AMD platforms), AMD AVIC will be supported by all Ryzen models. I suppose that this should also apply to the future Ryzen 3 and 5.

AVIC is AMD counterpart to Intel APICv, for APIC Virtualization. Intel provides this feature since Ivy Bridge-E, but only in the LGA 2011/2011-3 platform. HEDT Core i7s based on those platforms also include it, but no consumer LGA 1155/1150/1151 Processor has it (Including Xeons E3).

More technical info about AVIC here:

https://es.slideshare.net/xen_com_mgr/introduction-of-amd-virtual-interrupt-controller

AVIC should be supported since Linux Kernel 4.7:

http://www.phoronix.com/scan.php?page=news_item&px=KVM-AVIC-Linux-4.7

It may required to be manually enabled, but no idea if that is still the case. According to this:

https://lkml.org/lkml/2016/3/4/746

Currently, this patch series does not enable AVIC by default.
Users can enable SVM AVIC by specifying avic=1 during insmod kvm-amd.

Also worthy of mention regarding AVIC, is that with APICv, it was required to NOT enable the hv-vapic HyperV Enlightnement in QEMU to use it because it seems that VMs with Windows defaults to hv-vapic and ignores the APIC Virtualization support, thus in platforms with APICv it performed worse with hv-vapic enabled than without. Since I have absolutely no idea how AVIC is supposed to be enabled and put to work in QEMU (Besides the previously mentioned avic=1 for kvm_amd Kernel Module), I don't know if you need to do that or something else. I'm not even sure if you are required to disable/not enable hv-vapic.

3) AMD-Vi/IOMMU

The IOMMU seems to work out of the box.

I believe that you still have to manually enable it with a Kernel Parameter like amd_iommu=on, iommu=1, iommu=pt or something like that. No idea on specifics.

4) Devices FLR

Sadly, none of the interesing integrated devices (The two SATA Controllers, the two USB Controllers, and Azalia Audio) supports FLR, all shows FLReset- in lspci -vvv

5) IOMMU Groups/PCIe ACS

Patrick from ServeTheHome provided lspci -vvv, lspci-t and find /sys/kernel/iommu_groups/ outputs from his ASUS PRIME B350-PLUS:
https://forums.servethehome.com/index.php?threads/amd-ryzen-7-1700x-linux-benchmarks.13537/page-2#post-130063

https://www.asus.com/us/Motherboards/PRIME-B350-PLUS/

...so it was possible to me to build a composite PCIe Topology, also basing myself on the little info available of Motherboards Block Diagrams. The results are here:

http://imgur.com/a/36pAT

Don't panic, IOMMU Groups are ugly, but not as bad as it seems. So far, it seems that the topology works in a sort of Host Bridge/PCI Bridge pairing which are always grouped together, and there is isolation between those pairs (So at the very top level, PCIe ACS seems to be working). Where things go wrong, is that there is no isolation below them, so all the Devices attached to a PCI Bridge gets grouped together.

In that tree, the Video Card is currently sitting at the PCIe 16x Slot made with 4 PCIe 2.0 Lanes coming from the Chipset, so it is packed in the ugly Group 0 with all the Chipset devices. However, previously he had it in the main PCIe 16x Slot (Currently occupied with a Intel XL710 NIC, Group 2), so it should have been easily passthroughed. In a Motherboard with a X370 Chipset that allows bifurcation of the main 16 PCIe Lanes into 8x/8x, according to my calculations, it should create another PCI Bridge below one of the "unused" Host Bridges (Like 00:04.x) in its own exclusive Group, thus for two Video Cards in the Processor PCIe Slots you may have full isolation even at this early stage.

Since there is a lack of technical documentation at this stage, there is no info regarding if PCIe ACS should be a feature or not.

It could be possible that there is incomplete Firmware support, like this:

https://www.redhat.com/archives/vfio-users/2016-September/msg00070.html

Or that there is an errata, like in Skylake and Kaby Lake Chipsets.

If you want me to relay a request to someone with Ryzen Hardware, tell me exactly what they should type in case that you want a PCI Registers dump or whatever.

6) Passthrough

No confirmed successfully passthrough attempts at this point, but things are looking good for a single Video Card. It would also be interesing to know if the grouping unfriendly devices works with the ACS override patch/hack or is a dead end.

Overally, Ryzen has a lot of potential. It has AVIC and decent isolation between Processor PCIe Slots that consumer Intel platforms does not have. The Ryzen 7 1700 also allows you to play with 8 Cores at an affordable price, which was something that we were severely needing.

Ryzen AM4 platform interesing potential is in the fact that you have two USB Controllers and two SATA Controllers, one in the Processor and another in the Chipset, so with a Ryzen 7 1700 with 8 Cores and these integrated controllers, you could literally make a two person multiseat computer with minimum effort (X370 with two Video Cards, maybe a Sound Card and a third cheap host Video Card, and you're done). Sadly, current isolation means that you can't pass the integrated controllers. I really expect this to be fixeable, or I will be rather dissapointed.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20170305/9616b3ae/attachment.htm>