[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [vfio-users] Mobo reccommendations/suggestions ?



On 18/06/16 16:42, Alex Williamson wrote:
On Sat, Jun 18, 2016 at 9:11 AM, Samuel Holland <samuel sholland org> wrote:
On 06/17/2016 02:00 PM, Kajim Shanti wrote:
If i decide to replace the mobo, which one would you reccommend as
giving  the least problems for devices passthrough?

ASRock generally has a good reputation for not breaking VT-d. Be aware
that any motherboard you buy with multiple PCIe x16 ports will have them
all in the same IOMMU group.

This is why the continuous recommendations for High End Desktop Processors and Xeon E5, where this is not the case.  Just be aware that standard Core-i3/5/7 are going to group all the processor root ports together, which either limits your configurations or requires the acs override patch.  The latter, I don't consider supportable.  There are plenty of usable configurations available for Core processors, but for more than one discrete GPU, you're going to need to use PCH-based root ports.


_______________________________________________
vfio-users mailing list
vfio-users redhat com
https://www.redhat.com/mailman/listinfo/vfio-users

I'd actually vote for cheaper variants of skylake hardware for consumer use as you aren't likely to get the x16 port grouping and there won't be lots of extra controllers with hardwired groups. The grouping problem for the x16 ports goes away because these cheaper boards don't offer sharing of the 16 CPU lanes but instead sub-divide the lanes between separate bridges as you add cards (i.e. 1 x16 card or 2 x8, etc.). The intel spec. says the host port will offer full isolation if there is no inter-device data sharing that doesn't go through the host port which your root bridges can either guarantee for you .. or in the cheaper case which you get for free by not letting them share lanes and grouping exceptions manually. For normal hardware this usually means you didn't plug the same device into multiple ports - or connect devices with a cable. ACS does offer optional services to cope even with those cases but simpler boards will tend to have isolation by that rule and Mine even has ACS registers that explicitly say it's isolated and give some optional services I've no idea about.

The exception is if you are running multiple Gen2 legacy devices that need x16 to hit their maximum throughput or naturally for SLI which is the only consumer use case I can think of that can hope to even saturate a single x8 Gen 3 port. SLI devices are clearly in the same IOMMU group so unless that's your intent don't buy the Z77 type boards.

I have little trouble from running two Vms with a different radeon card passed to each and the IGD on the host using no ACS isolation patches and qemu 2.6 / 4.7 kernel as well as a couple of minimal server VMs that run 24/7 but are low load. The board is an ASUS Z170 Pro (I forget which variant) with an i5 6600k with the IGD running on the host but no X. This is all firmly in the upper lower end of consumer grade territory and is only consistent with minimal VM use but it's great if you don't need scalability or high availability under all workloads. I don't see many crashes outside the Win10 VM and those never take other VMs with them. The IGD crashes the host a lot if you use it for accelerated X - which is why I don't. I'd turn it off if I could use a less cutting edge kernel.

For reference .. these are my IOMMU groupings and the relevant devices. If you want more specific information I can provide working examples from my hardware. I suspect it may vary a lot between the cheaper boards as only the first PEG port is in any way standardised.

IOMMU / Device
0    00:00.0 Host bridge [0600]: Intel Corporation Skylake Host Bridge/DRAM Registers [8086:191f] (rev 07)
1    00:01.0 PCI bridge [0604]: Intel Corporation Skylake PCIe Controller (x16) [8086:1901] (rev 07)
2    00:02.0 VGA compatible controller [0300]: Intel Corporation HD Graphics 530 [8086:1912] (rev 06)
3    00:14.0 USB controller [0c03]: Intel Corporation Sunrise Point-H USB 3.0 xHCI Controller [8086:a12f] (rev 31)
4    00:16.0 Communication controller [0780]: Intel Corporation Sunrise Point-H CSME HECI #1 [8086:a13a] (rev 31)
5    00:17.0 SATA controller [0106]: Intel Corporation Sunrise Point-H SATA controller [AHCI mode] [8086:a102] (rev 31)
6    00:1b.0 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Root Port #17 [8086:a167] (rev f1)
7    00:1c.0 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Express Root Port #1 [8086:a110] (rev f1)
8    00:1c.4 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Express Root Port #5 [8086:a114] (rev f1)
9    00:1d.0 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Express Root Port #9 [8086:a118] (rev f1)
10  00:1f.0 ISA bridge [0601]: Intel Corporation Sunrise Point-H LPC Controller [8086:a145] (rev 31)
10  00:1f.2 Memory controller [0580]: Intel Corporation Sunrise Point-H PMC [8086:a121] (rev 31)
10  00:1f.3 Audio device [0403]: Intel Corporation Sunrise Point-H HD Audio [8086:a170] (rev 31)
10  00:1f.4 SMBus [0c05]: Intel Corporation Sunrise Point-H SMBus [8086:a123] (rev 31)
10  00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (2) I219-V [8086:15b8] (rev 31)
1    01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970/8970 OEM / R9 280X] [1002:6798]
1    01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series] [1002:aaa0]
12  03:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller [1b21:1242]
13  04:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cayman XT [Radeon HD 6970] [1002:6718]
13  04:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cayman/Antilles HDMI Audio [Radeon HD 6900 Series] [1002:aa80]



[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]