[vfio-users] IOMMU group question - different pcie slots gives different configuration

[Alex Williamson]

On Mon, May 9, 2016 at 10:06 AM, Jonas Camillus Jeppesen <jonascj at sdu.dk>
wrote:

> My IOMMU grouping of devices change depending on which pci-e socket I
> insert my R9 290 GPU into. For the sake of purchasing a new system I wanted
> to discuss the different groupings so I can better choose my new hardware.
>
> Here are the IOMMU group which contain my GPU with the GPU inserted into
> the three different PCIE slots I have (for all groups in the different
> configs see attachments).
>
> #A: GPU in PCIE1
> IOMMU group 1
>     00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen
> Core Processor PCI Express x16 Controller [8086:0c01] (rev 06)
>     01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc.
> [AMD/ATI] Hawaii PRO [Radeon R9 290] [1002:67b1]
>     01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI]
> Hawaii HDMI Audio [1002:aac8]
>
> #B:  GPU in PCIE3
> IOMMU group 1
>     00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen
> Core Processor PCI Express x16 Controller [8086:0c01] (rev 06)
>     00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen
> Core Processor PCI Express x8 Controller [8086:0c05] (rev 06)
>     02:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc.
> [AMD/ATI] Hawaii PRO [Radeon R9 290] [1002:67b1]
>     02:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI]
> Hawaii HDMI Audio [1002:aac8]
>
> #C:  GPU in PCIE4
> IOMMU group 14
>     04:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc.
> [AMD/ATI] Hawaii PRO [Radeon R9 290] [1002:67b1]
>     04:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI]
> Hawaii HDMI Audio [1002:aac8]
>
> I have had success with Arch linux and stock kernel 4.4.1 with config #C
> (GPU in PCIE4), but not the others. Is that to be expected without patching
> the kernel  (i.e. the GPU needs to be in a group all by itself)?
>
> Can these groups be figured out without plugging the GPU into the
> different slots and looking at /sys/kernel/iommu_groups/; deduced from the
> chip set specification, inspecting /sys/ in more clever way, or similar? If
> yes, then it would be great to collect a list of hardware and its
> groupings. That way it would be easier to decide which motherboard and cpu
> to get for different setups.
>

It's very simple, unless you choose a Xeon E5+ or a High End Desktop
Processor, then all of the processor based root ports will be grouped
together.  The difference between #A and #B in your example is simply that
the BIOS disables the 00:01.1 root port if there's nothing installed in the
slot (it can't disable 00:01.0 in configuration #B because function #0 must
be present in order to discover function #1).  In example #C you've
installed the card in a PCH-based root port for which we have quirks in the
kernel to enable isolation.  We have quirks for most Intel chipsets to
enable this:

http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/quirks.c#n3877

Linux v4.7 should have quirks for Skylake PCH root ports:

https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/commit/drivers/pci/quirks.c?id=1bf2bf229b64540f91ac6fa3af37c81249037a0b

Often the more high-end motherboard manufacturers will have block diagrams
of the system layout so you can see which slots are hosted by which
device.  IIRC supermicro typically does this, not that I'm endorsing their
products.

All of the above configurations will work, #A and #B simply add
restrictions that you can't use the additional CPU-based slots for anything
else without (not recommended) using the ACS override patch.  For an
assigned GPU, it does not matter that the root port itself is included in
the group.  If however you were trying to make use of a multi-port NIC,
with each port assigned to a separate VM, or an SR-IOV device with multiple
virtual functions, installing it in such a slot would be a very bad choice
for those applications.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20160509/fbc82587/attachment.htm>