[vfio-users] IOMMU group question - different pcie slots gives different configuration

Mon May 9 16:30:20 UTC 2016

Alex,

I am running the Arch VFIO kernel from aur and the ACS patch is working for my Skylake system.  Will I not need the ACS patch with 4.7 or will I get a performance increase? 

On May 9, 2016 9:25:09 AM PDT, Alex Williamson <alex.l.williamson at gmail.com> wrote:
>On Mon, May 9, 2016 at 10:06 AM, Jonas Camillus Jeppesen
><jonascj at sdu.dk>
>wrote:
>
>> My IOMMU grouping of devices change depending on which pci-e socket I
>> insert my R9 290 GPU into. For the sake of purchasing a new system I
>wanted
>> to discuss the different groupings so I can better choose my new
>hardware.
>>
>> Here are the IOMMU group which contain my GPU with the GPU inserted
>into
>> the three different PCIE slots I have (for all groups in the
>different
>> configs see attachments).
>>
>> #A: GPU in PCIE1
>> IOMMU group 1
>>     00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th
>Gen
>> Core Processor PCI Express x16 Controller [8086:0c01] (rev 06)
>>     01:00.0 VGA compatible controller [0300]: Advanced Micro Devices,
>Inc.
>> [AMD/ATI] Hawaii PRO [Radeon R9 290] [1002:67b1]
>>     01:00.1 Audio device [0403]: Advanced Micro Devices, Inc.
>[AMD/ATI]
>> Hawaii HDMI Audio [1002:aac8]
>>
>> #B:  GPU in PCIE3
>> IOMMU group 1
>>     00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th
>Gen
>> Core Processor PCI Express x16 Controller [8086:0c01] (rev 06)
>>     00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th
>Gen
>> Core Processor PCI Express x8 Controller [8086:0c05] (rev 06)
>>     02:00.0 VGA compatible controller [0300]: Advanced Micro Devices,
>Inc.
>> [AMD/ATI] Hawaii PRO [Radeon R9 290] [1002:67b1]
>>     02:00.1 Audio device [0403]: Advanced Micro Devices, Inc.
>[AMD/ATI]
>> Hawaii HDMI Audio [1002:aac8]
>>
>> #C:  GPU in PCIE4
>> IOMMU group 14
>>     04:00.0 VGA compatible controller [0300]: Advanced Micro Devices,
>Inc.
>> [AMD/ATI] Hawaii PRO [Radeon R9 290] [1002:67b1]
>>     04:00.1 Audio device [0403]: Advanced Micro Devices, Inc.
>[AMD/ATI]
>> Hawaii HDMI Audio [1002:aac8]
>>
>> I have had success with Arch linux and stock kernel 4.4.1 with config
>#C
>> (GPU in PCIE4), but not the others. Is that to be expected without
>patching
>> the kernel  (i.e. the GPU needs to be in a group all by itself)?
>>
>> Can these groups be figured out without plugging the GPU into the
>> different slots and looking at /sys/kernel/iommu_groups/; deduced
>from the
>> chip set specification, inspecting /sys/ in more clever way, or
>similar? If
>> yes, then it would be great to collect a list of hardware and its
>> groupings. That way it would be easier to decide which motherboard
>and cpu
>> to get for different setups.
>>
>
>It's very simple, unless you choose a Xeon E5+ or a High End Desktop
>Processor, then all of the processor based root ports will be grouped
>together.  The difference between #A and #B in your example is simply
>that
>the BIOS disables the 00:01.1 root port if there's nothing installed in
>the
>slot (it can't disable 00:01.0 in configuration #B because function #0
>must
>be present in order to discover function #1).  In example #C you've
>installed the card in a PCH-based root port for which we have quirks in
>the
>kernel to enable isolation.  We have quirks for most Intel chipsets to
>enable this:
>
>http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/quirks.c#n3877
>
>Linux v4.7 should have quirks for Skylake PCH root ports:
>
>https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/commit/drivers/pci/quirks.c?id=1bf2bf229b64540f91ac6fa3af37c81249037a0b
>
>Often the more high-end motherboard manufacturers will have block
>diagrams
>of the system layout so you can see which slots are hosted by which
>device.  IIRC supermicro typically does this, not that I'm endorsing
>their
>products.
>
>All of the above configurations will work, #A and #B simply add
>restrictions that you can't use the additional CPU-based slots for
>anything
>else without (not recommended) using the ACS override patch.  For an
>assigned GPU, it does not matter that the root port itself is included
>in
>the group.  If however you were trying to make use of a multi-port NIC,
>with each port assigned to a separate VM, or an SR-IOV device with
>multiple
>virtual functions, installing it in such a slot would be a very bad
>choice
>for those applications.
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>vfio-users mailing list
>vfio-users at redhat.com
>https://www.redhat.com/mailman/listinfo/vfio-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20160509/923d8f8d/attachment.htm>