[vfio-users] How to split iommu group on [xeon e3-1220 v5 + supermicro x11ssh-ln4f]

[Tran The Anh]

Dear Mr. Williamson,

Proxmox ve 4.1 has your ACS-override-patch included in the kernel.
When "pcie_acs_override=downstream" added to kernel parameter, dmesg 
says "PCIe ACS overrides enabled; ..."
However, iommu group 4 is still big.

I have patched the pve-kernel with your Skylake-PCH-patch.
It works now. IOMMU groups are separated. Each pcie device is in its own 
group.

Thank you very much.

On 4/5/2016 9:12 PM, Alex Williamson wrote:
> On Tue, Apr 5, 2016 at 3:13 AM, Tran The Anh 
> <anh.tranthe at thegioididong.com <mailto:anh.tranthe at thegioididong.com>> 
> wrote:
>
>     Hi,
>
>     I would like to passthrough some intel i210 ports to KVM's VM, but
>     has encountered the iommu group issue:
>
>         /sys/kernel/iommu_groups/0/devices/0000:00:00.0
>         /sys/kernel/iommu_groups/1/devices/0000:00:14.0 <= usb
>         /sys/kernel/iommu_groups/1/devices/0000:00:14.2
>         /sys/kernel/iommu_groups/2/devices/0000:00:16.0
>         /sys/kernel/iommu_groups/2/devices/0000:00:16.1
>         /sys/kernel/iommu_groups/3/devices/0000:00:17.0 <= sata
>         /sys/kernel/iommu_groups/4/devices/0000:00:1c.0 <= pch port
>         /sys/kernel/iommu_groups/4/devices/0000:00:1c.1 <= pch port
>         /sys/kernel/iommu_groups/4/devices/0000:00:1c.2 <= pch port
>         /sys/kernel/iommu_groups/4/devices/0000:00:1c.3 <= pch port
>         /sys/kernel/iommu_groups/4/devices/0000:00:1c.6 <= pch port
>         /sys/kernel/iommu_groups/4/devices/0000:01:00.0 <= i210
>         /sys/kernel/iommu_groups/4/devices/0000:02:00.0 <= i210
>         /sys/kernel/iommu_groups/4/devices/0000:03:00.0 <= i210
>         /sys/kernel/iommu_groups/4/devices/0000:04:00.0 <= i210
>         /sys/kernel/iommu_groups/4/devices/0000:05:00.0
>         /sys/kernel/iommu_groups/4/devices/0000:06:00.0 <= vga aspeed 2400
>         /sys/kernel/iommu_groups/5/devices/0000:00:1f.0
>         /sys/kernel/iommu_groups/5/devices/0000:00:1f.2
>         /sys/kernel/iommu_groups/5/devices/0000:00:1f.4
>
>
>     My system is:
>
>         Intel Xeon e3-1220 v5
>         Supermicro x11ssh-ln4f
>
>
>     According to board's diagram:
>     http://www.supermicro.com/manuals/motherboard/C236/MNL-1778.pdf
>     Those i210s are connected to PCH.
>
>     It seems that PCH ports have ACS capabilities:
>
>         lspci -vvvnk | egrep -i 'Access|Rev'
>         00:17.0 0106: 8086:a102 (rev 31) (prog-if 01 [AHCI 1.0])
>         00:1c.0 0604: 8086:a110 (rev f1) (prog-if 00 [Normal decode])
>                 Capabilities: [140 v1] Access Control Services
>         00:1c.1 0604: 8086:a111 (rev f1) (prog-if 00 [Normal decode])
>                 Capabilities: [140 v1] Access Control Services
>         00:1c.2 0604: 8086:a112 (rev f1) (prog-if 00 [Normal decode])
>                 Capabilities: [140 v1] Access Control Services
>         00:1c.3 0604: 8086:a113 (rev f1) (prog-if 00 [Normal decode])
>                 Capabilities: [140 v1] Access Control Services
>         00:1c.6 0604: 8086:a116 (rev f1) (prog-if 00 [Normal decode])
>                 Capabilities: [140 v1] Access Control Services
>
>     I have tried [proxmox ve 4.0, proxmox 4.1, centos 7, ubuntu 1404,
>     ubuntu 1510] + add "pci_acs_override=downstream" to kernel parameters.
>     But the iommu group 4 is always the same.
>
>
> The ACS override patch is not upstream, you'd actually need to run a 
> kernel with support for it for the kernel command line option to 
> work.  You have a Sunrise Point PCH where Intel has a broken 
> implementation of ACS.  I've posted quirks[1] for this, but they 
> probably won't be available until a v4.7 kernel.  Your choices are to 
> either patch your kernel with the ACS override patch or patch your 
> kernel with the quirks (preferred) until support is merged into the 
> kernel.  Thanks,
>
> Alex
>
> [1] https://lkml.org/lkml/2016/3/31/1112



---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20160407/e96f2fae/attachment.htm>