[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [vfio-users] fitlet I211 PCI passthrough



On Sat, Mar 26, 2016 at 5:28 AM, YAEGASHI Takeshi <yaegashi debian org> wrote:
Hello,

I have a fitlet http://www.fit-pc.com/web/products/fitlet/ with 4 GbE
(I211) ports and wanted to passthrough some of them to KVM guests, ie
assign one I211 to each guests.  I've tried various configurations
with libvirt/kvm but no luck so far.

After reading http://vfio.blogspot.jp/2014/08/iommu-groups-inside-and-out.html
I've got sure that there's a problem in IOMMU groups.  Actually all of
I211 and AMD's PCI bridge [1022:156b] share the same iommu_group 2.

$ lspci -tv
-[0000:00]-+-00.0  Advanced Micro Devices, Inc. [AMD] Device 1566
           +-00.2  Advanced Micro Devices, Inc. [AMD] Device 1567
           +-01.0  Advanced Micro Devices, Inc. [AMD/ATI] Mullins [Radeon R6 Graphics]
           +-01.1  Advanced Micro Devices, Inc. [AMD/ATI] Kabini HDMI/DP Audio
           +-02.0  Advanced Micro Devices, Inc. [AMD] Device 156b
           +-02.2-[01]----00.0  Intel Corporation I211 Gigabit Network Connection
           +-02.3-[02]----00.0  Intel Corporation I211 Gigabit Network Connection
           +-02.4-[03]----00.0  Intel Corporation I211 Gigabit Network Connection
           +-02.5-[04]----00.0  Intel Corporation I211 Gigabit Network Connection
           +-08.0  Advanced Micro Devices, Inc. [AMD] Device 1537
           +-10.0  Advanced Micro Devices, Inc. [AMD] FCH USB XHCI Controller
           +-11.0  Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode]
           +-12.0  Advanced Micro Devices, Inc. [AMD] FCH USB EHCI Controller
           +-13.0  Advanced Micro Devices, Inc. [AMD] FCH USB EHCI Controller
           +-14.0  Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller
           +-14.2  Advanced Micro Devices, Inc. [AMD] FCH Azalia Controller
           +-14.3  Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge
           +-14.7  Advanced Micro Devices, Inc. [AMD] FCH SD Flash Controller
           +-18.0  Advanced Micro Devices, Inc. [AMD] Device 1580
           +-18.1  Advanced Micro Devices, Inc. [AMD] Device 1581
           +-18.2  Advanced Micro Devices, Inc. [AMD] Device 1582
           +-18.3  Advanced Micro Devices, Inc. [AMD] Device 1583
           +-18.4  Advanced Micro Devices, Inc. [AMD] Device 1584
           \-18.5  Advanced Micro Devices, Inc. [AMD] Device 1585

$ find /sys/kernel/iommu_groups -type l
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.1
/sys/kernel/iommu_groups/2/devices/0000:00:02.0
/sys/kernel/iommu_groups/2/devices/0000:00:02.2
/sys/kernel/iommu_groups/2/devices/0000:00:02.3
/sys/kernel/iommu_groups/2/devices/0000:00:02.4
/sys/kernel/iommu_groups/2/devices/0000:00:02.5
/sys/kernel/iommu_groups/2/devices/0000:01:00.0
/sys/kernel/iommu_groups/2/devices/0000:02:00.0
/sys/kernel/iommu_groups/2/devices/0000:03:00.0
/sys/kernel/iommu_groups/2/devices/0000:04:00.0
/sys/kernel/iommu_groups/3/devices/0000:00:08.0
/sys/kernel/iommu_groups/4/devices/0000:00:10.0
/sys/kernel/iommu_groups/5/devices/0000:00:11.0
/sys/kernel/iommu_groups/6/devices/0000:00:12.0
/sys/kernel/iommu_groups/7/devices/0000:00:13.0
/sys/kernel/iommu_groups/8/devices/0000:00:14.0
/sys/kernel/iommu_groups/8/devices/0000:00:14.2
/sys/kernel/iommu_groups/8/devices/0000:00:14.3
/sys/kernel/iommu_groups/8/devices/0000:00:14.7
/sys/kernel/iommu_groups/9/devices/0000:00:18.0
/sys/kernel/iommu_groups/9/devices/0000:00:18.1
/sys/kernel/iommu_groups/9/devices/0000:00:18.2
/sys/kernel/iommu_groups/9/devices/0000:00:18.3
/sys/kernel/iommu_groups/9/devices/0000:00:18.4
/sys/kernel/iommu_groups/9/devices/0000:00:18.5

I'm running Ubuntu 14.04 with xenial kernel 4.4.0-13, libvirt 1.2.2,
qemu 2.0.0.  Using vfio-pci simply failed:

qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x3: vfio: error, group 2 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.
qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x3: vfio: failed to get group 2
qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x3: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x3: Device 'vfio-pci' could not be initialized

Using legacy KVM device assignment with <driver name='kvm'/> also
failed with unclear reason "Invalid argument":

qemu-system-x86_64: -device pci-assign,configfd=24,host=01:00.0,id=hostdev0,bus=pci.2,addr=0x2: Failed to assign device "hostdev0" : Invalid argument
qemu-system-x86_64: -device pci-assign,configfd=24,host=01:00.0,id=hostdev0,bus=pci.2,addr=0x2: Device initialization failed.
qemu-system-x86_64: -device pci-assign,configfd=24,host=01:00.0,id=hostdev0,bus=pci.2,addr=0x2: Device 'kvm-pci-assign' could not be initialized

After some googling I've learned that legacy KVM assignment also
refuses to work for this configuration since kernel 4.2
https://lkml.org/lkml/2015/11/12/661 .  With older kernel 3.19 this
configuration worked as expected.

Is there any chance for me to achieve single I211 passthrough with
recent kernels?  Because the hardware has no flexibility on the bus
topologies, the only possible way would be patching the kernel (ACS
override or PCI quirks).  Is it safe?  Does anyone have any info on
the IOMMU capability of device/bridge/chipset in question?

The grouping occurs the way it does because 00:02.* is a multifunction device.  Without native hardware support for ACS on those functions, we must assume that routing within the multifunction device can result in non-IOMMU translated peer-to-peer DMA between the downstream endpoints.  You can use the ACS override patch, but the reason that patch is not upstream is because it is potentially unsafe, you may be declaring isolation where none really exists and the VMs could interact with each other in unexpected ways.  The only real solution to enabling assignment of these devices is to work with AMD to determine whether this internal re-routing is possible and if not, add quirks to the kernel to expose the devices as isolated.  Thanks,

Alex

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]