[vfio-users] problem with passthrough and unsafe interrupts

Sat Sep 12 17:57:19 UTC 2015

On Sat, Sep 12, 2015 at 11:04 AM, Janusz <januszmk6 at gmail.com> wrote:

> Hello,
>
> I have a question about allowing unsafe interrupts, what exactly is
> happening when we allow it? why its unsafe?
>

On a standard PC, Message Signaled Interrupts (MSI) are triggered via a DMA
write by the device to a special address range.  In the early revisions of
VT-d, the IOMMU did not protect this range, which allowed devices to
effectively spoof interrupts from other devices and potentially run attacks
against the host using DMA writes to this interrupt block.  Later versions
of VT-d introduced the interrupt remapping feature which, among other
things, protects this range so that a device can only signal the interrupts
programmed for it.

Another thing that interrupt remapping does is provide a translation
between IOAPIC and X2APIC interrupt domains on the system.  Pretty much all
new Intel processors support X2APIC, which necessitates interrupt remapping
support.  So, I'd be really surprised if your new skylake system doesn't
have interrupt remapping.  You should really only need to enable the unsafe
interrupts option if you try to assign the device, it fails, and dmesg
tells you that you need it.  And only then if you trust your guests not to
be malicious to the host.

> I am asking because I am not able to passthrough my gpu without allowing
> unsafe interrupts, and have problems in running my virtual machine as it
> sometimes rebooting its self before its able to run windows. I know that
> there is also problem with using iGPU and not uefi bios, but its also
> happening with OVMF and passing uefi rom for my gpu (except that with
> OVMF sometimes I also get very high load). without passing VGA,
> everyting works fine. Is it possible that its because those unsafe
> interrupts?
> My hardware: i7 6700k, MSI z170a M7, Sapphire R9 290
>

Congratulations, you're the first person to show up with a Skylake system,
welcome to trailblazing a new Intel platform.  When you say you're not able
to assign the GPU without the unsafe interrupts option, does that mean when
you run the VM QEMU exits and dmesg reports:

"No interrupt remapping support.  Use the module param
"allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform"

That would be pretty epic if Intel dropped interrupt remapping support on
Skylake.  Or perhaps you just mean that you have something closer to
working with unsafe interrupts.  If the hardware supports interrupt
remapping, then the unsafe option makes absolutely no difference.  It's
only meant to allow an otherwise un-allowed scenario by making the user
opt-in to the security risk.

those are my options for non-uefi:
> -enable-kvm -m 10000 -cpu host -smp 8,cores=4,threads=2,sockets=1
> -device
> ioh3420,bus=pci.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1
> -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on
> -device vfio-pci,host=01:00.1
>

Looks like you're adding an ioh3420 for absolutely no reason.  Same with
the multifunction option.

> and for uefi:
>
> -drive if=pflash,format=raw,readonly,file=/home/janusz/uefi/OVMF_CODE.fd
> -drive if=pflash,format=raw,file=/home/janusz/uefi/OVMF_VARS.fd
> -enable-kvm -m 10000 -cpu host -smp 8,cores=4,threads=2,sockets=1
> -device
>
> vfio-pci,host=01:00.0,romfile=/home/janusz/uefi/uefi-vga.bin,multifunction=on
> -device vfio-pci,host=01:00.1
>

Useless multifunction option again.  Why does this one specify a ROM file?
Does the on-card ROM not support UEFI?

> + options with assigning /dev/sdb and two usb devices
>
> I am using ovmf build recently from master, kernel 4.2.0 and qemu-2.4.50
>
>
> My iommu groups:
>
> /sys/kernel/iommu_groups/0/devices/0000:00:00.0
>

> /sys/kernel/iommu_groups/1/devices/0000:00:01.0
> /sys/kernel/iommu_groups/1/devices/0000:01:00.0
> /sys/kernel/iommu_groups/1/devices/0000:01:00.1
>

So this ought to be the group we care about.

> /sys/kernel/iommu_groups/2/devices/0000:00:02.0
> /sys/kernel/iommu_groups/3/devices/0000:00:08.0
> /sys/kernel/iommu_groups/4/devices/0000:00:14.0
> /sys/kernel/iommu_groups/4/devices/0000:00:14.2
> /sys/kernel/iommu_groups/5/devices/0000:00:15.0
> /sys/kernel/iommu_groups/5/devices/0000:00:15.1
> /sys/kernel/iommu_groups/6/devices/0000:00:16.0
> /sys/kernel/iommu_groups/7/devices/0000:00:17.0
>

> /sys/kernel/iommu_groups/8/devices/0000:00:1c.0
> /sys/kernel/iommu_groups/8/devices/0000:00:1c.2
> /sys/kernel/iommu_groups/8/devices/0000:00:1c.7
> /sys/kernel/iommu_groups/8/devices/0000:02:00.0
> /sys/kernel/iommu_groups/8/devices/0000:03:00.0
> /sys/kernel/iommu_groups/8/devices/0000:04:00.0
>

Lovely, all the PCH root ports are grouped together, people looking for
hardware please take note.

> /sys/kernel/iommu_groups/9/devices/0000:00:1e.0
> /sys/kernel/iommu_groups/10/devices/0000:00:1f.0
> /sys/kernel/iommu_groups/10/devices/0000:00:1f.2
> /sys/kernel/iommu_groups/10/devices/0000:00:1f.3
> /sys/kernel/iommu_groups/10/devices/0000:00:1f.4
>
> and lspci:
>
> 00:00.0 Host bridge: Intel Corporation Sky Lake Host Bridge/DRAM
> Registers (rev 07)
>

> 00:01.0 PCI bridge: Intel Corporation Sky Lake PCIe Controller (x16)
> (rev 07)
>

> 00:02.0 VGA compatible controller: Intel Corporation Sky Lake Integrated
> Graphics (rev 06)
>

> 00:08.0 System peripheral: Intel Corporation Sky Lake Gaussian Mixture
> Model
>

> 00:14.0 USB controller: Intel Corporation Sunrise Point-H USB 3.0 xHCI
> Controller (rev 31)
> 00:14.2 Signal processing controller: Intel Corporation Sunrise Point-H
> Thermal subsystem (rev 31)
>

Don't you like how they've put the USB3 controller on a multifunction
device with some random thermal management device, without isolation of
course.  Wonder what chaos assigning that to a VM would cause.

> 00:15.0 Signal processing controller: Intel Corporation Sunrise Point-H
> LPSS I2C Controller #0 (rev 31)
> 00:15.1 Signal processing controller: Intel Corporation Sunrise Point-H
> LPSS I2C Controller #1 (rev 31)
>

> 00:16.0 Communication controller: Intel Corporation Sunrise Point-H CSME
> HECI #1 (rev 31)
>

> 00:17.0 SATA controller: Intel Corporation Device a102 (rev 31)
>

> 00:1c.0 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root
> Port #1 (rev f1)
> 00:1c.2 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root
> Port #3 (rev f1)
> 00:1c.7 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root
> Port #8 (rev f1)
> 00:1e.0 Signal processing controller: Intel Corporation Sunrise Point-H
> LPSS UART #0 (rev 31)
>

> 00:1f.0 ISA bridge: Intel Corporation Sunrise Point-H LPC Controller
> (rev 31)
> 00:1f.2 Memory controller: Intel Corporation Sunrise Point-H PMC (rev 31)
> 00:1f.3 Audio device: Intel Corporation Sunrise Point-H HD Audio (rev 31)
> 00:1f.4 SMBus: Intel Corporation Sunrise Point-H SMBus (rev 31)
>

Oh look, the audio controller that used to be a nice separate device is now
also buried into a multifunction device without isolation, no more
assigning that to a VM.

> 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc.
> [AMD/ATI] Hawaii PRO [Radeon R9 290]

01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device aac8
>

> 02:00.0 USB controller: ASMedia Technology Inc. Device 1242
> 03:00.0 Ethernet controller: Qualcomm Atheros Device e0a1 (rev 10)
> 04:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5721
> Gigabit Ethernet PCI Express (rev 21)
>
>
> If its not problem of unsafe interrupts, does anyone know why this can
> happen?
>

I don't see any reason to implicate unsafe interrupts.  It would be nice to
understand exactly what happens without specifying unsafe interrupts.  It
is possible to turn off interrupt remapping support with kernel config
options and boot options, so please make sure you have CONFIG_IRQ_REMAP=y
in your kernel and aren't disabling it on the commandline.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20150912/75333287/attachment.htm>