[vfio-users] Radeon passthrough causes host lockups when using vfio

[Arjen]

Apologies for replying to myself, but I would like to post the IMHO 
non-obvious solution for others who might run into this.

The problem was resolved by using a virtual machine with a PCIe bus and 
not using that PCIe bus for the vfio-pci pass-through. This works with 
both OVMF and SeaBIOS (for me) in passing through the AMD Radeon HD 7750 
as a secondary GPU to a Linux guest.

I tried VFIO pass-through before with and without PCIe, however it seems 
you need to do both at the same time. The drm module will crash in 
drm_pcie_get_speed_cap_mask unless I use -machine q35 (add PCIe bus). 
The radeon module gives 'ring 0 stalled' and 'GPU lockup' errors  after 
starting Xorg, unless I use -device vfio-pci,bus=pci.0,host=... (do NOT 
use PCIe bus).

kind regards, Arjen

On 11/07/2016 10:48 PM, Arjen wrote:
> Hi,
>
> I'm getting ring 0 stalls and GPU lockups errors in the radeon driver
> when starting Xorg. When passing through a AMD Radeon 7750 using VFIO.,
> the display looks garbled with random-looking patterns and colors. The
> guest tries to reset the GPU for a few seconds, until the host freezes
> to the point where the reset button does not even work.
>
> Passthrough of other devices such as on-board audio, USB3 controller
> work fine with VFIO and linux guests. I have a VM that runs Fedora 21
> and works fine on a kernel 2.6.32 (RedHat+OpenVZ+Proxmox) using
> pci-assign. The same VM experiences the ring 0 stall, GPU lockup and a
> host freeze with kernel 4.4.21 (Ubuntu+Proxmox) using vfio-pci.
>
> The differences in kvm command-line parameters are:
> Linux 2.6.32 + QEMU 2.2.1:  -cpu SandyBridge,+x2apic
>  -device
> 'pci-assign,host=01:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,multifunction=on'
>
>  -device
> 'pci-assign,host=01:00.1,id=hostpci0.1,bus=ich9-pcie-port-1,addr=0x0.1'
>  -device 'pci-assign,host=00:1b.0,id=hostpci1,bus=pci.0,addr=0x11'
> Linux 4.4.21 + QEMU 2.7.0:  -cpu
> 'SandyBridge,+kvm_pv_unhalt,+kvm_pv_eoi,enforce,vendor=GenuineIntel'
>  -device
> 'vfio-pci,host=01:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,multifunction=on'
>
>  -device
> 'vfio-pci,host=01:00.1,id=hostpci0.1,bus=ich9-pcie-port-1,addr=0x0.1'
>  -device 'vfio-pci,host=00:1b.0,id=hostpci1,bus=pci.0,addr=0x11'
>
> I have tried to use the same kvm command-line in 4.4, but it complains
> about an invalid argument for hostpci0.0 and I cannot find out which or
> why. (It also does not support +x2apic anymore)
>
> I have tested also SeaBIOS (as secondary VGA) and OVMF (with single
> x-vga=1) using Ubuntu Gnome 16.10, using machine q35 because the radeon
> driver assumes a PCIE root, with the same results. According to lspci,
> MSI-X is enabled. I have tried various kernel module settings for radeon
> and vfio-pci that I found on this mailinglist  to no avail
> (radeon.hard_reset=1, .dpm=1, ..., vfio-pci.disable_idle_d3).
>
> Has someone experienced similar issues and found a fix for it? Can
> someone advise me on things to investigate or try to get my desktop VM
> running with VFIO (or how to fall-back to pci-assign)?
>
> I noticed people having more luck with NVidia, but I replaced mine with
> AMD because I could get it to work with pci-assign some years back, and
> would like to keep using open-source drivers.
>
> Thank you and kind regards,
>     Arjen