[vfio-users] No video output from dedicated GPU when doing PCI passthrough

Sun Apr 16 11:00:56 UTC 2017

Hi!

I've been trying to get PCI/GPU passthrough to work for a few days but 
I've hit a wall. I can't get any video output from the passed GPU. I'm 
trying to virtualize Windows 10 Home Edition. I've been using the 
following guides:

  - Arch Wiki: https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF
  - Reddit: 
https://www.reddit.com/r/linuxmasterrace/comments/2z30dq/gpu_passthrough_or_how_to_play_any_game_at_near/?st=j1kioetn&sh=a198702b
  - http://vfio.blogspot.com/
  - Level1Techs: 
https://forum.level1techs.com/t/play-games-in-windows-on-linux-pci-passthrough-quick-guide/108981

My hardware should support this:

  - OS: Arch Linux with kernel version 4.10.9-1-ARCH
  - CPU: Intel Core i5-6600K (supports VT-d and VT-x)
  - GPU: Intel HD Graphics 530 (iGPU, used by Linux)
  - GPU: AMD Radeon R9 380 (passed through)
  - Mobo: Asus Z170 Pro Gaming (if I'm interpreting the manual 
correctly, hardware virtualization should be supported)
  - RAM: 2x4GB Kingston Savage DDR4-3000

VT-d is enabled from the UEFI BIOS and IOMMU is enabled in the kernel. 
This can be confirmed with `dmesg | grep -i iommu`:

     [    0.000000] Command line: BOOT_IMAGE=/vmlinuz-linux 
root=UUID=c6c3c8f3-030c-4dce-be63-8938386ac963 rw quiet intel_iommu=on
     [    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-linux 
root=UUID=c6c3c8f3-030c-4dce-be63-8938386ac963 rw quiet intel_iommu=on
     [    0.000000] DMAR: IOMMU enabled
     ...

The Radeon GPU and its HDMI audio controller are using the vfio-pci 
driver and are in the IOMMU group 1 with a PCIe controller. Output of 
`lspci -nnk`:

     ...
     01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, 
Inc. [AMD/ATI] Tonga PRO [Radeon R9 285/380] [1002:6939] (rev f1)
         Subsystem: Micro-Star International Co., Ltd. [MSI] Device 
[1462:2015]
         Kernel driver in use: vfio-pci
         Kernel modules: amdgpu
     01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] 
Tonga HDMI Audio [Radeon R9 285/380] [1002:aad8]
         Subsystem: Micro-Star International Co., Ltd. [MSI] Device 
[1462:aad8]
         Kernel driver in use: vfio-pci
         Kernel modules: snd_hda_intel
     ...

vfio-pci is running properly, as proved by `dmesg | grep -i vfio`:

     [    0.773952] VFIO - User Level meta-driver version: 0.3
     [    0.774978] vfio-pci 0000:01:00.0: vgaarb: changed VGA decodes: 
olddecodes=io+mem,decodes=io+mem:owns=none
     [    0.789351] vfio_pci: add [1002:6939[ffff:ffff]] class 
0x000000/00000000
     [    0.806016] vfio_pci: add [1002:aad8[ffff:ffff]] class 
0x000000/00000000
     [    2.767507] vfio-pci 0000:01:00.0: vgaarb: changed VGA decodes: 
olddecodes=io+mem,decodes=io+mem:owns=none
     [ 1310.412510] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003)

QEMU, libvirt, OVMF and virt-manager are installed and libvirtd-daemon 
is running. Output of `systemctl status libvirtd`:

     ● libvirtd.service - Virtualization daemon
        Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; 
enabled; vendor preset: disabled)
        Active: active (running) since Sat 2017-04-15 12:16:01 EEST; 24h ago
          Docs: man:libvirtd(8)
                http://libvirt.org
      Main PID: 469 (libvirtd)
         Tasks: 21 (limit: 4915)
        CGroup: /system.slice/libvirtd.service
                ├─469 /usr/bin/libvirtd
                ├─708 /usr/bin/dnsmasq 
--conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro 
--dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper
                ├─709 /usr/bin/dnsmasq 
--conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro 
--dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper
                ├─754 /usr/bin/dnsmasq 
--conf-file=/var/lib/libvirt/dnsmasq/wireless.conf --leasefile-ro 
--dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper
                └─755 /usr/bin/dnsmasq 
--conf-file=/var/lib/libvirt/dnsmasq/wireless.conf --leasefile-ro 
--dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper

I setup my VM with virt-manager like this (exact steps):

  1. Create a new QEMU/KVM VM from local install media
  2. Use a Windows 10 ISO image found from the hard disk. OS type and 
version are detected properly.
  3. Use default settings for RAM and CPU: 4096 MiB and 2 cores
  4. Create custom storage in /opt/os-storage/win10.qcow2 with size of 
60 GiB
  5. Check "Customize configuration before install" and click Finish
  6. In "Overview":
     - Change Firmware to UEFI x86_64: /usr/share/ovmf/x64/ovmf_x64.bin
     - Chipset is "i440FX" and Architecture "x86_64"
  7. In "CPUs":
     - Change Model to "host-passthrough"
  8. In "IDE Disk 1":
     - Change Disk bus to "VirtIO"
     - Change Cache mode to "writeback"
  9. In "IDE CDROM 1":
     - Connect it to the Win10 ISO
     - Change Disk bus to "SATA"
  10. In "NIC ...":
     - Change Device model to "virtio"
  11. In "Controller USB":
     - Change Model to "USB 3"
  12. Click "Add Hardware"
     - Add Storage (Disk device) in 
$HOME/os-images/storage/win10-data.qcow2 with size of 400 GiB
     - Change Bus type to "VirtIO"
     - Change Cache mode to "writeback"
  13. Click "Add Hardware"
     - Add Storage (CDROM device) connected to virtio drivers from Red 
Hat website
     - Change Bus type to "SATA"
     - Change Cache mode to "Hypervisor default"
  14. In "Boot Options":
     - Check "Enable boot menu"
     - Check "VirtIO Disk 1" and "SATA CDROM 1" from the list

I clicked "Begin Installation". It booted successfully to the 
installation media. I was able to load the VirtIO SCSI driver and see my 
virtual drives. Then I stopped the installation.

  15. Remove "Tablet", "Display Spice", "Sound: ich6", "Serial 1", 
"Channel spice", "Video QXL", "USB Redirector 1" and "USB Redirector 2"
  16. Click "Add Hardware" and add two "PCI Host Device"s:
     - 0000:01:00:0 ... Tonga PRO...
     - 0000:01:00.1 ... Tonga HDMI Audio...

I clicked the big play-button and changed my monitor input to the 
dedicated GPU, and it detected no signal. Virtual console said 
"Graphical console not configured for guest". The system is clearly 
starting up: it's allocating RAM and using the processor. I tried the 
same in Fedora 24 earlier this week but it failed in the same stage.

Here's the log file from /var/log/libvirt/qemu/win10.log:
   https://pastebin.com/DSCW0VXP
I don't know if there are some other log files I'd need to include since 
I'm not too familiar with KVM.

What I've tried:
  - Different Linux distros: Fedora 24 earlier this week, now Arch
  - Different chipsets: i440FX and Q35, with same results
  - pci-stub instead of vfio-pci
  - It's now faulty cable, I'm using it right now
  - Adding `<vendor_id state='on' value='whatever'/>` to `virsh edit`
  - Probably many more things I can't recall right now

I asked the same question over at SuperUser but figured that there'd be 
more knowledgeable people here.

Thank you,
Elias Kosunen