[vfio-users] Passthrough Primary GPU

[Asher Gottlieb]

Hi, 
I'm currently having issues passing one of two discrete gpus to a
Windows guest (I have no iGPU).
I'm not sure exactly where the issue lies, but I believe it has to do
with the fact that the motherboard uses the gpu I want to passthrough
as its output gpu during BIOS/UEFI boot. There is no option to switch
the default gpu in its settings.

I have disabled the framebuffer of the gpu I want to passthrough (a gtx
1070) by using these kernel parameters:
video=efifb:off,vesafb:off

There is now only /dev/fb0, my secondary gpu, running with nouveau.
There are no longer two framebuffers.

X is also setup to run on the secondary gpu (a gtx 660) and is working
perfectly.

The Windows 10 guest is passed an SSD, as opposed to an image file, so
I can also boot off the SSD directly by changing the boot order of my
machine. When booting from the Windows 10 SSD directly, everything
works smoothly, the gtx 1070 and all.

I can startup Windows 10 as a guest and connect to it over vnc.
However, when looking in device manager, I get error 43 for the gtx
1070. 
(Before I start the guest, the monitor is on with a black screen. When
I start the guest, the monitor enters power saving mode. vfio-pci must
have been giving it a signal before I started the guest, which is
interesting.)

In dmesg, I noticed these odd messages:
[   46.241022] pmd_set_huge: Cannot satisfy [mem 0xb0000000-0xb0200000] 
with a huge-page mapping due to MTRR override.
[   55.165255] vfio-pci 0000:01:00.0: Invalid PCI ROM header signature:
expecting 0xaa55, got 0xffff
[   55.165287] vfio-pci 0000:01:00.0: Invalid PCI ROM header signature:
expecting 0xaa55, got 0xffff

Google tells me that the "pmd_set_huge" message is more of a warning
than a serious issue, but the message doesn't occur when I start the
Windows guest without the 1070 passed through. (Both configurations use
2 hugepages of size 1GB).

The vfio-pci messages I assume are because of the fact that the uefi
has booted using the 1070. I think that is the root of the issue but I
am not really sure...

Here is my configuration for libvirt:
http://pastebin.com/raw/PFmQSy6N

As you can see, the hidden state is on and there is a vendor id. 
(As a side note, I had no content in the hyperv tags to begin with,
unlike other users it seems, but I haven't looked into this very much.
If these tags would help increase the performance of the guest, please
let me know which ones I should add.)

Looking at the QEMU logs, I also get this suspicious message:
Domain id=1 is tainted: host-cpu

I'm not quite sure what to make of it...
I noticed an email chain in the archives from about a month ago about
passing through the primary gpu, but I couldn't replicate their steps
sucessfully.

I'm new to this stuff, and so any help is appreciated!

Thanks,
Asher