[vfio-users] QEMU VGA passthrough hangs with no error - NVIDIA GTX 580

Wed Feb 10 03:12:09 UTC 2016

So I've made some progress. I pci stubbed all of my graphics devices and I
was able to succesfully bind my 580 and boot into Seabios. I attempted to
bind Nouveau to each of my other cards but it failed and crashes my system.
Clearly the drivers were fighting each other there.

Oddly enough, the NVIDIA drivers actually worked fine with the VM. I'm
currently successfully running the VM on card 2 and Arch on the other 2
cards. I did this by booting with all cards pci stubbed, using a standard
vfio bind script to bind card 2, starting the VM, and then binding card 1
and 3 using an nvidia bind script (same script but replaces "vfio-pci" with
"nvidia"). Unfortunately this setup doesn't stand up to a restart of the
VM, it gives an error that it "Cannot read device rom" when I try to boot
the VM again. Clearly the nvidia driver is claiming some part of the guest
card and blocking access to its bios.

As a next step I'm considering trying this arbiter patch:
https://lkml.org/lkml/2014/5/25/94

I noticed it's fairly old though, has that functionality already been
implemented into the kernel? Is there a way that I can check if it has on
my own? Are there any other nvidia specifc patches? I've seen ones for
significantly older drivers but I'm using the most up to date ones
currently. Thanks.
On Feb 8, 2016 11:50 PM, "Ben J" <btpprograms at gmail.com> wrote:

> Switching over to Nouveau did the trick, at least for this issue. I was
> able to start the VM without hanging and had a different error which I
> fixed by passing it a ROM file for my GPU. I'm still not getting an output
> from the passed card though. Any ideas on how I would fix the arbiter issue
> then? Obviously I'd prefer to keep using the normal nvidia driver on linux
> if possible, although I suppose I could pass the other card to a Linux VM
> if all else fails. Thanks for the help so far, I'll continue working on
> this tomorrow.
> On Feb 8, 2016 11:19 PM, "Alex Williamson" <alex.williamson at redhat.com>
> wrote:
>
>> On Mon, 8 Feb 2016 22:42:48 -0500
>> Ben J <btpprograms at gmail.com> wrote:
>>
>> > Hi, I've been working to get VGA passthrough working for a Windows 7 VM
>> for
>> > the past few days. My setup is as follows:
>> >
>> > Processor: I7-3960X
>> > Graphics: 2 NVIDIA GTX 580's and one NVIDIA GT 420
>> > Monitors: 1 Monitor hooked up to the 420, 1 monitor hooked up to both
>> 580's
>> > Motherboard: ASUS Rampage IV Extreme
>> > OS: Arch Linux
>> > Kernel: Standard, 4.4.1
>> >
>> > My end goal is to bind one 580 to the VM and the other to Arch. The 420
>> is
>> > just to allow X to start initially since I haven't found a way to
>> disable
>> > only one 580(They have the same vendor and model ID) at a time. I was
>> > initially following this guide
>> > <
>> https://www.reddit.com/r/pcmasterrace/comments/3lno0t/gpu_passthrough_revisited_an_updated_guide_on_how/
>> >.
>> > My guest card was in its own IOMMU group and I'm not using integrated
>> > graphics so from what I understand I don't need the patched kernel. My
>> > cards bound to vfio-pci fine, but I got stuck when I realized that the
>> GTX
>> > 580 doesn't support UEFI booting. I changed over to booting with Seabios
>> > but any time I've enabled "x-vga=on" QEMU just hangs. The QEMU monitor
>> > window freezes, there is no video output to my other monitor, and the
>> only
>> > way to get out of the window is to force close it in htop. It uses 0%
>> cpu
>> > during this. The only log put out into dmesg is "kvm: zapping shadow
>> pages
>> > for mmio generation wraparound" which looks fairly unrelated from my
>> > searches.
>> >
>> > I've tested the following things:
>> > Enabling vga cirrus and x-vga - Opens an empty, black graphical window,
>> > still hangs
>> > Enabling vga cirrus and no x-vga - Boots as expected, but obviously no
>> > passthrough
>> > Passing in a ROM for the graphics card - Hangs
>> > Enabling unsafe interrupts - Hangs
>> > Using linux-vfio kernel from AUR - Hangs
>> > Lowering RAM (based on this post
>> > <https://bugzilla.kernel.org/show_bug.cgi?id=107561>) - Hangs
>> >
>> > Here are my boot options, booting using rEFInd:
>> > "root=UUID=4accaa3c-6d16-40e9-bb95-e78d160962a5 ro intel_iommu=on
>> > vfio_iommu_type1.allow_unsafe_interrupts=1
>> pci-stub.ids=10de:1080,10de:0e09"
>> >
>> > The two pci-stub ID's are for the GTX 580 and its audio card, it
>> disables
>> > both 580's.
>> >
>> > Here is my QEMU startup script:
>> >
>> > qemu-system-x86_64 -enable-kvm -m 1024 -cpu host,kvm=off \
>> > -smp 4,sockets=1,cores=4,threads=1 \
>> > -bios /usr/share/qemu/bios.bin  -vga none \
>> > -device vfio-pci,host=02:00.0,multifunction=on,x-vga=on \
>> > -device vfio-pci,host=02:00.1 \
>> > -drive
>> file=/mnt/LinuxStorage/VMs/VGAWin7.qcow2,id=disk,format=qcow2,if=virtio \
>> > -boot menu=on \
>> >
>> > At this point I don't know of any log files to look at so I'm
>> essentially
>> > taking shots in the dark. Are there any other logs that could be useful?
>> > Has anyone else experienced issues like this? I've done a lot of
>> searching
>> > around and haven't come across anyone who has had this kind of hang
>> without
>> > an error message. Any ideas would be appreciated, thanks.
>>
>> Possibly a VGA arbiter deadlock with the host graphics not allowing the
>> the assigned graphics a chance to access VGA space.  What happens if
>> you pci-stub all the NVIDIA devices (pci-stub.ids=10de:ffffffff)?  X
>> obviously won't start on the host, but you should be able to login from
>> another system (VGA text mode on the 420 will get really dicey once the
>> VM starts, but the goal is just to test if the driver with X running on
>> the 420 is the problem).  There used to be a problem with this using
>> the nvidia.ko driver and a workaround was to hack on the wrapper script
>> to prevent it from permanently grabbing the VGA arbiter resources.
>> That seems to be fixed for most folks, but perhaps it still exits with
>> the older driver you might be using with a 4-series card.  Using
>> nouveau in the host may also avoid it.  5-series cards seem to cause
>> folks quite a bit of trouble when trying to assign them.  Good luck,
>>
>> Alex
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20160209/39eb66bd/attachment.htm>