[vfio-users] QEMU VGA passthrough hangs with no error - NVIDIA GTX 580

[Ben J]

Edit: the failure to find rom after restart issue actually occurs
regardless of whether I start the nvidia driver or not. I've seen a lot of
questions regarding that issue elsewhere so I'll start looking into it.
On Feb 9, 2016 10:12 PM, "Ben J" <btpprograms at gmail.com> wrote:

> So I've made some progress. I pci stubbed all of my graphics devices and I
> was able to succesfully bind my 580 and boot into Seabios. I attempted to
> bind Nouveau to each of my other cards but it failed and crashes my system.
> Clearly the drivers were fighting each other there.
>
> Oddly enough, the NVIDIA drivers actually worked fine with the VM. I'm
> currently successfully running the VM on card 2 and Arch on the other 2
> cards. I did this by booting with all cards pci stubbed, using a standard
> vfio bind script to bind card 2, starting the VM, and then binding card 1
> and 3 using an nvidia bind script (same script but replaces "vfio-pci" with
> "nvidia"). Unfortunately this setup doesn't stand up to a restart of the
> VM, it gives an error that it "Cannot read device rom" when I try to boot
> the VM again. Clearly the nvidia driver is claiming some part of the guest
> card and blocking access to its bios.
>
> As a next step I'm considering trying this arbiter patch:
> https://lkml.org/lkml/2014/5/25/94
>
> I noticed it's fairly old though, has that functionality already been
> implemented into the kernel? Is there a way that I can check if it has on
> my own? Are there any other nvidia specifc patches? I've seen ones for
> significantly older drivers but I'm using the most up to date ones
> currently. Thanks.
> On Feb 8, 2016 11:50 PM, "Ben J" <btpprograms at gmail.com> wrote:
>
>> Switching over to Nouveau did the trick, at least for this issue. I was
>> able to start the VM without hanging and had a different error which I
>> fixed by passing it a ROM file for my GPU. I'm still not getting an output
>> from the passed card though. Any ideas on how I would fix the arbiter issue
>> then? Obviously I'd prefer to keep using the normal nvidia driver on linux
>> if possible, although I suppose I could pass the other card to a Linux VM
>> if all else fails. Thanks for the help so far, I'll continue working on
>> this tomorrow.
>> On Feb 8, 2016 11:19 PM, "Alex Williamson" <alex.williamson at redhat.com>
>> wrote:
>>
>>> On Mon, 8 Feb 2016 22:42:48 -0500
>>> Ben J <btpprograms at gmail.com> wrote:
>>>
>>> > Hi, I've been working to get VGA passthrough working for a Windows 7
>>> VM for
>>> > the past few days. My setup is as follows:
>>> >
>>> > Processor: I7-3960X
>>> > Graphics: 2 NVIDIA GTX 580's and one NVIDIA GT 420
>>> > Monitors: 1 Monitor hooked up to the 420, 1 monitor hooked up to both
>>> 580's
>>> > Motherboard: ASUS Rampage IV Extreme
>>> > OS: Arch Linux
>>> > Kernel: Standard, 4.4.1
>>> >
>>> > My end goal is to bind one 580 to the VM and the other to Arch. The
>>> 420 is
>>> > just to allow X to start initially since I haven't found a way to
>>> disable
>>> > only one 580(They have the same vendor and model ID) at a time. I was
>>> > initially following this guide
>>> > <
>>> https://www.reddit.com/r/pcmasterrace/comments/3lno0t/gpu_passthrough_revisited_an_updated_guide_on_how/
>>> >.
>>> > My guest card was in its own IOMMU group and I'm not using integrated
>>> > graphics so from what I understand I don't need the patched kernel. My
>>> > cards bound to vfio-pci fine, but I got stuck when I realized that the
>>> GTX
>>> > 580 doesn't support UEFI booting. I changed over to booting with
>>> Seabios
>>> > but any time I've enabled "x-vga=on" QEMU just hangs. The QEMU monitor
>>> > window freezes, there is no video output to my other monitor, and the
>>> only
>>> > way to get out of the window is to force close it in htop. It uses 0%
>>> cpu
>>> > during this. The only log put out into dmesg is "kvm: zapping shadow
>>> pages
>>> > for mmio generation wraparound" which looks fairly unrelated from my
>>> > searches.
>>> >
>>> > I've tested the following things:
>>> > Enabling vga cirrus and x-vga - Opens an empty, black graphical window,
>>> > still hangs
>>> > Enabling vga cirrus and no x-vga - Boots as expected, but obviously no
>>> > passthrough
>>> > Passing in a ROM for the graphics card - Hangs
>>> > Enabling unsafe interrupts - Hangs
>>> > Using linux-vfio kernel from AUR - Hangs
>>> > Lowering RAM (based on this post
>>> > <https://bugzilla.kernel.org/show_bug.cgi?id=107561>) - Hangs
>>> >
>>> > Here are my boot options, booting using rEFInd:
>>> > "root=UUID=4accaa3c-6d16-40e9-bb95-e78d160962a5 ro intel_iommu=on
>>> > vfio_iommu_type1.allow_unsafe_interrupts=1
>>> pci-stub.ids=10de:1080,10de:0e09"
>>> >
>>> > The two pci-stub ID's are for the GTX 580 and its audio card, it
>>> disables
>>> > both 580's.
>>> >
>>> > Here is my QEMU startup script:
>>> >
>>> > qemu-system-x86_64 -enable-kvm -m 1024 -cpu host,kvm=off \
>>> > -smp 4,sockets=1,cores=4,threads=1 \
>>> > -bios /usr/share/qemu/bios.bin  -vga none \
>>> > -device vfio-pci,host=02:00.0,multifunction=on,x-vga=on \
>>> > -device vfio-pci,host=02:00.1 \
>>> > -drive
>>> file=/mnt/LinuxStorage/VMs/VGAWin7.qcow2,id=disk,format=qcow2,if=virtio \
>>> > -boot menu=on \
>>> >
>>> > At this point I don't know of any log files to look at so I'm
>>> essentially
>>> > taking shots in the dark. Are there any other logs that could be
>>> useful?
>>> > Has anyone else experienced issues like this? I've done a lot of
>>> searching
>>> > around and haven't come across anyone who has had this kind of hang
>>> without
>>> > an error message. Any ideas would be appreciated, thanks.
>>>
>>> Possibly a VGA arbiter deadlock with the host graphics not allowing the
>>> the assigned graphics a chance to access VGA space.  What happens if
>>> you pci-stub all the NVIDIA devices (pci-stub.ids=10de:ffffffff)?  X
>>> obviously won't start on the host, but you should be able to login from
>>> another system (VGA text mode on the 420 will get really dicey once the
>>> VM starts, but the goal is just to test if the driver with X running on
>>> the 420 is the problem).  There used to be a problem with this using
>>> the nvidia.ko driver and a workaround was to hack on the wrapper script
>>> to prevent it from permanently grabbing the VGA arbiter resources.
>>> That seems to be fixed for most folks, but perhaps it still exits with
>>> the older driver you might be using with a 4-series card.  Using
>>> nouveau in the host may also avoid it.  5-series cards seem to cause
>>> folks quite a bit of trouble when trying to assign them.  Good luck,
>>>
>>> Alex
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20160209/55cb41a8/attachment.htm>