[vfio-users] VM doesn't boot if I use GPU passthrough

Tue Jan 26 16:33:55 UTC 2016

Next up would be Kernel, it sounds like...

On Tue, Jan 26, 2016 at 8:27 AM, Ryan Flagler <ryan.flagler at gmail.com>
wrote:

> Thanks for this info Will. Tried matching your qemu/libvirt versions and I
> still get the driver crashes. I'm not sure what else to try.
>
> On Mon, Jan 25, 2016 at 9:20 PM Will Marler <will at wmarler.com> wrote:
>
>> Hey Ryan,
>>
>> Here are the answers to your questions:
>>
>> 20:06:27 will ~% uname -a
>> Linux haze 4.3.3-2-ARCH #1 SMP PREEMPT Wed Dec 23 20:09:18 CET 2015
>> x86_64 GNU/Linux
>> 20:07:01 will ~% pacman -Q | egrep '^linux|^libvirt|^qemu'
>> libvirt 1.3.1-1
>> libvirt-glib 0.2.2-1
>> libvirt-python 1.3.1-1
>> linux 4.3.3-2
>> linux-api-headers 4.1.4-1
>> linux-firmware 20151207.bbe4917-1
>> qemu 2.4.1-2
>>
>> And here is the pastebin to my XML file: http://pastebin.com/nB3DPkEr
>>
>> As far as the guest drivers are concerned, they're the "GeForce Game
>> Ready Driver" version 361.43.
>>
>> HTH!
>>
>> On Mon, Jan 25, 2016 at 10:12 AM, Ryan Flagler <ryan.flagler at gmail.com>
>> wrote:
>>
>>> Thanks Will. Here is my info with the guest that crashes.
>>>
>>> Host OS Info
>>>  ubuntu - 14.04.03
>>>  kernel - 3.19.0-47
>>>
>>> virsh version
>>>  Compiled against library: libvirt 1.2.18
>>>  Using library: libvirt 1.2.18
>>>  Using API: QEMU 1.2.18
>>>  Running hypervisor: QEMU 2.5.0
>>>
>>> patches
>>>  I did not manually apply any patches to Qemu. Built directly from
>>> source.
>>>
>>> Guest Info
>>>  Windows 10
>>>  nVidia Graphics Driver 361.43
>>>
>>> Guest Event Viewer Entry On Driver Crash
>>>  Source - nvlddmkm
>>>  Event ID - 14
>>>  Info - \Device\Video3  CMDre 00000004 0000011c bad0011f 00000000
>>> 00d0011f
>>>
>>> Guest XML - Attached
>>>
>>>
>>> On Mon, Jan 25, 2016 at 10:18 AM Will Marler <will at wmarler.com> wrote:
>>>
>>>> On Mon, Jan 25, 2016 at 9:07 AM, Ryan Flagler <ryan.flagler at gmail.com>
>>>> wrote:
>>>>
>>>>> Will, could you tell us the following?
>>>>>
>>>>> What Linux distribution on host?
>>>>>
>>>> Arch
>>>>
>>>>> What kernel are you using on host?
>>>>> What libvirt version on host?
>>>>> What qemu version on host?
>>>>>
>>>> Will have to check when I'm home from work & the kids are asnooze, but
>>>> it's whatever's latest (and I'm not using the linux-vfio-lts kernel)
>>>>
>>>>> What OS on guest?
>>>>>
>>>> Windows 10.
>>>>
>>>>> What nvidia graphics driver version on guest?
>>>>>
>>>> Again, I'll have to check. But the latest or nearly latest.
>>>>
>>>>> My machines gpu driver crashes constantly and I'm trying to narrow
>>>>> down why. Thanks!
>>>>>
>>>> How frustrating : (. I'll also get a pastebin of my XML for you, in
>>>> case that will help. I've been running "stable" since mid 2015. I use the
>>>> quotes because some things tripped me up (guest machine can't "sleep," can
>>>> only power on & power off; when host machine goes to sleep with guest
>>>> running, on host wake-up the guest is non-responsive and 100% CPU).
>>>>
>>>> Will
>>>>
>>>>
>>>>>
>>>>> On Mon, Jan 25, 2016, 10:02 AM Will Marler <will at wmarler.com> wrote:
>>>>>
>>>>>> This is discussed in
>>>>>> http://vfio.blogspot.com/2015/05/vfio-gpu-how-to-series-part-4-our-first.html.
>>>>>> You have to do more than <kvm><hidden state='on'/></kvm>:
>>>>>>
>>>>>> "The GeForce card is nearly as easy, but we first need to work around
>>>>>> some of the roadblocks Nvidia has put in place to prevent you from using
>>>>>> the hardware you've purchased in the way that you desire (and by my reading
>>>>>> conforms to the EULA for their software, but IANAL).  For this step we
>>>>>> again need to run virsh edit on the VM.  Within the <features> section,
>>>>>> remove everything between the <hyperv> tags, including the tags
>>>>>> themselves.  In their place add the following tags:
>>>>>>
>>>>>>     <kvm>
>>>>>>       <hidden state='on'/>
>>>>>>     </kvm>
>>>>>>
>>>>>> Additionally, within the <clock> tag, find the timer named
>>>>>> hypervclock, remove the line containing this tag completely.  Save and exit
>>>>>> the edit session."
>>>>>>
>>>>>> I can confirm it works, I've been getting a lot of mileage from my
>>>>>> passed-through 750Ti lately since getting a Steam Link :-D.
>>>>>>
>>>>>> On Sun, Jan 24, 2016 at 7:32 AM, Ruben Felgenhauer <
>>>>>> 4felgenh at informatik.uni-hamburg.de> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> finally I had time to this again. I tried out virt-manager and after
>>>>>>> a bit of playing around with it, it /somewhat/ worked:
>>>>>>>
>>>>>>> The machine is at least booting. I still have a standard vga card
>>>>>>> enabled in the virt-manager config window.
>>>>>>> After the machine has booted, I can see that the device gets
>>>>>>> recognized as 750ti.
>>>>>>> However, the gpu doesn't get used, because of 'Code 43'.
>>>>>>> Code 43 is a generic error, so any idea what it could mean in this
>>>>>>> case?
>>>>>>>
>>>>>>> Of course I added the <kvm><hidden state='on'/></kvm> lines at the
>>>>>>> associated position.
>>>>>>>
>>>>>>> Best regards,
>>>>>>> Ruben
>>>>>>>
>>>>>>>
>>>>>>> Am 18.01.2016 um 22:27 schrieb Will Marler:
>>>>>>>
>>>>>>> I'm not sure what correct command-line syntax is. Have you tried
>>>>>>> using libvirt and VirtManager to handle your VM rather than command line,
>>>>>>> and modifying the XML rather than the command line? I think that's
>>>>>>> generally the preferred method these days (it's certainly easier from my
>>>>>>> point of view, and the way I got my 750 Ti to pass through).
>>>>>>>
>>>>>>> On Mon, Jan 18, 2016 at 11:04 AM, Ruben Felgenhauer <
>>>>>>> 4felgenh at informatik.uni-hamburg.de> wrote:
>>>>>>>
>>>>>>>> Hi, Alex!
>>>>>>>>
>>>>>>>> Thanks for your reply!
>>>>>>>> My GPU indeed has a seperate audio device located at 01:00.1.
>>>>>>>>
>>>>>>>> However, just adding -device vfio-pci,host=01:00.1 doesn't seem to
>>>>>>>> do the trick.
>>>>>>>> Of course the corresponding device is already blacklisted and bound
>>>>>>>> to vfio.
>>>>>>>>
>>>>>>>> The Debian Wiki entry about VGA passthrough (
>>>>>>>> <https://wiki.debian.org/VGAPassthrough>
>>>>>>>> https://wiki.debian.org/VGAPassthrough) mentions QEMU arguments
>>>>>>>> like "-device
>>>>>>>> vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=...
>>>>>>>> -device vfio-pci,host=01:00.1,bus=pcie.0" which seems to address GPUs with
>>>>>>>> audio devices, but if I try to do something similar, the buses 'root' and
>>>>>>>> 'pcie' couldn't be found. Maybe I missed something very important?
>>>>>>>>
>>>>>>>> On the same article, it says that the "HDMI soundcard [...] needs
>>>>>>>> to be unbound from its driver":
>>>>>>>> # echo '0000:01:00.1' | sudo tee
>>>>>>>> /sys/bus/pci/devices/0000:01:00.1/driver/unbind
>>>>>>>> I figured the vfio-bind script from the Arch Linux Forum thread (
>>>>>>>> https://bbs.archlinux.org/viewtopic.php?id=162768) would do
>>>>>>>> exactly this thing, so I didn't explicitly do so for the audio device. Is
>>>>>>>> that okay?
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>> Ruben
>>>>>>>>
>>>>>>>>
>>>>>>>> Am 18.01.2016 um 08:31 schrieb Alexander Petrenz:
>>>>>>>>
>>>>>>>> Hi Ruben,
>>>>>>>>
>>>>>>>> I guess your 750ti also has some audio device. You should pass
>>>>>>>> through this too. It should be something like 01:00.1. There are many
>>>>>>>> command line examples you can find about that.
>>>>>>>> Also I´m not quite sure, if you should remove the x-vga=on.
>>>>>>>>
>>>>>>>> Regards
>>>>>>>> Alex
>>>>>>>>
>>>>>>>> On Sun, Jan 17, 2016 at 11:12 PM, Ruben Felgenhauer <
>>>>>>>> <4felgenh at informatik.uni-hamburg.de>
>>>>>>>> 4felgenh at informatik.uni-hamburg.de> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I am trying to pass my nVidia GTX 750ti to my QEMU guest.
>>>>>>>>>
>>>>>>>>> Problem is: After the QEMU monitor pops up, nothing happens. The
>>>>>>>>> GPU's output is dead, and the vm won't be accessible via SSH anymore, so
>>>>>>>>> it's very likely that the VM isn't booting up at all. Also, there are no
>>>>>>>>> error messages from QEMU on the console whatsoever which makes debugging it
>>>>>>>>> especially hard.
>>>>>>>>>
>>>>>>>>> This is how I start the vm with normal vga emulation:
>>>>>>>>> qemu-system-x86_64 -hda vm.ovl -boot c -enable-kvm -m 1024 -cpu
>>>>>>>>> host,kvm=off -smp cores=4,threads=2 -redir tcp:5022::22
>>>>>>>>> Everything runs fine in this case. To do the passthrough, I add
>>>>>>>>> this:
>>>>>>>>> -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on -vga none
>>>>>>>>> This brings said problems with it. I also tried out multiple
>>>>>>>>> different combinations of -device's arguments or even adding a romfile for
>>>>>>>>> the GPU, but none of these steps changed anything at all.
>>>>>>>>>
>>>>>>>>> Obviously, I am using a BIOS installation and I'm well-aware with
>>>>>>>>> this bug: <https://bugzilla.kernel.org/show_bug.cgi?id=107561>
>>>>>>>>> https://bugzilla.kernel.org/show_bug.cgi?id=107561, but neither
>>>>>>>>> using less RAM (as you can see I am using 1GB now) nor switching to an
>>>>>>>>> older Kernel changed anything about the problem. I have tried Kernel 4.1.0
>>>>>>>>> and 4.3.0.
>>>>>>>>>
>>>>>>>>> Host is Debian testing with QEMU 2.5.0.
>>>>>>>>> I tried both Debian and Windows 7 as a guest, but both are showing
>>>>>>>>> exactly the same behaviour.
>>>>>>>>> Mainboard is an ASUS Z87-PLUS. The 750ti is produced by ASUS
>>>>>>>>> aswell.
>>>>>>>>>
>>>>>>>>> Any idea how I could get passthrough running?
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> vfio-users mailing list
>>>>>>>>> vfio-users at redhat.com
>>>>>>>>> https://www.redhat.com/mailman/listinfo/vfio-users
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> vfio-users mailing list
>>>>>>>> vfio-users at redhat.com
>>>>>>>> https://www.redhat.com/mailman/listinfo/vfio-users
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> _______________________________________________
>>>>>> vfio-users mailing list
>>>>>> vfio-users at redhat.com
>>>>>> https://www.redhat.com/mailman/listinfo/vfio-users
>>>>>>
>>>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20160126/6d2eb5f8/attachment.htm>