[vfio-users] VM doesn't boot if I use GPU passthrough

Tue Jan 26 17:56:26 UTC 2016

Thanks for the encouragement guys. I think I'm going to try some scrounge
for some other hardware just to make sure my GPU isn't the problem. The
only other cards I have are AMD which besides rebooting actually work
solidly.

On Tue, Jan 26, 2016 at 11:36 AM Ruben Felgenhauer <
4felgenh at informatik.uni-hamburg.de> wrote:

> Hi, Ryan!
>
> Installing an older Kernel is probably easier than you might think.
> On Ubuntu you should be able to find out which kernels are in the repos
> with apt-cache,
> but I sadly don't know the params, so maybe take a look at the manpage.
> And afterwards you should be able to install a specific version with
> 'apt-get install packagename=version'
>
> On Debian there is simply http://snapshot.debian.org/package/linux/ which
> is how I downgraded from 4.3 to 4.1 on my Debian testing.
> You can just download the deb files there and install them with dpkg.
> Maybe if you search for a testing system that is similar to Ubuntu, you
> could give that a try.
>
> But keep in mind that this doesn't uninstall the old kernel, so you will
> have a fallback.
> You might need to select the right kernel at GRUB though.
>
> Best regards,
> Ruben
>
>
> Am 26.01.2016 um 18:11 schrieb Will Marler:
>
> Well, you run Linux and you're experimenting with VGA passthrough ...
> you're resourceful! What about picking up a 16GB SSD for $15
> <http://www.amazon.com/Samsung-16GB-Solid-State-Drive/dp/B003YMJPE8/ref=sr_1_3?ie=UTF8&qid=1453827934&sr=8-3&keywords=16GB+SSD> and
> installing Arch (or Fedora, or Gentoo... whatever suits) side by side with
> Ubuntu? Presumably your VM can be launched either way without any
> configuration changes ... when you get tired/frustrated of the
> Arch/Fedora/Gentoo way you reboot back. If it works, you've found the
> answer, if it doesn't, you've improved your Linux-fu for not much
> (monetary) cost.
>
>
> On Tue, Jan 26, 2016 at 10:03 AM, Ryan Flagler <ryan.flagler at gmail.com>
> wrote:
>
>> Yea, that's just a major jump. Wish I had a dedicated test system to try
>> more things. ;)
>>
>> On Tue, Jan 26, 2016 at 10:34 AM Will Marler <will at wmarler.com> wrote:
>>
>>> Next up would be Kernel, it sounds like...
>>>
>>> On Tue, Jan 26, 2016 at 8:27 AM, Ryan Flagler <ryan.flagler at gmail.com>
>>> wrote:
>>>
>>>> Thanks for this info Will. Tried matching your qemu/libvirt versions
>>>> and I still get the driver crashes. I'm not sure what else to try.
>>>>
>>>> On Mon, Jan 25, 2016 at 9:20 PM Will Marler <will at wmarler.com> wrote:
>>>>
>>>>> Hey Ryan,
>>>>>
>>>>> Here are the answers to your questions:
>>>>>
>>>>> 20:06:27 will ~% uname -a
>>>>> Linux haze 4.3.3-2-ARCH #1 SMP PREEMPT Wed Dec 23 20:09:18 CET 2015
>>>>> x86_64 GNU/Linux
>>>>> 20:07:01 will ~% pacman -Q | egrep '^linux|^libvirt|^qemu'
>>>>> libvirt 1.3.1-1
>>>>> libvirt-glib 0.2.2-1
>>>>> libvirt-python 1.3.1-1
>>>>> linux 4.3.3-2
>>>>> linux-api-headers 4.1.4-1
>>>>> linux-firmware 20151207.bbe4917-1
>>>>> qemu 2.4.1-2
>>>>>
>>>>> And here is the pastebin to my XML file: http://pastebin.com/nB3DPkEr
>>>>>
>>>>> As far as the guest drivers are concerned, they're the "GeForce Game
>>>>> Ready Driver" version 361.43.
>>>>>
>>>>> HTH!
>>>>>
>>>>> On Mon, Jan 25, 2016 at 10:12 AM, Ryan Flagler <ryan.flagler at gmail.com
>>>>> > wrote:
>>>>>
>>>>>> Thanks Will. Here is my info with the guest that crashes.
>>>>>>
>>>>>> Host OS Info
>>>>>>  ubuntu - 14.04.03
>>>>>>  kernel - 3.19.0-47
>>>>>>
>>>>>> virsh version
>>>>>>  Compiled against library: libvirt 1.2.18
>>>>>>  Using library: libvirt 1.2.18
>>>>>>  Using API: QEMU 1.2.18
>>>>>>  Running hypervisor: QEMU 2.5.0
>>>>>>
>>>>>> patches
>>>>>>  I did not manually apply any patches to Qemu. Built directly from
>>>>>> source.
>>>>>>
>>>>>> Guest Info
>>>>>>  Windows 10
>>>>>>  nVidia Graphics Driver 361.43
>>>>>>
>>>>>> Guest Event Viewer Entry On Driver Crash
>>>>>>  Source - nvlddmkm
>>>>>>  Event ID - 14
>>>>>>  Info - \Device\Video3  CMDre 00000004 0000011c bad0011f 00000000
>>>>>> 00d0011f
>>>>>>
>>>>>> Guest XML - Attached
>>>>>>
>>>>>>
>>>>>> On Mon, Jan 25, 2016 at 10:18 AM Will Marler <will at wmarler.com>
>>>>>> wrote:
>>>>>>
>>>>>>> On Mon, Jan 25, 2016 at 9:07 AM, Ryan Flagler <
>>>>>>> ryan.flagler at gmail.com> wrote:
>>>>>>>
>>>>>>>> Will, could you tell us the following?
>>>>>>>>
>>>>>>>> What Linux distribution on host?
>>>>>>>>
>>>>>>> Arch
>>>>>>>
>>>>>>>> What kernel are you using on host?
>>>>>>>> What libvirt version on host?
>>>>>>>> What qemu version on host?
>>>>>>>>
>>>>>>> Will have to check when I'm home from work & the kids are asnooze,
>>>>>>> but it's whatever's latest (and I'm not using the linux-vfio-lts kernel)
>>>>>>>
>>>>>>>> What OS on guest?
>>>>>>>>
>>>>>>> Windows 10.
>>>>>>>
>>>>>>>> What nvidia graphics driver version on guest?
>>>>>>>>
>>>>>>> Again, I'll have to check. But the latest or nearly latest.
>>>>>>>
>>>>>>>> My machines gpu driver crashes constantly and I'm trying to narrow
>>>>>>>> down why. Thanks!
>>>>>>>>
>>>>>>> How frustrating : (. I'll also get a pastebin of my XML for you, in
>>>>>>> case that will help. I've been running "stable" since mid 2015. I use the
>>>>>>> quotes because some things tripped me up (guest machine can't "sleep," can
>>>>>>> only power on & power off; when host machine goes to sleep with guest
>>>>>>> running, on host wake-up the guest is non-responsive and 100% CPU).
>>>>>>>
>>>>>>> Will
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Jan 25, 2016, 10:02 AM Will Marler <will at wmarler.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> This is discussed in
>>>>>>>>> http://vfio.blogspot.com/2015/05/vfio-gpu-how-to-series-part-4-our-first.html.
>>>>>>>>> You have to do more than <kvm><hidden state='on'/></kvm>:
>>>>>>>>>
>>>>>>>>> "The GeForce card is nearly as easy, but we first need to work
>>>>>>>>> around some of the roadblocks Nvidia has put in place to prevent you from
>>>>>>>>> using the hardware you've purchased in the way that you desire (and by my
>>>>>>>>> reading conforms to the EULA for their software, but IANAL).  For this step
>>>>>>>>> we again need to run virsh edit on the VM.  Within the <features> section,
>>>>>>>>> remove everything between the <hyperv> tags, including the tags
>>>>>>>>> themselves.  In their place add the following tags:
>>>>>>>>>
>>>>>>>>>     <kvm>
>>>>>>>>>       <hidden state='on'/>
>>>>>>>>>     </kvm>
>>>>>>>>>
>>>>>>>>> Additionally, within the <clock> tag, find the timer named
>>>>>>>>> hypervclock, remove the line containing this tag completely.  Save and exit
>>>>>>>>> the edit session."
>>>>>>>>>
>>>>>>>>> I can confirm it works, I've been getting a lot of mileage from my
>>>>>>>>> passed-through 750Ti lately since getting a Steam Link :-D.
>>>>>>>>>
>>>>>>>>> On Sun, Jan 24, 2016 at 7:32 AM, Ruben Felgenhauer <
>>>>>>>>> 4felgenh at informatik.uni-hamburg.de> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> finally I had time to this again. I tried out virt-manager and
>>>>>>>>>> after a bit of playing around with it, it /somewhat/ worked:
>>>>>>>>>>
>>>>>>>>>> The machine is at least booting. I still have a standard vga card
>>>>>>>>>> enabled in the virt-manager config window.
>>>>>>>>>> After the machine has booted, I can see that the device gets
>>>>>>>>>> recognized as 750ti.
>>>>>>>>>> However, the gpu doesn't get used, because of 'Code 43'.
>>>>>>>>>> Code 43 is a generic error, so any idea what it could mean in
>>>>>>>>>> this case?
>>>>>>>>>>
>>>>>>>>>> Of course I added the <kvm><hidden state='on'/></kvm> lines at
>>>>>>>>>> the associated position.
>>>>>>>>>>
>>>>>>>>>> Best regards,
>>>>>>>>>> Ruben
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Am 18.01.2016 um 22:27 schrieb Will Marler:
>>>>>>>>>>
>>>>>>>>>> I'm not sure what correct command-line syntax is. Have you tried
>>>>>>>>>> using libvirt and VirtManager to handle your VM rather than command line,
>>>>>>>>>> and modifying the XML rather than the command line? I think that's
>>>>>>>>>> generally the preferred method these days (it's certainly easier from my
>>>>>>>>>> point of view, and the way I got my 750 Ti to pass through).
>>>>>>>>>>
>>>>>>>>>> On Mon, Jan 18, 2016 at 11:04 AM, Ruben Felgenhauer <
>>>>>>>>>> 4felgenh at informatik.uni-hamburg.de> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi, Alex!
>>>>>>>>>>>
>>>>>>>>>>> Thanks for your reply!
>>>>>>>>>>> My GPU indeed has a seperate audio device located at 01:00.1.
>>>>>>>>>>>
>>>>>>>>>>> However, just adding -device vfio-pci,host=01:00.1 doesn't seem
>>>>>>>>>>> to do the trick.
>>>>>>>>>>> Of course the corresponding device is already blacklisted and
>>>>>>>>>>> bound to vfio.
>>>>>>>>>>>
>>>>>>>>>>> The Debian Wiki entry about VGA passthrough (
>>>>>>>>>>> https://wiki.debian.org/VGAPassthrough) mentions QEMU arguments
>>>>>>>>>>> like "-device
>>>>>>>>>>> vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=...
>>>>>>>>>>> -device vfio-pci,host=01:00.1,bus=pcie.0" which seems to address GPUs with
>>>>>>>>>>> audio devices, but if I try to do something similar, the buses 'root' and
>>>>>>>>>>> 'pcie' couldn't be found. Maybe I missed something very important?
>>>>>>>>>>>
>>>>>>>>>>> On the same article, it says that the "HDMI soundcard [...]
>>>>>>>>>>> needs to be unbound from its driver":
>>>>>>>>>>> # echo '0000:01:00.1' | sudo tee
>>>>>>>>>>> /sys/bus/pci/devices/0000:01:00.1/driver/unbind
>>>>>>>>>>> I figured the vfio-bind script from the Arch Linux Forum thread (
>>>>>>>>>>> https://bbs.archlinux.org/viewtopic.php?id=162768) would do
>>>>>>>>>>> exactly this thing, so I didn't explicitly do so for the audio device. Is
>>>>>>>>>>> that okay?
>>>>>>>>>>>
>>>>>>>>>>> Best regards,
>>>>>>>>>>> Ruben
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Am 18.01.2016 um 08:31 schrieb Alexander Petrenz:
>>>>>>>>>>>
>>>>>>>>>>> Hi Ruben,
>>>>>>>>>>>
>>>>>>>>>>> I guess your 750ti also has some audio device. You should pass
>>>>>>>>>>> through this too. It should be something like 01:00.1. There are many
>>>>>>>>>>> command line examples you can find about that.
>>>>>>>>>>> Also I´m not quite sure, if you should remove the x-vga=on.
>>>>>>>>>>>
>>>>>>>>>>> Regards
>>>>>>>>>>> Alex
>>>>>>>>>>>
>>>>>>>>>>> On Sun, Jan 17, 2016 at 11:12 PM, Ruben Felgenhauer <
>>>>>>>>>>> 4felgenh at informatik.uni-hamburg.de> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> I am trying to pass my nVidia GTX 750ti to my QEMU guest.
>>>>>>>>>>>>
>>>>>>>>>>>> Problem is: After the QEMU monitor pops up, nothing happens.
>>>>>>>>>>>> The GPU's output is dead, and the vm won't be accessible via SSH anymore,
>>>>>>>>>>>> so it's very likely that the VM isn't booting up at all. Also, there are no
>>>>>>>>>>>> error messages from QEMU on the console whatsoever which makes debugging it
>>>>>>>>>>>> especially hard.
>>>>>>>>>>>>
>>>>>>>>>>>> This is how I start the vm with normal vga emulation:
>>>>>>>>>>>> qemu-system-x86_64 -hda vm.ovl -boot c -enable-kvm -m 1024 -cpu
>>>>>>>>>>>> host,kvm=off -smp cores=4,threads=2 -redir tcp:5022::22
>>>>>>>>>>>> Everything runs fine in this case. To do the passthrough, I add
>>>>>>>>>>>> this:
>>>>>>>>>>>> -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on -vga
>>>>>>>>>>>> none
>>>>>>>>>>>> This brings said problems with it. I also tried out multiple
>>>>>>>>>>>> different combinations of -device's arguments or even adding a romfile for
>>>>>>>>>>>> the GPU, but none of these steps changed anything at all.
>>>>>>>>>>>>
>>>>>>>>>>>> Obviously, I am using a BIOS installation and I'm well-aware
>>>>>>>>>>>> with this bug:
>>>>>>>>>>>> https://bugzilla.kernel.org/show_bug.cgi?id=107561, but
>>>>>>>>>>>> neither using less RAM (as you can see I am using 1GB now) nor switching to
>>>>>>>>>>>> an older Kernel changed anything about the problem. I have tried Kernel
>>>>>>>>>>>> 4.1.0 and 4.3.0.
>>>>>>>>>>>>
>>>>>>>>>>>> Host is Debian testing with QEMU 2.5.0.
>>>>>>>>>>>> I tried both Debian and Windows 7 as a guest, but both are
>>>>>>>>>>>> showing exactly the same behaviour.
>>>>>>>>>>>> Mainboard is an ASUS Z87-PLUS. The 750ti is produced by ASUS
>>>>>>>>>>>> aswell.
>>>>>>>>>>>>
>>>>>>>>>>>> Any idea how I could get passthrough running?
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> vfio-users mailing list
>>>>>>>>>>>> vfio-users at redhat.com
>>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/vfio-users
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> vfio-users mailing list
>>>>>>>>>>> vfio-users at redhat.com
>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/vfio-users
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> vfio-users mailing list
>>>>>>>>> vfio-users at redhat.com
>>>>>>>>> https://www.redhat.com/mailman/listinfo/vfio-users
>>>>>>>>>
>>>>>>>>
>>>>>
>>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20160126/3338aba4/attachment.htm>