[vfio-users] GTX 580 Passthrough - Successes and Failures

[Ben J]

Hi, I got some help here a while ago on passing through one of my GTX
580's, since then I've made a lot of progress.

My setup:
Processor: I7-3960X
Graphics: 2 NVIDIA GTX 580's
RAM: Corsair Dominator GT, 16 GB, 1866 MHz
Monitors: 1 Monitor hooked up to the host 580, 1 monitor hooked up to both
580's
Motherboard: ASUS Rampage IV Extreme
Host OS: Arch Linux
Guest OS: Windows 7 Ultimate 64 Bit
Kernel: Standard, 4.4.1
Host Graphics Driver: nvidia 361.18

My current launch script:
QEMU_PA_SAMPLES=4096 QEMU_AUDIO_DRV=pa qemu-system-x86_64 -enable-kvm -m
8192 -cpu host,kvm=off \
-smp 4,sockets=1,cores=4,threads=1 \
-bios /usr/share/qemu/bios.bin  -vga none \
-device vfio-pci,host=02:00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=02:00.1 \
-usb -usbdevice host:046d:c534 \
-soundhw hda \
-boot menu=on \
-device virtio-scsi-pci,id=scsi \
-drive file=/mnt/4TB/Windows7Ultimate64Bit.iso,id=isocd,format=raw,if=none
-device scsi-cd,drive=isocd \
-drive file=/mnt/4TB/windows7.raw,id=disk,format=raw,if=none,cache=none
-device scsi-hd,drive=disk \
-drive
file=/home/benny/ISOs/Windows/Drivers/virtio.iso,id=virtiocd,if=none,format=raw
-device ide-cd,bus=ide.1,drive=virtiocd

Other details:
The GTX 580 gives the error "Cannot read device rom at 0000:02:00.0" when
trying to restart the VM for a second time. This happens on the second
launch with or without the NVIDIA driver being launched on the other card.
I'm able to get around the issue by using "systemctl suspend" to power
cycle the card; the VM boots fine after I wake up the computer from the
suspend. I've seen a lot of people having issues with reset issues on their
cards so I bet that this could be useful for them.

So in my tests so far I've gotten Windows 7 to install on a qcow2 partition
using a Windows ISO file. However, it took me 4 to 5 reboots to get the
install to finish, on most attempts it would freeze on "Expanding Windows
Files X%" at a different percent each time. When this happens the installer
still moves so it isn't completely frozen. I also tried installing by
passing through /dev/sr0 with a Windows 7 Professional disk inserted but I
had the same result. Once I got Windows successfully installed I had
various hangs while installing DirectX and .Net and trying to run Windows
Update. These hangs allow mouse movement within the guest but nothing else
responds and Control + Alt + Del doesn't work. The NVIDIA drivers installed
fine as did Steam. I've experienced no instability except for during
installs so I suspect that I'm having some kind of issues with disk I/O.
I'm assuming that the same issue that I'm getting on my qcow2 partition is
causing the Windows installer to hang so I've been trying to tweak my VM to
get another successful install onto a fresh RAW partition since I don't
need the extra features from qcow2.

I've tried the following to improve the disk I/O on the VM:
Put both the install ISO and the raw image on a hybrid SSHD

Checked the permissions on both images

Writeback and No Caching modes on the raw image

Changing the amount of cores between 1, 4, and 6. My CPU actually has 6

Changing the amount of threads to 1 and 2

Adding and removing RAM in various amounts (I have 16 GB)


At this point I'm running out of ideas. I'm not sure where the logs are
stored for the VM, I've checked the tmp directory but it seems that logs
are only made when the VM is run using virt-manager? No error messages are
being put out through the terminal or QEMU monitor.

I have an SSD which is currently holding my Windows partition. Would there
be a large benefit to putting the raw image on that drive? How does the
performance compare between placing the raw image on its filesystem, using
dd to put the raw image directly onto the drive, and passing through a sata
controller with that drive attached? I could probably do any of the 3 if it
would make a significant performance difference.

Thanks for any help in advance. Sorry for the long post, I just want to
make sure that I don't leave any important details out.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20160216/82191388/attachment.htm>