[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [vfio-users] GTX 580 Passthrough - Successes and Failures

I have been experiencing the precise symptoms that you describe in this thread and your previous thread. I have resolved the issue of the card not resetting after running the VM once, but I have not investigated the I/O issues yet. Have you since identified the issue with I/O?

I thought I should reply since I have very similar hardware to you, but with some differences. I've discovered that some of the GPU issues are caused by a feature of our motherboards.

My current setup:
Processor: i7-4960X
	GeForce GTX 580 (EVGA GeForce GTX 580 FTW Hydro Copper 2)
	GeForce 210 (ASUS GeForce 210 SILENT)
RAM: 4x Corsair Vengeance Black, total 32 GB, set to 1333 MHz
	Monitor [1] hooked up to 580 DVI 1, and TITAN DVI
	Monitor [2] hooked up to 580 DVI 2
	Monitor [3] hooked up to 210 DVI
Motherboard: ASUS Rampage IV Extreme
Host OS: Debian (sid)
Guest OS: Windows 10 Education 64 Bit
Kernel: 4.4.0-1-amd64
Host Graphics Driver: nouveau 1.0.12

Many of my problems with VGA passthrough have been caused by the ASUS Rampage IV Extreme motherboard. The board has the following PCI slots:

One year ago, I had two GTX 580 cards---one in PCIE_X16_1 and another in PXIE_X16/8_3. I put the 210 in PCIE_X8_4 and set it as the primary adapter, with the intention of passing both 580s in SLi to the VM. Unfortunately, this board CANNOT perform VGA passthrough with any card placed in PCIE_X16_1 (we established this as part of the discussion in the Arch forum). Since the motherboard does not have integrated graphics, it always performs some initialization on the card in the first PCIE slot; even pci-stub will not prevent the first card from being "claimed". To address this problem, I bought the TITAN X and discarded one of the 580s (having all three in at once was not physically possible).

In my current setup, I have the 580 in PCIE_X16_1, the 210 in PCIE_X8_2A, the TITAN in PCIE_X16/8_3, and a Blackmagic Intensity Pro (capture card) in PCIE_X8_4. PCIE_X8_2B and PCIEX1_1 are empty.

I mention all of these details because they are relevant to my problem with the TITAN not being reset after launching the VM. The Rampage IV Extreme includes a very handy feature: DIP switches that allow the PCI lanes to be turned on and off. I was using these switches to quickly "unplug" graphics cards while I was testing various configurations; since my system uses open-loop liquid cooling, it is impractical to frequently move the cards between slots.

It turns out that VGA passthrough does not work well with the Rampage IV Extreme's PCI lane switches.

When I left all of the lanes turned on except for PCIE_X8_4 (I haven't used the Blackmagic card for some time, so I just leave it off), I experienced similar issues: "Invalid ROM contents" messages, errors about not bringing the card out of the D3 power state, and a VM that only worked a fraction of the time. Additionally, my TITAN card would appear to switch between bus 03:00 and bus 04:00, even within the same session if I "removed" the device and then ran /sys/bus/pci/rescan. These issues stopped as soon as I turned all of the PCI lane DIP switches to the "on" position---both with the Blackmagic card plugged in and with it physically removed. The TITAN stopped switching between buses, and the VGA passthrough started working 100% of the time, even without reboots. "systemctl suspend" never resolved the issue for me.

Hopefully this helps other people who may have had strange problems with this motherboard. I am planning to reinstall the guest shortly, but I HAVE experienced the same I/O issues as you have (i.e., the Windows installer freezing on multiple "expanding files" percentages, and freezes when performing Windows update). However, I have always been assigning a physical HDD to the guest using "-drive file=/dev/sdf,format=raw". I would be interested to hear if you identify the cause of this problem.

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]