[vfio-users] GTX 580 Passthrough - Successes and Failures

Tue Feb 23 21:25:20 UTC 2016

I found the dip switches and they're all on. Damn. Although I've just
realized I still have an SLI cable on the cards. There's a good chance
that's causing some of my errors with the cards claiming each other...
On Feb 23, 2016 4:21 PM, "Ben J" <btpprograms at gmail.com> wrote:

> I wasn't even aware of those PCI switches. I'll look into it. To be clear,
> if I'm only passing through the card in the second slot, I shouldn't have
> any issues? What do those switches look like and where are they?
>
> I fixed my IO for the most part by switching over to a raw partition on my
> SSD and installing the redhat virtio drivers(if=virtio). I also set
> cache=none which removes some extra IO that you shouldn't need when passing
> through a drive. The freezing issue was on the latest virtio driver, when I
> switched over to stable my installs stopped pausing so I suspect there was
> something off with the latest driver.
>
> I also tried to pass the Asmedia SATA controller directly to the VM to get
> native performance but apparently it doesn't support boot drives at all.
> Anyway, once I installed the virtio drivers I got fairly close to the
> expected speeds on the drive, although my random read is still sitting in
> the 20s. I've since resolved the Windows Update issue by letting it run for
> around 40 hours straight, it's just extremely bloated but if you wait long
> enough it will eventually install the updates. I don't know why you'd have
> that issue on 10 though, I've only heard of it on 7.
>
> Thanks for the advice!
> On Feb 23, 2016 4:01 PM, "vfio" <vfio at taintedbit.com> wrote:
>
>> I have been experiencing the precise symptoms that you describe in this
>> thread and your previous thread. I have resolved the issue of the card not
>> resetting after running the VM once, but I have not investigated the I/O
>> issues yet. Have you since identified the issue with I/O?
>>
>> I thought I should reply since I have very similar hardware to you, but
>> with some differences. I've discovered that some of the GPU issues are
>> caused by a feature of our motherboards.
>>
>> My current setup:
>> Processor: i7-4960X
>> Graphics:
>>         GeForce GTX 580 (EVGA GeForce GTX 580 FTW Hydro Copper 2)
>>         GeForce 210 (ASUS GeForce 210 SILENT)
>>         GeForce GTX TITAN X (EVGA GeForce GTX TITAN X HYDRO COPPER)
>> RAM: 4x Corsair Vengeance Black, total 32 GB, set to 1333 MHz
>> Monitors:
>>         Monitor [1] hooked up to 580 DVI 1, and TITAN DVI
>>         Monitor [2] hooked up to 580 DVI 2
>>         Monitor [3] hooked up to 210 DVI
>> Motherboard: ASUS Rampage IV Extreme
>> Host OS: Debian (sid)
>> Guest OS: Windows 10 Education 64 Bit
>> Kernel: 4.4.0-1-amd64
>> Host Graphics Driver: nouveau 1.0.12
>>
>> Many of my problems with VGA passthrough have been caused by the ASUS
>> Rampage IV Extreme motherboard. The board has the following PCI slots:
>>         PCIE_X16_1
>>         PCIE_X8_2A
>>         PCIE_X8_2B
>>         PCIE_X16/8_3
>>         PCIEX1_1
>>         PCIE_X8_4
>>
>> One year ago, I had two GTX 580 cards---one in PCIE_X16_1 and another in
>> PXIE_X16/8_3. I put the 210 in PCIE_X8_4 and set it as the primary adapter,
>> with the intention of passing both 580s in SLi to the VM. Unfortunately,
>> this board CANNOT perform VGA passthrough with any card placed in
>> PCIE_X16_1 (we established this as part of the discussion in the Arch
>> forum). Since the motherboard does not have integrated graphics, it always
>> performs some initialization on the card in the first PCIE slot; even
>> pci-stub will not prevent the first card from being "claimed". To address
>> this problem, I bought the TITAN X and discarded one of the 580s (having
>> all three in at once was not physically possible).
>>
>> In my current setup, I have the 580 in PCIE_X16_1, the 210 in PCIE_X8_2A,
>> the TITAN in PCIE_X16/8_3, and a Blackmagic Intensity Pro (capture card) in
>> PCIE_X8_4. PCIE_X8_2B and PCIEX1_1 are empty.
>>
>> I mention all of these details because they are relevant to my problem
>> with the TITAN not being reset after launching the VM. The Rampage IV
>> Extreme includes a very handy feature: DIP switches that allow the PCI
>> lanes to be turned on and off. I was using these switches to quickly
>> "unplug" graphics cards while I was testing various configurations; since
>> my system uses open-loop liquid cooling, it is impractical to frequently
>> move the cards between slots.
>>
>> It turns out that VGA passthrough does not work well with the Rampage IV
>> Extreme's PCI lane switches.
>>
>> When I left all of the lanes turned on except for PCIE_X8_4 (I haven't
>> used the Blackmagic card for some time, so I just leave it off), I
>> experienced similar issues: "Invalid ROM contents" messages, errors about
>> not bringing the card out of the D3 power state, and a VM that only worked
>> a fraction of the time. Additionally, my TITAN card would appear to switch
>> between bus 03:00 and bus 04:00, even within the same session if I
>> "removed" the device and then ran /sys/bus/pci/rescan. These issues stopped
>> as soon as I turned all of the PCI lane DIP switches to the "on"
>> position---both with the Blackmagic card plugged in and with it physically
>> removed. The TITAN stopped switching between buses, and the VGA passthrough
>> started working 100% of the time, even without reboots. "systemctl suspend"
>> never resolved the issue for me.
>>
>> Hopefully this helps other people who may have had strange problems with
>> this motherboard. I am planning to reinstall the guest shortly, but I HAVE
>> experienced the same I/O issues as you have (i.e., the Windows installer
>> freezing on multiple "expanding files" percentages, and freezes when
>> performing Windows update). However, I have always been assigning a
>> physical HDD to the guest using "-drive file=/dev/sdf,format=raw". I would
>> be interested to hear if you identify the cause of this problem.
>>
>> _______________________________________________
>> vfio-users mailing list
>> vfio-users at redhat.com
>> https://www.redhat.com/mailman/listinfo/vfio-users
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20160223/d51ac96e/attachment.htm>