[vfio-users] Finding a suitable replacement mainboard for GPU passthrough

Martin Schrodt martin at schrodt.org
Wed Sep 28 22:36:56 UTC 2016


On 09/28/2016 10:10 PM, Alex Williamson wrote:
>     Oh, thanks for the hint, I did not know this before. I did not want to
>     ACS-override anyway, and thought there would be mainboard layouts that
>     would separate those 2 slots even on Z170. Is this not the case?
> 
> 
> This is not the case.  The processor does not support ACS itself and
> there is no configuration of processor root ports or downstream switches
> that can fix that.  On the consumer grade hardware only the PCH-based
> root ports can be configured for isolation.  Those will often still give
> you one or more x16 slots that can host a graphics card, but it is a
> longer path to memory and the CPU.

I see. So, can I pretty much assume that every X99 board on the market
has two PCIe-slots directly connected to the CPU, and isolated in
different IOMMU-groups?

>     If I try to replicate this, it'll go to sleep but not wake up on the
>     second command. Maybe that's also because of the passed through ASMedia
>     Device, I didn't test yet.
> 
> 
> Ok, that's a start, but I don't even have a lot of confidence in
> suspending and resuming the VM with a device assigned.

Hmm, could you explain in 3 short sentences why that is? I really would
like to deepen my knowledge.

Ah, and before I forget, the most important point for me was this (from
the first post):

4) When resuming from suspend, the BIOS (I presume, someone with another
board from ASRock has the same problems) fucks up TSC, which causes the
kernel to turn it off, and the VM to stutter badly afterwards. I have to
reboot to get a performing VM. (*2)

This is because on a fresh boot, my host decides to use TSC as it's
clock source.

[    0.074613] TSC deadline timer enabled
[    1.980617] tsc: Refined TSC clocksource calibration: 4007.999 MHz

When I suspend it, and wake it up again, *something* alters the TSC
counter, making my kernel reject it from now on:

[  440.964713] TSC synchronization [CPU#0 -> CPU#1]:
[  440.964713] Measured 6260509273 cycles TSC warp between CPUs, turning
off TSC clock.
[  440.964715] tsc: Marking TSC unstable due to check_tsc_sync_source failed

Then when I start the VM, kvm starts to rightfully complain:

[  463.175389] kvm: SMP vm created on host with unstable TSC; guest TSC
will not be reliable

and the VM (Windows 10) is unusable afterwards, until I reboot the host
again.

I can even take shutting down the VM before suspend, but I can't take
rebooting for every start of the VM.

I looked around and only found another person with the same problem on
another ASRock board, which makes me think this is something in their
bioses.

I need to get this solved, and I am even gonna spend money, if it's needed.






More information about the vfio-users mailing list