[vfio-users] Multi-gpu device passthrough, HD 6990

[Doug Johnson]

On Tue, 2 Jan 2018 18:35:49 +0100
Torbjorn Jansson <torbjorn.jansson at mbox200.swipnet.se> wrote:

> On 2018-01-02 12:41, Doug Johnson wrote:
> > 
> > Hi All,
> > 
> > I have an HD 6990 card which is a dual gpu card from 2011 (pre GCN) that I am trying to pass through with vfio.
> > 
> > I successfully passed both GPUs to a VM with QEMU, but seabios gets stuck on an option ROM (which I believe is attached to the second GPU). I can see the card initialize, though, so I know that works. If I specify an empty ROM file to the second GPU, the BIOS boots and I can get into windows. I can install drivers, but it appears that they only use one GPU (so, I cannot enable crossfile between them). I also downloaded a copy of the ROM file for my model (Gigabyte) from techpowerup.com and get the same behavior of getting stuck if I specify the file on both GPUs. The AMD drivers also, notably, don't seem to recognize the card properly saying the generic name of "HD 6900 Series," which I believe isn't the correct behavior, although I haven't tried it in a real windows machine to verify yet.
> > 
> > I suspect the issue may have to do with a PLX bridge chip that shows up in lspci (which is part of the card) but that isn't directly passed to the VM, and the GPU bios hangs trying to do something with that, but I am not entirely sure.
> > 
> > Attached is the output of my lspci -knn as well as my VM spinup script and the output of seabios debug... Any other relevant system details can be provided.
> > 
> > I am not sure where to go from here... Perhaps emulating the PCI topology of the card somehow? Maybe even there is a way to pass the PLX bridge chip?
> > 
> > Thanks for your help in advance
> 
> first of all i have no experience with radeon cards but as far as i know it is 
> not possible to get crossfire/sli working in a virtual machine.
> 
> also running cards with built in uefi support and os that uses uefi have a 
> higher chance of working since you avoid a lot of old legacy stuff.

SO I came across this blog post: https://nils.schimmelmann.us/post/154884462507/gaming-with-xdma-crossfire-passthrough-on-ubuntu

Which deals with getting crossfire working with XDMA which is a GCN 1.1 feature. The setup in the post also requires UEFI but alas this card is too old so I have to use legacy BIOS. I did noticed however the requirement that each GPU have its own root port switch:

-device ioh3420,port=0xe0,chassis=1,id=pcie.1,bus=pcie.0,multifunction=on,addr=0x1c \
-device ioh3420,port=0x18,chassis=2,id=pcie.2,bus=pcie.0,multifunction=on,addr=0x3 \

So I went ahead and modified my script with the following:

 -device ioh3420,port=0xe0,chassis=1,id=pcie.1,bus=pcie.0,multifunction=on,addr=0x1c \
 -device ioh3420,port=0xe0,chassis=2,id=pcie.2,bus=pcie.0,multifunction=on,addr=0x3 \
 -device vfio-pci,host=08:00.0,bus=pcie.1,addr=00.0,multifunction=on,x-vga=on \
 -device vfio-pci,host=08:00.1,bus=pcie.1,addr=00.1 \
 -device vfio-pci,host=09:00.0,bus=pcie.2,addr=00.0,multifunction=on,romfile= \
 -device vfio-pci,host=09:00.1,bus=pcie.2,addr=00.1 \

And to my surprise after windows booted and reinstalled the drivers with the new topology, it actually works. The drivers recognize one GPU as 'primary' and the other as 'linked.' I still had to specify an empty romfile on the secondary GPU, though, to avoid the lockup in seabios. I ran some benchmarks and it looks like the two GPUs are in fact running in tandem. 

-- 
Doug Johnson <dougvj at dougvj.net>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20180102/1faa56ce/attachment.sig>