[vfio-users] SRIOV & amdgpu

[Alex Williamson]

On Thu, 20 Oct 2016 13:01:07 +0200
Ward Poelmans <ward.poelmans at ugent.be> wrote:

> On 19-10-16 16:33, Alex Williamson wrote:
> > On Wed, 19 Oct 2016 16:03:59 +0200
> > Ward Poelmans <ward.poelmans at ugent.be> wrote:  
> >>
> >> I've looked at the AMDGPU module and I think it checks a table in the
> >> vbios to determine if it can do SRIOV. Upon testing it, the flag 'vBIOS
> >> is sr-iov ready' is set but the flag 'sr-iov is enabled on this GPU' is not.
> >>
> >> Anybody any idea how to activate this? It works with vmware, so I should
> >> be possible to do it with qemu too.  
> > 
> > Assuming they've implemented standard SR-IOV support in Linux, you
> > should find several sriov files in /sys/bus/pci/devices/$ADDRESS/,
> > namely srio_numvfs and sriov_totalvfs.  totalvfs should match the
> > SR-IOV capability and report 16, numvfs will report 0 and you can echo
> > values into numvfs to set the number of enabled VFs.  Note that due to
> > the way SR-IOV works, it is not possible change from one non-zero
> > number of VFs to another, for example adding one if VFs are already
> > enabled.  numvfs must be set to zero before any new non-zero value can
> > be set.  Let us know what you find, I haven't handled these cards
> > myself nor heard of users who have.  Thanks,  
> 
> Hi Alex,
> 
> I've already tried that and it doesn't work. The files are there but
> upon writing you get:
> pci 0000:04:00.0: Driver doesn't support SRIOV configuration via sysfs
> 
> I've been looking in the amdgpu driver and the header 'amdgpu_virt.h'
> [1] seems to point to some form of support. Currently, the virtual_caps
> is set to 1 (which means the vbios support sr-iov but it's not enabled).
> 
> So, I've tried to write to the pcie register to flip the sr-iov to enabled:
> setpci -v -s 04:00.0 ECAP_SRIOV+8.B=19
> gives us:
> IOVCtl:	Enable+ Migration- Interrupt- MSE+ ARIHierarchy+
> 
> Using the 'srio_numvfs' file still fails but if we now load the amdgpu
> driver, the second bit of virtual_caps is also on (meaning sr-iov is
> enabled). However, the driver fails with 'Fatal error during GPU init'
> (full output at [2]). It seems like the module might be doing too much?
> I think it should not do a full Post init if you want virtual functions?
> 
> We've also had a look at what AMD provides to activate it with vmware
> but there seems hardly to be anything there. It has some pci id's and
> some python scripts, but no goodies.
> 
> Any other ideas?

Nope, seems like AMD hasn't bothered to enable SR-IOV support for Linux
yet, there are clearly no calls to pci_enable_sriov() from the amdgpu
driver.  There's typically PF driver integration work for VF support,
enabling SR-IOV in the PF capability might make VFs appear on the bus,
but they're not likely to work.  Complain to AMD I guess.  Thanks,

Alex