[vfio-users] SRIOV & amdgpu

[Ward Poelmans]

On 19-10-16 16:33, Alex Williamson wrote:
> On Wed, 19 Oct 2016 16:03:59 +0200
> Ward Poelmans <ward.poelmans at ugent.be> wrote:
>>
>> I've looked at the AMDGPU module and I think it checks a table in the
>> vbios to determine if it can do SRIOV. Upon testing it, the flag 'vBIOS
>> is sr-iov ready' is set but the flag 'sr-iov is enabled on this GPU' is not.
>>
>> Anybody any idea how to activate this? It works with vmware, so I should
>> be possible to do it with qemu too.
> 
> Assuming they've implemented standard SR-IOV support in Linux, you
> should find several sriov files in /sys/bus/pci/devices/$ADDRESS/,
> namely srio_numvfs and sriov_totalvfs.  totalvfs should match the
> SR-IOV capability and report 16, numvfs will report 0 and you can echo
> values into numvfs to set the number of enabled VFs.  Note that due to
> the way SR-IOV works, it is not possible change from one non-zero
> number of VFs to another, for example adding one if VFs are already
> enabled.  numvfs must be set to zero before any new non-zero value can
> be set.  Let us know what you find, I haven't handled these cards
> myself nor heard of users who have.  Thanks,

Hi Alex,

I've already tried that and it doesn't work. The files are there but
upon writing you get:
pci 0000:04:00.0: Driver doesn't support SRIOV configuration via sysfs

I've been looking in the amdgpu driver and the header 'amdgpu_virt.h'
[1] seems to point to some form of support. Currently, the virtual_caps
is set to 1 (which means the vbios support sr-iov but it's not enabled).

So, I've tried to write to the pcie register to flip the sr-iov to enabled:
setpci -v -s 04:00.0 ECAP_SRIOV+8.B=19
gives us:
IOVCtl:	Enable+ Migration- Interrupt- MSE+ ARIHierarchy+

Using the 'srio_numvfs' file still fails but if we now load the amdgpu
driver, the second bit of virtual_caps is also on (meaning sr-iov is
enabled). However, the driver fails with 'Fatal error during GPU init'
(full output at [2]). It seems like the module might be doing too much?
I think it should not do a full Post init if you want virtual functions?

We've also had a look at what AMD provides to activate it with vmware
but there seems hardly to be anything there. It has some pci id's and
some python scripts, but no goodies.

Any other ideas?

Ward


[1]
https://github.com/torvalds/linux/blob/master/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h#L27
[2] https://gist.github.com/wpoely86/7ec963415a91acae8148f2092790abd4