[vfio-users] sr-iov support on main boards

[Torbjorn Jansson]

On 2017-08-21 19:56, Torbjorn Jansson wrote:
> On 2017-08-19 07:37, Torbjorn Jansson wrote:
>> On 2017-08-19 02:34, Taiidan at gmx.com wrote:
>>> On 08/15/2017 02:12 AM, Torbjorn Jansson wrote:
>>>
>>>> On 2017-08-15 02:44, Taiidan at gmx.com wrote:
>>>>> On 08/14/2017 12:28 AM, Torbjorn Jansson wrote:
>>>>>
>>>>>> yes i noticed that part was missing, but the card i have is an I350-T2 
>>>>>> and according to:
>>>>>> https://www.intel.com/content/www/us/en/support/network-and-i-o/ethernet-products/000005722.html 
>>>>>>
>>>>>>
>>>>>> the I350-T2 is supposed to have sr-iov support.
>>>>>> i guess it is possible they sent me the wrong type of card or something, 
>>>>>> i'll double check what is listed on the card itself.
>>>>> Some OEM's (ex: dell) shut off the SR-IOV ability on their NIC's because 
>>>>> reasons.
>>>>> Please provide an lspci -n and lspci -v -v for that device so we can see 
>>>>> what OEM it is.
>>>>>
>>>>> I would return it and have them send you a real one that isn't nerfed, 
>>>>> also FYI most NIC's on ebay are counterfeit so be careful what you buy.
>>>>> https://www.servethehome.com/investigating-fake-intel-i350-network-adapters/ 
>>>>> (real intel NIC ASIC and SR-IOV but crappy secondary components)
>>>>>
>>>>
>>>> if i look at the back of the card there is a sticker with: I350T2V2
>>>>
>>>> here is the output:
>>> Weird.
>>> I would say you almost definitely got a fake - either way send it back.
>>>
>>
>> yes very strange.
>> thing is, i bought it from a well known web shop I've been using forever.
>> i have opened a support ticket with them about this, hopefully i can return it.
>>
>> but then question is, how do i find a network card with sr-iov that works?
>> clearly specs on web shops can't always be trusted.
>>
>> also strange is that this page:
>> http://ark.intel.com/products/84804/Intel-Ethernet-Server-Adapter-I350-T2V2
>> clearly lists sr-iov and this is the model i got.
>>
> 
> i did a bit more searching and checking.
> found this pdf:
> https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/ethernet-controller-i350-datasheet.pdf 
> 
> section 6.2.25 looks interesting specifically the IOV Enable bit.
> 
> i used ethtool to dump the eeprom to file and a hex editor to check the 
> contents and this bit is set to a zero indicating IOV is not exposed.
> 
> but i'm definitely no expert on these things so i might be wrong too.
> but i wonder if there is a way to flip this bit and see what happens.
> 

i fixed it :)
i figured out how to use ethtool to flip the bit, rebooted and:
-------
03:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection 
(rev 01)
         Subsystem: Intel Corporation Ethernet Server Adapter I350-T2
         Physical Slot: 6-1
         Flags: bus master, fast devsel, latency 0, IRQ 61, NUMA node 0
         Memory at fad00000 (32-bit, non-prefetchable) [size=1M]
         Memory at fae04000 (32-bit, non-prefetchable) [size=16K]
         Capabilities: [40] Power Management version 3
         Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
         Capabilities: [70] MSI-X: Enable+ Count=10 Masked-
         Capabilities: [a0] Express Endpoint, MSI 00
         Capabilities: [100] Advanced Error Reporting
         Capabilities: [140] Device Serial Number a0-36-9f-ff-ff-ed-43-b0
         Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
         Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
         Capabilities: [1a0] Transaction Processing Hints
         Capabilities: [1c0] Latency Tolerance Reporting
         Capabilities: [1d0] Access Control Services
         Kernel driver in use: igb
         Kernel modules: igb
-------
as you can see above sr-iov is now listed in lspci, needed sriov entries is now 
listed under /sys and i can enable the virtual cards.
had to blacklist igbvf and libvirt kept complaining until i added the virtual 
cards to vfio too.

so now i finally got it all working and have a vm with a virtual card.

but i think it is really stupid of the manufacture(s) (intel?) to disable 
sr-iov in the eeprom and not provide any proper tool or documentation on how to 
enable it.