[vfio-users] sr-iov support on main boards

Sat Mar 31 08:04:26 UTC 2018

On 2017-08-27 18:03, Torbjorn Jansson wrote:
> On 2017-08-22 19:38, Alex Williamson wrote:
>> On Mon, 21 Aug 2017 23:09:55 +0200
>> Torbjorn Jansson <torbjorn.jansson at mbox200.swipnet.se> wrote:
>>
>>> On 2017-08-21 19:56, Torbjorn Jansson wrote:
>>>> On 2017-08-19 07:37, Torbjorn Jansson wrote:
>>>>> On 2017-08-19 02:34, Taiidan at gmx.com wrote:
>>>>>> On 08/15/2017 02:12 AM, Torbjorn Jansson wrote:
>>>>>>> On 2017-08-15 02:44, Taiidan at gmx.com wrote:
>>>>>>>> On 08/14/2017 12:28 AM, Torbjorn Jansson wrote:
>>>>>>>>> yes i noticed that part was missing, but the card i have is an I350-T2
>>>>>>>>> and according to:
>>>>>>>>> https://www.intel.com/content/www/us/en/support/network-and-i-o/ethernet-products/000005722.html 
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> the I350-T2 is supposed to have sr-iov support.
>>>>>>>>> i guess it is possible they sent me the wrong type of card or something,
>>>>>>>>> i'll double check what is listed on the card itself.
>>>>>>>> Some OEM's (ex: dell) shut off the SR-IOV ability on their NIC's because
>>>>>>>> reasons.
>>>>>>>> Please provide an lspci -n and lspci -v -v for that device so we can see
>>>>>>>> what OEM it is.
>>>>>>>>
>>>>>>>> I would return it and have them send you a real one that isn't nerfed,
>>>>>>>> also FYI most NIC's on ebay are counterfeit so be careful what you buy.
>>>>>>>> https://www.servethehome.com/investigating-fake-intel-i350-network-adapters/ 
>>>>>>>>
>>>>>>>> (real intel NIC ASIC and SR-IOV but crappy secondary components)
>>>>>>>
>>>>>>> if i look at the back of the card there is a sticker with: I350T2V2
>>>>>>>
>>>>>>> here is the output:
>>>>>> Weird.
>>>>>> I would say you almost definitely got a fake - either way send it back.
>>>>>
>>>>> yes very strange.
>>>>> thing is, i bought it from a well known web shop I've been using forever.
>>>>> i have opened a support ticket with them about this, hopefully i can 
>>>>> return it.
>>>>>
>>>>> but then question is, how do i find a network card with sr-iov that works?
>>>>> clearly specs on web shops can't always be trusted.
>>>>>
>>>>> also strange is that this page:
>>>>> http://ark.intel.com/products/84804/Intel-Ethernet-Server-Adapter-I350-T2V2
>>>>> clearly lists sr-iov and this is the model i got.
>>>>
>>>> i did a bit more searching and checking.
>>>> found this pdf:
>>>> https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/ethernet-controller-i350-datasheet.pdf 
>>>>
>>>>
>>>> section 6.2.25 looks interesting specifically the IOV Enable bit.
>>>>
>>>> i used ethtool to dump the eeprom to file and a hex editor to check the
>>>> contents and this bit is set to a zero indicating IOV is not exposed.
>>>>
>>>> but i'm definitely no expert on these things so i might be wrong too.
>>>> but i wonder if there is a way to flip this bit and see what happens.
>>>
>>> i fixed it :)
>>> i figured out how to use ethtool to flip the bit, rebooted and:
>>> -------
>>> 03:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection
>>> (rev 01)
>>>           Subsystem: Intel Corporation Ethernet Server Adapter I350-T2
>>>           Physical Slot: 6-1
>>>           Flags: bus master, fast devsel, latency 0, IRQ 61, NUMA node 0
>>>           Memory at fad00000 (32-bit, non-prefetchable) [size=1M]
>>>           Memory at fae04000 (32-bit, non-prefetchable) [size=16K]
>>>           Capabilities: [40] Power Management version 3
>>>           Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
>>>           Capabilities: [70] MSI-X: Enable+ Count=10 Masked-
>>>           Capabilities: [a0] Express Endpoint, MSI 00
>>>           Capabilities: [100] Advanced Error Reporting
>>>           Capabilities: [140] Device Serial Number a0-36-9f-ff-ff-ed-43-b0
>>>           Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
>>>           Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
>>>           Capabilities: [1a0] Transaction Processing Hints
>>>           Capabilities: [1c0] Latency Tolerance Reporting
>>>           Capabilities: [1d0] Access Control Services
>>>           Kernel driver in use: igb
>>>           Kernel modules: igb
>>> -------
>>> as you can see above sr-iov is now listed in lspci, needed sriov entries is now
>>> listed under /sys and i can enable the virtual cards.
>>> had to blacklist igbvf and libvirt kept complaining until i added the virtual
>>> cards to vfio too.
>>>
>>> so now i finally got it all working and have a vm with a virtual card.
>>>
>>> but i think it is really stupid of the manufacture(s) (intel?) to disable
>>> sr-iov in the eeprom and not provide any proper tool or documentation on how to
>>> enable it.
>>
>> Would you be so kind as to write-up instructions how you used ethtool
>> to resolve this so that others don't need to re-invent your solution?
>> Thanks,
>>
> 
> yes of course.
> 
> first of all, be careful when you poke around in the cards eeprom.
> below is what i did and it worked on my I350-T2V2 card.
> theoretically it might work on other I350 cards too but it is not something i 
> have tested.
> 
> before starting you need to know the device name of your ethernet device, it is 
> also useful to check output of lspci -v on your device and check that your card 
> is missing this line:
> Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
> 
> first make a backup of your eeprom:
> ethtool --eeprom-dump DEVNAME raw on >eth-eeprom.bin
> 
> i used this both as backup and to check the contents with a hex editor and to 
> make sure i understood the documentation correctly regarding endianess and 
> offsets.
> 
> next we want to check what the iov enable bit is currently set to.
> according to documentation i found, this is bit zero of word 0x25 in eeprom.
> so this makes byte offset 0x4a (0x25*2 documentation uses 16 bit words so we 
> need to multiply by 2 to get byte offset)
> so:
> 
> ethtool --eeprom-dump DEVNAME offset 0x4a length 1
> 
> in my case i got 0xf6 back.
> so bit zero is set to a zero which means IOV capability is not exposed 
> (expected based on earlier lspci -v).
> there is no point in continuing unless it is a zero.
> 
> take above value you just read out and set bit zero to 1 and then use your 
> newly calculated value in below command to change it:
> 
> ethtool --change-eeprom DEVNAME magic 0x15218086 offset 0x4a length 1 value 
> 0xYOUR_VALUE_GOES_HERE
> 
> the magic value to ethtool is i think deliberately not documented to prevent 
> you from accidentally screwing up your eeprom.
> in this case it is the pci vendor and device id (lspci -n).
> 
> once this is done, reboot computer and check output of lspci -v and look for:
> Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
> if everything worked out it should now show up and you can go ahead with the 
> rest of sr-iov configuration.
> 
> again, be careful when you do this.
> 
> 
> 
> if you need to use this then i would recommend contacting intel support about 
> this because i think this is not the proper way of doing things.
> i consider this more of a "hack" or a workaround than a proper solution.
> i would rather have a proper tool or documentation or something from intel 
> specifying what to expect regarding IOV enable/disable on these cards.
> 
> i found no documentation or tool to enable this or i would not have done this 
> via ethtool.
> 
> it is not fun to waste weeks on troubleshooting sr-iov only to find out that it 
> was disabled in the cards eeprom.
> it also doesn't help that when you lookup specs for this card it clearly lists 
> that it supports sr-iov, even the place i bought it from and their support said 
> it supported sr-iov (incorrectly, kind of)
> 

FYI the changes appear to not stick and have to be reapplied sometimes.
most likely when power is removed values goes away.
This happened to be several times.

Maybe something extra is needed to make it permanent.