[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [libvirt] two hostdev devices problem



Hi,

> First - I just got confirmation from the libnl maintainer that the bug I
> mentioned causing problems with max_vfs > 50 *is* an issue in libnl
> 3.2.16 - it is fixed in libnl 3.2.22. Try to upgrade to that version and
> it should solve at least some of your problems.

Upgrade libnl to version 3.2.22 helps.
Thanks.

> 1) when you have max_vfs = 10, you can assign the devices and mac
> addresses are set properly, i.e. everything works.
> 2) when you have max_vfs = 35, you can assign the devices, but the mac
> addresses are not set properly.
> 3) when you have max_vfs = 63, you can't assign the devices because you
> get this error:
>
>         internal error missing IFLA_VF_INFO in netlink response
>
> Is this correct?

Yes.
Before upgrade libnl :-)
Upgrede helps only for limit max_vfs.

> (BTW, it may get a bit confusing if you leave your networks named as
> "vnet0" and "vnet1" - although it's a different namespace, libvirt uses
> "vnetN" (where n is 0 - whatever) as the name for the tap device
> associated with each guest interface).

You're right.
I changed this.

> the output of "ip link show dev $PF".
Hmm,
> ip link show dev eth0
Shows only:
2: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq
master bond0 state UP qlen 1000
    link/ether b8:ca:3a:5b:a6:3a brd ff:ff:ff:ff:ff:ff

I see VFs like network interfaces eth2-eth127.
When VF is attached to VM it disappear from list on host.

--
Regards
Dominik


2013/5/24 Laine Stump <laine laine org>:
> On 05/24/2013 09:16 AM, Dominik Mostowiec wrote:
>
>>>> Do you have this problem with mac addresses when max_vfs is set to an
>> even lower number?
>
>
> First - I just got confirmation from the libnl maintainer that the bug I
> mentioned causing problems with max_vfs > 50 *is* an issue in libnl
> 3.2.16 - it is fixed in libnl 3.2.22. Try to upgrade to that version and
> it should solve at least some of your problems.
>
>
>>
>> No,
>> When max_fs=10, macs in VM are OK.
>
>
> Interesting.
>
> So now I'm curious about something - can you try adding a vlan tag to
> your networks and see if the vlan tag is set properly when max_vfs is a
> low number (10 or 7):
>
> <network>
>   <name>vnet0</name>
>   <uuid>ec49896a-a0b5-4944-a81f-9f0cdf578871</uuid>
>   <vlan>
>     <tag id='42'/>
>   </vlan>
>   <forward mode='hostdev' managed='yes'>
>     <pf dev='eth0'/>
>   </forward>
> </network>
>
>
> (you will have to do "virsh net-destroy vnet0; virsh net-start vnet0"
> after making the modification to the network). If the vlan tag is
> successfully set, it will show up next to the mac address for the VF in
> the output of "ip link show dev $PF".
>
> (BTW, it may get a bit confusing if you leave your networks named as
> "vnet0" and "vnet1" - although it's a different namespace, libvirt uses
> "vnetN" (where n is 0 - whatever) as the name for the tap device
> associated with each guest interface).
>
> To summarize:
>
> 1) when you have max_vfs = 10, you can assign the devices and mac
> addresses are set properly, i.e. everything works.
> 2) when you have max_vfs = 35, you can assign the devices, but the mac
> addresses are not set properly.
> 3) when you have max_vfs = 63, you can't assign the devices because you
> get this error:
>
>         internal error missing IFLA_VF_INFO in netlink response
>
> Is this correct?
>
>>
>> Phisical interfaces:
>>> ifconfig eth0
>> eth0      Link encap:Ethernet  HWaddr b8:ca:3a:5b:a6:38
>>           UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
>
> Okay, so that's not the problem.
>
>>
>>> ifconfig eth1
>> eth1      Link encap:Ethernet  HWaddr b8:ca:3a:5b:a6:38
>>           UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
>>
>>>> Hmm. ixgbe. I just got a report yesterday that setting of the vlan tag
>>>> for ixgbe vfs was not working properly (although it works fine for igb
>>>> devices)
>> Domain config:
>> <interface type="network">
>>      <alias name="hostdev0"/>
>>      <source network="vnet0"/>
>>      <mac address="52:54:0a:b1:48:fc"/>
>> </interface>
>> <interface type="network">
>>      <alias name="hostdev1"/>
>>      <source network="vnet1"/>
>>      <mac address="52:54:0a:b1:48:fc"/>
>>  </interface>
>>
>> Networks defined:
>> <network>
>>   <name>vnet0</name>
>>   <uuid>ec49896a-a0b5-4944-a81f-9f0cdf578871</uuid>
>>   <forward mode='hostdev' managed='yes'>
>>     <pf dev='eth0'/>
>>   </forward>
>> </network>
>>
>> <network>
>>   <name>vnet1</name>
>>   <uuid>6c8319e8-8b53-4382-9f4e-2400819b00a9</uuid>
>>   <forward mode='hostdev' managed='yes'>
>>     <pf dev='eth1'/>
>>   </forward>
>> </network>
>>
>> Regards
>> Dominik
>>
>> 2013/5/24 Laine Stump <laine laine org>:
>>> On 05/24/2013 04:45 AM, Dominik Mostowiec wrote:
>>>> Hi,
>>>> I have libnl3, kernel 3.8.8, os ubuntu 12.04, net:intel10G.
>>>> Upgrade libnl to 3.2.16 not helps,
>>>>
>>>> I don't know - this is probably another problem, when I have set
>>>> max_vfs=35,35 VF is attaching to VM but mac in VM is not set
>>>> propertly.
>>> Do you have this problem with mac addresses when max_vfs is set to an
>>> even lower number?
>>>
>>> Ah, another thing - did you "ifconfig up" the PF before you started the
>>> guest?
>>>> Libvirt probably set this macs because in syslog:
>>>> May 23 14:48:59 on-10-177-32-62 kernel: [12574.171792] ixgbe
>>>
>>> Hmm. ixgbe. I just got a report yesterday that setting of the vlan tag
>>> for ixgbe vfs was not working properly (although it works fine for igb
>>> devices)
>>>
>>>
>>>> 0000:01:00.0: setting MAC 52:54:0a:b1:48:f7 on VF 0
>>>> May 23 14:48:59 on-10-177-32-62 kernel: [12574.171799] ixgbe
>>>> 0000:01:00.0: Reload the VF driver to make this change effective.
>>>> May 23 14:48:59 on-10-177-32-62 kernel: [12574.175054] ixgbe
>>>> 0000:01:00.1: setting MAC 52:54:0a:b1:48:f7 on VF 0
>>>> May 23 14:48:59 on-10-177-32-62 kernel: [12574.175060] ixgbe
>>>> 0000:01:00.1: Reload the VF driver to make this change effective.
>>>> May 23 14:48:59 on-10-177-32-62 kernel: [12574.307172] pci-stub
>>>> 0000:01:12.4: enabling device (0000 -> 0002)
>>>> May 23 14:49:00 on-10-177-32-62 kernel: [12575.662890] assign device 0:1:12.4
>>>> May 23 14:49:00 on-10-177-32-62 kernel: [12575.665210] pci-stub
>>>> 0000:01:12.5: enabling device (0000 -> 0002)
>>>> May 23 14:49:00 on-10-177-32-62 kernel: [12575.765609] assign device 0:1:12.5
>>>> But in VM i have another macs probably old from VF.
>>>>
>>>> ---
>>>> Regards
>>>> Dominik
>>>>
>>>>
>>>>
>>>> 2013/5/24 Laine Stump <laine laine org>:
>>>>> On 05/22/2013 06:32 PM, Dominik Mostowiec wrote:
>>>>>> I tested on 1.0.5 patched version and vm with 2 vfs working fine.
>>>>>>
>>>>>> I have another problem
>>>>>> When i set max_vfs=63:
>>>>>> internal error missing IFLA_VF_INFO in netlink response
>>>>>>
>>>>>> Its working when max_vfs=31
>>>>>>
>>>>> What is the platform, kernel version, and what version of libnl are you
>>>>> using (libnl 1 or libnl3?)
>>>>>
>>>>> There have been multiple bugs recently filed and fixed in this area on
>>>>> RHEL6/CentOS6. In particular, there was a problem with max_vfs > 50 that
>>>>> had exactly this symptom. It was solved by a patch to libnl-1.1. If your
>>>>> platform uses libnl-1.1, this could be the source of the problem. It is
>>>>> fixed in the upstream libnl-1.1.4 maintenance release.
>>>>>
>>>>> I unfortunately am not sure where the libnl-1.1 git is (I only have the
>>>>> git tree for libnl3), so I can't give you the exact commit message, but
>>>>> in short the problem was that libnl was using too small of a buffer for
>>>>> the netlink socket, so when there were a lot of vfs, the netlink message
>>>>> containing vf info was truncated.
>>>>>
>>>>>
>>>>>> --
>>>>>> Dominik
>>>>>>
>>>>>> 21 maj 2013 15:19, "Dominik Mostowiec" <dominikmostowiec gmail com
>>>>>> <mailto:dominikmostowiec gmail com>> napisał(a):
>>>>>>
>>>>>>     Hmm,
>>>>>>     It seems to be working (after only simple tests).
>>>>>>
>>>>>>     --
>>>>>>     Dominik
>>>>>>
>>>>>>
>>>>>>     2013/5/21 Ján Tomko <jtomko redhat com <mailto:jtomko redhat com>>
>>>>>>
>>>>>>         On 05/21/2013 01:37 PM, Laine Stump wrote:
>>>>>>         > On 05/21/2013 04:03 AM, Ján Tomko wrote:
>>>>>>         >> On 05/21/2013 09:32 AM, Dominik Mostowiec wrote:
>>>>>>         >>> hi,
>>>>>>         >>> I try to add 2 VF by "hostdev".
>>>>>>         >>> Networks (vnet0, vnet1) with:
>>>>>>         >>>  <forward mode='hostdev' managed='yes'>
>>>>>>         >>>     <pf dev='eth1'/>
>>>>>>         >>> .....
>>>>>>         >>>
>>>>>>         >>> Domain:
>>>>>>         >>> <interface type="network">
>>>>>>         >>>              <source network="vnet0"/>
>>>>>>         >>> ....
>>>>>>         >>> <interface type="network">
>>>>>>         >>>              <source network="vnet1"/>
>>>>>>         >>> ....
>>>>>>         >>>
>>>>>>         >>> virsh create error:
>>>>>>         >>> "error: internal error process exited while connecting to
>>>>>>         monitor: kvm:
>>>>>>         >>> -device
>>>>>>         pci-assign,configfd=25,host=01:10.1,id=hostdev0,bus=pci.0,addr=0x4:
>>>>>>         >>> Duplicate ID 'hostdev0' for device"
>>>>>>         >>>
>>>>>>         >> Hi,
>>>>>>         >>
>>>>>>         >> it seems we have been assigning the same id to all network
>>>>>>         hostdevs until this
>>>>>>         >> recent commit (not yet released):
>>>>>>         >>
>>>>>>         >> http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=6597cc25
>>>>>>         >
>>>>>>         >
>>>>>>         > If that patch has such an effect, it's purely accidental.
>>>>>>         Have you
>>>>>>         > tested this? (I plan to test a before and after as soon as
>>>>>>         I've had
>>>>>>         > breakfast)
>>>>>>         >
>>>>>>
>>>>>>         I haven't tested it and looking at the code again, I might've
>>>>>>         been wrong :(
>>>>>>
>>>>>>         Sorry about that.
>>>>>>
>>>>>>         Jan
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>     --
>>>>>>     Pozdrawiam
>>>>>>     Dominik
>>>>>>
>>>>
>>
>>
>> --
>> Pozdrawiam
>> Dominik
>>
>



--
Pozdrawiam
Dominik


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]