[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [libvirt] two hostdev devices problem



Hi,
Mabye that's the problem ? :
May 27 09:13:44 on-10-177-32-62 kernel: [140204.533733] ixgbe
0000:01:00.0 eth0: VF Reset msg received from vf 13
May 27 09:13:44 on-10-177-32-62 kernel: [140204.533970] ixgbe
0000:01:00.0: VF 13 has no MAC address assigned, you may have to
assign one manually
May 27 09:13:44 on-10-177-32-62 kernel: [140204.552220] ixgbe
0000:01:00.1 eth1: VF Reset msg received from vf 13
May 27 09:13:44 on-10-177-32-62 kernel: [140204.552228] ixgbe
0000:01:00.1: VF 13 has no MAC address assigned, you may have to
assign one manually
May 27 09:13:45 on-10-177-32-62 kernel: [140205.674551] ixgbe
0000:01:00.1 eth1: VF Reset msg received from vf 12
May 27 09:13:45 on-10-177-32-62 kernel: [140205.694783] pci-stub
0000:01:13.1: irq 288 for MSI/MSI-X
May 27 09:13:45 on-10-177-32-62 kernel: [140205.726747] pci-stub
0000:01:13.1: irq 288 for MSI/MSI-X
May 27 09:13:45 on-10-177-32-62 kernel: [140205.726766] pci-stub
0000:01:13.1: irq 289 for MSI/MSI-X
May 27 09:13:45 on-10-177-32-62 kernel: [140205.786692] pci-stub
0000:01:13.1: irq 288 for MSI/MSI-X
May 27 09:13:45 on-10-177-32-62 kernel: [140205.786702] pci-stub
0000:01:13.1: irq 289 for MSI/MSI-X
May 27 09:13:45 on-10-177-32-62 kernel: [140205.786710] pci-stub
0000:01:13.1: irq 290 for MSI/MSI-X
May 27 09:13:45 on-10-177-32-62 kernel: [140206.254300] ixgbe
0000:01:00.0 eth0: VF Reset msg received from vf 13
May 27 09:13:45 on-10-177-32-62 kernel: [140206.286346] pci-stub
0000:01:13.2: irq 407 for MSI/MSI-X
May 27 09:13:45 on-10-177-32-62 kernel: [140206.306280] pci-stub
0000:01:13.2: irq 407 for MSI/MSI-X
May 27 09:13:45 on-10-177-32-62 kernel: [140206.306293] pci-stub
0000:01:13.2: irq 408 for MSI/MSI-X
May 27 09:13:45 on-10-177-32-62 kernel: [140206.354379] pci-stub
0000:01:13.2: irq 407 for MSI/MSI-X
May 27 09:13:45 on-10-177-32-62 kernel: [140206.354393] pci-stub
0000:01:13.2: irq 408 for MSI/MSI-X
May 27 09:13:45 on-10-177-32-62 kernel: [140206.354403] pci-stub
0000:01:13.2: irq 409 for MSI/MSI-X
May 27 09:13:49 on-10-177-32-62 kernel: [140210.174615] ixgbe
0000:01:00.1 eth1: VF Reset msg received from vf 13
May 27 09:13:49 on-10-177-32-62 kernel: [140210.207755] pci-stub
0000:01:13.3: irq 410 for MSI/MSI-X
May 27 09:13:49 on-10-177-32-62 kernel: [140210.251553] pci-stub
0000:01:13.3: irq 410 for MSI/MSI-X
May 27 09:13:49 on-10-177-32-62 kernel: [140210.251568] pci-stub
0000:01:13.3: irq 411 for MSI/MSI-X
May 27 09:13:49 on-10-177-32-62 kernel: [140210.331491] pci-stub
0000:01:13.3: irq 410 for MSI/MSI-X
May 27 09:13:49 on-10-177-32-62 kernel: [140210.331506] pci-stub
0000:01:13.3: irq 411 for MSI/MSI-X
May 27 09:13:49 on-10-177-32-62 kernel: [140210.331518] pci-stub
0000:01:13.3: irq 412 for MSI/MSI-X


Regards
Dominik

2013/5/26 Dominik Mostowiec <dominikmostowiec gmail com>:
> Hi,
>
>> First - I just got confirmation from the libnl maintainer that the bug I
>> mentioned causing problems with max_vfs > 50 *is* an issue in libnl
>> 3.2.16 - it is fixed in libnl 3.2.22. Try to upgrade to that version and
>> it should solve at least some of your problems.
>
> Upgrade libnl to version 3.2.22 helps.
> Thanks.
>
>> 1) when you have max_vfs = 10, you can assign the devices and mac
>> addresses are set properly, i.e. everything works.
>> 2) when you have max_vfs = 35, you can assign the devices, but the mac
>> addresses are not set properly.
>> 3) when you have max_vfs = 63, you can't assign the devices because you
>> get this error:
>>
>>         internal error missing IFLA_VF_INFO in netlink response
>>
>> Is this correct?
>
> Yes.
> Before upgrade libnl :-)
> Upgrede helps only for limit max_vfs.
>
>> (BTW, it may get a bit confusing if you leave your networks named as
>> "vnet0" and "vnet1" - although it's a different namespace, libvirt uses
>> "vnetN" (where n is 0 - whatever) as the name for the tap device
>> associated with each guest interface).
>
> You're right.
> I changed this.
>
>> the output of "ip link show dev $PF".
> Hmm,
>> ip link show dev eth0
> Shows only:
> 2: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq
> master bond0 state UP qlen 1000
>     link/ether b8:ca:3a:5b:a6:3a brd ff:ff:ff:ff:ff:ff
>
> I see VFs like network interfaces eth2-eth127.
> When VF is attached to VM it disappear from list on host.
>
> --
> Regards
> Dominik
>
>
> 2013/5/24 Laine Stump <laine laine org>:
>> On 05/24/2013 09:16 AM, Dominik Mostowiec wrote:
>>
>>>>> Do you have this problem with mac addresses when max_vfs is set to an
>>> even lower number?
>>
>>
>> First - I just got confirmation from the libnl maintainer that the bug I
>> mentioned causing problems with max_vfs > 50 *is* an issue in libnl
>> 3.2.16 - it is fixed in libnl 3.2.22. Try to upgrade to that version and
>> it should solve at least some of your problems.
>>
>>
>>>
>>> No,
>>> When max_fs=10, macs in VM are OK.
>>
>>
>> Interesting.
>>
>> So now I'm curious about something - can you try adding a vlan tag to
>> your networks and see if the vlan tag is set properly when max_vfs is a
>> low number (10 or 7):
>>
>> <network>
>>   <name>vnet0</name>
>>   <uuid>ec49896a-a0b5-4944-a81f-9f0cdf578871</uuid>
>>   <vlan>
>>     <tag id='42'/>
>>   </vlan>
>>   <forward mode='hostdev' managed='yes'>
>>     <pf dev='eth0'/>
>>   </forward>
>> </network>
>>
>>
>> (you will have to do "virsh net-destroy vnet0; virsh net-start vnet0"
>> after making the modification to the network). If the vlan tag is
>> successfully set, it will show up next to the mac address for the VF in
>> the output of "ip link show dev $PF".
>>
>> (BTW, it may get a bit confusing if you leave your networks named as
>> "vnet0" and "vnet1" - although it's a different namespace, libvirt uses
>> "vnetN" (where n is 0 - whatever) as the name for the tap device
>> associated with each guest interface).
>>
>> To summarize:
>>
>> 1) when you have max_vfs = 10, you can assign the devices and mac
>> addresses are set properly, i.e. everything works.
>> 2) when you have max_vfs = 35, you can assign the devices, but the mac
>> addresses are not set properly.
>> 3) when you have max_vfs = 63, you can't assign the devices because you
>> get this error:
>>
>>         internal error missing IFLA_VF_INFO in netlink response
>>
>> Is this correct?
>>
>>>
>>> Phisical interfaces:
>>>> ifconfig eth0
>>> eth0      Link encap:Ethernet  HWaddr b8:ca:3a:5b:a6:38
>>>           UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
>>
>> Okay, so that's not the problem.
>>
>>>
>>>> ifconfig eth1
>>> eth1      Link encap:Ethernet  HWaddr b8:ca:3a:5b:a6:38
>>>           UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
>>>
>>>>> Hmm. ixgbe. I just got a report yesterday that setting of the vlan tag
>>>>> for ixgbe vfs was not working properly (although it works fine for igb
>>>>> devices)
>>> Domain config:
>>> <interface type="network">
>>>      <alias name="hostdev0"/>
>>>      <source network="vnet0"/>
>>>      <mac address="52:54:0a:b1:48:fc"/>
>>> </interface>
>>> <interface type="network">
>>>      <alias name="hostdev1"/>
>>>      <source network="vnet1"/>
>>>      <mac address="52:54:0a:b1:48:fc"/>
>>>  </interface>
>>>
>>> Networks defined:
>>> <network>
>>>   <name>vnet0</name>
>>>   <uuid>ec49896a-a0b5-4944-a81f-9f0cdf578871</uuid>
>>>   <forward mode='hostdev' managed='yes'>
>>>     <pf dev='eth0'/>
>>>   </forward>
>>> </network>
>>>
>>> <network>
>>>   <name>vnet1</name>
>>>   <uuid>6c8319e8-8b53-4382-9f4e-2400819b00a9</uuid>
>>>   <forward mode='hostdev' managed='yes'>
>>>     <pf dev='eth1'/>
>>>   </forward>
>>> </network>
>>>
>>> Regards
>>> Dominik
>>>
>>> 2013/5/24 Laine Stump <laine laine org>:
>>>> On 05/24/2013 04:45 AM, Dominik Mostowiec wrote:
>>>>> Hi,
>>>>> I have libnl3, kernel 3.8.8, os ubuntu 12.04, net:intel10G.
>>>>> Upgrade libnl to 3.2.16 not helps,
>>>>>
>>>>> I don't know - this is probably another problem, when I have set
>>>>> max_vfs=35,35 VF is attaching to VM but mac in VM is not set
>>>>> propertly.
>>>> Do you have this problem with mac addresses when max_vfs is set to an
>>>> even lower number?
>>>>
>>>> Ah, another thing - did you "ifconfig up" the PF before you started the
>>>> guest?
>>>>> Libvirt probably set this macs because in syslog:
>>>>> May 23 14:48:59 on-10-177-32-62 kernel: [12574.171792] ixgbe
>>>>
>>>> Hmm. ixgbe. I just got a report yesterday that setting of the vlan tag
>>>> for ixgbe vfs was not working properly (although it works fine for igb
>>>> devices)
>>>>
>>>>
>>>>> 0000:01:00.0: setting MAC 52:54:0a:b1:48:f7 on VF 0
>>>>> May 23 14:48:59 on-10-177-32-62 kernel: [12574.171799] ixgbe
>>>>> 0000:01:00.0: Reload the VF driver to make this change effective.
>>>>> May 23 14:48:59 on-10-177-32-62 kernel: [12574.175054] ixgbe
>>>>> 0000:01:00.1: setting MAC 52:54:0a:b1:48:f7 on VF 0
>>>>> May 23 14:48:59 on-10-177-32-62 kernel: [12574.175060] ixgbe
>>>>> 0000:01:00.1: Reload the VF driver to make this change effective.
>>>>> May 23 14:48:59 on-10-177-32-62 kernel: [12574.307172] pci-stub
>>>>> 0000:01:12.4: enabling device (0000 -> 0002)
>>>>> May 23 14:49:00 on-10-177-32-62 kernel: [12575.662890] assign device 0:1:12.4
>>>>> May 23 14:49:00 on-10-177-32-62 kernel: [12575.665210] pci-stub
>>>>> 0000:01:12.5: enabling device (0000 -> 0002)
>>>>> May 23 14:49:00 on-10-177-32-62 kernel: [12575.765609] assign device 0:1:12.5
>>>>> But in VM i have another macs probably old from VF.
>>>>>
>>>>> ---
>>>>> Regards
>>>>> Dominik
>>>>>
>>>>>
>>>>>
>>>>> 2013/5/24 Laine Stump <laine laine org>:
>>>>>> On 05/22/2013 06:32 PM, Dominik Mostowiec wrote:
>>>>>>> I tested on 1.0.5 patched version and vm with 2 vfs working fine.
>>>>>>>
>>>>>>> I have another problem
>>>>>>> When i set max_vfs=63:
>>>>>>> internal error missing IFLA_VF_INFO in netlink response
>>>>>>>
>>>>>>> Its working when max_vfs=31
>>>>>>>
>>>>>> What is the platform, kernel version, and what version of libnl are you
>>>>>> using (libnl 1 or libnl3?)
>>>>>>
>>>>>> There have been multiple bugs recently filed and fixed in this area on
>>>>>> RHEL6/CentOS6. In particular, there was a problem with max_vfs > 50 that
>>>>>> had exactly this symptom. It was solved by a patch to libnl-1.1. If your
>>>>>> platform uses libnl-1.1, this could be the source of the problem. It is
>>>>>> fixed in the upstream libnl-1.1.4 maintenance release.
>>>>>>
>>>>>> I unfortunately am not sure where the libnl-1.1 git is (I only have the
>>>>>> git tree for libnl3), so I can't give you the exact commit message, but
>>>>>> in short the problem was that libnl was using too small of a buffer for
>>>>>> the netlink socket, so when there were a lot of vfs, the netlink message
>>>>>> containing vf info was truncated.
>>>>>>
>>>>>>
>>>>>>> --
>>>>>>> Dominik
>>>>>>>
>>>>>>> 21 maj 2013 15:19, "Dominik Mostowiec" <dominikmostowiec gmail com
>>>>>>> <mailto:dominikmostowiec gmail com>> napisał(a):
>>>>>>>
>>>>>>>     Hmm,
>>>>>>>     It seems to be working (after only simple tests).
>>>>>>>
>>>>>>>     --
>>>>>>>     Dominik
>>>>>>>
>>>>>>>
>>>>>>>     2013/5/21 Ján Tomko <jtomko redhat com <mailto:jtomko redhat com>>
>>>>>>>
>>>>>>>         On 05/21/2013 01:37 PM, Laine Stump wrote:
>>>>>>>         > On 05/21/2013 04:03 AM, Ján Tomko wrote:
>>>>>>>         >> On 05/21/2013 09:32 AM, Dominik Mostowiec wrote:
>>>>>>>         >>> hi,
>>>>>>>         >>> I try to add 2 VF by "hostdev".
>>>>>>>         >>> Networks (vnet0, vnet1) with:
>>>>>>>         >>>  <forward mode='hostdev' managed='yes'>
>>>>>>>         >>>     <pf dev='eth1'/>
>>>>>>>         >>> .....
>>>>>>>         >>>
>>>>>>>         >>> Domain:
>>>>>>>         >>> <interface type="network">
>>>>>>>         >>>              <source network="vnet0"/>
>>>>>>>         >>> ....
>>>>>>>         >>> <interface type="network">
>>>>>>>         >>>              <source network="vnet1"/>
>>>>>>>         >>> ....
>>>>>>>         >>>
>>>>>>>         >>> virsh create error:
>>>>>>>         >>> "error: internal error process exited while connecting to
>>>>>>>         monitor: kvm:
>>>>>>>         >>> -device
>>>>>>>         pci-assign,configfd=25,host=01:10.1,id=hostdev0,bus=pci.0,addr=0x4:
>>>>>>>         >>> Duplicate ID 'hostdev0' for device"
>>>>>>>         >>>
>>>>>>>         >> Hi,
>>>>>>>         >>
>>>>>>>         >> it seems we have been assigning the same id to all network
>>>>>>>         hostdevs until this
>>>>>>>         >> recent commit (not yet released):
>>>>>>>         >>
>>>>>>>         >> http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=6597cc25
>>>>>>>         >
>>>>>>>         >
>>>>>>>         > If that patch has such an effect, it's purely accidental.
>>>>>>>         Have you
>>>>>>>         > tested this? (I plan to test a before and after as soon as
>>>>>>>         I've had
>>>>>>>         > breakfast)
>>>>>>>         >
>>>>>>>
>>>>>>>         I haven't tested it and looking at the code again, I might've
>>>>>>>         been wrong :(
>>>>>>>
>>>>>>>         Sorry about that.
>>>>>>>
>>>>>>>         Jan
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>     --
>>>>>>>     Pozdrawiam
>>>>>>>     Dominik
>>>>>>>
>>>>>
>>>
>>>
>>> --
>>> Pozdrawiam
>>> Dominik
>>>
>>
>
>
>
> --
> Pozdrawiam
> Dominik



-- 
Pozdrawiam
Dominik


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]