[libvirt] RFC: Improve performance of macvtap device creation

[Laine Stump]

On 10/30/2015 11:56 AM, Tony Krowiak wrote:
> On 10/30/2015 06:49 AM, Michal Privoznik wrote:
>> On 29.10.2015 18:48, Laine Stump wrote:
>>> On 10/29/2015 12:49 PM, Tony Krowiak wrote:
>>>> For a guest domain defined with a large number of macvtap devices, it
>>>> takes an exceedingly long time to boot the guest. In a test of a guest
>>>> domain configured with 82 macvtap devices, it took over two minutes
>>>> for the guest to boot. An strace of the ioctl calls during guest start
>>>> up showed the SIOCGIFFLAGS ioctl literally being invoked 3,403 times.
>>>> I was able to isolate the source of the ioctl calls to
>>>> the*virNetDevMacVLanCreateWithVPortProfile*  function
>>>> in*virnetdevmacvlan.c*. The macvtap interface name is created by
>>>> looping over a counter variable, starting with zero, and appending the
>>>> counter value to 'macvtap'.
>>> I've wondered ever since the first time I saw that code why it was done
>>> that way, and why there had never been any performance complaints.
>>> Lacking any complaints, I promptly forgot about it (until the next time
>>> I went past the code for some other tangentially related reason.)
>>>
>>> Since you're the first to complain, you have the honor of fixing it :-)
>>>
>>>> With each iteration, a call is made to*virNetDevExists*  (SIOCGIFFLAGS
>>>> ioctl) to determine if a device with that name already exists, until a
>>>> unique name is created. In the test case cited above, to create an
>>>> interface name for the 82nd macvtap device, the*virNetDevExists*
>>>> function will be called for interface names 'macvtap0' to 'macvtap80'
>>>> before it is determined that 'mavtap81' can be used. So if N is the
>>>> number of macvtap interfaces defined for a guest, the SIOCGIFFLAGS
>>>> ioctl will be invoked (N x N + N)/2 times to find an unused macvtap
>>>> device names. That's assuming only one guest is being started, who
>>>> knows how many times the ioctl may have to be called in an
>>>> installation running a large number of guests defined with macvtap
>>>> devices.
>> Not only that, but unitl c0d162c68c2f19af8d55a435a9e372da33857048 (
>> contained v1.2.2~32) if two threads were starting a domain concurrently,
>> they even competed with each other in that specific area of the code.

That's only for the veth devices used by lxc, which is a separate thing 
from the macvtap devices (rough description: a veth is a pair of netdevs 
where the gazinta of one is the gazouta of the other and vice versa, 
while a macvtap is a chardev on one side, and is a netdev permanently 
attached "to the side" of another netdev at a point where the two can't 
see each others' traffic, but can both see all traffic going in and out 
the original device).

veth creation does suffer from the same problem though - each time you 
want to create a veth pair, libvirt will check for an unused device name 
by calling virNetDevExists() starting at vnet0, so it could benefit from 
the same code if it was made agnostic of basename (note that all 
standard tap devices would need to be marked in such a bitmap, since 
they also use the name "vnet%d").

(An aside: I'm a bit surprised to see that we are creating veths using 
"ip link add", since the same thing can be done with a netlink message 
and would remove the need to exec an external binary. Not that I haven't 
done the same thing myself when it was expedient :-)