[libvirt] UDP broadcasts vs. nat Masquerading issue

[Daniel P. Berrangé]

On Wed, Jul 03, 2019 at 09:07:10PM +0300, Nikolai Zhubr wrote:
> At the moment I'm running a quite outdated version 1.2.9 of libvirt, but
> because other than this issue it does its job pretty well I'd first consider
> some patching/backporting rather than totally replacing it with a new one.
> Anyway, I first need to better understand what is going on and what is wrong
> with it.

Not too bad an issue - the iptables rules libvirt creates have been
almost entirely unchanged since they were first introduced, until
I recently did some changes. My recent changes merely moved them
down into libvirt chains instead of the root chains, which shouldn't
be a functional change.

> I've already figured the source of trouble is anyway related to these rules
> added:
> 
> -A POSTROUTING -o br0 -j MASQUERADE
> -A POSTROUTING -o enp0s25 -j MASQUERADE
> -A POSTROUTING -o virbr2_nic -j MASQUERADE
> -A POSTROUTING -o vnet0 -j MASQUERADE

Is that really the full specification of the rules ?

AFAIK, libvirt would not create such rules - at very least lbivirt
MASQUERADE should always have a source + destination IP mask

If I start the libvirt 'default' virtual network the MASQUARADE
rules created by libvirt look like:

-A LIBVIRT_PRT -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
-A LIBVIRT_PRT -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
-A LIBVIRT_PRT -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE


In older libvirt it would be POSTROUTING instead of LIBVIRT_PRT but
otherwise the same


> Here, virbr2_nic and vnet0 are used by libvirt for arranging NAT-mode
> network configurations for VMs, unrelated to normal network stuff, so it
> looks ok. However, br0 (with enp0s25 in it) is a main interface of this host
> with primary ip address. And enp0s25 is a physical nic of this host, and it
> is used for all sorts of regular (unrelated to virtualization)
> communications as well. Also, br0 is used for attaching some bridge-mode (as
> opposed to NAT-mode) VMs managed by libvirt, but bridge mode is not supposed
> to employ address translation anyway.
> 
> So, clearly, libvirt somehow chooses to set up masquerading for literally
> all existing network interfaces here (except lo), but I can't see a reason
> for the first two rules in the list above. Furthermore, they corrupt UDP
> broadcats coming from outside and reaching this host (through enp0s25/br0)
> such that source address gets replaced by this hosts primary address (as per
> masquerading). I've verified this by arranging a hand-crafted UDP listener
> and printing the respective source addresses as seen by normal userspace.
> Obviously Samba server can not work correctly under such conditions.
> 
> Now I've discovered that I can "eliminate" the problem by e.g.:
> 
> 1. Removing "-A POSTROUTING -o br0 -j MASQUERADE" (manual brute force)
> 2. Inserting "-A POSTROUTING -s 192.168.0.0/24 -d 192.168.0.255/32 -j
> ACCEPT"
> (Of course correcting rules by hand is not a solution, just a test)
> 
> So question is, how the correct rules should ideally look like? And, is this
> issue known/fixed in most current libvirt?

I'm not convinced that libvirt is actually creating those rules.

Is there any other software on your host that could be responsible ?

Can you test

 - Disable all libvirt virtual networks

     virsh net-destroy NETNAME
     virsh net-autostart --disable NETNAME

 - Reboot the host OS

Now look at iptables rules. Are the MASQUERADE rules present ?

If they are not present then start a network with

   'virsh net-start NETNAME'

Are the MASQUERADE rules now present ?  If so can you show the
XML of the network (virsh net-dumpxml NETNAME).

Also do you have any hook scripts in /etc/libvirt/hooks  that
might be doing anything ?

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|