[libvirt-users] Guest-networking: Two subnets = Guests cannot connect to the other subnet

[mailinglists at florian-kaiser.net]

Hello list,

We have KVM setup using libvirt (0.9.0) as configuration device on a Debian
Squeeze Host. Networking is done in routed bridge mode and we have two
different subnets, as well as one main ip. Subnets get routed to the main ip
(this is done by our provider).

Network on hosts are done like this:
<network>
  <name>subnet_1</name>
  <uuid>7c60990d-ab31-c4a5-cd2b-5ce4c7c61d38</uuid>
  <forward dev='eth0' mode='route'/>
  <bridge name='virtbridge1' stp='off' delay='0' />
  <mac address='52:54:00:4D:83:2A'/>
  <ip address='xx.xx.240.121' netmask='255.255.255.248'>
  </ip>
</network>

<network>
  <name>subnet_2</name>
  <uuid>7c60990d-ab31-c4a5-cd2b-5ce4c7c61d38</uuid>
  <forward dev='eth0' mode='route'/>
  <bridge name='virtbridge1' stp='off' delay='0' />
  <mac address='52:54:00:4D:83:2A'/>
  <ip address='xx.xx.240.200' netmask='255.255.255.248'>
  </ip>
</network>

Routing on host
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use
Iface
xx.xx.122.33     0.0.0.0         255.255.255.255 UH    0      0        0
eth0
xx.xx.240.120   0.0.0.0         255.255.255.248 U     0      0        0
virtbridge1
xx.xx.239.200   0.0.0.0         255.255.255.248 U     0      0        0
virtbridge2
0.0.0.0         xx.xx.122.33     0.0.0.0         UG    0      0        0
eth0


Guests get this:
<interface type='network'>
  <mac address='de:ad:b6:83:85:5b'/>
  <source network='subnet_1'/>
  <model type='virtio'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
function='0x0'/>
</interface>

Guests put this into interfaces:
iface eth0 inet static
        address xx.xx.239.202
        netmask 255.255.255.248
        gateway xx.xx.239.201

Guest routing
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use
Iface
xx.xx.239.200   0.0.0.0         255.255.255.248 U     0      0        0 eth0
0.0.0.0         xx.xx.239.201   0.0.0.0         UG    0      0        0 eth0
	
	
Everything seemed to work just fine, that is: Guests could connect 
- to host, 
- to internet
- to other virtual maschines in the same subnet - all vice-versa. 

Internet can connect to any guest and our host can connect to any guest,
too.

Then, we needed one guest to connect to a guest from the other subnet. And
this does not work, what I get follows:

PING xx.xx.240.122 (xx.xx.240.122) 56(84) bytes of data.
>From xx.xx.239.201 icmp_seq=1 Destination Port Unreachable
>From xx.xx.239.201 icmp_seq=2 Destination Port Unreachable

Note, that I can ping the other subnets gateway just fine! But I cannot
connect to any of the other subnet IPs.

I believe this is just a routing problem, but I cannot figure out what route
to add where. Please ask if you need any more information.