[Fedora-xen] Weird routing problem
Dale Bewley
dlbewley at lib.ucdavis.edu
Wed Apr 16 22:34:08 UTC 2008
On Thu, 2008-04-10 at 14:55 +0200, Turbo Fredriksson wrote:
> I have two physical hosts (Correo and Alexander), running two XEN
> instances on one of them (Ferrari and Amarillo on Correo) and one
> on the other (Graham on Alexander)...
>
> Picture at http://bayour.com/misc/VoIP.jpg.
>
>
> On the firewall/gateway (192.168.1.1) I route 192.168.3.0/24 to Correo
> (192.168.1.7) and 192.168.4.0/24 to Alexander (192.168.1.6). This so
> that I can access the XEN hosts from the internal network. Very basic...
>
> And all my VoIP phones is on it's (about to be on a) separate network
> with the firewall/gateway as default gateway.
>
>
> On Alexander:
> =============
> * /etc/xen/graham.cfg
> kernel = '/boot/vmlinuz-2.6.18-5-xen-amd64'
> ramdisk = '/boot/initrd.img-2.6.18-5-xen-amd64'
> memory = '2500'
> root = '/dev/sda1 ro'
> disk = [ 'file:/home/xen/domains/graham/disk.img,sda1,w', 'file:/home/xen/domains/graham/swap.img,sda2,w' ]
> name = 'graham'
> vif = [ 'ip=192.168.4.11' ]
> on_poweroff = 'destroy'
> on_reboot = 'restart'
> on_crash = 'restart'
>
> * /etc/xen/xend-config.sxp
> (xend-http-server yes)
> (xend-unix-server yes)
> (xend-tcp-xmlrpc-server no)
> (xend-unix-xmlrpc-server yes)
> (xend-relocation-server yes)
> (xend-unix-path /var/lib/xend/xend-socket)
> (xend-port 8000)
> (xend-relocation-port 8002)
> (xend-address 'alexander')
> (xend-relocation-address 'alexander')
> (console-limit 1024)
> (network-script network-route)
> (vif-script vif-route)
> (dom0-min-mem 196)
> (dom0-cpus 2)
> (enable-dump yes)
> (vnc-listen '0.0.0.0')
>
> * ifconfig (trimmed - only 'lo' if removed)
> eth0 Link encap:Ethernet HWaddr 00:1C:23:C4:28:92
> inet addr:192.168.1.6 Bcast:192.168.1.255 Mask:255.255.255.0
> inet6 addr: fe80::21c:23ff:fec4:2892/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
>
> eth0:0 Link encap:Ethernet HWaddr 00:1C:23:C4:28:92
> inet addr:192.168.4.1 Bcast:192.168.4.255 Mask:255.255.255.0
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
>
> vif5.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
> inet addr:192.168.1.6 Bcast:192.168.1.255 Mask:255.255.255.255
> inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
>
> * route -n
> Kernel IP routing table
> Destination Gateway Genmask Flags Metric Ref Use Iface
> 192.168.4.11 0.0.0.0 255.255.255.255 UH 0 0 0 vif5.0
> 192.168.4.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
> 192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
> 0.0.0.0 192.168.1.1 0.0.0.0 UG 0 0 0 eth0
>
> * iptables -L -n
> Chain INPUT (policy ACCEPT)
> target prot opt source destination
>
> Chain FORWARD (policy ACCEPT)
> target prot opt source destination
> ACCEPT 0 -- 192.168.4.11 0.0.0.0/0 PHYSDEV match --physdev-in vif5.0
> ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif5.0 udp spt:68 dpt:67
>
> Chain OUTPUT (policy ACCEPT)
> target prot opt source destination
>
> * iptables -L -n -t nat
> Chain PREROUTING (policy ACCEPT)
> target prot opt source destination
>
> Chain POSTROUTING (policy ACCEPT)
> target prot opt source destination
>
> Chain OUTPUT (policy ACCEPT)
> target prot opt source destination
>
> On Graham:
> ==========
> * ifconfig (trimmed - only 'lo' if removed)
> eth0 Link encap:Ethernet HWaddr 00:16:3E:00:AB:28
> inet addr:192.168.4.11 Bcast:192.168.4.255 Mask:255.255.255.0
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
>
> * route -n
> Kernel IP routing table
> Destination Gateway Genmask Flags Metric Ref Use Iface
> 192.168.4.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
> 0.0.0.0 192.168.4.1 0.0.0.0 UG 0 0 0 eth0
>
> * iptables -L -n
> FATAL: Could not load /lib/modules/2.6.18-5-xen-amd64/modules.dep: No such file or directory
> iptables v1.3.6: can't initialize iptables table `filter': iptables who? (do you need to insmod?)
> Perhaps iptables or your kernel needs to be upgraded.
>
> Correo with the XEN hosts Ferrari and Amarillo basically look identical (only different
> networks).
>
> As seen, I do NOT use NAT here. I wanted to use true routed network... And it seems to work.
> My primary Asterisk server (the one that do all the routing - the one on Alexander only deals
> with the PSTN trafik) runs on Graham and it can be accessed from the outside - with port
> forwarding on the firewall/gateway and it can also contact external Asterisk servers (I run
> one at home to deal with my private VoIP).
>
>
> The DNS runs on Correo, but it can not be reached (queried) from Graham!
>
> ----- s n i p -----
> graham# ping -c 5 correo
> ping: unknown host correo
>
> graham# ping -c 5 192.168.1.7
> PING 192.168.1.7 (192.168.1.7) 56(84) bytes of data.
> 64 bytes from 192.168.1.7: icmp_seq=1 ttl=62 time=0.270 ms
> 64 bytes from 192.168.1.7: icmp_seq=2 ttl=62 time=0.260 ms
> 64 bytes from 192.168.1.7: icmp_seq=3 ttl=62 time=0.264 ms
> 64 bytes from 192.168.1.7: icmp_seq=4 ttl=62 time=0.273 ms
> 64 bytes from 192.168.1.7: icmp_seq=5 ttl=62 time=0.257 ms
>
> --- 192.168.1.7 ping statistics ---
> 5 packets transmitted, 5 received, 0% packet loss, time 4000ms
> rtt min/avg/max/mdev = 0.257/0.264/0.273/0.021 ms
>
> graham# traceroute -n 192.168.1.7
> traceroute to 192.168.1.7 (192.168.1.7), 30 hops max, 52 byte packets
> 1 192.168.1.6 0.285 ms 0.091 ms 0.090 ms
> 2 192.168.1.7 0.323 ms 0.262 ms 0.258 ms
>
> graham# telnet 192.168.1.7 53
> Trying 192.168.1.7...
> Connected to 192.168.1.7.
> Escape character is '^]'.
> correo
> Connection closed by foreign host.
>
> graham# host graham 192.168.1.7
> ;; reply from unexpected source: 192.168.1.1#53, expected 192.168.1.7#53
> ;; reply from unexpected source: 192.168.1.1#53, expected 192.168.1.7#53
> ;; connection timed out; no servers could be reached
> ----- s n i p -----
>
> Also, scp or ssh FROM Graham to Correo don't work, but the other way
> around works fine...
>
>
> Looking at the answer that 'host' gave me, I now see that the connection
> goes via the firewall/gateway which is not directly obvious - Alexander
> (which is Graham's default GW) is on the same network as Correo...
>
>
> PS. I solved this specific DNS problem with a caching DNS server on
> Alexander, but scp/ssh (etc) naturally still don't work because
> of this weird problem... I just can't see it! Maybe a set of
> (many :) extra eyes can... Thanx!
Did you figure it out yet?
I can not quite tell what you're doing. Who is that blue router in your
diagram? Maybe you are using a router icon to indicate a switch?
At any rate if I had to guess it sounds like you are expecting to speak
from eth0:0 IP when you are actually speaking from eth0 IP. This you
could problably confirm with a tcpdump on the target machine while you
probe from the other. If so, you could fix that with policy routing AKA
source routing. i.e. `ip rule help`. Look at your ARP tables too just in
case.
More information about the Fedora-xen
mailing list