RHEL 5, Xen, IP Change weirdness

[Denny Snyder]

Hi all,

I had installed RHEL 5 x86_64 with virtualization right off the bat.  
Everything ran beautifully with two Xen guests (also RHEL 5.)  All 3 IPs 
(host and 2 guests) were in the same subnet.

Then I had to change all the IP addresses to a different subnet (but 
again all 3 within the same subnet - trying to stay away from 802.1q 
stuff)...  so I modified each guest's /etc/hosts, etc/sysconfig/network, 
etc/sysconfig/network-scripts/ifcfg-eth0 and then shutdown both guests.  
Then modified the configs on the host and rebooted.

When it boots now it seems like it's fine but there is no network 
connectivity and the default route is pointing to the virbr0 interface!!
ifconfig shows eth0 but without an IP address

If I do a 'service network restart' then everything works fine again, 
eth0 has the correct IP and the default route is in place.  At that 
point everything seems fine with the exception of:

I get this every two minutes now:
libvir: Remote error : unable to connect to 
'/var/run/libvirt/libvirt-sock': Connection refused
libvir: warning : Failed to find the network: Is the daemon running ?
libvir: Remote error : unable to connect to 
'/var/run/libvirt/libvirt-sock': Connection refused
libvir: Remote error : unable to connect to 
'/var/run/libvirt/libvirt-sock': Connection refused
libvir: warning : Failed to find a node driver: Is the libvirtd daemon 
running ?
libvir: Remote error : unable to connect to 
'/var/run/libvirt/libvirt-sock': Connection refused
libvir: warning : Failed to find the network: Is the daemon running ?
libvir: Remote error : unable to connect to 
'/var/run/libvirt/libvirt-sock': Connection refused
libvir: Remote error : unable to connect to 
'/var/run/libvirt/libvirt-sock': Connection refused
libvir: warning : Failed to find a node driver: Is the libvirtd daemon 
running ?

virt-manager seems to work fine and lists my dom0 and my two xens which 
also seem to be working just fine.  virsh also seems to have no issues 
showing my xens:

virsh list
  Id Name                 State
----------------------------------
   0 Domain-0             running
   1 jhsympa1             idle
   2 jhsympa2             idle

xm list also shows no issues:
xm list
Name                                      ID Mem(MiB) VCPUs State   Time(s)
Domain-0                                   0    30050     8 r-----   2596.2
jhsympa1                                   1     1023     1 -b----    159.7
jhsympa2                                   2     1023     1 -b----   4125.3

'service libvirtd status' shows:
libvirtd dead but subsys locked

dmesg doesn't show anything that sticks out other than the fact that 
eth0 and eth1 seem to be pronounced to be DOWN, then UP, then DOWN, then 
UP, then DOWN then UP again and again in a time span of a few seconds as 
different modules start to act - mDNS, xend, bnx iSCSI support, 
whatever.  But when I compared that to the logs prior to the IP change, 
they're the same.

I do notice that when I launch virt-manager there is nothing listed in 
the 'virtual network' tab - it's all greyed out.

SELinux is disabled.  I can't find anything in any configs pertaining to 
xen or libvirt that are IP related.

I know if this server is rebooted for any reason I'm going to need to do 
a 'service network restart' before connectivity to/from the guests will 
work and that's not a good thing!

another error I noticed is:
virsh net-list --all
error: Failed to list active networks
error: this function is not supported by the hypervisor: 
virConnectNumOfNetworks

whereas xm gives me this:
#xm network-list Domain-0 comes back with nothing
#xm network-list jhsympa1
Idx BE     MAC Addr.     handle state evt-ch tx-/rx-ring-ref BE-path
0   0  00:16:36:32:6e:39    0     4      8     768  /57      
/local/domain/0/backend/vif/1/0
# xm network-list jhsympa2
Idx BE     MAC Addr.     handle state evt-ch tx-/rx-ring-ref BE-path
0   0  00:16:36:7b:f3:14    0     4      8     769  /768     
/local/domain/0/backend/vif/2/0

MORE INFO:
#brctl show
bridge name     bridge id               STP enabled     interfaces
virbr0          8000.000000000000       yes
xenbr0          8000.feffffffffff       no              vif2.0
                                                         vif1.0
                                                         peth0
                                                         vif0.0

This appears correct as far as I can tell

Where am I to fix this weirdness?  Changing IPs shouldn't involve all of 
this!  Especially when I'm only changing eth0 (eth1 is not configured) 
and they are all in the same subnet as before and nothing else was modified.


These guests are about to be put into production and I only have a few 
days to make changes.
Any help would be awesome!


Thanks


-- 
Denny Snyder
Systems Engineer
IT at Johns Hopkins
5801 Smith Ave Suite 3110C
Baltimore, MD 21209
(410) 735-7613: DESK
(410) 735-4660: FAX