17.1. A live migration example
Below is an example of how to setup a simple environment for live migration. This configuration is using NFS for the shared storage. NFS is suitable for demonstration environments but for a production environment a more robust shared storage configuration using Fibre Channel or iSCSI and GFS is recommended.
The configuration below consists of two servers (et-virt07 and et-virt08), both of them are using eth1 as their default network interface hence they are using xenbr1 as their Red Hat Virtualization networking bridge. We are using a locally attached SCSI disk (/dev/sdb) on et-virt07 for shared storage using NFS.
Create and mount the directory used for the migration:
# mkdir /xentest # mount /dev/sdb /xentest
Ensure the directory is exported with the correct options. If you are exporting the default directory /var/lib/xen/images/ make sure you only export /var/lib/xen/images/ and not/var/lib/xen/ as this directory is used by the xend daemon and other Xen components. Sharing /var/lib/xen/ will cause unpredictable behavior.
# cat /etc/exports /xentest *(rw,async,no_root_squash)
Verify it is exported via NFS:
# showmount -e et-virt07 Export list for et-virt07: /xentest *
The install command in the example used for installing the virtual machine:
# virt-install -p -f /xentest/xentestTravelVM01.dsk -s 5 -n\ xentesttravelvm01 --vnc -r 1024 -l http://porkchop.devel.redhat.com/rel-eng/RHEL5-\ Server-20070105.0/4.92/x86_64/os/ -b xenbr1
Make sure the Xen networking bridges are configured correctly and have the same name on both hosts:
[et-virt08 ~]# brctl show bridge name bridge id STP enabled interfaces xenbr1 8000.feffffffffff no peth1 vif0.1
[et-virt07 ~]# brctl show bridge name bridge id STP enabled interfaces xenbr1 8000.feffffffffff no peth1 vif0.1
Check the relocation parameters have been configured in the Xen config file on both hosts:
[et-virt07 ~]# grep xend-relocation /etc/xen/xend-config.sxp |grep -v '#' (xend-relocation-server yes) (xend-relocation-port 8002) (xend-relocation-address '') (xend-relocation-hosts-allow '')
[et-virt08 ~]# grep xend-relocation /etc/xen/xend-config.sxp |grep -v '#' (xend-relocation-server yes) (xend-relocation-port 8002) (xend-relocation-address '') (xend-relocation-hosts-allow '')
Make sure the Xen relocation server has started and is listening on the dedicated port for Xen migrations (8002):
[et-virt07 ~]# lsof -i :8002 COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME python 3445 root 14u IPv4 10223 TCP *:teradataordbms (LISTEN)
[et-virt08 ~]# lsof -i :8002 COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME python 3252 root 14u IPv4 10901 TCP *:teradataordbms (LISTEN)
Verify the NFS directory has been mounted on the other host and you can see and access the virtual machine image and file system:
[et-virt08 ~]# df /xentest Filesystem 1K-blocks Used Available Use% Mounted on et-virt07:/xentest 70562400 2379712 64598336 4% /xentest
[et-virt08 ~]# file /xentest/xentestTravelVM01.dsk /xentest/xentestTravelVM01.dsk: x86 boot sector; partition 1: ID=0x83, active, starthead 1, startsector 63, 208782 sectors; partition 2: ID=0x8e, starthead 0, startsector 208845, 10265535 sectors, code offset 0x48
[et-virt08 ~]# touch /xentest/foo [et-virt08 ~]# rm -f /xentest/foo
Start up the virtual machine (if it has not yet):
[et-virt07 ~]# xm li Name ID Mem(MiB) VCPUs State Time(s) Domain-0 0 1880 8 r----- 50.7
[et-virt07 ~]# xm create xentesttravelvm01 Using config file "/etc/xen/xentesttravelvm01". Going to boot Red Hat Enterprise Linux Server (2.6.18-1.2961.el5xen) kernel: /vmlinuz-2.6.18-1.2961.el5xen initrd: /initrd-2.6.18-1.2961.el5xen.img Started domain xentesttravelvm01
Verify the virtual machine is running:
[et-virt07 ~]# xm li Name ID Mem(MiB) VCPUs State Time(s) Domain-0 0 983 8 r----- 58.2 xentesttravelvm01 1 1024 1 -b---- 9.2
Save the virtual machine on the local host:
[et-virt07 xentest]# time xm save xentesttravelvm01 xentesttravelvm01.sav real 0m15.744s user 0m0.188s sys 0m0.044s
[et-virt07 xentest]# ls -lrt xentesttravelvm01.sav -rwxr-xr-x 1 root root 1075657716 Jan 12 06:46 xentesttravelvm01.sav
[et-virt07 xentest]# xm li Name ID Mem(MiB) VCPUs State Time(s) Domain-0 0 975 8 r----- 110.7
Restore the virtual machine on the local host:
[et-virt07 xentest]# time xm restore xentesttravelvm01.sav real 0m12.443s user 0m0.072s sys 0m0.044s
[et-virt07 xentest]# xm li Name ID Mem(MiB) VCPUs State Time(s) Domain-0 0 975 8 r----- 118.5 xentesttravelvm01 3 1023 1 -b---- 0.0
Run a simple loop inside the guest to print out time and hostname every 3 seconds.
The local host's clock is set to a different time 4hrs ahead of the remote host's clock.
# while true > do > hostname ; date > sleep 3 > done dhcp78-218.lab.boston.redhat.com Fri Jan 12 06:50:16 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 06:50:19 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 06:50:22 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 06:50:25 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 02:22:24 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 02:22:27 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 02:22:30 EST 2007
Verify the virtual machine is running:
[et-virt08 xen]# xm li Name ID Mem(MiB) VCPUs State Time(s) Domain-0 0 975 4 r----- 45.9 xentesttravelvm01 1 1023 1 -b---- 1.3
Initiate the live migration to et-virt08. in the example below et-virt07 is the hostname you are migrating to and <domain-id> muyst be replaced with a guest domain available to the host system.
[et-virt08 ~]# xm migrate --live <domain-id> et-virt07
Verify the virtual machine has been shut down on et-virt07
[et-virt07 xentest]# xm li Name ID Mem(MiB) VCPUs State Time(s) Domain-0 0 975 8 r----- 161.1
Verify the virtual machine has been migrated to et-virt08:
[et-virt08 ~]# xm li Name ID Mem(MiB) VCPUs State Time(s) Domain-0 0 975 4 r----- 46.3 xentesttravelvm01 1 1023 1 -b---- 1.6
Create the following script inside the virtual machine to log date and hostname during the migration. This script performs I/O tasks on the virtual machine's file system.
#!/bin/bash while true do touch /var/tmp/$$.log echo `hostname` >> /var/tmp/$$.log echo `date` >> /var/tmp/$$.log cat /var/tmp/$$.log df /var/tmp ls -l /var/tmp/$$.log sleep 3 done
Remember, that script is only for testing purposes and unnecessary for a live migration in a production environment.
Verify the virtual machine is running on et-virt08 before we try to migrate it to et-virt07:
[et-virt08 ~]# xm li Name ID Mem(MiB) VCPUs State Time(s) Domain-0 0 975 4 r----- 46.3 xentesttravelvm01 1 1023 1 -b---- 1.6
Initiate a live migration to et-virt07. You can add the time command to see how long the migration takes:
[et-virt08 ~]# time xm migrate --live xentesttravelvm01 et-virt07 real 0m10.378s user 0m0.068s sys 0m0.052s
run the script inside the guest:
# ./doit dhcp78-218.lab.boston.redhat.com Fri Jan 12 02:26:27 EST 2007 Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/VolGroup00-LogVol00 2983664 2043120 786536 73% / -rw-r--r-- 1 root root 62 Jan 12 02:26 /var/tmp/2279.log dhcp78-218.lab.boston.redhat.com Fri Jan 12 02:26:27 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 02:26:30 EST 2007 Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/VolGroup00-LogVol00 2983664 2043120 786536 73% / -rw-r--r-- 1 root root 124 Jan 12 02:26 /var/tmp/2279.log dhcp78-218.lab.boston.redhat.com Fri Jan 12 02:26:27 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 02:26:30 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 02:26:33 EST 2007 Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/VolGroup00-LogVol00 2983664 2043120 786536 73% / -rw-r--r-- 1 root root 186 Jan 12 02:26 /var/tmp/2279.log Fri Jan 12 02:26:45 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 02:26:48 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 02:26:51 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 06:54:57 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 06:55:00 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 06:55:03 EST 2007 Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/VolGroup00-LogVol00 2983664 2043120 786536 73% / -rw-r--r-- 1 root root 744 Jan 12 06:55 /var/tmp/2279.log dhcp78-218.lab.boston.redhat.com Fri Jan 12 02:26:27 EST 2007
Verify the virtual machine has been shut down on et-virt08:
[et-virt08 ~]# xm li Name ID Mem(MiB) VCPUs State Time(s) Domain-0 0 975 4 r----- 56.3
Verify the virtual machine has started up on et-virt07:
[et-virt07 xentest]# xm li Name ID Mem(MiB) VCPUs State Time(s) Domain-0 0 975 8 r----- 198.1 xentesttravelvm01 4 1023 1 -b---- 1.0
Run through another cycle migrating from et-virt07 to et-virt08. Initiate a migration from et-virt07 to et-virt08:
[et-virt07 xentest]# time xm migrate --live xentesttravelvm01 et-virt08 real 0m11.490s user 0m0.084s sys 0m0.028s
Verify the virtual machine has been shut down:
[et-virt07 xentest]# xm li Name ID Mem(MiB) VCPUs State Time(s) Domain-0 0 975 8 r----- 221.7
Before initiating the migration start the simple script in the guest and note the change in time when migrating the guest:
# ./doit dhcp78-218.lab.boston.redhat.com Fri Jan 12 06:57:53 EST 2007 Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/VolGroup00-LogVol00 2983664 2043120 786536 73% / -rw-r--r-- 1 root root 62 Jan 12 06:57 /var/tmp/2418.log dhcp78-218.lab.boston.redhat.com Fri Jan 12 06:57:53 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 06:57:56 EST 2007 Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/VolGroup00-LogVol00 2983664 2043120 786536 73% / -rw-r--r-- 1 root root 124 Jan 12 06:57 /var/tmp/2418.log dhcp78-218.lab.boston.redhat.com Fri Jan 12 06:57:53 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 06:57:56 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 06:58:00 EST 2007 Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/VolGroup00-LogVol00 2983664 2043120 786536 73% / -rw-r--r-- 1 root root 186 Jan 12 06:57 /var/tmp/2418.log dhcp78-218.lab.boston.redhat.com Fri Jan 12 06:57:53 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 06:57:56 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 06:58:00 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 02:30:00 EST 2007 Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/VolGroup00-LogVol00 2983664 2043120 786536 73% / -rw-r--r-- 1 root root 248 Jan 12 02:30 /var/tmp/2418.log dhcp78-218.lab.boston.redhat.com Fri Jan 12 06:57:53 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 06:57:56 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 06:58:00 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 02:30:00 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 02:30:03 EST 2007 Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/VolGroup00-LogVol00 2983664 2043120 786536 73% / -rw-r--r-- 1 root root 310 Jan 12 02:30 /var/tmp/2418.log dhcp78-218.lab.boston.redhat.com Fri Jan 12 06:57:53 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 06:57:56 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 06:58:00 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 02:30:00 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 02:30:03 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 02:30:06 EST 2007 Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/VolGroup00-LogVol00 2983664 2043120 786536 73% / -rw-r--r-- 1 root root 372 Jan 12 02:30 /var/tmp/2418.log
After the migration command completes on et-virt07 verify on et-virt08 that the virtual machine has started:
[et-virt08 ~]# xm li Name ID Mem(MiB) VCPUs State Time(s) Domain-0 0 975 4 r----- 67.3 xentesttravelvm01 2 1023 1 -b---- 0.4
and run another cycle:
[et-virt08 ~]# time xm migrate --live xentesttravelvm01 et-virt07 real 0m10.378s user 0m0.068s sys 0m0.052s
At this point you have successfully performed an offline and a live migration test.