[libvirt-users] Suggestions on building VM disks from scratch

Wed Apr 4 18:53:26 UTC 2012

On Wed, Apr 04, 2012 at 09:45:18AM +0100, Richard W.M. Jones wrote:
> > Then I created my blank 'disk' file and tried to
> > run virt-rescue on it. It crashed out with an 
> > error from febootstrap.
> 
> First of all, debug this properly:

===================================================================== 
> (1) What is the full error message?

I upgraded and installed a few more things and my error at least
changed, at least so I thought:

# virt-rescue -a recovery2.immortaldata.net.raw
mke2fs: Attempt to write block to filesystem resulted in short write while zeroing block 1048560 at end of filesystem

Could not write 6 blocks in inode table starting at 917506: Attempt to write block to filesystem resulted in short write
febootstrap-supermin-helper: /sbin/mke2fs -t ext2 -Fq '/var/tmp/guestfs.cZge1g/root': failed

But guestfish apparently finds my small root disk to be a 
tasty morsel of fish food and impolitely swallows it whole:

	4.1G    /var/tmp/guestfs.1PFF68
	3.5G    /var/tmp/guestfs.cZge1g
	4.1G    /var/tmp/guestfs.FYAm8O
	4.1G    /var/tmp/guestfs.JP2LNO
	4.1G    /var/tmp/guestfs.qsLaNA
	4.1G    /var/tmp/guestfs.wwKT58
	4.1G    /var/tmp/guestfs.ZueM9V

This maxed out the root partition. Once I found this was happening
I cleared this out and added:

	TMPDIR=/home/tmp

This got me back to the original error message:

# virt-rescue -a recovery2.immortaldata.net.raw
febootstrap-supermin-helper: ext2: parent directory not found: /lib: File not found by ext2_lookup

===================================================================== 
> (2) What is the complete, unedited output of 'libguestfs-test-tool'?

# libguestfs-test-tool

===== Test starts here =====
LIBGUESTFS_DEBUG=1
TMPDIR=(not set)
libguestfs: new guestfs handle 0x23dc190
library version: 1.14.8
guestfs_get_append: (null)
guestfs_get_attach_method: appliance
guestfs_get_autosync: 1
guestfs_get_direct: 0
guestfs_get_memsize: 500
guestfs_get_network: 0
guestfs_get_path: /usr/lib/guestfs
guestfs_get_pgroup: 0
guestfs_get_qemu: /usr/bin/kvm
guestfs_get_recovery_proc: 1
guestfs_get_selinux: 0
guestfs_get_smp: 1
guestfs_get_trace: 0
guestfs_get_verbose: 1
host_cpu: x86_64
Launching appliance, timeout set to 600 seconds.
libguestfs: [00000ms] febootstrap-supermin-helper --verbose -f checksum '/usr/lib/guestfs/supermin.d' x86_64
supermin helper [00000ms] whitelist = (not specified), host_cpu = x86_64, kernel = (null), initrd = (null), appliance = (null)
supermin helper [00000ms] inputs[0] = /usr/lib/guestfs/supermin.d
checking modpath /lib/modules/3.2.0-16-generic is a directory
picked vmlinuz-3.2.0-16-generic because modpath /lib/modules/3.2.0-16-generic exists
checking modpath /lib/modules/3.2.0-21-generic is a directory
picked vmlinuz-3.2.0-21-generic because modpath /lib/modules/3.2.0-21-generic exists
checking modpath /lib/modules/2.6.32-38-server is a directory
picked vmlinuz-2.6.32-38-server because modpath /lib/modules/2.6.32-38-server exists
supermin helper [00000ms] finished creating kernel
supermin helper [00098ms] visiting /usr/lib/guestfs/supermin.d
supermin helper [00098ms] visiting /usr/lib/guestfs/supermin.d/daemon.img
supermin helper [00098ms] visiting /usr/lib/guestfs/supermin.d/init.img
supermin helper [00098ms] adding kernel modules
supermin helper [00259ms] finished creating appliance
libguestfs: [00353ms] begin building supermin appliance
libguestfs: [00353ms] run febootstrap-supermin-helper
libguestfs: [00353ms] febootstrap-supermin-helper --verbose -f ext2 /usr/lib/guestfs/supermin.d x86_64 /var/tmp/guestfs.xtl6Jb/kernel /var/tmp/guestfs.xtl6Jb/initrd /var/tmp/guestfs.xtl6Jb/root
supermin helper [00000ms] whitelist = (not specified), host_cpu = x86_64, kernel = /var/tmp/guestfs.xtl6Jb/kernel, initrd = /var/tmp/guestfs.xtl6Jb/initrd, appliance = /var/tmp/guestfs.xtl6Jb/root
supermin helper [00000ms] inputs[0] = /usr/lib/guestfs/supermin.d
checking modpath /lib/modules/3.2.0-16-generic is a directory
picked vmlinuz-3.2.0-16-generic because modpath /lib/modules/3.2.0-16-generic exists
checking modpath /lib/modules/3.2.0-21-generic is a directory
picked vmlinuz-3.2.0-21-generic because modpath /lib/modules/3.2.0-21-generic exists
checking modpath /lib/modules/2.6.32-38-server is a directory
picked vmlinuz-2.6.32-38-server because modpath /lib/modules/2.6.32-38-server exists
supermin helper [00000ms] finished creating kernel
supermin helper [02595ms] finished mke2fs
supermin helper [02596ms] visiting /usr/lib/guestfs/supermin.d
supermin helper [02596ms] visiting /usr/lib/guestfs/supermin.d/daemon.img
supermin helper [02601ms] visiting /usr/lib/guestfs/supermin.d/init.img
febootstrap-supermin-helper: ext2: parent directory not found: /lib: File not found by ext2_lookup
libguestfs: error: external command failed, see earlier error messages
libguestfs-test-tool: failed to launch appliance
libguestfs: closing guestfs handle 0x23dc190 (state 0)

=====================================================================  
> (3) What version of libguestfs & febootstrap are using and where did
> you get them from?

Updated from official Ubuntu Precise Pangolin server last night.

	Package: libguestfs0
	Architecture: amd64
	Source: libguestfs
	Version: 1:1.14.8-1

	Package: febootstrap
	Architecture: amd64
	Version: 3.12-1

> Assuming the filesystem was in /tmp/root.tar.gz, the following code
> will do this:
> 
>   guestfish <<EOF
> 
>   sparse /tmp/test.img 11G
>   run
> 
>   part-init /dev/sda mbr
>   # 9GB sda1
>   part-add /dev/sda p 64 $(( 9*1024*1024*2 ))
>   # remainder in sda2
>   part-add /dev/sda p $(( 9*1024*1024*2 + 1 )) -64
> 
>   mkfs ext4 /dev/sda1
>   mkswap /dev/sda2
> 
>   mount /dev/sda1 /
>   tgz-in /tmp/root.tar.gz /
> 
>   EOF
> 
> Example:
> 
>   $ sh test.sh 
>   $ virt-df -a test.img -h
>   Filesystem                                Size       Used  Available  Use%
>   test.img:/dev/sda1                        9.0G       276M       8.3G    4%
>   $ ll -h test.img 
>   -rw-rw-r--. 1 rjones rjones 11G Apr  4 09:39 test.img
> 
> Whether this would actually boot is another question: you may also
> need to add some grub commands to set up the bootloader, *or* (better
> and easier IMHO) set up libvirt so that it boots from an external
> kernel + initrd.

If I can get the grub installed it should, since the 
original are in most cases virtual servers built on vmdk 
disks on a licensed VMware ESX server.

The other way I found to get to this particular point was:

	$ dd if=/dev/zero of=/recover2.us.net.raw count=20M
	$ losetup /dev/loop0 recover2.us.net.raw
	$ cfdisk /dev/loop0
		create a 9G bootable linux partition and a 1G swap
	$ kpartx -a -v /dev/loop0
	$ mkfs.ext4 /dev/mapper/loop0p1 -m 0.01 -L "RecoverTest"
	$ mkdir tmp
	$ mount /dev/mapper/loop0p1 tmp
   From the backup server:
	$ cd <to the lastest dirvish backup set>
	$ rsync -av tree/ root at rebuild.server.net:/srv/idcpool/tmp/

But I could not figure out a safe way to install grub that
would not put the boot disk of a critical server at risk
of getting its MBR clobbered. Grub is very good at doing
such things and very difficult to convince to do what *you*
want rather than what *it* wants.

I much prefer the technique of just using a guestfish script to
do it all as you showed, but I still need to load up the backup 
set and install grub properly.

> > It also lacks access to rsync.
> 
> It's not the first time that someone has asked for rsync, and it
> wouldn't be too hard to add.  However note that rsync really gives you
> no benefit when you're creating a filesystem from scratch, because
> there's no original to rsync against.  If you are updating a
> filesystem image then rsync makes sense.

Actually, what the situation is that for our restoration
of *last resort*, we have a server that does incremental
daily and for some key infrastructure 4 times daily backups
via dirvish. So what we need dirvish for is so that we
can connect to the backup server and copy the most recent
copy of the failed device. 

So yes, we do have a good reason to use rsync as it gives
us a very simple secure data transfer of the files from
one server to the new image.

The alternative is to do the rsync over to the rebuild
server, being careful not to get links messed up, tar.gz
the result and then pray the result will actually recreate
things, links, dev files, uid.gid's, modes, etc on when
unpacked. I have an inherent distrust of adding extra
steps into things if I can avoid them, particularly
when discussing a mission critical last resort failsafe.

Oh, and yes, we do separately back up database files via
the appropriate snapshotting and those would have to
be overloaded seperately after the main restoration.

That's one problem, the big one I am working on. The other
I can talk to you about after I get this one sorted, and
that is using virt-rescuse/guestfish in the virtualization
of an old SuSE server on end of life hardware. It of course
has lots of LVM's on the system disk just to make things
interesting...