Q: How to move filesystems (including root and boot) from one drive to another

[Philip Prindeville]

Mikkel L. Ellertson wrote:
> Philip Prindeville wrote:
>   
>> I recently decided that my build server was running too slow (disk
>> bound) and went out and bought a 3ware 9650SE controller and disks.
>>
>> Got the array configured as a single RAID 5 logical unit (yes, I know,
>> RAID 5 has slow writes...) and ran fdisk on the unit to create partitions.
>>
>> I formatted the partitions...
>>
>> I mounted each of the partitions and did the following:
>>
>> # cd /old-fs-mnt-point
>> # find . -depth -mount -print0 | cpio --null -pdv /new-fs-mnt-point
>>
>> then I ran:
>>
>> # mkswap ...
>> # hal-device | less
>>
>> to get the UUID's.  I edited the new /boot/grub/grub.conf and /etc/fstab
>> and rebooted.  Oops.  Didn't quite work.  Needed to change the BIOS
>> drive assignment order so that the array was now /dev/sda.
>>
>> Rebooted.
>>
>>     
> You also have to tell Grub about the BIOS drive changes. Grub use
> the BIOS to access the drives. The first stage is dumb, and only
> know the physical location of stage 1.5, and if you change the BIOS
> ordering, it looks on the wrong drive.
>
>   
>> System hung: forgot to install the boot blocks.
>>
>> Hmmm...  Tried to run "install-grub /dev/sda" from the Live CD, but that
>> didn't work.  Something about not being able to figure out the BIOS
>> drive number... don't understand why that's relevant, but that's a
>> different story...
>>
>>     
> Grub uses the BIOS to access the hard drive. So you need to know the
> BIOS number of the drive as Grub will see it. This may not be the
> same numbering as when you boot from a CD/DVD.
>
>   
>> So I reran the installer, using a small partition that I had left for
>> whatever... and it wrote the boot blocks and MBR.
>>
>> Editted /boot/grub/grub.conf to change back the default to what I
>> wanted... and rebooted.
>>
>> Now the system hangs at:
>>
>> Red Hat nash version 6.0.52 starting
>> Unable to access resume device (/dev/sda3)
>> mount: error mounting /dev/root on /sysroot as ext3: No such file or
>> directory
>> setuproot: moving /dev failed: No such file or directory
>> setuproot: error mounting /proc: No such file or directory
>> setuproot: error mounting /sys: No such file or directory
>> Mount failed for selinuxfs: on /selinux: No such file or directory
>> switchroot: mount failed: No such file or directory
>>
>>
>>
>> This is FC9 (updated) x86_64 on a Phenom/NVidia MB.
>>
>> What have I forgotten?
>>
>>     
> You need to rebuild the initrd.
>
>   
>> Moving filesystems used to be a lot easier (these are plain ol' ext3
>> filesystems... I'm not using LVM)... and I thought the whole UUID
>> support was to simplify moving drives around, etc.
>>
>> Doesn't seem to be the case.  (Ah, for the days when just about
>> everything that caused a booting system to hang was pretty much
>> self-evident and transparent...)
>>
>> There's a certain amount of tweaking I've done of the configuration, the
>> rpm's I've installed, accounts created, kernel variables that I tune...
>> it's kind of painful to have to do a reinstall from scratch.
>>
>> Also, I wanted to peek inside the initrd.*.img files and see if there
>> was anything in there choking up... but I couldn't figure out how to
>> mount them as loopback filesystems.
>>
>> Any revelation of these mysteries appreciated.
>>
>> -Philip
>>
>>     
> One problem with changing the root file system to a drive on a
> different controller is that the driver may not be in the initrd, so
> the kernel can not access the drive. This is inherent with building
> a kernel with all the driver controllers as modules. But with all
> the different controllers, the only other way to handle it would be
> to have a much larger kernel with lots of unneeded drivers build in,
> or have a initrd with all the drivers. Both would wast a lot of memory.
>
> One other thing you are going to run into if you use SELinux is the
> the new drive is going to have the wrong contexts. You are going to
> have to relabel the files.
>
> Mikkel
>   

And as I found out late last night (too bad you hadn't yet written 
this!), you are exactly right on both counts.

I needed to rerun mkinitrd:

mkinitrd --with=3w-9xxx --with=shpchp --with=libata --with=ahci 
/boot/initrd-`uname -r`.img `uname -r`

then I unpacked the .img file:

mkdir /tmp/xx
cd $!
gunzip -c /boot/initrd-`uname -r`.img | cpio -idv

and edited the "init" file.  Turning off "setquiet" helped a lot (if 
"init" understood command-line args, passing in -v -- or something 
similar -- would have been really handy here).  Verified the "mkrootdev" 
and "resume" lines...

I'm a little disappointed that "init" can't just grab the "root=" 
argument from the command line.

Embedding the actual device name in the "resume":  not sure why this 
happens and why we can't just use UUID.  The man page for "nash" doesn't 
mention "resume" as a built-in, and the .img doesn't contain sbin/resume 
or bin/resume ... Odd.

So, the bottom line is that I was bitten in the butt by not rerunning 
"mkinitrd" with the drivers that I would need to have to reboot on the 
3ware controller.

Plus, as you also astutely point out, I needed to do:

touch /.autorelabel

before rebooting.

It would be great to have a tool (or even a How-To) to make sure a 
transition easier... but as you point out, you don't always know what 
the drive assignments are going to be until after the fact (which is a 
good argument for having a FreeDOS partition... too bad FreeDOS can't 
boot from USB or understand USB drives).

-Philip