[linux-lvm] RAID support...
Kirby C. Bohling
kbohling at birddog.com
Mon Dec 31 00:28:02 UTC 2001
Colin,
My apologies, this message is way to long. You don't read it unless you
want to know the specifics of how some idiot transformed a stock RH7.1
no raid install into a RAID1/LVM configuration by hacking it out by hand.
I have failed to ever get the installer to work correctly to install
directly onto the machine. Disks are cheap so I have around 150GB of WD
7200 drives.
/dev/hda -- Root filesystem/install disk
/dev/hdb -- half of /dev/md0
/dev/hdc -- half of /dev/md0
/dev/hdd -- half of /dev/md1
/dev/hde -- half of /dev/md1
(Okay in real life that isn't the case they are really different letters,
all the drives are on seperate controllers for performance reasons).
I normally install RedHat onto a regular disk drive, no mirror, no LVM,
nothing special. Basically it is a 4 click version of the install after
I get done partitioning the drives. After getting that all done up I
config the 4 drives into 2 seperate mirrors (read the how-to).
Partition /dev/hd[bcde] into on giant partition whose type is OxFD.
Basics of that are edit /etc/raidtab to this:
raiddev /dev/md0
raid-level 1
nr-raid-disks 2
chunc-size 32
persistant-superblock 1
device /dev/hdb1
raid-disk 0
device /dev/hdc1
raid-disk 1
mkraid /dev/md0
The persistant-superblock is important. You really want that working or
you might have trouble getting the LVM to autostart during the boot
sequence.
And similar for /dev/md1.
Then do the pvcreate on /dev/md0 and /dev/md1. To be honest, I have to
just type pvcreate and read the help line to figure it out.
vgcreate to create the volume group, nothing special, again read the help.
I have never gone to the trouble of moving the root onto LVM, in my
opinion it isn't worth the hassle (kernel/modules/boot sequence hell
ensues if you screw anything up), it becomes a real hassle because I
hate boot floppies and I generally never make one. If I can't recover
using a RedHat CD I generally don't configure the system that way (yes I
that am lazy).
When I install I am sure to create the following filesystems:
/tmp
/boot
/usr
/usr/local
/var
All of these were installed onto /dev/hda not onto any mirrors or any
partitions.
Once you get here it gets a little scary, but you are almost done. Log
in as root, change to single user mode. Now I make a logical volume
that matches each of the above filesystems. I make a mountpoint for
each non-LVM version of the filesystem in called xxx.old where xxx is
the original filesystem. I configure all that up in /etc/fstab so it is
documented in case I forget something. Make the logical volumes, and
/etc/fstab entries for both. Oh yeah, the xxx.old are mount read only.
Now do the following
mkfs -t xxx /dev/vg/lv
mount /tmp
( cd /tmp.old && tar cf . - ) | ( cd /tmp && tar xvpf - )
Do that once for each filesystem. Note that the tar will only exist in
the pipe so you don't need enough disk space to store it, and the p in
the second tar ensures that ownership and permissions are preserved.
Okay so this screws up one thing that is important. /tmp has special
permissions. The bit t is set on the world execute bit which is
important, it is the sticky bit and controls who is allowed to delete
which files in a directory ( I don't know the exact meaning I just know
it is important to get it right on /tmp ).
If you don't like using tar, you can use dd or dump/restore. On the dd
version, you have to make the logical volumes the same size as the
original partitions or bigger. If they are bigger you will have to use
a filesystem resize command in order to not waste space. (The dd
version of the command scared me to death, but it appears to work just
fine.) Anybody with real experience care to comment? The machine I am
typing this email on had the dd command used to move filesystems.
cat /proc/partitions | egrep "block|/dev/hda6"
(I believe the block size under Linux is 512 bytes don't quote me on that)
dd if=/dev/hda6 of=/dev/vg/lv bs=512 count="3rd number on the grep"
Actually I believe on my system I just did
dd if=/dev/hda6 of=/dev/vg/lv,
but I am not sure, basically I dinked with it until it mounted correctly
and returned the same amount of space used, and then fsck'ed correctly.
You can use dump and restore, but make sure the filesystems are mounted
read only. The reason for this is that under a 2.4 kernel it is my
understanding that dump doesn't work any more. See lkml history for the
reason why. My very limited understanding is that the cache for
directories moved, and that dump doesn't know about it. So you could
change a directory and it would only be in the cache, but dump would
look on disk and in the old cache and not find the change in the new
cache. Linus said dump was no longer safe, and I believe him.
My guess is that having it mounted read only makes it safe (that is my
guess don't take it as truth, I am not a kernel hacker I wouldn't know).
I haven't ever done it this way, but it should work fine if you are
comfortable/good with dump and restore.
I normally have raid1 and lvm built into my kernel. I use ext3 as my
filesystem (I run RH7.2). I have used reiserfs, but I like some of the
stuff that ext3 does a lot (like the fact that it can be configured to
journal all data as opposed to meta-data only) yes it is slower that
way. I haven't had the nerve to try the off device journal just yet. I
will get to it someday, just not today.
I have done it using either grub or lilo, with and w/o initrd images. I
have done all of them as a module thru initrd images (using that it
should be easy to do a LVM root filesystem, no I have never done it
makes recovery a pain). If you are going to do that I would have an LVM
partition that you boot to, and a duplicate non-LVM root in case of
emergency that has the proper tools on it to recover the LVM and
re-build and install a kernel from it. I normally have at least one
partition I can boot from that has all the appropriate modules and user
land tools to do recovery from in case all hell breaks loose on the
mirrors or LVM during a kernel recompile/install cycle.
I then remove all of the change the the *.old filesystems to be noauto
and leave them around in cause I want to do a "restore" of a machine.
To really do that correctly you have to re-mount the root filesystem
read only while in single user mode and then store it way somewhere too.
I submitted patches to mount and e2fsprogs to support disk labels on LVM
so hopefully that should get become standard, I apply my patches and
install mount and e2fsprogs. Then I label all of the ext2/3 filesystems
and happy mounting (the LVM guys should have done that forever ago, LVM
is much easier with that working).
About the only configuration I have never tried is to get it all up and
running using devfs.
Colin, if you have a specific setup you want to achieve just let me know
I have a couple of 40 GB drives lying around at home and a machine to
tinker with. I would happily figure out how to get an all raid all LVM
configuration up and running and write up a mini-HOW-TO on my experince.
To be honest, I have never used LVM in a production environment, it is
just something I tinker with at home because it is fun, and is the
easiest way to convert from ext3 -> reiserfs and back. If nothing else,
I could hack a RH7.1/7.2 installer to do LVM. That will take longer to
get a working version of that, but it would be much more fun. CC me
personally to get my attention if you want more specifics or want me to
tinker with a specific setup. Just because the recovery would be a
pain, doesn't mean I won't figure it out if somebody would find it useful.
It might also be a bit of a total hack depending on how I go about
getting it up and running.
Thanks,
Kirby
Colin Coe wrote:
> Hi all
>
> Kirby, can you describe how you use software raid and lvm? I have been
> trying to install RH7.1 directly onto software raid and lvm without success.
>
> Any help much appreciated.
>
> CC
>
>
>>-----Original Message-----
>>From: Kirby C. Bohling [SMTP:kbohling at birddog.com]
>>Sent: Saturday, December 29, 2001 2:36 AM
>>To: linux-lvm at sistina.com
>>Subject: Re: [linux-lvm] RAID support...
>>
>>Michael,
>>
>> I am replying to you personally, and not forwarding it to the list,
>>not
>>sure if that is correct netiquetee or not. Here is the a link to the
>>FAQ entry that discuss precisely the issue. Basically, the primary
>>developer disagrees with you.
>>
>>I happen to agree with him, but that is because I end up using software
>>RAID and mirror everything and then build LVM devices out of them.
>>Technically there is not speed advantage for doing it for anything
>>except for RAID0 in the LVM. It causes a lot of code duplication
>>between the software RAID and the LVM.
>>
>> A strong case could be made for a more GUI/command line capable disk
>>
>>volume management that encompassed RAID/LVM/GFS/ext3/XFS/JFS/ReiserFS/nb
>>and every other thing under the sun to make administration easier, but
>>well, somebody will get to it, just not today.
>>
>>http://tech.sistina.com/cgi-bin/lvm-faqprocessor.cgi?q=Why%20does%20LVM%20
>>not%20support%20mirroring%3F
>>
>>
>> Kirby
>>
>>
>>Michael O'Rourke wrote:
>>
>>
>>>To whom it may concern,
>>>
>>>Not that this is a show stopper, but it seems that the LVM is missing
>>>important capabilities such as RAID 1 & 5. I see that you already
>>>support RAID 0, but that's it. In my opinion, it makes sense to merge
>>>the RAID support (seperate product) with the LVM product. Because the
>>>two really complement each other.
>>>Sincerely,
>>>Michael O'Rourke
>>>Unix System Admin
>>>Per-Se' Technologies
>>>
>>>
>>>
>>>_______________________________________________
>>>linux-lvm mailing list
>>>linux-lvm at sistina.com
>>>http://lists.sistina.com/mailman/listinfo/linux-lvm
>>>read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html
>>>
>>
>>--
>>Real Programmers view electronic multimedia files with a hex editor.
>>
>>
>>_______________________________________________
>>linux-lvm mailing list
>>linux-lvm at sistina.com
>>http://lists.sistina.com/mailman/listinfo/linux-lvm
>>read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html
>>
>
> **************************************************************************
> Bunnings Legal Disclaimer:
>
> 1) This document is confidential and may contain legally privileged
> information. If you are not the intended recipient you must not
> read, copy, distribute or act in reliance on it.
> If you have received this document in error, please telephone
> us immediately on (08) 9365-1555.
>
> 2) All e-mails sent to and sent from Bunnings Building Supplies are
> scanned for content. Any material deemed to contain inappropriate
> subject matter will be reported to the e-mail administrator of
> all parties concerned.
>
> **************************************************************************
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm at sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html
>
--
Real Programmers view electronic multimedia files with a hex editor.
More information about the linux-lvm
mailing list