[linux-lvm] Please help: LVM Recovery on base RHEL3 system.

[Pete Toscano]

A friend of mine recently built a fileserver from some spare stuff he had
around along with a spare RHEL3 CD from his work.  He has no RHEL
entitlements, so no binary updates.  (I would have encouraged him to at
least use an upgradeable system, but that's neither here nor there, now.)
As for the system, it's very basic with a system hard drive, a DVD drive,
PCI IDE card, and five drives dedicated to a RAID5 volume.

He followed the software RAID and LVM how-tos, eventually building a 930+GB
RAID5 volume md0.  He then used md0 as the sole PV to build a "raid_discs"
VG.  This VG was split into three LVs: home, music, and pics.  Each LV was
formatted EXT3 and set to mount in /etc/fstab:

/dev/raid_discs/music  /raid/music              ext3    defaults        0 0
/dev/raid_discs/pictures  /raid/pictures        ext3    defaults        0 0
/dev/raid_discs/home  /raid/home                ext3    defaults        0 0

This done, he proceeded to copy 260G of media to the server.  Thinking
everything we living safely on a RAID5 device now, he proceeded to delete
the sources.  *sigh*

Everything worked fine until he rebooted the server (cleanly, I'm told).
After the machine came up, it wouldn't mount his LVs.  The RAID device is
there:

[root at Server lvmconf]# cat /proc/mdstat
Personalities : [raid5]
read_ahead 1024 sectors
Event: 1
md0 : active raid5 hdh[4] hdg[3] hdf[2] hde[1] hdc[0]
      976793600 blocks level 5, 128k chunk, algorithm 2 [5/5] [UUUUU]

unused devices: <none>

>From what I can tell, though, it doesn't look like LVM is seeing the RAID
device as a PV anymore:

[root at Server lvmconf]# pvdisplay -vv /dev/md0
pvdisplay -- checking physical volume name "/dev/md0"
pvdisplay -- reading data of physical volume "/dev/md0" from disk
pvdisplay -- no physical volume identifier on "/dev/md0"

Am I reading this correctly?

I tried to do a vgcfgrestore and got this:

[root at Server lvmconf]# vgcfgrestore -n raid_discs -vv  /dev/md0
vgcfgrestore -- INFO: using backup file "/etc/lvmconf/raid_discs.conf"
vgcfgrestore -- locking logical volume manager
vgcfgrestore -- restoring volume group "raid_discs" from
"/etc/lvmconf/raid_discs.conf"
vgcfgrestore -- checking existence of "/etc/lvmconf/raid_discs.conf"
vgcfgrestore -- reading volume group data for "raid_discs" from
"/etc/lvmconf/raid_discs.conf"
vgcfgrestore -- reading physical volume data for "raid_discs" from
"/etc/lvmconf/raid_discs.conf"
vgcfgrestore -- reading logical volume data for "raid_discs" from
"/etc/lvmconf/raid_discs.conf"
vgcfgrestore -- checking volume group consistency of "raid_discs"
vgcfgrestore -- checking volume group consistency of "raid_discs"
vgcfgrestore -- reading physical volume "/dev/md0"
vgcfgrestore -- ERROR "pv_read(): PV identifier invalid" reading physical
volume "/dev/md0"

Is there anything that we can do to save his data?  I know that with
partition tables, if you know them in advance, you can re-write them and, if
you're exact, you won't lose any data.  Is it possible to do the same with
pvcreate?  Could I re-init the PV, then have the VG suddenly be happy?
Suggestions?

Would it help matters to boot off a live CD like Helix and try recovery that
way?  I'm a bit hesitant to do that as I'd rather try to get things working
with the tools that created the VG in the first place and I'm not sure how
nicely LVM2 would play with LVM.  OTOH, since we're dealing with base RHEL3,
we could also be hitting a bug that was later patched.  (After recovery, I'm
getting him to change the OS ASAP.)

Thank you so much.  I really hope he hasn't borked his pics of his kids.

pete
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-lvm/attachments/20070227/7dec0765/attachment.htm>