[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

[linux-lvm] LVM2, NFS and random device (major:minor) numbers




I have a NFS server still running CentOS 4.0 (kernel 2.6.9-5.0.5.ELsmp, lvm2-2.00.31-1.0.RHEL4). It has two 3ware 9500S-8 controllers translating to two RAIDs at /dev/sda and /dev/sdb. The OS is on a small regular IDE disk. Both sda and sdb are 2TBs, have a GPT partition table and have one LVM flagged partition taking up the whole 2TBs -- /dev/sda1 and /dev/sdb1 each made into PVs. I created two volume groups, vg1 on sda1 and vg2 on sda2. No other PVs involved.

I started making volumes on vg1 and no problems. Recently I made a couple volumes on vg2 and started using them with no problems. Then yesterday I rebooted the box for the first time since creating volumes on vg2. Suddenly I get reports from my users who have NFS mounted the volumes that data is missing or wrong. Looking closely I see that volumes off this server are suddenly mounted in the wrong spots off by two (volume1 is mounted where volume3 is supposed to be, volume2 is mounted where volume 4 is supposed to be, and son on).

NFS depends on what the server says the underlying device id is of the exported volume and if that changes, then NFS mounts are going to change out from under themselves. And everytime I add a volume and reboot the device ids change.

I can see that vgscan always sees vg2 before vg1 but gives no way to change the order it sees things. I suspect if I had filled up vg2 first and then made volumes on vg1, I never would have discovered this problem unless
I went back and deleted a volume on vg2.

After frantically search the net about what the hell is going on I first find a Changelog for lvm1 saying a bug just like this got fixed. But this is lvm2 and it seems to not be fixed. I try upgrading to CentOS 4.2 but testing the issue again by creating more volumes on vg2 and the rebooting show the underlying device ids change once again.

I finally come upon this old post to this list:

http://www.redhat.com/archives/linux-lvm/2005-May/msg00029.html

QUESTION 1: My first question is why aren't major/minor assignments persistent in the first place by default? I cannot see a reason why not and the current default behavior is just asking for trouble for anyone doing NFS export of their LVM volumes. It also screwes up many incremental backup programs that depend on persistent device numbers so the exports fsid= thing is not a full solution.

It seems like something about this should be screamed as a loud warning in the LVM HOWTO if things really are supposed to work way, but I could find no reference to it at all.

On my server, I turned off the NFS server, umounted all the volumes and desperately tried to think of a way I could get my volumes back to their original device IDs without having to possibly reboot over 500 Linux clients to clear the issue. My first attempt was to use

   lvchange -My --major ### --minor ###

on my volumes. On the first volume in vg1 I did

   lvchange -My --major 253 --minor 1

and on the second I did the same with minor of 2, and so on.  Then in
vg2 I decided the easiest thing was give it a different major, so
on its first volume I did:

   lvchange -My  --major 254 --minor 1

and so on on the other five volumes in vg2. Then reboot. And discovered that my first five volumes in vg1 refused to mount complaining another
volume was already using its major/minor numbers.

Doing a 'lvdisplay -v' on the first volume in vg2, I could see it was
still using major 253 despite having given it major 254 above.  In
fact the lvdisplay showed inconsistent data like this:

  Persistent major       254
  Persistent minor       1
  Block device           253:101

QUESTION 2: I then saw in the mail post above that kernel 2.6 ignores
the major number given.  But the lvchange command insisted that I give
it anyway giving me the illusion that it mattered.  So I think this
is a bug in lvchange that should be fixed.  But is the major number
something that could still be in flux and someday I find all my
volumes have changed device number again due to the device mapper giving
all LVM volumes sam random new major?

So I ending up going to scheme where on vg2 volumes I still use major 253 and start the minor numbers at 100+<volnum> so the first volume on vg2 has minor 101. This finally seemed to work okay after reboot (assuming
the major number is never going to change).

QUESTION 3: So it seems as standard practice on using lvcreate from now on I need to -My to prevent these problems. Is this what everyone else is doing?

QUESTION 4: I have several other severs configured just like the one above. And on one I know have created new volumes on the 2nd volume group and have not rebooted it since. I am now afraid to do so and I am trying to figure out what I can do. Can I do 'lvdisplay' on each live, NFS exported volume to get its current major:minor and then run 'lvchange -My' on it also live to set it to persistant? Or will it have to be offline?


--
---------------------------------------------------------------
Paul Raines                email: raines at nmr.mgh.harvard.edu
MGH/MIT/HMS Athinoula A. Martinos Center for Biomedical Imaging
149 (2301) 13th Street     Charlestown, MA 02129	    USA



[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]