[linux-lvm] harddisk dies while pvmove is in progress

[Robert Schöftner]

hi!

i recently noticed some read errors off of one of my "storage" 
harddisks. layout was something like 1*2tb + 1*1,5tb in a linear 
mapping, 1 VG (storage_1), 2 LVs (shares, homevideo). the 1,5gig 
harddisk failed.

so i ordered some disks, built a raid5 out of 4 2tb disks, and added it 
to storage_1. I pvmoved the 2tb disk to the new raid. then i tried to 
move the failed 1,5tb disk to the new raid. this process stopped when 
pvmove was at about 50%, when the drive went completely dead. server was 
rebooted.

it seems pvmove was able to move eveything of LV shares to the raid.

situation as of now:

* LVs fail to activate (needed lvchange -a y --partial)
* after activation, homevideo is completely dead, shares seems to work 
without any problems (though it generates lots of device-mapper: raid1: 
Unable to read primary mirror during recovery messages)
* percentage-counter counting, unsure if actual disk activity happens
* pvmove complains:

# pvmove
   /dev/dm-4: read failed after 0 of 4096 at 0: Eingabe-/Ausgabefehler
   /dev/dm-4: read failed after 0 of 4096 at 971224580096: 
Eingabe-/Ausgabefehler
   /dev/dm-4: read failed after 0 of 4096 at 971224637440: 
Eingabe-/Ausgabefehler
   /dev/dm-4: read failed after 0 of 4096 at 4096: Eingabe-/Ausgabefehler
   /dev/dm-5: read failed after 0 of 4096 at 1500298280960: 
Eingabe-/Ausgabefehler
   /dev/dm-5: read failed after 0 of 4096 at 1500298338304: 
Eingabe-/Ausgabefehler
   Couldn't find device with uuid u1F6AW-pvCR-gM1c-O1c5-oWa0-s1Nd-oKTkf5.
   Cannot change VG storage_1 while PVs are missing.
   Consider vgreduce --removemissing.


# lvs -a -o +devices storage_1
   /dev/dm-4: read failed after 0 of 4096 at 0: Eingabe-/Ausgabefehler
   /dev/dm-4: read failed after 0 of 4096 at 971224580096: 
Eingabe-/Ausgabefehler
   /dev/dm-4: read failed after 0 of 4096 at 971224637440: 
Eingabe-/Ausgabefehler
   /dev/dm-4: read failed after 0 of 4096 at 4096: Eingabe-/Ausgabefehler
   /dev/dm-5: read failed after 0 of 4096 at 1500298280960: 
Eingabe-/Ausgabefehler
   /dev/dm-5: read failed after 0 of 4096 at 1500298338304: 
Eingabe-/Ausgabefehler
   Couldn't find device with uuid u1F6AW-pvCR-gM1c-O1c5-oWa0-s1Nd-oKTkf5.
   LV        VG        Attr   LSize Origin Snap%  Move           Log 
Copy%  Convert Devices
   homevideo storage_1 -wI-a- 
1,08t                                                 pvmove0(126141)
   homevideo storage_1 -wI-a- 
1,08t                                                 /dev/md127(553732)
   [pvmove0] storage_1 p-C-ao 1,36t               unknown device      
38,07         unknown device(0),/dev/md127(604932)
   [pvmove0] storage_1 p-C-ao 1,36t               unknown device      
38,07         unknown device(126141),/dev/md127(731073)
   shares    storage_1 -wI-a- 
2,59t                                                 /dev/md127(128000)
   shares    storage_1 -wI-a- 
2,59t                                                 pvmove0(0)
   shares    storage_1 -wI-a- 
2,59t                                                 /dev/md127(485699)
   shares    storage_1 -wI-a- 
2,59t                                                 /dev/md127(0)

# dmsetup table
storage_1-pvmove0-missing_1_0: 0 1896923136 error
storage_1-shares: 0 2930270208 linear 9:127 1048579072
storage_1-shares: 2930270208 1033347072 linear 252:5 0
storage_1-shares: 3963617280 557326336 linear 9:127 3978849280
storage_1-shares: 4520943616 1048576000 linear 9:127 3072
storage_1-homevideo: 0 1896923136 linear 252:5 1033347072
storage_1-homevideo: 1896923136 419430400 linear 9:127 4536175616

# dmsetup info -c
storage_1-pvmove0             252   5 L--w    2    2      0 
LVM-ONKgAr1yc19fIeDyMClPFjDv92vxDEAFyOZtSj8iJexF2g6PMNRff3rYytIZ5RYJ
storage_1-pvmove0-missing_1_0 252   4 L--w    1    1      0 
LVM-ONKgAr1yc19fIeDyMClPFjDv92vxDEAFyOZtSj8iJexF2g6PMNRff3rYytIZ5RYJ-missing_1_0
storage_1-shares              252   6 L--w    0    4      0 
LVM-ONKgAr1yc19fIeDyMClPFjDv92vxDEAFvbEPv7JJKoctUb2kNtGqjFrfsrHiS0v3
storage_1-homevideo           252   7 L--w    0    2      0 
LVM-ONKgAr1yc19fIeDyMClPFjDv92vxDEAFwFm52QuLABa7OKlHdYp3yEe8wxp6K07I

# dmsetup status
storage_1-pvmove0: 0 1033347072 linear
storage_1-pvmove0: 1033347072 1896923136 mirror 2 252:4 9:127 
89145/1852464 1 SA 1 core
storage_1-pvmove0-missing_1_0: 0 1896923136 error
storage_1-shares: 0 2930270208 linear
storage_1-shares: 2930270208 1033347072 linear
storage_1-shares: 3963617280 557326336 linear
storage_1-shares: 4520943616 1048576000 linear
storage_1-homevideo: 0 1896923136 linear
storage_1-homevideo: 1896923136 419430400 linear

[irrelevant VGs removed from output]

So, LV homevideo is definitely lost, but shares seems to be OK, modulo 
the "running"/interrupted pvmove. Is there a way to clean up this mess 
without creating another LV and copying the contents of shares over?

thanx
   Robert