[linux-lvm] LVM thin LV filesystem superblock corruption

[Andres Toomsalu]

On 25.03.2013, at 21:45, Mike Snitzer wrote:

> On Fri, Mar 22 2013 at 11:12am -0400,
> Andres Toomsalu <andres at active.ee> wrote:
> 
>> Update! Issue seems to be active only with PERC H800 and MD1200 disks - local raid with PERC H700 and lvm thin lv-s work fine without corrupting on reboot.
>> 
>> 
>> We stumbled on strange lvm thinly provisioned LV filesystem corruption case - here are steps that reproduce the issue:
>> 
>> lvcreate --thinpool pool -L 8T --poolmetadatasize 16G VolGroupL1
>> lvcreate -T VolGroupL1/pool -V 2T --name thin_storage
>> mkfs.ext4 /dev/VolGroupL1/thin_storage
>> mount /dev/VolGroupL1/thin_storage /storage/
>> reboot
> 
> Couple things:
> 1) mkfs.ext4 does buffered IO so there is no gaurantee the superblock or
>   any other block group destriptors, have actually been committed to
>   non-volatile storage when mkfs.ext4 completes

I see. While this could have been true during the tests I did later after discovering the issue - its a bit unlikely the case for first time the issue appeared - as there was about 24h time window between mkfs and host reboot then - and data was copied into new /storage LV.

One more strange thing about the issue - during the tests I made repeatedly thin LV setups cycles with different pool, pool metadata and lv sizes - from 50G to 8TB - which seemed not to affect anything.
Once it failed - it failed repeatedly - until the moment it started to work again and then it worked repeatedly - that behaviour actually could support the buffered IO theory…
Right now I cant reproduce the issue anymore at will - waiting for failure again.

> 2) reboot sequence is very distro specific; /storage may not have been
>   unmounted before reboot -- if it was unmounted then all data
>   should've been pushed out to non-volatile storage

Distro is CentOS 6.4 - should unmount LV-s correctly as far as I know.

> 
> So if you add this to command before "reboot" do you no longer have
> missing data after the system reboots?:
> 
> echo 3 > /proc/sys/vm/drop_caches

will try next time

> 
>> # NB! without host reboot unmount/mount succeeds!
>> 
>> [root at node3 ~]# mount /dev/VolGroupL1/thin_storage /storage/
>> mount: you must specify the filesystem type
>> 
>> Tried also to set poolmetadatasize to 2G, 14G, 15G and pool size to 1T, 2T - no change - corruption still happens.
>> 
>> Hardware setup: 
>> * Underlaying block device (sdb) is hosted by PERC H800 controller and disks are coming from SAS disk expansion box (DELL MD1200).
> ...
>> What could be the issue here?
> 
> I assume by "reboot" you mean the host (with the PERC card) never loses
> power?

Yes - soft reboot - no power cut.

> 
> What layers of hardware writeback caching are in place in the
> H800+MD1200 case vs H700+localraid?

H800 has RAID10 array with cache set to 'writethrough'
H700 has RAID10 array with cache set to 'writeback'