[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [linux-lvm] Data corruption on large, multi-device filesystem



joe eiler net wrote:

I have recently run into this problem also.  I have seen it happen on SuSe 9.2,
Fedora Core 2 and 3, and vanilla kernels 2.6.8.1, 2.6.9, and 2.6.10.
All of my tests were using xfs.

It happens whenever 2 or more devices are striped together with a total volume
size greater than 2TB.  I have played with a single 4TB raid (12x 400GB RAID5)
and did not see any corruption (but I did not fill the disk either).

I initially saw the problem running video files over samba. But have recreated
the problem by simply copying some large (5GB+) files and then checking
md5sums.

I don't see any corruption on the files unless I specify the -i option to
lvcreate.  I usually see data corruption within an hour using my current tests.



To verify, this corruption you are seeing only happens when you have a LV larger than 2TB
and when you use striping specifically with lvcreate -i.
Has anyone experienced data corruption with >2TB LV and no striping?


Randall
-

Let me know if I can be of any assistance.
Joe


Quoting Jens Beyer <jbe webde-ag de>:




Hi,

I get severe data corruption using an logical volume larger
then 2 TB. Finally I was able to track down device mappper or
lvm as last suspects.

My first guess where problems with filesystems but recently
I tried using md / RAID0 - and didnt have any errors of any
kind. I would prefer using LVM since we want to use snapshots
to simplify backup, but I have no clue how to further debug.

On a system with 3 devices each larger then 1 TB and a logical
volume striped over all devices some data gets corrupted while
written (or read ?) from disk. This shows up as md5 or crc sums
changes on sequenced reads of files if filecache is not involved
(by reading a lot data).
On ext2fs there are error while writing data (kernel: EXT2-fs error
(device dm-0): ext2_new_block: Allocating block in system zone -
block = 722239884), on other filesystems successive fsck/repairs
shows corrupted metadata.

The system setup is
- Three 29160B Adaptec scsi-controller each with one
 ATA-Disk Raid sized 1240 GB, (dual PIII, HP DL360 G2, 2 GB Ram)
- Volume group over all three devices, logical volume stripped
 full size (3.7 TB)
- Filesystem either ext2fs/ext3fs (1.34), reiserfs (3.6.13) or
 xfs (2.6.25)

- host:~ # lvm version
 LVM version:     2.00.33 (2005-01-07)
 Library version: 1.00.21-ioctl (2005-01-07)
 Driver version:  4.3.0
- 2.6.10 vanilla + 2.6.10-udm1 patches

The problems where initially discovered on 2.6.8, tracked on 2.6.9-udm
and also occurs if only 2 devices (sum 2.4 TB) are used.

For a limited time I will be able to further debug the system though
it takes some time to generate more then 2 TB of data
(max seq read/write rate is ~80 MB/s).

Jens

--
Nur tote Fische schwimmen mit dem Strom



--
..:.::::
Randall Jones     GST      NASA Goddard Space Flight Center
HPC Visualization Support       http://hpcvis.gsfc.nasa.gov
Scientific Visualization Studio    http://svs.gsfc.nasa.gov
rajones svs gsfc nasa gov      Code 610.3      301-286-2239



[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]