[dm-devel] dm-cache fs corruption
Vladimir Smolensky
arizal at gmail.com
Wed Jan 22 01:04:28 UTC 2014
Okay, update on the problem.
Tested with
Linux v-5-231-d1862-150 3.13.0-rc8 #1 SMP Fri Jan 17 17:55:34 GMT 2014
x86_64 Intel(R) Xeon(R) CPU E5620 @ 2.40GHz GenuineIntel GNU/Linux
dm-cache breaks when cache gets full, again
dmesg:
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589304] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589310] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589312] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589315] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589317] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589319] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589322] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589324] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589326] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589328] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589331] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589333] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589335] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589337] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589339] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589342] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589344] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589346] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589349] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589351] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589353] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589355] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589357] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589360] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589362] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589364] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589366] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589368] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589371] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589373] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589375] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589377] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589448] XFS (dm-0):
metadata I/O error: block 0x2aeb4ee40 ("xfs_trans_read_buf_map") error 117
numblks 16
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589455] XFS (dm-0):
xfs_imap_to_bp: xfs_trans_read_buf() returned error 117.
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590202] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590206] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590209] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590212] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590214] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590216] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590218] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590221] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590223] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590225] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590227] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590229] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590232] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590234] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590236] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590238] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590241] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590243] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590245] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590248] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590250] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590252] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590254] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
and so on... it just repeats the same
blockdev --report
RO RA SSZ BSZ StartSec Size Device
rw 256 512 4096 0 26843414528 /dev/sda
rw 256 512 4096 2048 4094689280 /dev/sda1
rw 256 512 4096 7999488 22479372288 /dev/sda2
rw 2048 512 4096 0 20973392756736 /dev/sdb
rw 2048 512 4096 2048 20973390659584 /dev/sdb1
rw 2048 512 4096 0 2397799710720 /dev/sdc
rw 2048 512 4096 2048 1023410176 /dev/sdc1
rw 2048 512 4096 2000896 2396774203392 /dev/sdc2
rw 2048 512 512 0 20973390659584 /dev/dm-0
dmsetup table
storage: 0 40963653632 cache 8:33 8:34 8:17 8192 1 writeback default 0
dmsetup status
storage: 0 40963653632 cache 1152/249856 86805905 9729090 1164931 47563307
0 525209 525209 0 1 writeback 2 migration_threshold 10000000 4
random_threshold 4 sequential_threshold 10000000
Cache size ~2.2T, origin ~20TB
If you need more specific info, please tell me what.
reagrds.
On Wed, Nov 13, 2013 at 12:24 PM, Vladimir Smolensky <arizal at gmail.com>wrote:
> Hello,
> I've been testing dm-cache for use with static web content.
> It appears that, when using big cache ~3TB(+20TB origin dev) dm-cache
> device corrupts the filesystem after block eviction starts to happen.
> If I set smaller cache size - single 480GB ssd(again with 20TB origin
> dev), the dm-cache dev works just fine.
>
>
> dmesg error:
>
> XFS (dm-0): Corruption detected. Unmount and run xfs_repair
> XFS (dm-0): metadata I/O error: block 0x3d4f534e0
> ("xfs_trans_read_buf_map") error 117 numblks 16
> XFS (dm-0): xfs_imap_to_bp: xfs_trans_read_buf() returned error 117.
> XFS (dm-0): Corruption detected. Unmount and run xfs_repair
> XFS (dm-0): Corruption detected. Unmount and run xfs_repair
>
>
> ext4 also got broken.
>
>
> Tested with 3.10.11-gentoo, fedora 19- kernel 3.11.something, xfs, ext4,
> writeback and writethrough modes.
>
>
>
> regards,
> Vladimir Smolensky
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/dm-devel/attachments/20140122/649bc704/attachment.htm>
More information about the dm-devel
mailing list