Filesystem gone readonly
Hiren Joshi
josh at moonfruit.com
Sat Apr 11 00:32:30 UTC 2009
I've got a really weird situation here. I'm using RHEL 4 and connecting
to an EMC storage device using fibre and qla2300.
The luns are put into LVM and we have a number of 400G partitions coming
off that, I made a snapshot and ran fsck -yn on it with the following
output:
fsck 1.35 (28-Feb-2004)
e2fsck 1.35 (28-Feb-2004)
Pass 1: Checking inodes, blocks, and sizes
Inode 2392655 has illegal block(s). Clear? no
Illegal block #9 (4101620032) in inode 2392655. IGNORED.
Inode 2392655, i_blocks is 672, should be 664. Fix? no
Inode 29507663 has illegal block(s). Clear? no
Illegal block #9 (4101620032) in inode 29507663. IGNORED.
Inode 32918223 has illegal block(s). Clear? no
Illegal block #9 (4101620032) in inode 32918223. IGNORED.
Inode 33768943 has illegal block(s). Clear? no
Illegal block #9 (4101620032) in inode 33768943. IGNORED.
Inode 33771951 has illegal block(s). Clear? no
Illegal block #9 (4101620032) in inode 33771951. IGNORED.
Duplicate blocks found... invoking duplicate block passes.
Pass 1B: Rescan for duplicate/bad blocks
Illegal block number passed to ext2fs_test_block_bitmap #4101620032 for
multiply claimed block map
Illegal block number passed to ext2fs_test_block_bitmap #4101620032 for
multiply claimed block map
Illegal block number passed to ext2fs_test_block_bitmap #4101620032 for
multiply claimed block map
Illegal block number passed to ext2fs_test_block_bitmap #4101620032 for
multiply claimed block map
Illegal block number passed to ext2fs_test_block_bitmap #4101620032 for
multiply claimed block map
Duplicate/bad block(s) in inode 37063007: 74791522
Duplicate/bad block(s) in inode 37063010: 74791523 74791524 74791525
Duplicate/bad block(s) in inode 37389296: 74791530
Duplicate/bad block(s) in inode 37389297: 74791531
Duplicate/bad block(s) in inode 37389298: 74791532 74791533 74791537
74791538 74791543
Duplicate/bad block(s) in inode 37389300: 74791544
Duplicate/bad block(s) in inode 37389304: 74791521
Duplicate/bad block(s) in inode 37389305: 74791527
Duplicate/bad block(s) in inode 37389307: 74791520
Duplicate/bad block(s) in inode 37389325: 74791529 74791545 74791547
Duplicate/bad block(s) in inode 37389367: 74791520 74791521 74791522
74791523 74791524 74791525 74791527 74791529 74791530 74791531 74791532
74791533 74791537 74791538 74791543 74791544 74791545 74791547
Pass 1C: Scan directories for inodes with dup blocks.
Pass 1D: Reconciling duplicate blocks
(There are 11 inodes containing duplicate/bad blocks.)
File /8/004/006/176/558/pages/html-4523408551 (inode #37063007, mod time
Wed Apr 8 19:25:11 2009)
has 1 duplicate block(s), shared with 1 file(s):
/8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun
Nov 9 13:49:59 2008)
Clone duplicate/bad blocks? no
Delete file? no
File /8/004/006/921/648/pages/html-4532555619 (inode #37063010, mod time
Wed Apr 8 18:48:15 2009)
has 3 duplicate block(s), shared with 1 file(s):
/8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun
Nov 9 13:49:59 2008)
Clone duplicate/bad blocks? no
Delete file? no
File /8/004/006/990/488/pages/4533377522-Apr__6_2009__8:10:49:616AM
(inode #37389296, mod time Mon Apr 6 09:11:33 2009)
has 1 duplicate block(s), shared with 1 file(s):
/8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun
Nov 9 13:49:59 2008)
Clone duplicate/bad blocks? no
Delete file? no
File /8/004/006/990/488/pages/4533377531-Apr__6_2009__8:12:10:800AM
(inode #37389297, mod time Mon Apr 6 09:12:00 2009)
has 1 duplicate block(s), shared with 1 file(s):
/8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun
Nov 9 13:49:59 2008)
Clone duplicate/bad blocks? no
Delete file? no
File /8/004/006/990/488/pages/4533372293-Apr__6_2009__8:12:30:586AM
(inode #37389298, mod time Mon Apr 6 09:12:20 2009)
has 5 duplicate block(s), shared with 1 file(s):
/8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun
Nov 9 13:49:59 2008)
Clone duplicate/bad blocks? no
Delete file? no
File /8/004/006/990/488/pages/4533372293-Apr__6_2009__8:15:43:736AM
(inode #37389300, mod time Mon Apr 6 09:15:33 2009)
has 1 duplicate block(s), shared with 1 file(s):
/8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun
Nov 9 13:49:59 2008)
Clone duplicate/bad blocks? no
Delete file? no
File /8/004/006/931/218/pages/4532668438-Apr__6_2009__8:08:37:743PM
(inode #37389304, mod time Mon Apr 6 21:12:07 2009)
has 1 duplicate block(s), shared with 1 file(s):
/8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun
Nov 9 13:49:59 2008)
Clone duplicate/bad blocks? no
Delete file? no
File /8/004/006/875/928/pages/4533008375-Apr__7_2009_10:22:30:456AM
(inode #37389305, mod time Tue Apr 7 11:26:00 2009)
has 1 duplicate block(s), shared with 1 file(s):
/8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun
Nov 9 13:49:59 2008)
Clone duplicate/bad blocks? no
Delete file? no
File /8/004/006/875/928/images/4525650407_pre.jpg (inode #37389307, mod
time Tue Apr 7 11:29:54 2009)
has 1 duplicate block(s), shared with 1 file(s):
/8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun
Nov 9 13:49:59 2008)
Clone duplicate/bad blocks? no
Delete file? no
File /8/004/006/875/928/pages/4532395783-Apr__7_2009_12:04:08:120PM
(inode #37389325, mod time Tue Apr 7 13:07:38 2009)
has 3 duplicate block(s), shared with 1 file(s):
/8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun
Nov 9 13:49:59 2008)
Clone duplicate/bad blocks? no
Delete file? no
File /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time
Sun Nov 9 13:49:59 2008)
has 18 duplicate block(s), shared with 10 file(s):
/8/004/006/875/928/pages/4532395783-Apr__7_2009_12:04:08:120PM (inode
#37389325, mod time Tue Apr 7 13:07:38 2009)
/8/004/006/990/488/pages/4533372293-Apr__6_2009__8:15:43:736AM (inode
#37389300, mod time Mon Apr 6 09:15:33 2009)
/8/004/006/990/488/pages/4533372293-Apr__6_2009__8:12:30:586AM (inode
#37389298, mod time Mon Apr 6 09:12:20 2009)
/8/004/006/990/488/pages/4533377531-Apr__6_2009__8:12:10:800AM (inode
#37389297, mod time Mon Apr 6 09:12:00 2009)
/8/004/006/990/488/pages/4533377522-Apr__6_2009__8:10:49:616AM (inode
#37389296, mod time Mon Apr 6 09:11:33 2009)
/8/004/006/875/928/pages/4533008375-Apr__7_2009_10:22:30:456AM (inode
#37389305, mod time Tue Apr 7 11:26:00 2009)
/8/004/006/921/648/pages/html-4532555619 (inode #37063010, mod time Wed
Apr 8 18:48:15 2009)
/8/004/006/176/558/pages/html-4523408551 (inode #37063007, mod time Wed
Apr 8 19:25:11 2009)
/8/004/006/931/218/pages/4532668438-Apr__6_2009__8:08:37:743PM (inode
#37389304, mod time Mon Apr 6 21:12:07 2009)
/8/004/006/875/928/images/4525650407_pre.jpg (inode #37389307, mod time
Tue Apr 7 11:29:54 2009)
Clone duplicate/bad blocks? no
Delete file? no
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences: -4806293
Fix? no
Free blocks count wrong for group #2282 (0, counted=18).
Fix? no
Free blocks count wrong (22802537, counted=22802555).
Fix? no
Inode bitmap differences: -10549863 -10549865 -(10549871--10549873)
-(10549876--10549879) -10549883 -(10549885--10549888)
Fix? no
Free inodes count wrong for group #643 (15860, counted=15846).
Fix? no
Free inodes count wrong (47980634, counted=47980620).
Fix? no
webspace2: ********** WARNING: Filesystem still has errors **********
webspace2: 4448166/52428800 files (3.4% non-contiguous),
82055063/104857600 blocks
So, I took the partition offline and ran fsck -yf on it with the
following output:
fsck 1.35 (28-Feb-2004)
e2fsck 1.35 (28-Feb-2004)
Pass 1: Checking inodes, blocks, and sizes
Inode 2392655 has illegal block(s). Clear? yes
Illegal block #9 (4101620032) in inode 2392655. CLEARED.
Inode 2392655, i_blocks is 672, should be 664. Fix? yes
Inode 29507663 has illegal block(s). Clear? yes
Illegal block #9 (4101620032) in inode 29507663. CLEARED.
Inode 32918223 has illegal block(s). Clear? yes
Illegal block #9 (4101620032) in inode 32918223. CLEARED.
Inode 33768943 has illegal block(s). Clear? yes
Illegal block #9 (4101620032) in inode 33768943. CLEARED.
Inode 33771951 has illegal block(s). Clear? yes
Illegal block #9 (4101620032) in inode 33771951. CLEARED.
Duplicate blocks found... invoking duplicate block passes.
Pass 1B: Rescan for duplicate/bad blocks
Duplicate/bad block(s) in inode 37062984: 74791523
Duplicate/bad block(s) in inode 37063012: 74791524 74791525
Duplicate/bad block(s) in inode 37290160: 74791522
Duplicate/bad block(s) in inode 37389296: 74791530
Duplicate/bad block(s) in inode 37389297: 74791531
Duplicate/bad block(s) in inode 37389298: 74791532 74791533 74791537
74791538 74791543
Duplicate/bad block(s) in inode 37389300: 74791544
Duplicate/bad block(s) in inode 37389304: 74791521
Duplicate/bad block(s) in inode 37389305: 74791527
Duplicate/bad block(s) in inode 37389307: 74791520
Duplicate/bad block(s) in inode 37389325: 74791529 74791545 74791547
Duplicate/bad block(s) in inode 37389367: 74791520 74791521 74791522
74791523 74791524 74791525 74791527 74791529 74791530 74791531 74791532
74791533 74791537 74791538 74791543 74791544 74791545 74791547
Pass 1C: Scan directories for inodes with dup blocks.
Pass 1D: Reconciling duplicate blocks
(There are 12 inodes containing duplicate/bad blocks.)
File /8/004/006/730/448/pages/html-4530801363 (inode #37062984, mod time
Thu Apr 9 15:12:45 2009)
has 1 duplicate block(s), shared with 1 file(s):
/8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun
Nov 9 13:49:59 2008)
Clone duplicate/bad blocks? yes
File /8/004/006/176/558/pages/4523428551-Apr__9_2009_10:22:19:750AM
(inode #37063012, mod time Thu Apr 9 11:25:50 2009)
has 2 duplicate block(s), shared with 1 file(s):
/8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun
Nov 9 13:49:59 2008)
Clone duplicate/bad blocks? yes
File /8/004/006/389/058/pages/html-4527727459 (inode #37290160, mod time
Thu Apr 9 14:37:51 2009)
has 1 duplicate block(s), shared with 1 file(s):
/8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun
Nov 9 13:49:59 2008)
Clone duplicate/bad blocks? yes
File /8/004/006/990/488/pages/4533377522-Apr__6_2009__8:10:49:616AM
(inode #37389296, mod time Mon Apr 6 09:11:33 2009)
has 1 duplicate block(s), shared with 1 file(s):
/8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun
Nov 9 13:49:59 2008)
Clone duplicate/bad blocks? yes
File /8/004/006/990/488/pages/4533377531-Apr__6_2009__8:12:10:800AM
(inode #37389297, mod time Mon Apr 6 09:12:00 2009)
has 1 duplicate block(s), shared with 1 file(s):
/8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun
Nov 9 13:49:59 2008)
Clone duplicate/bad blocks? yes
File /8/004/006/990/488/pages/4533372293-Apr__6_2009__8:12:30:586AM
(inode #37389298, mod time Mon Apr 6 09:12:20 2009)
has 5 duplicate block(s), shared with 1 file(s):
/8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun
Nov 9 13:49:59 2008)
Clone duplicate/bad blocks? yes
File /8/004/006/990/488/pages/4533372293-Apr__6_2009__8:15:43:736AM
(inode #37389300, mod time Mon Apr 6 09:15:33 2009)
has 1 duplicate block(s), shared with 1 file(s):
/8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun
Nov 9 13:49:59 2008)
Clone duplicate/bad blocks? yes
File /8/004/006/931/218/pages/4532668438-Apr__6_2009__8:08:37:743PM
(inode #37389304, mod time Mon Apr 6 21:12:07 2009)
has 1 duplicate block(s), shared with 1 file(s):
/8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun
Nov 9 13:49:59 2008)
Clone duplicate/bad blocks? yes
File /8/004/006/875/928/pages/4533008375-Apr__7_2009_10:22:30:456AM
(inode #37389305, mod time Tue Apr 7 11:26:00 2009)
has 1 duplicate block(s), shared with 1 file(s):
/8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun
Nov 9 13:49:59 2008)
Clone duplicate/bad blocks? yes
File /8/004/006/875/928/images/4525650407_pre.jpg (inode #37389307, mod
time Tue Apr 7 11:29:54 2009)
has 1 duplicate block(s), shared with 1 file(s):
/8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun
Nov 9 13:49:59 2008)
Clone duplicate/bad blocks? yes
File /8/004/006/875/928/pages/4532395783-Apr__7_2009_12:04:08:120PM
(inode #37389325, mod time Tue Apr 7 13:07:38 2009)
has 3 duplicate block(s), shared with 1 file(s):
/8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun
Nov 9 13:49:59 2008)
Clone duplicate/bad blocks? yes
File /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time
Sun Nov 9 13:49:59 2008)
has 18 duplicate block(s), shared with 11 file(s):
/8/004/006/875/928/pages/4532395783-Apr__7_2009_12:04:08:120PM (inode
#37389325, mod time Tue Apr 7 13:07:38 2009)
/8/004/006/990/488/pages/4533372293-Apr__6_2009__8:15:43:736AM (inode
#37389300, mod time Mon Apr 6 09:15:33 2009)
/8/004/006/990/488/pages/4533372293-Apr__6_2009__8:12:30:586AM (inode
#37389298, mod time Mon Apr 6 09:12:20 2009)
/8/004/006/990/488/pages/4533377531-Apr__6_2009__8:12:10:800AM (inode
#37389297, mod time Mon Apr 6 09:12:00 2009)
/8/004/006/990/488/pages/4533377522-Apr__6_2009__8:10:49:616AM (inode
#37389296, mod time Mon Apr 6 09:11:33 2009)
/8/004/006/875/928/pages/4533008375-Apr__7_2009_10:22:30:456AM (inode
#37389305, mod time Tue Apr 7 11:26:00 2009)
/8/004/006/176/558/pages/4523428551-Apr__9_2009_10:22:19:750AM (inode
#37063012, mod time Thu Apr 9 11:25:50 2009)
/8/004/006/730/448/pages/html-4530801363 (inode #37062984, mod time Thu
Apr 9 15:12:45 2009)
/8/004/006/389/058/pages/html-4527727459 (inode #37290160, mod time Thu
Apr 9 14:37:51 2009)
/8/004/006/931/218/pages/4532668438-Apr__6_2009__8:08:37:743PM (inode
#37389304, mod time Mon Apr 6 21:12:07 2009)
/8/004/006/875/928/images/4525650407_pre.jpg (inode #37389307, mod time
Tue Apr 7 11:29:54 2009)
Duplicated blocks already reassigned or cloned.
Pass 2: Checking directory structure
Directory inode 29507663 has an unallocated block #9. Allocate? yes
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences: -4806293
Fix? yes
Free blocks count wrong for group #2 (16059, counted=16040).
Fix? yes
Free blocks count wrong for group #146 (7244, counted=7245).
Fix? yes
Free blocks count wrong for group #2282 (0, counted=18).
Fix? yes
Inode bitmap differences: -10549863 -10549865 -(10549871--10549873)
-(10549876--10549879) -10549883 -(10549885--10549888)
Fix? yes
webspace2: ***** FILE SYSTEM WAS MODIFIED *****
webspace2: 4451892/52428800 files (3.5% non-contiguous),
82037602/104857600 blocks
Then I saw this in /var/log/messages:
Apr 9 21:53:56 kernel: EXT3-fs error (device dm-4): ext3_readdir:
directory #29507663 contains a hole at offset 4096
Apr 9 21:53:56 kernel: Aborting journal on device dm-4.
Apr 9 21:53:56 kernel: EXT3-fs error (device dm-4): ext3_readdir:
directory #29507663 contains a hole at offset 8192
Apr 9 21:53:56 kernel: EXT3-fs error (device dm-4): ext3_readdir:
directory #29507663 contains a hole at offset 12288
Apr 9 21:53:56 kernel: ext3_abort called.
Apr 9 21:53:56 kernel: EXT3-fs error (device dm-4):
ext3_journal_start_sb: Detected aborted journal
Apr 9 21:53:56 kernel: Remounting filesystem read-only
So I took it off-line and ran fsck -yf again with the following output:
fsck 1.35 (28-Feb-2004)
e2fsck 1.35 (28-Feb-2004)
webspace2: recovering journal
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Directory inode 29507663 has an unallocated block #1. Allocate? yes
Directory inode 29507663 has an unallocated block #2. Allocate? yes
Directory inode 29507663 has an unallocated block #3. Allocate? yes
Directory inode 29507663 has an unallocated block #4. Allocate? yes
Directory inode 29507663 has an unallocated block #5. Allocate? yes
Directory inode 29507663 has an unallocated block #6. Allocate? yes
Directory inode 29507663 has an unallocated block #7. Allocate? yes
Directory inode 29507663 has an unallocated block #8. Allocate? yes
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Free blocks count wrong for group #2 (16029, counted=16021).
Fix? yes
Free blocks count wrong (22763960, counted=22763952).
Fix? yes
webspace2: ***** FILE SYSTEM WAS MODIFIED *****
webspace2: 4454345/52428800 files (3.5% non-contiguous),
82093648/104857600 blocks
What on earth is going on?! Can I expect the partition to go read-only
again?
Also, over the last week a number of other partitions from the same lvm
and EMC setup and they all came out with:
Illegal block #9 (4101620032) in inode 803439. CLEARED.
In the fsck......
More information about the Ext3-users
mailing list