[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Filesystem gone readonly



I've got a really weird situation here. I'm using RHEL 4 and connecting
to an EMC storage device using fibre and qla2300.

The luns are put into LVM and we have a number of 400G partitions coming
off that, I made a snapshot and ran fsck -yn on it with the following
output:

fsck 1.35 (28-Feb-2004)
e2fsck 1.35 (28-Feb-2004)
Pass 1: Checking inodes, blocks, and sizes
Inode 2392655 has illegal block(s).  Clear? no

Illegal block #9 (4101620032) in inode 2392655.  IGNORED.
Inode 2392655, i_blocks is 672, should be 664.  Fix? no

Inode 29507663 has illegal block(s).  Clear? no

Illegal block #9 (4101620032) in inode 29507663.  IGNORED.
Inode 32918223 has illegal block(s).  Clear? no

Illegal block #9 (4101620032) in inode 32918223.  IGNORED.
Inode 33768943 has illegal block(s).  Clear? no

Illegal block #9 (4101620032) in inode 33768943.  IGNORED.
Inode 33771951 has illegal block(s).  Clear? no

Illegal block #9 (4101620032) in inode 33771951.  IGNORED.
Duplicate blocks found... invoking duplicate block passes.
Pass 1B: Rescan for duplicate/bad blocks
Illegal block number passed to ext2fs_test_block_bitmap #4101620032 for multiply claimed block map Illegal block number passed to ext2fs_test_block_bitmap #4101620032 for multiply claimed block map Illegal block number passed to ext2fs_test_block_bitmap #4101620032 for multiply claimed block map Illegal block number passed to ext2fs_test_block_bitmap #4101620032 for multiply claimed block map Illegal block number passed to ext2fs_test_block_bitmap #4101620032 for multiply claimed block map
Duplicate/bad block(s) in inode 37063007: 74791522
Duplicate/bad block(s) in inode 37063010: 74791523 74791524 74791525
Duplicate/bad block(s) in inode 37389296: 74791530
Duplicate/bad block(s) in inode 37389297: 74791531
Duplicate/bad block(s) in inode 37389298: 74791532 74791533 74791537 74791538 74791543
Duplicate/bad block(s) in inode 37389300: 74791544
Duplicate/bad block(s) in inode 37389304: 74791521
Duplicate/bad block(s) in inode 37389305: 74791527
Duplicate/bad block(s) in inode 37389307: 74791520
Duplicate/bad block(s) in inode 37389325: 74791529 74791545 74791547
Duplicate/bad block(s) in inode 37389367: 74791520 74791521 74791522 74791523 74791524 74791525 74791527 74791529 74791530 74791531 74791532 74791533 74791537 74791538 74791543 74791544 74791545 74791547
Pass 1C: Scan directories for inodes with dup blocks.
Pass 1D: Reconciling duplicate blocks
(There are 11 inodes containing duplicate/bad blocks.)

File /8/004/006/176/558/pages/html-4523408551 (inode #37063007, mod time Wed Apr 8 19:25:11 2009)
 has 1 duplicate block(s), shared with 1 file(s):
/8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008)
Clone duplicate/bad blocks? no

Delete file? no

File /8/004/006/921/648/pages/html-4532555619 (inode #37063010, mod time Wed Apr 8 18:48:15 2009)
 has 3 duplicate block(s), shared with 1 file(s):
/8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008)
Clone duplicate/bad blocks? no

Delete file? no

File /8/004/006/990/488/pages/4533377522-Apr__6_2009__8:10:49:616AM (inode #37389296, mod time Mon Apr 6 09:11:33 2009)
 has 1 duplicate block(s), shared with 1 file(s):
/8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008)
Clone duplicate/bad blocks? no

Delete file? no

File /8/004/006/990/488/pages/4533377531-Apr__6_2009__8:12:10:800AM (inode #37389297, mod time Mon Apr 6 09:12:00 2009)
 has 1 duplicate block(s), shared with 1 file(s):
/8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008)
Clone duplicate/bad blocks? no

Delete file? no

File /8/004/006/990/488/pages/4533372293-Apr__6_2009__8:12:30:586AM (inode #37389298, mod time Mon Apr 6 09:12:20 2009)
 has 5 duplicate block(s), shared with 1 file(s):
/8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008)
Clone duplicate/bad blocks? no

Delete file? no

File /8/004/006/990/488/pages/4533372293-Apr__6_2009__8:15:43:736AM (inode #37389300, mod time Mon Apr 6 09:15:33 2009)
 has 1 duplicate block(s), shared with 1 file(s):
/8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008)
Clone duplicate/bad blocks? no

Delete file? no

File /8/004/006/931/218/pages/4532668438-Apr__6_2009__8:08:37:743PM (inode #37389304, mod time Mon Apr 6 21:12:07 2009)
 has 1 duplicate block(s), shared with 1 file(s):
/8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008)
Clone duplicate/bad blocks? no

Delete file? no

File /8/004/006/875/928/pages/4533008375-Apr__7_2009_10:22:30:456AM (inode #37389305, mod time Tue Apr 7 11:26:00 2009)
 has 1 duplicate block(s), shared with 1 file(s):
/8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008)
Clone duplicate/bad blocks? no

Delete file? no

File /8/004/006/875/928/images/4525650407_pre.jpg (inode #37389307, mod time Tue Apr 7 11:29:54 2009)
 has 1 duplicate block(s), shared with 1 file(s):
/8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008)
Clone duplicate/bad blocks? no

Delete file? no

File /8/004/006/875/928/pages/4532395783-Apr__7_2009_12:04:08:120PM (inode #37389325, mod time Tue Apr 7 13:07:38 2009)
 has 3 duplicate block(s), shared with 1 file(s):
/8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008)
Clone duplicate/bad blocks? no

Delete file? no

File /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008)
 has 18 duplicate block(s), shared with 10 file(s):
/8/004/006/875/928/pages/4532395783-Apr__7_2009_12:04:08:120PM (inode #37389325, mod time Tue Apr 7 13:07:38 2009) /8/004/006/990/488/pages/4533372293-Apr__6_2009__8:15:43:736AM (inode #37389300, mod time Mon Apr 6 09:15:33 2009) /8/004/006/990/488/pages/4533372293-Apr__6_2009__8:12:30:586AM (inode #37389298, mod time Mon Apr 6 09:12:20 2009) /8/004/006/990/488/pages/4533377531-Apr__6_2009__8:12:10:800AM (inode #37389297, mod time Mon Apr 6 09:12:00 2009) /8/004/006/990/488/pages/4533377522-Apr__6_2009__8:10:49:616AM (inode #37389296, mod time Mon Apr 6 09:11:33 2009) /8/004/006/875/928/pages/4533008375-Apr__7_2009_10:22:30:456AM (inode #37389305, mod time Tue Apr 7 11:26:00 2009) /8/004/006/921/648/pages/html-4532555619 (inode #37063010, mod time Wed Apr 8 18:48:15 2009) /8/004/006/176/558/pages/html-4523408551 (inode #37063007, mod time Wed Apr 8 19:25:11 2009) /8/004/006/931/218/pages/4532668438-Apr__6_2009__8:08:37:743PM (inode #37389304, mod time Mon Apr 6 21:12:07 2009) /8/004/006/875/928/images/4525650407_pre.jpg (inode #37389307, mod time Tue Apr 7 11:29:54 2009)
Clone duplicate/bad blocks? no

Delete file? no

Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences:  -4806293
Fix? no

Free blocks count wrong for group #2282 (0, counted=18).
Fix? no

Free blocks count wrong (22802537, counted=22802555).
Fix? no

Inode bitmap differences: -10549863 -10549865 -(10549871--10549873) -(10549876--10549879) -10549883 -(10549885--10549888)
Fix? no

Free inodes count wrong for group #643 (15860, counted=15846).
Fix? no

Free inodes count wrong (47980634, counted=47980620).
Fix? no


webspace2: ********** WARNING: Filesystem still has errors **********

webspace2: 4448166/52428800 files (3.4% non-contiguous), 82055063/104857600 blocks



So, I took the partition offline and ran fsck -yf on it with the
following output:

fsck 1.35 (28-Feb-2004)
e2fsck 1.35 (28-Feb-2004)
Pass 1: Checking inodes, blocks, and sizes
Inode 2392655 has illegal block(s).  Clear? yes

Illegal block #9 (4101620032) in inode 2392655.  CLEARED.
Inode 2392655, i_blocks is 672, should be 664.  Fix? yes

Inode 29507663 has illegal block(s).  Clear? yes

Illegal block #9 (4101620032) in inode 29507663.  CLEARED.
Inode 32918223 has illegal block(s).  Clear? yes

Illegal block #9 (4101620032) in inode 32918223.  CLEARED.
Inode 33768943 has illegal block(s).  Clear? yes

Illegal block #9 (4101620032) in inode 33768943.  CLEARED.
Inode 33771951 has illegal block(s).  Clear? yes

Illegal block #9 (4101620032) in inode 33771951.  CLEARED.
Duplicate blocks found... invoking duplicate block passes.
Pass 1B: Rescan for duplicate/bad blocks
Duplicate/bad block(s) in inode 37062984: 74791523
Duplicate/bad block(s) in inode 37063012: 74791524 74791525
Duplicate/bad block(s) in inode 37290160: 74791522
Duplicate/bad block(s) in inode 37389296: 74791530
Duplicate/bad block(s) in inode 37389297: 74791531
Duplicate/bad block(s) in inode 37389298: 74791532 74791533 74791537 74791538 74791543
Duplicate/bad block(s) in inode 37389300: 74791544
Duplicate/bad block(s) in inode 37389304: 74791521
Duplicate/bad block(s) in inode 37389305: 74791527
Duplicate/bad block(s) in inode 37389307: 74791520
Duplicate/bad block(s) in inode 37389325: 74791529 74791545 74791547
Duplicate/bad block(s) in inode 37389367: 74791520 74791521 74791522 74791523 74791524 74791525 74791527 74791529 74791530 74791531 74791532 74791533 74791537 74791538 74791543 74791544 74791545 74791547
Pass 1C: Scan directories for inodes with dup blocks.
Pass 1D: Reconciling duplicate blocks
(There are 12 inodes containing duplicate/bad blocks.)

File /8/004/006/730/448/pages/html-4530801363 (inode #37062984, mod time Thu Apr 9 15:12:45 2009)
 has 1 duplicate block(s), shared with 1 file(s):
/8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008)
Clone duplicate/bad blocks? yes

File /8/004/006/176/558/pages/4523428551-Apr__9_2009_10:22:19:750AM (inode #37063012, mod time Thu Apr 9 11:25:50 2009)
 has 2 duplicate block(s), shared with 1 file(s):
/8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008)
Clone duplicate/bad blocks? yes

File /8/004/006/389/058/pages/html-4527727459 (inode #37290160, mod time Thu Apr 9 14:37:51 2009)
 has 1 duplicate block(s), shared with 1 file(s):
/8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008)
Clone duplicate/bad blocks? yes

File /8/004/006/990/488/pages/4533377522-Apr__6_2009__8:10:49:616AM (inode #37389296, mod time Mon Apr 6 09:11:33 2009)
 has 1 duplicate block(s), shared with 1 file(s):
/8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008)
Clone duplicate/bad blocks? yes

File /8/004/006/990/488/pages/4533377531-Apr__6_2009__8:12:10:800AM (inode #37389297, mod time Mon Apr 6 09:12:00 2009)
 has 1 duplicate block(s), shared with 1 file(s):
/8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008)
Clone duplicate/bad blocks? yes

File /8/004/006/990/488/pages/4533372293-Apr__6_2009__8:12:30:586AM (inode #37389298, mod time Mon Apr 6 09:12:20 2009)
 has 5 duplicate block(s), shared with 1 file(s):
/8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008)
Clone duplicate/bad blocks? yes

File /8/004/006/990/488/pages/4533372293-Apr__6_2009__8:15:43:736AM (inode #37389300, mod time Mon Apr 6 09:15:33 2009)
 has 1 duplicate block(s), shared with 1 file(s):
/8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008)
Clone duplicate/bad blocks? yes

File /8/004/006/931/218/pages/4532668438-Apr__6_2009__8:08:37:743PM (inode #37389304, mod time Mon Apr 6 21:12:07 2009)
 has 1 duplicate block(s), shared with 1 file(s):
/8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008)
Clone duplicate/bad blocks? yes

File /8/004/006/875/928/pages/4533008375-Apr__7_2009_10:22:30:456AM (inode #37389305, mod time Tue Apr 7 11:26:00 2009)
 has 1 duplicate block(s), shared with 1 file(s):
/8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008)
Clone duplicate/bad blocks? yes

File /8/004/006/875/928/images/4525650407_pre.jpg (inode #37389307, mod time Tue Apr 7 11:29:54 2009)
 has 1 duplicate block(s), shared with 1 file(s):
/8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008)
Clone duplicate/bad blocks? yes

File /8/004/006/875/928/pages/4532395783-Apr__7_2009_12:04:08:120PM (inode #37389325, mod time Tue Apr 7 13:07:38 2009)
 has 3 duplicate block(s), shared with 1 file(s):
/8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008)
Clone duplicate/bad blocks? yes

File /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008)
 has 18 duplicate block(s), shared with 11 file(s):
/8/004/006/875/928/pages/4532395783-Apr__7_2009_12:04:08:120PM (inode #37389325, mod time Tue Apr 7 13:07:38 2009) /8/004/006/990/488/pages/4533372293-Apr__6_2009__8:15:43:736AM (inode #37389300, mod time Mon Apr 6 09:15:33 2009) /8/004/006/990/488/pages/4533372293-Apr__6_2009__8:12:30:586AM (inode #37389298, mod time Mon Apr 6 09:12:20 2009) /8/004/006/990/488/pages/4533377531-Apr__6_2009__8:12:10:800AM (inode #37389297, mod time Mon Apr 6 09:12:00 2009) /8/004/006/990/488/pages/4533377522-Apr__6_2009__8:10:49:616AM (inode #37389296, mod time Mon Apr 6 09:11:33 2009) /8/004/006/875/928/pages/4533008375-Apr__7_2009_10:22:30:456AM (inode #37389305, mod time Tue Apr 7 11:26:00 2009) /8/004/006/176/558/pages/4523428551-Apr__9_2009_10:22:19:750AM (inode #37063012, mod time Thu Apr 9 11:25:50 2009) /8/004/006/730/448/pages/html-4530801363 (inode #37062984, mod time Thu Apr 9 15:12:45 2009) /8/004/006/389/058/pages/html-4527727459 (inode #37290160, mod time Thu Apr 9 14:37:51 2009) /8/004/006/931/218/pages/4532668438-Apr__6_2009__8:08:37:743PM (inode #37389304, mod time Mon Apr 6 21:12:07 2009) /8/004/006/875/928/images/4525650407_pre.jpg (inode #37389307, mod time Tue Apr 7 11:29:54 2009)
Duplicated blocks already reassigned or cloned.

Pass 2: Checking directory structure
Directory inode 29507663 has an unallocated block #9.  Allocate? yes

Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences:  -4806293
Fix? yes

Free blocks count wrong for group #2 (16059, counted=16040).
Fix? yes

Free blocks count wrong for group #146 (7244, counted=7245).
Fix? yes

Free blocks count wrong for group #2282 (0, counted=18).
Fix? yes

Inode bitmap differences: -10549863 -10549865 -(10549871--10549873) -(10549876--10549879) -10549883 -(10549885--10549888)
Fix? yes


webspace2: ***** FILE SYSTEM WAS MODIFIED *****
webspace2: 4451892/52428800 files (3.5% non-contiguous), 82037602/104857600 blocks


Then I saw this in /var/log/messages:

Apr 9 21:53:56 kernel: EXT3-fs error (device dm-4): ext3_readdir: directory #29507663 contains a hole at offset 4096

Apr  9 21:53:56 kernel: Aborting journal on device dm-4.

Apr 9 21:53:56 kernel: EXT3-fs error (device dm-4): ext3_readdir: directory #29507663 contains a hole at offset 8192

Apr 9 21:53:56 kernel: EXT3-fs error (device dm-4): ext3_readdir: directory #29507663 contains a hole at offset 12288

Apr  9 21:53:56 kernel: ext3_abort called.

Apr 9 21:53:56 kernel: EXT3-fs error (device dm-4): ext3_journal_start_sb: Detected aborted journal

Apr  9 21:53:56 kernel: Remounting filesystem read-only




So I took it off-line and ran fsck -yf again with the following output:

fsck 1.35 (28-Feb-2004)
e2fsck 1.35 (28-Feb-2004)
webspace2: recovering journal
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Directory inode 29507663 has an unallocated block #1.  Allocate? yes

Directory inode 29507663 has an unallocated block #2.  Allocate? yes

Directory inode 29507663 has an unallocated block #3.  Allocate? yes

Directory inode 29507663 has an unallocated block #4.  Allocate? yes

Directory inode 29507663 has an unallocated block #5.  Allocate? yes

Directory inode 29507663 has an unallocated block #6.  Allocate? yes

Directory inode 29507663 has an unallocated block #7.  Allocate? yes

Directory inode 29507663 has an unallocated block #8.  Allocate? yes

Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Free blocks count wrong for group #2 (16029, counted=16021).
Fix? yes

Free blocks count wrong (22763960, counted=22763952).
Fix? yes


webspace2: ***** FILE SYSTEM WAS MODIFIED *****
webspace2: 4454345/52428800 files (3.5% non-contiguous), 82093648/104857600 blocks




What on earth is going on?! Can I expect the partition to go read-only
again?


Also, over the last week a number of other partitions from the same lvm
and EMC setup and they all came out with:

Illegal block #9 (4101620032) in inode 803439.  CLEARED.


In the fsck......


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]