EXT3 errors and / filesystem readonly.

[Narayanan, Karthikeyan]

Hello,

 

I have a production RHEL ES 4U6 server encountered EXT3 filesystem
errors. Subsequently local / Root (/dev/cciss/c0d0p2) and one emcpower
device went readonly. I had to reboot the server and FSCK and then
remount both the filesystems back to normal. I have both Linux running
in local hdd and other filesystems mounted from EMC clariion.

 

I see the below errors in the /var/log/messages. 

 

Aug 26 20:45:41 zmcs kernel: Error:Mpx:Path Bus 0 Tgt 1 Lun 6 to
APM00083500788 is dead.

Aug 26 20:45:41 zmcs kernel: Error:Mpx:Path Bus 0 Tgt 1 Lun 6 to
600601607D0122005484F535FE81DE11 is marked as dead; volume ID has
changed.

Aug 26 20:45:43 zmcs kernel: Error:Mpx:Path Bus 0 Tgt 0 Lun 6 to
APM00083500788 is dead.

Aug 26 20:45:43 zmcs kernel: Error:Mpx:Path Bus 0 Tgt 0 Lun 6 to
600601607D0122005484F535FE81DE11 is marked as dead; volume ID has
changed.

Aug 26 20:46:43 zmcs kernel: Error:Mpx:All paths to
600601607D0122005484F535FE81DE11 are dead.

Aug 26 20:46:43 zmcs kernel: Error:Mpx:600601607D0122005484F535FE81DE11
is dead.

....

.......

.........

Aug 27 02:12:04 zmcs kernel: EXT3-fs error (device dm-2):
ext3_find_entry: reading directory #55511780 offset 1

Aug 27 02:12:04 zmcs kernel: Aborting journal on device dm-2.

Aug 27 02:12:04 zmcs kernel: EXT3-fs error (device dm-2):
ext3_find_entry: reading directory #55511780 offset 2

Aug 27 02:12:04 zmcs kernel: EXT3-fs error (device dm-2):
ext3_find_entry: reading directory #55511780 offset 3

Aug 27 02:13:06 zmcs kernel: EXT3-fs error (device dm-2):
ext3_get_inode_loc: unable to read inode block - inode=130155080,
block=260309012

Aug 27 02:13:42 zmcs kernel: EXT3-fs error (device dm-2): ext3_readdir:
directory #123682826 contains a hole at offset 4096

......

.........

............

Aug 27 02:16:47 zmcs kernel: ext3_abort called.

Aug 27 02:16:47 zmcs kernel: EXT3-fs error (device dm-2):
ext3_journal_start_sb: Detected aborted journal

Aug 27 02:16:47 zmcs kernel: Remounting filesystem read-only

 

 

My questions are,

1.       Why does the above behaviour happen suddenly? I believe there
is a problem with the SAN, please confirm this.

2.       Although the above messages seems to be related only to EMC
power device, why did the / (local harddisk) filesystem became readonly?

 

Thanks in advance!

 

Thanks

KK