[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

[linux-lvm] directory # contains a hole



I'm not certain this is LVM related but at this point I'm grasping at
straws. Every so often, ranging from 14 days to a little over a month,
the System will through the following error to messages:

kernel: EXT3-fs error (device dm-6): ext3_readdir: directory #30933792
contains a hole at offset 28672

After that message shows up we get numerous "Journal Aborted" errors
(messages clip at the bottom). The File system is marked Read-Only and
that in turn initiates a fail-over of the application to the standby
node. Then on the standby node, the filesystem is fsck'ed and is
easily cleaned up without requiring any special intervention.

Out SAN admin isn't seeing any errors on his end, and I don't see
anything that jumps out at me in out messages. Every time this has
happened the inode mentioned in the above error is different. We have
rebuilt the VG and the subsequent filesystem from scratch once. we've
turned the firmware parameters on the QLogic card to match the
recommendations for EMC for our HW config and OS version. The failure
occurs on both nodes (primary and standby) in the cluster. We've got a
ticket open with EMC and another open with RedHat.

Any help and/or ideas would be much appreciated....

Here's the system config:
Red Hat Enterprise Linux AS release 4 (Nahant Update 2)
Kernel: 2.6.9-22.ELsmp
Qlogic HBA's Connected to an EMC Clariion
Using QLogic's drivers and EMC Powerpath
About 350Gig presented to the system
LVM version:     2.01.14 (2005-08-04)
Library version: 1.01.04 (2005-08-02)
Driver version:  4.4.0
-------------------------------------------------------------------------------------------
# vgdisplay -v vg01
   Using volume group(s) on command line
   Finding volume group "vg01"
 --- Volume group ---
 VG Name               vg01
 System ID             xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 Format                lvm2
 Metadata Areas        7
 Metadata Sequence No  3
 VG Access             read/write
 VG Status             resizable
 MAX LV                256
 Cur LV                1
 Open LV               1
 Max PV                256
 Cur PV                7
 Act PV                7
 VG Size               349.56 GB
 PE Size               32.00 MB
 Total PE              11186
 Alloc PE / Size       11186 / 349.56 GB
 Free  PE / Size       0 / 0
 VG UUID               74biJE-Jn9V-i7jD-4Mng-2Pje-01YH-piE51u

 --- Logical volume ---
 LV Name                /dev/vg01/data
 VG Name                vg01
 LV UUID                000000-0000-0000-0000-0000-0000-000000
 LV Write Access        read/write
 LV Status              available
 # open                 1
 LV Size                349.56 GB
 Current LE             11186
 Segments               7
 Allocation             normal
 Read ahead sectors     1024
 Block device           253:6

 --- Physical volumes ---
 PV Name               /dev/emcpowera1
 PV UUID               DcRtrM-NtY6-ispA-uPwD-sJO1-SF0A-7iZlmm
 PV Status             allocatable
 Total PE / Free PE    1598 / 0

 PV Name               /dev/emcpowerb1
 PV UUID               VIrhHw-LqEt-KNyZ-W86T-3oyY-9F25-Zu7zMA
 PV Status             allocatable
 Total PE / Free PE    1598 / 0

 PV Name               /dev/emcpowerc1
 PV UUID               ySMNRg-m3XQ-ifF0-fWZx-ztAe-lhOX-seqJ9G
 PV Status             allocatable
 Total PE / Free PE    1598 / 0

 PV Name               /dev/emcpowerd1
 PV UUID               puTa3Z-ZRrv-mRRS-orL0-Ndpz-dW4T-E1x46s
 PV Status             allocatable
 Total PE / Free PE    1598 / 0

 PV Name               /dev/emcpowere1
 PV UUID               tsmFWq-YSVA-gHbI-ZGKG-4V57-766T-G8R6R2
 PV Status             allocatable
 Total PE / Free PE    1598 / 0

 PV Name               /dev/emcpowerf1
 PV UUID               WipA0D-GQXE-fDWL-OPNK-tRbx-7BUO-bAAJHY
 PV Status             allocatable
 Total PE / Free PE    1598 / 0

 PV Name               /dev/emcpowerg1
 PV UUID               jbcB7h-qh8O-LkAB-wcJz-CxKy-GIIm-xwylk2
 PV Status             allocatable
 Total PE / Free PE    1598 / 0
-------------------------------------------------------------------------------------------

Messages Clip Below:
-------------------------------------------------------------------------------------------
Dec  8 17:50:01 localhost logger: 17:50:01 up 17 days, 4:17, 2 users,
load average: 3.07, 4.30, 4.54
Dec  8 17:50:04 localhost kernel: EXT3-fs error (device dm-6):
ext3_readdir: directory #30933792 contains a hole at offset 28672
Dec  8 17:50:04 localhost kernel: Aborting journal on device dm-6.
Dec  8 17:50:04 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Readonly filesystem
Dec  8 17:50:04 localhost kernel: Aborting journal on device dm-6.
Dec  8 17:50:04 localhost kernel: ext3_abort called.
Dec  8 17:50:04 localhost kernel: ext3_abort called.
Dec  8 17:50:04 localhost kernel: ext3_abort called.
Dec  8 17:50:04 localhost kernel: EXT3-fs error (device dm-6):
ext3_journal_start_sb: Detected aborted journal
Dec  8 17:50:04 localhost kernel: Remounting filesystem read-only
Dec  8 17:50:04 localhost kernel: EXT3-fs error (device dm-6):
ext3_journal_start_sb: <2>EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:04 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:05 localhost kernel: EXT3-fs error (device dm-6):
ext3_journal_start_sb: Detected aborted journalDetected aborted
journal
Dec  8 17:50:05 localhost kernel:
Dec  8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:05 localhost last message repeated 2 times
Dec  8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in
ext3_new_block: Journal has aborted
Dec  8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in
ext3_prepare_write: Journal has aborted
Dec  8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:05 localhost last message repeated 5 times
Dec  8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in
ext3_reserve_inode_write: Journal has aborted
Dec  8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in
ext3_new_inode: Journal has aborted
Dec  8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in
ext3_create: Journal has aborted
Dec  8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:05 localhost last message repeated 7 times
Dec  8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in
ext3_new_block: Journal has aborted
Dec  8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in
ext3_prepare_write: Journal has aborted
Dec  8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:05 localhost last message repeated 2 times
Dec  8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in
ext3_reserve_inode_write: Journal has aborted
Dec  8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in
ext3_dirty_inode: Journal has aborted
Dec  8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:06 localhost last message repeated 11 times
Dec  8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in
ext3_new_block: Journal has aborted
Dec  8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in
ext3_prepare_write: Journal has aborted
Dec  8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:06 localhost last message repeated 7 times
Dec  8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in
ext3_reserve_inode_write: Journal has aborted
Dec  8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in
ext3_dirty_inode: Journal has aborted
Dec  8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:06 localhost last message repeated 5 times
Dec  8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:06 localhost last message repeated 6 times
Dec  8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in
add_dirent_to_buf: Journal has aborted
Dec  8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in
ext3_create: Journal has aborted
Dec  8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in
ext3_new_inode: Journal has aborted
Dec  8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in
ext3_create: Journal has aborted
Dec  8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:06 localhost last message repeated 37 times
Dec  8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:06 localhost last message repeated 44 times
Dec  8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in
ext3_new_inode: Journal has aborted
Dec  8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in
ext3_create: Journal has aborted
Dec  8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:06 localhost last message repeated 4 times
Dec  8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in
ext3_new_inode: Journal has aborted
Dec  8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in
ext3_create: Journal has aborted
Dec  8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:06 localhost last message repeated 3 times
Dec  8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:07 localhost last message repeated 8 times
Dec  8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in
ext3_ordered_commit_write: IO failure
Dec  8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:07 localhost last message repeated 3 times
Dec  8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:07 localhost last message repeated 16 times
Dec  8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in
add_dirent_to_buf: Journal has aborted
Dec  8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in
ext3_rename: Journal has aborted
Dec  8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:07 localhost last message repeated 6 times
Dec  8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in
ext3_new_block: Journal has aborted
Dec  8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in
ext3_prepare_write: Journal has aborted
Dec  8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:07 localhost last message repeated 7 times
Dec  8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:07 localhost last message repeated 3 times
Dec  8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal hasEXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:07 localhost last message repeated 3 times
Dec  8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:07 localhost last message repeated 3 times
Dec  8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:08 localhost last message repeated 2 times
Dec  8 17:50:08 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:08 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:08 localhost last message repeated 2 times
Dec  8 17:50:08 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:08 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:08 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:08 localhost last message repeated 2 times
Dec  8 17:50:08 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:08 localhost last message repeated 33 times
Dec  8 17:50:08 localhost kernel: EXT3-fs freeing b_committed_data
Dec  8 17:50:08 localhost kernel: __journal_remove_journal_head:
freeing b_committed_data
Dec  8 17:50:09 localhost last message repeated 26 times
Dec  8 17:50:09 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:09 localhost kernel: __journal_remove_journal_head:
freeing b_committed_data
Dec  8 17:50:09 localhost last message repeated 4 times
Dec  8 17:50:09 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:09 localhost kernel: __journal_remove_journal_head:
freeing b_committed_data
Dec  8 17:50:09 localhost last message repeated 34 times
Dec  8 17:50:09 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:10 localhost last message repeated 1779 times
Dec  8 17:50:11 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:25 localhost last message repeated 51 times
Dec  8 17:50:25 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:26 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:27 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:36 localhost last message repeated 11 times
Dec  8 17:50:38 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:50:56 localhost last message repeated 16 times
Dec  8 17:51:01 localhost logger: 17:51:01 up 17 days, 4:18, 2 users,
load average: 3.81, 4.62, 4.65
Dec  8 17:51:04 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:51:04 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:51:04 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:51:04 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec  8 17:51:17 localhost kernel: __journal_remove_journal_head:
freeing b_committed_data
Dec  8 17:51:17 localhost last message repeated 8 times
Dec  8 17:52:01 localhost logger: 17:52:01 up 17 days, 4:19, 0 users,
load average: 1.77, 3.92, 4.41


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]