[Linux-cluster] GFS fail with iozone

[Andrew Holway]

Aslo,

IOzone gave this error: Error writing block 29813, fd= 3

GFS2: fsid=nimble_cluster:gfs_test.0: jid=3: Trying to acquire journal lock...
GFS2: fsid=nimble_cluster:gfs_test.0: jid=3: Looking at journal...
GFS2: fsid=nimble_cluster:gfs_test.0: jid=3: Acquiring the transaction lock...
GFS2: fsid=nimble_cluster:gfs_test.0: jid=3: Replaying journal...


GFS seemed to repair itself and things carried on working.

thanks,

Andrew

On Sep 20, 2012, at 4:08 PM, Andrew Holway wrote:

> Hello,
> 
> I have set up a 4 node cluster. They are interconnected with an IPoIB (connected mode)
> 
> Whist running a benchmark with IOzone I got the following errors:
> 
> IO seems to have halted.
> 
> Thanks,
> 
> Andrew
> 
> Sep 20 16:01:57 node001 kernel: INFO: task iozone:15816 blocked for more than 120 seconds.
> Sep 20 16:01:57 node001 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Sep 20 16:01:57 node001 kernel: iozone        D 0000000000000011     0 15816  15374 0x00000080
> Sep 20 16:01:57 node001 kernel: ffff880fd5ebbac8 0000000000000086 ffff880fd5ebba38 ffffffff81276b66
> Sep 20 16:01:57 node001 kernel: 0000000000000096 ffff881ff95588d8 ffff880fd5ebba58 ffffffff81091f97
> Sep 20 16:01:57 node001 kernel: ffff880fe238c638 ffff880fd5ebbfd8 000000000000fb88 ffff880fe238c638
> Sep 20 16:01:57 node001 kernel: Call Trace:
> Sep 20 16:01:57 node001 kernel: [<ffffffff81276b66>] ? __prop_inc_single+0x46/0x60
> Sep 20 16:01:57 node001 kernel: [<ffffffff81091f97>] ? bit_waitqueue+0x17/0xd0
> Sep 20 16:01:57 node001 kernel: [<ffffffffa0943570>] ? gfs2_glock_holder_wait+0x0/0x20 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffffa094357e>] gfs2_glock_holder_wait+0xe/0x20 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffff814fe97f>] __wait_on_bit+0x5f/0x90
> Sep 20 16:01:57 node001 kernel: [<ffffffffa0943570>] ? gfs2_glock_holder_wait+0x0/0x20 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffff814fea28>] out_of_line_wait_on_bit+0x78/0x90
> Sep 20 16:01:57 node001 kernel: [<ffffffff81092110>] ? wake_bit_function+0x0/0x50
> Sep 20 16:01:57 node001 kernel: [<ffffffff810538b6>] ? enqueue_task+0x66/0x80
> Sep 20 16:01:57 node001 kernel: [<ffffffffa09454f5>] gfs2_glock_wait+0x45/0x90 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffffa09468f7>] gfs2_glock_nq+0x237/0x3d0 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffffa09540ec>] gfs2_permission+0xec/0x100 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffffa09540e4>] ? gfs2_permission+0xe4/0x100 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffff8118982d>] __link_path_walk+0xad/0x1030
> Sep 20 16:01:57 node001 kernel: [<ffffffff8118aa3a>] path_walk+0x6a/0xe0
> Sep 20 16:01:57 node001 kernel: [<ffffffff8118ac0b>] do_path_lookup+0x5b/0xa0
> Sep 20 16:01:57 node001 kernel: [<ffffffff8118b877>] user_path_at+0x57/0xa0
> Sep 20 16:01:57 node001 kernel: [<ffffffff811804ac>] vfs_fstatat+0x3c/0x80
> Sep 20 16:01:57 node001 kernel: [<ffffffff8118061b>] vfs_stat+0x1b/0x20
> Sep 20 16:01:57 node001 kernel: [<ffffffff81180644>] sys_newstat+0x24/0x50
> Sep 20 16:01:57 node001 kernel: [<ffffffff8150326e>] ? do_page_fault+0x3e/0xa0
> Sep 20 16:01:57 node001 kernel: [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
> Sep 20 16:01:57 node001 kernel: INFO: task iozone:15818 blocked for more than 120 seconds.
> Sep 20 16:01:57 node001 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Sep 20 16:01:57 node001 kernel: iozone        D 0000000000000011     0 15818  15374 0x00000080
> Sep 20 16:01:57 node001 kernel: ffff880fbe5b3ac8 0000000000000082 0000000000000000 ffff881ff95587a0
> Sep 20 16:01:57 node001 kernel: ffff881000000002 ffff88100ee13048 00000000be5b3a58 00000040ffffffff
> Sep 20 16:01:57 node001 kernel: ffff88100eed1ab8 ffff880fbe5b3fd8 000000000000fb88 ffff88100eed1ab8
> Sep 20 16:01:57 node001 kernel: Call Trace:
> Sep 20 16:01:57 node001 kernel: [<ffffffffa0943570>] ? gfs2_glock_holder_wait+0x0/0x20 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffffa094357e>] gfs2_glock_holder_wait+0xe/0x20 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffff814fe97f>] __wait_on_bit+0x5f/0x90
> Sep 20 16:01:57 node001 kernel: [<ffffffffa0943570>] ? gfs2_glock_holder_wait+0x0/0x20 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffff814fea28>] out_of_line_wait_on_bit+0x78/0x90
> Sep 20 16:01:57 node001 kernel: [<ffffffff81092110>] ? wake_bit_function+0x0/0x50
> Sep 20 16:01:57 node001 kernel: [<ffffffffa09454f5>] gfs2_glock_wait+0x45/0x90 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffffa09468f7>] gfs2_glock_nq+0x237/0x3d0 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffffa09540ec>] gfs2_permission+0xec/0x100 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffffa09540e4>] ? gfs2_permission+0xe4/0x100 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffff8118982d>] __link_path_walk+0xad/0x1030
> Sep 20 16:01:57 node001 kernel: [<ffffffff8118aa3a>] path_walk+0x6a/0xe0
> Sep 20 16:01:57 node001 kernel: [<ffffffff8118ac0b>] do_path_lookup+0x5b/0xa0
> Sep 20 16:01:57 node001 kernel: [<ffffffff8118b877>] user_path_at+0x57/0xa0
> Sep 20 16:01:57 node001 kernel: [<ffffffff811804ac>] vfs_fstatat+0x3c/0x80
> Sep 20 16:01:57 node001 kernel: [<ffffffff8118061b>] vfs_stat+0x1b/0x20
> Sep 20 16:01:57 node001 kernel: [<ffffffff81180644>] sys_newstat+0x24/0x50
> Sep 20 16:01:57 node001 kernel: [<ffffffff8150326e>] ? do_page_fault+0x3e/0xa0
> Sep 20 16:01:57 node001 kernel: [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
> Sep 20 16:01:57 node001 kernel: INFO: task iozone:15820 blocked for more than 120 seconds.
> Sep 20 16:01:57 node001 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Sep 20 16:01:57 node001 kernel: iozone        D 0000000000000008     0 15820  15374 0x00000080
> Sep 20 16:01:57 node001 kernel: ffff880ffed7bac8 0000000000000086 0000000000000000 ffffffff81276b66
> Sep 20 16:01:57 node001 kernel: 0000000000000002 ffff881ff95588d8 00000000fed7ba58 00000040ffffffff
> Sep 20 16:01:57 node001 kernel: ffff880fbdd51ab8 ffff880ffed7bfd8 000000000000fb88 ffff880fbdd51ab8
> Sep 20 16:01:57 node001 kernel: Call Trace:
> Sep 20 16:01:57 node001 kernel: [<ffffffff81276b66>] ? __prop_inc_single+0x46/0x60
> Sep 20 16:01:57 node001 kernel: [<ffffffffa0943570>] ? gfs2_glock_holder_wait+0x0/0x20 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffffa094357e>] gfs2_glock_holder_wait+0xe/0x20 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffff814fe97f>] __wait_on_bit+0x5f/0x90
> Sep 20 16:01:57 node001 kernel: [<ffffffffa0943570>] ? gfs2_glock_holder_wait+0x0/0x20 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffff814fea28>] out_of_line_wait_on_bit+0x78/0x90
> Sep 20 16:01:57 node001 kernel: [<ffffffff81092110>] ? wake_bit_function+0x0/0x50
> Sep 20 16:01:57 node001 kernel: [<ffffffff810538b6>] ? enqueue_task+0x66/0x80
> Sep 20 16:01:57 node001 kernel: [<ffffffffa09454f5>] gfs2_glock_wait+0x45/0x90 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffffa09468f7>] gfs2_glock_nq+0x237/0x3d0 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffffa09540ec>] gfs2_permission+0xec/0x100 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffffa09540e4>] ? gfs2_permission+0xe4/0x100 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffff8118982d>] __link_path_walk+0xad/0x1030
> Sep 20 16:01:57 node001 kernel: [<ffffffff8118aa3a>] path_walk+0x6a/0xe0
> Sep 20 16:01:57 node001 kernel: [<ffffffff8118ac0b>] do_path_lookup+0x5b/0xa0
> Sep 20 16:01:57 node001 kernel: [<ffffffff8118b877>] user_path_at+0x57/0xa0
> Sep 20 16:01:57 node001 kernel: [<ffffffff811804ac>] vfs_fstatat+0x3c/0x80
> Sep 20 16:01:57 node001 kernel: [<ffffffff8118061b>] vfs_stat+0x1b/0x20
> Sep 20 16:01:57 node001 kernel: [<ffffffff81180644>] sys_newstat+0x24/0x50
> Sep 20 16:01:57 node001 kernel: [<ffffffff8150326e>] ? do_page_fault+0x3e/0xa0
> Sep 20 16:01:57 node001 kernel: [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
> Sep 20 16:01:57 node001 kernel: INFO: task iozone:15822 blocked for more than 120 seconds.
> Sep 20 16:01:57 node001 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Sep 20 16:01:57 node001 kernel: iozone        D 0000000000000011     0 15822  15374 0x00000080
> Sep 20 16:01:57 node001 kernel: ffff880fd5dddac8 0000000000000086 0000000000000000 ffffffff81276b66
> Sep 20 16:01:57 node001 kernel: 0000000000000096 ffff881ff95588d8 ffff880fd5ddda58 ffffffff81091f97
> Sep 20 16:01:57 node001 kernel: ffff880fe238d098 ffff880fd5dddfd8 000000000000fb88 ffff880fe238d098
> Sep 20 16:01:57 node001 kernel: Call Trace:
> Sep 20 16:01:57 node001 kernel: [<ffffffff81276b66>] ? __prop_inc_single+0x46/0x60
> Sep 20 16:01:57 node001 kernel: [<ffffffff81091f97>] ? bit_waitqueue+0x17/0xd0
> Sep 20 16:01:57 node001 kernel: [<ffffffffa0943570>] ? gfs2_glock_holder_wait+0x0/0x20 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffffa094357e>] gfs2_glock_holder_wait+0xe/0x20 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffff814fe97f>] __wait_on_bit+0x5f/0x90
> Sep 20 16:01:57 node001 kernel: [<ffffffffa0943570>] ? gfs2_glock_holder_wait+0x0/0x20 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffff814fea28>] out_of_line_wait_on_bit+0x78/0x90
> Sep 20 16:01:57 node001 kernel: [<ffffffff81092110>] ? wake_bit_function+0x0/0x50
> Sep 20 16:01:57 node001 kernel: [<ffffffff810538b6>] ? enqueue_task+0x66/0x80
> Sep 20 16:01:57 node001 kernel: [<ffffffffa09454f5>] gfs2_glock_wait+0x45/0x90 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffffa09468f7>] gfs2_glock_nq+0x237/0x3d0 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffffa09540ec>] gfs2_permission+0xec/0x100 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffffa09540e4>] ? gfs2_permission+0xe4/0x100 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffff8118982d>] __link_path_walk+0xad/0x1030
> Sep 20 16:01:57 node001 kernel: [<ffffffff8118aa3a>] path_walk+0x6a/0xe0
> Sep 20 16:01:57 node001 kernel: [<ffffffff8118ac0b>] do_path_lookup+0x5b/0xa0
> Sep 20 16:01:57 node001 kernel: [<ffffffff8118b877>] user_path_at+0x57/0xa0
> Sep 20 16:01:57 node001 kernel: [<ffffffff811804ac>] vfs_fstatat+0x3c/0x80
> Sep 20 16:01:57 node001 kernel: [<ffffffff8118061b>] vfs_stat+0x1b/0x20
> Sep 20 16:01:57 node001 kernel: [<ffffffff81180644>] sys_newstat+0x24/0x50
> Sep 20 16:01:57 node001 kernel: [<ffffffff8150326e>] ? do_page_fault+0x3e/0xa0
> Sep 20 16:01:57 node001 kernel: [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
> Sep 20 16:01:57 node001 kernel: INFO: task iozone:15824 blocked for more than 120 seconds.
> Sep 20 16:01:57 node001 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Sep 20 16:01:57 node001 kernel: iozone        D 0000000000000011     0 15824  15374 0x00000080
> Sep 20 16:01:57 node001 kernel: ffff880fbe5edac8 0000000000000086 0000000000000000 ffffffff81276b66
> Sep 20 16:01:57 node001 kernel: 0000000000000002 ffff881ff95588d8 00000000be5eda58 00000040ffffffff
> Sep 20 16:01:57 node001 kernel: ffff880ff69085f8 ffff880fbe5edfd8 000000000000fb88 ffff880ff69085f8
> Sep 20 16:01:57 node001 kernel: Call Trace:
> Sep 20 16:01:57 node001 kernel: [<ffffffff81276b66>] ? __prop_inc_single+0x46/0x60
> Sep 20 16:01:57 node001 kernel: [<ffffffffa0943570>] ? gfs2_glock_holder_wait+0x0/0x20 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffffa094357e>] gfs2_glock_holder_wait+0xe/0x20 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffff814fe97f>] __wait_on_bit+0x5f/0x90
> Sep 20 16:01:57 node001 kernel: [<ffffffffa0943570>] ? gfs2_glock_holder_wait+0x0/0x20 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffff814fea28>] out_of_line_wait_on_bit+0x78/0x90
> Sep 20 16:01:57 node001 kernel: [<ffffffff81092110>] ? wake_bit_function+0x0/0x50
> Sep 20 16:01:57 node001 kernel: [<ffffffff810538b6>] ? enqueue_task+0x66/0x80
> Sep 20 16:01:57 node001 kernel: [<ffffffffa09454f5>] gfs2_glock_wait+0x45/0x90 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffffa09468f7>] gfs2_glock_nq+0x237/0x3d0 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffffa09540ec>] gfs2_permission+0xec/0x100 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffffa09540e4>] ? gfs2_permission+0xe4/0x100 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffff8118982d>] __link_path_walk+0xad/0x1030
> Sep 20 16:01:57 node001 kernel: [<ffffffff8118aa3a>] path_walk+0x6a/0xe0
> Sep 20 16:01:57 node001 kernel: [<ffffffff8118ac0b>] do_path_lookup+0x5b/0xa0
> Sep 20 16:01:57 node001 kernel: [<ffffffff8118b877>] user_path_at+0x57/0xa0
> Sep 20 16:01:57 node001 kernel: [<ffffffff811804ac>] vfs_fstatat+0x3c/0x80
> Sep 20 16:01:57 node001 kernel: [<ffffffff8118061b>] vfs_stat+0x1b/0x20
> Sep 20 16:01:57 node001 kernel: [<ffffffff81180644>] sys_newstat+0x24/0x50
> Sep 20 16:01:57 node001 kernel: [<ffffffff8150326e>] ? do_page_fault+0x3e/0xa0
> Sep 20 16:01:57 node001 kernel: [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
> Sep 20 16:01:57 node001 kernel: INFO: task iozone:15826 blocked for more than 120 seconds.
> Sep 20 16:01:57 node001 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Sep 20 16:01:57 node001 kernel: iozone        D 0000000000000011     0 15826  15374 0x00000080
> Sep 20 16:01:57 node001 kernel: ffff880fbe7cfac8 0000000000000086 0000000000000000 ffffffff81276b66
> Sep 20 16:01:57 node001 kernel: 0000000000000096 ffff881ff95588d8 ffff880fbe7cfa58 ffffffff81091f97
> Sep 20 16:01:57 node001 kernel: ffff88100ddcbab8 ffff880fbe7cffd8 000000000000fb88 ffff88100ddcbab8
> Sep 20 16:01:57 node001 kernel: Call Trace:
> Sep 20 16:01:57 node001 kernel: [<ffffffff81276b66>] ? __prop_inc_single+0x46/0x60
> Sep 20 16:01:57 node001 kernel: [<ffffffff81091f97>] ? bit_waitqueue+0x17/0xd0
> Sep 20 16:01:57 node001 kernel: [<ffffffffa0943570>] ? gfs2_glock_holder_wait+0x0/0x20 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffffa094357e>] gfs2_glock_holder_wait+0xe/0x20 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffff814fe97f>] __wait_on_bit+0x5f/0x90
> Sep 20 16:01:57 node001 kernel: [<ffffffffa0943570>] ? gfs2_glock_holder_wait+0x0/0x20 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffff814fea28>] out_of_line_wait_on_bit+0x78/0x90
> Sep 20 16:01:57 node001 kernel: [<ffffffff81092110>] ? wake_bit_function+0x0/0x50
> Sep 20 16:01:57 node001 kernel: [<ffffffff810538b6>] ? enqueue_task+0x66/0x80
> Sep 20 16:01:57 node001 kernel: [<ffffffffa09454f5>] gfs2_glock_wait+0x45/0x90 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffffa09468f7>] gfs2_glock_nq+0x237/0x3d0 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffffa09540ec>] gfs2_permission+0xec/0x100 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffffa09540e4>] ? gfs2_permission+0xe4/0x100 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffff8118982d>] __link_path_walk+0xad/0x1030
> Sep 20 16:01:57 node001 kernel: [<ffffffff8118aa3a>] path_walk+0x6a/0xe0
> Sep 20 16:01:57 node001 kernel: [<ffffffff8118ac0b>] do_path_lookup+0x5b/0xa0
> Sep 20 16:01:57 node001 kernel: [<ffffffff8118b877>] user_path_at+0x57/0xa0
> Sep 20 16:01:57 node001 kernel: [<ffffffff811804ac>] vfs_fstatat+0x3c/0x80
> Sep 20 16:01:57 node001 kernel: [<ffffffff8118061b>] vfs_stat+0x1b/0x20
> Sep 20 16:01:57 node001 kernel: [<ffffffff81180644>] sys_newstat+0x24/0x50
> Sep 20 16:01:57 node001 kernel: [<ffffffff8150326e>] ? do_page_fault+0x3e/0xa0
> Sep 20 16:01:57 node001 kernel: [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
> Sep 20 16:01:57 node001 kernel: INFO: task iozone:15828 blocked for more than 120 seconds.
> Sep 20 16:01:57 node001 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Sep 20 16:01:57 node001 kernel: iozone        D 0000000000000011     0 15828  15374 0x00000080
> Sep 20 16:01:57 node001 kernel: ffff88100684bac8 0000000000000086 0000000000000000 ffffffff81276b66
> Sep 20 16:01:57 node001 kernel: 0000000000000002 ffff881ff95588d8 000000000684ba58 00000040ffffffff
> Sep 20 16:01:57 node001 kernel: ffff88100edc7af8 ffff88100684bfd8 000000000000fb88 ffff88100edc7af8
> Sep 20 16:01:57 node001 kernel: Call Trace:
> Sep 20 16:01:57 node001 kernel: [<ffffffff81276b66>] ? __prop_inc_single+0x46/0x60
> Sep 20 16:01:57 node001 kernel: [<ffffffffa0943570>] ? gfs2_glock_holder_wait+0x0/0x20 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffffa094357e>] gfs2_glock_holder_wait+0xe/0x20 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffff814fe97f>] __wait_on_bit+0x5f/0x90
> Sep 20 16:01:57 node001 kernel: [<ffffffffa0943570>] ? gfs2_glock_holder_wait+0x0/0x20 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffff814fea28>] out_of_line_wait_on_bit+0x78/0x90
> Sep 20 16:01:57 node001 kernel: [<ffffffff81092110>] ? wake_bit_function+0x0/0x50
> Sep 20 16:01:57 node001 kernel: [<ffffffff810538b6>] ? enqueue_task+0x66/0x80
> Sep 20 16:01:57 node001 kernel: [<ffffffffa09454f5>] gfs2_glock_wait+0x45/0x90 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffffa09468f7>] gfs2_glock_nq+0x237/0x3d0 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffffa09540ec>] gfs2_permission+0xec/0x100 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffffa09540e4>] ? gfs2_permission+0xe4/0x100 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffff8118982d>] __link_path_walk+0xad/0x1030
> Sep 20 16:01:57 node001 kernel: [<ffffffff8118aa3a>] path_walk+0x6a/0xe0
> Sep 20 16:01:57 node001 kernel: [<ffffffff8118ac0b>] do_path_lookup+0x5b/0xa0
> Sep 20 16:01:57 node001 kernel: [<ffffffff8118b877>] user_path_at+0x57/0xa0
> Sep 20 16:01:57 node001 kernel: [<ffffffff811804ac>] vfs_fstatat+0x3c/0x80
> Sep 20 16:01:57 node001 kernel: [<ffffffff8118061b>] vfs_stat+0x1b/0x20
> Sep 20 16:01:57 node001 kernel: [<ffffffff81180644>] sys_newstat+0x24/0x50
> Sep 20 16:01:57 node001 kernel: [<ffffffff8150326e>] ? do_page_fault+0x3e/0xa0
> Sep 20 16:01:57 node001 kernel: [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
> Sep 20 16:01:57 node001 kernel: INFO: task iozone:15830 blocked for more than 120 seconds.
> Sep 20 16:01:57 node001 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Sep 20 16:01:57 node001 kernel: iozone        D 0000000000000000     0 15830  15374 0x00000080
> Sep 20 16:01:57 node001 kernel: ffff880fbdd0fac8 0000000000000082 0000000000000000 ffffffff81276b66
> Sep 20 16:01:57 node001 kernel: 0000000000000002 ffff881ff95588d8 00000000bdd0fa58 00000040ffffffff
> Sep 20 16:01:57 node001 kernel: ffff88100de93098 ffff880fbdd0ffd8 000000000000fb88 ffff88100de93098
> Sep 20 16:01:57 node001 kernel: Call Trace:
> Sep 20 16:01:57 node001 kernel: [<ffffffff81276b66>] ? __prop_inc_single+0x46/0x60
> Sep 20 16:01:57 node001 kernel: [<ffffffffa0943570>] ? gfs2_glock_holder_wait+0x0/0x20 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffffa094357e>] gfs2_glock_holder_wait+0xe/0x20 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffff814fe97f>] __wait_on_bit+0x5f/0x90
> Sep 20 16:01:57 node001 kernel: [<ffffffffa0943570>] ? gfs2_glock_holder_wait+0x0/0x20 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffff814fea28>] out_of_line_wait_on_bit+0x78/0x90
> Sep 20 16:01:57 node001 kernel: [<ffffffff81092110>] ? wake_bit_function+0x0/0x50
> Sep 20 16:01:57 node001 kernel: [<ffffffff810538b6>] ? enqueue_task+0x66/0x80
> Sep 20 16:01:57 node001 kernel: [<ffffffffa09454f5>] gfs2_glock_wait+0x45/0x90 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffffa09468f7>] gfs2_glock_nq+0x237/0x3d0 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffffa09540ec>] gfs2_permission+0xec/0x100 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffffa09540e4>] ? gfs2_permission+0xe4/0x100 [gfs2]
> Sep 20 16:01:57 node001 kernel: [<ffffffff8118982d>] __link_path_walk+0xad/0x1030
> Sep 20 16:01:57 node001 kernel: [<ffffffff8118aa3a>] path_walk+0x6a/0xe0
> Sep 20 16:01:57 node001 kernel: [<ffffffff8118ac0b>] do_path_lookup+0x5b/0xa0
> Sep 20 16:01:57 node001 kernel: [<ffffffff8118b877>] user_path_at+0x57/0xa0
> Sep 20 16:01:57 node001 kernel: [<ffffffff811804ac>] vfs_fstatat+0x3c/0x80
> Sep 20 16:01:57 node001 kernel: [<ffffffff8118061b>] vfs_stat+0x1b/0x20
> Sep 20 16:01:57 node001 kernel: [<ffffffff81180644>] sys_newstat+0x24/0x50
> Sep 20 16:01:57 node001 kernel: [<ffffffff8150326e>] ? do_page_fault+0x3e/0xa0
> Sep 20 16:01:57 node001 kernel: [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
> 
>