[Linux-cluster] Issue with GFS2 medata block on two nodes cluster.

[Theophanis Kontogiannis]

Hello all,

I have a two node cluster with kernel-2.6.18-53.1.21.el5 and drbd8.2.

The LV is formatted with GFS2.

However suddenly one of the nodes keeps on reporting:


Jun 23 12:56:05 tweety1 kernel: GFS2: fsid=tweety:gfs2-00.0: fatal: invalid
metadata block
Jun 23 12:56:05 tweety1 kernel: GFS2: fsid=tweety:gfs2-00.0:   bh = 21879736
(magic number)
Jun 23 12:56:05 tweety1 kernel: GFS2: fsid=tweety:gfs2-00.0:   function =
gfs2_meta_indirect_buffer, file = fs/gfs2/meta_io.c, line = 438
Jun 23 12:56:05 tweety1 kernel: GFS2: fsid=tweety:gfs2-00.0: about to
withdraw this file system
Jun 23 12:56:05 tweety1 kernel: GFS2: fsid=tweety:gfs2-00.0: telling LM to
withdraw


Jun 23 12:56:06 tweety1 kernel: GFS2: fsid=tweety:gfs2-00.0: withdrawn
Jun 23 12:56:06 tweety1 kernel:
Jun 23 12:56:06 tweety1 kernel: Call Trace:
Jun 23 12:56:06 tweety1 kernel:  [<ffffffff8865715e>]
:gfs2:gfs2_lm_withdraw+0xc1/0xd0
Jun 23 12:56:06 tweety1 kernel:  [<ffffffff80014cca>] sync_buffer+0x0/0x3f
Jun 23 12:56:06 tweety1 kernel:  [<ffffffff80062a3f>]
out_of_line_wait_on_bit+0x6c/0x78
Jun 23 12:56:06 tweety1 kernel:  [<ffffffff8009ba34>]
wake_bit_function+0x0/0x23
Jun 23 12:56:06 tweety1 kernel:  [<ffffffff8866895b>]
:gfs2:gfs2_meta_check_ii+0x2c/0x38
Jun 23 12:56:06 tweety1 kernel:  [<ffffffff8865adbb>]
:gfs2:gfs2_meta_indirect_buffer+0x1e3/0x284
Jun 23 12:56:06 tweety1 kernel:  [<ffffffff88655a88>]
:gfs2:gfs2_inode_refresh+0x22/0x2b9
Jun 23 12:56:06 tweety1 kernel:  [<ffffffff88654eff>]
:gfs2:inode_go_lock+0x29/0x57


After this, the node halts.
The other node works perfectly.

If I boot single user the problematic node, what steps should I take to
recover??

Thank you all for your time.

Theophanis Kontogiannis