[Linux-cluster] nightly problems with gfs

[pguerra <guerra_peter bah com>]

We are getting errors nightly on one node in our 5 member cluster. The error 
is: 
GFS: Assertion failed on line 1088 of file fs/gfs/inode.c
GFS: Assertion "rgd"
GFS: time = 1118380504
fsid = webserver:documents.1

and then we get a kernel panic and it locks up. 

# cat /proc/cluster/nodes
Node Votes Exp Sts Name
1  1  2 M web2
2  1  2 M logs
3  1  2 M web1
4  1  2 M web3
5  1  2  M web4

# cat /proc/cluster/status
Version: 2.0.1
Config version: 1
Cluster name: webserver
Cluster ID: 56660
Membership state: Cluster-Member
Nodes: 5
Expected_votes: 2
Total_votes: 5
Quorum: 3
Active subsystesm: 12
Node address: 1.1.1.1

We noticed that the expected votes is 2 but we thought it should be 5, since 
there are 5 nodes in the cluster. Is that correct? When we try to reset it 
with cman_tool expected -e 5, it doesn't do it. 

Setup: 
Debian 3.0 , Kernel 2.6.7
5 Dell PowerEdge 2650s
Optical Cards in each are using the QLogic 2300 series using qla2300 kernel 
module. 
All attached to a SAN. 

This started to happen 3 weeks ago and has persisted through this morning. Any 
suggestions would be helpful, we are running out of ideas why this is 
happening and we can't upgrade our kernel at this time. 

-- 
Thanks,
Peter