[Linux-cluster] GFS2 hangs with dlm messages

[Sridhar Ramaswamy (srramasw)]

I'm trying to get some performance numbers out of GFS2. Before
describing the problem, I got to mention the previous dlm_sendd/recvd
spinning issue is no longer seen after moving to 2.6.21-rc4 kernel on
FC6.
 
I've a two node GFS2 setup sharing a disk off node1 using GNBD. I'm
running meta-data heavy tests (i.e create/read/delete tons of small 8k
files) from node2. The test kind of hangs in the middle. I see the
following log mesgs in node1,
 
Mar 29 15:16:23 cfs1 gnbd_serv[2723]: startup succeeded
Mar 29 15:16:37 cfs1 gnbd_clusterd[2729]: connected
Mar 29 15:17:15 cfs1 kernel: GFS2: fsid=: Trying to join cluster
"lock_dlm", "ciscogfs2:sridhar"
Mar 29 15:17:15 cfs1 kernel: dlm: connecting to 2
Mar 29 15:17:15 cfs1 kernel: dlm: got connection from 2
Mar 29 15:17:15 cfs1 kernel: GFS2: fsid=ciscogfs2:sridhar.1: Joined
cluster. Now mounting FS...
Mar 29 15:17:16 cfs1 kernel: GFS2: fsid=ciscogfs2:sridhar.1: jid=1,
already locked for use
Mar 29 15:17:16 cfs1 kernel: GFS2: fsid=ciscogfs2:sridhar.1: jid=1:
Looking at journal...
Mar 29 15:17:16 cfs1 kernel: GFS2: fsid=ciscogfs2:sridhar.1: jid=1: Done
Mar 30 09:58:34 cfs1 kernel: dlm: sridhar: remove fr 2 none
Mar 30 09:58:34 cfs1 kernel: dlm: message size 5457 from 2 too big, buf
len 4632
Mar 30 09:58:34 cfs1 kernel: dlm: sridhar: remove fr 2 none
Mar 30 09:58:35 cfs1 last message repeated 51 times
Mar 30 09:58:35 cfs1 kernel: dlm: message size 13880 from 2 too big, buf
len 85072
Mar 30 09:58:35 cfs1 kernel: dlm: sridhar: remove fr 2 none
Mar 30 09:58:35 cfs1 last message repeated 3 times
Mar 30 09:58:35 cfs1 kernel: dlm: message size 13880 from 2 too big, buf
len 93760
Mar 30 09:58:37 cfs1 kernel: dlm: message size 8224 from 2 too big, buf
len 101136
Mar 30 09:58:37 cfs1 kernel: dlm: message size 8224 from 2 too big, buf
len 101248
Mar 30 09:58:39 cfs1 kernel: dlm: message size 8224 from 2 too big, buf
len 101472

sar/iostat shows there is no major network/disk-io traffic going on
after this problem. strace'ing any of the GFS process hangs. Except I
see tons of activity in 'aisexec' process with lots of sendmsg/recvmsg
going on. It seems some cluster level component - cman or dlm error -
causes GFS2 to lock up.
 
Previous tests with random file size (range 0 to 1MB) went thru' fine.
But I also remember one of previous block test (create/read/rw a1GB
file) had similar problem.
 
Anyone seen such a problem? Any clues to resolve?
 
thanks,
Sridhar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20070330/ae0e1d6b/attachment.htm>