[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

[Linux-cluster] dlm: message size from 3 too big



Hi all,

I've got a three node cluster running CentOS 4.8, GFS-6.1.19-1.el4_8 (GFS 1 filesystems), kernel 2.6.9-89.0.19.ELsmp. I've seen messages like those below a couple times in the last couple weeks. Node 3 doesn't go down, so it doesn't get fenced; but DLM is unable to negotiate locks, so the load average on each node spikes and the cluster can't serve anything out through NFS. Has anyone seen anything like this? Any idea what to do about it? Shooting node 3 in the head has caused the cluster to recover, but I'd like to know how to fix it rather than work around it.

Thanks,

James

[[Operating normally prior to this point]]
May 20 04:52:50 s12n01 clurgmgrd[7467]: <err> #48: Unable to obtain cluster lock: Connection timed out May 20 04:53:41 s12n03 clurgmgrd[7476]: <err> #48: Unable to obtain cluster lock: Connection timed out May 20 04:54:09 s12n03 kernel: dlm: message size from 3 too big 34560(pkt len=386) May 20 04:54:09 s12n03 kernel: 00 00 00 00 00 00 00 87-00 00 00 23 00 00 00 00 May 20 04:54:09 s12n03 kernel: 30 b0 d2 84 02 01 00 00-30 b0 d2 84 02 01 00 00 May 20 04:54:09 s12n03 kernel: 8e 64 13 80 ff ff ff ff-f0 7d ee 81 02 01 00 00 May 20 04:54:09 s12n03 kernel: f0 7d ee 81 02 01 00 00-02 02 00 00 00 00 00 00 May 20 04:54:09 s12n03 kernel: a4 de eb 81 02 01 00 00-00 40 01 00 00 01 00 00 May 20 04:54:09 s12n03 kernel: b8 7c ee 81 02 01 00 00-ff ff ff ff ff ff ff ff May 20 04:54:09 s12n03 kernel: 82 01 00 00 00 00 00 00-90 de eb 81 02 01 00 00 May 20 04:54:09 s12n03 kernel: 00 10 00 00 00 00 00 00-00 00 00 00 00 01 00 00 May 20 04:54:09 s12n03 kernel: b7 6d db b6 6d db b6 6d-be cb 36 a0 ff ff ff ff May 20 04:54:09 s12n03 kernel: 60 fa 06 01 00 01 00 00-82 01 00 00 00 00 00 00 May 20 04:54:09 s12n03 kernel: 82 d1 0d 2a 00 01 00 00-7e 0e 00 00 00 00 00 00 May 20 04:54:09 s12n03 kernel: 00 d0 0d 2a 00 01 00 00-00 00 00 00 00 00 00 00 May 20 04:54:09 s12n03 kernel: 00 00 00 00 00 00 00 00-00 00 00 00 03 00 00 00 May 20 04:54:09 s12n03 kernel: 68 7e ee 81 02 01 00 00-02 00 00 00 00 00 00 00 May 20 04:54:09 s12n03 kernel: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 May 20 04:54:09 s12n03 kernel: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 May 20 04:54:09 s12n03 kernel: 00 00 00 00 00 00 00 00-90 de eb 81 02 01 00 00 May 20 04:54:09 s12n03 kernel: 01 00 00 00 00 00 00 00-10 50 38 a0 ff ff ff ff May 20 04:54:09 s12n03 kernel: fc ff ff ff 00 00 00 00-98 3c eb 81 02 01 00 00 May 20 04:54:09 s12n03 kernel: b0 c9 14 80 ff ff ff ff-b2 d1 36 a0 ff ff ff ff May 20 04:54:09 s12n03 kernel: 00 00 00 00 00 00 00 00-91 d0 36 a0 ff ff ff ff May 20 04:54:09 s12n03 kernel: a8 3c eb 81 02 01 00 00-87 c9 14 80 ff ff ff ff May 20 04:54:09 s12n03 kernel: ff ff ff ff ff ff ff ff-98 3c eb 81 02 01 00 00 May 20 04:54:09 s12n03 kernel: 30 3c eb 81 02 01 00 00-c0 86 f2 af 00 01 00 00
May 20 04:54:09 s12n03 kernel: 12
May 20 04:54:09 s12n03 kernel: 02
May 20 04:54:09 s12n03 kernel: dlm: midcomms: bad header version 0
May 20 04:54:09 s12n03 kernel: dlm: midcomms: cmd=0, flags=0, length=1024, lkid=1711276032, lockspace=0 May 20 04:54:09 s12n03 kernel: dlm: midcomms: base=000001002a0dd000, offset=1024, len=810, ret=1024, limit=00001000 newbuf=0 May 20 04:54:09 s12n03 kernel: 00 00 00 00 00 00 00 04-00 00 00 66 00 00 00 00 May 20 04:54:09 s12n03 kernel: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 May 20 04:54:09 s12n03 kernel: 00 00 00 00 00 00 00 00-00 00 00 00 01 00 01 00 May 20 04:54:09 s12n03 kernel: 03 00 72 00 c0 00 a4 23-17 00 00 01 6a 01 c9 26 May 20 04:54:09 s12n03 kernel: 00 00 00 00 08 00 00 00-00 00 00 00 00 00 00 00 May 20 04:54:09 s12n03 kernel: 00 00 00 00 00 00 00 00-00 00 00 00 84 34 00 00 May 20 04:54:09 s12n03 kernel: 00 00 00 00 ff 03 01 16-19 70 00 00 ff 52 00 00 May 20 04:54:09 s12n03 kernel: 00 00 00 00 00 00 00 00-00 00 00 00 00 a6 00 00 May 20 04:54:09 s12n03 kernel: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 May 20 04:54:09 s12n03 kernel: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 01 00 May 20 04:54:09 s12n03 kernel: 01 00 03 00 72 00 3d 00-7a 2a 17 00 00 01 13 00 May 20 04:54:09 s12n03 kernel: 42 2c 00 00 00 00 08 00-00 00 00 00 00 00 00 00 May 20 04:54:09 s12n03 kernel: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 84 34 May 20 04:54:09 s12n03 kernel: 00 00 00 00 00 00 ff 03-01 16 19 70 00 00 8c 0a May 20 04:54:09 s12n03 kernel: 00 00 00 00 00 00 00 00-00 00 00 37 00 00 00 53 May 20 04:54:09 s12n03 kernel: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 May 20 04:54:09 s12n03 kernel: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 May 20 04:54:09 s12n03 kernel: 01 00 01 00 03 00 72 00-49 03 b5 26 17 00 00 01 May 20 04:54:09 s12n03 kernel: 56 01 69 26 00 00 00 00-08 00 00 00 00 00 00 00 May 20 04:54:09 s12n03 kernel: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 May 20 04:54:09 s12n03 kernel: 84 34 00 00 00 00 00 00-ff 03 01 16 19 70 00 00 May 20 04:54:09 s12n03 kernel: dc fc 00 00 00 00 00 00-00 00 00 00 00 0f 00 00 May 20 04:54:09 s12n03 kernel: 00 6d 00 00 00 00 00 00-00 00 00 00 00 00 00 00 May 20 04:54:09 s12n03 kernel: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 May 20 04:54:09 s12n03 kernel: 00 00 01 00 01 00 03 00-72 00 75 00 b7 26 17 00 May 20 04:54:09 s12n03 kernel: 00 01 93 02 4f 26 00 00-00 00 08 00 00 00 00 00 May 20 04:54:09 s12n03 kernel: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 May 20 04:54:09 s12n03 kernel: 00 00 84 34 00 00 00 00-00 00 ff 03 01 16 19 70 May 20 04:54:09 s12n03 kernel: 00 00 62 a7 00 00 00 01-00 00 00 00 00 00 00 50 May 20 04:54:09 s12n03 kernel: 00 00 00 76 00 00 00 00-00 00 00 00 00 00 00 00 May 20 04:54:09 s12n03 kernel: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 May 20 04:54:09 s12n03 kernel: 00 00 00 00 01 00 01 00-03 00 72 00 8e 02 2d 26 May 20 04:54:09 s12n03 kernel: 17 00 00 01 81 03 85 27-00 00 00 00 08 00 00 00 May 20 04:54:09 s12n03 kernel: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 May 20 04:54:09 s12n03 kernel: 00 00 00 00 84 34 00 00-00 00 00 00 ff 03 01 16 May 20 04:54:09 s12n03 kernel: 19 70 00 00 6c a4 00 00-00 00 00 00 00 00 00 00 May 20 04:54:09 s12n03 kernel: 00 44 00 00 00 49 00 00-00 00 00 00 00 00 00 00 May 20 04:54:09 s12n03 kernel: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 May 20 04:54:09 s12n03 kernel: 00 00 00 00 00 00 01 00-01 00 03 00 72 00 5b 00 May 20 04:54:09 s12n03 kernel: f0 21 17 00 00 01 3a 02-fb 2b 00 00 00 00 08 00 May 20 04:54:09 s12n03 kernel: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 May 20 04:54:09 s12n03 kernel: 00 00 00 00 00 00 84 34-00 00 00 00 00 00 ff 03 May 20 04:54:09 s12n03 kernel: 01 16 19 70 00 00 ff 74-00 00 00 00 00 00 00 00 May 20 04:54:10 s12n03 kernel: 00 00 00 00 00 00 00 83-00 00 00 00 00 00 00 00 May 20 04:54:10 s12n03 kernel: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 May 20 04:54:10 s12n03 kernel: 00 00 00 00 00 00 00 00-01 00 01 00 03 00 72 00 May 20 04:54:10 s12n03 kernel: 9a 02 18 23 17 00 00 01-b9 02 f5 2d 00 00 00 00 May 20 04:54:10 s12n03 kernel: 08 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 May 20 04:54:10 s12n03 kernel: 00 00 00 00 00 00 00 00-84 34 00 00 00 00 00 00 May 20 04:54:10 s12n03 kernel: ff 03 01 16 19 70 00 00-ff 83 00 00 00 00 00 00 May 20 04:54:10 s12n03 kernel: 00 00 00 00 00 00 00 00-00 75 00 00 00 00 00 00 May 20 04:54:10 s12n03 kernel: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 May 20 04:54:10 s12n03 kernel: 00 00 00 00 00 00 00 00-00 00 01 00 01 00 03 00 May 20 04:54:10 s12n03 kernel: 72 00 1b 01 86 2a 17 00-00 01 56 02 d8 28 00 00 May 20 04:54:10 s12n03 kernel: 00 00 08 00 00 00 00 00-00 00 00 00 00 00 00 00 May 20 04:54:10 s12n03 kernel: 00 00 00 00 00 00 00 00-00 00 84 34 00 00 00 00 May 20 04:54:10 s12n03 kernel: 00 00 ff 03 01 16 19 70-00 00 fe 82 00 00 00 00 May 20 04:54:10 s12n03 kernel: 00 00 00 00 00 00 00 00-00 00 00 64 00 00 00 00 May 20 04:54:10 s12n03 kernel: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 May 20 04:54:10 s12n03 kernel: 00 00 00 00 00 00 00 00-00 00 00 00 01 00 01 00 May 20 04:54:10 s12n03 kernel: 03 00 72 00 c6 01 fc 27-17 00 00 01 e0 00 f6 28 May 20 04:54:10 s12n03 kernel: 00 00 00 00 08 00 00 00-00 00 00 00 00 00 00 00 May 20 04:54:10 s12n03 kernel: 00 00 00 00 00 00 00 00-00 00 00 00 84 34 00 00
May 20 04:54:10 s12n03 kernel: 00 00 00 00
May 20 04:54:10 s12n03 kernel: ff 03 01 16
May 20 04:54:10 s12n03 kernel: 19 70 00 00
May 20 04:54:10 s12n03 kernel: f7
May 20 04:54:10 s12n03 kernel: a3
May 20 04:54:10 s12n03 kernel: 00
May 20 04:54:10 s12n03 kernel: 00
May 20 04:54:10 s12n03 kernel: dlm: lowcomms: addr=000001002a0dd000, base=0, len=1834, iov_len=3710, iov_base[0]=000001002a0dd72a, read=1448 May 20 04:54:50 s12n01 clurgmgrd[7467]: <err> #50: Unable to obtain cluster lock: Connection timed out May 20 04:56:41 s12n03 clurgmgrd[7476]: <err> #50: Unable to obtain cluster lock: Connection timed out May 20 05:02:13 s12n02 clurgmgrd[7527]: <err> #48: Unable to obtain cluster lock: Connection timed out May 20 05:05:13 s12n02 clurgmgrd[7527]: <err> #50: Unable to obtain cluster lock: Connection timed out May 20 05:08:13 s12n02 clurgmgrd[7527]: <err> #48: Unable to obtain cluster lock: Connection timed out
[...]

When I say the load spikes, this is what I mean:

Linux 2.6.9-89.0.19.ELsmp (s12n01)      05/20/2010

12:00:01 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
[...]
04:00:01 AM         0       820      0.18      1.84      3.66
04:10:01 AM         0       820      2.72      4.03      4.04
04:20:01 AM         0       820      3.57      4.62      4.64
04:30:01 AM         0       820     11.42      7.35      5.44
04:40:01 AM         0       820      4.20      7.51      7.10
04:50:01 AM         0       820      1.69      2.18      4.33
05:00:01 AM         0       820    513.68    406.40    205.61
05:10:01 AM         0       820    530.02    513.44    360.00
05:20:01 AM         0       820    530.06    527.83    440.93
05:30:01 AM         0       820    530.12    529.75    483.33
05:40:01 AM         0       820    530.07    530.04    505.57
05:50:01 AM         0       820    530.08    530.05    517.21
06:00:01 AM         0       820    530.02    530.03    523.29
[...]


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]