[Linux-cluster] GFS Locking questions?

[Matthew B. Brookover]

I have a GFS cluster set up for testing.  The source code was pulled
from CVS around 2 weeks ago, I believe the release number is 1095350377.

I set up 3 logical volumes, lv1, lv2, and lv3.  lv1 was mounted on a
host named eightoften as /var/spool/mqueue, lv2 was mounted on host a
host named nineoften as /var/spool/mqueue.  Lv3 was mounted on both
eightoften and nineoften as as /var/spool/mail.

The goal is to have one mailbox for each user on either host and have a
load balancer distribute the incoming between the 2 hosts.  Under light
loads, things work well, but under heavy loads sendmail has some locking
problems:

Oct  1 15:57:00 nineoften sendmail[29243]: NOQUEUE: SYSERR(root): cannot lockf(qfi91LpkrM029041, fd=4, type=5, omode=0, euid=0): Operation not permitted
Oct  1 15:57:00 nineoften sendmail[29243]: NOQUEUE:   4: fl=0x0, mode=100600: dev=253/3, ino=290, nlink=1, u/gid=0/55, size=1188
Oct  1 15:57:00 nineoften sendmail[29243]: NOQUEUE: SYSERR(root): cannot lockf(qfi91Lpl88029050, fd=4, type=5, omode=0, euid=0): Operation not permitted
Oct  1 15:57:00 nineoften sendmail[29243]: NOQUEUE:   4: fl=0x0, mode=100600: dev=253/3, ino=193, nlink=1, u/gid=0/55, size=1187
Oct  1 15:57:00 nineoften sendmail[29243]: NOQUEUE: SYSERR(root): cannot lockf(qfi91Lpik7029006, fd=4, type=5, omode=0, euid=0): Operation not permitted
Oct  1 15:57:00 nineoften sendmail[29243]: NOQUEUE:   4: fl=0x0, mode=100600: dev=253/3, ino=297, nlink=1, u/gid=0/55, size=1187


Other host:

Oct  1 15:56:52 eightoften sendmail[7288]: NOQUEUE: SYSERR(root): cannot lockf(qfi91LpiRJ006964, fd=4, type=5, omode=0, euid=0): Operation not permitted
Oct  1 15:56:52 eightoften sendmail[7288]: NOQUEUE:   4: fl=0x0, mode=100600: dev=253/1, ino=156, nlink=1, u/gid=0/55, size=1188
Oct  1 15:56:52 eightoften sendmail[7288]: NOQUEUE: SYSERR(root): cannot lockf(qfi91Lpkw1006986, fd=4, type=5, omode=0, euid=0): Operation not permitted
Oct  1 15:56:52 eightoften sendmail[7288]: NOQUEUE:   4: fl=0x0, mode=100600: dev=253/1, ino=203, nlink=1, u/gid=0/55, size=1203
Oct  1 15:56:52 eightoften sendmail[7288]: NOQUEUE: SYSERR(root): cannot lockf(qfi91Lpt0k007183, fd=4, type=5, omode=0, euid=0): Operation not permitted


I wrote a couple of test programs that call fcntl to get a read lock or
a write lock.  The test program work fine on ext2, or reiser file
systems, but return an operation not permitted on a GFS file system
under some cases: 

If you call fcntl with F_SETLKW, things appear to work, calling fnctl
with F_SETLK will return with a -1 and set errno to 1.

readlock and writelock were compiled to call fcntl with F_SETLK:
First, I ran writelock on nineoften:

[mbrookov at nineoften locktest]$ ./writelock /var/spool/mail/test/afile
Have write lock, hit return to free write lock on /var/spool/mail/test/afile and exit


Then ran readlock on eightoften:

[mbrookov at eightoften locktest]$ ./readlock /var/spool/mail/test/afile
Could not get read lock on /var/spool/mail/test/afile errno=1:Operation not permitted
[mbrookov at eightoften locktest]$


Similar problems happen if you run readlock and writelock on the same
host.

Out of 500 mail messages sent, only 216 arrived in the mail box.  I have
a load balancer set up to distribute the mail messages between the 2
hosts.

Any ideas?

thanks

Matt

Academic Computing and Networking
Colorado School of Mines
mbrookov at mines.edu
303-273-3436


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20041001/26203c40/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: readlock.c
Type: text/x-c
Size: 1261 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20041001/26203c40/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: writelock.c
Type: text/x-c
Size: 1266 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20041001/26203c40/attachment-0001.bin>