[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [Linux-cluster] Locking in GFS



Chris Joelly wrote:
Hello,

i have a question on locking issues on GFS:

how do GFS lock files on the filesystem. I have found one posting to
this list which states that locking occurs "more or less" on file level. Is this true? or does some kind of locking occur on directory
level too?

You may view GFS(1) internal lock granularity is on system call level - that is, when either a write or read (say pwrite() or pread()) is issued, the associated file is locked until the system call returns. There are few simple things that will be helpful if you keep them in mind:

1. a write requires an exclusive lock (i.e., when there is a write going on, every access to that file has to wait). 2. a read needs a shared lock (i.e. many reads to the same file will not be stalled). 3. a write may involve directory lock (e.g. a "create" would need a write lock of the parent directory). 4. local locking (two writes compete the same lock on the same node) is always much better than inter-node (different nodes) locking (ping-pong the same write lock between different nodes is very expensive). 5. standard APIs (such as fcntl() and flock()) precedes GFS(1) internal locking if used correctly (e.g. upon obtaining an exclusive flock, other access to that file will be stalled, assuming every instance of the executables running on different nodes has the correct flock implemented and honored).

One exception to (5) is posix byte-range locks. Say there are two processes running on different nodes, each obtaining its own byte range locks. Process A locks byte offset 0 to 10K; process B locks byte 10K+1 to 40K. When both have writes issued, one of them has to wait until other's write system call completes before it can continue - a result of its posix locking implementation that is done by a separate module outside its internal filesystem locking.

The question arrose because i was thinking on whats would happen when
using GFS on an file server with home directories for lets say 1000
users. how do i setup this directory structure to avoid locking issues
as best as possible.

is a directory layout like the following ok when /home is a GFS file system:

/home/a/b/c/abc
/home/d/e/f/def
/home/g/h/i/ghi
...
/home/x/y/z/xyz


Hope above statements have helped you understanding that on GFS(1),

1. A short-and-fat directory structure will work (much) better than tall-and-skinny ones. 2. If possible, the directory setup should avoid ping-pong directory and/or write locks between different nodes

For GFS2, after browsing thru its newest source code few minutes ago - it reminds me of a "sea horse" shape with linux page locks on its curly "belly" :). It is difficult (for me) to describe so I'll skip.

-- Wendy


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]