[Linux-cluster] GFS cluster / DLM locking - Mostly idle but high load

[Nikolas Lam]

On Tue, 2007-10-16 at 03:28 -0400, Gordan Bobic wrote:
> Hi,
> 
> I have a cluster (3 nodes at the moment, may grow up to 16) for handling a 
> lot of small files (Maildir). When I test the system by sending around 3-5 
> messages/second I see the load on the cluster nodes go up to about 20-30, 
> even though the CPUs on the cluster are about 90% idle at all times.
> 
> I am guessing that this is due to the clustered machines waiting for DLM 
> locks to be established, which causes a lot of processes to be fighting to 
> run, but since they don't get to run very soon, they back up and cause the 
> load averages to go up.
> 
> Assuming the DLM runs over the interface specified by IP and MAC in 
> cluster.conf, it is running over gigabit ethernet.
> 
> Are there any configuration changes or tuning parameters I can apply to 
> DLM to alleviate this condition? The machine I'm running the test from 
> (the one sending messages) is about 1/4 of the spec of each of the cluster 
> nodes, and it's running a load average of about 0.4. It seems crazy that a 
> single low-spec node should be able to completely overwhelm a cluster 12x 
> it's spec several times over.
> 
> Thanks.
> 
> Gordan



Hi Gordan,

I don't know alot about GFS but since no one else has replied yet, my
understanding is that it's not suitable for an applications like what
you describe (many small files being opened frequently). I think GFS2,
which is still a tech preview, has been redesigned to improve this
situation.

Regards,

Nik