[Linux-cluster] optimising DLM speed?

[Steven Whitehouse]

Hi,

On Tue, 2011-02-15 at 21:07 +0100, Marc Grimme wrote:
> Hi Steve,
> I think lately I observed a very similar behavior with RHEL5 and gfs2.
> It was a gfs2 filesystem that had about 2Mio files with sum of 2GB in a directory. When I did a du -shx . in this directory it took about 5 Minutes (noatime mountoption given). Independently on how much nodes took part in the cluster (in the end I only tested with one node). This was only for the first time running all later executed du commands were much faster.
> When I mounted the exact same filesystem with lockproto=lock_nolock it took about 10-20 seconds to proceed with the same command.
> 
> Next I started to analyze this with oprofile and observed the following result:
> 
> opreport --long-file-names:
> CPU: AMD64 family10, speed 2900.11 MHz (estimated)
> Counted CPU_CLK_UNHALTED events (Cycles outside of halt state) with a unit mask of 0x00 (No unit mask) count 100000
> samples  %        symbol name
> 200569   46.7639  search_rsb_list
The resource table size is by default 256 entries in size. Assuming that
you have enough ram that all 4m locks (for 2m files) are in memory at
the same time, that is approx 15625 resources per hash chain, so it
would make sense that this would start to slow things down a bit.

There is a config option to increase the resource table size though, so
perhaps you could try that?

> 118905   27.7234  create_lkb
This reads down a hash chain in the lkb table. That table is larger by
default (1024), which is probably why there is less cpu time burned
here. On the other hand, the hash chain might be read more than once if
there is a collision on the lock ids. Again it is a config option, so it
should be possible to increase the size of the table.

> 32499     7.5773  search_bucket
> 4125      0.9618  find_lkb
> 3641      0.8489  process_send_sockets
> 3420      0.7974  dlm_scan_rsbs
> 3184      0.7424  _request_lock
> 3012      0.7023  find_rsb
> 2735      0.6377  receive_from_sock
> 2610      0.6085  _receive_message
> 2543      0.5929  dlm_allocate_rsb
> 2299      0.5360  dlm_hash2nodeid
> 2228      0.5195  _create_message
> 2180      0.5083  dlm_astd
> 2163      0.5043  dlm_find_lockspace_global
> 2109      0.4917  dlm_find_lockspace_local
> 2074      0.4836  dlm_lowcomms_get_buffer
> 2060      0.4803  dlm_lock
> 1982      0.4621  put_rsb
> ..
> 
> opreport --image /gfs2
> CPU: AMD64 family10, speed 2900.11 MHz (estimated)
> Counted CPU_CLK_UNHALTED events (Cycles outside of halt state) with a unit mask of 0x00 (No unit mask) count 100000
> samples  %        symbol name
> 9310     15.5600  search_bucket
This should get better in RHEL6.1 and above, due to the new design of
glock hash table. The patch is already in upstream. The glock hash table
is much larger than the dlm hash table, though there are still
scalability issues due to the locking and that we cannot currently grow
the hash table.

> 6268     10.4758  do_promote
The result in do_promote is interesting, as I wouldn't have expected
that to show up here really, so I'll look into that when I have a moment
and try to figure out what is going on.

> 2704      4.5192  gfs2_glock_put
> 2289      3.8256  gfs2_glock_hold
> 2286      3.8206  gfs2_glock_schedule_for_reclaim
> 2204      3.6836  gfs2_glock_nq
> 2204      3.6836  run_queue
> 2001      3.3443  gfs2_holder_wake
> ..
> 
> opreport --image /dlm
> CPU: AMD64 family10, speed 2900.11 MHz (estimated)
> Counted CPU_CLK_UNHALTED events (Cycles outside of halt state) with a unit mask of 0x00 (No unit mask) count 100000
> samples  %        symbol name
> 200569   46.7639  search_rsb_list
> 118905   27.7234  create_lkb
> 32499     7.5773  search_bucket
> 4125      0.9618  find_lkb
> 3641      0.8489  process_send_sockets
> 3420      0.7974  dlm_scan_rsbs
> 3184      0.7424  _request_lock
> 3012      0.7023  find_rsb
> 2735      0.6377  receive_from_sock
> 2610      0.6085  _receive_message
> 2543      0.5929  dlm_allocate_rsb
> 2299      0.5360  dlm_hash2nodeid
> 2228      0.5195  _create_message
> ..
> 
> This very much reminded me on a similar test we've done years ago with gfs (see http://www.open-sharedroot.org/Members/marc/blog/blog-on-dlm/red-hat-dlm-__find_lock_by_id/profile-data-with-diffrent-table-sizes).
> 
> Does this not show that during the du command 46% of the time the kernel stays in the dlm:search_rsb_list function while looking out for locks. It still looks like the hashtable for the lock in dlm is much too small and searching inside the hashmap is not constant anymore?
> 
> I would be really interesting how long the described backup takes when the gfs2 filesystem is mounted exclusively on one node without locking.
> For me it looks like you're facing a similar problem with gfs2 that has been worked around with gfs by introducing the glock_purge functionality that leads to a much smaller glock->dlm->hashtable and makes backups and the like much faster.
> 
> I hope this helps.
> 
> Thanks and regards
> Marc.
> 
Many thanks for this information, it is really helpful to get feedback
like this which helps identify issues in the code,

Steve.