[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

RE: [Linux-cluster] GFS performance



Paul,

 

Does the GFS lock traffic have its own interface on each server and segmented switch network?

 

What does the perl script do?

 

Regards,

 

Britt Treece

 


From: linux-cluster-bounces redhat com [mailto:linux-cluster-bounces redhat com] On Behalf Of Paul n McDowell
Sent: Wednesday, July 12, 2006 10:49 AM
To: linux-cluster redhat com
Subject: [Linux-cluster] GFS performance

 


Hi,

Can anyone provide techniques or suggestions to improve GFS performance?

The problem we have is best summarized by the output of a perl script that one of our frustrated developers has written:

The script simply creates a file, then reads it, then removes it and prints out the time it takes for each of these instructions to complete.  Sometimes it takes only a second to do all three, while sometimes it takes as long as 15 seconds to do these 3 simple instructions.  I've run the script on all of the five GFS file systems and see the same response characteristics.  I've run the script when the systems and when they are fairly quiet and still see the same issue.


To summarize the environment:

RH ES3 update 6 (GFS 6.0) 11 node cluster environment with 3 redundant lock managers and five GFS file systems mounted on each participating servers.  The GFS file systems range from 100GB to 1.5 TB.

The storage array is an EMC CX700 attached to a dual redundant SAN consisting four 2GB Brocade 3900 SAN switches.  

The HBA's in all servers are Qlogic qla2340's  Firmware version:  3.03.14, Driver version 7.07.00

The servers are all HP DL585 64bit AMD opteron server class machines each configured with between 8GB and 32GB of memory.


.....................................................................................................................................................................................................................................................

I've raised a support call with RedHat but according to their experts our configuration seems already to be set for optimum performance.

RedHat provide a utility to get and set tunable gfs file systems parameters but there is next to no supporting documentation.

So, is there anything I can do or am I missing something obvious that is just plainly mis-configured?

Shown below is the GFS configuration summary derived from lock_gulmd -C and gfs_tool df for each file system.

I'll be happy to supply any other information if it will help.

Thanks to all in advance

Paul McDowell



lock_gulmd -C

# hashed: 0x44164246
cluster {
 name = "cra_gfs"
 lock_gulm {
  heartbeat_rate = 15.000
  allowed_misses = 3
  coreport = 40040
  new_connection_timeout = 15.000
  # server cnt: 3
  # servers = ["iclc1g.cra.applera.net", "iclc2g.cra.applera.net", "ccf001g.cra.applera.net"]
  servers = ["172.20.8.21", "172.20.8.22", "172.20.8.51"]
  lt_partitions = 4
  lt_base_port = 41040
  lt_high_locks = 20971520
  lt_drop_req_rate = 300
  prealloc_locks = 5000000
  prealloc_holders = 11000000
  prealloc_lkrqs = 60
  ltpx_port = 40042
 

#gfs_tool df


/crx:
  SB lock proto = "lock_gulm"
  SB lock table = "cra_gfs:cra_crx"
  SB ondisk format = 1308
  SB multihost format = 1401
  Block size = 4096
  Journals = 11
  Resource Groups = 1988
  Mounted lock proto = "lock_gulm"
  Mounted lock table = "cra_gfs:cra_crx"
  Mounted host data = "">
  Journal number = 0
  Lock module flags = async
  Local flocks = FALSE
  Local caching = FALSE

  Type           Total          Used           Free           use%          
  ------------------------------------------------------------------------
  inodes         933593         933593         0              100%
  metadata       943899         121868         822031         13%
  data           128274180      58546879       69727301       46%


[root iclc1g tmp]# gfs_tool df /crx/data
/crx/data:
  SB lock proto = "lock_gulm"
  SB lock table = "cra_gfs:cra_crxdata"
  SB ondisk format = 1308
  SB multihost format = 1401
  Block size = 4096
  Journals = 11
  Resource Groups = 5970
  Mounted lock proto = "lock_gulm"
  Mounted lock table = "cra_gfs:cra_crxdata"
  Mounted host data = "">
  Journal number = 0
  Lock module flags = async
  Local flocks = FALSE
  Local caching = FALSE

  Type           Total          Used           Free           use%          
  ------------------------------------------------------------------------
  inodes         3296091        3296091        0              100%
  metadata       2649271        616186         2033085        23%
  data           385236382      310495360      74741022       81%

[root iclc1g tmp]# gfs_tool df /crx/home
/crx/home:
  SB lock proto = "lock_gulm"
  SB lock table = "cra_gfs:cra_crxhome"
  SB ondisk format = 1308
  SB multihost format = 1401
  Block size = 4096
  Journals = 11
  Resource Groups = 3978
  Mounted lock proto = "lock_gulm"
  Mounted lock table = "cra_gfs:cra_crxhome"
  Mounted host data = "">
  Journal number = 0
  Lock module flags = async
  Local flocks = FALSE
  Local caching = FALSE

  Type           Total          Used           Free           use%          
  ------------------------------------------------------------------------
  inodes         3477487        3477487        0              100%
  metadata       3162164        341627         2820537        11%
  data           254032093      157709829      96322264       62%

[root iclc1g tmp]# gfs_tool df /usr/local
/usr/local:
  SB lock proto = "lock_gulm"
  SB lock table = "cra_gfs:cra_usrlocal"
  SB ondisk format = 1308
  SB multihost format = 1401
  Block size = 4096
  Journals = 11
  Resource Groups = 394
  Mounted lock proto = "lock_gulm"
  Mounted lock table = "cra_gfs:cra_usrlocal"
  Mounted host data = "">
  Journal number = 0
  Lock module flags = async
  Local flocks = FALSE
  Local caching = FALSE

  Type           Total          Used           Free           use%          
  ------------------------------------------------------------------------
  inodes         765762         765762         0              100%
  metadata       582989         22854          560135         4%
  data           24393837       9477084        14916753       39%

[root iclc1g tmp]# gfs_tool df /data
/data:
  SB lock proto = "lock_gulm"
  SB lock table = "cra_gfs:cra_GQ"
  SB ondisk format = 1308
  SB multihost format = 1401
  Block size = 4096
  Journals = 11
  Resource Groups = 1298
  Mounted lock proto = "lock_gulm"
  Mounted lock table = "cra_gfs:cra_GQ"
  Mounted host data = "">
  Journal number = 0
  Lock module flags = async
  Local flocks = FALSE
  Local caching = FALSE

  Type           Total          Used           Free           use%          
  ------------------------------------------------------------------------
  inodes         10026          10026          0              100%
  metadata       282680         189037         93643          67%
  data           103761726      94277221       9484505        91%


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]