[Linux-cluster] GFS filesystem "hang" with cluster-1.03.00

[Ramon van Alteren]

Some additional info,

I ran gfs_tool counters <mountpoint> on all stuck nodes
They all seem to have a large amount of outstanding BIO calls.

Any way I can find out what is causing this ?
Any other information to look for ?

As far as I can see, there's no bottleneck at the shared coraid storage AoE.

gfs_tool counters output on: mrcluster1

                                   locks 3024
                              locks held 108
                           incore inodes 12
                        metadata buffers 166
                         unlinked inodes 0
                               quota IDs 0
                      incore log buffers 1
                          log space used 0.15%
               meta header cache entries 9998
                      glock dependencies 33
                  glocks on reclaim list 0
                               log wraps 8
                    outstanding LM calls 0
                   outstanding BIO calls 805
                        fh2dentry misses 0
                        glocks reclaimed 5992
                          glock nq calls 4784596
                          glock dq calls 4784236
                    glock prefetch calls 25
                           lm_lock calls 6341
                         lm_unlock calls 5269
                            lm callbacks 11907
                      address operations 254155990
                       dentry operations 1978
                       export operations 0
                         file operations 1457916
                        inode operations 5489
                        super operations 1577443
                           vm operations 0
                         block I/O reads 136200
                        block I/O writes 81780453
gfs_tool counters output on: mrcluster2

                                   locks 3020
                              locks held 107
                           incore inodes 10
                        metadata buffers 604
                         unlinked inodes 0
                               quota IDs 0
                      incore log buffers 1
                          log space used 0.15%
               meta header cache entries 10000
                      glock dependencies 34
                  glocks on reclaim list 0
                               log wraps 8
                    outstanding LM calls 0
                   outstanding BIO calls 490
                        fh2dentry misses 0
                        glocks reclaimed 3725
                          glock nq calls 4755003
                          glock dq calls 4754287
                    glock prefetch calls 12
                           lm_lock calls 4364
                         lm_unlock calls 3017
                            lm callbacks 8140
                      address operations 252523873
                       dentry operations 1957
                       export operations 0
                         file operations 1444785
                        inode operations 5425
                        super operations 1564779
                           vm operations 0
                         block I/O reads 135658
                        block I/O writes 81574696
gfs_tool counters output on: mrcluster3

                                   locks 3018
                              locks held 135
                           incore inodes 9
                        metadata buffers 1
                         unlinked inodes 0
                               quota IDs 0
                      incore log buffers 1
                          log space used 0.15%
               meta header cache entries 9997
                      glock dependencies 20
                  glocks on reclaim list 0
                               log wraps 25
                    outstanding LM calls 0
                   outstanding BIO calls 191
                        fh2dentry misses 0
                        glocks reclaimed 11097
                          glock nq calls 15308139
                          glock dq calls 15307573
                    glock prefetch calls 13
                           lm_lock calls 8734
                         lm_unlock calls 7813
                            lm callbacks 17167
                      address operations 941469125
                       dentry operations 5308
                       export operations 0
                         file operations 4730084
                        inode operations 17157
                        super operations 5170894
                           vm operations 0
                         block I/O reads 333851
                        block I/O writes 4449228
gfs_tool counters output on: mrcluster4

                                   locks 3017
                              locks held 206
                           incore inodes 7
                        metadata buffers 2945
                         unlinked inodes 0
                               quota IDs 2
                      incore log buffers 5
                          log space used 0.24%
               meta header cache entries 9343
                      glock dependencies 54
                  glocks on reclaim list 0
                               log wraps 2
                    outstanding LM calls 0
                   outstanding BIO calls 249
                        fh2dentry misses 0
                        glocks reclaimed 2075
                          glock nq calls 1485236
                          glock dq calls 1485023
                    glock prefetch calls 0
                           lm_lock calls 1967
                         lm_unlock calls 1326
                            lm callbacks 3657
                      address operations 66603382
                       dentry operations 1747
                       export operations 0
                         file operations 457700
                        inode operations 4821
                        super operations 484938
                           vm operations 0
                         block I/O reads 28969
                        block I/O writes 21973543

Grtz Ramon