[Linux-cluster] GFS2 reads eventually cause writes to slow

[lcl at nym.hush.com]

Hello,

My team has been having a problem while testing a cluster in a lab in which write operations are extremely slow after reads have been performed continuously for some period of time.

We eventually isolated the problem to where we can replicate it on only one node.  The other two nodes are powered on, but no filesystems are mounted on those nodes and no operations are performed on those nodes.

To replicate, we first reboot everything, then start a number of threads (300+) doing random reads of 8K files in a large directory structure.  This goes well for up to 45 minutes.  (We're not expecting the reads to be that fast, given they are not cached at that point, but they are within expectations.)  We don't do any writes at this time.

Then something changes, and we can see that glock_manager is generally at 98-99% in iotop.  At this point, however, the reads are still fast enough.

Once the node has gotten into this state, an attempt to write an 8K file will usually take several seconds.  Note that it takes several seconds to write a file even if it is written on a different filesystem from the one on which we are doing the reads.

This bad condition persists until reads are stopped.  After reads are stopped, the node recovers in a few minutes, after which writes can be performed quickly.  After that, once the test is restarted, it will once again take up to 45 minutes to get the node into the bad state again.

Our hypothesis at this point is that there is some cleanup that is not getting performed as long as intensive reads are ongoing.  Because that cleanup has not been done, writes are extremely slow.  Once the reads stop, the necessary cleanup gets performed, and then it is a long time to cause the problem again.

We've tried various tuning options and are starting to dig into source code to find out more, but I thought I'd find out if anyone has any insight into this.

We're testing on CentOS 5 with kernel 2.6.18-238.12.1.el5, with gfs2-kmod-debuginfo.x86_64 1.92-1.1.el5_2.2.

Thanks,
Brian