[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

[Linux-cluster] 32 nodes limit?


Just what I hope will be a quick question regarding the cluster suite.

The current lock manager FAQ states ...

"CMAN in RHEL4 has known problems when you have more than 32 nodes in the cluster. We're working to resolve those issues, but until then use GULM if you have more than 32 nodes."

... while the pre-Wiki version of this document refered to DLM instead of CMAN in RHEL4. Which one is it? DLM makes more sense to me.

In any case, I gather that this issue has been resolved. If so, can you tell me the minimum version of the cluster suite and/or upstream kernel that would allow for more than 32 nodes (with DLM)? A pointer to a patch or patches that I could use would be ideal.

More details ...

I'm trying to move a 5TB filespace from NFS to GFS2. I have a P4 (the current NFS server) and 33 Opteron nodes, all running a stock 2.6.22 kernel, OpenAIS 0.80.3, and a 2.00.00 cluster suite. For now, I've dummied out fencing and set expected_votes to 1. I can start/stop cman on all nodes no problem. With all cman's running, I've formatted, mounted and populated the filesystem using the P4. Proceeding through the Opterons to mount the filesystem succeeds until the 32nd node, at which point mount.gfs2 hangs (in "D" according to `ps ax`). Going back, the first 16 systems that have mounted the filesystem can still `ls` the top level directory, but attempts to do so on the remaining systems also get stuck in "D". Any attempt to unmount the filesystem throws the entire setup in "D".

Due to various considerations, moving to more recent versions is not the preferred option at this point. Hence my question.

Any ideas?



|  Marc Aurele La France           |  work:   1-780-492-9310          |
|  Academic Information and        |  fax:    1-780-492-1729          |
|    Communications Technologies   |  email:  tsi ualberta ca         |
|  352 General Services Building   +----------------------------------+
|  University of Alberta           |                                  |
|  Edmonton, Alberta               |    Standard disclaimers apply    |
|  T6G 2H1                         |                                  |
|  CANADA                          |                                  |

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]