[Linux-cluster] Interfacing csnap to cluster stack

Daniel McNeil daniel at osdl.org
Fri Oct 8 22:30:28 UTC 2004


On Fri, 2004-10-08 at 14:25, Daniel Phillips wrote:
> On Friday 08 October 2004 15:16, Lon Hohberger wrote:
> > On Thu, 2004-10-07 at 13:58 -0400, Daniel Phillips wrote:
> > > Suppose that the winner of the race to get the exclusive lock is a
> > > bad choice to run the server.  Perhaps it has a fast connection to
> > > the net but is connected to the disk over the network instead of
> > > directly like the other nodes.   How do you fix that, within this
> > > model?
> >
> > Let me see if I am getting this use-case picture right...
> > Are either of those close?
> 
> No, the arranagement I was describing is:
> 
>     SAN                       GigE
>       | <---> iSCSI/GNBD <---> |
>       |                        |
>       | <---> Client <-------> |  Node 1
>       |                        |
>       | <---> Client <-------> |  Node 2
>                                |
>               Client <-------> |  Node 3
>               Server <-------> |  Node 3
> 
> Node 3 won the race to get the EX lock because the lock is mediated over 
> the GigE network.  But Node 3 is a bad choice because it is two hops 
> away from the disk.  The DLM chose Node 3 because the DLM doesn't know 
> anything about network topology, just who got there first to grab the 
> lock.
> 
> > (1) Don't set up your csnap server in such a way that some the nodes
> > exhibit a bottleneck on disk I/O and some do not.
> 
> But what prevents it?  How do you "set up your csnap server"?  Why would 
> you want to introduce new rules about cluster topology instead of 
> fixing the code?

Fixing which code?

The csnap server is the best one to know what is best.
See my previous posting on using 2 dlm locks to allow
different priorities.  A directly connected node will
be selected if there is one, otherwise one of the "other"
nodes will be selected.  The csnap server can just do
the right thing.

> 
> > (2) Have the administrator make an intelligent decision as to whether
> > or not to relocate the csnap master server again as [s]he tries to
> > fix the problem that caused the failover.  I.E. Don't worry about it
> > if the csnap master server is running slowly.
> 
> The administrator is normally asleep or busy with girlfriend when 
> anything goes wrong.
> 
> > Your clients still work, and the csnap server is available, albeit at
> > a potentially degraded state.
> 
> Well...
> 
> > (3) Don't use the cluster-lock model.  It has its shortcomings.  Its
> > strengths are in its simplicity; not its flexibility.
> 

Actually, the DLM can be used in simple ways or very complex
ways.  It is very flexible.  It does have a different programming
model that takes time to get use to.

> Yes, that's the one.  We need real resource management, even if it 
> initially just consists of an administrator setting up config files.  
> Something has to read those config files[1] and respond to server 
> instance requests from csnap agents accordingly.
> 
> [1] At cluster bring-up time.  The resource manager has to be able to 
> operate without reading files during failover.
> 

IMHO, a resource manager is NOT the right way to do this:

- cluster services should avoid config files if at all possible.
   If they are are not set up right, the whole cluster can get
   messed up.  If the cluster changes, the config files might
   need to change.  The config files will be your single point
   of failure.  From previous experience, cluster configuration
   is one of the biggest sources of cluster failure, and you
   won't know it until a failure -- the worst possible time.

- It makes a low-level function dependent on a higher-level
   function.  As you say above, the resource manager has to
   operate very carefully to avoid dead locking.  This is
   asking for trouble.

Daniel 





More information about the Linux-cluster mailing list