[Cluster-devel] cluster/cman/qdisk README
lhh at sourceware.org
lhh at sourceware.org
Fri Jul 21 17:55:20 UTC 2006
CVSROOT: /cvs/cluster
Module name: cluster
Changes by: lhh at sourceware.org 2006-07-21 17:55:19
Modified files:
cman/qdisk : README
Log message:
Add man pages for qdisk
Patches:
http://sourceware.org/cgi-bin/cvsweb.cgi/cluster/cman/qdisk/README.diff?cvsroot=cluster&r1=1.3&r2=1.4
--- cluster/cman/qdisk/README 2006/06/23 16:05:33 1.3
+++ cluster/cman/qdisk/README 2006/07/21 17:55:19 1.4
@@ -1,274 +1 @@
-qdisk 1.0 - a disk-based quorum algorithm for Linux-Cluster
-
-(C) 2006 Red Hat, Inc.
-
-1. Overview
-
-1.1. Problem
-
-In some situations, it may be necessary or desirable to sustain
-a majority node failure of a cluster without introducing the need for
-asymmetric (client-server, or heavy-weighted voting nodes).
-
-1.2. Design Requirements
-
-* Ability to sustain 1..(n-1)/n simultaneous node failures, without the
-danger of a simple network partition causing a split brain. That is, we
-need to be able to ensure that the majority failure case is not merely
-the result of a network partition.
-
-* Ability to use external reasons for deciding which partition is the
-the quorate partition in a partitioned cluster. For example, a user may
-have a service running on one node, and that node must always be the master
-in the event of a network partition. Or, a node might lose all network
-connectivity except the cluster communication path - in which case, a
-user may wish that node to be evicted from the cluster.
-
-* Integration with CMAN. We must not require CMAN to run with us (or
-without us). Linux-Cluster does not require a quorum disk normally -
-introducing new requirements on the base of how Linux-Cluster operates
-is not allowed.
-
-* Data integrity. In order to recover from a majority failure, fencing
-is required. The fencing subsystem is already provided by Linux-Cluster.
-
-* Non-reliance on hardware or protocol specific methods (i.e. SCSI
-reservations). This ensures the quorum disk algorithm can be used on the
-widest range of hardware configurations possible.
-
-* Little or no memory allocation after initialization. In critical paths
-during failover, we do not want to have to worry about being killed during
-a memory pressure situation because we request a page fault, and the Linux
-OOM killer responds...
-
-
-1.3. Hardware Configuration Considerations
-
-1.3.1. Concurrent, Synchronous, Read/Write Access
-
-This daemon requires a shared block device with concurrent read/write
-access from all nodes in the cluster. The shared block device can be
-a multi-port SCSI RAID array, a Fiber-Channel RAID SAN, a RAIDed iSCSI
-target, or even GNBD. The quorum daemon uses O_DIRECT to write to the
-device.
-
-1.3.2. Bargain-basement JBODs need not apply
-
-There is a minimum performance requirement inherent when using disk-based
-cluster quorum algorithms, so design your cluster accordingly. Using a
-cheap JBOD with old SCSI2 disks on a multi-initiator bus will cause
-problems at the first load spike. Plan your loads accordingly; a node's
-inability to write to the quorum disk in a timely manner will cause the
-cluster to evict the node. Using host-RAID or multi-initiator parallel
-SCSI configurations with the qdisk daemon is unlikely to work, and will
-probably cause administrators a lot of frustration. That having been
-said, because the timeouts are configurable, most hardware should work
-if the timeouts are set high enough.
-
-1.3.3. Fencing is Required
-
-In order to maintain data integrity under all failure scenarios, use of
-this quorum daemon requires adequate fencing, preferrably power-based
-fencing.
-
-
-1.4. Limitations
-
-* At this time, this daemon only supports a maximum of 16 nodes.
-
-* Cluster node IDs must be statically configured in cluster.conf and
-must be numbered from 1..16 (there can be gaps, of course).
-
-* Cluster node votes should be more or less equal.
-
-* CMAN must be running before the qdisk program can start. This
-limitation will be removed before a production release.
-
-* CMAN's eviction timeout should be at least 2x the quorum daemon's
-to give the quorum daemon adequate time to converge on a master during a
-failure + load spike situation.
-
-* The total number of votes assigned to the quorum device should be
-equal to or greater than the total number of node-votes in the cluster.
-While it is possible to assign only one (or a few) votes to the quorum
-device, the effects of doing so have not been explored.
-
-* Currently, the quorum disk daemon is difficult to use with CLVM if
-the quorum disk resides on a CLVM logical volume. CLVM requires a
-quorate cluster to correctly operate, which introduces a chicken-and-egg
-problem for starting the cluster: CLVM needs quorum, but the quorum daemon
-needs CLVM (if and only if the quorum device lies on CLVM-managed storage).
-One way to work around this is to *not* set the cluster's expected votes
-to include theh quorum daemon's votes. Bring all nodes online, and start
-the quorum daemon *after* the whole cluster is running. This will allow
-the expected votes to increase naturally.
-
-
-2. Algorithms
-
-2.1. Heartbeating & Liveliness Determination
-
-Nodes update individual status blocks on the quorum disk at a user-
-defined rate. Each write of a status block alters the timestamp, which
-is what other nodes use to decide whether a node has hung or not. If,
-after a user-defined number of 'misses' (that is, failure to update a
-timestamp), a node is declared offline. After a certain number of 'hits'
-(changed timestamp + "i am alive" state), the node is declared online.
-
-The status block contains additional information, such as a bitmask of
-the nodes that node believes are online. Some of this information is
-used by the master - while some is just for performace recording, and
-may be used at a later time. The most important pieces of information
-a node writes to its status block are:
-
- - timestamp
- - internal state (available / not available)
- - score
- - max score
- - vote/bid messages
- - other nodes it thinks are online
-
-
-2.2. Scoring & Heuristics
-
-The administrator can configure up to 10 purely arbitrary heuristics, and
-must exercise caution in doing so. By default, only nodes scoring over
-1/2 of the total maximum score will claim they are available via the
-quorum disk, and a node (master or otherwise) whose score drops too low
-will remove itself (usually, by rebooting).
-
-The heuristics themselves can be any command executable by 'sh -c'. For
-example, in early testing, I used this:
-
- <heuristic program="[ -f /quorum ]" score="10" interval="2"/>
-
-This is a literal sh-ism which tests for the existence of a file called
-"/quorum". Without that file, the node would claim it was unavailable.
-This is an awful example, and should never, ever be used in production,
-but is provided as an example as to what one could do...
-
-Typically, the heuristics should be snippets of shell code or commands which
-help determine a node's usefulness to the cluster or clients. Ideally, you
-want to add traces for all of your network paths (e.g. check links, or
-ping routers), and methods to detect availability of shared storage.
-
-
-2.3. Master Election
-
-Only one master is present at any one time in the cluster, regardless of
-how many partitions exist within the cluster itself. The master is
-elected by a simple voting scheme in which the lowest node which believes
-it is capable of running (i.e. scores high enough) bids for master status.
-If the other nodes agree, it becomes the master. This algorithm is
-run whenever no master is present.
-
-If another node comes online with a lower node ID while a node is still
-bidding for master status, it will rescind its bid and vote for the lower
-node ID. If a master dies or a bidding node dies, the voting algorithm
-is started over. The voting algorithm typically takes two passes to
-complete.
-
-Master deaths take marginally longer to recover from than non-master
-deaths, because a new master must be elected before the old master can
-be evicted & fenced.
-
-
-2.4. Master Duties
-
-The master node decides who is or is not in the master partition, as
-well as handles eviction of dead nodes (both via the quorum disk and via
-the linux-cluster fencing system by using the cman_kill_node() API).
-
-
-2.5. How it All Ties Together
-
-When a master is present, and if the master believes a node to be online,
-that node will advertise to CMAN that the quorum disk is avilable. The
-master will only grant a node membership if:
-
- (a) CMAN believes the node to be online, and
- (b) that node has made enough consecutive, timely writes to the quorum
- disk.
-
-
-3. Configuration
-
-3.1. The <quorumd> tag
-
-This tag is a child of the top-level <cluster> tag.
-
- <quorumd
- interval="1" This is the frequency of read/write cycles
- tko="10" This is the number of cycles a node must miss
- in order to be declared dead.
- votes="3" This is the number of votes the quorum daemon
- advertises to CMAN when it has a high enough
- score.
- log_level="4" This controls the verbosity of the quorum daemon
- in the system logs. 0 = emergencies; 7 = debug
- log_facility="local4" This controls the syslog facility used by the
- quorum daemon when logging.
- status_file="/foo" Write internal states out to this file
- periodically ("-" = use stdout).
- min_score="3" Absolute minimum score to be consider one's
- self "alive". If omitted, or set to 0, the
- default function "floor((n+1)/2)" is used.
- device="/dev/sda1" This is the device the quorum daemon will use.
- This device must be the same on all nodes.
- label="mylabel"/> This overrides the device field if present.
- If specified, the quorum daemon will read
- /proc/partitions and check for qdisk signatures
- on every block device found, comparing the label
- against the specified label. This is useful in
- configurations where the block device name
- differs on a per-node basis.
-
-
-3.2. The <heuristic> tag
-
-This tag is a child of the <quorumd> tag.
-
- <heuristic
- program="/test.sh" This is the program used to determine if this
- heuristic is alive. This can be anything which
- may be executed by "/bin/sh -c". A return value
- of zero indicates success.
- score="1" This is the weight of this heuristic. Be careful
- when determining scores for heuristics.
- interval="2"/> This is the frequency at which we poll the
- heuristic.
-
-3.3. Example
-
- <quorumd interval="1" tko="10" votes="3" device="/dev/gnbd/qdisk">
- <heuristic program="ping routerA -c1 -t1" score="1" interval="2"/>
- <heuristic program="ping routerB -c1 -t1" score="1" interval="2"/>
- <heuristic program="ping routerC -c1 -t1" score="1" interval="2"/>
- </quorumd>
-
-3.4. Heuristic score considerations
-
-* Heuristic timeouts should be set high enough to allow the previous run
-of a given heuristic to complete.
-
-* Heuristic scripts returning anything except 0 as their return code
-are considered failed.
-
-* The worst-case for improperly configured quorum heuristics is a race
-to fence where two partitions simultaneously try to kill each other.
-
-3.5. Creating a quorum disk partition
-
-3.5.1. The mkqdisk utility.
-
-The mkqdisk utility can create and list currently configured quorum disks
-visible to the local node.
-
- mkqdisk -L List available quorum disks.
-
- mkqdisk -f <label> Find a quorum device by the given label.
-
- mkqdisk -c <device> -l <label>
- Initialize <device> and name it <label>. This
- will destroy all data on the device, so be careful
- when running this command.
+See qdisk(5) for setup and other information
More information about the Cluster-devel
mailing list