[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [Linux-cluster] cluster is not quorate. refusing connection



starting ccsd with the -v flag should print to the log the plugin it is using to connect to the cluster infrastructure. Something on the order of:
"Connected to cluster infrastructure via: CMAN/SM Plugin v1.1"


This will at least tell you that ccsd is able to connect to the cluster manager - and therefore, should know whether the cluster is quorate or not.

brassow

On Dec 10, 2004, at 4:16 PM, Matthew B. Brookover wrote:

On Fri, 2004-12-10 at 00:17, David Teigland wrote:
This sounds similar to a problem I have if I run fence_tool without ccsd
running.


Check /proc/cluster/status while it's waiting to see if the cluster
actually has quorum or not.  Also, I've added some extra checking and
debugging to fence_tool that should help narrow down where things are
stuck.  Please update from cvs and rebuild at least the stuff in
cluster/fence; then use "fence_tool join -D".

Usually things get stuck talking to ccs when ccs/magma libraries are out
of sync, but this case sounds different.

Ok, I pulled the updates from CVS and rebuilt the code and the kernel.


On node fiveoften, fence_tool printed out some errors and exited with
status 1.  The errors are below.

On node fouroften, fence_tool did not print any messages and it did
not exit.  I am guessing that fence_tool did not exit because of a
feature of the -D flag.  Fence_tool did startup fenced on fouroften.

Ccsd started up and is running on both nodes.

According to /proc/cluster/status and nodes on both fouroften and
fiveoften, the cluster is up and has quorum.

fence_tool printed these messages on node fiveoften:
+ fence_tool join -D
fence_tool: cannot connect to ccs -111

fence_tool: wait for quorum
fence_tool: waiting for cluster quorum
fence_tool: waiting for cluster quorum
fence_tool: waiting for cluster quorum
fence_tool: get our node name
fence_tool: connect to ccs
fence_tool: waiting for ccs connection -111
fence_tool: waiting for ccs connection -111
fence_tool: waiting for ccs connection -111
fence_tool: waiting for ccs connection -111
fence_tool: waiting for ccs connection -111
fence_tool: waiting for ccs connection -111
fence_tool: waiting for ccs connection -111
fence_tool: waiting for ccs connection -111
fence_tool: waiting for ccs connection -111
fence_tool: waiting for ccs connection -111

Log entries on node fiveoften:
Dec 10 14:08:39 fiveoften kernel: Lock_Harness <CVS> (built Dec 10 2004
09:14:45) installed
Dec 10 14:08:39 fiveoften kernel: GFS <CVS> (built Dec 10 2004 09:14:04)
installed
Dec 10 14:08:39 fiveoften kernel: CMAN <CVS> (built Dec 10 2004
09:51:59) installed
Dec 10 14:08:39 fiveoften kernel: NET: Registered protocol family 30
Dec 10 14:08:39 fiveoften kernel: DLM <CVS> (built Dec 10 2004 09:52:25)
installed
Dec 10 14:08:39 fiveoften kernel: Lock_DLM (built Dec 10 2004 09:14:25)
installed
Dec 10 14:08:40 fiveoften kernel: CMAN: Waiting to join or form a
Linux-cluster
Dec 10 14:09:11 fiveoften kernel: CMAN: sending membership request
Dec 10 14:09:11 fiveoften kernel: CMAN: got node fouroften
Dec 10 14:09:11 fiveoften kernel: CMAN: quorum regained, resuming
activity
Dec 10 14:09:11 fiveoften ccsd[3391]: Cluster is not quorate. Refusing
connection.
Dec 10 14:09:11 fiveoften ccsd[3391]: Error while processing connect:
Connection refused
Dec 10 14:09:12 fiveoften ccsd[3391]: Cluster is not quorate. Refusing
connection.
Dec 10 14:09:12 fiveoften ccsd[3391]: Error while processing connect:
Connection refused
Dec 10 14:09:13 fiveoften ccsd[3391]: Cluster is not quorate. Refusing
connection.
Dec 10 14:09:13 fiveoften ccsd[3391]: Error while processing connect:
Connection refused
Dec 10 14:09:14 fiveoften ccsd[3391]: Cluster is not quorate. Refusing
connection.
Dec 10 14:09:14 fiveoften ccsd[3391]: Error while processing connect:
Connection refused
Dec 10 14:09:15 fiveoften ccsd[3391]: Cluster is not quorate. Refusing
connection.
Dec 10 14:09:15 fiveoften ccsd[3391]: Error while processing connect:
Connection refused
Dec 10 14:09:16 fiveoften ccsd[3391]: Cluster is not quorate. Refusing
connection.
Dec 10 14:09:16 fiveoften ccsd[3391]: Error while processing connect:
Connection refused
Dec 10 14:09:17 fiveoften ccsd[3391]: Cluster is not quorate. Refusing
connection.
Dec 10 14:09:17 fiveoften ccsd[3391]: Error while processing connect:
Connection refused
Dec 10 14:09:18 fiveoften ccsd[3391]: Cluster is not quorate. Refusing
connection.
Dec 10 14:09:18 fiveoften ccsd[3391]: Error while processing connect:
Connection refused
Dec 10 14:09:19 fiveoften ccsd[3391]: Cluster is not quorate. Refusing
connection.
Dec 10 14:09:19 fiveoften ccsd[3391]: Error while processing connect:
Connection refused
Dec 10 14:09:20 fiveoften ccsd[3391]: Cluster is not quorate. Refusing
connection.
Dec 10 14:09:20 fiveoften ccsd[3391]: Error while processing connect:
Connection refused


The logs stopped when fence_tool exited.

On node fiveoften, /proc/cluster/status and /proc/cluster/nodes contain:

[mbrookov fiveoften ~]$ more /proc/cluster/status
Protocol version: 4.0.1
Config version: 6
Cluster name: CSMTEST
Cluster ID: 9374
Membership state: Cluster-Member
Nodes: 2
Expected_votes: 1
Total_votes: 2
Quorum: 1
Active subsystems: 0
Node addresses: 138.67.4.25

[mbrookov fiveoften ~]$ more /proc/cluster/nodes
Node  Votes Exp Sts  Name
   1    1    1   M   fouroften
   2    1    1   M   fiveoften
[mbrookov fiveoften ~]$

On node fouroften, /proc/cluster/status and /proc/cluster/nodes contain:

[mbrookov fouroften ~]$ more /proc/cluster/status
Protocol version: 4.0.1
Config version: 6
Cluster name: CSMTEST
Cluster ID: 9374
Membership state: Cluster-Member
Nodes: 2
Expected_votes: 1
Total_votes: 2
Quorum: 1
Active subsystems: 1
Node addresses: 138.67.4.24

[mbrookov fouroften ~]$ more /proc/cluster/nodes
Node  Votes Exp Sts  Name
   1    1    1   M   fouroften
   2    1    1   M   fiveoften

Log entries on node fouroften:
Dec 10 14:08:36 fouroften kernel: Lock_Harness <CVS> (built Dec 10 2004
09:14:45) installed
Dec 10 14:08:36 fouroften kernel: GFS <CVS> (built Dec 10 2004 09:14:04)
installed
Dec 10 14:08:36 fouroften kernel: CMAN <CVS> (built Dec 10 2004
09:51:59) installed
Dec 10 14:08:36 fouroften kernel: NET: Registered protocol family 30
Dec 10 14:08:36 fouroften kernel: DLM <CVS> (built Dec 10 2004 09:52:25)
installed
Dec 10 14:08:36 fouroften kernel: Lock_DLM (built Dec 10 2004 09:14:25)
installed
Dec 10 14:08:37 fouroften kernel: CMAN: Waiting to join or form a
Linux-cluster
Dec 10 14:09:09 fouroften kernel: CMAN: forming a new cluster
Dec 10 14:09:09 fouroften kernel: CMAN: quorum regained, resuming
activity
Dec 10 14:09:09 fouroften kernel: CMAN: got node fiveoften



/etc/cluster/cluster.conf: <?xml version="1.0"?> <cluster name="CSMTEST" config_version="6">

<cman two_node="1" expected_votes="1">
</cman>

<clusternodes>
	<clusternode name="fouroften" votes="1">
		<fence>
			<method name="cascade1">
				<device name="human" ipaddr="fouroften"/>
			</method>
		</fence>
	</clusternode>

	<clusternode name="fiveoften" votes="1">
		<fence>
			<method name="cascade1">
				<device name="human" ipaddr="fiveoften"/>
			</method>
		</fence>
	</clusternode>

</clusternodes>

<fencedevices>
        <fencedevice name="human" agent="fence_manual"/>
</fencedevices>

</cluster>

Both nodes are running Fedora Core 3 with the 2.6.9 kernel from
kernel.org.

Thanks for you time!

Matt
mbrookov mines edu


-- Linux-cluster mailing list Linux-cluster redhat com http://www.redhat.com/mailman/listinfo/linux-cluster



[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]