[Linux-cluster] service state unchanged when host crashes

[Martin Waite]

Hi Jakov,

I managed to get fencing working - at least enough for my experiments.  

Sure enough, I hit the same problem:

clusternode30 is running service "SENTINEL" - and then is powered down
at ~ 18:19

Oct 27 18:19:55 clusternode27 clurgmgrd[2785]: <debug> Membership Change
Event
Oct 27 18:19:55 clusternode27 clurgmgrd[2785]: <info> State change:
clusternode30 DOWN
Oct 27 18:19:55 clusternode27 clurgmgrd[2785]: <debug> Membership Change
Event
Oct 27 18:19:55 clusternode27 clurgmgrd[2785]: <debug> Membership Change
Event
Oct 27 18:19:55 clusternode27 fenced[2760]: clusternode30 not a cluster
member after 0 sec post_fail_delay
Oct 27 18:19:55 clusternode27 fenced[2760]: fencing node "clusternode30"
Oct 27 18:19:55 clusternode27 fenced[2760]: can't get node number for
node p<CA>@#001
Oct 27 18:19:55 clusternode27 fenced[2760]: fence "clusternode30"
success
Oct 27 18:19:55 clusternode27 clurgmgrd[2785]: <debug> 22 rules loaded

(The "can't get node number" looks suspicious, but fenced claims to
succeed).

Next morning - it still hasn't relocated the service:

Cluster Status for testcluster @ Wed Oct 28 11:13:55 2009
Member Status: Quorate

 Member Name                                                     ID
Status
 ------ ----                                                     ----
------
 clusternode27                                                      27
Online, Local, rgmanager
 clusternode28                                                      28
Online, rgmanager
 clusternode30                                                      30
Offline

 Service Name                                                     Owner
(Last)                                                     State
 ------- ----                                                     -----
------                                                     -----
 service:SCRIPT
clusternode28                                                    started
 service:SENTINEL
clusternode30                                                    started
 service:VIP
clusternode27                                                    started
 service:mysql_authdb_service
clusternode27                                                    started


I am going to strip my config down later on so that SENTINEL is the only
running service.   My fencing mechanism is pretty pathetic - I have
added a new fence agent that does nothing but always succeeds (which I
hope is enough for this stage in my education) - but my understanding is
that the sequence of events should be something like this:

1. <node fails>
2. cman notices  (and groupd)
3. fencing is applied to the node
4. the service is relocated - or marked as failed

regards,
Martin

-----Original Message-----
From: linux-cluster-bounces at redhat.com
[mailto:linux-cluster-bounces at redhat.com] On Behalf Of Jakov Sosic
Sent: 28 October 2009 10:04
To: linux-cluster at redhat.com
Subject: Re: [Linux-cluster] service state unchanged when host crashes

On Tue, 27 Oct 2009 09:57:50 -0000
"Martin Waite" <Martin.Waite at datacash.com> wrote:

> I am running Debian Lenny 64-bit.   Is that going to be a problem for
> me ?

Well maybe. Last time I tried RedHat Cluster Suite on Debian Lenny was
two months ago, and then I had stumbled upon the following bug:

http://www.mail-archive.com/linux-cluster@redhat.com/msg06018.html

I don't know if they have fixed that bug... but it resembles totally to
your problem... Node goes down, node gets fenced, service is seen as
down by rgmanager, but there is no action to relocate it to a live
cluster member. That was a start of a project for me, so after that I
migrated to CentOS 5 (which is a free RHEL fork).


> I think you have given me enough of a pointer - ie. I haven't
> configured fencing properly - to get me going again.  Thanks.

I can see that from the logs now :) If you get to the point where bug
that I explained earlier pops up, please share that information here so
that we know the state of RHCS on Debian.



-- 
|    Jakov Sosic    |    ICQ: 28410271    |   PGP: 0x965CAE2D   |
=================================================================
| start fighting cancer -> http://www.worldcommunitygrid.org/   |

--
Linux-cluster mailing list
Linux-cluster at redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster