[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [Linux-cluster] Unkillable clurgmgrd





On 11/12/07, Lon Hohberger <lhh redhat com> wrote:
On Sun, 2007-11-11 at 23:57 +0100, Jos Vos wrote:
> Hi,
>
> I have a node that has an unkillable (kill -9 doesn't work) clurgmgrd
> running.  I have fenced it now for the third time, with the same
> result after startup...
>
> Stracing clutstat gives:
>
> [...]
> socket(PF_FILE, SOCK_STREAM, 0)         = 5
> connect(5, {sa_family=AF_FILE, path="/var/run/cluster/rgmanager.sk"}, 110) = -1 ENOENT (No such file or directory)
> close(5)                                = 0
> dup(2)                                  = 5
> fcntl(5, F_GETFL)                       = 0x8002 (flags O_RDWR|O_LARGEFILE)
> fstat(5, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0
> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2aaaaaaac000
> lseek(5, 0, SEEK_CUR)                   = -1 ESPIPE (Illegal seek)
> write(5, "msg_open: No such file or direct"..., 36msg_open: No such file or directory
> ) = 36
> close(5)                                = 0
> munmap(0x2aaaaaaac000, 4096)            = 0
> [...]
>
> How to get this node back up again???
>
> This is on a RHEL 5.0 clone.

If it's unkillable, it's stuck waiting on the kernel for something.

echo 1 > /proc/sys/kernel/sysrq
echo t > /proc/sysrq-trigger

dmesg > foo.out

reply + attach foo.out ;)


I observed a similar problem on the test cluster. It appears the clurgmgrd deadlocks in some cases in groups.c:count_resource_groups(). It does not happen every time but it is reproducible. Surviving node calls rg_lock(service:mysql) @ groups.c:101 and gets stuck. The other node resource manager waits indefinitely for the lock:

{3965} rg_lock(service:mysql) @ groups.c:101
[3460] debug: Sending service states to CTX0xa2a7fd0
no key for rg="service:mysql"
no key for rg="service:test"
[3460] debug: Sending node states to CTX0xa2a7fd0
[3460] debug: Sending service states to CTX0xa2a7fd0
no key for rg="service:mysql"
no key for rg="service:test"

To the original poster: the surviving node clurgmgrd is "unkillable" as well.
You can try to reboot the surviving node - it will release the lock and
resource manager on the fenced node will be unblocked and start just fine.
Unfortunately, once you reboot the node the situation may reverse (resource manager will hang on the rebooted node).


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]