[Linux-cluster] cman_tool kill, cluster stuck

[Dan B. Phung]

Hi all, my cluster seems to be stuck!  one of the nodes
went down, and I see a message on one of the live nodes
that repeatedly says:

  fencing node "blade12"
  fence "blade12" failed

so I take a view of my nodes:

cluster # cman_tool nodes
Node  Votes Exp Sts  Name
   1    1    1   M   blade01
   4    1    1   M   blade04
   9    1    1   M   blade09
  10    1    1   M   blade10
  11    1    1   M   blade11
  12    1    1   X   blade12

blade09 and 10 report in, but they don't come all the way up (can't ssh
in) I think because it's hanging on the fencing of blade12.  so I
try:

cluster # cman_tool kill -n12
Can't kill node 12 : No such file or directory

cluster # cman_tool kill -nblade12
kill node failed: Invalid argument

is there a better way to repair my cluster without rebooting everybody?

thanks,
dan

--