[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

[Linux-cluster] recovering from "resource groups locked" error?




If clustat reports rgmanager as online, why would any clusvcadm operation fail with "Try again (resource groups locked)" ?

Is there any way to recover from that rgmanger failure/error besides resetting the entire cluster?

Details --

Yesterday evening a technician connected a Netgear GS748T switch to my network. The new switch somehow caused a storm of traffic that in turn caused a disruption of network connectivity across the entire LAN, including to all of my CS/GFS cluster nodes, for a few minutes until the new switch was removed from the network.

This morning when I finally had a chance to investigate I found that all of the cluster members that are supposed to be online were online and that the cluster was quorate. But rgmanager would not work and services running under rgmanager were hung. (The cluster must have become inquorate and blocked access to the shared GFS volume while the outage was in progress. But some of the services and rgmanager never recovered?)

I first tried resetting the "lead" member. (This is a pool of mirrored storage servers where the lead member creates a rsync batch off of a main fileserver and all of the other members then replay the rsync batch that is on a shared filesystem against their local filesystem mirror of the main fileserver)

No matter what I did rgmanager would not start. cman_tool services would report code "S-1,80,4" --

root gfs05:~
(0)>cman_tool services
Service          Name                              GID LID State     Code
Fence Domain:    "default"                           1   2 run       -
[2 1 4 3]

DLM Lock Space:  "clvmd"                             2   3 run       -
[2 1 4 3]

User:            "usrm::manager"                     0   4 join      S-1,80,4
[]

Other cluster members would report rgmanager as online, yet when I tried to operate on member services, the operation would fail with "Try again (resource groups locked)".

root gfs06:~
(1)>clustat
Member Status: Quorate

  Member Name                              Status
  ------ ----                              ------
  gfs04                                    Online, rgmanager
  gfs05                                    Online
  gfs06                                    Online, Local, rgmanager
  gfs07                                    Online, rgmanager
  gfs08                                    Offline

  Service Name         Owner (Last)                   State
  ------- ----         ----- ------                   -----
  mapsmirror1          gfs05                          started
  mapsmirror2          gfs06                          started
  mapsmirror3          gfs07                          started
  mapsmirror4          gfs04                          started
  mapsmirror5          (none)                         stopped
root gfs06:~
(0)>clusvcadm -d mapsmirror1
Member gfs06 disabling mapsmirror1...failed: Try again (resource groups locked)

Eventually I just gave up and power cycled all cluster members at ounce. Everything, including rgmanger, then came back online OK.

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]