[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [Linux-cluster] GFS2: processes stuck in "just schedule"





On 12/04/2009 04:39 AM, Steven Whitehouse wrote:
Hi,

On Thu, 2009-12-03 at 17:30 -0500, Allen Belletti wrote:
Hi All,

After Steve and the RedHat guys dug into my nasty crashdump (thanks
all!), I believe I'm down to the last GFS2 problem on our mail cluster,
but it's a common one.

I've always had trouble with processes getting stuck on GFS2 access and
queuing up.  Since the 5.4 upgrade and moving the proper GFS2 kernel
module, it's changed but not gone away.  Ever few days now, I'm seeing
processes getting stuck with WCHAN=just_schedule.  Once this starts
happening, both cluster nodes will accumulate them rapidly which
eventually brings IO to a halt.  The only way I've found to escape is
via a reboot, sometimes of one, sometimes of both nodes.

Since there's no crash, I don't get any useful debug information.
Outside of this one repeating glitch, performance is great and all is
well.  If anyone can suggest ways of gathering more data about the
problem, or possible solutions, I would be grateful.

Thanks,
Allen


This would be typical for what happens when there is contention on a
glock between two (or more) nodes. There is a mechanism which is
supposed to try and mitigate the issue (by allowing each node to hold on
to a glock for a minimum period of time which is designed to ensure that
some work is done each time a node acquires a glock) but if your storage
is particularly slow, and/or possibly depending upon the exact I/O
pattern, it may not always be 100% effective.

In the first instance though, see if you can find an inode which is
being contended from both nodes as that will most likely be the culprit,
We've got a 3-4 year old Sun 3510 FC array shared between the two nodes. The utilization on it is generally quite reasonable, so I doubt that this would qualify as "particularly slow". Also, the very busiest times for the mail system are usually during the night rsync backups and it rarely if ever gets wedged during those times.

Can you give me some hints as to how I might go about finding a inode that's being contended for by both nodes? I assume that would be useful to confirm what the problem is at least.

Thanks,
Allen

--
Allen Belletti
allen isye gatech edu                             404-894-6221 Phone
Industrial and Systems Engineering                404-385-2988 Fax
Georgia Institute of Technology


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]