[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [Linux-cluster] OOM failures with GFS, NFS and Samba on a cluster with RHEL3-AS



/proc/meminfo:
        total:    used:    free:  shared: buffers:  cached:
Mem:  4189741056 925650944 3264090112        0 18685952 76009472
Swap: 2146787328        0 2146787328
MemTotal:      4091544 kB
MemFree:       3187588 kB
MemShared:           0 kB
Buffers:         18248 kB
Cached:          74228 kB
SwapCached:          0 kB
Active:         107232 kB
ActiveAnon:      50084 kB
ActiveCache:     57148 kB
Inact_dirty:      1892 kB
Inact_laundry:   16276 kB
Inact_clean:     16616 kB
Inact_target:    28400 kB
HighTotal:     3276544 kB
HighFree:      3164096 kB
LowTotal:       815000 kB
LowFree:         23492 kB
SwapTotal:     2096472 kB
SwapFree:      2096472 kB
Committed_AS:    72244 kB
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     2048 kB

When a bunch of locks become free, lowmem seems to recover somewhat. However, shutting down lock_gulmd entirely does NOT return lowmem to what it probably should be (though I'm not sure if the system is just keeping all of that memory cached until something else needs it or not).

jonathan

Jonathan Woytek wrote:
Michael Conrad Tadpol Tilstra wrote:

On Sun, Jan 23, 2005 at 01:45:28PM -0500, Jonathan Woytek wrote:

Additional information:

I enabled full output on lock_gulmd, since my dead top sessions would often show that process near the top of the list around the time of crashes. The machine was rebooted around 10:50AM, and was down again at



Not suprising that lock_gulmd is working hard when gfs is under heavy use. Its it busy processing all those lock requests. What would be more useful from gulm for this than the logging messages, is to query the locktable every so often for its stats. `gulm_tool getstats <master>:lt000` The 'locks = ###' line is how many lock structures are current held. gulm is very greedy about memory, and you are running the lock servers on the same nodes you're mounting from.


Here are the stats from the master lock_gulmd lt000:

I_am = Master
run time = 9436
pid = 2205
verbosity = Default
id = 0
partitions = 1
out_queue = 0
drpb_queue = 0
locks = 20356
unlocked = 17651
exclusive = 15
shared = 2690
deferred = 0
lvbs = 17661
expired = 0
lock ops = 107354
conflicts = 0
incomming_queue = 0
conflict_queue = 0
reply_queue = 0
free_locks = 69644
free_lkrqs = 60
used_lkrqs = 0
free_holders = 109634
used_holders = 20366
highwater = 1048576


Something keeps eating away at lowmem, though, and I still can't figure out what exactly it is.



also, just to see if I read the first post right, you have
samba->nfs->gfs?


If I understand your arrows correctly, I have a filesystem mounted with GFS that I'm sharing via NFS to another machine that is sharing it via Samba. I've closed that link, though, to try to eliminate that as a problem. So now I'm serving the GFS filesystem directly through Samba.

jonathan


-- Jonathan Woytek w: 412-681-3463 woytek+ cmu edu NREC Computing Manager c: 412-401-1627 KB3HOZ PGP Key available upon request


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]