Memory use...

["Sean" <seanlkml sympatico ca>]

Yesterday, we had a couple processes OOM killed while the box should have
been almost unused.   So we're trying to look at memory use a bit closer. 
What are the best tools?

On this RHEL4 box "top" currently shows about 1.3Gb in use:

top - 07:42:52 up 1 day, 10:29,  8 users,  load average: 0.07, 0.02, 0.00
Tasks:  79 total,   1 running,  78 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.3% us,  0.3% sy,  0.0% ni, 98.1% id,  0.3% wa,  0.0% hi,  0.0% si
Mem:   3090664k total,  2108212k used,   982452k free,   138204k buffers
Swap:  2096472k total,        8k used,  2096464k free,   592180k cached


Adding the virtual memory size (ps -e -o vsz) of all the processes only
totals: 996Mb.  And presumably this represents less physical memory still.

Could some of this be accounted for by kernel slab use (or are they
counted as buffers/cache)?

 Active / Total Objects (% used)    : 876797 / 1045021 (83.9%)
 Active / Total Slabs (% used)      : 136543 / 136544 (100.0%)
 Active / Total Caches (% used)     : 86 / 124 (69.4%)
 Active / Total Size (% used)       : 482447.22K / 501425.41K (96.2%)
 Minimum / Average / Maximum Object : 0.02K / 0.48K / 128.00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
300870 152539  50%    0.09K   6686       45     26744K buffer_head
292816 292816 100%    0.23K  18301       16     73204K dentry_cache
200514 200514 100%    0.59K  33419        6    133676K proc_inode_cache
 99652  99652 100%    1.94K  49826        2    199304K task_struct
 89624  89624 100%    0.82K  22406        4     89624K ext3_inode_cache
 23751  15970  67%    0.52K   3393        7     13572K radix_tree_node
[...]

200Mb for task_struct seems rather large given that there are only 80
processes running, doesn't it?

Anyway, is there a better way to dig into the memory to see how it is
being used?   It seems like we're using way more memory than we'd expect.

Thanks,
Sean