malloc and 'Active' memory

[Matthijs van der Klip]

On Fri, 29 Jul 2005, Rick Stevens wrote:
> Matthijs van der Klip wrote:
> > This all raises a new question however. In the case of fillmem the Active
> > memory is released again. Why is this not happening after a configure /
> > make / make install episode? I have a particular compile ready that leaves 
> > close to 1GB of Active memory after each run. After running this a couple 
> > of times, the total amount of Active memory rises to close of 6GB again. 
> > Does this mean something in this compile process is leaking memory? Would 
> > that explain why the Active memory is not released again as soon as the 
> > compiler, linker etc. have finished their work?
> > 
> > I am not the first one to experience this problem by the way:
> > 
> > http://lists.debian.org/debian-kernel/2004/12/msg00410.html
> 
> Well, malloc() will fail if you request a chunk of memory and there
> isn't a SINGLE chunk available of that size.  So if memory gets fragged,
> there isn't a single 7GB chunk available and malloc() will fail.
> fillmem allocates in smaller chunks, then releases it all so the
> memory defragger can clean things up.

I see what you mean and this was entirely my first thought when I ran into 
this problem. However I was told (true or not true) that the malloc 
implementation on Fedora Core 4 could not suffer from memory fragmentation 
in the way I described (e.g. the same way you describe). I've checked the 
documentation and found some interesting references. From the 'malloc' 
manpage:

'By default, Linux follows an optimistic memory allocation strategy. This 
means that when malloc() returns non-NULL there is no guarantee that 
the memory really is available. This is a really  bad  bug.  In case it 
turns out that the system is out of memory, one or more processes will be 
killed by the infamous OOM killer.  In case Linux is employed under 
circumstances where it would be less desirable to suddenly lose some 
randomly picked processes, and moreover the kernel version is sufficiently  
recent, one can switch off this  overcommitting behavior using a command like
    # echo 2 > /proc/sys/vm/overcommit_memory
See also the kernel Documentation directory, files  
vm/overcommit-accounting and sysctl/vm.txt.'

Unfortunately sysctl/vm.txt from the kernel-doc-2.6.12-1.1398_FC4 rpm 
describes kernel 2.2:

'This file contains the documentation for the sysctl files in
/proc/sys/vm and is valid for Linux kernel version 2.2.'

So while this is interesting with respect to the overcommitting behaviour 
(that's probably why I can allocate two chunks of 6GB while physical 
memory is limited at 8GB), it doesn't provide much of an answer to my 
questions surrounding the 'Active' memory:

'Memory that has been used more recently and usually not reclaimed unless 
absolutely necessary.'

Maybe I should let go of that and accept your explenation of the memory 
fragmentation, but I'm still bothered by it. What is that 'Active' memory 
doing there and why can't I use it? Furthermore why does a build process 
leave so much of that darned 'Active' memory and can a single run of 
fillmem 'magically' free it al?

I've been experimenting with a setting of '2' for the
/proc/sys/vm/overcommit_memory tuneable, and as it turns out the largest
block I can allocate directly after a clean boot is roughly 3.8GB. So it
seems FC4 is suffering badly from memory fragmentation after all. The
overcommit situation makes it a bit less visible, but it's still there.


> Ideally, that's what mysql should do.  Or start off at some huge
> size and keep trying progressively smaller chunks until it gets some,
> e.g. try 8GB.  If that fails, try 6GB, then 4, then 2, you get the
> idea.  It could then link those together and manage them.

I will propose this on the MySQL list, I hope I can explain why they have 
to do their own high level memory management.


Best regards,

-- 
Matthijs van der Klip
System Administrator
Spill E-Projects
The Netherlands