Re: kswapd taking 100% CPU - only there's no swap?

[Don MacAskill <don smugmug com>]

Tom Sightler wrote:
> On Mon, 2006-03-06 at 16:27 -0800, Don MacAskill wrote:
> > We have an 8-way RHEL4 AS system (4 dual-core Opterons) with no swap 
> > (because swapping is still broken for DB loads.  It swaps the DB out in 
> > preference of disk cache).
> > ...
> > We're obviously not starving for RAM (3.4GB in disk cache), and we don't 
> > have any swap enabled, so what on earth is it doing and why is it 
> > pegging my CPU?
>
> I think these two statements together basically outline your problem.
> You assume that when swap is enabled and the systems starts swapping,
> that the system is broken.  Then, you disable swap to fix the
> "brokeness" and wonder why kswapd uses lots of CPU.  Well, it's kswapd's
> job to monitor free memory and make sure memory pages are free.  Your
> system has 40MB free (cache usage still counts as used memory, even if
> you don't want to count it), which is probably low enough that kswap
> thinks it needs to free up some memory.  It's busily scanning memory
> looking for stuff to swap out except, hey, what's this, there's no swap,
> so it's a wasted effort but that won't keep it from trying again and
> again.

There was a big discussion about this problem on taroon-list a few years 
ago.   We all spent quite a bit of time trying to tune the swap 
parameters and couldn't do it.  Turning of swap fixed the problem 100%.

Fact:  There was major brokenness under DB loads with swap enabled.  It 
*is* broken.

Fact:  As soon as we disabled swap, our DB boxes saw a 2X boost in 
performance.

I've been running RHEL4 on this box for awhile, ran into a similar 
problem, played with the tuneables, finally disabled swap, my problems 
went away.  Figured it would have been fixed in RHEL4, but no such luck.

Recently, though, I updated to U2.  I *think* (I can't be sure, alas, 
since I got busy) this new problem surfaced.  Seems like I'm stuck - 
swap on, performance is bad.  Swap off, I lose one of my CPU cores.

> If your system is swapping instead of reclaiming cache, don't just
> remove all swap, tune the parameters that control the behavior, namely
> swappiness and vfs_cache_pressure.  Have you already attempted tuning
> with vfs_cache_pressure and not been successful?

Yes, I've tried tuning all of the tuneables I can find.  No real 
difference.  I definitely played with swappiness, but I can't recall if 
I played with vfs_cache_pressure.  I probably did, but it's possible I 
overlooked it - I'll look.

I didn't just turn swap off as a first effort.  It was a last resort - 
and I wasn't alone.

Don