Re: "Out of Memory: Killed process" errors on server running Oracle

[David Knight clubcorp com]

What is the status if the SGA in oracle?
are you running ocfs? do you have plenty of disk space?

Gene <gene udder org>
Sent by: taroon-list-bounces redhat com

07/12/2007 04:17 PM

Please respond to
"Discussion of Red Hat Enterprise Linux 3 (Taroon)" <taroon-list redhat com>

To	"Discussion of Red Hat Enterprise Linux 3 (Taroon)" <taroon-list redhat com>
cc
Subject	"Out of Memory: Killed process" errors on server running Oracle

Hola,

  Every 5-6 days I'm getting "Out of Memory: Killed process"
messages on a Redhat AS 3 server running Oracle 9.20.  These errors
will occur for a couple hours and will eventually lead to Oracle shutting
down.  The load on the server did increase recently, but there's still
1.8GB memory free on the system during these errors.

  The box is running the smp kernel.  Should I do something like
increase the stack size or use the hugemem kernel?

$ uname -a 
Linux prddb02 2.4.21-15.0.3.ELsmp #1 SMP Tue Jun 29 18:04:47 EDT 2004 i686
i686 i386 GNU/Linux

$ ulimit -a
core file size        (blocks, -c) 0
data seg size         (kbytes, -d) unlimited 
file size             (blocks, -f) unlimited
max locked memory     (kbytes, -l) 4
max memory size       (kbytes, -m) unlimited
open files                  
 (-n) 65536
pipe size          (512 bytes, -p) 8 
stack size            (kbytes, -s) 10240
cpu time             (seconds, -t) unlimited
max user processes            (-u) 16384
virtual memory        (kbytes, -v) unlimited

>From /var/log/messages:

Jul 11 19:06:31 prddb02 kernel: Mem-info:
Jul 11 19:06:31 prddb02 kernel: Zone:DMA freepages:  2549 min:  
  0 low:     0 high:     0
Jul 11 19:06:31 prddb02 kernel: Zone:Normal freepages:  1254 min:
 1279 low:  4544 high:  6304 
Jul 11 19:06:31 prddb02 kernel: Zone:HighMem freepages:501745 min:  
255 low: 20992 high: 31488
Jul 11 19:06:31 prddb02 kernel: Free pages:      505548
(501745 HighMem)
Jul 11 19:06:31 prddb02 kernel:   aa:0 ac:0 id:0 il:0 ic:0 fr:2549
Jul 11 19:06:31 prddb02 kernel:   aa:0 ac:40 id:82 il:73 ic:0 fr:1254
Jul 11 19:06:31 prddb02 kernel:   aa:530276 ac:82904 id:17203 il:6249
ic:8419 fr:501745
Jul 11 19:06:31 prddb02 kernel: 1*4kB 2*8kB 2*16kB 1*32kB 0*64kB 1*128kB
1*256kB 1*512kB 1*1024kB 0*2048kB 2*4096kB = 10196kB) 
Jul 11 19:06:31 prddb02 kernel: 137*4kB 121*8kB 45*16kB 19*32kB 6*64kB
2*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 5020kB)
Jul 11 19:06:31 prddb02 kernel: 35001*4kB 11806*8kB 4209*16kB 1479*32kB
1490*64kB 513*128kB 291*256kB 48*512kB 15*1024kB 5*2048kB 335*4096kB =
2006980kB) 
Jul 11 19:06:31 prddb02 kernel: Swap cache: add 0, delete 0, find 0/0,
race 0+0
Jul 11 19:06:31 prddb02 kernel: 192549 pages of slabcache
Jul 11 19:06:31 prddb02 kernel: 1192 pages of kernel stacks
Jul 11 19:06:31 prddb02 kernel: 0 lowmem pagetables, 127372 highmem pagetables
Jul 11 19:06:31 prddb02 kernel: Free swap:       8192980kB
Jul 11 19:06:31 prddb02 kernel: 1572864 pages of RAM
Jul 11 19:06:31 prddb02 kernel: 1277920 pages of HIGHMEM
Jul 11 19:06:31 prddb02 kernel: 89860 reserved pages 
Jul 11 19:06:31 prddb02 kernel: 24863915 pages shared
Jul 11 19:06:31 prddb02 kernel: 0 pages swap cached
Jul 11 19:06:31 prddb02 kernel: Out of Memory: Killed process 8302 (oracle).

$ sar -r -f /var/log/sa/sa11
10:00:00 AM kbmemfree kbmemused  %memused kbbuffers  kbcached
kbswpfree kbswpused  %swpused  kbswpcad
05:00:00 PM   1901836   4030180     67.94    
 4228   1976412   8192980         0
     0.00         0
05:10:00 PM   1755672   4176344     70.40    
 1456   2115832   8192980         0
     0.00         0
05:20:00 PM   1901120   4030896     67.95    
 2952   1976168   8192980         0
     0.00         0
05:30:00 PM   1896664   4035352     68.03    
 2392   1975920   8192980         0
     0.00         0
05:39:59 PM   1892232   4039784     68.10    
 2200   1975900   8192980         0
     0.00         0 
05:50:00 PM   1889808   4042208     68.14    
 2408   1976076   8192980         0
     0.00         0
06:00:00 PM   1897984   4034032     68.00    
 2336   1976244   8192980         0
     0.00         0
06:10:00 PM   1831708   4100308     69.12    
 1192   2040788   8192980         0
     0.00         0
06:20:01 PM   1832808   4099208     69.10    
 1372   2040912   8192980         0
     0.00         0
06:30:03 PM   1844548   4087468     68.91    
 1284   2040988   8192980         0
     0.00         0
06:40:01 PM   1895092   4036924     68.05    
 1304   2041120   8192980         0
     0.00         0 
06:50:00 PM   1909852   4022164     67.80    
 1300   2041248   8192980         0
     0.00         0
07:00:01 PM   1906980   4025036     67.85  
   1104   2041300   8192980        
0      0.00         0
07:10:01 PM   4363412   1568604     26.44  
  13816    726432   8192980        
0      0.00         0

-Gene
gene udder org--
Taroon-list mailing list
Taroon-list redhat com
https://www.redhat.com/mailman/listinfo/taroon-list