[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: ext3 performance issue with a Berkeley db application



Greg Louis <glouis dynamicro on ca> wrote:
>
> cd /root_of_tree
> ./autogen.sh --arguments-to-configure
> (such as --prefix=/usr or whatever; you don't really need args for this
> test)

Doh, OK, I needed to update automake & autoconf.

> 
> That was a bit optimistic.  The script Matthias was thinking of is a
> shuffle filter; takes stdin and outputs the lines in random order.  I
> can't give you a script that creates the input you need to reproduce
> the problem.  However, if you can download a 4Mb file from
>   http://www.bgl.nu/bogofilter/randlist.txt.bz2
> and uncompress it, then you can do
>   mkdir scratch
>   ./bogoutil -l scratch/newdb <randlist.txt
> and that will take ca half an hour on ext3/ordered but around 4 minutes
> on ext2; a bit longer than 4 minutes on ext3/writeback.  Fast drives
> might shrink those times in proportion.

Ah lovely, thanks.

I tried this on my 800MHz/256M laptop (running 2.5.59-mm7++) and yeah, it
sucks.  So I did:

	mount /dev/hda6 -o remount,commit=120

and lo, 

bogoutil -l newdb < ~/randlist.txt  19.06s user 28.04s system 95% cpu 49.468 total

Look at the `vmstat 1' output during the normal commit=5 run:

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  2      0   4544  26476 177576    0    0     4    11   12    11  0  0 99  1
 0  2      0   4544  26476 177576    0    0     0  1080 1246    96  0  1  0 99
 0  2      0   5384  26460 176552    0    0     4   784 1172    81  0  2  0 98
 0  2      0   5392  26460 176552    0    0     0   880 1197    70  0  1  0 99
 0  2      0   5392  26460 176552    0    0     0   876 1204    79  0  0  0 100
 0  2      0   5392  26460 176552    0    0     0   916 1211    80  0  1  0 99
 0  2      0   5392  26460 176556    0    0     0   792 1182    60  0  1  0 99
 0  2      0   5392  26460 176568    0    0     0   840 1187    84  1  1  0 98
 0  2      0   5392  26460 176572    0    0     0   992 1216    75  1  2  0 97

Very dribbly, and it's doing a huge amount of I/O.

vmstat 1 with commit=120:

 1  0      0   5920  26576 176260    0    0     0     0 1004     9 38 62  0  0
 1  0      0   5500  26580 176664    0    0     0     0 1004    11 38 62  0  0
 1  0      0   4980  26580 177176    0    0     0     0 1004     9 38 62  0  0
 1  0      0   4568  26580 177584    0    0     0     0 1003     9 38 62  0  0
 1  0      0   5280  26552 176868    0    0     0     0 1004    11 39 61  0  0
 1  0      0   4832  26552 177324    0    0     0     0 1005     9 40 60  0  0
 1  0      0   4440  26552 177696    0    0     0     0 1004    11 39 61  0  0
 1  0      0   5112  26524 177060    0    0     0     0 1004    11 40 60  0  0
 1  0      0   4608  26524 177552    0    0     0     0 1004     9 41 59  0  0
 1  0      0   5224  26496 176944    0    0     0     0 1004    11 39 61  0  0
 1  0      0   4776  26500 177380    0    0     0     0 1004     9 39 61  0  0
 1  0      0   5392  26444 176868    0    0     0     0 1004    13 36 64  0  0
 1  0      0   4832  26444 177428    0    0     0     0 1004     9 39 61  0  0
 1  0      0   5448  26396 176796    0    0     0     0 1004    11 38 62  0  0
 1  0      0   4888  26396 177348    0    0     0     0 1004     9 36 64  0  0
 1  0      0   4496  26396 177748    0    0     0     0 1003     9 38 62  0  0
 0  1      0   5616  25704 176948    0    0     0 15188 1003    14 34 56  0 10
 0  1      0   5616  25704 176948    0    0     0  7696 1075    27  0  2  0 98
 0  1      0   5740  25704 176948    0    0     0   520 1106    50  0  1  0 99
 0  0      0   6400  25712 176948    0    0     0    52 1037    40  0  2 74 24
 0  0      0   6400  25712 176948    0    0     0     0 1004     9  0  0 100  0

blam.

Now, looking at the enormous amount of system time which the commit=120 run
took, I assume that the application is doing a _ton_ of overwriting. 
Redirtying the same pages again and again and again.  So poor old ext3 keeps
rewriting them again and again.

You'll hit similar problems with ext2 - on a slower computer, or on a larger
database, or on a system with the kupdate interval decreased from the 30
second default.

My recommendation is to work out why the application is performing so much
overwriting, and make it stop.  

Now without a kernel profile that's all speculation.  I need to reboot this
machine to run a profile, so I'm going to hit send on this lot ;)






[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]