[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: ext3 performance issue with a Berkeley db application

On Sun, 02 Feb 2003, Andrew Morton wrote:

> > Here's the ext3 part.  On another machine I got the following times to
> > build that list of 530,000 tokens, starting by creating empty db files
> > on a partition mounted as:
> How large are the generated output files?

Some ten MB, like 30.

> > Some further findings:
> > 
> > o  It doesn't matter if the source (mbox) file is on ext3/ordered or
> >    ext2; the difference in time is insignificant.
> > 
> > o  Just processing a message to classify it takes about four times as
> >    long if the .db files are on ext3/ordered as it does if the .db
> >    files are on ext2.
> I would be suspecting that the database is opening the files with O_SYNC or
> is running fsync or such.  Maybe.

The graphs on my web site have been created by running strace against
bogofilter and taking the pwrite() offsets, divided by 4096 to give the
"page number", and plot the page numbers over the line number in the
strace. There is exactly one fsync(), and it directly precedes the

In either case, the data base file is opened with O_RDWR|O_LARGEFILE, no
O_SYNC (I straced to figure this). No fdatasync().

> > o  Dumping the tokens and counts from the database in text form and
> >    reloading them into a new database file is not subject to serious
> >    performance problems; on the machine that needs 24 minutes to
> >    build from a 200-Mb mbox, rebuilding the database from a list of
> >    tokens took eight seconds -- this was on ext3 in ordered mode.
> How does this operation differ from the operation which is "slow"?

The access pattern changes a lot, because the data base is dumped in
traversal order which makes reinserting them into a fresh tree have MUCH
better data locality. Most writes are then in sequential order in
respect to the file offset, with some excursions to offset #0 and #4096
(pages #0 and #1), as you can see on

http://mandree.home.pages.de/bogofilter/bogoutil.png   <- write positions
http://mandree.home.pages.de/bogofilter/bogoutil-f.png <- page frequency

There are also fewer write accesses altogether.

> > These data were obtained on machines running linux kernels
> > 2.4.21-pre3-ac4 and -ac5 and 2.4.21-pre4-ac1; kernel 2.4.20-ac2 appears
> > to give similar results though this has not been thorougly tested. 
> > Results like those reported were initially obtained with db-3.1.17; the
> > tests shown here used db-4.1.25.
> > 
> > More info available on request; tuning hints most gratefully received
> > and tested.
> If you can suggest an easy way in which I can reproduce this, that would be
> efficient.

If it's acceptable for you to build the current bogofilter package
http://bogofilter.sourceforge.net/ then Greg could provide you with a
Perl script to create a proper input.

If that's too much an effort for you which I'd perfectly understand,
just state so and I'll ask Greg to send me an strace of his "slow"
program and create a short monolithic C or perhaps Perl program that
just exactly reproduces the scattered pwrite() sequence pattern we
observe in our application.

Are you aware of a module that applies to recent kernel versions and
that traces block numbers of ll_rw_block()? It might turn up some useful
information -- then we'd easier know the ext3 "output" to the hard disk;
we already know the "input" from the application at the syscall level.

BTW: what's the status of the dirsync patches in respect to 2.4.21-pre?
Is further testing needed or just a "ping" to get them merged? Or
does Marcelo not want the patch?

Matthias Andree

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]