[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: how to counteract slowdown



Daniel Pittman wrote:
> 
> On Mon, 12 Nov 2001, Andrew Morton wrote:
> 
> [...]
> 
> > A fragmented journal would be bad, of course.  It's better to
> > create a journal on an empty disk if one possibly can.
> >
> > However in this case, I'm theorising that the blockage is on
> > checkpoint writeback: ie, the data which has already been
> > written into the journal and which is now being written into
> > the main fs - we can't recycle the journal space until this data
> > has hit disk, and we write it _all_ out.
> >
> > Interesting.  Thanks.  We need to start checkpointing earlier,
> > non-blockingly.  hmm.
> 
> Please.  At the moment, performance on my machine has some notable
> hiccups when fetching mail.
> 
> I use fetchmail and it takes around 30 seconds to pull down 250 messages
> oven the SSH tunnel it's using. So, I see the mail fetching running fine
> for five seconds -- then stall as data is written out to disk. ~1.5k
> write operations, in fact, stalling the system for ~3/4 seconds.

Is this filesytem mounted in ordered-data mode?

Does fetchmail write 250 files, or one?

If both, then a write of 1500 potentially very discontiguous
blocks will certainly take some time.  This normally won't
cause the writer to block.  But if that same process needs
to _read_ something, the write activity will certainly delay
that.   

The ideal fix for this is to not spread the data all over the
disk.   is all the write activity to files in the same directory,
or to multiple ones?

Journalled data may provide some small improvement here.

> The fetch then picks up again, happily, until the next five second
> burst.
> 
> The setup is that mail is fetched and fed to a local Postfix process
> that drops it into it's internal queue system, which is on a disk with a
> 100MB journal and data=journaled.

Postfix does lots of fsync()s.
 
> Fetchmail continues doing this for ~30 seconds, fetching mail from a
> remote system and feeding it to Postfix fairly quickly.
> 
> Postfix, in the meantime, shuffles the file name around between a few
> directories on that first disk, then appends the data to ~/Mailbox. This
> is on a second partition, but the same disk, with a 100MB journal and
> data=journaled.
> 
> The total data fetched is well under 5MB. Even accounting for ten copies
> of that, it should *never* have filled the journal more than 50% full --
> and the mail fetch has been the *only* read/write activity (other than
> inode atime) going on at that stage...

Well, there are a lot of synchronous writes going on here.  They cost.
 
> I can't for the life of me see why my system ends up blocking on writes
> every five seconds during this process. It strikes me that the data
> should all be hitting the journal and, at worst, starting to flush
> gently. No blocking of anything, if possible.

That's what should be happening for journalled data mode.  We write
everything in a nice slurp into the journal and then leave it for
kupdate writeback.   Unless we come under pressure for journal space
or memory, in which case a complete flush is forced.

Either way around, the data has to be written out into the main
filesystem sometime, somehow.

 
> It would be *really* nice to see a two or three level flush implemented,
> based on the degree to which the journal has filled.
> 
> More importantly, it would be good to see the synchronous flush at the 5
> second mark (write-back delay in the kjournald thread) stop happening
> unless it actually *needs* to be done.

We commit when there's data which is mor ethan five seconds old.
That should be OK.   The advantages of delaying 30 seconds, which
is what ext2 will do, are fairly small for normal non-benchmark
workloads.


>From your description, if the first fs is in ordered data mode
I don't think there's anything wrong or fixable here.   The best way
to improve is to be able to lay the data out better on disk, which
is currently being looked at.


Or am I being complacent here?  You provided a great description.
Please, just a little more ;)





[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]