Re: ext3 sucks

[Marc Michelsen <marc u washington edu>]

Joshua Baker-LePain wrote:

> On Mon, 24 Oct 2005 at 8:05pm, Arjan van de Ven wrote
>
> > On Mon, 2005-10-24 at 13:49 -0400, Joshua Baker-LePain wrote:
> >
> > > Well, on my servers with 3ware 7500-8 boards in hardware RAID5 mode it
> > > does, and I thought the subject line might get a bit of attention.
> > > Basically, no matter how I tune, I can't get anything even resembling
> > > decent write speed out of hardware that used to run quite well.  I've
> > > played with tuning both 'blockdev --setra' as well as
> > > /sys/block/$DEVICE/queue/nr_requests, and nothing increases ext3 write
> > > speed over 20MB/s on a single board.  To prove it isn't hardware, I 
> > > also
> > > tested using the centosplus kernel and XFS on the exact same array.  
> > > XFS
> > > was able to get over 40MB/s on a single board.  When doing a RAID0 
> > > over 2
> > > boards, ext3 maxed out at 35MB/s while XFS was able to do >100MB/s.  
> > > All
> > > these tests were done using bonnie++ and a filesize >4x the amount of
> > > system RAM.
> >
> >
> > did you set the journalling mode to write-back?
> > The default ext3 journalling mode is more strict than the XFS one, but
> > slower (so the trade off is data integrity vs speed); XFS is more
> > comparable with the writeback type. The performance difference is mostly
> > visible on new file creation...
> First off, I'd like to apologize for the subject line.  It's been a 
> bad few days, but that's no excuse to start a thread out like that.  
> *sigh*
>
> Thanks for the suggestions.  I tested with 3 different sets of my 
> "tweak settings", and setting 'data=writeback' actually slowed things 
> down (if it had any effect).
>
> Also, to answer the question about versions, all ext3 testing was done 
> with 2.6.9-11.ELsmp, and all XFS testing was done with 
> 2.6.9-11.106.unsupportedsmp (centosplus) -- the box is essentialy at 
> U1. And I'm using bonnie++-1.03a.
>
> If there are any more suggestions/tweaks/patches, I'd love to hear 
> 'em. Thanks again.
From previous benchmarking I've done in the past I'd bet that
if you only used one drive on each of your 3ware cards
that you'd probably get 40-45MB/s to each drive and ~90MB/s
with a software raid-0 to both drives using an ext3 filesystem
in all cases. I assume XFS would be similar. If this is the
case then ext3 by itself isnt causing the problem but perhaps
how the ext3 journal and filesystem are layed out interacts with
the underlying hardware raid-5. I'm thinking
it might be doing more raid-5 updates (where only one
block in the stripe is updated, parity is read and updated and written back
out along with the updated block) than full stripes (where
all blocks are written out with new parity).

Have you tweaked the -E stride=stripe-size option (formerly -R)
to mkfs.ext3? I can't recall getting too much improvement,
but I switched from 3wares hardware raid-5 to linux's software raid-5
first and that improved performance.

Marc