ext3 sucks

[Joshua Baker-LePain <jlb17 duke edu>]

Well, on my servers with 3ware 7500-8 boards in hardware RAID5 mode it 
does, and I thought the subject line might get a bit of attention. 
Basically, no matter how I tune, I can't get anything even resembling 
decent write speed out of hardware that used to run quite well.  I've 
played with tuning both 'blockdev --setra' as well as 
/sys/block/$DEVICE/queue/nr_requests, and nothing increases ext3 write 
speed over 20MB/s on a single board.  To prove it isn't hardware, I also 
tested using the centosplus kernel and XFS on the exact same array.  XFS 
was able to get over 40MB/s on a single board.  When doing a RAID0 over 2 
boards, ext3 maxed out at 35MB/s while XFS was able to do >100MB/s.  All 
these tests were done using bonnie++ and a filesize >4x the amount of 
system RAM.

I'd love to use XFS, but I don't trust it with 4K stacks, so I need to 
get ext3 working at something resembling decent speeds.  Any ideas where I 
should start looking for this problem?  Any more tuning parameters that 
may make a difference?  I've tried 'mount -o noreservation' (found in a 
vaguly promising looking bugzilla entry) and that made no difference.

Thanks.

--
Joshua Baker-LePain
Department of Biomedical Engineering
Duke University