[Consult-list] Re: [dm-devel] dm-multipath has great throughput but we'd like more!

[Nicholas C. Strugnell]

On Thu, 2006-05-18 at 08:44 +0100, Bob Gautier wrote:
> On Thu, 2006-05-18 at 02:25 -0500, Jonathan E Brassow wrote:
> > The system bus isn't a limiting factor is it?  64-bit PCI-X will get 
> > 8.5 GB/s (plenty), but 32-bit PCI 33MHz got 133MB/s.
> > 
> > Can your disks sustain that much bandwidth? 10 striped drives might get 
> > better than 200MB/s if done right, I suppose.
> > 

This is an HDS Lightning - 64GB of mirrored write cache - I doubt if any
of the writes even see disk :-)

> > Don't the switches run at 2 Gbits/s?  2 Gbits/s / 10 (throw in 2 bits 
> > for protocol) ~= 200MB/s.
> > 
> 
> Thanks for the fast responses:
> 
> The card is a 64-bit PCI-X, so I don't think the bus is the bottleneck,
> and anyway the vendor specifies a maximum throughput of 200Mbyte/s per
> card.
> 
> The disk array does not appear to be the bottleneck because we get
> 200Mbyte/s when we use *two* HBAs in load-balanced mode.
> 
> The question is really about why we only see O(100Mbyte/s) with one HBA
> when we can achieve O(200MByte/s) with two cards, given that one card
> should be able to achieve that throughput.
> 

We've just done _exactly_ the same test against an EVA 8000 with 8
active paths - theoretically we should be able to get 4Gb/s via two HBAs
but in fact we saw max 200MB/s with ext2, dropping to 160MB/s with ext3
- this was to a fairly slow RAID5 but that is irrelevant as we have a
16GB write cache and we were only writing 4GB files with bonnie++

I'm not sure where the overhead is. The fact that we see a 20%
performance drop when we switch journalling on suggests that the
overhead might be in the filesystem perhaps?

It might make sense to test raw writes to a device with dd and see if
that gets comparable performance figures - I'll just try that myself
actually.

Nick 

-- 
M: +44 (0)7736 665171           Skype: nstrug
http://europe.redhat.com
GPG FPR: 9C6C 093C 756A 6C57 49A1  E211 BBBA F5F5 C440 5DE0

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 191 bytes
Desc: This is a digitally signed message part
URL: <http://listman.redhat.com/archives/dm-devel/attachments/20060518/789a6fa2/attachment.sig>