[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [linux-lvm] pvmove painfully slow on parity RAID

On 12/30/2010 03:42 AM, Stuart D. Gathman wrote:
On Wed, 29 Dec 2010, Spelic wrote:
I tried multiple times for every device with consistent results, so I'm pretty
sure these are actual numbers.
What's happening?
Apart from the amazing difference of parity raid vs nonparity raid, with
parity raid it seems to vary randomly with the number of devices and the
This is pretty much my experience with parity raid all around.  Which
is why I stick with raid1 and raid10.

Parity raid goes fast for me for normal filesystem operations, that's why I suppose there is some strict sequentiality is enforced here.

That said, the sequential writes of pvmove should be fast for raid5 *if*
the chunks are aligned so that there is no read/modify/write cycle.

1) Perhaps your test targets are not properly aligned?

aligned to zero yes (arrays are empty now), but all raids have different chunk sizes and stripe sizes as I reported, which are all bigger than the lvm chunksize which is 1M for the VG.

2) Perhaps the raid5 implementation (hardware? linux md?
    experimental lvm raid5?) does a read modify write even when it
    doesn't have to.

Your numbers sure look like read/modify/write is happening for some reason.

Ok but strict sequentiality is probably enforced too much. There must be some barrier or flush & wait thing going on here at each tiny bit of information (at each lvm chunk maybe?). Are you a lvm devel? Consider that a sequential dd write goes hundreds of megabytes per second on my arrays, not hundreds of... kilobytes! Even random io goes *much* faster than this, if one stripe does not have to wait for another stripe to be fully updated (i.e. sequentiality not enforced from the application layer). If pvmove ouputed 100MB before every sync or flush, I'm pretty sure I would see speeds almost 100 times higher.

Also there is still the mystery of why times appear *randomly* related to the number of devices, chunk sizes, and stripe sizes! if the rmw cycle was the culprit, how come I see:
raid5, 4 devices, 16384k chunk: 41sec (4.9MB/sec)
raid5, 6 device, 4096k chunk: 2m18sec ?!?! (1.44 MB/sec!?)
the first has much larger stripe size of 49152K , the second has 20480K !

Thank you

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]