[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [linux-lvm] pvmove painfully slow on parity RAID

On 12/30/2010 08:12 PM, Stuart D. Gathman wrote:
On Thu, 30 Dec 2010, Spelic wrote:

Also there is still the mystery of why times appear *randomly* related to the
number of devices, chunk sizes, and stripe sizes! if the rmw cycle was the
culprit, how come I see:
raid5, 4 devices, 16384k chunk: 41sec (4.9MB/sec)
raid5, 6 device, 4096k chunk: 2m18sec ?!?! (1.44 MB/sec!?)
the first has much larger stripe size of 49152K , the second has 20480K !
Ok, next theory.  Pvmove works by allocating a mirror for each
contiguous segment of the source LV, update metadata

Ok never mind, I found the problem:
LVM probably uses O_DIRECT, right?
Well it's absymally slow on MD parity raid (I checked with dd on the bare MD device just now) and I don't know why it's so slow. It's not because of the rmw because it's slow even the second time I try, when it does not read anything anymore because all reads are in cache already.

I understand this is probably to be fixed at MD side (and I will report the problem to linux-raid, but I see it has already been discussed without much results)
...is there any chance you might fix it at lvm side too, changing LVM to use nondirect IO so to "support" MD? In my raid5 array between direct and nondirect (dd bs=1M or smaller) there's the difference of 2.1MB/s to 250MB/sec, and would probably be greater on larger arrays. Also in raid10 nondirect is much faster for small transfer sizes like bs=4K (28MB/sec to 160MB/sec) but not at 1M, however LVM probably uses low transfer sizes, right?

Thank you

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]