[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [linux-lvm] RAID chunk size & LVM 'offset' affecting RAID stripe alignment

On Jun 25, 2010, at 4:36 AM, Linda A. Walsh wrote:
Doug Ledford wrote:
Correction: all reads benefit from larger chunks now a days. The only
reason to use smaller chunks in the past was to try and get all of
your drives streaming data to you simultaneously, which effectively
made the total aggregate throughput of those reads equal to the
throughput of one data disk times the number of data disks in the
array.  With modern drives able to put out 100MB/s sustained by
themselves, we don't really need to do this any more, ....
	I would regard 100MB/s as moderately slow.  For files in my
server cache, my Win7 machine reads @ 110MB/s over the network, so as
much as file-io slows down network response, 100MB would be on
the slow side.  I hope for at least 2-3 times that with software
RAID, but with hardware raid 5-6X that is common. Write speeds run maybe 50-100MB/s slower?

In practice you get better results than that. Maybe not a fully linear scale up, but it goes way up. My test system anyway was getting 400-500MB/s under the right conditions.

and if we aren't
attempting to get this particular optimization (which really only
existed when you were doing single threaded sequential I/O anyway,
which happens to be rare on real servers), then larger chunk sizes
benefit reads because they help to ensure that reads will, as much as
possible, only hit one disk. If you can manage to make every read you
service hit one disk only, you maximize the random I/O ops per second
that your array can handle.
	I was under the impression that rule of thumb was that IOPs of
a RAID array were generally equal to that of 1 member disk, because normally they operate as 1 spindle.

With a small chunk size, this is the case, yes.

 It seems like in your case, you
are only using the RAID component for the redundancy rather than the

No, I'm trading off some speed up in sequential throughput for a speed up in IOPs.

	If you want to increase IOPs, above the single spindle
rate, then I had the impression that using a multi-level RAID would
accomplish that -- like RAID 50 or 60?    I.e. a RAID0 of 3 RAID5's
would give you 3X the IOP's  (because, like in your example, any
read would likely only use a fraction of a stripe), but you would
still benefit from using multiple devices for a read/write to get

In truth, whether you use a large chunk size, or smaller chunk sizes and stacked arrays, the net result is the same: you make the average request involve fewer disks, trading off maximum single stream throughput for IOPs.

My argument in all of this is that single threaded streaming performance is such a total "who cares" number, that you are silly to ever chase that particular beast. Almost nothing in the real world that is doing I/O at speeds that we even remotely care about, is doing that I/O in a single stream. Instead, it's various different threads of I/O to different places in the array and what we care about is that the array be able to handle enough IOPs that the array stays ahead of the load. An exception to this rule might be something like the data acquisition equipment at CERN's HADRON collider. That stuff dumps data in a continuous stream so fast that it makes my mind hurt.

 I seem to remember something about multiprocessor checksumming
going into some recent kernels that could allow practical multi-level
RAID in software.

Red herring. You can do multilevel raid without this feature, and this feature is currently broken so I wouldn't recommend using it.

in response to my
observation that my 256K-data wide stripes (4x64K chunks) would be
skewed by a
chunk size on my PV's that defaulted to starting data at offset 192K
So, we end up touching two stripes instead
of one and we have to read stuff in, introducing a latency delay,
before we can write our data out.
Duh...missing the obvious, I am! Sigh. I think I got it write...oi veh! If not, well... dumping and restoring that much data just takes WAY too long. (beginning to think 500-600MB read/writes are too slow...
actually for dump/restore -- I'm lucky when I get an 8th of that).

linux-lvm mailing list
linux-lvm redhat com
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]