[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

[dm-devel] LVM/DM transfer size scaling issue



[Not sure if this is the right forum - apologies in advance if it's not...]

I'm running some LVM/DM scaling performance runs on a 4-way IA64 box (HP RX2600), that is equipped with four dual-U320 SCSI host bus adapters. Each bus has 7 U320 disk drives (15K) attached for a total of 56 drives. [Each dual-U320 HBA is capable of doing on the order of 512MB/sec sustained throughput, I've measured the system using direct IO (no LVM) at being able to read data at up to 1.8GB/sec.] The system has RHEL4 Update 1 Beta bits installed (2.6.9 based kernel).

The test program is the SPEW benchmark, and was run with the DIRECT IO flag set.

There were ten (10) LVM volumes used, they were all constructed with a 16KB stripe size. The volumes were -

Volume 1 - 1 disk
Volume 2 - 2 disks, each disk on a separate U320 bus
Volume 3 - 3 disks, each disk on a separate U320 bus
...
Volume 8 - 8 disks, each disk on a separate U320 bus
Volume 9 - 9 disks, 1 disk on 7 buses, 2 disks each on 1 U320 bus
Volume 10 - 10 disks, 1 disk on 6 buses, 2 disks each on 2 U320 buses

I then ran 4 groups of tests, with each group consisting of a separate run against each of the 10 volumes. The groups had a block size associated with it based upon the number of disks in a volume. Thus the first group of runs had a block size of 16KB times the number of disks in the volume. [The idea being to have a single transfer split into one 16KB against each of the disks in the volume.] The second group had a block size of 32KB times the number of disks in the volume. [The idea here was to see the effects of the wrapping - for example, for Volume 2 with 2 disks (striped at 16KB) we would expect each 64KB initial transfer to be split into four 16KB-transfers, with the first and third delivered to the first drive in the volume, and the second and fourth transfer going to the second drive in the volume.] The third group had a block size of 48KB times the number of disks, and the fourth had a block size of 64KB times the number of disks in the volume.

The expectation was to see that the overall performance would be at least as good or hopefully even better as the overall transfer size increased. [If for no other reasons except that (a) the amount of time spent doing IO transfers at the device level should dwarf any overhead at the application/LVM(DM) layers, and (b) the targetted devices could be given multiple commands and thus reduce idle time.] However, that is *not* what was seen. The accompanying xmgrace-generated chart illustrates this (I've also attached the xmgrace file itself, in case you are having problems with the .png file included...):



Here we see the black line showing scaling along the first group (16KB per segmenet/disk per transfer) - in fact we see good scaling up to 4 disks (pretty linear), with a slight hiccup, and then it continues onwards at less of an increase. Something strange at 5 disks happens, and again at 9/10 disks - the latter perhaps due to the fact that we're wrapping on the disks now (some adapters have two disks busy at a time). I'm going to look into the actual bus/target mappings for the volumes to see if they play a role.

But note, when we go to group 2 (32KB per segment per transfer), the performance is always slightly lower up to 3 disks, and then starting at 4 disks we see almost a flattening out (with a big dip at volume 6). We do see some (slight) scaling again from 32KB to 48KB and then 64KB, but we never again attain the rate of simply issuing the 16KB per segment initial requests.

It's interesting to see the dip at 6-disks per volume for all the transfer sizes that wrap.

I've been investigating this, but was wondering if anyone had noticed something like this - or perhaps had a ready explanation for this...

Alan D. Brunelle
HP

PNG image

# Grace project file
#
@version 50117
@page size 792, 612
@page scroll 5%
@page inout 5%
@link page off
@map font 0 to "Times-Roman", "Times-Roman"
@map font 1 to "Times-Italic", "Times-Italic"
@map font 2 to "Times-Bold", "Times-Bold"
@map font 3 to "Times-BoldItalic", "Times-BoldItalic"
@map font 4 to "Helvetica", "Helvetica"
@map font 5 to "Helvetica-Oblique", "Helvetica-Oblique"
@map font 6 to "Helvetica-Bold", "Helvetica-Bold"
@map font 7 to "Helvetica-BoldOblique", "Helvetica-BoldOblique"
@map font 8 to "Courier", "Courier"
@map font 9 to "Courier-Oblique", "Courier-Oblique"
@map font 10 to "Courier-Bold", "Courier-Bold"
@map font 11 to "Courier-BoldOblique", "Courier-BoldOblique"
@map font 12 to "Symbol", "Symbol"
@map font 13 to "ZapfDingbats", "ZapfDingbats"
@map color 0 to (255, 255, 255), "white"
@map color 1 to (0, 0, 0), "black"
@map color 2 to (255, 0, 0), "red"
@map color 3 to (0, 255, 0), "green"
@map color 4 to (0, 0, 255), "blue"
@map color 5 to (255, 255, 0), "yellow"
@map color 6 to (188, 143, 143), "brown"
@map color 7 to (220, 220, 220), "grey"
@map color 8 to (148, 0, 211), "violet"
@map color 9 to (0, 255, 255), "cyan"
@map color 10 to (255, 0, 255), "magenta"
@map color 11 to (255, 165, 0), "orange"
@map color 12 to (114, 33, 188), "indigo"
@map color 13 to (103, 7, 72), "maroon"
@map color 14 to (64, 224, 208), "turquoise"
@map color 15 to (0, 139, 0), "green4"
@reference date 0
@date wrap off
@date wrap year 1950
@default linewidth 1.0
@default linestyle 1
@default color 1
@default pattern 1
@default font 0
@default char size 1.000000
@default symbol size 1.000000
@default sformat "%.8g"
@background color 0
@page background fill on
@timestamp off
@timestamp 0.03, 0.03
@timestamp color 1
@timestamp rot 0
@timestamp font 0
@timestamp char size 1.000000
@timestamp def "Wed Jun 29 16:48:16 2005"
@r0 off
@link r0 to g0
@r0 type above
@r0 linestyle 1
@r0 linewidth 1.0
@r0 color 1
@r0 line 0, 0, 0, 0
@r1 off
@link r1 to g0
@r1 type above
@r1 linestyle 1
@r1 linewidth 1.0
@r1 color 1
@r1 line 0, 0, 0, 0
@r2 off
@link r2 to g0
@r2 type above
@r2 linestyle 1
@r2 linewidth 1.0
@r2 color 1
@r2 line 0, 0, 0, 0
@r3 off
@link r3 to g0
@r3 type above
@r3 linestyle 1
@r3 linewidth 1.0
@r3 color 1
@r3 line 0, 0, 0, 0
@r4 off
@link r4 to g0
@r4 type above
@r4 linestyle 1
@r4 linewidth 1.0
@r4 color 1
@r4 line 0, 0, 0, 0
@g0 on
@g0 hidden false
@g0 type XY
@g0 stacked false
@g0 bar hgap 0.000000
@g0 fixedpoint off
@g0 fixedpoint type 0
@g0 fixedpoint xy 0.000000, 0.000000
@g0 fixedpoint format general general
@g0 fixedpoint prec 6, 6
@with g0
@    world 0.5, 0, 10.5, 400
@    stack world 0, 0, 0, 0
@    znorm 1
@    view 0.150000, 0.150000, 1.150000, 0.850000
@    title "IA64 4-way w/ U320 SPEW LVM"
@    title font 0
@    title size 1.500000
@    title color 1
@    subtitle ""
@    subtitle font 0
@    subtitle size 1.000000
@    subtitle color 1
@    xaxes scale Normal
@    yaxes scale Normal
@    xaxes invert off
@    yaxes invert off
@    xaxis  on
@    xaxis  type zero false
@    xaxis  offset 0.000000 , 0.000000
@    xaxis  bar on
@    xaxis  bar color 1
@    xaxis  bar linestyle 1
@    xaxis  bar linewidth 1.0
@    xaxis  label "Segments per volume"
@    xaxis  label layout para
@    xaxis  label place auto
@    xaxis  label char size 1.000000
@    xaxis  label font 0
@    xaxis  label color 1
@    xaxis  label place normal
@    xaxis  tick on
@    xaxis  tick major 1
@    xaxis  tick minor ticks 0
@    xaxis  tick default 6
@    xaxis  tick place rounded true
@    xaxis  tick in
@    xaxis  tick major size 1.000000
@    xaxis  tick major color 1
@    xaxis  tick major linewidth 1.0
@    xaxis  tick major linestyle 1
@    xaxis  tick major grid off
@    xaxis  tick minor color 1
@    xaxis  tick minor linewidth 1.0
@    xaxis  tick minor linestyle 1
@    xaxis  tick minor grid off
@    xaxis  tick minor size 0.500000
@    xaxis  ticklabel on
@    xaxis  ticklabel format decimal
@    xaxis  ticklabel prec 0
@    xaxis  ticklabel formula ""
@    xaxis  ticklabel append ""
@    xaxis  ticklabel prepend ""
@    xaxis  ticklabel angle 0
@    xaxis  ticklabel skip 0
@    xaxis  ticklabel stagger 0
@    xaxis  ticklabel place normal
@    xaxis  ticklabel offset auto
@    xaxis  ticklabel offset 0.000000 , 0.010000
@    xaxis  ticklabel start type auto
@    xaxis  ticklabel start 0.000000
@    xaxis  ticklabel stop type auto
@    xaxis  ticklabel stop 0.000000
@    xaxis  ticklabel char size 1.000000
@    xaxis  ticklabel font 0
@    xaxis  ticklabel color 1
@    xaxis  tick place both
@    xaxis  tick spec type none
@    yaxis  on
@    yaxis  type zero false
@    yaxis  offset 0.000000 , 0.000000
@    yaxis  bar on
@    yaxis  bar color 1
@    yaxis  bar linestyle 1
@    yaxis  bar linewidth 1.0
@    yaxis  label "MBytes per second"
@    yaxis  label layout para
@    yaxis  label place auto
@    yaxis  label char size 1.000000
@    yaxis  label font 0
@    yaxis  label color 1
@    yaxis  label place normal
@    yaxis  tick on
@    yaxis  tick major 100
@    yaxis  tick minor ticks 3
@    yaxis  tick default 6
@    yaxis  tick place rounded true
@    yaxis  tick in
@    yaxis  tick major size 1.000000
@    yaxis  tick major color 1
@    yaxis  tick major linewidth 1.0
@    yaxis  tick major linestyle 3
@    yaxis  tick major grid on
@    yaxis  tick minor color 1
@    yaxis  tick minor linewidth 1.0
@    yaxis  tick minor linestyle 2
@    yaxis  tick minor grid on
@    yaxis  tick minor size 0.500000
@    yaxis  ticklabel on
@    yaxis  ticklabel format general
@    yaxis  ticklabel prec 5
@    yaxis  ticklabel formula ""
@    yaxis  ticklabel append ""
@    yaxis  ticklabel prepend ""
@    yaxis  ticklabel angle 0
@    yaxis  ticklabel skip 0
@    yaxis  ticklabel stagger 0
@    yaxis  ticklabel place both
@    yaxis  ticklabel offset auto
@    yaxis  ticklabel offset 0.000000 , 0.010000
@    yaxis  ticklabel start type auto
@    yaxis  ticklabel start 0.000000
@    yaxis  ticklabel stop type auto
@    yaxis  ticklabel stop 0.000000
@    yaxis  ticklabel char size 1.000000
@    yaxis  ticklabel font 0
@    yaxis  ticklabel color 1
@    yaxis  tick place both
@    yaxis  tick spec type none
@    altxaxis  off
@    altyaxis  off
@    legend on
@    legend loctype view
@    legend 0.22, 0.8
@    legend box color 1
@    legend box pattern 1
@    legend box linewidth 1.0
@    legend box linestyle 1
@    legend box fill color 0
@    legend box fill pattern 1
@    legend font 0
@    legend char size 0.800000
@    legend color 1
@    legend length 4
@    legend vgap 1
@    legend hgap 1
@    legend invert false
@    frame type 0
@    frame linestyle 1
@    frame linewidth 1.0
@    frame color 1
@    frame pattern 1
@    frame background color 0
@    frame background pattern 0
@    s0 hidden false
@    s0 type xy
@    s0 symbol 1
@    s0 symbol size 1.000000
@    s0 symbol color 1
@    s0 symbol pattern 1
@    s0 symbol fill color 1
@    s0 symbol fill pattern 0
@    s0 symbol linewidth 1.0
@    s0 symbol linestyle 1
@    s0 symbol char 65
@    s0 symbol char font 0
@    s0 symbol skip 0
@    s0 line type 1
@    s0 line linestyle 1
@    s0 line linewidth 1.0
@    s0 line color 1
@    s0 line pattern 1
@    s0 baseline type 0
@    s0 baseline off
@    s0 dropline off
@    s0 fill type 0
@    s0 fill rule 0
@    s0 fill color 1
@    s0 fill pattern 1
@    s0 avalue off
@    s0 avalue type 2
@    s0 avalue char size 1.000000
@    s0 avalue font 0
@    s0 avalue color 1
@    s0 avalue rot 0
@    s0 avalue format general
@    s0 avalue prec 3
@    s0 avalue prepend ""
@    s0 avalue append ""
@    s0 avalue offset 0.000000 , 0.000000
@    s0 errorbar on
@    s0 errorbar place both
@    s0 errorbar color 1
@    s0 errorbar pattern 1
@    s0 errorbar size 1.000000
@    s0 errorbar linewidth 1.0
@    s0 errorbar linestyle 1
@    s0 errorbar riser linewidth 1.0
@    s0 errorbar riser linestyle 1
@    s0 errorbar riser clip off
@    s0 errorbar riser clip length 0.100000
@    s0 comment "/work/common/data/lvm/2005_06_29/results.dat"
@    s0 legend  "16KB per segment transfers"
@    s1 hidden false
@    s1 type xy
@    s1 symbol 1
@    s1 symbol size 1.000000
@    s1 symbol color 2
@    s1 symbol pattern 1
@    s1 symbol fill color 2
@    s1 symbol fill pattern 0
@    s1 symbol linewidth 1.0
@    s1 symbol linestyle 1
@    s1 symbol char 65
@    s1 symbol char font 0
@    s1 symbol skip 0
@    s1 line type 1
@    s1 line linestyle 1
@    s1 line linewidth 1.0
@    s1 line color 2
@    s1 line pattern 1
@    s1 baseline type 0
@    s1 baseline off
@    s1 dropline off
@    s1 fill type 0
@    s1 fill rule 0
@    s1 fill color 1
@    s1 fill pattern 1
@    s1 avalue off
@    s1 avalue type 2
@    s1 avalue char size 1.000000
@    s1 avalue font 0
@    s1 avalue color 1
@    s1 avalue rot 0
@    s1 avalue format general
@    s1 avalue prec 3
@    s1 avalue prepend ""
@    s1 avalue append ""
@    s1 avalue offset 0.000000 , 0.000000
@    s1 errorbar on
@    s1 errorbar place both
@    s1 errorbar color 2
@    s1 errorbar pattern 1
@    s1 errorbar size 1.000000
@    s1 errorbar linewidth 1.0
@    s1 errorbar linestyle 1
@    s1 errorbar riser linewidth 1.0
@    s1 errorbar riser linestyle 1
@    s1 errorbar riser clip off
@    s1 errorbar riser clip length 0.100000
@    s1 comment "/work/common/data/lvm/2005_06_29/results.dat"
@    s1 legend  "32KB"
@    s2 hidden false
@    s2 type xy
@    s2 symbol 1
@    s2 symbol size 1.000000
@    s2 symbol color 3
@    s2 symbol pattern 1
@    s2 symbol fill color 3
@    s2 symbol fill pattern 0
@    s2 symbol linewidth 1.0
@    s2 symbol linestyle 1
@    s2 symbol char 65
@    s2 symbol char font 0
@    s2 symbol skip 0
@    s2 line type 1
@    s2 line linestyle 1
@    s2 line linewidth 1.0
@    s2 line color 3
@    s2 line pattern 1
@    s2 baseline type 0
@    s2 baseline off
@    s2 dropline off
@    s2 fill type 0
@    s2 fill rule 0
@    s2 fill color 1
@    s2 fill pattern 1
@    s2 avalue off
@    s2 avalue type 2
@    s2 avalue char size 1.000000
@    s2 avalue font 0
@    s2 avalue color 1
@    s2 avalue rot 0
@    s2 avalue format general
@    s2 avalue prec 3
@    s2 avalue prepend ""
@    s2 avalue append ""
@    s2 avalue offset 0.000000 , 0.000000
@    s2 errorbar on
@    s2 errorbar place both
@    s2 errorbar color 3
@    s2 errorbar pattern 1
@    s2 errorbar size 1.000000
@    s2 errorbar linewidth 1.0
@    s2 errorbar linestyle 1
@    s2 errorbar riser linewidth 1.0
@    s2 errorbar riser linestyle 1
@    s2 errorbar riser clip off
@    s2 errorbar riser clip length 0.100000
@    s2 comment "/work/common/data/lvm/2005_06_29/results.dat"
@    s2 legend  "48KB"
@    s3 hidden false
@    s3 type xy
@    s3 symbol 1
@    s3 symbol size 1.000000
@    s3 symbol color 4
@    s3 symbol pattern 1
@    s3 symbol fill color 4
@    s3 symbol fill pattern 0
@    s3 symbol linewidth 1.0
@    s3 symbol linestyle 1
@    s3 symbol char 65
@    s3 symbol char font 0
@    s3 symbol skip 0
@    s3 line type 1
@    s3 line linestyle 1
@    s3 line linewidth 1.0
@    s3 line color 4
@    s3 line pattern 1
@    s3 baseline type 0
@    s3 baseline off
@    s3 dropline off
@    s3 fill type 0
@    s3 fill rule 0
@    s3 fill color 1
@    s3 fill pattern 1
@    s3 avalue off
@    s3 avalue type 2
@    s3 avalue char size 1.000000
@    s3 avalue font 0
@    s3 avalue color 1
@    s3 avalue rot 0
@    s3 avalue format general
@    s3 avalue prec 3
@    s3 avalue prepend ""
@    s3 avalue append ""
@    s3 avalue offset 0.000000 , 0.000000
@    s3 errorbar on
@    s3 errorbar place both
@    s3 errorbar color 4
@    s3 errorbar pattern 1
@    s3 errorbar size 1.000000
@    s3 errorbar linewidth 1.0
@    s3 errorbar linestyle 1
@    s3 errorbar riser linewidth 1.0
@    s3 errorbar riser linestyle 1
@    s3 errorbar riser clip off
@    s3 errorbar riser clip length 0.100000
@    s3 comment "/work/common/data/lvm/2005_06_29/results.dat"
@    s3 legend  "64KB"
@target G0.S0
@type xy
1 70.52
2 138.21
3 201.13
4 271.44
5 275.3
6 319.66
7 356
8 366.55
9 327.93
10 362.51
&
@target G0.S1
@type xy
1 64.62
2 103.68
3 190.05
4 202.64
5 210.48
6 160.71
7 171.36
8 170.12
9 188.88
10 206.91
&
@target G0.S2
@type xy
1 70.32
2 106.69
3 190.09
4 209.55
5 251.58
6 185.25
7 210.11
8 224.61
9 218.05
10 219.72
&
@target G0.S3
@type xy
1 70.41
2 109.8
3 190.6
4 216.79
5 264.03
6 202.39
7 225.53
8 240.77
9 252.18
10 245.6
&

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]