[dm-devel] Shell Scripts or Arbitrary Priority Callouts?

[Pasi Kärkkäinen]

On Tue, Mar 24, 2009 at 05:01:04PM +0200, Pasi Kärkkäinen wrote:
> On Tue, Mar 24, 2009 at 08:21:45AM -0400, John A. Sullivan III wrote:
> > > 
> > > Core-iscsi developer seems to be active developing at least the 
> > > new iSCSI target (LIO target).. I think he has been testing it with
> > > core-iscsi, so maybe there's newer version somewhere? 
> > > 
> > > > We did play with the multipath rr_min_io settings and smaller always
> > > > seemed to be better until we got into very large numbers of session.  We
> > > > were testing on a dual quad core AMD Shanghai 2378 system with 32 GB
> > > > RAM, a quad port Intel e1000 card and two on-board nvidia forcedeth
> > > > ports with disktest using 4K blocks to mimic the file system using
> > > > sequential reads (and some sequential writes).
> > > > 
> > > 
> > > Nice hardware. Btw are you using jumbo frames or flow control for iSCSI
> > > traffic? 
> > > 
> 
> Dunno if you noticed this.. :) 
> 
> 
> > > > 
> > > 
> > > When you used dm RAID0 you didn't have any multipath configuration, right? 
> > Correct although we also did test successfully with multipath in
> > failover mode and RAID0.
> > > 
> 
> OK.
> 
> > > What kind of stripe size and other settings you had for RAID0?
> > Chunk size was 8KB with four disks.  
> > > 
> 
> Did you try with much bigger sizes.. 128 kB ?
> 
> > > What kind of performance do you get using just a single iscsi session (and
> > > thus just a single path), no multipathing, no DM RAID0 ? Just a filesystem
> > > directly on top of the iscsi /dev/sd? device.
> > Miserable - same roughly 12 MB/s.
> 
> OK, Here's your problem. Was this btw reads or writes? Did you tune
> readahead-settings? 
> 
> Can paste your iSCSI session settings negotiated with the target? 
> 
> > > 
> > > Sounds like there's some other problem if invidual throughput is bad? Or did
> > > you mean performance with a single disktest IO thread is bad, but using multiple
> > > disktest threads it's good.. that would make more sense :) 
> > Yes, the latter.  Single thread (I assume mimicking a single disk
> > operation, e.g., copying a large file) is miserable - much slower than
> > local disk despite the availability of huge bandwidth.  We start
> > utilizing the bandwidth when multiplying concurrent disk activity into
> > the hundreds.
> > 
> > I am guessing the single thread performance problem is an open-iscsi
> > issue but I was hoping multipath would help us work around it by
> > utilizing multiple sessions per disk operation.  I suppose that is where
> > we run into the command ordering problem unless there is something else
> > afoot.  Thanks - John
> 
> You should be able to get many times the throughput you get now.. just with
> a single path/session.
> 
> What kind of latency do you have from the initiator to the target/storage? 
> 
> Try with for example 4 kB ping:
> ping -s 4096 <ip_of_the_iscsi_target>
> 
> 1000ms divided by the roundtrip you get from ping should give you maximum
> possible IOPS using a single path.. 
> 
> 4 kB * IOPS == max bandwidth you can achieve.
> 

Maybe I should have been more clear about that.. assuming you're measuring 
4 kB IO's with disktest, and you have 1 outstanding IO at a time, then the
above is max throughput you can get.

Higher block/IO size and higher number of outstanding IOs will give you
better thoughput. 

I think CFQ disk elevator/scheduler has a bug that prevents queue depths
bigger than 1 outstanding IO.. so don't use that. "noop" might be a good idea.

-- Pasi