[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

[linux-lvm] LVM performance anomalies



Hello,

I have some rather odd performance anomalies when using direct i/o towards lvm logical volumes.

I have a setup with 10 external scsi disks attached to the external scsi port of a DELL PERC 4DC RAID controller. Each disk is configured as its own logical disk, hence not using raid functionality in the adapter .

* The disks is pvcreated with "pvcreate /dev/sd[d-m]" .
* The vg is created with vgcreate vg_perc /dev/sd[d-m] .
* Logical volumes is created using for i in b2 b3 b4 b5 ; do lvcreate -L 50g -n lv_${i} -I 64 -i 10 vg_perc ; done

I use fio [1] to generate workload, with this configuration:

[global]
rw=read
size=1280m
blocksize=64k
ioengine=sync
write_lat_log
write_bw_log
direct=1
[job1]
filename=/dev/mapper/vg_perc-lv_b2
[job2]
filename=/dev/mapper/vg_perc-lv_b3
[job3]
filename=/dev/mapper/vg_perc-lv_b4
[job4]
filename=/dev/mapper/vg_perc-lv_b5

Normal, and expected aggregated throughput is 35-40 MB/s . ( Approx 8.5 MB/s each of the LVs sharing the aggragated bandwidth)

I can run this job many times after each other (I ran it continously for 12 Hour once) and get the expected results.

completion latency and bandwidth log plots (from fio) of a job with normal throughput (some anomalies in the start though) is seen here:

http://folk.uio.no/jb/strange-lvm/good-clat.png
http://folk.uio.no/jb/strange-lvm/good-bw.png

Suddenly, in the middle of one the runs, the throughput drops to 128-256kB/s pr LV. If I stop the job and start a job reading directly from the underlying disks I get 39MB/s aggregated . If I start the job against LVs I still get maximum 512kB/s .

completion latency and bandwidth log plots (from fio) of a job with poor throughput is seen here:

http://folk.uio.no/jb/strange-lvm/bad-clat.png
http://folk.uio.no/jb/strange-lvm/bad-bw.png

Summary output from fio after the bad condition is here:
http://folk.uio.no/jb/strange-lvm/bad-fio.out

Summary output from fio after the good condition is here:
http://folk.uio.no/jb/strange-lvm/good-fio.out

Complete blktraces of the lvm devices for bad condition is here:
http://folk.uio.no/jb/strange-lvm/bad-blktraces.tgz

Complete blktraces of the lvm devices for good condition is here:
http://folk.uio.no/jb/strange-lvm/good-blktraces.tgz

I have not been able to find out what i triggering the problem. It happens at random times. If I leave to run in poor condition. Sometimes it recovers again after some time.

If anyone would help me shed some light on what is going on, or suggest how to proceed, it would be much appreciated .

I have tried doing read through an ext3 filesystem instead of directly to the device, and it is the same problem.

[1] http://freshmeat.net/projects/fio/

Best rgds
Jarle Bjørgeengen


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]