[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

[dm-devel] dm-thin vs lvm performance



Hi,
I  recently started using dm-thin module and first of all, thank you for the good work. It works great, well documented and the comments in the code are very useful.  I tried to run some performance tests using a high performance flash device and the performance of dm thin is about 30% of LVM performance on a full allocated thin provisioned volume. ( ie after all pages/blocks are allocated using dd). Performance test does only reads and no writes. 

To reproduce the scenario easily, I tried same operations using ramdisk and dd. 

To read 3.2G of data using dd, 

   Raw Ramdisk takes - 2.5 seconds
   LVM Lun - takes 4 seconds
   TP Lun ( after fully allocated) - takes 18 seconds



Steps involved
------------------
1. Linux 3.2.0.rc4 version from kernel.org
 3.2.0-rc4 #1 SMP Fri Dec 16 13:25:02 EST 2011 x86_64 x86_64 x86_64 GNU/Linux
2. LVM2.2.02.88
3. compiling and installing dm thin modules.
4. Create a ramdisk of size 4G.

Ramdisk Raw Read Test
--------------------------
[root lab-dm-cn4 ~]# dd of=/dev/null if=/dev/ram bs=4096 count=786432
786432+0 records in
786432+0 records out
3221225472 bytes (3.2 GB) copied, 2.56607 s, 1.3 GB/s

LVM SETUP
------------------------

Create PV with 8M metadata on ramdisk , chunksize=8M in VG, lv uses 100%VG.

root lab-dm-cn4 ~]# dd of=/dev/null if=/dev/6C067CWX00132-1/engineering bs=4096 count=$BLOCKS
                                786432+0 records in
                                786432+0 records out
                                3221225472 bytes (3.2 GB) copied, 4.06687 s, 792 MB/s



DM THIN SETUP
-----------------------
Create metadata device of size 8M and data device of size 3G using dm-linear on ramdisk.
Create TP Pool using the above devices with page/block size of 8M.
Create TP LUN in the TP Pool.
Using dd, write 4k to every page in the TP LUN so that all pages are allocated.
using "dmsetup status" verify that all pages in the pool are allocated.

[root lab-dm-cn4 ~]# dd of=/dev/null if=/dev/mapper/thin1 bs=4096 count=$BLOCKS
                                          786432+0 records in
                                          786432+0 records out
                                          3221225472 bytes (3.2 GB) copied, 18.2166 s, 177 MB/s



I have attached the output file and the script file used for this test.  Any ideas / suggestions on what may be the bottle neck in dm-thin performance is much appreciated.

Thanks,
Jagan.



[root lab-dm-cn4 ~]# #constants
[root lab-dm-cn4 ~]# let ONEMEG=1024*2; #translated to sectors
[root lab-dm-cn4 ~]# 
[root lab-dm-cn4 ~]# #user defined
[root lab-dm-cn4 ~]# DMSETUP=/home/jagan/src/LVM2.2.02.88/tools/dmsetup
[root lab-dm-cn4 ~]# LUN=1
[root lab-dm-cn4 ~]# META_DEVICE_START_SECTOR=0 #on $DEVICE_PATH this is where meta data starts
[root lab-dm-cn4 ~]# DEVICE_PATH=/dev/ram
[root lab-dm-cn4 ~]# let META_DEV_SIZE=1*1024*$ONEMEG; #1G translated to sectors used in pvcreate command
[root lab-dm-cn4 ~]# let PAGESIZE=8*$ONEMEG #page size is 8M in sectors
[root lab-dm-cn4 ~]# let DATA_DEV_SIZE=3*1024*$ONEMEG+$PAGESIZE; #3G translated to sectors
[root lab-dm-cn4 ~]# let LVM_DEVICE_SIZE=$META_DEV_SIZE+$DATA_DEV_SIZE;
[root lab-dm-cn4 ~]# let DD_BS=4*1024 # dd block size = 4k  bytes
[root lab-dm-cn4 ~]# let BLOCKS=($DATA_DEV_SIZE-$PAGESIZE)*512/$DD_BS
[root lab-dm-cn4 ~]# 
[root lab-dm-cn4 ~]# #system defined
[root lab-dm-cn4 ~]# let DATA_DEVICE_START_SECTOR=$META_DEVICE_START_SECTOR+$META_DEV_SIZE
[root lab-dm-cn4 ~]# let PAGES=$DATA_DEV_SIZE/$PAGESIZE
[root lab-dm-cn4 ~]# let SEEK=$PAGESIZE*512/$DD_BS
[root lab-dm-cn4 ~]# 
[root lab-dm-cn4 ~]# echo $DMSETUP LUNID=$LUN DEVICE=$DEVICE_PATH DEVICE_START=$META_DEVICE_START_SECTOR META_DEV_SIZE=$META_DEV_SIZE DATA_DEV_SIZE=$DATA_DEV_SIZE  PAGESIZE=$PAGESIZE DD_BS=$DD_BS DATA_DEVICE_START_SECTOR=$DATA_DEVICE_START_SECTOR PAGES=$PAGES SEEK=$SEEK BLOCKS=$BLOCKS LVM_DEVICE_SIZE=$LVM_DEVICE_SIZE
/home/jagan/src/LVM2.2.02.88/tools/dmsetup LUNID=1 DEVICE=/dev/ram DEVICE_START=0 META_DEV_SIZE=2097152 DATA_DEV_SIZE=6307840 PAGESIZE=16384 DD_BS=4096 DATA_DEVICE_START_SECTOR=2097152 PAGES=385 SEEK=2048 BLOCKS=786432 LVM_DEVICE_SIZE=8404992
[root lab-dm-cn4 ~]# 
[root lab-dm-cn4 ~]# pvcreate  -M2 --dataalignment 4K --metadatacopies 1 --metadatasize 8M --zero y -v /dev/ram
    Set up physical volume for "/dev/ram" with 8388608 available sectors
    Zeroing start of device /dev/ram
  Writing physical volume data to disk "/dev/ram"
  Physical volume "/dev/ram" successfully created
[root lab-dm-cn4 ~]# vgcreate -l 16384 -s 8M -c n -A n 6C067CWX00132-1 /dev/ram
  WARNING: This metadata update is NOT backed up
  Volume group "6C067CWX00132-1" successfully created
[root lab-dm-cn4 ~]# lvcreate -A n -C n -l 100%VG -Z y -n engineering -r None 6C067CWX00132-1
  WARNING: This metadata update is NOT backed up
  WARNING: This metadata update is NOT backed up
  Logical volume "engineering" created
[root lab-dm-cn4 ~]# 
[root lab-dm-cn4 ~]# # run dd test on several devices
[root lab-dm-cn4 ~]# 
[root lab-dm-cn4 ~]# #write something
[root lab-dm-cn4 ~]# dd if=/dev/zero of=/dev/6C067CWX00132-1/engineering bs=4096 count=$BLOCKS
786432+0 records in
786432+0 records out
3221225472 bytes (3.2 GB) copied, 4.63318 s, 695 MB/s
[root lab-dm-cn4 ~]# 
[root lab-dm-cn4 ~]# #note the time on lvm device
[root lab-dm-cn4 ~]# dd of=/dev/null if=/dev/6C067CWX00132-1/engineering bs=4096 count=$BLOCKS
786432+0 records in
786432+0 records out
3221225472 bytes (3.2 GB) copied, 4.06687 s, 792 MB/s
[root lab-dm-cn4 ~]# 
[root lab-dm-cn4 ~]# lvremove -f /dev/6C067CWX00132-1/engineering
  Logical volume "engineering" successfully removed
[root lab-dm-cn4 ~]# vgremove 6C067CWX00132-1
  Volume group "6C067CWX00132-1" successfully removed
[root lab-dm-cn4 ~]# pvremove /dev/ram
  Labels on physical volume "/dev/ram" successfully wiped
[root lab-dm-cn4 ~]# lvdisplay
  No volume groups found
[root lab-dm-cn4 ~]# vgdisplay
  No volume groups found
[root lab-dm-cn4 ~]# pvdisplay
[root lab-dm-cn4 ~]# ls -l /dev/mapper
total 0
crw-rw----. 1 root root 10, 236 Jan 11 11:34 control
[root lab-dm-cn4 ~]# 
[root lab-dm-cn4 ~]# 
[root lab-dm-cn4 ~]# 
[root lab-dm-cn4 ~]# #constants
[root lab-dm-cn4 ~]# let ONEMEG=1024*2; #translated to sectors
[root lab-dm-cn4 ~]# 
[root lab-dm-cn4 ~]# #user defined
[root lab-dm-cn4 ~]# DMSETUP=/home/jagan/src/LVM2.2.02.88/tools/dmsetup
[root lab-dm-cn4 ~]# LUN=1
[root lab-dm-cn4 ~]# META_DEVICE_START_SECTOR=0 #on $DEVICE_PATH this is where meta data starts
[root lab-dm-cn4 ~]# DEVICE_PATH=/dev/ram
[root lab-dm-cn4 ~]# let META_DEV_SIZE=8*$ONEMEG; #8M translated to sectors
[root lab-dm-cn4 ~]# let DATA_DEV_SIZE=3*1024*$ONEMEG; #3G translated to sectors
[root lab-dm-cn4 ~]# let PAGESIZE=8*$ONEMEG #page size is 8M in sectors
[root lab-dm-cn4 ~]# let DD_BS=4*1024 # dd block size = 4k  bytes
[root lab-dm-cn4 ~]# 
[root lab-dm-cn4 ~]# #system defined
[root lab-dm-cn4 ~]# let DATA_DEVICE_START_SECTOR=$META_DEVICE_START_SECTOR+$META_DEV_SIZE
[root lab-dm-cn4 ~]# let PAGES=$DATA_DEV_SIZE/$PAGESIZE
[root lab-dm-cn4 ~]# let SEEK=$PAGESIZE*512/$DD_BS # how many blocks we need to seek in dd to go to next page
[root lab-dm-cn4 ~]# let BLOCKS=$DATA_DEV_SIZE*512/$DD_BS
[root lab-dm-cn4 ~]# 
[root lab-dm-cn4 ~]# echo $DMSETUP LUNID=$LUN DEVICE=$DEVICE_PATH DEVICE_START=$META_DEVICE_START_SECTOR META_DEV_SIZE=$META_DEV_SIZE DATA_DEV_SIZE=$DATA_DEV_SIZE  PAGESIZE=$PAGESIZE DD_BS=$DD_BS DATA_DEVICE_START_SECTOR=$DATA_DEVICE_START_SECTOR PAGES=$PAGES SEEK=$SEEK BLOCKS=$BLOCKS
/home/jagan/src/LVM2.2.02.88/tools/dmsetup LUNID=1 DEVICE=/dev/ram DEVICE_START=0 META_DEV_SIZE=16384 DATA_DEV_SIZE=6291456 PAGESIZE=16384 DD_BS=4096 DATA_DEVICE_START_SECTOR=16384 PAGES=384 SEEK=2048 BLOCKS=786432
[root lab-dm-cn4 ~]# 
[root lab-dm-cn4 ~]# echo 0 $META_DEV_SIZE linear $DEVICE_PATH $META_DEVICE_START_SECTOR | $DMSETUP create tp_meta_disk$LUN
[root lab-dm-cn4 ~]# 
[root lab-dm-cn4 ~]# #zero out first 4k of meta data disk
[root lab-dm-cn4 ~]# dd if=/dev/zero of=/dev/mapper/tp_meta_disk$LUN bs=4096 count=1
1+0 records in
1+0 records out
4096 bytes (4.1 kB) copied, 3.9319e-05 s, 104 MB/s
[root lab-dm-cn4 ~]# 
[root lab-dm-cn4 ~]# #create data disk
[root lab-dm-cn4 ~]# echo 0 $DATA_DEV_SIZE linear $DEVICE_PATH  $DATA_DEVICE_START_SECTOR | $DMSETUP create tp_data_disk$LUN
[root lab-dm-cn4 ~]# 
[root lab-dm-cn4 ~]# 
[root lab-dm-cn4 ~]# #create thin pool
[root lab-dm-cn4 ~]# $DMSETUP create pool$LUN --table "0 $DATA_DEV_SIZE thin-pool /dev/mapper/tp_meta_disk$LUN /dev/mapper/tp_data_disk$LUN  $PAGESIZE 0 " #1G page with 100 with no water mark
[root lab-dm-cn4 ~]# 
[root lab-dm-cn4 ~]# #verify that 0 pages are allocated
[root lab-dm-cn4 ~]# $DMSETUP status pool$LUN
0 6291456 thin-pool 0 9/2048 0/384 -
[root lab-dm-cn4 ~]# $DMSETUP message /dev/mapper/pool$LUN 0 "create_thin 0"
[root lab-dm-cn4 ~]# $DMSETUP create thin$LUN --table "0 $DATA_DEV_SIZE thin /dev/mapper/pool$LUN 0"
[root lab-dm-cn4 ~]# #by now no pages should be allocated
[root lab-dm-cn4 ~]# $DMSETUP status pool$LUN
0 6291456 thin-pool 0 10/2048 0/384 -
[root lab-dm-cn4 ~]# 
[root lab-dm-cn4 ~]# 
[root lab-dm-cn4 ~]# #touch all pages on thin provisoned data disk so that all pages are allocated
[root lab-dm-cn4 ~]# i=0;while [ $i -lt $PAGES ]; do let j=$i*$SEEK; echo $i; dd if=/dev/zero of=/dev/mapper/thin$LUN bs=4096 seek=$j count=1; let i=$i+1; done >&/dev/null
[root lab-dm-cn4 ~]# #by now all pages should be allocated
[root lab-dm-cn4 ~]# $DMSETUP status pool$LUN
0 6291456 thin-pool 0 18/2048 384/384 -
[root lab-dm-cn4 ~]# 
[root lab-dm-cn4 ~]# 
[root lab-dm-cn4 ~]# #note the time on thin device
[root lab-dm-cn4 ~]# dd of=/dev/null if=/dev/mapper/thin1 bs=4096 count=$BLOCKS
786432+0 records in
786432+0 records out
3221225472 bytes (3.2 GB) copied, 18.2166 s, 177 MB/s
[root lab-dm-cn4 ~]# 
[root lab-dm-cn4 ~]# 
[root lab-dm-cn4 ~]# #cleanup
[root lab-dm-cn4 ~]# $DMSETUP remove thin$LUN
[root lab-dm-cn4 ~]# $DMSETUP message /dev/mapper/pool$LUN 0 "delete 0"
[root lab-dm-cn4 ~]# $DMSETUP remove pool$LUN
[root lab-dm-cn4 ~]# $DMSETUP remove tp_data_disk$LUN
[root lab-dm-cn4 ~]# $DMSETUP remove tp_meta_disk$LUN

Attachment: lvm_thin_compare.sh
Description: Bourne shell script


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]