[linux-lvm] LVM Read/Write speed <10% drive's normal speed

[André Gillibert]

Stephen Hamer <Stephen.Hamer at jhuapl.edu> wrote:
> Hello all,
> 
> I've been searching through the archives and the internet for a while
> now, and can't seem to find anything that helps me out. I hope you don't
> my posting this to both xen-users and linux-lvm simultaneously, but i
> figured it'd help keep the solution in one place... even though it'll be
> the same across two places... Anyways:
> 
> My setup is this:
>         Xen 3.3 hypervisor (from the ubuntu repositories)
>         Ubuntu 9.04 server
>         Kernel and module listed here:
> http://www.infohit.net/blog/post/running-xen-on-ubuntu-intrepid-and-jaunty.html
> 
> The dom0 and domUs in my Xen setup are all Ubuntu 9.04, using that
> kernel. (I'm not sure if the kernel has something to do with it, so i
> included it just in case.) All the domUs are set up to use LVM. The dom0
> uses a straight ext3 fs on /dev/sda1. The domUs use ext3 fs's within
> LVMs (using LVM2) set up on /dev/sda5.
> 
> I have noticed a massive file I/O problem on all of my domUs. While I
> can peak file operations around 100 MB/s within the dom0, I can't get
> anything more than 3 MB/s read OR write out of a file operation on the
> domUs. I am using values reported by "iotop" to make this distinction.
> I'm using both the deployment of a Jboss server and the "dd" command to
> benchmark this. The Jboss slowdown is how I found this. On the dom0, the
> server takes 56s to come up. On a domU, the same operation takes 15
> minutes.
> 
> It should also be noted that I attempted to change the filesystems to
> ext2, because I noticed that 'kjournald' was chugging away taking up
> most of the i/o percentage, but only writing at KB/s! Actually, when I
> removed the journaling is when the process slowed from 7 minutes to 15.
> Even without the kjournald there, the java process took forever to load
> and never wrote more than 3 MB/s.
> 
> I have messed around with the 'blockdev' and 'hdparm' commands (the
> second of which doesn't even interact with any partition on my computer,
> lvm or no, instead simply failing with an "input/output error" message),
> but have had no performance increases after messing around with the read
> ahead speed of the devices. I also tried the "lvchange -r" command to
> set the readahead higher. I think there's something else missing here,
> though.
> 
> I'm fairly fresh to xen, this being only my second installation, and I'm
> extremely fresh to lvm. I've played with what seems like every option
> within "xm" and the 'lv*' commands. The LVMs themselves I haven't played
> with all too much yet, as I don't want to nuke a certain domU before it
> needs to be used heavily tomorrow. I've already killed a test domU that
> I brought up just for this.
> 
> -Stephen

Are you sure the file systems in DomUs are not mounted with the sync option, either in their fstab entry or in their default mount options as set with tune2fs? That could explain poor write performance, but not poor read performance. :|
When using dd to create a file, does the block size change performances significantly?

i.e. compare:
dd if=/dev/zero of=testfile bs=1M count=10
to:
dd if=/dev/zero of=testfile bs=1K count=10240

You should individually test the performance (in a test DomU) of the file system on the LVM partition in the DomU, the performance of the LVM logical volume and the performance of the physical volume containing the LVM logical volume, so that you see which layer is responsible.

That's general advice, but I hope it may help.

-- 
André Gillibert