[linux-lvm] Re: [Xen-users] LVM Read/Write speed <10% drive's normal speed: SOLVED

[Stephen Hamer <Stephen Hamer jhuapl edu>]

WOW. Thank you all for the overwhelming response!!! I'm going to answer you all in order of who replied first. 
(NOTE: the solution is in the 2nd section. Thanks Oliver!)
-Stephen



==================================
On Thu, 2009-09-17 at 03:41 -0400, Robert Dunkley wrote: 
>     >Hi Stephen,
>     >Can you please post your DomU config file to the list. 
>     >Rob

Sure, without any of the comments, here it is (without the true names or IPs, of course):

kernel      = '/boot/vmlinuz-2.6.26-2-xen-686'
ramdisk     = '/boot/initrd.img-2.6.26-2-xen-686'
memory      = '1024'
root        = '/dev/hda2 ro'
disk        = [ 'phy:/dev/xenvg/testVM1-swap,hda1,w',
                  'phy:/dev/xenvg/testVM1-root,hda2,w', ]

name        = 'testVM1'

vif = [ 'ip=192.168.0.100' ]


on_reboot   = 'restart'
on_crash    = 'restart'

# Needs to be inserted to allow console to attach to VM
extra="console=hvc0 xencons=tty"


This was generated by xen-tools (after a little modification to xen-tools xm.tmpl). I like how easy xen-tools is to expand upon. For example, I used 'xen-update-image' as a model to create my own 'xen-image-cmd' which can pass a single command to multiple offline domUs. Works just like 'xen-update-image', except for the fact that you can pass ANY command that the domU can execute AND you can force the output to be redirected to a file INSIDE the chrooted image. I was sort of proud of myself for that scripting voodoo. =D


==================================
On Thu, 2009-09-17 at 03:45 -0400, Olivier B. wrote:
> >Hi,
>
> >what mount options do you use on the DomU ?
> >Under Debian there is xen-tools example with the "sync" mount options : 
> >if you use it, writes will be really slow.
>
> >Olivier

THIS is where the money is. Once I removed the 'sync' option from the fstab for my root partition, the I/O speed sailed. Makes sense, really. This got the VM's performance to actually BEAT the old bare metal machine that we were running this server on before. 
Thanks for your help!


==================================
On Thu, 2009-09-17 at 06:21 -0400, Bryn M. Reeves wrote: 
> What is the configuration of the guest's storage? I.e. what type of virt
> driver are you using to expose the dom0 storage to the domUs? This can
> have a massive impact on the performance levels you can achieve.
>
> You should also test the performance of these underlying devices in the
> domUs - i.e. benchmark read (and if possible write) performance of
> the /dev/sd*, /dev/hd* or /dev/xvd* devices seen in domU (or whatever
> other type of virtual disks you are using).
>
> Regards,
> Bryn.

I'm not PRECISELY sure what you mean here, but I think you're referring to the xen driver used to pass the storage in as a block dev. to the domU. If that is what you mean, I used the 'phy:' command. The performance of the /dev/hda* inside the domU is what I was mentioning trying to benchmark simplistically with 'dd'. 
- A 'dd' command in my dom0 on the true physical /dev/sda1 got 100 MB/s. 
- Before I turned off 'sync', a 'dd' command on /dev/hda2 inside my domU (physically a 10GB LVM on the physical device /dev/sda5) got 3 MB/s. 
I noted later that running a DD command directly to another LVM on that same VG as the others got comparable performance to the 100 MB/s. That was my first indicator that the domUs were doing something nasty. 
- After I turned off 'sync', a 'dd' command on that same /dev/hda2 inside the same domU got 100 MB/s!


==================================
On Thu, 2009-09-17 at 10:59 -0400, Jeff Sturm wrote: 
>     Your domU's--are they paravirt, or HVM?
I believe they're paravirtualized, unless I'm doing something silly. They're running off the modified kernel, and I'm not doing any hardware forwarding magic. See config file at the top.

==================================
On Thu, 2009-09-17 at 11:16 -0400, Fajar A. Nugraha wrote:
> I suggest you try using official xen.org's 2.6.18 kernel, or other
> vendor-supported 2.6.18 kernel-xen (like RHEL5's kernel-xen, or even
> Etch's kernel) for both domU and dom0 and see if the problem persists.
I've noticed a couple of quirks in the startup sequence of my domUs, I figured it had something interesting to do with the kernel. I'll look into using one of those kernels. Thanks!