[fedora-virt] slightly OT: Comparing apples with apples?

[Daniel P. Berrange]

On Fri, Apr 24, 2009 at 09:10:47AM +0100, Mark McLoughlin wrote:
> > The most common complain was that competing for the host's resources
> > was not fair, something along the line: Fedora starts first becasue
> > has no grub, grabs some critical resource and Ubuntu has to wait for
> > it before continuing.
> > 
> > So basically the question is: do you think there could be any reason
> > why such a test can be unfair to one of the VMs?
> 
> Okay, seriously - I think it's a reasonable experiment.
> 
> However, the fact that you have two guests competing for resources is
> always going to make people suspicious. It may well be a deterministic
> experiment, but you're always going to have a hard time convincing
> people of that.

This would be particularly true if the VMs were over-committing on
CPU resources. eg if you had a 2 cpu host, and gave both VMs 2 vcpus
each, then you'd have 4 vcpus total comparing for 2 pcpus. They
will also likely compete on disk I/O during boot and impact each
other that way. Which reminds me, that you need to be careful with
the QEMU disk caching modes. By default QEMU will use the host OS
disk cache, so if one of the VM's data was already in the host cache
and the other's wasn't, one would have an unfair I/O advtange.

I'd recommend running with cache=off for the -driver parameters to
ensure they a guarenteed to be using Direct IO,avoiding any cache
on the host OS.
 
> Personally, I'd do it by timing each VM on its own and comparing the
> boot times.

You've also got to be sure both VMs are being run with comparable
configs. eg, it'd be totally unfair to compare Fedora VM with IDE
disk and RTL8139 nic against a Ubuntu VM using virtio disk & net,
or vica-verca.


Daniel
-- 
|: Red Hat, Engineering, London   -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org  -o-  http://virt-manager.org  -o-  http://ovirt.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-  F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|