[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [libvirt] [PATCH RFC 0/8] qemu: Cache results of parsing qemu help output



On Sun, Mar 11, 2012 at 02:44:44PM -0400, Lee Schermerhorn wrote:
> Stracing libvirtd shows that the qemu driver is executing 2 different
> qemu binaries 3 times each to fetch the version, capabilities [supported
> devices], and cpu models each time VM state is queried.  E.g., [lines wrapped]:
> 
> 6471  17:15:26.561890 execve("/usr/bin/qemu",
> 	["/usr/bin/qemu", "-cpu", "?"], [/* 2 vars */]) = 0
> 6472  17:15:26.626668 execve("/usr/bin/qemu",
> 	["/usr/bin/qemu", "-help"], [/* 2 vars */]) = 0
> 6473  17:15:26.698104 execve("/usr/bin/qemu",
> 	["/usr/bin/qemu", "-device", "?", "-device", "pci-assign,?",
> 	"-device", "virtio-blk-pci,?"], [/* 2 vars */]) = 0
> 6484  17:15:27.267770 execve("/usr/bin/qemu-system-x86_64",
> 	["/usr/bin/qemu-system-x86_64", "-cpu", "?"],
> 	/* 2 vars */]) = 0
> 6492  17:15:27.333177 execve("/usr/bin/qemu-system-x86_64",
> 	["/usr/bin/qemu-system-x86_64", "-help"],
> 	[/* 2 vars */]) = 0
> 6496  17:15:27.402280 execve("/usr/bin/qemu-system-x86_64",
> 	["/usr/bin/qemu-system-x86_64", "-device", "?", "-device",
> 	"pci-assign,?", "-device", "virtio-blk-pci,?"],
> 	[/* 2 vars */]) = 0
> 
> ~1sec per libvirt api call.  Not a killer, but on a heavily loaded
> host -- several 10s of VMs -- a periodic query of all VM state, such
> as from a cloud compute manager, can take a couple of minutes to
> complete.
> 
> Because the qemu binaries on the host do not change all that often,
> the results of parsing the qemu help output from the exec's above
> can be cached.  The qemu driver already does some caching of
> capabilities, but it does not prevent the execs above.
> 
> This series is a work in progress.  I'm submitting it as an RFC because I
> saw Eric mention the frequent execing of qemu binaries and I have been
> working on this to eliminate the overhead shown above.
> 
> The series caches the parse results of:
> 
> + qemuCapsExtractVersionInfo
> + qemuCapsProbeMachineTypes
> + qemuCapsProbeCPUModels
> 
> by splitting these functions into two parts.  The existing function
> name fetches the cached parse results for the specified binary and returns
> them.  The other half, named "qemuCapsCacheX", where X is one of
> ExtractVersionInfo, ProbeMachineTypes, and ProbeCPUModels, exec's the
> emulator binary and caches the results.  The act of fetching the
> cached results will fill or refresh the cache as necessary in a new
> function qemuCapsCachedInfoGet().  A few auxilliary function have been
> added -- e.g., virCapabilitiesDupMachines() to duplicate a cached list
> of machine types and virBitmapDup() to duplicate cached capabilities
> flags.

  yes, thanks, in general that's a point we really need to improve,
thanks for going though this,

> The series does not attempt to integrate with nor remove the existing
> capabilities caching.  TBD.
> 
> The series was developed and tested in the context of the Ubuntu 11.04 natty
> libvirt_0.8.8-1ubuntu6.7 package using quilt to manage patches in the
> debian/patches directory.  In that context, it builds, passes all
> "make check" tests [under pbuilder] and some fairly heavy, overlapping VM
> launch tests where it does eliminate all but a few initial exec's of the
> various qemu* and kvm binaries.
> 
> The version here, rebased to libvirt-0.9.10, builds cleanly under mock on
> Fedora 16 in the context of a modified libvirt-0.9.10-1.fc16 source package.
> I.e., no errors and warning-for-warning compatible with build of the
> libvirt-0.9.10 fc16 srpm downloaded from libvirt.org.  I placed the modified
> spec file [applies the patches] and the build logs at:
> 
> 	http://downloads.linux.hp.com/~lts/Libvirt/
> 
> I have installed the patched libvirt on a fedora 16 system and successfully
> defined and launched a vm.  Testing in progress.  I'll place an annotated
> test log ont the site above when complete.
> 
> I also need to rebase atop the current mainline sources, but I wanted to get
> this series out for review to see if the overall approach would be acceptable.

  Sounds a good idea, but we would really prefer patches against git
head as that where development is really done. running make check and
make syntax-check should be done to the patches in that context,

  thanks in advance !

Daniel

-- 
Daniel Veillard      | libxml Gnome XML XSLT toolkit  http://xmlsoft.org/
daniel veillard com  | Rpmfind RPM search engine http://rpmfind.net/
http://veillard.com/ | virtualization library  http://libvirt.org/


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]