[augeas-devel] improving performance of aug_get() and aug_match() with large datasets

[David Lutterkort]

On Mon, Oct 5, 2015 at 6:54 AM, Laine Stump <laine at redhat.com> wrote:

>
> augeas netcf  libvirt list   dump    virt-manager
> ------ -----  ------- ------ ------- ------------
> 1.4.0  0.2.8  1.2.20  1:37.6 13:46.6 15:37
> upstrm 0.2.8  1.2.20  1:04.7 07:34.8 08:41
> upstrm upstrm 1.2.20  0:03.7 06:40.3 06:46
> upstrm upstrm upstrm  0:02.0 06:39.5 06:39
>
> (the upstream change in netcf is to call aug_load() only 1/sec, and to
> libvirt is to avoid calling ncf_if_mac_string() multiple times for each
> interface during iface-list).
>
> In the case of virt-manager, the application becomes responsive after ~
> the "list" time, then stops using CPU at the end of the "virt-manager"
> time, so the netcf change has a big effect on the amount of time until
> virt-manager is usable, and the augeas change has an even bigger effect on
> how long it takes for the system to settle down.
>
>
> So just as an experiment, I tried removing the most complicated term:
>
>   "MASTER = ../*[BRIDGE = 'br1']/DEVICE"
>
> from the search expression. When I did this, the time for a "virsh
> iface-dumpxml" of all interfaces dropped from 6min39.5sec down to 15.3
> seconds!
>
> upstrm upstrm upstrm  0:02.1 0:15.3 00:17
>
> Any bright ideas on how to either make that search term execute faster, or
> alternately replace it with something simpler? (A first thought is that
> maybe it would be faster to do a two-staged search where we first look for
> everything with BRIDGE=="br1", then retrieve the DEVICE of all those
> matches, then search for MASTER==[any of the DEVICEs found in the first
> step]. Or maybe not; hard for me to say without trying.) *EDIT - David
> already replied showing how to do exactly this and indicating another
> drastic improvement, so now I'll be implementing his suggested changes in
> netcf)


This all looks very encouraging - are these times satisfactory for users or
do we need to look more into speeding them up further ?

Just as a heads-up: I posted a PR for Augeas (PR 305) that should make
testing different queries easier. It adds a '--timing' flag to augtool
which makes it print how long a command took after each command.

David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/augeas-devel/attachments/20151005/921f91ac/attachment.htm>