[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [augeas-devel] improving performance of aug_get() and aug_match() with large datasets

On Mon, Oct 5, 2015 at 6:54 AM, Laine Stump <laine redhat com> wrote:

augeas netcf  libvirt list   dump    virt-manager
------ -----  ------- ------ ------- ------------
1.4.0  0.2.8  1.2.20  1:37.6 13:46.6 15:37
upstrm 0.2.8  1.2.20  1:04.7 07:34.8 08:41
upstrm upstrm 1.2.20  0:03.7 06:40.3 06:46
upstrm upstrm upstrm  0:02.0 06:39.5 06:39

(the upstream change in netcf is to call aug_load() only 1/sec, and to libvirt is to avoid calling ncf_if_mac_string() multiple times for each interface during iface-list).

In the case of virt-manager, the application becomes responsive after ~ the "list" time, then stops using CPU at the end of the "virt-manager" time, so the netcf change has a big effect on the amount of time until virt-manager is usable, and the augeas change has an even bigger effect on how long it takes for the system to settle down.

So just as an experiment, I tried removing the most complicated term:

  "MASTER = ../*[BRIDGE = 'br1']/DEVICE"

from the search _expression_. When I did this, the time for a "virsh iface-dumpxml" of all interfaces dropped from 6min39.5sec down to 15.3 seconds!

upstrm upstrm upstrm  0:02.1 0:15.3 00:17

Any bright ideas on how to either make that search term execute faster, or alternately replace it with something simpler? (A first thought is that maybe it would be faster to do a two-staged search where we first look for everything with BRIDGE=="br1", then retrieve the DEVICE of all those matches, then search for MASTER==[any of the DEVICEs found in the first step]. Or maybe not; hard for me to say without trying.) *EDIT - David already replied showing how to do exactly this and indicating another drastic improvement, so now I'll be implementing his suggested changes in netcf)

This all looks very encouraging - are these times satisfactory for users or do we need to look more into speeding them up further ?

Just as a heads-up: I posted a PR for Augeas (PR 305) that should make testing different queries easier. It adds a '--timing' flag to augtool which makes it print how long a command took after each command.


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]