[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [augeas-devel] improving performance of aug_get() and aug_match() with large datasets



On Sat, Oct 3, 2015 at 3:05 PM, David Lutterkort <lutter watzmann net> wrote:
On Thu, Oct 1, 2015 at 11:44 AM, Laine Stump <laine redhat com> wrote:

But 13 (or even 8) minutes is still a very long time, so I played around a bit in gdb and found that most of the time now seems to be spent in one call to aug_match():


  r = aug_match(aug, path, "/files/etc/sysconfig/network-scripts/*[ DEVICE = 'br1' or BRIDGE = 'br1' or MASTER = 'br1' or MASTER = ../*[BRIDGE = 'br1']/DEVICE ]/DEVICE");

Whoever wrote that code must have thought they were incredibly clever with this query ;)

There's a few ways in which I think this can be sped up: for one, rather than use 'or', we can build an intermediate nodeset for the first three nodesets by matching

(1) /files/etc/sysconfig/network-scripts/*[(DEVICE|BRIDGE|MASTER) = 'br1']/DEVICE

The last term in that 'or' is very expensive since it constitutes a nested loop, with "/files/etc/sysconfig/network-scripts/*" being the outer loop ("for each ifcfg file") and "../*[BRIDGE = 'br1']/DEVICE" being the inner loop ("for each ifcfg file see if it is a BRIDGE and return its DEVICE"). That can be made a little more targetted by using

(2) /files/etc/sysconfig/network-scripts/*/MASTER[ . = ../*[BRIDGE = 'br1']/DEVICE ]

so that we only trigger the inner loop for ifcfg files that actually have a MASTER entry. This helps if you don't have bonds - I suspect, if there are any bonds on the system, the query will still be very expensive.

I shouldn't computer on weekends: this part is total nonsense. It needs to be

 /files/etc/sysconfig/network-scripts/*[MASTER][MASTER = ../*[BRIDGE = 'brvlan42']/DEVICE ]

The first '[MASTER]' makes sure we only run the second predicate "[MASTER = ../*[BRIDGE = 'brvlan42']/DEVICE ]" with the expensive loop on files that actually have a MASTER entry.

Attached updated augcmds.txt and output - the timings are roughly in the same ballpark as mentioned in my previous email.


citron:augeas (master)>./src/try -e -r /var/tmp/bridges-for-vlans/root/
augtool> # Original
augtool> match /files/etc/sysconfig/network-scripts/*[ DEVICE = 'brvlan42' or BRIDGE = 'brvlan42' or MASTER = 'brvlan42' or MASTER = ../*[BRIDGE = 'brvlan42']/DEVICE ]/DEVICE
aug_match(/files/etc/sysconfig/network-scripts/*[ DEVICE = 'brvlan42' or BRIDGE = 'brvlan42' or MASTER = 'brvlan42' or MASTER = ../*[BRIDGE = 'brvlan42']/DEVICE ]/DEVICE) = 2
Time: 674ms
/files/etc/sysconfig/network-scripts/ifcfg-p14p1.42/DEVICE = p14p1.42
/files/etc/sysconfig/network-scripts/ifcfg-brvlan42/DEVICE = brvlan42
augtool> #
augtool> #
augtool> # Turn first three 'or' terms into an intermediate nodeset, and triggeraugtool> # the inner loop for MASTER only if there actually are bonds
augtool> match (/files/etc/sysconfig/network-scripts/*[(DEVICE|BRIDGE|MASTER) = 'brvlan42']|/files/etc/sysconfig/network-scripts/*[MASTER][MASTER = ../*[BRIDGE = 'brvlan42']/DEVICE ])/DEVICE
aug_match((/files/etc/sysconfig/network-scripts/*[(DEVICE|BRIDGE|MASTER) = 'brvlan42']|/files/etc/sysconfig/network-scripts/*[MASTER][MASTER = ../*[BRIDGE = 'brvlan42']/DEVICE ])/DEVICE) = 2
Time: 36ms
/files/etc/sysconfig/network-scripts/ifcfg-p14p1.42/DEVICE = p14p1.42
/files/etc/sysconfig/network-scripts/ifcfg-brvlan42/DEVICE = brvlan42
augtool> #
augtool> #
augtool> # Assuming we have two bonds on the system, using 'or'
augtool> match (/files/etc/sysconfig/network-scripts/*[(DEVICE|BRIDGE|MASTER) = 'brvlan42']|/files/etc/sysconfig/network-scripts/*[MASTER = 'bond0' or MASTER = 'bond1' ])/DEVICE
aug_match((/files/etc/sysconfig/network-scripts/*[(DEVICE|BRIDGE|MASTER) = 'brvlan42']|/files/etc/sysconfig/network-scripts/*[MASTER = 'bond0' or MASTER = 'bond1' ])/DEVICE) = 2
Time: 5ms
/files/etc/sysconfig/network-scripts/ifcfg-p14p1.42/DEVICE = p14p1.42
/files/etc/sysconfig/network-scripts/ifcfg-brvlan42/DEVICE = brvlan42
augtool> #
augtool> #
augtool> # Assuming there are no bonds on the system
augtool> match (/files/etc/sysconfig/network-scripts/*[(DEVICE|BRIDGE|MASTER) = 'brvlan42'])/DEVICE
aug_match((/files/etc/sysconfig/network-scripts/*[(DEVICE|BRIDGE|MASTER) = 'brvlan42'])/DEVICE) = 2
Time: 3ms
/files/etc/sysconfig/network-scripts/ifcfg-p14p1.42/DEVICE = p14p1.42
/files/etc/sysconfig/network-scripts/ifcfg-brvlan42/DEVICE = brvlan42
augtool>
# Original
match /files/etc/sysconfig/network-scripts/*[ DEVICE = 'brvlan42' or BRIDGE = 'brvlan42' or MASTER = 'brvlan42' or MASTER = ../*[BRIDGE = 'brvlan42']/DEVICE ]/DEVICE
#
#
# Turn first three 'or' terms into an intermediate nodeset, and trigger
# the inner loop for MASTER only if there actually are bonds
match (/files/etc/sysconfig/network-scripts/*[(DEVICE|BRIDGE|MASTER) = 'brvlan42']|/files/etc/sysconfig/network-scripts/*[MASTER][MASTER = ../*[BRIDGE = 'brvlan42']/DEVICE ])/DEVICE
#
#
# Assuming we have two bonds on the system, using 'or'
match (/files/etc/sysconfig/network-scripts/*[(DEVICE|BRIDGE|MASTER) = 'brvlan42']|/files/etc/sysconfig/network-scripts/*[MASTER = 'bond0' or MASTER = 'bond1' ])/DEVICE
#
#
# Assuming there are no bonds on the system
match (/files/etc/sysconfig/network-scripts/*[(DEVICE|BRIDGE|MASTER) = 'brvlan42'])/DEVICE

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]