[augeas-devel] Match operations over long file too slow

[Štěpán Balážik]

Hi again,

so I was using Augeas version 1.4.0 from Ubuntu repositories. Installing 
latest stable version 1.8.0 made the process (literally) 100x faster.

I am sorry I was bothering you all.

Štěpán


On 14.6.2017 14:40, Josef Reidinger wrote:
> On Wed, 14 Jun 2017 14:13:31 +0200
> "Štěpán Balážik" <stepan.balazik at nic.cz> wrote:
>
>> Hi,
>>
>> we want to use Augeas to programmatically parse and generate tests for
>> Deckard <https://gitlab.labs.nic.cz/knot/deckard>[1] (our test harness
>> for DNS resolvers).
>>
>> I wrote a lens
>> <https://gitlab.labs.nic.cz/knot/deckard/blob/augeas_wip/pydnstest/deckard.aug>[2]
>> which works fine but on longer config files such as this one
>> <https://gitlab.labs.nic.cz/knot/deckard/blob/augeas_wip/sets/resolver/iter_badraw.rpl>[3]
>> the match operation on /scenario/step nodes takes about 3 ms per call on
>> my machine using both augtool and the python binding. This sums up to
>> over 10 s for the whole file which too slow for our purposes.
>>
>> We have no idea what is the cause of this slowness. Are Augeas matches
>> always so slow on "shallow and wide" trees?
>>
>> Thanks in advance for any help.
>> Štěpán Balážik @ CZ.NIC
>>
>> [1] https://gitlab.labs.nic.cz/knot/deckard
>> [2]
>> https://gitlab.labs.nic.cz/knot/deckard/blob/augeas_wip/pydnstest/deckard.aug
>> [3]
>> https://gitlab.labs.nic.cz/knot/deckard/blob/augeas_wip/sets/resolver/iter_badraw.rpl
>>
> Hi Stepan,
> we face similar issue when parsing /etc/hosts that are auto-filled by some blocking software ( so around 10k entries there ). In the end solution for us is to introduce caching of match results and creating own map. We use ruby, but I think it is so similar to python, that you get idea.[1]
> If you need any explanation, do not hesitate to ask. Speed up in our case is approximately half of time spend at parsing.
>
> Josef
>
> [1] https://github.com/config-files-api/config_files_api/blob/master/lib/cfa/augeas_parser/keys_cache.rb