[augeas-devel] Improving Augeas' idempotence in Puppet

[Dominic Cleal]

On 06/04/12 09:26, Raphaël Pinson wrote:
> As of today, the way the Augeas provider in Puppet manages idempotence
> can be somehow problematic. The `onlyif` statements can only be
> evaluated on the client, so all Augeas resources are generally run on
> the client, and whether to run them cannot be easily determined in the
> catalog, unless it relies on other resources as dependencies or on
> custom facts (possibly using the Augeas ruby library).

I agree that idempotency with the Augeas provider is a big problem for
users today.  I'm not sure that moving the problem completely to the
master is the right solution though.

If the resource is removed from the catalog if you know it's already in
sync (via the idea below) then reports, no-op mode etc will no longer
contain any information about it.  Dashboard, the Foreman,
compliance/auditing functionality will no longer show the resource state.

Some users also like the catalog to be more permanent - a feature we're
seeing in the Puppet "Telly" faces, where a catalog can more easily be
cached and re-run at a later date.  Ensuring the state is verified on
the client rather than the master makes more sense here.

> Here is a proposition to improve this, by making the Augeas tree
> available on the Puppet server during catalog compilation.
> 
> A new aug_to_xml() API call has been added in Augeas 0.10.0 allowing
> to export the whole (or part of the) Augeas tree as an XML document.
> 
> The idea is the following:
> 
>   * Before compiling the catalog, make a request to retrieve the
> Augeas tree as XML (using facter or another system);

Ken Barber's been working on adding structured data support to Facter,
so potentially it could even be transported through this:
http://projects.puppetlabs.com/issues/4561

I'd also wonder about the security and sense of sending the contents of
every file up as a fact though to be stored, as the dump-xml output of
the 741 parsed files here is about 6.7MB.

>   * On the server, reinject the XML in a local Augeas instance (using
> a yet-to-code aug_from_xml() call);
>   * Evaluate the onlyif statements (or a new kind of statement) using
> this local copy of the client's Augeas tree;

This is the bit that gets me, I'm not sure that this is an improvement
over doing it on the client.  You'll still need a way to write the
evaluation that's equivalent to the "onlyif" statements in use today.

The only way I see it being an improvement was if you then generated the
Augeas commands using ERB or similar, based on the tree sent through
Facter.  You could probably solve most idempotency problems like this,
but it'd be rather complex to write.

I don't have any solutions at the moment for idempotency.  I've been
thinking about creating new aug_srun/augtool commands representing more
complex operations that could be used in Puppet - things you'd normally
do in a proper language via the APIs trivially.  David's also suggested
this in the past (Puppet issue #2696) to create conditionals in the
provider language (complexity I think we should avoid).

Christos Bountalis created a config "merging" implementation last year
for GSoC which got me wondering if the Puppet resource should instead
contain one tree that's merged into the file.  Unfortunately the
implementation was only designed for simple filetypes and couldn't merge
complex trees (I think the data on *how* to merge complex trees is
missing in the lens definition).

I saw a comment this week from David on R.I.Pienaar's puppet-concat blog
entry which shows an idea to write the tree out in the resource itself
and then have the provider create it.  I think this would be even easier
now with hash support in the Puppet DSL.

http://www.devco.net/archives/2010/02/19/building_files_from_fragments_with_puppet.php/comment-page-1#comment-5096

Regards,

-- 
Dominic Cleal
Red Hat Consulting
m: +44 (0)7817 878113