[katello-devel] Katello Backend Service Relationships and Architecture

[Lukas Zapletal]

On Mon, Dec 17, 2012 at 08:48:49AM -0500, Eric Helms wrote:
> * Lazy accessors vs. Remote resources
>     - Lazy accessors are the original method of defining a set of
> attributes in the glue layer that access the object only when an
> attribute is requested
>     - Remote resources encompass using a thin model to represent
> what the resource looks like in the backend service

I think this is a bit out of the orchestration code, but as you say it
goes hand in hand. IMHO lazy accessors is great idea and it works for
read only fields - it's very convenient to have the possibility to fetch
these without additional code. And it should go through our new
integration layer - whatever it is - as well as data changes.

In all architectures I have been working on we used to pass EVERYTHING
through the integration layer (the bus), so everything was logged in the
message store. Big advantage - when you are solving an issue, you are
very often interested what data does a backend engine return. And since
you have also read-only request-reply messages stored in the log, you
can see it immediately (and identify a bug in an orchestration
transformation where a value is changed improperly for example).

Please note integration and orchestration are two different things in
this architecture. Orchestration box lives in the integration box and
the only way to interact with orchestration is via integration (bus).
And all the parties involved need to use that integration layer too.
And there is audit, log, routing and other services that we can offer if
we stick with this rule - not from the day one maybe, but possibility to
have it is a big win.

It *may* look scary, but we do not want to implement full-blown ESB, we
just need to stick with basic attributes of ESB (Enterprise Service Bus)
to have all the possibilities, architecture and design patterns which
are possible for ESBs.

Typical ESB uses reliable messaging for inside communication between
components, have routing and orchestration engine inside to do
transformations and orchestration, have several other services like
audit or message store and adapters for integration with other systems
(backend services in our case). They can be web services, REST or just
messaging bridges.

Katello would be good candidate for direct communication via messages
while backend engines could be integrated via REST HTTP adapters for the
first phase.

ESB projects and products are usually too big for us, even "lightweight"
projects like Apache ServiceMix (FUSE now acquired by Red Hat). And that
is the reason why I'd prefer to build something light with the same
architecture.

> * Backend namespacing and proxied entities vs. top level Katello
> entities for every model
>     - Keeping backend resources and functionality relegated to the
> glue layer and lazy accessors/remote resource, and treating all
> entities as Katello entities that may use a backend service to do
> the heavy lifting
>     - Exposing proxied entities (those without an ActiveRecord
> model) to controllers and namespacing at each level of the
> application

If we are planning to change our integration and orchestration
architecture, I think ActiveResource will disappear, because we will
need to create a thin layer that would send message (either request or
just notification) through the integration layer. Only the integration
layer will be able to contact the backend services, either via REST HTTP
or via other means.

But this is very important point - integration layer should be the glue
point, the single point of contact for all parties. And this is
something I will need to insist for OrchestrationNG (ONG from now on).
If we agree this is the right way to do it, there will be no proxied
entities - you just ask for a resource state through integration layer
(or send change there).

The goal is to move from spaghetti integration to robust and centralized
integration. I think your pictures shows this very clearly that is is
the time to at least start designing change like this.

> * Glue layer interactions with base Katello model:  modules vs. classes
>    - Treating glue layer as modules that enhance the base Katello model
>    - Creating a glue layer class for each entity that functions as
> an observer of the base Katello model

Orchestration must be separate from our integration layer. Advantages
are described on the ONG wiki page - like testability, loose coupling
and other things. We need to totally separate this, it needs a surgery.

-- 
Later,

 Lukas "lzap" Zapletal
 #katello #systemengine