[katello-devel] Foreman integration and Elastic search

[Eric Helms]

On 11/27/2012 03:46 AM, Petr Chalupa wrote:
>
>
> On 26.11.12 17:42, Dmitri Dolguikh wrote:
>> On 26/11/12 04:15 PM, Mike McCune wrote:
>>> On 11/22/2012 01:08 AM, Tomas Strachota wrote:
>>>> On 11/21/2012 04:22 PM, Dmitri Dolguikh wrote:
>>>>> On 21/11/12 03:13 PM, Bryan Kearney wrote:
>>>>>> On 11/20/2012 02:28 PM, Lukas Zapletal wrote:
>>>>>>> On Tue, Nov 20, 2012 at 05:04:54PM +0200, Ohad Levy wrote:
>>>>>>>> On 11/20/2012 04:05 PM, Dmitri Dolguikh wrote:
>>>>>>>>> We are about to start using Elastic search for lookups on
>>>>>>>>> Foreman data
>>>>>>>>> on Katello side. Up until this moment Katello didn't persist any
>>>>>>>>> Foreman-related data at all.
>>>>>>>>>
>>>>>>>>> This introduces what is essentially a long-term cache, but
>>>>>>>>> without any
>>>>>>>>> form of cache control or cache validation. Another issue is this
>>>>>>>>> breaks
>>>>>>>>> Foreman modularization, with Katello now being responsible for
>>>>>>>>> some of
>>>>>>>>> the Foreman data.
>>>>>>>>>
>>>>>>>>> I think this functionality should be moved to Foreman (or 
>>>>>>>>> Foreman's
>>>>>>>>> existing functionality improved) and exposed via introduction of
>>>>>>>>> search
>>>>>>>>> functionality in Foreman's API.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thoughts?
>>>>>>>> If we introduce ES updates as callback filters would that work?
>>>>>>>>
>>>>>>>> if that works, it sounds like a plan for foreman rails engine /
>>>>>>>> plugin which adds more callbacks?
>>>>>>>
>>>>>>> I like this more than integrating ES with Foreman in Katello, but
>>>>>>> please
>>>>>>> let's do NOT use HTTP/REST for that and leverage messaging for 
>>>>>>> this.
>>>>>>> This is typical case for fire-and-forget pattern which is much more
>>>>>>> reliable than playing around with http requests.
>>>>>>>
>>>>>> Dmitri, is the issue here that we have no object inthe katello db?
>>>>>
>>>>> One of the issues, yes. The larger problem is that we are losing a
>>>>> clear
>>>>> separation between Katello and Foreman (if only on UI side). It 
>>>>> appears
>>>>> that we might need tighter integration (Justin pointed out that
>>>>> combining elastic search results in the model I proposed would be 
>>>>> hard)
>>>>> -- perhaps we should look into adopting Foreman-as-rails-engine
>>>>> approach: integration would be simpler (faster too, I'd wager).
>>>>>
>>>>> -d
>>>>
>>>> Pardon my ignorance but I'm afraid I still didn't understand the 
>>>> issue.
>>>> How is it different from what we do with records in Pulp? Some models
>>>> are also stored completely on Pulp's side without having a bit in
>>>> Katello db but still we index them with ES.
>>>>
>>>
>>> I had the exact same question .. why is what we are doing with
>>> Foreman's searchable data any different than what we have done in the
>>> past with Pulp's?  We decided some time ago that ES is, in fact, a
>>> long-term cache of data we need to make searchable and display very
>>> quickly in the UI using elasticsearch.
>> Our integration with foreman is such that we have no foreman-related
>> data in Katello local db (unlike with pulp) and are essentially caching
>> remote system data ignoring any cache-controls mandated by that remote
>> system. I think this breaks ES usage pattern that was established
>> earlier, and we should revise how ES is being used/interfaced with.
>>
>>> I'm also curious what it really means to have foreman-as-rails-engine
>>> as an alternate approach. Do you have something we could look at that
>>> explains how that would work?
>>>
>> http://guides.rubyonrails.org/engines.html. In short, rails engine is a
>> way to embed a slice of functionality into another rails application. If
>> I understood this correctly, foreman team has actually started looking
>> into this.
>
> I think to use Foreman as a rails engine is not easy thing to do. 
> Rails application are typically not developed as embeddable - they 
> pollute Ruby global namespace a lot. There would be a lot of 
> collisions in classes (e.g. User in Katello and Foreman), in database 
> tables and other problems.
>
> I would rather not go this way.
>
> Petr
>
This can be alleviated by properly name-spacing everything in your 
engine.  I recently ported Alchemy (which used to be ConvergeUI) to a 
Rails engine and after my first crack at it found that name-spacing was 
a must to prevent collisions and to provide clarity about where a 
component is originating from (from the original application or the engine).

When it comes to engines the question is mountable vs. non-mountable.  
Mountable means that you 'mount' or 'bolt-on' the engine beside your 
main application.  Mountable engines are designed to add functionality, 
not integrate functionality.  Non-mountable engines are designed to 
provide a focused set of functionality that an application can hook into 
it's own infrastructure.  For example, Alchemy is a non-mountable engine 
as it will provide assets, helpers, views, etc. to be consumed and used 
the way Katello sees fit.  Where as something likes an authorization 
engine could be mountable since it would control and handle all the 
Users/Roles/Permissions for you.

All that being said, these sort of debates make me wonder if we lack 
clarity (a mission statement of sorts) in what Katello actually is. My 
understanding of Katello:

Katello seeks to bring together best-in-breed backend tools to provide a 
higher level of abstraction and interaction to allow users a more 
seamless and unified systems management platform.

Ok, so maybe that sounds like some market speak, but the point is, we 
have always held that the backend engine is a means to an end and not a 
first-class citizen.  We brought in ES to handle the search queries and 
scenarios that we needed to perform the actions and interactions we 
present to our users.  As Justin pointed out, we index more than just 
our own data to provide a fast, fluid, consistent query interface to our 
users.  So let me ask, given that we have always treated our backend 
services as just that, services:

Why would  we suddenly deviate from our architecture? Simply because 
Foreman is a Rails project and CAN be mounted as an engine? Or because 
drawing Foreman in as a first-class mounted engine or as a non-mounted 
engine provides some paradigm shift that we need or want?

I think we need to answer the fundamental question, understand the 
implications and go forward making the decision decisions and solutions 
that achieve our desired integration architecture from the point of view 
of Katello.

-Eric