[katello-devel] Foreman integration and Elastic search

[Mike McCune]

On 11/22/2012 01:08 AM, Tomas Strachota wrote:
> On 11/21/2012 04:22 PM, Dmitri Dolguikh wrote:
>> On 21/11/12 03:13 PM, Bryan Kearney wrote:
>>> On 11/20/2012 02:28 PM, Lukas Zapletal wrote:
>>>> On Tue, Nov 20, 2012 at 05:04:54PM +0200, Ohad Levy wrote:
>>>>> On 11/20/2012 04:05 PM, Dmitri Dolguikh wrote:
>>>>>> We are about to start using Elastic search for lookups on Foreman data
>>>>>> on Katello side. Up until this moment Katello didn't persist any
>>>>>> Foreman-related data at all.
>>>>>>
>>>>>> This introduces what is essentially a long-term cache, but without any
>>>>>> form of cache control or cache validation. Another issue is this
>>>>>> breaks
>>>>>> Foreman modularization, with Katello now being responsible for some of
>>>>>> the Foreman data.
>>>>>>
>>>>>> I think this functionality should be moved to Foreman (or Foreman's
>>>>>> existing functionality improved) and exposed via introduction of
>>>>>> search
>>>>>> functionality in Foreman's API.
>>>>>>
>>>>>>
>>>>>> Thoughts?
>>>>> If we introduce ES updates as callback filters would that work?
>>>>>
>>>>> if that works, it sounds like a plan for foreman rails engine /
>>>>> plugin which adds more callbacks?
>>>>
>>>> I like this more than integrating ES with Foreman in Katello, but please
>>>> let's do NOT use HTTP/REST for that and leverage messaging for this.
>>>> This is typical case for fire-and-forget pattern which is much more
>>>> reliable than playing around with http requests.
>>>>
>>> Dmitri, is the issue here that we have no object inthe katello db?
>>
>> One of the issues, yes. The larger problem is that we are losing a clear
>> separation between Katello and Foreman (if only on UI side). It appears
>> that we might need tighter integration (Justin pointed out that
>> combining elastic search results in the model I proposed would be hard)
>> -- perhaps we should look into adopting Foreman-as-rails-engine
>> approach: integration would be simpler (faster too, I'd wager).
>>
>> -d
>
> Pardon my ignorance but I'm afraid I still didn't understand the issue.
> How is it different from what we do with records in Pulp? Some models
> are also stored completely on Pulp's side without having a bit in
> Katello db but still we index them with ES.
>

I had the exact same question .. why is what we are doing with Foreman's 
searchable data any different than what we have done in the past with 
Pulp's?  We decided some time ago that ES is, in fact, a long-term cache 
of data we need to make searchable and display very quickly in the UI 
using elasticsearch.

I'm also curious what it really means to have foreman-as-rails-engine as 
an alternate approach. Do you have something we could look at that 
explains how that would work?

Mike
-- 
Mike McCune
mmccune AT redhat.com
Red Hat Engineering       | Portland, OR
Systems Management        | 650-254-4248