[katello-devel] Another reason for manifest-import to be asynchronous

[Mike McCune]

On 05/25/2012 06:52 AM, Brad Buckingham wrote:
> On 05/25/2012 09:06 AM, Ivan Nečas wrote:
>> On 05/25/2012 02:58 PM, Bryan Kearney wrote:
>>> On 05/24/2012 08:43 AM, Ivan Nečas wrote:
>>>> Hi,
>>>>
>>>> I've been working on z stream bug
>>>> https://bugzilla.redhat.com/show_bug.cgi?id=823890, the solution
>>>> (https://github.com/Katello/katello/pull/144) is to remote the products
>>>> (including repositories) that were removed from the newly-imported
>>>> manifest. This slows down of the manifest import - time is needed for
>>>> removing the repositories which is not big problem for unsynchronized
>>>> repos, but there might be significant change for repositories already
>>>> synchronized.
>>>>
>>>> I would like to get more ideas on this. Is this the right time to move
>>>> manifest import to asynchronous state.
>>>>
>>>> Also removing repositories (not only in this case) causes records about
>>>> their promotions in changeset to vanish. We probably need other
>>>> (softer)
>>>> mechanism for removing repositories that still leaves some marks that
>>>> the repository was there once.
>>>>
>>> The user can really not do much else while the import is going on.
>>> But.. i would be happy to add this to the backlog if the team wants to.
>> Yeah, but it's always a bit risky to let longer running tasks being
>> synchronous: in case of connection time-out the user is not properly
>> reported about the finish of the request.
>>
>> -- Ivan
> I am +1 for making it async, given that it is a really long running
> task.  If we do that, we'll need to make sure the UI is updated so that
> users are able to determine that an import is in progress, if they
> navigate away from and back to the import page.
>
> That said, I thought there was some mention in the process of import
> changing in the future which would make it much faster.  (e.g. Not
> importing repos that aren't going to be enabled).

yes, if we go this route and skip the pulp repo creation during import 
the actual transaction time is quite fast.  On a manifest I edited to 
only contain one repo the manifest imports in 2-3 seconds.

If we remove the pulp repo creation, the need to make this task async is 
much lessened but may still be helpful in the case that Lukas mentions 
above where we need to go and delete repositories that are no longer in 
the manifest.

The priority of work should be:

1) take out pulp repo creation during manifest import

2) then decide if we should make it async

Mike