[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [Pulp-list] Coordinator Usage for Repositories

On 03/16/2012 12:18 AM, Jay Dobies wrote:
That has the potential to be really annoying if you have a lot of
scheduled syncs. You could be in the middle of some admin operations
when one or more scheduled syncs kicks in and basically takes over all
of Pulp's processing capabilities. That may be alleviated by suggested
usage of off-hours synccing.

PulpDist definitely needs to be able to configure new repos while sync operations are running in the background. (For the initial copy of big trees or when dealing with slow remote servers, PulpDist jobs can take a long time to complete)

And that's just within repo operations. To be delayed from creating a
new repo or updating one because I've triggered a handful of consumer
operations is also a rough user experience (I say delayed meaning the
coordinator isn't the one blocking it, just the sheer lack of open
threads in the task pool).

That said, I haven't fully thought through what a multiple queue setup
would look like. It probably gets really tricky very fast. I just want
to make sure we understand how that user experience is going to change
now that many more things that previously didn't are now reliant on an
open thread in the task pool.

Perhaps it would make sense to have a two-tier task pool? It's similar to your creative math idea, but a bit more formalised.

The basic concept would be to have a "reserved" pool within the larger task pool. Say you had a total task queue size of 10 points, and a reserved weight of 4 points.

Then "background" tasks (like sync jobs or consumer operations) would only be handed over to a task thread if the total weight of the background tasks in progress would remain under the lower non-reserved threshold of 6. "Foreground" tasks (like repo creation and configuration updates) would be allowed up to the full weight of 10 (counting both foreground and background tasks).

That way, the administrators of a particular Pulp installation would have direct control over how they wanted to balance the responsiveness for repo manipulation commands vs content distribution commands.


Nick Coghlan
Red Hat Engineering Operations, Brisbane

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]