[Pulp-list] Asynchronous Task Dispatching

[Jason L Connor]

On Thu, 2011-04-14 at 13:28 -0400, Jay Dobies wrote:
> You lost me after the 30 seconds part. How will the on demand
> dispatcher
> resolve handling situations where there are other tasks ahead of it in
> line?
> 
> So let's take a case of a single thread doing stuff. It just popped a
> new task off the queue, beginning the 30 second timer. Left on the
> queue
> are tasks B, C, and D.
> 
> During that 30 second timeframe, a request to sync a repo comes in.
> I'd
> expect it to be put on the queue to execute after D, but the way I
> read
> your comment it'll jump to the front of the line by being run by the
> on
> demand dispatcher.
> 

I think you're thinking of the queue as a fifo, when it's actually a
priority queue ordered by the tasks's scheduled_time field.

> I'm probably missing something though, so maybe some more explanation
> will help clear it up.
> 

Ok, so let's take your example. Let's assume that the tasking system is
setup to run several task concurrently. This means at the last time the
dispatcher woke up, task A was able to run, but the rest of the tasks'
scheduled_times were not yet up.

When the repo sync comes in (assuming it's not a scheduled sync) it'll
get "scheduled" to run immediately and enqueued. This will wake up the
dispatcher thread (the on-demand part) which will examine the tasks at
the head of the queue, find the repo sync, and kick it off. As well as
kick off any other tasks that have become ready.

> Also (again, I don't know how you're implementing it), based on the
> discussions in this thread it sounds like the queue of tasks will only
> exist in the database and not in memory. How is that going to affect
> the
> uniqueness check? Does that become a series of database retrievals or
> can we do that check atomically in the DB? 

The uniqueness check is start a tad naive, as it's basically pulling all
the tasks from the db and iterating over them. I have plans to convert
the uniqueness criteria into a db spec and pushing it completely into
the database. However, I need to pickle the arguments in order to do so
and haven't gotten around to doing that yet.

-- 
Jason L Connor
linear on freenode #pulp
http://pulpproject.org/
RHCE: 805010912355231
GPG Fingerprint: 2048R/CC4ED7C1
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: This is a digitally signed message part
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20110414/2c5d5a60/attachment.sig>