[Ovirt-devel] Re: Thoughts about taskomatic redesign

[Ian Main]

So I've been doing some thinking on this, and here's what I've come up with to date. As usual any input is appreciated.

I wanted to make sure we had some basic requirements we could discuss.  Taskomatic should:

- Execute many tasks in a timely manner (presumably via distributed/multi threaded setup)
- Execute tasks in the correct order.
- Implement transaction support (if we can/want it)
- Have good error reporting.
- Include authentication and encryption.
- It would be nice to be able to see the state of the queues in the WUI.
- Implement at least the following tasks: 
   - start/stop/create/destroy VMs
   - clear host functionality.  Destroy all VMs/pools.
   - Migrate vms.
   - Storage pool creation/destruction/management


Now, if we break it down the system into basic components, I'm thinking:

- Task producer: This is the WUI now but could also be some other script in the future.  Creates task and adds them to the queue.

- Task ordering system: At some point I think a single process needs to separate out the tasks and order them.  I think it would be useful to actually move them into separate independent queues that can be executed in parallel.  This would be done such that each action in a given queue would need to be done in order, but each queue will not have dependencies on anything in another queue and so they can be worked on in parallel.  While this could be a bottleneck I think that the logic required to order the tasks will not be onerous and should keep up with high loads.

- Task implementation system: This would take a given queue and implement the tasks within it, dispatching requests to hosts as needed.  Any errors occurring in this system will be reported and possibly we could implement rollback.

- Host/vm/pool State: In addition to the above, in order to implement queue ordering and determine for certain that a given task succeeded, we'll require host/vm/storage pool state information that is as up to date as possible.

So in terms of implementing this, a lot of it comes down to technology selection.

Queues could continue to be implemented in postgresql.  It would be nice however to have something that was event driven and did not require polling.  It is possible we could use a python stored procedure to alert the consumer but that is postgresql specific and may have its own problems.  We may also consider using qpid for this, as it can do 'durable' queues which are stored on disk and survive across reboots.  Somewhere however, it needs to have a complete view of all queues so it can keep track of things.

I think a single ruby process could be used to order the tasks and place them in per-thread/process queues.  If using a DB I think we could either migrate the entries to a new 'in-progress' table, or update the row with the ID of the process/thread and possibly the sequence number to be used in implementing the queue.  Again however, this is another polling point and another point where we could shift to qpid msg queues.  Actually we may be best off to get commands from the wui via qpid and then have the separate queues in the database as it would allow the WUI to easily display the status of the work qeueus.  Jobs could be marked as 'completed' rather than removed until the queue is complete.  Individual processes could be awoken using one of various methods.

One possibility is that the task implementers be a mix of ruby processes running on the wui and C or C++ applications running on the node that use qpid modeling to represent the host, VMs, and storage pools.  The managed node daemon would model the host/vm/storage pools as objects and the wui-side process could call higher level functions to get things done.  eg vm->start() kind of thing where the daemon would then implement this using the libvirt C api.  State information for the host, VMs and storage pools would also be made available to the wui side and could be subscribed to by the implementers and the task ordering system.

Alternatively at this point, we could stick with libvirt.  The only problems I see with this is the libvirt backport issue, the potentially long timeout issue (looks like amqp can specify timeouts, or one could go async..).  It would be nice to have up to date status information available from the managed nodes as well.  It would also give us the opportunity to clean up a good deal of the daemons we have currently implementing various things.

Anyway, that's what I'm thinking for now.  I left out a lot of details for the sake of brevity.  

Thoughts?

    Ian