[libvirt] Applying for GSoC project 'Introducing job control to the storage driver' in libvirt

[Taowei Luo]

Hello, everyone.

I'm Taowei Luo. A participator in Google Summer of Code last year for
the project rewriting vbox driver in libvirt. I am so glad that my
project isn't found in this year's GSoC project list :)

Having a nice experience last year, libvirt is my top option to join
the GSoC 2015. And my interesting project is “Introducing job control
to the storage driver”.

I have read some materials about this project. Including I noticed
that Tucker had attempted it last year. It seems he didn't make it
through the final evaluation. During his working, some discussion were
made in mail-list. It is really inspiring to me. So I have some basic
ideas for now.

The project requires job control functions (at least, reporting the
progress) in storage driver. Obviously, it contains two part. First
find codes that really do the storage work which may take a long
period and can be asynchronized. Then, extract it to the job control
part so make it under the control. It would not be hard to find codes
that really need asynchronization. Maybe by dumping the debug message
and tracing the function calls. So the big part is design the APIs for
job control. I think the goal for API design is not only make it
workable but also make it reasonable and extendable.

To achieve this, a lot of details were discussed in last year's
proceeding. I summered it and add my own opinion.

Parallel jobs: The idea result is that several jobs work in parallel
and libvirt monitors and controls it. There are two ways for parallel:
thread and process. I prefer process. In process, we can easier
implement the idea of *control* by signal. Process has better
independence than thread. What's more, it is a low coupling design.

Synchronization: process can use system level lock to make sure it
obtain the resources. If the process can't obtain it could exit with
failure (or wait). In process, we can leave most resource competition
handling by OS. If thread is used instead, we need to think about
resource competition between libvirt and other process, and at the
same time, those competitions in libvirt thread.

Management: We execute those asynchronous codes in a new process. In
libvirt, it invokes those processes with parsed arguments. Libvirtd
would have a process pool to store the pids and some attached
information for each process. Signal would be used to communicate
between process and libvirt.

Expandability: Some other jobs like domain migration could be in
implemented under this design. It's all about creating new jobs with
parsed arguments, which tells the child process what to do and what
resources they need.

Privacy: If new jobs are created in process, user may access the
process directly and not noticing libvirt. Function sigaction(), which
provides the pid of sending process, could be used to register
responding functions. Meanwhile, atexit() register functions that
execute when the process is going to exit. It is helpful on notifying
libvirtd the end of the job.

What already have: Besides the discussion, some other resources would
be helpful for this project. qemu has a prototype for job control in
migration. We already have gnulib and tools in util/virprocess.h and
util/virthread.h to achieve parallel.

When undertaking, I would firstly implement the job control part. It
would have some basic functions. Make one storage API work in it.
Then, adding job control support for the rest APIs.

This is all about what I came up with. Maybe those ideas are old and
repeating. But I think it is a workable plan. Waiting for feedback.

Regards,
Taowei