[Libguestfs] Splitting the large libguestfs repo

[Pino Toscano]

On Tuesday, 15 October 2019 18:39:32 CEST Richard W.M. Jones wrote:
> On Tue, Oct 15, 2019 at 03:59:04PM +0200, Pino Toscano wrote:
> > On Tuesday, 15 October 2019 10:01:28 CEST Richard W.M. Jones wrote:
> > > I got a little way into this.  The two attached patches are
> > > preliminary work.
> > 
> > I see the work was done already, so I guess providing alternative ideas
> > (or opinions, apparently) is of no use now...
> 
> It's always valued.

Of limited use though, if things are already rolling toward a specific
direction.

> > > My proposed split is:
> > > 
> > >       libguestfs.git
> > >           common -> git submodule libguestfs-common.git
> > >           generator/
> > >           lib/
> > >           all language bindings
> > >           C based tools (eg. virt-df, virt-edit, guestfish)
> > >     
> > >       guestfs-tools.git
> > >           common -> git submodule libguestfs-common.git
> > >           virt-builder, virt-customize, virt-sparsify, virt-sparsify, etc
> > 
> > I do not think splitting these tools in an own repository makes much
> > sense.  What is the goal/advantage you get by splitting them in an own
> > repository, compared to the ones left in libguestfs.git?  Users do not
> > care about virt-customize written in OCaml rather than C, and it makes
> > harder to eventually rewrite a C tool in OCaml.
> 
> This part of the split isn't absolutely necessary, I really wanted to
> concentrate on virt-v2v and get that done, partly just because dealing
> with virt-v2v inside the bigger repo is such a pain.

TBH I would have concentrated on virt-v2v only, just like I handled
virt-p2v on its own.

> You're right that people don't care about what a tool is written in,
> so another idea might be to put the C _and_ OCaml tools into the
> guestfs-tools.git repo (leaving lib + daemon + language bindings only
> in libguestfs.git).  In fact I like this better now I think about it.

Still some of the questions I wrote above apply: what do we gain from
splitting the tools in an own repository? Having them in an own
repository means that, unles you use the API yourself (in C/Python/etc)
then you need to build another repository to do image
manipulation/customization.

The tools (other than v2v) are quite stable, and they do not have many
changes these days. Looking at the release notes for libguestfs 1.38,
and 1.40 (the latest two stable series), the biggest changes were
- add --key in all the tools
- output selection for --machine-readable in OCaml tools
whose main implementations were in common code anyway.

This also crosses another topic: how is virt-v2v going to be versioned,
and released? Similar question for the potential split of the tools.
If the answer is "packed together in a single tarball like now", then
splitting makes the work more complex only to developers, i.e. us.

> > > The current common/ subdirectory would become a git submodule.  While
> > > git submodules are awkward, they do solve this particular problem with
> > > having common code shared across the repositories, there's only one
> > > git submodule and it's under our control.  It does mean that any time
> > > there's a change to common/, we would need to add a commit to the
> > > other 3 repos updating the submodule hash.
> > 
> > The current common/ subdirectory is a giant mixup of different
> > components needed by some or just one tool each; few examples:
> > - common/mlaugeas -> only for the daemon
> > - common/mllibvirt -> only for v2v
> > - common/mlxml -> only for v2v
> > - some of the ml modules are need by any OCaml stuff
> > - some of the ml modules are used by 1/2 tools
> > - etc
> 
> It's a step on the path rather than the end point.  We can definitely
> move mllibvirt & mlxml to virt-v2v in future.  We can also try to
> organize it better or split it in future.

Considering that the changes are deep in the structure of the
libguestfs sources, having a bit more light on the proposed path would
certainly help to understand it better, and check whether the direction
is sound.

> The alternatives to the submodule are somehow packaging everything up
> into a library, which would then become a build dep of the tools.
> This isn't particularly nice because (as you say) it's a big mix of
> miscellaneous "stuff", essentially a core library of missing C and
> OCaml functionality that we happened to need.  So therefore not useful
> as an independent library.

There is also the fact that this set of common code has no concept of
compatibility, so this potentially needs branches for each stable
series. Also, this crosses the topic above of releasing, w.r.t. bundled
or independent releases.

> I'm not overjoyed by git submodules, but in this particular instance I
> think it can make sense, and there's only one of them (at the moment,
> we could change that).

Right, I'm not thrilled by git submodules either, and so far I have a
bad experience with them.

-- 
Pino Toscano
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: This is a digitally signed message part.
URL: <http://listman.redhat.com/archives/libguestfs/attachments/20191016/883fc098/attachment.sig>