[fedora-virt] Improving release / update process for virt packages

[Daniel P. Berrange]

This mail is a braindump of some ideas we've been kicking around to
try and improve the way we push new releases of upstream virt 
packages to Fedora. We want to try and improve the quality of stable
Fedora branches, and also improve testing of rawhide. 

Taking libvirt as an example, currently new libvirt upstream releases
get immediately built and pushed to both

 - Fedora rawhide
 - Fedora stable updates-testing

Depending on the development phase rawhide is in, we get somewhere 
between 'zero' and 'lots' of feedback for new builds in rawhide. 
Typically before rawhide beta, we get near zero feedback, and from
beta onwards we get alot of feedback.

We typically always get significant feedback when the new builds are
pushed to updates-testing.

The obvious problem with what we do for libvirt at the moment, is
that we are introducing major new features into the stable release
stream here. This risks destabalizing the stable Fedora streams, 
and also compromises our 'Feature' process because stuff we're 
pushing for Fedora 11 appears in Fedora 9 / 10 before F11 even comes
out.

I think it would be desirable to get the stable Fedora releases onto a 
pretty strong bugfix only policy, but we can only do that if we figure
out a way to get more people testing the stuff we're pushing for rawhide.
There simply isn't enough testing of rawhide prior to beta point. The
bug reports / testing from the stable update-testing repo far exceeds
the feedback we get from rawhide users.

We can keep telling people that rawhide is supposed to be functional
enough for daily use, but frankly history shows otherwise and I don't
see any sign of this ever changing. Things like the mass rebuild,
rpm upgrade, xorg <-> kernel interface are inevitably changing and
cause instability. Personally I do pretty much all of my day-to-day
libvirt work on a stable Fedora 9 / 10 distro, only ever using rawhide
if there is a specific thing I can't get in F9/10 packages.

There are many people who wish to test new virtualization pieces. They
do not wish to have to debug changed RPM, or changed Xorg or changed
kernel packages, and the rest of rawhide at the same time. While it is
true that problems with rawhide can be resolved when they occur, it is
still a significant waste of time. If I'm interested in testing virt,
I only expect to have to debug virt packages,  not the entire OS.

What we need is a way to expose new virtualization features to people
running the current latest stable Fedora release stream. So if they 
are interested in testing the new features, they can update to those
packages without having to include & debug the rest of rawhide.

Debian distros have this sort of idea in the 'backports' stream which
gives optional early access to new app releases, separate from the
normal updates and unstable streams. This is distro wide though.

We currently have people doing various adhoc personal repos from scratch
builds, so my suggestion is that formalize this under the umbrella of the
Fedora Virtualization SIG. Specifically, to provide a 'virt-preview' YUM
repo for the most recent stable stream (ie F10, but not F9).

In the designated virtualization package set, anytime we do a build of a
new release for rawhide, we would also do a build for the 'preview' stream.
We would explicitly not do these builds for the 'updates' repos, at least
not without a very significant period of testing & positive feedback.

To allow us to do builds in both the preview stream, and updates stream,
which are not scratch builds, this would unfortunately require another 
CVS branch. We'd also need an NEVR that satisfies

 - Newer than current stable stream NEVR
 - Older than the new build in rawhide
 - Allows for newer build to be later made in 'updates' repo

One simple option is to have

 - CVS branch of F-10-VIRT_PREVIEW
 - Release field for all builds in F-10VIRT-PREVIEW must have leading 0

The leading 0, allows us to later do a build in the main F-10 stable
branch with newer NEVR, without risking that it then become newer
than the rawhide build.

A convenient Makefile rule in the 'devel' branch could automate the
initial work of copying the contents of devel/ to F-10-VIRT-PREVIEW
and setting the N-E-V-R to suitable value. A few manual changes 
might be needed, but hopefully not much before the maintainer could
do a build in koji.

There would then need to be a way to build a YUM repo from all builds
in the F-10-VIRT-PREVIEW branch. It should automatically take all the
latest builds from the branch - we want it to be a mini rawhide of
just virt packages, not use the updates infrastructure.

If we could get Fedora infrastructure support for the branching that
would be great, but we could easily do this with a private branch like
(private-F-10-VIRT-PREVIEW) and a cron job to build the YUM repo and
publish in a well-known location, eg http://virtmaint.fedorapeople.org/

So in summary

 - All new upstream releases built in rawhide
 - New upstream releases also built in stable preview branch if possible
 - Only bugfixes built in stable updates/updates-testing branch
 - In exceptional circumstances, rebase for preview branch can be
   built to updates/updates-testing after alot of positive testing

This would

 - Ensure users of stable Fedora release have high confidence in 
   quality of the updates/updates-testing stream
 - Allow users to trivially test new upstream rebases while 
   staying on the stable distro stream
 - Improve testing coverage of major new rawhide features without using
   the stable release stream users as guinea pigs


Regards,
Daniel
-- 
|: Red Hat, Engineering, London   -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org  -o-  http://virt-manager.org  -o-  http://ovirt.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-  F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|