Testing Fedora - small (?) suggestion.

[Gilboa Davara]

On Sat, 2006-11-11 at 00:28 -0900, Jeff Spaleta wrote:
> On 11/10/06, Gilboa Davara <gilboad at gmail.com> wrote:
> > A. Create more mile-stone releases. Once the tree reaches build
> > integrity (no missing packages), spin a test release. (Fixes P1, P2)
> 
> Logic fault... we dont have enough testers right now... more
> mile-stone releases will actually mean even fewer people will be
> testing each of those releases.  We don't magically gain more manhours
> of testing by having 14 individual isosets in the wild.


You've missed my point.
Take Mozilla - each and every night they release a nightly build.
Do all of these builds get used? -Far- from it.
So why do they do it?
Simple, because they want to -lower- the amount of voodoo magic required
by a tester in-order to get the damn thing (excuse my language)
built/installed.

Point of reason:
A. The biggest issue with any Linux distribution is the installer.
B. We don't have enough rawhide testers.
C. Trying to install Rawhide from scratch (in-order to test the
Anaconda) is a -very- frustrating experience. (As I said, at least for
me, it's has a 50% chance of blowing up).
E. A tester that had his Rawhide installer die due to missing package,
is a tester that may never try Rawhide again. So as a result...
F. ...most rawhide users avoid the frustration by installing the latest
release and using yum-to-development to jump (plus massive --exclude) to
get Rawhide installed. However....
G. ..install Rawhide using yum doesn't test the installer.

But...

H. Spinning ISO is a labor intensive task.

Solution:
A. Create* a tool that's capable of detecting tree integrity.
B. Execute this tool once a day.
C. Once the said tools finds no broken packages (for arch N), copy the
packages to a different directory, label it by date and...
D. Send an automated message to fedora-devel, testers, that a new label
is out.

* I assume that such capability already exists within yum.


> > B. Change the terms that are being used to describe each test release.
> > Whether we like it or not, people are used to the "Alpha", "Beta" and
> > "RC" terms, and tend to consider "Test release" as "Alpha release". I
> > understand that the term "Test" was used to differentiate the
> > ever-rolling Fedora from the release-based RHEL, but Fedora has aged
> > enough to be viewed as an entity by itself and we can drop the "Test"
> > term.
> 
> Complete redherring. Changing the naming scheme...again... 

Why again?

> will only cause additional confusion because it will require yet another change
> in things such as mirror url locations and updating available
> documentation on the procecss. 

I doubt it.
Current test releases are called <Rel>.9<TestID>, and it had been so
since, baah, RedHat 4?
URL won't change.

As for documentation - I don't know enough to comment about that.

>  This is a pointless change for the
> sake of change simply because its an easy thing to do, without
> quantifiable metrics as to the importance of this particular issue.  I
> decree this is completely not important.
> I'm not even sure how valuable technical feedback is from people who
> are confused by the naming scheme. At best I expect "those" people to
> bitch about colorschemes, font glyphs and other stylist aspects which
> are not of an important technical nature.

While my sampling group is small, not more then 15 Linux using friends
and coworkers.
They all work in Hi-Tech companies.
Most of them are active in local LUGs and/or contribute time/code/etc to
the OSS world.
Most of them are using unstable distributions - Debian Sid, Unstable
Gentoo (assuminh there's a stable one...) and OpenSUSE (Though I'd
assume OpenSUSE will drop like a stone now) 
Neither of them is "ugly-font-joe-six-pack"... far from it.

Come to think about it, "ugly-font-joe-six-pack" doesn't know what
Alpha/beta/RC means - he may understand what test means.
Anyone living in the software development world is used to judge the
stability of software by the term Alpha/Beta/RC and lack of them is
confusing.

> 
> > C. Once Fedora hits RC, only bug fixes go into the tree. No last minute
> > 2.6.39 kernel that break Anaconda, SCSI, and USB two days before the
> > release. Nada. New features can always enter the tree as updates once
> > the release ISOs have been sent.
> 
> Easier said than done... and in fact against the very nature of how
> kernel updates are done once is a release is out.  Say it with me...
> Upstream  Upstream Upstream!  I think you underestimate the amount of
> work and time it would require to hand pick individual patches and
> backport them instead of lifting the next upstream kernel which
> includes a superset of issues identified in Fedora based testing.

One of the basic thumb rules in software development is that if you
change basic-component-X within product-Z, most/all of the testing that
has been conducted up to this day should be repeated.

Put a new glibc and/or kernel, 4 days before the release, and you'll
need at least one major RC release to clean up the mess or you may end
up with a broken release. (And being a long time FC user, I do remember
a couple of those...)

> 
> The current kernel maintainer may want to comment on this particular
> issue in more detail, but in an effort to save him his valuable time,
> I would strongly suggest that you take a look back through the history
> of fedora-devel mailinglist for kernel maintainer comments concerning
> the overall goal of reducing the amount of patches being applied to
> fedora kernels and to stay as close to upstream as is reasonable.  I
> don't think what you are suggesting here as a remedy to the kernel in
> particular is realistically possible.

The problem is not upstream vs. patch-set.
The problem is "release a new kernel/glibc two days before the release
without any type of meaningful testing"

> >
> > Here's a mock Fedora release schedule:
> > T-4 Months: Alpha1
> > T-X Months: AlphaX
> > T-1 Months: Beta
> > T-3 weeks: RC1 - Tree go into lock mode.
> > T-1.5 weeks: RC2
> > T+n weeks: unexpected RC3.
> > T: release. Part time.
> 
> Forget for a second the substantial additional burden on the release
> engineering team associated with the scheduling associated with all
> the new self-consistent isos you want to see spun up.  Or the fact
> that you have to institute additional freeze deadlines which make it
> more difficult for individual maintainers to get new tech into the
> tree at all.  Do you really expect a significant number of testers to
> install even half of these images and do the due dilligence associated
> with testing of the installer looking for regressions?  And if a
> significant number of people aren't going to be doing the regression
> testing..then you haven't solved the problem you were aiming to solve.

See above.

Question: Can pungi be used to auto-create the ISOs?

> 
> 
> -jef"Every single iso image deadline in that mock up that you expect
> to pass through release engineering will slip by a week..garunteed.
> You want a to see 8 milestone isos.. that 2 months of delay associated
> with any strawman schedule. Hold your breath."spaleta

Gilboa "unless the ISO is being mastered and uploaded automatically by a
script every time depcheck decides that there's no missing packages"
Davara