Update/install experience

Wed Dec 16 15:00:20 UTC 2009

Jumping in because it's a topic of great interest for me, even though
I'm not deeply involved in daily QA work.  QA folks, forgive me if I
go astray or misrepresent anything.

On Tue, Dec 15, 2009 at 04:48:38PM -0700, Kevin Fenzi wrote:
> On Tue, 15 Dec 2009 11:06:33 -0500
> Will Woods <wwoods at redhat.com> wrote:
> > > Who is testing them "as a single unit" ?
> > 
> > Anyone who's interested in testing updates or getting early access to
> > new fixes and features. Collectively referred to as "QA". I'm sure
> > you've heard of us?
> 
> Sure. :) 
> 
> I am just worried that this is a "well, everyone will jump in and test"
> type of thing. If they aren't doing so now, how does this setup
> increase QA folks? Or you think the current folks in QA will be able to
> test all updates in the flow if it was just organized a bit better?

I would think it's more of both -- the latter in the short term, being
able to effectively test something after release day; and in the
longer term, the ability to grow a QA community as a result of more
discrete tasks available that can be accurately described, documented,
and picked up by willing contributors.  Right now, once a release is
out the door, there's no effective way to test the stable release
midstream, and have that be meaningful for other people.

Problems in the help channels sometimes come down to users having to
run 'rpm -q <bunches_o'_stuff>' because when you ask "What Fedora are
you running?" the answer isn't all that helpful starting a few weeks
after release day.  (Or maybe even the day after, for that matter.)
The first answer is, "Have you updated?" when there's no real answer
to whether that will actually help the situation, or might cause an
unrelated problem -- typically it's asked because the helper is using
latest updates, and this advice is the only way to make sure the user
is on the same set of software.  I'm not saying that's not a sane
approach the way things are right now, and this is not a slam in any
way, shape, or form toward the intrepid folks who help users with
problems.  They're doing the best they can based on the way our
releases and updates run right now.

But what if we did start offering a collection on a regular basis that
rolled in these updates?  Reducing the multiplicity of updates and
resulting test matrices could make it possible for us to start
cataloging fixes and catching regressions before they impact users.
There'd be more certainty when helping a user as to what components
they actually had installed.  Perhaps this wouldn't look like a
respin, so I'm purposely not calling it that.

> How many problems or bugs are due to integration with other updates?
> In the cases of new package A using new package B, wouldn't you
> currently have to have both installed to test A anyhow?

Which is, I think, one reason why we could, for the moment, step away
from thinking about packages and just consider components as the
original whiteboard suggested.

> > > How much time for testing would there be? If another update shows up
> > > the day before the weekly push, it would be deferred to the next
> > > one? How about 2 days? 3?  
> > 
> > Policy remains to be set here. I'd personally advocate monthly update
> > pushes with a freeze at least one week before the release. Stuff that
> > comes in after the freeze goes into the next push.
> 
> So, this would be essentially "12.1", "12.2"? 
> Ie, minor point releases of stable release X. 

I guess that would be one way to do it, sure.  The only problem here
is that the original release looks like a '.0' and I'd hate to
accumulate all the unsubstantiated baggage that implies.  Maybe that's
unavoidable though.  Already, plenty of people using every Linux
distro under the sun wait for their favorite to supposedly "stabilize"
before they upgrade to a new release, regardless of whether that
approach is actually doing anything for stability.

> > > Would security updates hold for the next weekly push? 
> > 
> > Absolutely not; security updates will always go out as soon as they're
> > ready. Note that this would mean that we have *much* more available
> > manpower to test the security updates.
> 
> Why? Wouldn't the people who test be also testing the next monthly
> blob? Or there would be no testing on it until it's frozen?
> 
> > > Or push out as they are done? If so, wouldn't that mess up in
> > > progress testing? 
> > 
> > Not really. Security updates are usually small, targeted changes, and
> > they're pretty uncommon - I don't have exact numbers but I'd estimate
> > something like 10/month across the entire distribution; the number of
> > security updates for a typical install will be a subset of that. 
> 
> yeah, much smaller to be sure. 

Hopefully, having updates out on a more predictable, less frequent
basis would make it easier to break time off for security update
testing.  It's not a matter of creating more time with a miracle, just
being able to better plan the teamwork.  I can't speak for the QA
folks, but I would think that a constantly moving target (such as
daily updates to a stable release) makes it really difficult to
accurately plan in that way.

> > The destabilizing effect is much, much less than (e.g.) the daily
> > changes we get during the freezes for Alpha/Beta. It's a manageable
> > amount.
> 
> So wait... are we talking rawhide here or stable releases? or Both?

Interesting question to be sure.  Again, I'd call back to
Christopher's post about sheriffs and things that he believes need to
be fixed in Fedora's approach if we want to provide a better
environment for advancing FOSS, especially including our many
developers.  Right now we have a lot of people who are (or could be)
great Fedora contributors who don't use Rawhide until quite late in
the cycle, when it's really in pre-release status.  That means we miss
out on a lot of insightful bugs, comments, and other contributions.
Right now, you not only need to be brave to use Rawhide, you
frequently need to have a lot of time on your hands even if you want
to use Rawhide to work on something that's more at the development or
application level.

I tend to think that Rawhide itself could use some rethinking -- while
keeping it open (up to some point) for radical changes, making those
changes in some unit fashion that would ensure someone could actually
install and use the system daily to a reasonable degree.  It would be
ludicrous to say "No bugs allowed in Rawhide," but there's probably a
basic level of functionality we could ensure, so that it'd be more
usable for daily work and development by people who can still live
with more bugs than you'd expect in a stable release.  For instance,
the system boots on certain definite sets of hardware and a VM, and
lets you login, start a web browser, and visit pre-defined web pages.

-- 
Paul W. Frields                                http://paul.frields.org/
  gpg fingerprint: 3DA6 A0AC 6D58 FEC4 0233  5906 ACDB C937 BD11 3717
  http://redhat.com/   -  -  -  -   http://pfrields.fedorapeople.org/
  irc.freenode.net: stickster @ #fedora-docs, #fedora-devel, #fredlug