Request for Comments: Proposed changes to Fedora development cycle

[Jesse Keating]

On Wed, 17 Oct 2007 07:55:28 +0200
Thorsten Leemhuis <fedora at leemhuis.info> wrote:

> >  We have to stop changes in order to finalize
> > the release and get it out the door.  
> 
> A week IMHO should suffice. If it doesn't I'd say we need more point
> in the middle to bring rawhide at least in a better shape beforehand.

I think you severely understimate the amount of time it takes to
compose trees, including all the Live images, validate that the compose
itself went fine, final check that all the test cases still work, move
the bits around to stage for mirrors, get mirrors populated, send gold
bits off to pressers to produce media to use at events, finalize
release notes web content, etc, etc, etc...  A week is a severely short
amount of time to accomplish all of this.

> > [...]  
> >> Normal, non-critical
> >> updates for the release (once it happens) stay updates candidates
> >> and thus don't get much tested either.  
> > They can be pushed to updates-testing at the 0day mark and thus get
> > testing there.  
> 
> Depends on how much lands into updates-testing over those weeks (and
> how many weeks we are taking of; three to four afaics). IOW: it IMHO
> looks bad to do a release and have more than <n> updates in
> updates-proper and <n> in updates testing (<n> something like 10 or
> 20 each in both cases).

I'm sorry but I just can't fathom being able to accomplish all we need
to do to get the release out in just a week's time, especially if there
have been open commits and builds right up until that week's time.

> >> Some packagers will thus likely just push their updates into the
> >> current release without testing them in rawhide first.   
> > But rawhide isn't the place to test them,  
> 
> Sure it is -- at least for me. Here is what I do when I get a new
> upstream release in Fedora:
> 
> - test locally (but that's of course always limited, as apps get used
> differently by different people)
> - update cvs for devel and build for rawhide, as rawhide users are
> those that won't yell to much if upstream released something new with
> multiple bugs in it
> - leave it there -- depending how crucial the package is for a few
> days a week some weeks
> - if nothing bad turned up in rawhide (e.g. if the new upstream
> release didn't contain any new bugs) I ship it in F7 updates-testing
> for some days
> - if nothing bad turns up move it to F7 updates proper
> - if that package would be of interest for F6 I would put it in
> updates-testing for F6 at this point (if one would exist)
> - if nothing bad turned some days later up push to F6 updates proper
> 
> Shipping a update in multiple branches at the same time or in F7
> earlier then in rawhide (because it is blocked for release) seems
> like a bad idea to me (I know, that's the standard behavior for a lot
> of people) -- if upstream got a bad bug in it all our users are
> effected by it and that can easily be prevented by above scheme.

However rawhide is not the place to test F7 issues.  You have different
sets of users in rawhide, F8-testing, F7-testing, etc...  You need to
have testing by each of those user sets before you can go final updates
for those releases.  They are different releases with different
userlands and different kernels/runtimes/etc... and quite likely there
are different interactions you need to validate.  Using rawhide as a
test case for these may find you the really really glaring problems
(which IMHO should be found in local testing anyway) but won't get you
near those other issues.  Releasing it as a testing update for F7
before it shows up in rawhide is perfectly fine.  Again different sets
of users, different problems, different test cases.

> >> That's bad in
> >> general already; but it's even more bad here, as it also create
> >> EVR-update troubles once the release is done.  
> > How can it create EVR-update troubles?  Only if maintainers push
> > updates to previous releases before pushing them for the pending
> > release, and even so this is not a new problem, as updates to
> > previous releases happen all the time while the release is a static
> > package set that quickly has "lower" nvrs than the fully updated
> > previous release. Not a new problem, [...]  
> 
> No, but with the new scheme it becomes bigger afaics.

I don't see how it will become bigger.  We're /shortening/ the amount
of time where builds don't automatically go somewhere, not lengthening
it.

> 
> >>> In order to accommodate a
> >>> release candidate set plus the final release creation the freeze
> >>> point for this should happen between the current Test3 dates and
> >>> the final deep freeze dates.    
> >> This proposal would IMHO be a bit easier to understand if you could
> >> write a potential F9 schedule with release, freeze and branching
> >> dates.  
> > I can add some example dates, but they shouldn't be counted on until
> > the Board and FESCo et al set the schedule.  
> 
> thx for doing that!
> 
> > [...]  
> >> Yes, sure, I know out mirrors are important, but not that
> >> important to delay the release that 5 days when the test window is
> >> four weeks (the usual cycle we had between test releases up to
> >> now).  
> > That puts an extreme pressure on the mirrors that I'd not like to
> > do. We've frequently seen feedback from them that if we just do
> > torrent first and delay the mirrors that they'd just as likely stop
> > mirroring us which doesn't help anybody.  The only /real/ way to
> > get releases out there quicker is to not have unstable trees all
> > the time.  
> 
> I strongly disagree with the "only" in that sentence. OpenSuse and
> Ubuntu afaics seem to get their test-release out way more quickly. We
> should be as well for *test*releases. For the final, yeah, the old
> behavior might fine. I'd say most mirror admins will understand that
> we need to get test releases out more quickly.
> 
> >> Further: I know multiple people that don't run rawhide because we
> >> always tell everyone "once rawhide always rawhide"  
> > Where exactly do "we" tell everyone this?  
> 
> It's in peoples heads afaics and they mention it on the list if
> questions come up "can I run a test release and later the final
> without reinstalling?" -> "No".

Find me a place that states this in official Fedora documentation and
we'll change it.  It's just absolutely untrue.

> Of did we document somewhere in between how to switch from rawhide to
> release before rawhide moves on? If yes please point me to, then I'll
> blog about it to get the old scheme out of peoples head.

As I said before, we may not have the process documented, but to the
best of my knowledge we certainly don't have the opposite written down
either, that you /can't/ do it.

> >> and "rawhide easts
> >> babies". That' IMHO are our biggest problems. Thus i suggest:
> >> - work hard to make rawhide not eat babies and kill kittens in the
> >> 5 weeks approaching a release. Tell the developers so they are more
> >> careful during that timeframe (e.g. as careful as they would be
> >> for a update for a released version) and send ninjas if they are
> >> not.  
> > Why 5 weeks, why not just all the time?  
> 
> Well, that's what we tried. Seemed it didn't work to well -- see for
> example OpenEXR.

Right.  It's hard to get into /every single/ person's head to not
disrupt rawhide without warnings or without doing some legwork to see
if what you're doing is going to cause undue problems.  We can only try
to get better each and every release.  I don't think putting a timeline
on it helps at all, I think it should be an all the time thing.

> >> - we need a way for testers that want to test the release that they
> >> want to continue to run afterwards without reinstalling it. Two
> >> ideas follow, I think we should do both:
> >> (1.) support and document the "rawhide -> stable" way -- e.g. at
> >> the point where rawhide on the servers becomes targeting the next
> >> Fedora version give users the choice "Do you want to continue
> >> using rawhide or stick to the release that is happening now?".  
> > Not sure how to "give them the choice"  
> 
> Yum-Plugin? Or something that disables all repos and asks which ones
> to activate again: rawhide or release.

Potentially.  Let me know when this code falls out of the air (:

> 
> > but we certainly can document
> > how to switch to the final release.   
> 
> thx -- in the past it seemed that was discouraged.

Maybe it depended upon who you talked to, but yes we've had an overall
paradigm shift to be more friendly to rawhide users and be more helpful
to get them from a release to rawhide and from rawhide to a release.

-- 
Jesse Keating
Fedora -- All my bits are free, are yours?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/fedora-devel-list/attachments/20071017/198f0424/attachment.sig>