fedora 7 schedule (was Re: Fedora 7 planing)

[Luis Villa]

On 12/13/06, Dave Jones <davej at redhat.com> wrote:
> On Tue, Dec 12, 2006 at 10:48:54PM -0500, Luis Villa wrote:
>
>  > > * Xen being horked and taking forever to get working.
>  > >   Not particularly easy to fix given we're at times dependant upon
>  > >   an unresponsive upstream.
>  >
>  > (0) Prevent feature X from going into distro trunk until feature X was
>  > actually ready-ish, such that there was never risk of delay from
>  > feature X.
>  > (1) back feature X out completely, or at least to the FC6 state.
>  > (2) admit that feature X will not work in FC7.
>  > (3) delay FC7.
>  >
>  > I would have suggested in the Xen case that you should have done (2),
>  > since I presume Xen is too tightly tied to the kernel to allow for (0)
>  > or (1). Why did you do (3) instead? Commitment to feature based
>  > releases over time-based releases? Some other reason?
>
> It was deemed "must have" functionality for a number of reasons.
> Two obvious ones being:
> 1. dropping it would be a regression vs FC5.
> 2. It's a major line item for RHEL5, and Fedora is supposedly
>    where stuff like this gets beaten into shape first.

(1) makes sense, though in that case there is an obvious question
about 'why was that last, broken patchset allowed in so late?'

As far as (2) goes, I thought Fedora was an independent project?

>  > Focusing on the future, what is Fedora's plan for the Feature X (which
>  > will almost inevitably occur) in FC7? (0)? (1)? (2)? (3)? None of the
>  > above? [Note that (3) doesn't seem to be an option, given the hard
>  > deadline, which suggests FC needs to assess 0-2 and set policies for
>  > how to choose between 0, 1, and 2.]
>
> Right now, as far as Xen goes, I'm sitting on the fence waiting to see
> where things land.  With most of our virtualisation team tied up in
> hammering out RHEL5, and Xensource sitting on their thumbs wrt pushing
> stuff upstream, I'm not optimistic at all that anything notable will happen.
> The grim meathook future currently looks like this:
>
> * Xensource's Xen tree is based on 2.6.16 (Yes, sixteen)
> * FC6 has a 2.6.18 kernel and a rebase of Xen that Red Hat did.
> * kernel.org current stable is .19
>   We can't move to this as an FC6 update until we get Xen in shape.
>   Juan mentioned that he's going to jump on this soon, but its probably
>   at least a weeks work.
> * devel/ is current on the road toward .20rc1, and hasn't got Xen
>   applied at all (zillions of rejects).
>
> I find it hard to talk about Xen in person without cursing. Really.

That makes the question even more pressing, then. Assume Xen is
horribly broken come April 23rd. What will Fedora do? What process
will the board go through to make that decision? The earlier you know
what that process is, the better you can plan.

>  > I'd also note that (3) makes a ton of sense in an enterprise OS
>  > context, where you've made hard commitments to customers about feature
>  > lists, so that 0/1 (and particularly 2) are not options. This is the
>  > kind of thing where Fedora can be (should be?) different from internal
>  > RH engineering processes, I think. But that is an explicit policy
>  > choice- to be time-based, and not feature-based- that Fedora's
>  > leadership should explicitly think about and choose.
>
> tbh, I agree with you. Fedora should not be hostage to RHEL feature requirements.
> Merging Xen was the single biggest headache I've faced in kernel maintainence
> in the last 3.5 years.  Even NPTL against the RHL 2.4 kernels was a walk
> in the park compared to this fiasco.

So whose responsibility is it to make that call? (Or alternately, to
public admit that that call can't be made?) Again, the earlier the
policy is decided on, the better for everyone.

>  > Since it is specifically external legal liability that prevents (2),
>  > that suggests being as proactive as possible about all *legal* issues
>  > in order to avoid delays. Understanding that perfect foresight is
>  > impossible, who is in charge of assessing legal issues and being the
>  > best humanly possible lookout for legal icebergs? What are they doing
>  > right now to help meet this proposed schedule?
>
> Red Hat legal is pretty much the gatekeeper for such issues.

Gatekeepers are by definition stationary and deal with problems that
come to them :) Who from Fedora is responsible for pushing RH legal to
work proactively on this kind of thing, or alternately, responsible
for finding such problems and bringing them to RH legal ASAP?

>  > > * Discovery of late breaking nasty bugs (Like the ext3-went-boom bug)
>  > >   Whilst we'd all like it if this weren't to repeat itself ever again,
>  > >   it can't be ruled out. Sometimes things just fall out of testing late
>  > >   in the cycle, and right before release is when we really stress things
>  > >   as much as possible.
>  >
>  > Probably again a naive question, but why were changes to something as
>  > critical as the file system being made late enough in the game to
>  > delay the release?
>
> The bug in question had been there for months.  It turned out that
> it only affected filesystems made with 1K block sizes, which isn't the
> default, but we didn't know that at first.  As this was a corner case,
> it unsurprisingly didn't get a great deal of testing.

Ah. Sucky. Admittedly no good way to deal with that, sort of begging
for lots of testing, which I realize is hard (despite my other
questions :)

>  > Probably more usefully, if for some reason you can't have earlier
>  > freezes for critical, complex subsystems, who is in charge of
>  > encouraging early stress testing? Is there someone whose job it is to
>  > evangelize widespread testing?
>
> That'll be Will Woods <wwoods at redhat.com>, Fedora QA lead.

Oh, great, glad there is an answer to that. (/me waves at Will if he
is on this list.)

> Towards the end of FC6 dev cycle, we did a few things differently
> to improve things.
> * bi-weekly status meetings. Just a half hour concall to make
>   sure everyone knew the state of the onion.

Does Will have the power in these meetings to change priorities and
direct engineering resources towards the end of a cycle? e.g., during
the FC7 meetings, what will happen if he screams about Xen? :)

> * Will started hashing out test plans.
> * Some process documentation got put on the wiki.
> * I believe there are weekly(?) test team irc meetings.
>
> Something else that should see the light of day at some point will be
> the eventual opening of our internal QA test harnesses (along with
> some of the tests that we put a RHEL release through).
> Getting stuff like this into the hands of eager wannabe testers
> is going to be really useful.

Yeah, sounds like it.

>  > More questions that you could ask, besides 'why did it go wrong last
>  > time?' would be things like how does Fedora define a showstopper?
>
> Basically 'blocking' criteria is something like..
> * large class of machines doesn't boot
> * really common machine doesn't boot
> * installer falls over really easily with no workaround
> * common network cards don't work (preventing updates).
>
> It's spelled out a lot better than my feeble memory can remember
> somewhere on the fedoraproject.org wiki.
>
> Good questions.  I snipped some of them that I felt others could
> answer better than I could, but I hope the above helps.

Hope it at least starts discussion- that is all I can hope to do.

Luis (everyone wish me luck on my Torts exam tomorrow.)