On Wed, 2006-12-13 at 07:43 -0500, Luis Villa wrote: > On 12/13/06, Dave Jones <davej redhat com> wrote: > > On Tue, Dec 12, 2006 at 10:48:54PM -0500, Luis Villa wrote: > > > Probably more usefully, if for some reason you can't have earlier > > > freezes for critical, complex subsystems, who is in charge of > > > encouraging early stress testing? Is there someone whose job it is to > > > evangelize widespread testing? > > > > That'll be Will Woods <wwoods redhat com>, Fedora QA lead. > > Oh, great, glad there is an answer to that. (/me waves at Will if he > is on this list.) Howdy! It's good to meet you. > > Towards the end of FC6 dev cycle, we did a few things differently > > to improve things. > > * bi-weekly status meetings. Just a half hour concall to make > > sure everyone knew the state of the onion. > > Does Will have the power in these meetings to change priorities and > direct engineering resources towards the end of a cycle? e.g., during > the FC7 meetings, what will happen if he screams about Xen? :) Luckily for me, the people involved in the FC6 status meetings are all reasonable folk, so it was never necessary for me to scream about Xen being broken. It seemed we were all in agreement on what was OK and what needed love, and efforts were directed in the appropriate direction. In the end, though, I think Max has said that I have the power to press the Big Red Button on a release if something is absolutely broken beyond repair. > > * Will started hashing out test plans. > > * Some process documentation got put on the wiki. > > * I believe there are weekly(?) test team irc meetings. Yes indeed! The next one is at 1700UTC (Noon EST) tomorrow (Dec. 14): http://fedoraproject.org/wiki/QA/Meetings/20061214 > > Something else that should see the light of day at some point will be > > the eventual opening of our internal QA test harnesses (along with > > some of the tests that we put a RHEL release through). > > Getting stuff like this into the hands of eager wannabe testers > > is going to be really useful. > > Yeah, sounds like it. We're working hard at this stuff - the automated test tools are currently hosted at https://testing.108.redhat.com/ if you wanted to poke around. The new updates system should have hooks to run automated tests, and we will be maintaining a repo of contributed tests that the update system will pull from. There's all sorts of fun work going on here. > > > More questions that you could ask, besides 'why did it go wrong last > > > time?' would be things like how does Fedora define a showstopper? > > > > Basically 'blocking' criteria is something like.. > > * large class of machines doesn't boot > > * really common machine doesn't boot > > * installer falls over really easily with no workaround > > * common network cards don't work (preventing updates). > > > > It's spelled out a lot better than my feeble memory can remember > > somewhere on the fedoraproject.org wiki. http://fedoraproject.org/wiki/QA/ReleaseCriteria was the set of criteria proposed during FC6. For the most part, we managed to stick to it, but you'll notice it doesn't include anything about new features in the release (e.g. Xen in FC6). We had publicly committed to delivering Xen, though, so it had to work. The tree testing matrix required a Xen install: http://fedoraproject.org/wiki/QA/TreeTestingTemplate The real problem here was that we were already committed to delivering Xen, so we had no choice but to hammer on it until it was fixed. > That makes the question even more pressing, then. Assume Xen is > horribly broken come April 23rd. What will Fedora do? What process > will the board go through to make that decision? The earlier you know > what that process is, the better you can plan. Well, I think we need to add something to the Release Criteria about major features for a new release. I assume there will be two basic categories of features: some features that will be required (i.e. we will block until it works - stuff like SELinux, new GNOME releases, etc.) and some others that we plan to include, but may drop if we can't get it in a reasonable state for release (maybe Xen would have gone here in the past). We'll need some planning meetings to figure out the feature list and put the features into those two categories - I think there are normally meetings to decide on the feature list, so that's just a couple extra bits to discuss there. So then the Release Criteria just need to say that Required features must work before release, and other features need to be "sufficiently stable" to be included. Probably "sufficiently stable" will be need to be better defined, but basically it means "The release team thinks it's OK, and the testing/development community have tried it and aren't complaining." So then we know which features we slip for, and which ones we can either drop or release-note ("Feature X is b0rken. It'll be fixed by an update Real Soon Now. Really.") Does that make sense? Would that cover most cases? -w p.s. It's great to have you around - I'd love it if you could pop into a QA meeting or #fedora-testing sometime and we could discuss these things further.
Description: This is a digitally signed message part