FUDCON hackfest - Building a fedora test plan

[James Laska]

Ooops ... <ctrl> <enter> is too close to easy to hit when switching
view ports.  

Premature_email_send++

On Tue, 2008-06-17 at 08:29 -0400, James Laska wrote:
> On Mon, 2008-06-16 at 21:10 -0400, John Poelstra wrote:
> > 
> > I think this is a great idea.  It sounds like you are focusing 
> > specifically on test plans for these areas?  Would these also result
> > in 
> > test matrices which are filled out before we release and potentially 
> > block the release if broken?
> 
> You betcha!  I'm not looking to go through a deep dive of every
> application.  We'll start by getting a long list on a white board, then
> begin making some cuts for this first pass.  
> 
> > One thing that I think is still lacking in our overall release
> > criteria: 
> > http://fedoraproject.org/wiki/QA/ReleaseCriteria is a more detail 
> > surrounding the desktop applications... for example here 
> > http://fedoraproject.org/wiki/QA/ReleaseCriteria#Desktop.
> 
> I've gone to both extremes with respect to release criteria.  What I've
> found works well is the "keep it simple" model.  What we have now is
> fairly basic, but I might even suggest something more minimal.  Perhaps
> we start by keeping it close to home and in terms of things we all
> understand ... bugs.  
> 
>  Alpha - 
>    * All feature tracking bugs filed (and bound to feature wiki pages)
>    * No OPEN bugs with priority >= urgent AND severity >= urgent
>  Beta - 
>    * All feature tracking bugs in MODIFIED
>    * No OPEN bugs with priority >= high AND severity >= high
>  PreviewRelease -
>    * No OPEN bugs with priority >= high AND severity >= high
> 
> >From there, we define some rules on how to assess the impact of a bug
> and get it on the appropriate radar.  My preference would be to defer
> the decision to individual test leads.

and their development counterparts.  They're going to be closest to the
components/code and in the best position to provide an assessment on
impact to Fedora.

> There are two really solid definitions of bug severity out there 
>  * from Mark
> Cox ... http://www.redhatmagazine.com/2007/04/18/risk-report-two-years-of-red-hat-enterprise-linux-4
>  * from
> bugzilla ... https://bugzilla.redhat.com/page.cgi?id=fields.html#bug_severity

> Using these definitions, we can 

try to build some consistency on the bugs Fedora QA adds to the mix.
This is likely a separate large project with ties to Triage, but at
least something on my radar I'd like to devote some brain power to
during the upcoming release.  

Perhaps something to discuss in a future meeting/blog. :)

> > Do we REALLY do complete "click-through" tests for every application?
> > I 
> > don't think I've ever seen a test matrix created for this nor do I
> > think 
> > we want to create one.  I think it would be a better use of our time
> > to 
> > create a list of the more often used applications (perhaps via
> > mugshot) 
> > and add those to a matrix for specific testing.

Definitely, I'm in agreement here.  Let's collectively determine what's
most important ... and go from there.
 
> > Another thing we might want to brainstorm about at FUDCon is creating
> > a 
> > complete list of test matrices... including CD installs, live images, 
> > etc. that we need for each release.  I think we overlooked some of
> > these 
> > during F9.  I think the idea of creating test plans and test cases 
> > beyond just installation testing are a great step forward.

I'll be very happy if at the end of the hackfest QA session at FUDCON
we've:
 * defined a list of important use cases or applications
 * narrowed the list down to the "must haves" for F10
 * stub out some basic matrices for one or more of these use cases
 * solicit interested parties for plan ownership or feedback

Thanks,
James

> -- 
> ==========================================
>  James Laska         -- jlaska at redhat.com
>  Quality Engineering -- Red Hat, Inc.
> ==========================================
> 
-- 
==========================================
 James Laska         -- jlaska at redhat.com
 Quality Engineering -- Red Hat, Inc.
==========================================