FUDCON hackfest - Building a fedora test plan

[James Laska]

On Mon, 2008-06-16 at 21:10 -0400, John Poelstra wrote:
> 
> I think this is a great idea.  It sounds like you are focusing 
> specifically on test plans for these areas?  Would these also result
> in 
> test matrices which are filled out before we release and potentially 
> block the release if broken?

You betcha!  I'm not looking to go through a deep dive of every
application.  We'll start by getting a long list on a white board, then
begin making some cuts for this first pass.  

> One thing that I think is still lacking in our overall release
> criteria: 
> http://fedoraproject.org/wiki/QA/ReleaseCriteria is a more detail 
> surrounding the desktop applications... for example here 
> http://fedoraproject.org/wiki/QA/ReleaseCriteria#Desktop.

I've gone to both extremes with respect to release criteria.  What I've
found works well is the "keep it simple" model.  What we have now is
fairly basic, but I might even suggest something more minimal.  Perhaps
we start by keeping it close to home and in terms of things we all
understand ... bugs.  

 Alpha - 
   * All feature tracking bugs filed (and bound to feature wiki pages)
   * No OPEN bugs with priority >= urgent AND severity >= urgent
 Beta - 
   * All feature tracking bugs in MODIFIED
   * No OPEN bugs with priority >= high AND severity >= high
 PreviewRelease -
   * No OPEN bugs with priority >= high AND severity >= high

>From there, we define some rules on how to assess the impact of a bug
and get it on the appropriate radar.  My preference would be to defer
the decision to individual test leads.

There are two really solid definitions of bug severity out there 
 * from Mark
Cox ... http://www.redhatmagazine.com/2007/04/18/risk-report-two-years-of-red-hat-enterprise-linux-4
 * from
bugzilla ... https://bugzilla.redhat.com/page.cgi?id=fields.html#bug_severity

Using these definitions, we can 

> Do we REALLY do complete "click-through" tests for every application?
> I 
> don't think I've ever seen a test matrix created for this nor do I
> think 
> we want to create one.  I think it would be a better use of our time
> to 
> create a list of the more often used applications (perhaps via
> mugshot) 
> and add those to a matrix for specific testing.
> 
> Another thing we might want to brainstorm about at FUDCon is creating
> a 
> complete list of test matrices... including CD installs, live images, 
> etc. that we need for each release.  I think we overlooked some of
> these 
> during F9.  I think the idea of creating test plans and test cases 
> beyond just installation testing are a great step forward.
> 
-- 
==========================================
 James Laska         -- jlaska at redhat.com
 Quality Engineering -- Red Hat, Inc.
==========================================