Managing Fedora Testing

[Keith Lofstrom]

Some folks attempted to run Fedora Core 2 Test 2, and had an extremely
difficult time, and expressed that here.  They were roundly castigated
by others for expressing that frustration.

The purpose of the test process is to get a lot of different eyes on
a lot of different things.   The test releases are a good place to look
for bugs before they are enshrined in the release.  It is a very bad
place to play control games and squelch dissent.  Consider the "n" key
instead.

I am not a programmer.  I do, however, suffer from bugs and mis-features
that sail through the testing process into releases.  I *do* understand
testing stuff.  Because I am skilled at testing, I would like to
participate, and help isolate bugs, and describe them so that real
programmers can fix them.

I realize that pre-alpha releases are hard to get functioning.  But
I also understand good and bad test procedures.  If you are going to
test a product with 4000 modified components, you don't throw all your
pre-alpha code together in one place, then ask people to test it,
because they will only detect errors that crop up early.  What that
means is that you thrash out the problems in the first 100 components,
perhaps, and the latter 3900 components are mostly untested.  

In electronics engineering, when we want to test a new component design,
we plug it into a network of proven components.  That way, we can
quickly isolate the source of a problem.  When we combine too many new
components together in an attempt to save time on testing, the result
is usually the opposite.  Fault isolation becomes nearly impossible,
and the engineer in charge of component B spends all her time arguing
with the engineer for component A.  The technical term for managers
that set up such test programs is "f***ing idiots".  Anyone who has
worked in industry knows what I'm talking about.

If I am a volunteer distro tester, and want to look at how a new Mozilla
interacts with a new X windows, I do NOT need to be spending time
finding out that Anaconda can't find my IDE hard drive, or that yum is
configured to use the wrong mirrors.  I have limited time, there are
a limited number of testers, and there are a large number of packages
and an enormous number of interactions to look at.  If I quietly give
up in frustration, or am driven away by the rantings of arrogant twits,
then the kinds of things I can test go untested.  

Probably 50% of the people that attempt to participate in the Fedora
test process lack the skills to test effectively.  Of those, probably
half can be patiently taught to do a good job, and the other half
should be gently led towards contributing in other ways.  Of the folks
that *do* have the skills to test things, perhaps 10% have the necessary
skills to test *anything*, or to test advanced components in spite of
the basic components being broken.  The other 90% of the 50% can perform 
adequate testing but only in favorable circumstances.  At least, this
is my experience from other projects.  Breaking it down:

   5% can test almost anything in difficult circumstances
  45% can test some things under favorable circumstances
  25% can learn to perform effective tests but don't know how yet
  24% are unable to test but can do other important tasks
   1% are brain dead and should go play Windows Solitare instead

I would put myself in the 45% group, and include many of the folks that
are complaining about booting or mirror problems.  The Fedora test process
seems to be geared exclusively for the 5%.  As a result, Fedora will
release with less than 10% of the potentially available testing effort,
with the remaining effort disproportionally focused.

Well, the proof will be in the results.  Perhaps the high quality of
the 5% will overcome all odds and turn out a quality product.  Or 
perhaps Fedora Core 2 will be a buggy piece of crap.  I wish you all
luck, because in spite of the nattering twits, there are some truly
wonderful people working on Fedora and at Redhat, doing great work,
and I would be truly heartbroken to see all that talent wasted.  

Keith

P.S. The prescriptions would come here, but this post is already too
long.  I can provide those in a later post, if there is interest.

-- 
Keith Lofstrom           keithl at ieee.org         Voice (503)-520-1993
KLIC --- Keith Lofstrom Integrated Circuits --- "Your Ideas in Silicon"
Design Contracting in Bipolar and CMOS - Analog, Digital, and Scan ICs