Expectation Management for Test Releases

[Bill Diamond]

On Tue, 2004-04-20 at 13:05, Keith Lofstrom wrote:
> This weekend, at the party following LinuxFest Northwest, I found myself
> across the table from an avionics software project manager at Boeing. 
> He told me of some of his problems with Fedora.
> 
> He said he had big problems running Fedora Core 2 Test 1;  he was not
> mollified when I tried to point out that one can *expect* huge problems
> trying to run test releases.   He insisted that it was entirely
> unprofessional to release software that wouldn't even install, and that
> using the FC2T1 upgrade process on top of a working FC1 system made a
> real hash of it.  He had performed alpha and beta testing on Windows
> and never had such problems. 

Keith,
You could say I have feet planted firmly on both sides of the fence. 
There's no ethical way that I can see to release software without full
exposure to uncontrolled environments. The most valid, useful feedback
for stability, performance, and usability per task comes from beta
testers who are generally unrelenting in their bluntness.

Release Candidates are probably what your acquaintance from Boeing is
familiar with.  Those aren't betas.  They're feature locked, meant to
influence opinoin and generally have performance monitoring code left in
the system.  

In my day, betas were very tightly controlled at Microsoft.  What most
customers perceive today as a beta is a release candidate. 

Now, as a director of marketing for a software company, I'm often found
begging for beta testers.  Real, old school, mean as a snake and rude as
a fart in church beta testers.  They're hard to find, and really hard to
continue to attract.  The pre-boom (and pre-FASB changes) that allowed
us to give our software as free gifts have killed a lot of the
participation.  The increasing complexity of software and generally poor
management of beta programs has frustrated the real quality testers.

Modern beta testing has been taken over by professional beta testers.  
While my experience with Fedora has been very uneven, there is a heck of
a charge out of getting a problematic system to install and run.   I
don't think there's as good of monitoring and trace/reporting tools in
Fedora that would really help Red Hat and the Fedora Project understand
the behavior of the code "in the wild".  

That doesn't mean that I enjoyed seeing my very stable FC1 system wiped
out by an errant FC2T2 install.  But, it's part of the risk of what real
beta testing is.  Sure, I have a vested interest in participating.  Red
Hat is my preferred linux distro and I want to see Fedora succeed.

So, I participate.  What I think we need is a return to basics, an
education on what it means to beta test and we need old school folks to
run these programs again to make them valuable and to attract qualtiy
testers and to train the next generation of code curmudgeons.

Bill