crazy thought about how to ease QA testing

[Pekka Savola]

On Fri, 10 Feb 2006, Mike McCarty wrote:
> Jesse Keating wrote:
>> On Fri, 2006-02-10 at 17:54 +0200, Pekka Savola wrote:
>> 
>>> So, instead of adding more hoops ("please, install a virtual image of all 
>>> the other distros and do verify testing etc. there"), most focus should be 
>>> put on making participation easier. 
>> 
>> 
>> I am trying to make it easier.  I'm trying to make it so that people
>> don't have to use their production systems as package fodder for our QA
>
> That's the primary reason I haven't volunteered for testing for FC2.
> I have an FC2 machine which I don't want to clobber. Isn't that
> the *reason* Legacy exists? We don't want to clobber our machines.

I'd suspect most folks should have dozens if not hundreds of systems 
running, and are willing to experiment with a couple of them (or have 
a couple of experimental boxes set aside) in order to get the tested 
updates shipped to the rest of the systems once the updates have been 
approved.

Jim Popovitch wrote:
> I have to agree with Jesse, there is no way automated testing will 
> work.  There are just too many differing issues from patch to patch.

Jim, you're probably missing the fact that VERIFY QA doesn't include 
the steps "test if the patch worked; test if the vulnerability is 
fixed".  While some folks do perform more rigorous testing, it's not 
required, and for a good reason.

Which one is better, not shipping any updates at all (or after months 
and months of delays), or shipping "looks good" updates quickly and 
fixing them (if issues come up) even faster?

Aiming for perfection doesn't cut it.  Contrary to common beliefs, FL 
doesn't have the resources for thorough testing that some vendors have 
the luxury of.  That's why we employ those vendors' fixes directly :-)

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings