what needs to be tested at VERIFY

[Pekka Savola]

On Thu, 3 Mar 2005, Jim Popovitch wrote:
> It is NOT anywhere near easy.  Not by a long shot.  Thus the reason for
> my speaking up.  There really needs to be a list of:
>
>   What was broke
>   The proposed fix
>   Who fixed it

These should be available at the Fedora Legacy Test Notification mail? 
(the middle part, if you want it in unidiff, is available from 
comparing the sources.)

>   What should be tested

That's a good point, below.

>> Currently there are about 16 packages waiting for one or more VERIFY
>> votes.  It would be really useful to try help with these.  VERIFYING
>> is really trivial if you have ever used the application in question,
>> and you have access to the right RHL/FC versions.
>
> I do have access, I have "verified" packages (or have I? who decides?),
> and I probably will continue at some level.  The problem is FL needs
> more than just the current people and process, and what it being tested
> needs more rigorous (think: ISO 9001) guidelines and procedures.

That's exactly the problem, I think.

Different people have different things in mind when they think about 
"testing".

Mine is:
  - see that it installs
  - see that basic functionality works (5 or 10 minutes is enough)
  - if you are able to test the exploit, it's bonus but not necessary.

It seems clear that there is _NO_ way the project has resources for 
any _REAL_ QA testing.  So, we'll just have to be content with 
checking for the obvious issues (IMHO).

Otherwise we get nothing done, and that would be much worse.

>> For how to do VERIFY testing, take another look at
>>   http://www.fedoralegacy.org/wiki/index.php/QaTesting under
>>   "Testing packages for release to updates".
>
> That URL is a start, but there is no second step after it.  Everyone has
> read the above... where's the next chapter?

What would you like to see in the next chapter?  Discussion on what to 
test?  Something else?

> As I said;  It is NOT anything close to simple, and even then it is open
> to too much discernment.  A *kernel* has been released recently..who
> tested it in it's entirety?   Where is the list of fixes and the
> checkmarks that indicate "x" was fixed by Bob and "x" was tested by Sue?

>From above, I don't think the project has resources for this kind of 
testing.  It's IMHO enough to get a "warm fuzzy feeling", like "I 
installed it, the app still works, nothing seems to have broken."

There's no way we could do any more than that.  Besides, we're already 
using the patches from sources which have already had some QA (e.g., 
RHEL, Debian, ...) so they should not be typically broken.

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings