How to do QA

[Eric Rostetter]

Quoting Michael Schwendt <ms-nospam-0306 at arcor.de>:

> On Fri, 06 Feb 2004 02:14:19 +0100, Jonas Pasche wrote:
>
> > I have put together a page that might help people doing QA:
> >
> > http://jonaspasche.de/fedoralegacy/qa-howto.html
>
> In such a detailed form this goes into a wrong direction IMHO. Some

Unfortunately, Jonas and I are starting the only place we can.  We don't
know the top from the bottom, so we can't start either place.  So we
start with the fedora.us page, as was suggested many times on the mailing
list as the place to start.

>   Disclaimer in front: This is no attempt at stopping anyone from trying
>   to create a "how to do QA" document, although I think there's no general
>   recipe on how to do QA.

I think we need the doc.  I agree there is no standard way to do QA.  But
we need to provide something to get people started.  Right now we don't
have enough people doing QA.  The reason, according to those who speak
up, is they just don't know how.  So, either we fail to publish packages,
or we need to write such a how-to.  I don't want to fail in our mission;
to publish packages.  So, that means we need to provide such a document,
how ever nieve it may be.

Hopefully we can add enough text to suggest to the reader that it is
not a cut and dry follow these steps process.  But at the same time,
we need to push new members of the community in the right direction.

In addition, there has been a desire by the lead members to have a formal
(documented) set of policies and procedures.  These web pages will fill
that need, which until now is filled only be the mailing list archives,
which is not a convenient source for locating policy.


> If there are people who find such documents
>   useful and if it helps them actually to get started with package reviews
>   and approvals, I shut up.

There are (myself, maybe Jonas, and a few others who have spoke up on
the list).  But that doesn't mean you should shut up.

> At this time, however, I'd like to avoid that
>   the "QA hurdle" is set too high or that it is considered as set too
>   high.

Right now the hurdle is set too high.  The hurdle is people don't know
what to do.  The result is they do nothing.  Anything to help people
get involved can only be a Good Thing.

> Apart from that, copying some things from the fedora.us checklist
>   simply doesn't make any sense.

We've had no real other place to start.

> I like to point out that people who are new to reviewing packages should
> follow a top-down approach to testing packages (and build teams with
> people who look a level lower) instead of a bottom-up approach which
> starts with low-level technical package reviews.

That's probably true, and sounds like it is reasonable or plausible to
me, but at the same time I really have no idea what exactly it means...

> If Fedora Legacy wants to
> provide updated packages which fix security issues or critical bugs, the
> primary objective is to provide packages which fix those issues and which
> don't break anything else.

Correct.

> This requires testing the built binary
> packages.

Yes.  But someone has to build them also, before they can be tested.  And
they must be pre-tested for things like trojans, etc.

> If no binary packages are provided, it may be necessary to
> rebuild the source rpms. Like Eric Rostetter's page, an introduction to
> testing packages should start with stuff like how to rpmbuild a src.rpm as
> non-root user, how to extract a src.rpm, how to create an rpm diff against
> the previous src.rpm, and maybe later continue with additional checks at
> the level of the source rpm or binary rpm (e.g. ldd, rpm -qlvp and rpm -qR
> checks), meeting the requirements of a build system (=> completing the
> build dependencies).

That's good food for thought for the page(s).

> However, Fedora Legacy aims at modifying the source
> rpms as little as necessary, so reviewers don't generally need to spend
> any time reading the spec file beyond the diff against the previous
> release.

I think there are two levels of reviewers.  The top (expert) level needs
to check everything.  They make sure there are no trojans, that the
changes don't introduce additional bugs, that the spec file is correctly
updated, that the actual vulnerability is fixed, etc.  The bottom level
(normal mortals) test that the package builds (optional), installs correctly,
seems to work when they use it, works with other packages that interact
with it, etc.

Right now my draft QA docs don't treat these two groups as separate, but
I'd prefer that they do at some point.

> A checklist makes the package verification procedure appear overly
> complex, complicated [and maybe even pedantic].

Yes, but we need testers, and we don't have them.  So we need to provide
people with the info needed to become testers.  And part of that means
checklists...  They are not the end-all-be-all, but they are a start.

> Or people go through the
> list step by step and don't examine anything else.

Then we need the list to tell them (by implication) what else they
need to do besides what is explicitly on the list.

> Fedora.us has different
> requirements because it deals with entirely new package build scripts,
> which are often written from scratch or sometimes derived from other
> distributions. Sometimes a packager doesn't even take the time to
> understand what is done in the package build script and whether it does
> what it is supposed to do. Such circumstances require a different level of
> reviewing package submissions.

Which is why Jonas and I put our work up for comment and correction, rather
than publishing it on the fedoralegacy.org site.  We *want* input and help.
We want to be told what to remove, what to add, what to change.

> previous release?). Many of the steps in the linked/copied fedora.us
> checklist don't apply at all to Fedora Legacy.

Great, but which ones?  Please review my page at

http://rashi.ph.utexas.edu/docs/QA.php

and let me know what *doesn't* belong (as well as anything you think needs
to be added, if anything).

> I feel that starting with
> this list as a basis is approaching the "QA problem" from the wrong
> direction. Start at the top and refine the procedures and policies.

Again, we (Jonas and I) have no idea of what top or bottom is here.
We simply know that people, including ourselves, need help, and we're
trying to provide it to the best of our ability with what we have
available to us.  And what we have available to us, at this time,
is the fedora.us information...

--
Eric Rostetter