Linux examined: Fedora 9

[Jon Stanley]

2008/5/15 Paul W. Frields <stickster at gmail.com>:

> I think something our QA folks (Will et al.) should think about is a way
> to expose more details about the test configurations they use for
> banging on the installer.  These could be exposed to the media for our
> Alpha or Beta test release, as well as our larger user base, to make
> sure that people are expanding on those configurations, rather than
> duplicating them endlessly to the exclusion of others.

Our test matrix is available here:

http://fedoraproject.org/wiki/QA/TestResults/Fedora9Install/FinalRelease

Admittedly, we need A LOT more detail on the tested hardware in these,
but these are the high-level test cases that we use. Note the absence
of dual-boot systems in there - that very probably is a gap in our
test coverage, but all of these are manual test cases, since we
currently have no viable automated test harness, and every one of
these tests take time.  A dual-boot test would take even more time
(which is, unfortunately, a finite resource) A lot of testing is also
done in KVM/QEMU/VMWare (yes, that's me using proprietary software,
but given our usage profile, I think that it's essential test
coverage), and not on real hardware.  I know that one of the proposals
that we were floating in the QA team was requiring smolt profiles for
each test case, so that we could tell how it was tested and that the
test succeeded.  We'll probably be talking more about this internally
in QA in the future, no decision has yet been made to my knowledge
(but I think it's an excellent idea).