What is rawhide for?

[John Poelstra]

Rawhide serves several purposes in Fedora and yet I don't think we 
clearly delineate what they are.  It is often said in Fedoraland that 
the best way to test the Feodora is to "run rawhide."  There is a 
difference between installing and running rawhide to experience the 
absolute latest and greatest Fedora has to offer and intentionally 
testing it.  I just think we need to be clearer about our approach and 
use of rawhide for testing (specific QA) because sometimes it doesn't 
make sense.

Take a step back and consider what a strange testing target rawhide is:

1) An "officially" available version for reproducing bugs or test 
results only exists for one day--contrast that with our build 
environment that was created specifically with the intention of always 
being able to recreate binaries in their original environment.

2) For most people, the economical way to get rawhide is a mirror--there 
is no simple definitive way to determine that you have "the originally 
composed version" of rawhide for that day.  There are a few methods that 
can give you "reasonable certainty", but nothing like a single check sum 
or a way to confirm that the tree you've downloaded is exactly the same 
as what was composed.

3) There is no "last known good version" when things completely get 
messed up and you need to reinstall from scratch except the Alpha, Beta, 
and Preview Release.

4) You never know when it will install or not.  It isn't smoke 
tested--we leave that for *everyone* to do themselves based on their own 
install attempt.  Even if Fedora hosted an automated smoke test to 
determine if it installs you still have the problem of mirrors being out 
of sync and not knowing if what you have is the exact same group of 
packages that passed the smoke test.

5) It is the community's only access point for obtaining the (what is 
close to) the Release Candidate for testing.  I know several people will 
disagree with me immediately here--we've had the argument several times 
on IRC.  I still think the underlying assumptions that it is "close 
enough" and "most likely the same thing" are not good enough.  I think 
we have been lucky so far and there was a situation with F9 where the RC 
contained a different package version than what was in rawhide.

6) We often place a higher value on daily rawhide than the Alpha and 
Beta releases by proclaiming that they "don't really matter that much 
because they are 'simply snapshots' of rawhide."  The community at large 
seems to focus more on the Alpha, Beta, and Preview releases as 
evidenced by spikes in traffic on fedora-test-list after these releases.

7) We consider rawhide our primary testing target yet there is no 
practical way to create a test matrix around it because it changes every 
day.  Instead we create test matrices for the Alpha, Beta, and Preview 
releases which..... see the previous point.  How do we know when we have 
completed a full test run?  How can you thoroughly test a moving target?

I know I will hear the stock replies of "but it is rawhide" or "this is 
Fedora, not an enterprise distro like RHEL" and while I agree that 
"rawhide is rawhide" (a circular, but well understood argument for long 
term rawhide veterans) and that Fedora is not striving to be an 
"enterprise distro" that doesn't mean one of Fedora's goals has to be 
emphasizing a testing process that is flawed and could be better if we 
all put our collective brains together to come up with something better 
:-)  We innovate in so many other areas... why not innovate here?

I would like to advocate that we reconsider the value we place on 
rawhide and the emphasis we place around the Alpha, Beta, and Preview 
Release.  I think a good place to start would be document in our test 
plan where using rawhide for test results makes sense and where it does 
not.  I believe rawhide does have its place, but I think we are trying 
to use it to cover too many bases and could do more effective testing 
with a more refined approach which in the end makes Fedora better!

What do other people think?  Is there something here worth throwing 
around here on this list with a following up discussing at FUDCon later 
in the week?