What is rawhide for?

Tue Jun 17 17:43:04 UTC 2008

Will Woods said the following on 06/17/2008 11:12 AM Pacific Time:
> On Mon, 2008-06-16 at 10:16 -0400, John Poelstra wrote:
> 
>> 1) An "officially" available version for reproducing bugs or test 
>> results only exists for one day--contrast that with our build 
>> environment that was created specifically with the intention of always 
>> being able to recreate binaries in their original environment.
> 
> The difference between rawhide on any two days is, on average, somewhere
> between 0.1% and 0.5%. If this *blazing* development pace is too much
> for you, you're welcomed and encouraged to keep multiple copies of the
> rawhide repos. 

Rapid amount of change is not the issue.  I'm speaking more to the 
situation of encountering a problem, reporting a bug, being asked for 
more information on that bug, but not being able to troubleshoot further 
because the exact same tree used to install is gone... there is no history.

> It'd definitely be nice if we had some good scripts for maintaining
> multiple local copies of rawhide, with hardlinks to save space. 
> 
>> 2) For most people, the economical way to get rawhide is a mirror--there 
>> is no simple definitive way to determine that you have "the originally 
>> composed version" of rawhide for that day.  There are a few methods that 
>> can give you "reasonable certainty", but nothing like a single check sum 
>> or a way to confirm that the tree you've downloaded is exactly the same 
>> as what was composed.
> 
> This is just plain wrong. The timestamp in .treeinfo is unique
> per-tree. 

Just because .treeinfo is present doesn't mean all the other packages 
associated with it from that day are present on the mirror.

<<CUT>>

>> 4) You never know when it will install or not.  It isn't smoke 
>> tested--we leave that for *everyone* to do themselves based on their own 
>> install attempt.  Even if Fedora hosted an automated smoke test to 
>> determine if it installs you still have the problem of mirrors being out 
>> of sync and not knowing if what you have is the exact same group of 
>> packages that passed the smoke test.
> 
> This is the problem that the rawhide dashboard page is supposed to
> solve. We're automating SNAKE to do smoke tests on rawhide every night,
> and post results on the dashboard. So you *will* know whether it
> installs or not.

Excellent. I didn't realize that was in process.  Could the dashboard 
also contain a historical record of which days install and which days 
don't?  I think this would be useful information to have.

> We've got a prototype rawhide dashboard page that shows whether boot
> images are present, at least: 
> http://wwoods.fedorapeople.org/rawhide.html

snake-treeinfo can tell this too, right?

> As for the mirrors being out of sync - see above.
> 
>> 5) It is the community's only access point for obtaining the (what is 
>> close to) the Release Candidate for testing.  I know several people will 
>> disagree with me immediately here--we've had the argument several times 
>> on IRC.  I still think the underlying assumptions that it is "close 
>> enough" and "most likely the same thing" are not good enough.
> 
> I think you're wrong. I also think you underestimate the time it
> requires to put out an RC versus the turnaround time on testing it. 
> 
> Given the amount of time it adds to the schedule and lengthens the
> development freeze, plus the amount of work it takes from every part of
> the release team, what would we actually *gain* from doing all that
> work? 

I think we can create a more polished release with less critical items 
on the "known issues" wiki page for that release.  I realize the 
"criteria" and "how much time is too much" is somewhat subjective and 
something we do not agree on.

>> 6) We often place a higher value on daily rawhide than the Alpha and 
>> Beta releases by proclaiming that they "don't really matter that much 
>> because they are 'simply snapshots' of rawhide."  The community at large 
>> seems to focus more on the Alpha, Beta, and Preview releases as 
>> evidenced by spikes in traffic on fedora-test-list after these releases.
> 
> I think you're mixing up cause and effect here. 
 >
> People don't focus on the milestones because they just inherently *love*
> milestones. They focus on the milestones because we go to the effort of
> making them easy to consume, and we publicize them and test them to
> ensure they'll be installable and put up ISOs on torrents and the
> mirrors. So *obviously* more people use them.

I was attempting to respond to past situations when the alpha and beta 
haven't been in that great of shape and it has been said that it doesn't 
matter that much because "beta == snapshot of rawhide".

> 
>> 7) We consider rawhide our primary testing target yet there is no 
>> practical way to create a test matrix around it because it changes every 
>> day.  Instead we create test matrices for the Alpha, Beta, and Preview 
>> releases which..... see the previous point.  How do we know when we have 
>> completed a full test run?  How can you thoroughly test a moving target?
> 
> I completely disagree with the assertion that "there is no practical way
> to create a test matrix for Rawhide". 
> 
> First of all, the only reason we don't create test matrices for every
> day's rawhide is that it's time-consuming to create the matrix itself.
> The old wiki's pretty slow, remember?
> 
> Better test run tracking (with Testopia, for instance) will make it
> trivial to create a matrix for every day's rawhide. Further, automating
> our test cases will let us fill in most of that matrix automatically.
> 
> The test plan for rawhide is, obviously going to be somewhat less
> exhaustive than the test plans for milestone releases or the final
> release. So we can complete a "full test run" by looking at the
> (mostly-full, auto-created) matrix for that day's rawhide and.. filling
> in the blanks.
> 
>> that doesn't mean one of Fedora's goals has to be 
>> emphasizing a testing process that is flawed and could be better if we 
>> all put our collective brains together to come up with something better 
>> :-)  We innovate in so many other areas... why not innovate here?
> 
>> I would like to advocate that we reconsider the value we place on 
>> rawhide and the emphasis we place around the Alpha, Beta, and Preview 
>> Release.  
> 
> Innovate.. by emphasizing the traditional milestones? I think you'll
> find that "innovation" and "tradition" are actually *opposites*. 

Innovate by taking a different approach we haven't considered before.  I 
didn't suggest that "traditional milestones" are the answer.

> Rawhide might be a strange testing target from the point of view of
> someone coming from RHEL or proprietary systems or other traditional
> milestone-focused development, but I don't think Rawhide is that strange
> in the Open Source world. 

I'll give that some more thought.

> You can apply nearly all of your arguments to, say, the way the kernel
> is developed - it changes too quickly! Not enough freeze time! We need
> more milestones! You can't possibly test it!
 >
> You know what would *really* be innovative? Engineering our test efforts
> to match the pace and reality of typical Open Source development, rather
> than working the other way around.
> 
>> I think a good place to start would be document in our test 
>> plan where using rawhide for test results makes sense and where it does 
>> not.  I believe rawhide does have its place, but I think we are trying 
>> to use it to cover too many bases and could do more effective testing 
>> with a more refined approach which in the end makes Fedora better!
> 
> Okay. So start documenting where you think it does and doesn't make
> sense and we'll discuss *that*. But so far all you've done is rehashed a
> bunch of complaints and offered no solutions.

I guess that is the tricky part. I'd like to think that being a 
contributor to Fedora does not mean you always have to have the solution 
to raise what you perceive as something that is wrong.  Hopefully my 
contributions in other areas Fedora make up for it here.

>> What do other people think?  Is there something here worth throwing 
>> around here on this list with a following up discussing at FUDCon later 
>> in the week?
> 
> Sure, if we go to the effort of actually *defining* problems and
> discussing *solutions*, it's totally worth it.
> 

I agree.

Or if the problems are misunderstood then we need to do a better 
explaining why our current approach really does make sense--I am not the 
only one who has echoed these same questions.  Maybe someone else can 
explain the issues I'm trying to get at in a better way?

John