Re: Help with distribution choice

[Matt Drew <mdrew redhat com>]

Okay.

> First of all , i have no reason to start a flamewar, as you would have
> noted after reading the message my criticisms were contructive and well
> founded, I for one are glad you responded because, since you are from
> redhat you really have a moral obligation to answer questions and
> criticisms which are constructive of course and not flame mail.

Fine, I'll ignore the Rathead comments. :)

> > since we try to focus on that sort of thing, we necessarily are not on the
> > cutting edge (currently defined in my mind by running a 2.3.x or 2.4.x
> > kernel, among many other things).
>
> Well at the time of the release of Rh 7.x you were certain on the cutting
> edge.

No, we weren't.  We included a very stable 2.2.16 kernel, and held off on
many things that were still iffy in the 2.2 series (ATA 66/100 support,
for example).  There were some things that we pushed (xinetd) because we
believed they were better and more stable than their predecessors, and we
considered them mature enough to replace the older software and move to a
better overall situation.

This makes me curious though.  What would you consider specifically to be
on the cutting edge?  (Anyone feel free to answer this -- I really am
interested in hearing about this).

> >We have backported some things at the
> > urgent request of our partners, such as USB.
>
> Things which RH should have thought at the time of the original release
> instead of rushing to include unstable development releases of major
> software.

Which would be ... let me see:

BIND 8.2.2  (not 9.x)
Apache 1.3.12 (not anything mind-blowing there)
XFree86 3.3.6/4.0.1 (a little racy, but the older servers are there)
wu-ftpd 2.6.1 (stable)
glibc-2.1.92-14 (updated to 2.2-9 recently, racy but had to be done)
gcc-2.96 (which our guys say is much better than 2.95.2 -- YMMV)


> > Personally, I've been running six different machines for three months on
> > 7.0, and haven't had any problems with installation or stability (other
> > than one kickstart issue which was my fault). I would say that (in my
> > biased, Red Hat employee opinion) that 7.0 is, with all of its faults, one
> > of our better releases, and certainly better than 6.0.
>
>
> I dont think so, I am sure many old timers here would agree that rh 6.x
> series was much more stabler and better (you yourself recommened the 6.x
> series as a stable release and its true).

I actually recommended 6.1 or 6.2 if my memory serves me, and that is what
I meant.  I would not recommend 6.0 to anyone these days.  Since 6.2 was
the last release in the 6.x series, it is the most stable, and
developement on that platform can move to 7.x relatively easily (which is
why Alan recommends it).

>
> >  Do I think we
> > succeeded in all of our goals?  Hardly.  At least, however, we aren't
> > afraid to fail, since we seem to be pretty good at getting up, dusting
> > ourselves off, and moving on.
>
> Yes, but you should try to learn from mistakes and not trying to rush
> through with shiny new software without thinking of its implecations, why
> couldnt RH wait a small time and do some more testing on rh 7.0 and
> release it.

At some point, you have to call it and release it.  We release early and
often, just like most open source software.  There is a certain amount of
roughness that comes with that, regardless of who you are.  Do I think we
did the best job we could with 7?  No.  Do I think we did the best job we
could in the time we had? No.  Do I think we learned from that experience?
Try the beta and then 7.1 and you decide.  Again, I'm curious about what
the "shiny new software" would be, since I didn't see a whole lot of that
in 7, actually (excepting gcc, of course).

> considered downloading it through my measly 56K modem, but I decided to
> watch and wait, and when I learnt more about what was included and the gcc
> 2.96 and xf86 4.0 inclusion, and reading all the trouble and bugs that
> people were experiencing, I was very disappointed and felt letdown, it
> looked like RH was releasing such untested brand new software because it
> wanted to try to knock out the competition (mandrake and debian et al),
> and RH 7,0 had all the hallmarks of a rushed through release without
> adequate testing because of commercial reasons :(

So...you've never tried it?  Everyone knows about all the problems we have
with every release -- part of being open source.  From my point of view,
things have been relatively quiet, even on bugzilla and the news groups --
lots of people complaining, but very few concrete problems.

> So one day one of my friends urged me to try debians potato, and I
> installed it and it was astounding, for the first time in my life I could
> update everything automatically and painlessly, and the config file layout
> was very logical. The updating is the strongest point because I dont have
> to search for new versions etc, when new versions are released and put on
> the debian ftp site by the maintainers all I have to do is to execute a
> "apt-get update && app-get upgrade", and lo and behold all the necessary
> packages would be updated :) So with such a good and stable linux I
> decided not to touch rh 7.x at least till it becomes more stable and the
> bugs are sorted out.

If you haven't tried it, I can't speak to that.  Your call.  RHN can do
pretty much the same thing (not as well yet), and a lot more.  Stay tuned.
:)

> What exactly were you hoping to get by XF86 4.x, true there are issues
> like DRI, and 3d support which make xf86 4.x a attractive option, but were
> these so great as to warrant such a major and risky jump instead of
> sticking to XF86 3.x.x.

Because our customers wanted 4.x .  We shipped both versions to try and
satisfy as many people as we could -- without the SVGA bug (which was
unrelated) we would've been golden.  Aside from the gaming market, many
people are utilizing 3D functionality, and we felt that it was important
(I personally agree -- 4.x is a much better platform for that).

> I did not read it on slashdot, but after you said so above I visited it
> and saw the mail from the gcc steering comitee and realised that RH were
> far more stupider than I thought :(

Thanks.:)  Try this:

http://lwn.net/2000/1005/a/rh-tools.php3

Which is Richard Henderson's posting about the reasoning behind 2.96.
Pretty good, I think, and I didn't expect anything else from the steering
committee, since that is their job.

I think the confusion arises from perception.  In our minds, Red Hat is
pretty far from the real bleeding edge (ATA 100 support in Fisher,
finally).  However, from a corporate perspective, we are on the "corporate
bleeding edge" for lack of a better phrase, since we, like all other
distros, constantly release new and updated products at a bewildering
speed to those used to Microsoft's much longer release and patch cycle (it
hasn't been regression tested yet, but...).

> > The choice was:
> >
> > 1)ship a broken C/C++ compiler (2.95.2 or earlier) that was an official
> > release and deal with the consequences.
>
> Well what it soo wrong with gcc 2.95.2? You are forgetting that this so
> called "broken" gcc-2.95.2 is used on sevral major linux releases, for
> example my debian potato too has this "broken" compiler in it,

2.95.2's C++ API is incomplete, and allows broken code to compile (see
Richard Henderson's post).  Talking with the gcc guys I know, most of them
stay away from 2.95.2 these days, because of all the problems, especially
on non-x86 platforms.

> kalum spartacus:~$ gcc -v
> Reading specs from /usr/lib/gcc-lib/i386-linux/2.95.2/specs
> gcc version 2.95.2 20000220 (Debian GNU/Linux)
>
> I have been using 2.95.2 for a long time to compile kernels, and hundreds
> of other source packages and I still have to find a situation where it has
> let me down, actually the gcc maintainers advice you to use 2.95.2 if you
> have any doubt over using a development version, a advice that RH should
> have stuck to IMHO which would have avoided this unpleasant stuation.

It was a lot less unpleasant than it would have been if we had shipped
2.95.2, because of all the issues there.  SuSE shipped 2.95.2-117 with
their 7.0 product; have you ever seen a package with a 117 minor number?
Even worse, all of that effort was wasted, since 2.95.2 is a development
dead end.  I can't speak for this -- I don't do much compiling.  But I
would tend to believe the gcc guys who work here, since they work with the
code day in and day out.  Source packages are, of course, going to
generally work, because they've been tested before release -- that isn't
an accurate estimate of the problems of a compiler.


> > 2)ship a snapshot of the current development that was less broken, and
> > deal with the consequences.
> >
> > We chose number 2.  It was a decision that was not made lightly.
>
> Not lightly but hastily I think, did you contact the gcc comitee about
> such a inclusion to a mojor distribution and possible adverse effects that
> could arise due to it?

Not hastily either, and with the full knowledge and support of many gcc
developers.  We (apparently, I don't know) did not think that approval
from the committee was required, at least not formally.  As it turned out,
that was a bad idea.  A mistake, as I've said before.

> > The only real beef was with the unfortunately chosen name, which was dumb
> > and which we have apologized for.
>
> Apology expected, but how could you deal with the following problem,
> "Current snapshots of GCC, and any version labeled 2.96, produce object
> files that are not compatible with those produced by either GCC 2.95.2 or
> the forthcoming GCC 3.0.  Therefore, programs built with these snapshots
> will not be compatible with any official GCC release."

However, 2.96 is source-code (API) compatible with 3.0, and 2.95.2 is not.
If your code is built on 2.96, all it takes a recompile (excepting any ABI
issues) and you are good to go on 3.0, which cannot be said about 2.95.2.
Since 2.95.2 object code is not backwards or forwards compatible *either*,
we aren't losing much and gain source-code API compatibility by switching
to 2.96.

> > through any problems that they encounter.  Judging from your posts here
> > (and the fact that you still subscribe) I would say that you also fall
> > into the category of wanting people to have a good experience with Linux,
> > and Red Hat Linux in particular.
>
> Well I certainly hoped to have a good experience with rh 7.x because of
> the lovely 6.1 experience (a memorable release that I really salute you),
> but 7.x bought a lot of disapointment.

So...you did install it?

> The reason I am still on this list is that, not all of the questions on
> this list are RH specific, and that I think I can still help someone in
> there problems, since I have gone though them myself and learnt solve them
> the hard way.

Ditto, and good man.

> > I don't know of any statistics on redhat-install-list, but there were 153
> > bugs submitted with "rpm" in the summary for all versions of 6.2 in
> > bugzilla, and 128 bugs submitted with "rpm" in the summary for all
> > versions of Red Hat Linux 7.  Considering the transition from rpm 3 to rpm
> > 4, I'd say that is pretty good.  It could also mean that people are
> > digusted with Red Hat and are no longer submitting as many bugs -- a
> > possibility (although I would disagree).
>
> But you would have to consider it as a possibilty.

Yes.

> > The reason for the release of Red Hat Linux 7 was the same reason for the
> > release of every other version of Red Hat Linux -- we release about every
> > six months, and have obligations and reasons for doing so.
>
> Yes, but then why does RH 7.x appear (not appear it actually is)  to be
> under tested, hastily pushed through without considering implecations of
> new software etc..Surely a few more months of testing could have been
> done, or you could have stuck with more stabler releases of the software
> that you were including if you did not have time for testing.

It did appear that way to me, but then again all Linux products do -- spit
and polish is a low priority across the board in my experience.  It sure
was a lot easier to install and work with (IMHO) than 6.2.  Again, in
hindsight there were some things that we should've caught, and we've
admitted that, and we are working to make things better, which is about
all one can really ask for.

> >As a supporter
> > of open source products, I'm proud that Debian is doing well, and I hope
> > that Red Hat can learn from that and include it in our own distribution.
> > After all, that's what it is all about, right?
>
> Yes exactly, true there is commercial pressures and commitments, but in
> the end its the user that matters the most, and he should get a good deal.

And when the vast majority of your paying customers users are commercial
entities, they matter -- not less than all of our other users, but they
matter, and we have to take them into account.  We also have to take into
account the burgeoning market of new users, who have computers with modern
features that they believe (hell I believe)  Linux should support. *cough*
(USB).

> Or move to much better designed and stable distribution like Debian or
> mandrake etc..

If that is what you want.  We offer a product to the market -- the market
decides whether or not it is good.  I recommend to anyone to choose the
distribution they like the most, but at least to give us a chance and try
us out.

> Yes but what I dont understand is your contradictory claims, you say that
> RH 7.x is stable and reliable etc, then when later you comment about

No, I said we strive for stability and reliability, but we must also
advance quickly to support our customers, and balance those things.  We do
not go all out and run with the latest and greatest, but neither do we
always stick with the tried and true at the expense of the new and usually
vastly improved.

> potato you say "The other is called Potato, and is the most stable version
> of Linux that you can find.  However, there are many features that most
> people would consider key that are missing from Potato because they aren't
> stable enough yet to be included.", thus implying that RH 7.x has included
> unstable software in it thus invalidating your previous claims of
> stability etc...why this?

...stable enough to be included in Potato, not Red Hat, since Potato's
overriding goal is stability and reliability. Since Debian offers Woody
for those interested in new features, it isn't a big deal.  We, however,
only have one distribution, so we do the best we can. And yes, stability
and reliability are primary goals for us, but not the only goals -- and
sometimes, a little pain now can alleviate a lot of pain later, so we try
to smooth the way for upcoming development as well.  Who knows?  Maybe
we'll move to the Debian way of doing things, although I doubt it.  Time
will tell.

> If you dont criticise something you never will make it
> better..constructive criticism is the best way to improve things.

The problem I have with this is that I don't see specific criticisms in
your email.  I see "2.96" this and "XFree86 4.x" that, with no bug reports
and no details.  Since I work for support here at Red Hat, I've seen
pretty much everything there is to see with this release (please god let
them get the 3.3.6 SVGA server fixed) and from my experiences the 7
release was pretty okay, but far from perfect.

> not me alone but a large amount of people, if you walk around the usenet
> you can find a huge amount of people disappointed with redhat going the
> microsoft way, it was the last thing that many of us thought would happen
> and unfortunately the wheels have started grinding already.

And again, with no specifics and no valid reasoning other than "I saw on
Slashdot..." and the like. If Red Hat folded every time someone bitched
about our product we would've disappeared a long time ago.

We put our professional pride, our fortunes, and our careers on the line
every day (who would hire a developer from a failed OS company?).  I tend
to respect similar people's opinions much more than the vague commentary
on Slashdot from people that I don't know, and who seem to know very
little about what is really at stake and the decisions that were made. I
respect the opinions of people who actually work and help the community
fix and improve Linux.  I respect the opinion of those who help new users
get started (which is why we are having this discussion in the first
place).  I'm very disappointed (and, as you can see, a little tweaked)
when someone second-guesses the decision of the guys on the ground doing
the work and the coding, especially with comments like "Rathead",
"Microsoft of Linux", and "stupid".

To quote one of my favorite authors, Richard Marcinko -- "The path to
glory is littered with fuck-ups."  Red Hat must not be afraid to fail,
again and again, because that is the only way that we can truly produce a
product that shines.  Constructive criticism is very useful, but vague
diatribes simply insult our judgement and dedication.

> But it is RH's duty to try to minimise bug reports by *adequately
> testing out there software before releasing it*, ie it is obvious that
> including developement or brand new versions of software would include
> much more bugs in the distribution, so why do that and palm the buck
> off to the user to discover the bugs, when you could either do that
> yourselves, or include not so new stable versions of software in the
> first place.

> I would be happy if you responded to this since a couple of key issues
> have been mentioned here, but if you think your time is better spent doing
> something else then its fine by me, I too have programming to do and I
> understand how time is precious.

I think this is important enough to answer, and I welcome further
conversation on this.  Have a good weekend everyone.

Matt

-- 
Matt Drew
Executive Officer
Red Hat Consumer Services