Re: A criticism of RedHat's strategy

[Charles Hixson <charleshixsn earthlink net>]

Scott Doty wrote:

> Today, I got up early and decided to check out the story with Red Hat.  Good
> comments from Mike and Gerald, but there is a slight problem with
> semantics...
>
> On Sun, Oct 15, 2000 at 02:37:22AM -0400, Mike A. Harris wrote:
> > On Sat, 14 Oct 2000, Gerald Henriksen wrote:
> >
>   [Scott Doty wrote]:
> > >>That may be the case, but I wonder how many _weren't_ fixed.  I think
> > >>we can agree that rushing the release out the door with outstanding
> > >>bugs isn't in anyone's best interests (except perhaps marketing folks).
> > >
> > >In an ideal and perfect world every release would be bug free.
> > >
> > >We don't however live in a perfect world and as such any software
> > >release will ship with bugs.
> >
> > That is very true.  No Linux distribution will *EVER* ship
> > without bugs.
>
> Having used open source software since 1992, I understand the situation.
> Perhaps I wasn't clear:  my problem is with shipping software with _known_
> bugs.  Please evaluate your discussion in this context:  _known_ bugs
> should not appear in a new release, period.
>
> [bugs, known and otherwise, in new releases]
> > This rule is for ANY distribution however.  Debian holds back
> > releases quite a while until all bugs are stomped out, but again
> > there are no guarantees, and bugs do get found.
>
> True, but since I am concerned about _known_ bugs, I suggest that Debian's
> strategy is precisely the strategy that should be adopted at Red Hat.
> Unfortunately, my questions about Red Hat's strategy decisions remain
> unanswered -- and I don't think this is the wrong venue for that discussion.
> If this is an engineering support company, why would Red Hat be reticent to
> discuss their strategy with their staunchest supporters?
>
> Think about it:  if you were the CTO of Red Hat, wouldn't you monitor the
> mailing lists set up for new betas and new releases?  If someone questioned
> your strategy decisions, wouldn't you either defend them, or admit that
> change was necessary?
>
> Make no bones about it, I am -><- that close to firing Red Hat.  I don't
> want to hear about how many bugs Red Hat fixed before their 7.0 release -- I
> want to hear why it was released with known bugs.  I don't want to hear how
> Red Hat's support structure will, indeed, look somthing like Cisco's -- I
> want to see that they start with a known baseline release from which my
> purchase of that support becomes viable.
>
> > It's really too bad that Debian folks are no longer supporting their
> > non-recent distribution too.
>
> That is a point in Red Hat's favor.  I am very pleased that they still
> update 5.2, since that is our current standard release.  However, as 5.2
> grows older, I am concerned that that support will disappear.  I'm afraid
> this has become important in our company, more anon.
>
> > Even with their methodology of holding back releases until bugs are found
> > and fixed and tested, bugs do occur and always will.
>
> Understood.  I have no problem with that, as these appear to not be known
> bugs.
>
> > I think people who are worried about security and stability, need
> > to actually learn about it first, and not use the latest greatest
> > version of ANYTHING.
>
> Right with you.  I already pointed out the versions of the components we
> use on our network -- "new" doesn't mean anything to me, I want "secure" and
> "stable".
>
> > >>Yes, it will take more person-hours to fix every bug, but all that means
> > >>is that the release date is delayed.  While others may disagree, I think
> > >>I would rather wait for a bug-free release.
> > >
> > >Which you will never get.  Your typical Linux distribution is just far
> > >too big to be bug free.
>
> _Known_ bugs, sir, _known_ bugs.  You pointed out that Debian is careful
> to avoid that problem in their releases -- should we hold Red Hat to a
> different standard?
>
> > >The best you can get is what Red Hat does which is to actively
> > >maintain an errata list.
> >
> > Which works extremely well too.
>
> I disagree.  Red Hat ignores known bugs -- and if ignored, the don't appear
> in errata.
>
> > The fact is though if they hold back a release, NOBODY is using it, and
> > hence nobody is testing it as good as if it were released.
>
> Nevertheless, those concerned with the security, reliability, and
> performance of the product -- and are considering purchasing support from
> Red Hat -- will make darn sure their particular systems are well-tested in
> the beta releases.  Consider our case: I had a concern, tested pinstripe,
> found the same bug in the 7.0 beta as I found in 6.2...and Red Hat ignored
> that.  They released 7.0 with known bugs, and the only explanation I heard
> was "it works fine on Red Hat's network."
>
> If you want me to explain the problem with this, it would be my pleasure
> to articulate it -- but let's cut out the BS and get down to brass tacks,
> shall we?
>
> > As a result no matter when the release occurs, bugs will be found
> > afterward that couldn't turn up in internal lab testing.
>
> As far as I can tell, Red Hat conducted no "internal lab testing" of YP.
> >From where I stand, this appears systemic, and a flaw in their strategy.
>
> > No dist maker has every piece of hardware out there, and can't possibly
> > test all the permutations of everything.  The permutations of just the
> > kernel builds alone are enough to cause extreme nightmares.
>
> Since I don't subscribe to a philosophy of grabbing the "latest and
> greatest", deciding on a Linux kernel for our servers becomes:  why
> _wouldn't_ we want to use 2.0.38?
>
> > The fact that there are updates immediately following a release,
> > IMHO shows Red Hat's strong point of customer support.
>
> It also implies that Red Hat released a production distribution with
> _known_ bugs.  Consider my position:  if you wanted to purchase
> support from Red Hat, would you accept a distribution with _known_ bugs?
> Sir, that would be most unfair -- and if Red Hat can't make the grade, I'll
> find somebody who will.
>
> Nor do I want special treatment.  I want Red Hat to adopt a strategy that
> will ensure that they have made their new releases as bug-free as possible.
> When folks start telling me about "crunch time", or "maybe they had to meet
> a CD-vendor commitment", or "so-and-so came from Cygnus", I'm afraid this
> will give me pause for thought.  What does this have to do with releasing
> software with known bugs?  I don't see any plainer way of describing the
> situation.
>
> > They are fast to release fixes, and always have been.  This is one major
> > reason I switched from Slackware to Red Hat back in 1997.
>
> I switched from MCC to Red Hat's "Halloween" release back in...was that,
> 1993?  No matter:  we switched because it seemed like a good idea at the
> time.  Marc Ewing gave us a hand with some important issues, and we were
> happy to work with him, including submitting new packages.
>
> So I hope you can understand why I don't take the prospect of "firing Red
> Hat" lightly.  If the right people applied themselves, that might be
> avoided, but I don't see that happening at this late date.  If we didn't
> have a history with the company, I wouldn't be bothering to offer this
> much-needed criticism -- but you know, I only have so much patience, and
> I've been working on keeping us with Red Hat for over a month.  They just
> aren't helping.
>
> > >>Well, rather than point fingers and assign blame, I would instead like to
> > >>see _something_ happen, whatever it may be, that meets the goal: a RedHat
> > >>release that we would be comfortable deploying on our production servers.
> > >>6.x and 7 don't meet that goal, so it appears to be a trend.
> > >
> > >If none of the last 4 releases have met your needs for a production
> > >server, then you are I feel either going to have to make your own
> > >distribution or modify your needs.
>
> Or go with another distribution.  I've considered building our own
> distribution, which would be very satisfying -- but let's face it, I don't
> have time for that.  With plenty of Linux distributions to choose from, I
> think Red Hat, our current distribution, can meet the challenge.  (If you
> think I've spent too much time on this already, please consider our history
> with Red Hat -- I think they deserve the chance.)
>
> > > While I am sure that you have valid reasons for rejecting those
> > >releases, those same releases have served a large number of people very
> > >adequately.
>
> Ya know, when we met with with a certian Red Hat distributer at an ISPCON,
> they were surprised to learn we were serving so many web sites on a single
> Linux box.  That was the first inkling that our Linux network might be
> larger than usual.
>
> Back then, our philosophy was to install a Linux distribution, and then
> install our own builds of server software.  It was only recently (while
> discussing the situation on this list) that I realized that _all_ server
> software on our Red Hat systems were local builds.
>
> I wonder if we dare rely on a vendor for that software, as I daresay our
> server software stability may have little to do with Red Hat.  In some
> cases, we've built software on a Red Hat 5.2 box "static", and then deployed
> the binary on a system running an earlier release.
>
> Personally, I think this empirical knowledge is invaluable to any company
> hoping to release stable software.  I'd love to tell the right people at
> Red Hat what we did, and why we did it -- but dammit man, we can't even get
> off square one, which is Red Hat admitting that they need to change their
> strategy.
>
> To illustrate, consider the following post in our "sonic.os.unix" (a local
> newsgroup):
>
>    [Regarding Linux in general] But I wonder how well and how long it will
>    scale.  Many small ISP's that started with Linux have eventually
>    migrated to other operating systems.
>
> So there I was, defending Linux in general, and pointing out our experience
> with Red Hat.  I already have criticism from people I respect telling me
> that I'm "slow to adopt change", that "we should adopt a different
> distribution."  I'm trying very hard to stay with Red Hat, but I'm not
> getting any help from their company.  I have invested a lot of time
> in constructive criticism, and got nothing for my effort.
>
> I don't have time for this.  All I can do now is tell you _my_ deadline:
>
> On Monday, October 16th, at 3pm PDT, Sonic.net will be holding our
> twice-weekly ops meeting, where I will render my decision.  Hopefully, Red
> Hat will jump in here and talk about their strategy before then -- if not,
> I'm afraid the prospects for Red Hat are very bleak.
>
> > Yes, and there is and never will be ONE OS that will satisfy
> > every single person's needs.
>
> Linux is our OS, and it works fine.  My criticism is with Red Hat.
>
> Finally, two items:
>
>   * bugs not affecting "the majority of users out there".
>     Nonsense.  Any large-scale deployment will run into the same
>     bugs we have.  The only thing special about our deployment, in
>     the context of this discussion, is that we have chosen to use Linux --
>     and we have chosen to use Red Hat 5.2 as our Linux distribution.
>
>   * "That is one reason I don't use Debian - which is a very
>      good distribution.  It's packages are quite out of date."
>     Sir, this is a point in their favor.
>
>  -Scott
>
> _______________________________________________
> Guinness-list mailing list
> Guinness-list@redhat.com
> https://listman.redhat.com/mailman/listinfo/guinness-list

But if 5.2 currently suits you, then I would think it quite clear that 6.2 was
to be preferred over 7.0.  It *IS* still available.  If Red Hat doesn't sell it,
I'm sure you could get a version from a repackager, e.g., CheapBytes.  This will
mean that you end up having to upgrade nearly as often as major upgrades are
released, but you will always be upgrading to a well debugged version.  So I
don't really understand your concern.

Possibly I should say it this way:  No company can ever, under any circumstance,
no matter how much they spend, but a distribution to the kind of testing that a
real release puts it to.

Given the preceeding sentence, if you want a secure system, then you strategy is
obviously to adopt the one just past.  That will even save you some money, as
there are often sales on "obsolete" versions.  (Of course, others just keep
selling them for full price, but look for what you want, not what you don't
want.)