Why don't they "provide it universally in RHEL?" -- WAS: Thoughts from last meeting

[Bryan J Smith]

On Fri, May 25, 2012 at 5:45 PM, inode0 <inode0 at gmail.com> wrote:
> I don't see any reason for them to just remove it from EPEL while not providing it
> universally in RHEL.

Define "universally"?  I can understand people wondering how Fedora
will handle details.  That's difficult enough at times.

But I'm surprised how many are trying to think and re-think for Red
Hat, or comment on what it should do, or what they think it's up to.
Everyone should always feel free to reach out to Red Hat and ask.
I've always done this myself over the years.  Alan Cox himself took
many hours out to explain it to me, several things, over the years.
Others did as well.

It's been the customers and IHV/ISV needs for certification and
support has driven most everything.  This is especially in the case
where customers have requested Red Hat to take on products and
codebases with certification and support.  Not everyone uses them, and
it quickly becomes a question of X for Y, where economies of scale of
the add-on aren't like the core.

But then you have customers who want the $49 Client, or the basic
Server for a couple hundred, and they don't want to pay for full
support of the kitchen sink either.  So what does Red Hat do?  This
has happened through RHEL4, RHEL5 and RHEL6.  Red Hat listens to
customers who want the certification and support options.  It happens.
 Not everyone wants to pay one entitlement rate for what they actually
need in certification and support.

DISCLAIMER:  I'm going to just point out things from watching Red Hat
a long time.  I'm sure this will be demonized.  I'm sure some will
just run and do what they do.  That's fine.  But I'll point it out
this once.  This is 100% from customer perspectives over the last
decade of Enterprise options from Red Hat.  If you search the
Internet, you can see I was screaming for Red Hat to add XFS (Scalable
File System) back in 2000+.  And yes, if you're wondering why it's
called "Scalable File System," remember that trademarks are something
every commercial entity deals with (even Red Hat).  ;)

It's all software engineering 101 to me.  Sustainment is 85% cost.
Just because most commercial models charge for features up front, and
then spend a lot of money maintaining and backporting for 5 or 7+
years, doesn't mean it's the right way.  Red Hat is the opposite.
SuSE actually perfected it first.  Charge for sustaining engineering,
but leading edge is low-cost or, as Red Hat put forth first, free
(Fedora).  At the same time, Red Hat has always attempted to maintain
a balance by releasing the Source RPMS so everyone can rebuilt and
strive for bit-for-bit compatibility, something SuSE did not until
after Novell's acquisition IIRC (more on SuSE in a bit).

> As an EPEL consumer I find this all rather confusing.

Actually, it's really been going on all along, well before EPEL.
Advanced Server, "AS" and "Advanced Platform" (AP) were already
"layered products."  Several "EL Rebuilds" would build that code.
EPEL came around and avoid them early on.  But since then, Red Hat has
heavily expanded that.  It's no longer a half-dozen-plus (6+) child
channels, it's now up to three dozen or so (~36) if you add them all
up.

Remember, Red Hat has radically changed from Red Hat Enterprise Linux
3 some eight (8) years ago when it had around a half-dozen.  It's no
longer a few hundred employees, but over 5,000.  There are a lot of
maintainers not just on a few subsystems, but many that did not exist
even five (5) years ago.  Red Hat is no longer doing just one (1)
option, but usually several (if not stewardship of an entire swath of
code, even multiple options).

A lot of people are still living pre-2003, again, well before EPEL,
even before Fedora (when Red Hat had both Enterprise Linux and Linux
-- yes, they co-existed for well over two years).  And even EPEL
avoided "AS" "Advanced Platform" (AP) -- that's a half-dozen "layered
products" right there.  This is nothing new.  The thing is that there
are more and more layered products.  It's still a shock to me that Red
Hat has been able to scale and deliver added certification and
support, but it has.

Understand Red Hat hasn't been just an OS company for a long time.
And even the platform level has greatly expanded on its own.  It's an
entire ecosystem that competes with partners, from EMC to Oracle, and
still growing, to meet customer requests.

> I don't want to have to know which layered products are protected and which aren't.

Define "protected"?  I think you're looking at that from the wrong
angle.  I've never looked at Red Hat "protecting" anything.

And even from the Fedora angle, I don't consider EPEL trying to
"protect" anything.  EPEL is trying to serve the needs of the
community of users, customers, everyone.  That's a tall order.  It has
had its policies.  It will continue to do so.  Red Hat just did not
ship a lot of components at all, components that many in the
community, including Red Hat customers, did not have.  EPEL was the
result and why I tapped it immediately.

One thing that I continually see missed is that Red Hat releases
everything.  The Fedora Project is a _superset_ of all of the
technologies in Red Hat Enterprise products.  Of course Fedora rebases
more than Red Hat Enterprise releases.  People want to avoid rebasing,
and that's why they look to "Enterprise" products for long-term,
sustaining engineering,  ABI/API longevity, etc...

That was the huge difference between even Red Hat Linux and Red Hat
Enterprise Linux before as well.  I even remember when Red Hat
clarified at the release of Red Hat Linux 5.0 about 4.2 being
supported another year, and re-clarified after 7.1 that updates would
be maintained for a year.  Rebasing and end-of-life (EOL) was a
reality of more leading edge.

Understand SuSE was the first first that showed -- with SuSE Linux
Enterprise Server version 7 (SLES7) -- that such sustaining
engineering was not cheap to dom but customers and IHVs/ISVs wanted.
This was back when Red Hat did it's first "Enterprise" option called
Red Hat Linux 6.2 Enterprise, which had a three (3) year SLA, but
rebased software.  Yes, sounds like "Long Term Support" of a more
leading edge distribution, doesn't it?

Red Hat Advanced Server / Enterprise Linux then grew up alongside Red
Hat Linux for several years,.  There were the "child" channels that
were in a much more costly form.  Then customers wanted more.  The
rest is history.

It's costly enough that, today, a "smaller" SuSE is now rebasing
components in their Enterprise line now.  Red Hat is now the only one
doing 10+ years of kernel/core ABI sustainment (much less no one seems
to be doing just even 5+ now).  That long-term kABI and related
ABI/API is nothing no one wants to attempt, and even virtualization
and whitebox products are using that sustaining effort, other than Red
Hat.  And then you have more and more code built around that, that is
maintained 3, 5 even 7+ years long-term.  Customers then want that
too.

It's a hard balance to strike.  Every time I've brought it up with Red
Hat over the years, people have listened.

> I think I'd rather live with a simpler uniform policy regarding layered products.

Going with your point above, is it really a "simpler, uniform policy"
for Fedora?
Or are you really inquiring why Red Hat has "layered entitlements" in
the first place?
Seriously, I'd rather not dance around the issue, but see a direct
statement made.

All I can point out is that little ISPs with mixes of unpaid/rebuild
"EL" and paid/certified/supported Enterprise Linux are not going to
want to pay for the same entitlements for certification and support as
someone who does utilize all of the storage, clustering, directory
services, deployment/management, etc... options on the platform (not
even considering middleware, various middleware options, etc...).  So
I'm curious what the statement of "uniform" means from the standpoint
of Fedora?  EPEL?  Or even Red Hat?

How does Fedora and the EPEL SIG deal with all that?  I don't know.
But let's recognize why it exists, not question in the Fedora project
why it does, or how it cannot be more uniform.  Just my $0.02, but I
could be wrong and/or biased.

-- Bryan

P.S.  This will be the only time I point all this out on the EPEL
list.  I know as the years go by, some haven't been exposed to the
same history.  That's the only reason why I point it out, because I
hope it explains a lot of things that are outside of Fedora.