Red Hat and the Clone Wars II: A history of the early 2000s Linux landscape

After Saturday’s post I wanted to take a step back and talk about some history that many have either forgotten or weren’t familiar with in the first place. Some may remember it quite well, but haven’t quite gotten the lessons right the first time around. Let’s talk about Red Hat Linux and the early days of Red Hat Enterprise Linux before it was even called that.

Red Hat sets the standard

My Linux journey started with Slackware Linux in 1996, completely by accident. By that I mean that I had never heard of Linux or sought it out, until I stumbled on a 4-CD set and decided I wanted to learn more. I was studying English lit and Communications/Journalism at a state school in the northeast corner of Missouri. Nobody I knew cared much about computers beyond games or running Word to write their papers. It was literally years before I met someone else who was an avid Linux user.

It was a surprise to learn, a bit later, that Slackware was a Linux and that many distributions existed. As I learned more and more about Linux, though, something became clear: Red Hat was the popular choice. Red Hat was the Coca-Cola of Linux, even before its IPO in mid-1999.

Everybody wants a piece

The company I started working for in March 1999 offered a Red Hat-based Linux distro called “Linux Pro” that included some proprietary bits (like a professional X server with CDE), and more. We also sold 99 cent CDs of Red Hat Linux (and other distros) that were copied from the FTP servers where Red Hat distributed Red Hat Linux. From day one, Red Hat has been more open than is required, which has had its benefits and drawbacks.

Note that Linux Mall wasn’t the only company to sell the cheap CDs, and Red Hat Linux wasn’t the only distro that got the cheap CD treatment. Red Hat Linux also got bundled with books that were quick and dirty compilations of the Linux HOWTOs thrown together and shoved into bookstores to capitalize on Linux’s growing popularity.

Most users still couldn’t easily download Linux, they depended on retail boxen or cheap CDs. College students, however, could often grab Linux more easily thanks to university Internet. As an early adopter of DSL in 2000, I was pleased to be able to download a single ISO image overnight, but high speed Internet was far far far from mainstream.

All these other methods competed directly with Red Hat Linux’s primary retail offering at the time: the Red Hat Linux boxed product. Part of the value you received from buying a Red Hat Linux box set was literally just being able to get the bits in an installable form without having to try to download it yourself. Yes, Red Hat used to come in boxes. And that was convenient! (It’s kind of a shame that there’s no longer a boxed version of RHEL. Like vinyl, it’s not the most practical way to get it, but it’d be nice to have something physical to remember each release.)

Red Hat hadn’t yet launched Red Hat Enterprise Linux (RHEL), but that was coming and people weren’t happy about it when it happened.

Red Hat goes public, Linux in the spotlight

Even in the earliest days, Red Hat was contending with a lot of players looking to make a buck of its product. But this also reinforced Red Hat’s popularity, and helped to establish its brand.

Let’s set the stage here. Red Hat was a public company in late 1999 and now had to answer to shareholders. Everybody was starting to realize that there was money in Linux (and open source) and/or that Linux and open source posed a threat to their current business.

And IBM, the company that people love to sling rocks at for every Red Hat decision they don’t like, was early to bet on Linux and open source with a commitment of one billion USD and helping to establish the OSDL (later Linux Foundation) in 2000. It also helped found the Apache Software Foundation (ASF) in 1999, and so on.

But we’re here to talk Red Hat before its IBM days.

Red Hat Linux was becoming something of a standard, in as much as there was a standard. Trying to get Linux vendors and projects to all go in one direction, even for the benefit of all, was like trying to herd cats. On meth. And Linux was still young. It was in many ways not yet ready for prime time.

Linux still had a lot of maturing to do to compete with proprietary UNIX, but folks in Redmond were already starting to sweat this whole open source movement. Open source was far from mainstream, and had a reputation as unprofessional, untested, anti-commercial, risky, buggy, lazy copies of proprietary software. Most people had never heard of Linux or open source, and those who had were just as likely to be against them as for them.

So here’s Red Hat: trying to go head to head with an untrusted product against proprietary UNIX and Windows NT with industry giants ahead of it, a gaggle of businesses quick to scoop up its product and sell it cheaper, as well as number of competitors all jockeying to be the number one Linux or at least ride in Red Hat’s wake.

Building a brand

Red Hat’s technology was easy to copy, and still is. And it needed to deliver revenue, reliably, quarter after quarter after quarter. This is not to say that the other businesses had no need for revenue, but Red Hat actually came under scrutiny as a public company from investors (and the SEC…) that others didn’t have to cope with.

A small business could happily trail in Red Hat’s wake by cloning Red Hat. Case in point, Kevin’s Red Hat Uber Distribution (KRUD) was a popular Red Hat Linux clone in the late 90s and early 2000s. You could subscribe to updates by CD to KRUD that would provide the latest errata. It also included additional software not in Red Hat Linux like CD burning tools, a Microsoft Word viewer, and OpenSSH. Yes, friends, apparently OpenSSH wasn’t part of a default Red Hat Linux install back in 2000.

Not that KRUD was likely to pose a huge risk to Red Hat, but Red Hat must have taken notice that as it gathered steam other companies would attach their barnacles to the U.S.S. Red Hat.

Mandriva Linux was another contender, a fork of Red Hat Linux 5.1 that embraced KDE and provided a really polished (for the time) desktop experience. SUSE, which hopefully needs little introduction, had started as a Slackware fork but evolved into another KDE-based distro with really good engineering and a lot of popularity in Europe. Mandriva and SUSE also embraced RPMs, but as mentioned, the meth cats couldn’t agree on the implementation of RPM or anything that would allow RPMs to be reliably compatible across distributions.

So, Linux is gaining in popularity and evolving quickly. But you certainly couldn’t bank on an application compiled for one Linux flavor to run reliably on another. Or even install, honestly. But Red Hat was gaining ground and becoming something of a de facto standard, and popular to derive other distros from.

What SUSE and Mandriva didn’t have? Popular clones, and a brand that was effectively “the standard” for a large swath of Linux enthusiasts, professionals and (over time) businesses.

Red Hat throws the curveball

Red Hat, better than others, understood that it wasn’t just building an operating system and that technology alone wasn’t going to get it done. It was building a brand, it was selling products, and that reputation was going to be instrumental in success or failure.

People assumed that things would just continue as they were, and of course they never do. Red Hat announced the end of Red Hat Linux in July 2003. Red Hat promised a six-month release cycle for a community developed Linux with more open development, but details were in short supply and there was much confusion and criticism. The initial announcement landed in July 2003, and then Red Hat announced its new direction in September 2003.

That announcement brought together Red Hat and the Fedora Linux Project. Fedora Linux, at the time, was a separate project from Red Hat started by an undergrad student named Warren Togami. That was a project providing additional packages for Red Hat Linux, but the September 2003 announcement started the journey that led to the Fedora Project as we know it today.

History doesn’t repeat itself, but it does rhyme. Red Hat’s changes, coupled with terse communications and incomplete details, sparked some anger and discussion of alternatives.

SUSE, Mandrake Linux, Lindows, Xandros, and Debian were all contenders for the throne. Much was said about the importance of developer and enthusiast happiness to ensure success. Red Hat’s fumblings with Fedora opened the door to Ubuntu, but we’ll cover that soon enough.

Surely, this misstep was going to be the end of Red Hat’s reign, yes? Not so much.

SUSE was sold to Novell in November of 2003, broke and still pursuing the retail market. Credit where due, SUSE has survived so many twists and turns and is still at it in 2023.

Lindows and Xandros are long gone. At the same time Mandrake, which did make developers and enthusiasts quite happy, was in bankruptcy at the time and wound up becoming Mandriva after a trademark lawsuit. Mandriva is also long gone. So is Caldera, so is Stormix, so too are any number of other distributions that failed to reach critical mass and commercial success.

Debian, as a pure vendor-neutral project, keeps chugging away and is the basis of Ubuntu and its offshoots.

As a Red Hat Linux clone, KRUD was thrown for a loop. Tummy.com, which sold and supported KRUD, faced the uphill battle of providing a clone of Red Hat’s new offerings that would require de-branding and finding a JVM that it could license.

In addition, everybody realized the same thing: The operating system is just there to run applications, and vendors target specific operating systems.

As one user put it on the KRUD users list, “Unless I can get them to certify KRUD’s distro, I am just screwed here. I am going to have to go and spend money, every damn year to RedHat, whom I don’t like at all, and not use Tummy, whom I really really like…” Without the ability to refer back to a specific Red Hat release, not just the technology but the very specific product, the code loses a lot of value.

I’m not sure when KRUD retired, but the KrudUsers list archives end on Archive.org around 2006. The Kevin from which KRUD derived its moniker is now the Fedora Infrastructure Lead. Although not completely relevant to the topic at hand, I’d like to note that he’s also a great person to work with and probably under-appreciated in terms of his contributions to Fedora.

What the past tells us about today

There are lessons for everyone here, if only we care to learn them and adapt.

• Better communication would help a lot. Lots of people loved Red Hat, and then they were angry and felt like they had the rug pulled out from under them. Thing is, Red Hat didn’t promise them a rug, but they felt like it had. Red Hat won back a lot of goodwill over the years by chopping wood and carrying water, and by slowly turning Fedora into a real community project. But it also wasted a fair amount of goodwill unnecessarily. The announcement last week could’ve been clearer, more explicit, more helpful, and at least 30% more palatable to “the community” even if Red Hat’s actions were otherwise identical. It’s entirely possible that Red Hat chose not to paint a crystal clear picture, and we can discuss that in a future post.
• Quality alone won’t save you or make you a standard. In this time period I used all of these distributions. SUSE and Mandrake were quality distributions, in some ways more polished than Red Hat (for the desktop, at least). Unfortunately they didn’t understand the market or failed to execute well enough to flourish. Red Hat focused on market fit, branding, lifecycle, and sales.This isn’t to say young Red Hat never made mistakes. It dipped its toe into PostgreSQL early on, for example, and just as quickly got out of the water.
• Go where the money is. Red Hat had a laser focus on the market that would produce the deals it needed to grow. Red Hat did and does take criticism for not focusing on the desktop, but that’s not where the money was or is.
• Understand the motives and perspective of those giving you advice and criticism. Lots of people have advice and critiques for Red Hat, which is fine. But, it’s worth asking whether those critics have the information at hand to really give good advice or critiques. Understand that that advice and criticism is shaped by what is good for them and not necessarily what’s good for Red Hat short or long term. They may not want to spend money “every damn year” with Red Hat, but that’s how Red Hat stays in business. If Red Hat listened to the masses in 2003, the present would look very different.
• Standards and certifications matter. As I wrote yesterday, the thing people are clamoring for is the thing that really has value – specific “binary compatible” RHEL releases. If you can’t tell the vendor you’re running RHEL 8.5 or something they consider equivalent to RHEL 8.5, then they don’t want to take your support call. This was true in 2003, and it’s true in 2023 – only the names and version numbers have changed.
• It’s not (just) Red Hat vs. the community. I’ve tried to illustrate that, even from early days, Red Hat has faced a lot of drag from clones and copies in its attempts to create and maintain a business. Rarely are those factors considered by “the community” that really just wants a Red Hat compatible Linux, Red Hat’s business be damned. Red Hat’s actions may impact you, the community, but consider that the battle Red Hat is fighting isn’t against you – and the other players rarely get named or blamed for Red Hat’s reactions.
• Choose your upstream carefully. If you’re betting on a vendor-controlled project as an upstream (and I include businesses running KRUD or CentOS here) understand that you are extremely vulnerable and have no one to blame but yourself if they make disruptive changes. (See also: Give nothing, expect nothing.) This isn’t a “rug pull,” this is Red Hat refusing to set itself on fire just to keep others warm.
• Debian is always an option. Debian was producing a fine, stable Linux distro 20 years ago, and it’s still producing a fine, stable Linux distro now. If free software purity and a community first attitude is what you seek, you’re never going to get that by hectoring vendors to be something they’re not. There are tradeoffs in choosing a vendor-neutral operating system, and you might not always like Debian’s technical choices, but if you want a project you can directly influence without being a customer then choose Debian.

OK, enough about the distant past for now. The context is important, though, to understand the present. Plenty more to write about.