Red Hat and the Kernel Kerfluffle

Red Hat’s changes to the way it distributes its kernel have made headlines all over the tech press. The company has taken a lot of heat, but what about the competitors that are freeloading on Red Hat’s work?

If you haven’t followed the issue, let me sum up. Prior to Red Hat Enterprise Linux (RHEL) 6, Red Hat would ship its kernel source with the patches it applies broken out separately. With RHEL 6, Red Hat ships the source with the patches pre-applied. What difference does that make for the vast majority of the people who use or interact with RHEL? None at all. But it does pose a barrier for other developers who want to study the RHEL kernel and see what patches it has applied to the stock kernel.

Wait, isn’t Red Hat shipping the 2.6.32 kernel? Well, yes-ish. Red Hat tweaks the kernel quite a bit with backported patches from later kernels, fixes that it has applied in response to customer issues, and some patches that may not have made it into the vanilla upstream kernel.

Since a lot of people watch Red Hat’s kernel for a number of reasons — varying from developers from other distros studying the kernel to see why something works well on Red Hat (or doesn’t) to competing companies that want to provide support for Red Hat’s distribution.

How many people does this affect? Well, to put it in perspective, it took about four months for this to come to light in the tech press. That happened after Debian kernel team member Maximilian Attems made a comment about Red Hat’s “obfuscated kernel.”

Patchless Considered Harmful

People hate change when it seems like they’re losing something they had before — even if they weren’t actively using it. So it’s not surprising that Red Hat drew a bunch of criticism off the bat.

But the way that Red Hat is now distributing its kernel is legitimately frowned upon by the developers who actually do work with the kernel, like Attems. It’s sort of like asking someone for a recipe for the family’s chocolate chip cookies, and getting cookie batter instead. Or for more writerly folk, it’s like the difference between getting a final manuscript versus the Word or ODF file with changes highlighted. It’s much more difficult to study.

Free Ride on RHEL

Which is, of course, exactly why Red Hat has done it this way. Bryan Stevens, Red Hat’s CTO and VP of worldwide engineering, laid the blame for the change at the feet of Red Hat’s competitors:

Why did we make this change? To speak bluntly, the competitive landscape has changed. Our competitors in the Enterprise Linux market have changed their commercial approach from building and competing on their own customized Linux distributions, to one where they directly approach our customers offering to support RHEL.

Frankly, our response is to compete. Essential knowledge that our customers have relied on to support their RHEL environments will increasingly only be available under subscription. The itemization of kernel patches that correlate with articles in our knowledge base is no longer available to our competitors, but rather only to our customers who have recognized the value of RHEL and have thus indirectly funded Red Hat’s contributions to open source that will advance their business now and in the future.

It’s a shame that Red Hat has gone in a direction that is unfriendly to the developer community. I’m sure that some or many of Red Hat’s engineers are displeased with the decision, and it’s unfortunate if this hinders collaboration in the kernel community.

Red Hat has taken some lumps for the change, but what I haven’t seen is more than a passing mention of Oracle and Novell’s role in this.

Increasingly, Red Hat’s major competition is just saying “hey, we’ll let Red Hat do the engineering work and just provide support.” This is true of Oracle, and Novell to a lesser extent. Oracle has been riding on Red Hat’s coattails for years and has said it would do just that, saying that companies should compete “purely on the support side of the business.”

Novell still does its own very fine Linux distro, but if you look carefully, the amount of new features and energy put into its Linux distro the past few years has waned a bit. The company has ramped up support offerings for RHEL in the meantime, and put a lot more energy into things like SUSE Studio.

Oracle is just content to leech off of Red Hat. While Novell is trying to woo customers over to SLES by supporting RHEL as a bridge to SLES, Oracle just freeloads off of Red Hat’s distribution.

It’s a good thing that the GPL and other open source licenses allow companies to jump in and provide support for competitor’s products. But this trend isn’t healthy for the larger community — and it’s not something that can or should be solved by licensing. Companies in the market for Linux services should exercise a little forward thinking and reward the companies that are doing the most to maintain the ecosystem. Here’s a hint: It ain’t Oracle. Even if that Oracle support contract is a bit cheaper than Red Hat’s right now, what do you think Oracle’s going to charge if it manages to marginalize Red Hat?

Bottom line: If you want to snipe at Red Hat for its admittedly community unfriendly change, at least recognize that the company is still doing more than its share of the work.

Red Hat is still complying with the GPL. It’s still doing its work upstream with the kernel community. Any competitor that is putting together its own distribution from scratch and participating in the kernel community is going to be working side by side with Red Hat’s kernel engineers.

Red Hat is responding to competitive pressure by refusing to share notes with its competitors. As a side-effect, it’s also having an effect on the rest of the community — and that truly sucks. But let’s not put all the heat on Red Hat, even if you feel chafed by the change.

You’ll notice I didn’t mention CentOS or Scientific Linux here. They’ve been around for years, and I doubt they are putting much of a pinch on Red Hat. The CentOS community has even gone out of its way to be supportive of Red Hat by directing users to Red Hat’s beta programs rather than packaging and releasing the betas as they came out. And they serve a valuable purpose for the users and organizations that aren’t addressed by Red Hat’s commercial offerings. (Red Hat could do much better here by offering RHEL subscriptions appropriate for individuals who want to maintain their technical chops, run home servers, etc.)

Props to Red Hat

I’d also like to give props to Red Hat for being extremely forthright about the change when asked. The company didn’t hem, haw, and offer up no comment. Whether the decision was right or wrong, they’ve owned it and were honest about why they did it. That’s pretty rare, so Red Hat deserves some credit for being blunt about their reasoning. You may not agree with them, but at least give Red Hat credit for being up-front about what it’s doing.

Red Hat’s new kernel policies may not be entirely in “the spirit” of the GPL, but they are (as far as I understand it, not being a lawyer and all) GPL-compliant. In the larger picture, Red Hat is still doing right by the community by channeling a lot of the money it makes into upstream development and working on new features and projects through Fedora that will all be open source. The company makes mistakes from time to time, but if they’re going to get dinged for this I do hope that their competition gets some of the “credit” as well.