Red Hat and the Kernel Kerfluffle

Recent changes to how RHEL kernel patches are distributed is creating barriers for developers. Why did Red Hat make this change? To stay competitive.

Red Hat’s changes to the way it distributes its kernel have made headlines all over the tech press. The company has taken a lot of heat, but what about the competitors that are freeloading on Red Hat’s work?

If you haven’t followed the issue, let me sum up. Prior to Red Hat Enterprise Linux (RHEL) 6, Red Hat would ship its kernel source with the patches it applies broken out separately. With RHEL 6, Red Hat ships the source with the patches pre-applied. What difference does that make for the vast majority of the people who use or interact with RHEL? None at all. But it does pose a barrier for other developers who want to study the RHEL kernel and see what patches it has applied to the stock kernel.

Wait, isn’t Red Hat shipping the 2.6.32 kernel? Well, yes-ish. Red Hat tweaks the kernel quite a bit with backported patches from later kernels, fixes that it has applied in response to customer issues, and some patches that may not have made it into the vanilla upstream kernel.

Since a lot of people watch Red Hat’s kernel for a number of reasons — varying from developers from other distros studying the kernel to see why something works well on Red Hat (or doesn’t) to competing companies that want to provide support for Red Hat’s distribution.

How many people does this affect? Well, to put it in perspective, it took about four months for this to come to light in the tech press. That happened after Debian kernel team member Maximilian Attems made a comment about Red Hat’s “obfuscated kernel.”

Patchless Considered Harmful

People hate change when it seems like they’re losing something they had before — even if they weren’t actively using it. So it’s not surprising that Red Hat drew a bunch of criticism off the bat.

But the way that Red Hat is now distributing its kernel is legitimately frowned upon by the developers who actually do work with the kernel, like Attems. It’s sort of like asking someone for a recipe for the family’s chocolate chip cookies, and getting cookie batter instead. Or for more writerly folk, it’s like the difference between getting a final manuscript versus the Word or ODF file with changes highlighted. It’s much more difficult to study.

Free Ride on RHEL

Which is, of course, exactly why Red Hat has done it this way. Bryan Stevens, Red Hat’s CTO and VP of worldwide engineering, laid the blame for the change at the feet of Red Hat’s competitors:

Why did we make this change? To speak bluntly, the competitive landscape has changed. Our competitors in the Enterprise Linux market have changed their commercial approach from building and competing on their own customized Linux distributions, to one where they directly approach our customers offering to support RHEL.

Frankly, our response is to compete. Essential knowledge that our customers have relied on to support their RHEL environments will increasingly only be available under subscription. The itemization of kernel patches that correlate with articles in our knowledge base is no longer available to our competitors, but rather only to our customers who have recognized the value of RHEL and have thus indirectly funded Red Hat’s contributions to open source that will advance their business now and in the future.

It’s a shame that Red Hat has gone in a direction that is unfriendly to the developer community. I’m sure that some or many of Red Hat’s engineers are displeased with the decision, and it’s unfortunate if this hinders collaboration in the kernel community.

Red Hat has taken some lumps for the change, but what I haven’t seen is more than a passing mention of Oracle and Novell’s role in this.

Increasingly, Red Hat’s major competition is just saying “hey, we’ll let Red Hat do the engineering work and just provide support.” This is true of Oracle, and Novell to a lesser extent. Oracle has been riding on Red Hat’s coattails for years and has said it would do just that, saying that companies should compete “purely on the support side of the business.”

Novell still does its own very fine Linux distro, but if you look carefully, the amount of new features and energy put into its Linux distro the past few years has waned a bit. The company has ramped up support offerings for RHEL in the meantime, and put a lot more energy into things like SUSE Studio.

Oracle is just content to leech off of Red Hat. While Novell is trying to woo customers over to SLES by supporting RHEL as a bridge to SLES, Oracle just freeloads off of Red Hat’s distribution.

It’s a good thing that the GPL and other open source licenses allow companies to jump in and provide support for competitor’s products. But this trend isn’t healthy for the larger community — and it’s not something that can or should be solved by licensing. Companies in the market for Linux services should exercise a little forward thinking and reward the companies that are doing the most to maintain the ecosystem. Here’s a hint: It ain’t Oracle. Even if that Oracle support contract is a bit cheaper than Red Hat’s right now, what do you think Oracle’s going to charge if it manages to marginalize Red Hat?

Bottom line: If you want to snipe at Red Hat for its admittedly community unfriendly change, at least recognize that the company is still doing more than its share of the work.

Red Hat is still complying with the GPL. It’s still doing its work upstream with the kernel community. Any competitor that is putting together its own distribution from scratch and participating in the kernel community is going to be working side by side with Red Hat’s kernel engineers.

Red Hat is responding to competitive pressure by refusing to share notes with its competitors. As a side-effect, it’s also having an effect on the rest of the community — and that truly sucks. But let’s not put all the heat on Red Hat, even if you feel chafed by the change.

You’ll notice I didn’t mention CentOS or Scientific Linux here. They’ve been around for years, and I doubt they are putting much of a pinch on Red Hat. The CentOS community has even gone out of its way to be supportive of Red Hat by directing users to Red Hat’s beta programs rather than packaging and releasing the betas as they came out. And they serve a valuable purpose for the users and organizations that aren’t addressed by Red Hat’s commercial offerings. (Red Hat could do much better here by offering RHEL subscriptions appropriate for individuals who want to maintain their technical chops, run home servers, etc.)

Props to Red Hat

I’d also like to give props to Red Hat for being extremely forthright about the change when asked. The company didn’t hem, haw, and offer up no comment. Whether the decision was right or wrong, they’ve owned it and were honest about why they did it. That’s pretty rare, so Red Hat deserves some credit for being blunt about their reasoning. You may not agree with them, but at least give Red Hat credit for being up-front about what it’s doing.

Red Hat’s new kernel policies may not be entirely in “the spirit” of the GPL, but they are (as far as I understand it, not being a lawyer and all) GPL-compliant. In the larger picture, Red Hat is still doing right by the community by channeling a lot of the money it makes into upstream development and working on new features and projects through Fedora that will all be open source. The company makes mistakes from time to time, but if they’re going to get dinged for this I do hope that their competition gets some of the “credit” as well.

Comments on "Red Hat and the Kernel Kerfluffle"


I’ll add my “amen” to that. I can’t afford a real Red Hat support contract, so I end up running CentOS and Fedora for my servers, but I’m not going to shed any tears over Oracle and Novel and other commercial vendors having to work a little harder on their own to build and support their own for-hire offerings.


    We use Centos, Redhat, Suse/OpenSuse, and Ubuntu, depending on the server/service(PITA only try it at home..). I’m not upset about it given RedHat is the biggest company that contributes to the kernel, IBM is right behind according to a study released and I think mentioned on this website. They are making the patches upstream, and they are fixing things to other projects as well.

    Most distros use a newer kernel, and it is understandable in some respects how much backporting they do as they don’t want to break any installations with patches.

    I don’t know how many changes Suse/Novell are making especially upstream. Apparently not the kernel, but they maybe making changes to other associated projects. I have seen Suse’s work pre-novell days, make it upstream. I haven’t seen much lately, but I may not be aware of it.

    It would be nice if people jumped on companies like NVidia for their Tegra support. :)


    Yes, many organizations including government agencies have chosen CentOS as their choice of Operating System in the past few years. CentOS has lot of community support and its getting popular.


    Me and this article, sitting in a tree, L-A—REN-I-N-G!


Does anyone know if this change affects the kernel sources shipped with Fedora? One could conceivably look at the patches broken out in the Fedora kernel source RPM if they’re still being provided there.


    I don’t think that this affects Fedora at all. However, I suspect that you’d have better luck following the LKML and watching what is submitted upstream than teasing out patches to Fedora’s kernel.

    I could be wrong, but I believe what happens is 1) customer reports a problem/makes a request to RHAT, 2) they implement a fix and submit patch upstream, 3) patch is backported to enterprise kernels.

    Remember that they went with 2.6.32 when RHEL6 shipped, but the upstream kernel was at 2.6.35 or .36 when RHEL6 shipped. (I think that’s right.)

    Hmmm. Though maybe it would be worth watching Fedora as well. Not sure. I’ll look into that.


    Hmm it appears like your site ate my first comment (it was super long) so I guess I&l;#1782l just sum it up what I wrote and say, I’m thoroughly enjoying your blog. I too am an aspiring blog writer but I’m still new to the whole thing. Do you have any tips for newbie blog writers? I’d really appreciate it.


Friends don’t let friends use Red Hat.


I guess I have a question for Joe specifically…. since he might have more insight into the amount of kernel copying that Novell did. This article, and many others on this subject say that one reason that Red Hat did this was to prevent (specifically) Novell from using Red Hat patches in their SLES. Is this REALLY true? I mean, RHEL and SLES have never really sync’d well with regards to kernel versions on their releases. In fact, you might even be able to argue that it would be harder to use a Red Hat patch on a kernel version that was significantly different (might actually require MORE work). So… just curious…


If you look at this from a long-term perspective, this is the inevitable evolutionary fragmentation that comes about from competitive forces. It’s very true that this is now localized to something Red Hat is doing, but the claimed necessity to make life harder for independent developers as a result of competitor’s actions I think skirts the underlying philosophy of the GPL.


I agree fully with this! For anyone to complain about what Red Hat is doing is almost blasphemy! If you look at contributions over they lifespan of Linux, you will see that Red Hat is always the top contributor, usually almost at around 30%-50% of the overall contributions. I would say that if Linus Torvalds were a company it would be called Red Hat. Red Hat is the other carrier of everything that Linux has become.

Companies like Oracle, Novell and Canonical SHOULD be contributing to the betterment of Linux as well, not just their own respective companies. Red Hat does this more than anyone, and now that they are protecting themselves from the leaches, people are crapping on them. I understand it is common practice to do this, most projects are completed by a couple of people and everyone else gets the credit. Imagine how amazing Linux would be now if the companies that use Linux helped in building a better machine, rather then just painting over what everyone else built! Red Hat builds the machine and we all benefit from their work. Everyone else just paints over the Red Hat logo and calls it their own.

I think it’s time that these other companies are held more accountable for their upstream efforts. Part of the beauty of Open Source is that you are allowed to do this, it doesn’t make it morally right though! If you make money from something, it only makes sense to help build it. You will end up making more money the more people use it.

I know everything I’m saying will probably hit a brick wall, but I’m still saying it.