Kernel Driver

The Guru himself gives us his take on Open Source Software, kernel hacking, and why he'll never work in Redmond.

Kernel Driver

red hat

When Linus Torvalds thanks the thousands of contributors who have made Linux what it is today, one who is uppermost in his mind is Alan Cox. Since he first loaded Linux as a replacement for his Amiga in 1992, Cox has gone on to write much of the networking and SMP (symmetric multiprocessor) code for Linux. But his most important work is as “kernel maintainer.” Right now, that means maintaining the cutting-edge “AC” version of the kernel — a test-bed for the latest and most innovative (and sometimes unstable) patches to Linux.

Alan Cox Opener
After Linus himself, Alan Cox may be the most influential man working on Linux.


Though Torvalds is the more visible public figure, Alan Cox clearly commands the respect of the Linux community, and a comment in his widely-read Diary can cause a serious ripple in the world of Linux hackerdom. The 30 year-old Cox met with Linux Magazine Publisher Adam Goodman and Executive Editor Robert McMillan at last May’s Linux Expo in Raleigh, NC.

LINUX MAGAZINE: What was your first computer?

ALAN COX: The first computer I actually programmed was a school computer — a Commodore PET. The first I actually owned was a Sinclair ZX-81.

LM: And what was the first program you wrote?

AC: The first vaguely useful thing I wrote was a very bad lunar landing clone. I wrote that on the PET.

LM: Did you know right away that you were destined to program?

AC: Computing just kind of fit. I found that I had no real problem picking it up and seeing how everything fit together.

LM: When did you start working professionally with computers?

AC: The first serious work I did was with a school work experience program. I have been told by cynics that the point of these programs is to prove to students that they really do want to go to university. A lot of people at my school had gone to big computer companies and basically made coffee for two weeks. I had better luck.

I wanted to get into computer game programming, so I phoned a couple of Birmingham game companies and eventually got hired by Venture International, UK. I spent two weeks there helping them write bits of code — support code, that kind of stuff — before I was hired on for the rest of the summer. I was about 18 years old at the time.

LM: When did you first encounter Linux?

AC: When I first saw Linux, I was doing some UNIX work on the Amiga.I’d written most of a multi-user game, a bulletin board, and various other bits of UNIX software while at Swansea University, and I was developing a new version of the multi-user game on my Amiga. I had added all this crazy stuff to the Amiga to make it more UNIX-like. I wasn’t so conceited as to think I could have written the entire operating system! I didn’t even try.

LM: You were an Amiga-head?

AC: Sort of. [laughs] But when Linux came along, I thought “Now I can buy a 386 because there is something useful for me to run on it.” It wasn’t that I hated Windows. It was just that Windows and DOS weren’t useful to me. I didn’t see the point of buying the hardware without the software. It was as simple as that.

In the spring of 1992 I ordered a 386 PC with 4 megs of RAM and cobbled together some old disks to build a computer. I ended up picking Linux over BSD because at the time FreeBSD required a floating point chip. Although BSD was obviously far more advanced, the floating point chips were pretty expensive back then.

LM: So it was a practical decision?

AC: The original choice was pretty much a practical one, yeah.

LM: Why do you hack the kernel today?

AC: It’s fun. It’s something I enjoy doing. It’s as simple as that [laughs].

LM: How do you explain your job to people?

AC: It’s kind of hard to explain. When I say that I’m paid to give away software, it tends to confuse people. Most of what I do with the kernel involves testing and coordinating patches. I look for code that fits together, stuff that is good but needs testing — the stuff that Linus has missed. Basically, I hoover up all of the patches that appear, test them together, throw out bad stuff and then feed the good stuff to Linus.

LM: Does anyone else pick up the patches from the kernel list? Or is that officially your job?

AC: There’s no “official” anything. It seems to have become my job. There are other people who will do it — they’ll see stuff on the kernel list and ask, “Why hasn’t this patch been included?” Sometimes I find patches this way. There’s this continuous army of people looking and cross-checking for stuff. A huge chunk of it goes straight to Linus. I hoover up all the other stuff.

LM: Is there anyone else that has a comparable role to yours?

AC: There are lots of people doing specific areas. Andrea Arcangeli is doing a lot with memory management. There are people like Dave Miller and Alexey Kuznetsov, who do the network code. When I get patches for TCP/IP, I feed them to Dave and Dave will simply throw it out or approve it, and then he’ll feed it all to Linus. And there are other people who deal specifically with things like CD-ROM drivers, PS/2, bits of SCSI and strange machines. Some of them feed code to me. Some feed stuff directly to Linus.

LM: What’s your relationship with Linus? Are you two friends?

AC: Um. Kind of. Linus is a Finn, which means he doesn’t actually say anything [laughs]. Linus is very, very quiet, both by e-mail and in person.

LM: How many times have the two of you met?

AC: Four or five times, I guess. The first time I met Linus was at the first Linux Congress in 1994. By then we’d been trading stuff and working on code for quite awhile.

LM: So what do you think of him?

AC: Some of the time he drives me up the wall, but most of the time he’s right, which is even more infuriating. A large part of Linus’s job is to say “no” to things. And that can be quite a hard job. I have it much easier because I’m looking for stuff that perhaps ought to be done, and Linus’s job is to say “No that’s not going in. No that’s crap, rewrite it.”

LM: So your job is to say “yes,” while Linus’s is to say “no?”

AC: I do a lot of filtering. Sometimes you get stuff that’s a great idea, but dreadfully implemented. Other times people have done a good job, but they’ve done it the wrong way. Sometimes people write code that fights the kernel. They give you long complicated alternative ways to achieve something. There, the right thing to do is say, “Okay that’s a great idea, great interface. Throw the bit in the middle away; rewrite it to fit the way the kernel wants to work.”

Alan Cox 1

Alan Cox 2

Alan Cox 3

LM: Linus gets asked a lot about what will happen if he ever gets hit by a bus…

AC: I can’t see that happening; he lives in California.

LM: The answer Linus generally gives is that he won’t care. He’ll be dead. But you will care. So what happens if Linus gets hit by a bus?

AC: We’d probably have the largest funeral in California. I don’t know. The Linux community is not totally dependent on one person. It would shake itself out in awhile. I guess I could end up in charge of Linux, or somebody else could, or a group of people would take over bits.

LM: How would you feel if you were suddenly responsible for all of Linux?

AC: Slightly unnerved. In many ways it’s a good thing that there’s no official policy if Linus dies. It means that whatever happens, something will sort itself out. There is a clear idea of what group of people will be directly involved. Any time people leave — for example when the guy doing the Linux IPv6 work, Pedro Roque, left and went to work for Cisco — people just kind of take over the work.

The biggest problem is likely to be that all the key people now work for Linux vendors, so there might be a little bit of concern from the big vendors about that.

LM: Do you think that would lead to a balkanization of Linux?

AC: No. I was maintaining 2.0.35-36 kernel stuff even when companies other than Red Hat were shipping it. There’s a sufficient level of trust there, so that isn’t really a problem.

LM: Is there any scenario that could lead to a balkanization of Linux?

AC: I don’t really think so. You may get commercial pressures to do that, but I don’t think any of the key Linux people would allow it because it would just make their lives hard. So if anyone wanted to split something off, I think it would just curl up and die.

LM: So what is your greatest fear for Linux?

AC: My main concern is that however hard you try to keep the kernel lean, things gradually creep in. It’s very easy to put something in a kernel; it’s really hard to get it out. That’s the real thing: making sure you keep the gradual growth under control. Getting that wrong could be very bad.

LM: With all the Linux development that is being done these days, do you think that having one person — Linus Torvalds — okay every single addition to the kernel, will continue to work?

AC: There was a time when I thought not. But now I think the answer is “yes.” Linus’s role is changing slightly because there are now more people collecting patches and feeding things on to him. So he’s actually got less work to do for each patch. He still makes the final decisions about the direction of the kernel. I don’t see that changing because he is just exceedingly good at it.

LM: Linus seems very focused on the desktop and on making Linux appeal more to “regular” users. Is that the direction you want to take?

AC: I don’t think any one individual shapes where Linux is going. A lot of people want Linux on the desktop. The people who want Linux improved on the high-end tend to be the big vendors, because the high-end is where all the money is. These people influence kernel development because they either provide drivers or they fund the people who write them. The low-end stuff is being done because individuals contribute code. The high-end stuff is being done because people are putting money behind it.

LM: But aren’t there fundamental architectural decisions to be made that favor either the server or the desktop?

AC: Sometimes. You know you’ve go tit right when the same decision is correct for everything from the PalmPilot upwards. That is going to get harder. Because, for the low-end, people are starting to seriously use embedded Linux — things like RT-Linux [A real-time version of Linux]. They’re doing things like embedding Linux in tiny MP3 car stereos. At the high end, companies like SGI are starting to show an interest in Linux. So as the range of hardware support grows, it becomes more and more awkward to find things that work across all the cases.

LM: So keeping Linux unified will become more of an issue?

AC: I certainly think so. We’ll see how it plays out.

LM: How would you have it play out?

AC: I’m hoping we could find algorithms that could work for all cases. Or we may end up with situations where there are configuration decisions for small or very large machines. The real difficulty is with the very big multiprocessor machines — the NUMA [Non-Uniform Memory Architecture] machines, the clustering machines.

LM: How would you describe the relationship between big commercial vendors and Linux developers?

AC: It can be awkward. Large companies are coming into something they don’t really understand. They’re trying to do the right thing; sometimes they get it right, sometimes they get it wrong. With one company, I found myself helping one division write
a driver, while another part of the
same company was trying to stop me.

The same is true the other way. Quite a lot of the problem, especially in the past, has been because Linux people don’t always know the best way of dealing with these big companies. You often need to understand things from their point of view. A company like SGI might say “We could possibly help you with this machine, but we don’t sell it anymore, there are very few of them left, it would take somebody two or three days digging around in technical libraries to find the documentation. It’s a lot of commitment on our side for something we don’t benefit from.”

There are other vendors who genuinely have problems with Free Software. For a very small number of products, the real value is in the software not the hardware. And these guys are making $10 or $15 chips, but there are tens of thousands of man-hours of work in the software that goes with it. For them, Linux gets awkward. Because if you give away your 10,000 man-hours of software, then obviously some other vendor can come along and clone your $15 chip. In that case, the challenge is understanding why the vendor has this problem and what to do about it. And the answer is to ask, “Show us how to use the hardware. Don’t give away your 10,000 man-hour driver. If you tell us how to write the driver, then we’ll go and do our own 10,000 man hours of work.”

LM: Translucent Source.

AC: Yes.

LM: So, does this approach actually

AC: It varies. Sometimes I’ll say, “Tell us how the hardware works and we’ll do all the hard software.” And the vendor says, “Ugh! No.” Sometimes that’s an even worse prospect.

LM: This spring, Mindcraft, Inc. published the results of a Microsoft-funded study that found that NT outperformed Linux in Web and file serving. Mindcraft’s results were roundly dismissed by the Linux community, but was there a lesson there?

AC: There are a mix of things. There are chunks of the benchmark that are very badly done and chunks that look like obvious set-ups. There is a certain part of the benchmarking industry that basically ratifies the fact that a particular carefully-arranged set-up is genuinely faster than the other guy’s. That’s Mindcraft’s job. It’s not a particularly useful benchmark.

But some of the Mindcraft results are interesting — when you get machines with a lot of processors and very large amounts of RAM, other operating systems start to win on scalability. Solaris, especially, beats Linux flat on big multiprocessor Intel machines. We murder it on a single processor machine because we’re efficient. It murders us on a really big high-end machine because it is scalable. A lot of the 2.3 kernel work is going to be on scalability. It is an incentive, amongst other things, to make sure we beat everybody on every benchmark.

LM: So it was useful to you, in a sense?

AC: It would have been nicer if it had been a benchmark that had come from an established reputable standards body.

I have no problem with people saying that Linux isn’t particularly good at some jobs. I’ve had people come to me and say, “I want to put two terabytes of data on a Linux box.” And my answer is “At the moment, go buy an SGI or an AIX box or something. It’s not the right choice.”

Alan Cox 2

LM: What will be the big challenges for Linux in the next five years?

AC: I think the biggest one at the low-end is going to be all the user interface stuff. With things like GNOME [GNU Network Object Model Environment] the user interface is getting to be as easy to use as Windows. But that isn’t good enough. Windows is still the Black and Decker power tools of the computing world. We need to have something much simpler than that. A lot of people don’t want to learn how to use a computer. You shouldn’t ever have to read a manual. You shouldn’t have to deal with file managers. Why should you have to understand all this file stuff? That is the real challenge — not just for Linux, but for the whole computing community. Because if you want to take computing to the next phase, it’s got to be much, much simpler than anything currently available.

LM: Does Linux have any advantage in the race to achieve this?

AC: The obvious one is that it’s very flexible. You don’t have to fight with a vendor to customize it. You’re never in a situation of having to work around modules without source code to get a particular effect. You’ve got the entire source, so for any specialist application, it’s going to be simple. If there’s something inconvenient in the kernel, you can change it.

LM: Does Linux have any particular disadvantages in this area?

AC: I’m not sure. There are some user-interface issues at the moment. The X Window System is quite large. But of course Jim Gettys is now wandering around with X Windows on the Itsy [a prototypical PDA], and there are projects like NanoGUI that are trying to build very, very small user interfaces. People are currently working on a thing called ELKS, which is the Embeddable Linux Kernel Subset. It’s basically about getting Linux to run on the original IBM PCs and other suitable hardware. They’re working with very, very tight memory constraints. There’s no way to run X on that type of hardware.

LM: Analysts have said that Linux is too big to really take off in the embedded space.

AC: You can run Linux on a PalmPilot. For the very, very low-end devices they’re right. When you’re talking about 16-bit microcontrollers, Linux is too big. But at the point where you’re trying to support things like RealVideo, MPEG Audio, DVD playback, you are not using 16-bit microcontrollers anymore. With chips like the embedded StrongARM and the PowerPC, the price of a 32-bit processor is not much greater. So there we’ve been lucky. The technology has caught up with Linux.

LM: Is it true that Microsoft offered you a job?

AC: Yes. I’ve been offered a job with Microsoft.

LM: What was the offer?

AC: I never got that far. It got to, “We’d like to talk to you about working for Microsoft.” I said “No.” Some people saw all sorts of sinister connotations in this, but it happened because I knew people at Microsoft, and one of them thought I might be interested in some of their networking research stuff. There was no great sinister mystery about it. The timing, with respect to the Halloween documents was a bit odd [they came out the same week] but it was basically chance.

LM: What would you think if Microsoft did a Linux distribution?

AC: As long as they stuck to the rules and they were releasing GPL code, it would be fine.

LM: Do you think they would do that?

AC: I would be very surprised. If Microsoft wanted to get into the UNIX market, I suspect they would take something like FreeBSD, where the license allows them to say, “Thank you, good-bye. We’ll do this binary-only.”

LM: But doesn’t that presuppose that Microsoft doesn’t understand what is going on? Microsoft has shown more than any other company in history, an amazing capacity to learn, and a willingness to occasionally eat its own lunch if it has to. Wouldn’t they be willing to, in a sense, shoot themselves in the foot, if they could be relatively sure that the bullet would ricochet off the floor and kill everybody else in the room?

AC: I don’t think Microsoft is actually in a position to do a colossal amount of damage to Linux. Because a lot of people are now very much aware of the fact that if you buy OpenSource Software, you have choice. You can buy from one vendor, but if they screw you, you can go somewhere else. A lot of this revolves around risk. In the Microsoft case, you rely entirely on Microsoft delivering something; basically your business is resting on them. With Linux, you know you can go to another vendor. You know that in the worst case, you can hire somebody to get you out of whatever hole you’ve gotten into. So your actual risk is much lower.

LM: Did you get the sense that Microsoft “got it” from reading the Halloween documents?

AC: I think they get bits of it. I don’t think they get all of it.

LM: What do they not get?

AC: I don’t think they really understand the scale of the whole thing. The fact that you can now get code that is subject to peer review, that you can go to someone else if you don’t like your original vendor. If you buy a car from me and I screw you, you can go and buy a car from another company. You don’t have to buy a new road, a new garage, and everything else. You can just buy a new car.

LM: Would you ever consider working for Microsoft?

AC: Not Microsoft in its current form.

LM: In what form would you work for it?

AC: If it wasn’t a near-monopoly player and if it had some more reasonable business ethics. Even then I doubt it. I’ve never enjoyed working for a large company. I don’t like being in a situation where I can’t get something done because I’ve got to fill in three sheets of paper and wait a week for delivery. So working with Red Hat is fine, especially because I’m working as a contractor on the edge of Red Hat. If any piece of computer hardware blows up, I don’t have to fill out a form and wait for the service engineer to appear. I just go down the road to the computer shop, wave the broken hardware at someone and say “fix it.” I can get things done.

LM: What do you see your contracting firm, Building #3, doing in the future?

AC: At the moment Red Hat is paying me to help them with the support side of things. If you do a big support contract, you need someone at the very end to whom you can say, “Okay it’s broken. Fix it.” That’s why people like Red Hat need engineers like me, Dave Miller, Stephen Tweedie. But Red Hat is also paying me to make Linux better. That’s about as good a deal as I can hope for.

The Red Hat guys are very committed to Open Source and Free Software. Bob Young passionately believes that Linux is the right thing to be doing. So there are no real business clashes either. It’s this whole organization that believes in what they’re doing. There are a bunch of people there who don’t follow the “turn up wearing suits” thing.

Alan Cox 3

LM: Where does the name Building #3 come from?

AC: Red Hat already had buildings #1 and #2 in North Carolina at the time I signed on. Given the pain it causes FedEx delivery people, it may not have been quite a stroke of pure brilliance.

LM: What gives you your biggest sense of satisfaction?

AC: [thinks for a minute] Doing something because someone told you it was impossible is one of the good ones. The Macintosh port was really good fun. And getting a shell that was just about usable was really, really great. It was sort of like flipping twofingers to Apple. Despite the fact they gave us no documentation, no help, and basically told us to go away, we still did it anyway. That was fun.

A lot of the fun is in the strange bits of hardware you come across. There are always different challenges. Every piece of hardware I’ve come across is uniquely broken in some way.

LM: Who are your heroes?

AC: [He pauses] That’s an interesting question. [and then laughs] I’ve never really thought about that. Things that people have done stand out more to me. When I was first learning real computing, there was a system administrator at my university named Rob Ash. He was a really good system administrator, and he was very busy. But he was prepared to spend the time answering stupid questions. My class had gotten as far as we could with the documentation, and we were doing things not related to our course — trying to really find out how stuff worked. And he was prepared to sit and answer questions. That really made a difference.

LM: What is the most important lesson that you’ve learned in all the time you’ve been hacking?

AC: The most annoying one you eventually learn is that sometimes you have to just throw everything in the bin because you screwed up and did it the wrong way. And now the only way you’re ever going to get your project right is just to throw everything away and start again.

LM: How did you learn that lesson?

AC: By trying to rescue several programs that shouldn’t have been rescued. With Abermud, a multi-user game I worked on, I really learned that. I ended up rewriting the entire thing, after having spent at least a year patching on new bits, trying to fix this, that and the other thing. It was never quite right.

Fred Brooks wrote a wonderful book called The Mythical Man-Month,which I think everybody should read. Brooks had not only been through it all, but he managed to summarize it. The precise description of how not to run a software project: how it goes wrong and what you should be doing. The message you get from the book is, “be prepared to throw the first thing away.” And that’s definitely true.

LM: That brings up an interesting question. Are there any original design assumptions about Linux that are now causing the OS to bump up against a roof: the threading model, for example?

AC: I don’t think so. There are lots of cases where, to get the best performance, you want to really think how to redo stuff. What we’ve been doing there, especially with 2.2, is making drivers with all the nice locking stuff in them run much faster on a multiprocessor machine. If they don’t have these features, things still work. And that’s really important. It’s possible to write a very naive driver. And once you’ve got it working, you can improve it, optimize it, and effectively you can then tell the kernel, “Hey I’m clever, I want to take on all these roles. I’m going to screw up on my own without your help.”

LM: So will Linux eventually scale to an unlimited number of processors? How far can it go?

AC: There’s a certain point where conventional SMP machines cease to work. NUMA machines work a bit further above that. At 64 processors, the UltraSPARC is as far as anyone’s gotten with an SMP machine. Beyond that you get into clustering. At the moment, Linux does explicit clustering with software like Beowulf. With Beowulf, you design programs for the cluster. Now that’s how a lot of high-end stuff works. It’s the only way you get performance. But there are things like MOSIX, where you can take eight machines, chain them together with fast Ethernet, run the MOSIX modules on all of them, and share processes around all the machines. You won’t get the best cluster performance, but at least you are administrating a single computer.

LM: How long do you think people will be using Linux? What’s its life span?

AC: I have no idea. Look at how well DOS is doing at the moment. I think a long time. There’s a lot of incentive now not to write a new system. With a lot of research operating systems, you have one great idea and the rest is rather rough, because that wasn’t the point of the exercise. But with something like Linux, you can rip out the chunk you want to change and put your code into it. So at least initially there will be a lot of cases where people have a grand new idea they build around Linux.

Eventually someone will write a new kernel, borrowing freely from the Linux kernel. They might take our device drivers and networking stuff, but do their own memory management. Linux’s long-term future may well be as a kit of parts for some better future operating system.

LM: When you first started hacking Linux, did you ever foresee it becoming this grand commercial endeavor?

AC: No, it was this neat little hack project. I never expected it to be anything more than that.

LM: Are you surprised by what’s happened?

AC: It continuously surprises me. Every week, at the moment.

Robert McMillan is Executive Editor for Linux Magazine. He can be reached at bob@linux-mag.com.

Comments are closed.