He may be benevolent and he may be delegating more work, but the Linux kernel remains Linus's project.
The Linux kernel project is growing up. New contributors, job delegation, and a source code control system have changed the way the kernel gets hacked. The guy at the center of it all — Linus Torvalds — has changed, too. Gone are his days as poster boy for Open Source. He doesn’t do Comdex keynotes anymore; he’s not on the covers of business magazines; and he rarely gives interviews. No, these days, Linus is all about what he does best: hacking the kernel and keeping a sure and steady hand on the rudder of everyone’s favorite project.
LINUX MAGAZINE: Tell us about how your job has evolved in the last few years.
LINUS TORVALDS: One of the things that I wanted to do with Bitkeeper [the source code control system Linus recently adopted] was to see how easy it is to have other people in charge of their own areas. So far, [it's] worked really well with some people. Before, I had to merge patches, and I always did it, but it was still work for me. These days, with those people I know I can work with, it takes a second every week. It’s not a big deal anymore. I get an email that says, “Please pull.” I pull it. It’s done. And that’s very convenient.
It works very well within certain subsystems. USB is a perfect example of this. Greg [Kroah-Hartman] just handles all of USB. Nobody even tries to send me USB patches normally. He’s the guy. In part, networking and network drivers are the same way.
There is still a problem though. Fundamental patches that aren’t clearly in one sub-category still take some work, but there are a few people who handle that, too. Andrew Morton has been really good at memory management, which used to be a major point of contention.
LM: Why was it a point of contention?
LT: There are a lot of people who are interested in memory management. Memory management is just something so fundamental to just about any subsystem. From a user perspective, it’s a big deal how you page things up if you’re low on memory. But it ends up being a big deal for filesystems, for example, that maintain their journals and have tons of memory for journal data. But then, when the kernel needs memory for other things, the kernel has to have a good way of communicating with the filesystem, saying, “Hey, get rid of that data without the machine coming to a complete halt.”
So memory management tends to be caught in the middle of all these different things, which means that it ends up being one of the most complex pieces, and at the same time, the most fundamental. It needs to be 100 percent stable, and at the same time, there are many different people who have different things they want to do with it.
LM: Last year, you switched Linux’s virtual memory (VM) code in the midst of a stable release development cycle. That ruffled a lot of feathers. Now that you have some perspective, would you do the same thing again?
TORVALDS: I still think it went about as well as it could go. One of the things about the way it turned out was the fact that it forced people to take a real stand. Without the VM split in 2.4, I don’t think anything would have gotten done. That’s the reason that I think it was actually good.
LM: Were you thinking that at the time?
TORVALDS: That was the major reason. I mean, everybody had been ragging on the VM system for several months — this was at 2.4.9. It worked OK, but there were certain loads where it just did not work very well. And there were no solutions in sight. And then Andrea [Arcangeli] came along and said, “If you do this, it fixes the problem.” And it did. It broke a lot of stuff, too. It took just two releases to be quite useful again. But it was a big enough change that 50 percent of all kernel developers said, “Hey, that is way out of line. Don’t do that.” And to some degree that is true, but without [the change] the VM wouldn’t have gotten fixed.
And then the Red Hat people stayed at 2.4.9, and suddenly there was this fierce competition going on. The 2.4.9 VM lost every single benchmark out there, so you could point to benchmarks and say, “The 2.4.9 benchmark sucks, and this new kind of radical thing beats the pants off it.”
LM: So were you surprised by the reaction at all?
TORVALDS: Some people were more bitter than I expected them to be.
LM: Do you think that we could be in a situation like that again?
TORVALDS: Right now the other issue is the IDE layer, which has a lot of the same issues. The fundamental problem is that most drivers are very independent. There are no connections between different network drivers, which makes it much easier to maintain them. When you make a change to the EEPro driver, you don’t want to, by mistake, break two other drivers that just happen to depend on what you’ve changed. So network drivers rarely have concerns.
In the IDE layer, that’s not true. So you have different IDE chipsets assuming different things, and some of them work and some of them don’t, and when you try to fix one, you may break another. It’s kind of messy.
So during 2.5, I said, “OK, let’s go with this radical fairly open rewrite.” And again, a lot of people didn’t like the instability. In fact, they didn’t like it so much that Martin Dalecki, who did the thing, has basically thrown his hands up and said, “OK, I can’t take the abuse any more.” So that particular rewrite actually ended up going back to the old code just because it takes a certain kind of masochist to actually take the abuse.
The Next Kernel
LM: What will the next kernel version number be? 3.0 or 2.6?
TORVALDS: 2.6. There’s a lot of new stuff in the current kernel — especially scalability work — but for me, the decision whether it was 2.6 or 3.0 was going to be if there was a big push toward new kinds of architectures, especially NUMA. And there has been some work on that, but most scalability work has been done on SMP. IBM has been doing a really good job, but it’s not fundamentally different, so that’s why it’s 2.6.
LM: When is 2.6 expected?
TORVALDS: The feature freeze is Halloween.
LM: You always pick a memorable date.
TORVALDS: We talked this over at the OLS [Ottawa Linux Symposium]. Once we decided it was October something, you might as well just say Halloween. So Halloween is feature freeze. Practically speaking, the IDE situation, because it seems that we’re just going with the old stable code, that’s not going to be a big issue. There are a few other issues, which means that after feature freeze there will be a long time before we can do a real code freeze. The major one is async I/O. That is right now one of the major issues that has changed so much, that it had better be there comfortably before the feature freeze. And that’s the one that might take a while after the feature freeze to stabilize enough that we can say, “It’s a code freeze.” So, realistically, it will be at least six months after the feature freeze.
LM: So, improved SMP and async I/O are going to be the two most interesting 2.6 features, in your opinion?
TORVALDS: There are a lot of other things. Actually, fairly recently, Ingo [Molnar], who has been doing most of the scheduling work, and the process related work, started working together with the glibc guys on improving threading. We’ve had this situation for a long time where Linux has this cloned thread abstraction, and most of userland wants to really see a PT threads-compatible layer. And the glibc translation overhead was horribly bad, and there was nobody who was comfortable with both glibc and the kernel to try to figure out what the right abstraction layers were.
Oracle, and basically all the database vendors want this interface to have big pages. They want to get the OS memory manager out of the way. They want to say, “Hey I want one contiguous area — 256 meg — and I want to use all the big page TLB [Translation Lookaside Buffer] features because I know my locality sucks. So I want to have as little a TLB footprint as I can.”
LM: These seem like server, not desktop, features.
TORVALDS: The desktop turns out to be mostly about drivers. On the desktop side, most of the work is actually in userspace. So a lot of this stuff, has been people picking up the Xrender extension, and doing some of the 3D graphics work, and doing anti-alias fonts. And none of that has anything to do with the kernel. The 3D graphics has a kernel side to it, but it’s not something fundamental. It’s just a fairly complex device driver model. So in the kernel, there’s not a lot of support needed for desktop uses of the kernel.
Then there’s the embedded space. The embedded space has also grown up so much that you don’t have to be a 100K kernel to run there, and the standard Linux kernel actually works pretty well. There tends to be ARM support — which has been ongoing all the time — and device drivers for it. So there is very little fundamental stuff for the desktop. The threading stuff may actually make a big difference for the desktop, too.
Not His Cup of Itanium
LM: I read a kernel post where you said you were praying that AMD’s x86-64 chip architecture would succeed. So what have you got against Itanium?
TORVALDS: I think the main problem with Itanium is that I don’t care about the server market that much. To me Itanium is a complete dead end because Intel doesn’t even seem to be pushing it into the markets I care about.
One reason I really like the AMD approach — the way I’ve read it — is Optiron is going to be a desktop chip, [and] because of its 64-bit features, [it] can do better in the server market, too. I like that approach a lot more, because I’m a big believer in desktops. That’s what I use every single day. I refuse to have a rackmount in my home. I don’t want to have a coffee table either. I want to have a standard PC form factor — smaller if possible. It needs to fit under my desk, and it needs to be fairly quiet, and it needs to be small enough that I don’t hit it with my foot by mistake.
I’m actually hoping to get a desktop Itanium 2 soon enough, but I know that’s not what most people are going to use. This [Itanium] thing costs several thousand dollars. It’s not going to run x86 legacy apps very much, so most people on the desktop won’t buy one of those because it’s just too expensive for them. It’s for very high-end workstations.
The thing is, they’ve been doing well without me. I’ve not written a line of Itanium 2 code, and apparently the people who are actually working on Itanium 2 and Linux are really happy. It’s integrated. I don’t mind. I will support them.
LM: I wanted to ask you a little bit about your job. Transmeta laid off, what, 40 percent of its staff a few months ago?
TORVALDS: Bloody Thursday. I don’t think it was 40 percent, but it was close.
LM: What was that like?
TORVALDS: I’m not a manager. I used to be a long time ago, and I just said, “I never want to do it again.” But because I’m a Fellow at Transmeta, I actually ended up being one of the people who got to go around and tell people, “OK, I’m sorry but you’re laid off.” So, I did that to three friends of mine that I’d been working with for over five years.
It was one of those things where it was clear it had to be done. I think I actually preferred being one of the people telling others they had to be laid off, [instead of] just walking around being one of the people staying. Especially, with some of them, it was like, “OK, if somebody has to do this, it might as well be me. It’s just better that it’s me.”
LM: Had you brought people to Transmeta? Did you Encourage friends to get hired there?
TORVALDS: I hadn’t actually done that very much. Partly because all my personal friends tended to be from Finland, and a lot of my Linux contacts were there before me. So I didn’t have to feel that particular feeling of guilt.
LM: It must feel like such a different world from when you came here.
TORVALDS: It’s psychologically very different. But I think that’s true in Silicon Valley in general. At least I didn’t come in like a lot of people came in, in the last two years. Just as they came in, it was at the height of the craze, and some people had jobs for just a few months, and then they were told, “OK, I’m sorry. It didn’t work out after all. Go home.”
The Up-and-Comer from Redmond
LM: Did you go to the Microsoft booth at LinuxWorld this year?
TORVALDS: I walked past, and I looked. I mean, they should have shown off what they’re good at. I think they were showing off Korn shell running on NT. If you want to run Korn shell, don’t run NT.
LM: What do you think they were doing here?
TORVALDS: I have no idea. Maybe Bill has told them that they’re going to port all of their software to Linux, so they’d better get used to it. Maybe. Maybe not.
I think it’s cheap enough for Microsoft to come there, and maybe they can talk to some people and say, “Use NT instead. We have all the tools. If you like a shell, you can run that.” That’s what they’re doing. It’s probably not a big strategic decision.
LM: Maybe it’s just a PR stunt.
TORVALDS: Maybe that’s it. Maybe they just like being written up in the New York Times… and I’m sure, in Linux Magazine, too.
I found it kind of funny that, they were across the aisle from CodeWeavers, and how the LinuxWorld show guide said they were in a section for small, upcoming companies who hope to grow.
Robert McMillan is Editor at Large for Linux Magazine. He can be reached at firstname.lastname@example.org