Sparking the Embedded Revolution:

Does Linux really spell the end of proprietary embedded operating systems? Lineo CEO Bryan Sparks certainly thinks so -- and he's doing everything he can to make it happen.


Does Linux really spell the end of proprietary embedded operating systems? Lineo CEO Bryan Sparks certainly thinks so — and he’s doing everything he can to make it happen.

When all is said and done, Linux may have the greatest impact where it is the hardest to see — in the phones and PDAs and routers that, more and more, are being powered by the Penguin. One of the people at the forefront of the embedded Linux phenomenon is Bryan Sparks, CEO of Lindon, UT-based Lineo. After his Linux company Caldera bought DR-DOS from Novell in 1996, Sparks began building a new line of business selling software and services to the embedded market. Before long, Sparks realized that it was Linux and not DR-DOS that held the most potential. Two years later, the embedded business was spun off from Caldera, and Lineo was born. Lineo filed for an IPO in the fall of 2000. Due to unfortunate market conditions, they were forced to call it off in January 2001. Despite this, when Sparks recently sat down to dinner with Linux Magazine’s Publisher Adam Goodman and Contributing Editor Robert McMillan, he was highly optimistic about the future of both embedded Linux and his own company.

LINUX MAGAZINE: How did Lineo come to be spun out of Caldera?

BRIAN SPARKS: Two years ago we weren’t even called Lineo. We were running under the awkward name “Caldera Thin Clients.” We were building these TV set-tops, software solutions. We had a browser that we were writing just for TV output, and it was not even Linux- based. It was DOS-based. At that time, there was this guy building a DSL router who came to us thinking that we would do a DOS-based router deal. We said, “You don’t want to do that. Tell you what, we’ll help you with Linux.” So we did a deal with him and put embedded Linux on his device. Then we got approached to put Linux on another device by a big device manufacture. We did a flash driver for them and that’s when we said, “Gosh, what are we doing? You know what? We need to be doing embedded Linux.” So in early 1999, we started talking to Motorola and a bunch of other vendors. Motorola caught on first and said, “We’re thinking about doing Linux,” and they showed interest and helped us define what products we should build. So, that’s how we got here.

LM: In the last two years, has the embedded Linux market matured in ways that surprised you?

BS: Yes. Two years ago I thought I would find all our success in areas where Linux was strong: networking, routing, firewalls, gateways, VPN [Virtual Private Networks], those kinds of things. The big surprise for us has been the demand for Linux for consumer electronics. It was not what we had anticipated. Linux alone doesn’t have an intrinsic GUI for embedded systems. You read about X-based applications. Well, X doesn’t work on embedded devices. It’s a pig no matter how small you try to make it. It just doesn’t work. However, even without an intrinsic GUI, there was still interest for Linux in embedded, consumer electronics that have graphical user interfaces, simple as they might be.




LM: Why are these consumer electronics developers interested in Linux?

BS:I think it’s because we offer a richer technology base and a richer technology roadmap than the traditional RTOS [Real Time Operating System] vendors offer. It’s not that it’s open source, though that’s interesting too because it gives the OEM (Original Equipment Manufacturer) a little more freedom than they would have if they were tied to someone like Wind River, who charges them for fixes to bugs that they find.

Certainly cost is involved, because Linux is perceived to be cheaper and indeed is. But I really think that it’s a richer technology roadmap. I can do more, faster than an RTOS vendor, sometimes on an order of magnitude. We matched VxWorks’ [Wind River's embedded operating system] functionality in our first release of Embeddix [Lineo's embedded Linux distribution]. With our second release — man, we’re way ahead. I mean, we’re doing stuff I think they’re just now starting to talk about.

LM: Is that because there is a community of developers that are contributing to the same core technology?

BS: Of course — and there’s my ability to attract quality engineers and make them effective and productive sooner.

LM: So what is the difference between you and the other embedded Linux vendors? For example, you seem to approach real-time Linux in a different fashion from MontaVista.

BS: In order to provide hard real-time guaranteed determinism, there are several things we have to do. If you’re an application on Red Hat and you want hard real time on Red Hat 6.2, you can’t get that, because you are at the whim of the scheduler that schedules you. You can be a high-priority task, but if there are five other high-priority tasks, you’re dead. So, we’ve had to introduce a layer of determinism that architecturally sits underneath the Linux kernel. Computer science academics call it an executive. It’s a real-time executive that runs Linux as a task.

LM: Like a microkernel?

BS: Like a microkernel, yes. But, you see microkernels — Mach or Chorus or some of the others — those were micro only on paper. In actual implementation they were macro. Man, they were huge! Mach was just massive. And with Mach and Chorus you ran operating system personalities on top. We’re not that full featured but we do have this little layer underneath, this run-time executive. And it’s not an OS; we don’t provide all those services. We expose an API and we have a whole developer kit that allows tasks to register their desire for real-time services. That’s how we do it. That’s hard real time. We also do soft real time, which is really just having high priority tasks. That’s what Monte Vista does. Monte Vista went in and modified the Linux scheduler. We did the same thing to essentially create a higher priority granularity in the scheduler.

LM: It seems like there are a variety of different forks of the embedded Linux kernel out there.

BS:What we don’t want is… OK, we splintered Linux but not on purpose. We basically put Linux on devices that the community doesn’t have momentum on — like an ARM chip. The community, in general, has made no effort to put Linux on the ARM chip. In that sense we’re kind of a derivative. We don’t mean to be. We try to give back but we’re not normally picked up by the community because, again, there’s just not a lot of interest. So in some areas we do branch. In the real-time space we’re probably branching a bit too because, in general, the community doesn’t really care.

LM: I’ve wondered about this. Everyone talks about the embedded market as an important space for Linux, but I’m not sure it really means anything to the community running Linux on their desktops and servers. Do you think it will?

BS:I don’t know if it will or not. You think of the Linux community as a great big mass of people, right? So they all congregate around what the mass feels is interesting. Servers are interesting. For example, with Linux 2.4 there’s a great big push for a better SMP. Those kinds of things don’t really play in the embedded space. We play on ARM chips and Hitachi chips — things that are not interesting to the community at large. You can’t buy a Hitachi-based PC. So I’m not sure there really is a connection to the community.

LM: Are you saying you don’t really feel connected to the Linus Torvalds and Alan Coxes of the world?

BS: Oh, we do. We make changes and all of that. However, we do things that make the community in general go, “Boy, those are weird people over there.”

LM: Do you think that it’s inevitable that there will be many kernel forks with all these weird changes?

BS: We move to some different — I don’t want to say weird, but certainly divergent from the community — platforms. I’ll give you a concrete example. We run on devices that have chips that don’t have an onboard hardware MMU [Memory Management Unit]. So there isn’t an MMU, and so you have to modify the Linux kernel, which makes substantial use of hardware MMUs. For example, the exec system call is copy on write; that’s an MMU function. We have to write our own exec for MMU-less chips. That’s a fork.

LM: So do you have to consciously find ways to work with other embedded Linux vendors to minimize forking and try to get the most out of the work that you’re all doing?

BS: Well, you know what? That was the original intent of the Embedded Linux Consortium. Perhaps we did that wrong, but that was what we wanted to accomplish. It’s not going to do that unfortunately.

LM: Why not?

BS: I mean, Wind River is a member. They don’t even talk Linux anymore. I don’t even know what they are doing. The notion was, “Let’s create conventions since we’re in virgin territory for the Linux community.” One obvious thing is how to launch apps on embedded devices that don’t have command-line prompts? Let’s just come up with a convention. Nobody gets a competitive advantage out of it. Let’s just come up with one we all agree on. That was what I wanted from the Embedded Linux Consortium. It didn’t do that. It quickly turned into, “Well, I want my time,” and, “Oh, we should be marketing,” and…oh man!

So now we’re just kind of doing our own thing. I think we’d all agree that nobody sees a competitive advantage in the way we launch applications.

LM: Is the name Linux going to really mean anything in the long run if you guys have all these little forks and your competitors do things in different ways?

BS: It may not matter. Do we care if it’s Linux? Well yeah, I don’t want to shoot myself in the foot. My biggest differentiation right now is that I’m Linux. Linux is a means to an end for us, so it would be silly to drop Linux and do BSD. However, we are not a company that believes everything we build must be open source.

We don’t challenge the GPL. In fact, we’re working with the Free Software Foundation to actually define it a little better — but we’re going to add proprietary value and we think we should get paid for our value-add. If you don’t want it, that’s fine. Build it yourself. That’s capitalism in its purest sense.

So it’s not that we’re just forking Linux. I don’t want you to get the impression that we are. We’re just taking Linux into areas that the community in general isn’t going into. I don’t call that a fork. You might, but we don’t really care if the community picks up the things we add because we’re not publishing source to them.

LM: Then why did you choose Linux?

BS:The paradigm shift — If I’d done just another RTOS, I would have had no chance of knocking Wind River out.

LM: So you’re riding the momentum essentially?

BS: Well, we’re riding the momentum and taking advantage of the inherent qualities that differentiate us from our largest competitors. How do you take your competitor’s greatest strength and turn it into his weakness? You can’t change his strength.

Wind River’s strength is VxWorks. They can’t just swap that out, because $400 million a year in revenue would simply disappear. So you ask, “How can we take that out?” Well, you know what? We commoditize it. Wind River talks about raising per-unit value in time; they tell their investors that. I say we’re here to commoditize system software for embedded devices. That hurts them and gives me opportunity.

I can do exactly the same stuff — technology roadmap, feature set — with less resource leveraging, and then move faster and charge less for it. That’s a paradigm shift.

LM: What do you think of your competitors who are responding by embracing Linux and putting Linux APIs on top of their own operating systems?

BS: Well, I’m just sitting in my little office at work going, “Boy, if I was my competitor, what would I do now?” It’s a logical thing to do. But that doesn’t help them really. It doesn’t enrich their roadmap. They are just following at that point. The more they wait the harder it gets, because the more brand value we create the more customer relationships we establish.

LM: What exactly are you making your money from right now?

BS:We sell tools.

LM: Is that where you make the bulk of your revenue?

BS: At this stage, we make the bulk of our money in services. But all of them have to do with product sales, so we still have many design wins. We’re early in our company. We have a lot of design starts. Until those products actually ship and we start seeing residuals, we’ll fill out our revenue stream with “hand holding.” So we have three ways to make money: real product that we charge for — per unit — sometimes for our value add, tools, and professional services.

We’re not a job shop. We do professional services as it relates to a product sale. If somebody comes to us and says, “The only thing I want is for you to help me do this with that Linux device; I’ll never see you again,” we’ll say, “Go do it yourself. Good luck. You’ll be back.”

Robert McMillan is a contributing editor with Linux Magazine. He can be reached at bob@linux-mag.com.

Comments are closed.