The State of the Penguin: Telecom

From data centers to hand held receivers, Linux is changing how manufacturers and carriers do business — for the better.

If you want to get a idea of where the telecommunications industry may be heading, just take a look at where Haavard Nord has been. The chief executive officer of the Norwegian software company Trolltech, a company best known for writing the Qt graphics engine that lies at the heart of the open source graphical user interface KDE, Nord says he first noticed a sudden shift in travel destinations about four years ago.

Prior to that, his company had put together a stripped-down version of Qt for embedded devices such as personal digital assistants (PDAs). Licensed to Sharp for use in the Sauron PDA, the devices sold erratically, but Trolltech’s engineering strategy — writing the graphics directly into the frame buffer of the underlying Linux operating system instead of a standalone GUI — won raves from cellphone manufacturers eager to take a bite out of Nokia’s dominant market share.

By the end of 2003, Haavard says, the usual European and North American stamps in his passport book were being crowded out by Chinese, Japanese, and Indian stamps.

“I spent lots of time traveling, especially in Asia, noting the gradual shift in mobile telephony activity in the wake of the 2000-2001 technology bust. “I was very frequently asked about solutions written specifically for mobile phones. That’s what most of the customers were looking for.”

As the frequent flyer miles mounted, Haavard and Trolltech management decided to attempt yet another repositioning. Instead of building its graphics package— now dubbed Qtopia — specifically for PDAs, the company built it specifically for the new class of so-called “smart phones.” After all, says Haavard, the convergence in voice and data was creating nothing short of a generational shift, putting pressure on older hardware and software formats alike. In such an environment, companies would be hesitant to rewrite their software operating systems from scratch, so why not save them the trouble with a phone-ready Linux package?

That was 2003.

In 2004, Trolltech rolled out Qtopia for phones. Since then, things have gotten a little crazy. Judging by the number of companies that have contracted with Trolltech, Nord betrays his native Scandinavian’s penchant for pessimism and predicts that as many as 100 million phones could be running some version of Linux by the end of 2008.

“One year ago, we were working with two handset manufacturers,” Nord says. “Today, we’re working with more than 20.”

While some might chuckle at Nord’s prediction of 100 million phones — the current top-selling smart phone operating system, Symbian OS, posted a little more than 14 million shipments last year — the secondary observation that many manufacturers marketers and service providers are taking a sudden interest in Linux is coming in from all quarters. From server clusters to handheld devices to the internal tools once powered by proprietary software, Linux is gaining fast admirers in an industry where technology solutions rarely come cheap.

“That was really the key to our taking off,” notes Mark Spencer, founder and president of Digium, an Alabama company that develops and sells an open source private branch exchange, or PBX, running atop Linux. Dubbed Asterisk[ see http://www.linux-mag.com/2004-09], Digium’s open PBX is targeted towards businesses looking to shift their phone networks onto the Internet without paying proprietary software costs.

“Before 2001, people would spend money on telecom without ever thinking about it,” says Spencer, noting the year U.S. companies collectively put a brake on technology infrastructure spending. “When 2001 happened, people stopped for just a second and thought, ‘What else can I do? What other options could I consider before I write this check for $80,000 to Cisco?’”

Bill Weinberg, open source architecture specialist for the Open Source Development Laboratories, says even telecommunications businesses are starting to ask the “What else can I try?” question when it comes to the software. Again, a lot of it comes down to cost. From the spring of 2001 to the fall of 2002, the Nasdaq Telecommunications Index plummetted more than 90 percent. While the index performance has since doubled, its recovery continues to lag the overall Nasdaq. Such sluggishness is a reflection of the sector’s lingering troubles when it comes to excess bandwidth and under-utilized hardware, troubles which have had a secondary effect on research and development.

“When the telecommunications downturn happened, companies were forced to trim labor costs and to put a lot of designs on the shelf,” notes Weinberg. “Years later, these same companies don’t have the time to resurrect old technologies. As a result, the fastest way forward is to leverage common, off-the-shelf hardware and open source software.”

On the service provider side of the industry, that means replacing aging Unix systems with Linux-based systems, a process OSDL has attempted to facilitate through its Carrier Grade Linux (CGL) initiative. One of three development initiatives — the other two focus on the data center and the desktop environments respectively — CGL focuses entirely on increasing the viability of Linux as a telecommunications platform.

The term “carrier grade” alludes to the higher expectations telephony services impose on commercial software. Given the number of customers served, basic and specialized telecom service providers[ “carriers” in industry parlance] suffer severe economic, not to mention regulatory, penalties whenever their networks go down. For most companies and the hardware vendors that serve them, 99.9995 to 99.9999 percent uptime — that is, less than two minutes of unplanned outage time within a single calendar year — is the absolute minimum when it comes to network performance.

Linux, while not built to high availability specifications, can be sufficiently “hardened” to achieve this “6 nines” performance level as long as all companies and Linux vendors agree on what features and processes merit common priority. Hammering out that vision is where OSDL comes in. The Portland, Oregon non-profit has brought together telephone equipment manufacturers (Nokia, Ericsson, Alcatel, NEC), networking hardware manufacturers (Cisco, Intel, HP), and embedded software vendors (MontaVista Software, LynuxWorks, Wind River) to forge the first CGL specification in 2002. It has since shepherded through two major upgrades.

Weinberg hesitates to use the word “standard” when talking about CGL. Still, he sees CGL as a crucial step in the overall “IP-ification” of telephone services. Without a common vision for what a telephone-worthy version of Linux might look like, companies might lose valuable time to wasteful overlapping development. The entire reason so many companies have set aside their competitive differences and taken part in CGL, Weinberg says, is to take advantage of the rapid time-to-market cycles that emerge via a common, interoperable software platform.

“These companies have to turn to Linux to survive,” Weinberg says, noting that with Linux, companies gain the ability to outsource application development costs onto the open source community and channel the resulting savings into marketing and brand-differentiation.

Another reason telecoms have grown comfortable with the idea of running Linux on the network side is because many telecoms are already running high-end versions of Unix, an operating system born within and nutured by the telecommunications industry. Given Linux’s informal reputation as a Unix offshoot, many companies have found it natural next step when considering the switch from high performance hardware systems to lower cost, Intel hardware systems.

“We kind of view Linux as the next iteration of Unix,” says Ali Kafel, vice president of telecom business at Stratus, a Maynard, Massachusetts company specializing in low-cost, high availability network systems.

Stratus specializes in building networks that offer the same fault tolerance as traditional Unix networks at fraction of the cost. That means running Linux or Windows NT on servers and using lower cost Intel hardware in place of traditional RISC-based systems. To achieve the necessary fault tolerance —. the ability to handle single point server failures common to low-cost networks — the company assigns a pair of CPUs to each incoming call or customer transaction. Even with two machines handling each service request (such as call-forwarding, mobile email, three-way conferencing, and so on), the cost savings is enough to cut the average telecom service provider’s hardware budget by 90 percent.

“The price per system varies depending on how many disks you use and how much memory you need, but, in general, you’re talking tens of thousands rather than hundreds of thousands,” Kafels says.

Although both Linux and Windows can run atop Intel hardware, Kafel says the perceived Unix affiliation makes Linux the natural first choice for the telecommunications companies on the Stratus client list. Three years ago, 50 percent of responding customers would have deemed Linux unworthy for “prime time.” Today, the number has dropped below 25 percent.

“We’ve had some very nice experiences where the customer has been able to take their applications from one version of Unix to Linux without a whole lot of code change,” Kafel says. “All they’ve had to do is recompile, relink, and retest.”

Such application portability is similarly valuable on the handset side of the marketplace, where the latest upgrade in handset standards — the so-called third generation of 3G standards — is creating a disconnect between current and past software architectures. Where past handsets employed licensed, proprietary embedded software built to device and service network specification, today’s handsets require an added layer of Internet compatibility to handle data and, in some cases, voice over IP traffic. In this corner of the marketplace, Linux operating system’s reputation as a “network native” operating system puts it in the running with other mobile software platforms, namely Palm, Windows Mobile, and Symbian.

Neil Strother, a senior analyst with the communications market research firm In-Stat, currently pegs the London-based Symbian as the leader in the foot race. According to Symbian’s own news reports, manufacturers such as Fujitsu, Motorola, Nokia, and Panasonic shipped 14.4 million phones running the Symbian OS. Citing similar company-provided stats, Strother puts Research in Motion Ltd., the company behind the BlackBerry data delivery device in second place with 1.7 million. Bringing up the rear are Windows Mobile and Palm OS, systems still trying to find a place in the smartphone market.

“Right now I’d put Linux in the ‘other’ category,” Strother says, estimating current shipments at well below the 1 million mark.

That said, Strother sees big companies, most notably Motorola, keeping a foot in Linux development if only to prepare themselves for the day when today’s smart phones become tomorrow’s bargain-basement giveaways.

“I wouldn’t count Linux out,” says Strother, “but I think Linux has some formidable OSes and hardware makers still out there.”

MontaVista vice president of marketing Peder Ulander has a different view. Like Trolltech’s Nord, Ulander reports plenty of interest from the networking equipment providers and handset manufacturers hovering just below the name brand level. Ulander reports that 10 of the top 10 NEPS are running MontaVista’s embedded, carrier grade version of Linux, dubbed Hard Hat Linux. Meanwhile, the company has “engaged” six out of the top 10 device manufacturers.

“It’s gotten hot. There’s no question about it,” Ulander says.

Like Trolltech, MontaVista has responded to the heat by rebranding around the mobile telephony industry. That means reconfiguring its real time, continuously available operating system built for switches and other network components into a cell phone operating system. Most of the companies responding to this repositioning are manufacturers leery of Symbian, a company originally backed by Psion but now backed by market leader Nokia, yet also leery of investing too much into an untested strain of Linux

“The frustration for a lot of these companies is that many don’t want to do the heavy lifting that Motorola did two years ago,” says Ulander, noting Motorola’s 2003 move into Linux smartphones. “Linux mirrors what they would get from a Symbian or a Microsoft in that it comes 80 percent done.”

In some cases, Ulander notes, 80 percent is more than enough. It shouldn’t be surprising that most of the companies interested in Linux handsets reside in China, a country where all products move quickly to commodity prices. For Chinese companies who know they can’t compete with Nokia at the feature or brand-popularity level, the GPL provides room to move in the other direction: downmarket. Engineers can strip out unneeded software elements, shrinking the operating system down to the bare essentials, thus facilitating an equally minimal approach on the hardware side.

For more ambitious companies, the Linux kernel’s scalability offers a chance to market an entire family of phones, each positioned at a well-defined price point.

“Multiple platforms fragment development,” says Ulander. “That Linux can actually play on those single-chip $40 phones all the way up to those $1,000 multi-chip smart phones gives companies the ability to leverage engineering decisions across the board.”

Trolltech’s Nord illustrates the handset manufacturer perspective. “A large company like Motorola has a dozen different architectures and in-house systems,” he says. “If you want to innovate in one architecture, you can’t share information across other architectures.”

What’s more, says Nord, companies are losing the ability to differeniate products at the hardware level. The arrival of 32-bit chips has opened the market to commodity manufacturers such as Intel and Texas Instruments, pushing innovation to the software side. This push has, in turn, heightened the demand for ready applications and the operating systems flexible enough to run them.

“The problem with a home grown operating systems is there’s basically no ecosystem around it,” Nord says. “If you need a mail client, you have to build it yourself. If you need games you contract a game developer to write them for you. If you need a web browser you have to build it yourself. The Web is getting extremely complex. Not everybody can write a good web browser any longer.”

What companies can do, however, is create a compelling user interface that integrates common applications and creates a unique brand identity. Nord cites the case of Ningbo Bird, a company currently challenging Motorola for the number one slot in the booming Chinese cell phone market. One of the few domestic cell phone suppliers to invest heavily in research and design, Ningbo Bird uses Linux and Qtopia to speed market turnaround, revamping its user interface to keep pace with the latest songs, movies, and pop culture trends.

“The freedom and the control over their own business are very imporant,” says Nord. “When you combine that freedom with the power of Linux to scale across multiple devices, it forms an extremely compelling argument.”

Such scalability can be just as powerful on the service provider side of the business as well. Monica Kumar, senior manager for Linux strategy and solutions at Oracle, says most of her customers are service providers eager to squeeze every last bit of revenue out of each CPU cycle.

“Software is helping them do that,” Kumar says. “The mainstream operators of landlines are still upgrading their infrastructure, but the wireless carriers have really been leading the way with a lot of applications available now for managing information contacts, downloading data, and doing traditional types of computer actions over the cell phone or a WiFi laptop.”

Kumar sees three major milestones over the last three year marking companies’ increased acceptance of Linux. The first milestone was the 2002 release of Red Hat Advanced Server 2.1 with asynchronous I/O support, the first commercial Linux server product designed specifically for applications requiring high availability. “That was a big turning point in terms of momentum,” Kumar says.

The second milestone was Oracle’s own internal decision to shift its 9,000 member development team to the Linux platform internally, a move that psychologically underlined the operating system’s strategic value internally, while at the same time speeding the delivery of the company’s 10g database for Linux systems. The final milestone, Kumar says, was the October 2004 Gartner conference in which company analysts finally acknowledged the operating system’s maturity as an enterprise platform.

Oracle’s stake in this Linux marketplace currently revolves around its Oracle Database 10g, a relational database built to run on grid or cluster-based computer systems. In terms of telecommunications, 10g offers service providers the ability to track account data and user privileges, while at the same time providing scalable access to the business applications seeking to employ that data.

One telecom service provider taking advantage of the Linux and Oracle combination is the Reston, Virginia-based company Talk America. Selling local and long distance service to consumers and businesses in various regional markets, the company’s revenues amount to little more than a rounding error on the balance sheets of Verizon or SBC. Still, the company has managed to build a durable market niche through savvy marketing and customer service.

“We’re a small fish in the sea,” admits Larry Grant, Talk America’s vice president of enterprise computing. “We need to differentiate ourselves any way that we can.”

That means pushing customer service entirely onto the web and giving customers the ability to modify their accounts in real time. It also means giving sales teams the ability to slice customer data and monitor mailouts and other marketing with the same level of flexibility.

To balance both sides of this equation, Talk America’s network fuses Hewlett Packard Superdome hardware and Oracle 10g enterprise software at the datacenter. Just outside the datacenter, the company runs a constellation of Java applications running atop Linux servers. As both Grant and database systems director Vishal Anand readily admit, the thought of using Linux in such a mission critical environment would have been anathema as recently as four years ago. The maturity of the product just wasn’t there, Anand says.

Echoing Oracle’s Kumar and Trolltech’s Nord, Aand and Grant say internal attitudes towards Linux started to shift in the middle of the technology slump. By 2003, he says, Linux was starting to come across as a sensible option given the ongoing hassles that came with the company’s preexisting server system, Windows NT. Grant says Microsoft’s tight coupling of kernel and interface made it difficult to access the command-line and troubleshoot common problems. Similarly, the inability to treat operating system and the server as distinct entities, made it all but impossible to exit gracefully from server hangups. The company eventually switched to Apache. “Problem solved,” says Grant.

Such switches prompted reappraisal of Linux in general and Red Hat’s Advanced Server 2.1 in particular. Grant says he found the operating system lacking when it came to input/output scalability, especially when you factor Talk America’s 60 terabyte customer data warehouse. Still, running Linux servers in parallel provides more than enough computing horsepower to handle the sub-second transactions Talk America customers have grown to expect. It’s also frees up enough money to give Grant and the Talk America team the ability to tinker with novel approaches, such as cluster or grid computing at the data warehouse level to eke out still more hardware savings. Provided the next generation of Linux servers offer a way to get past the single SCSI and Fibre Channel bottlenecks that pushed the company toward HP’s ultra-redundant Superdome, Grant expects to see Linux in the data warehouse soon.

“If, instead of running a 32 processor machine, I can run four eight processor machines, it’ll buy us a lot of options,” he says. “We are seriously going to look at scaling towards the Linux architecture in 2006 when our HP contract ends with a possible purchase towards 2007.”

Sam Williams covers business and software technology for a number of publications. He is the author of two books, “Arguing A.I.” and “Free as in Freedom: Richard Stallman’s Free Software Crusade.”

Comments are closed.