How Not to Build a Linux PDA

Last month, I touched a little bit on HP's screwed up Linux PDA initiative, but perhaps I was a bit too harsh. Sure, they have a research arm that's completely underutilized and they have absolutely no clue as how to turn those efforts into a product, but HP is in no way unique in their absence from the PDA and Linux device cluetrain. For the most part, the entire industry needs a swift kick in the head to see how to build and market a successful Linux handheld and to learn how to properly support open source PDA developers. I learned how the hard way, and here's my painful perspective on the whole shebang.

Last month, I touched a little bit on HP’s screwed up Linux PDA initiative, but perhaps I was a bit too harsh. Sure, they have a research arm that’s completely underutilized and they have absolutely no clue as how to turn those efforts into a product, but HP is in no way unique in their absence from the PDA and Linux device cluetrain. For the most part, the entire industry needs a swift kick in the head to see how to build and market a successful Linux handheld and to learn how to properly support open source PDA developers. I learned how the hard way, and here’s my painful perspective on the whole shebang.

A Painful Lesson

From December of 2002 through February 2003, I was Software Developer Liaison for Sharp’s Electronics’ Zaurus, probably the first handheld Linux device to enjoy any commercial success at all (although its success has been extremely limited, and at the end of the day, only the hardcore Linux faithful have remained loyal to the product.)

While I can’t go into detail about how many units were sold in the United States, it was considerably lower than most public estimates. Within the first six months of the product’s launch, it was banished from superstore shelves, the product was re-purposed as an overpriced enterprise handheld for vertical market applications, and then was only for sale by a handful of resellers and small system integrators. Currently, the only way to get a new Zaurus is to import one directly from Japan through a handful of “grey market” resellers for about $600-$800. Today, Sharp is still trying to find to a way to yet again re-purpose the machine and generate interest.

Why did the Zaurus do so poorly? After all, the device received accolades from the Linux developer community for its great features and usability as a “real” Linux machine. In fact, during the early beta testing of the product, I reviewed the device for Linux Magazine (prior to my engagement at Sharp) and reported that it had enormous potential. So what happened?

Build it and They Will Come… NOT!

Part of the failure stemmed from the “If we build it, they will come” mentality that’s pervaded most Linux-based devices that have been marketed to date. Let’s face it: there have been several attempts at building a Linux device for the masses prior to the Zaurus, and every single one of them failed.There was the Agenda, with its woefully under-powered chip, and the Yopy with its “kwazy” [sic] keyboard and non-existent backlighting.

Both products came from small companies with extremely limited resources and capital, so at the time of the Zaurus’ introduction in 2002, it seemed that Sharp, a multi-billion dollar Japanese electronics giant, would have the moxie and the yen (in both senses of the word) to make their product a success.

But having cash and building a Linux device for the sake of Linux appeal alone are rarely good enough. Ultimately, many factors contributed to the Zaurus’ poor market performance. The crappy economy at the time of launch is partly at fault, but Sharp’s failure to work effectively with the Linux community — developers and end-users alike — sealed the fate of the machine. Sharp’s misperceptions greatly affected the quality of the Zaurus’ software stack, which impacted the quality and quantity of commercial and open source applications released, which in turn resulted in lackluster sales.

Keeping Your Developers Happy

To its credit, Sharp was able to successfully seed its developer base by providing a cheap developer version of its PDA to more than five thousand capable software developers in both the community and commerce who genuinely could build applications for the device. As proof, over five hundred significant open source applications and ports were released for the device within the first 6 months.

However, after the honeymoon, when the novelty of the device wore off, interest dropped considerably. Much of this was due to the fact that Sharp Japan’s support for its North American subsidiary’s developers was lackluster — it took over eight months for Sharp to put up its SourceForge-based Developer Zone (much of this due to a Byzantine accounting and approval process in Japan). And even when that site eventually did come online, the developers didn’t get access to any of the source code of the Qtopia GUI and Sharp-modified applications in the Zaurus ROM. Hence, the capabilities of those five thousand developers to debug and improve code were left completely untapped.

Updated ROM releases took months, and Sharp issued no periodic beta test releases of the ROMs to its developers. The Zaurus kernel source sometimes took months to get posted every time there was a ROM re-release, which frustrated developers working on device drivers and community-supported distributions like OpenZaurus.

To make matters worse, the internal workings of the Zaurus hardware was never completely exposed to the public. Only a handful of qualified developers, all under non-disclosure, were allowed copies of the service manual, which specified the actual JTAG pinouts. Even with this guarded information, the custom chips in the device were never completely documented and it made it nearly impossible for the community to develop open source versions of its own ROMs and drivers.

The reasoning made to my support staff by senior management was that if the complete hardware specifications of the machine were made public, nothing would stop a Korean or Taiwanese manufacturer from releasing a competing product for half the price. This completely closed mindset blew me and everyone else who heard this rationale away. Say what? Weren’t half the PDAs on the market designed by the Taiwanese anyway? If they wanted to release a competing product, couldn’t they just call up Lineo and Trolltech and do the same thing?

Many of the modules and drivers on the Zaurus had system integration done by Lineo and used third-party, licensed materials, and thus they could never be released. Case in point: the Lineo-designed SD driver interface on the Zaurus was completely closed. To get it to work on community-based ROMs, Sharp had to be convinced to let the community redistribute the SD binary kernel module. Note that this last issue wasn’t really Sharp’s fault — they and Lineo were bound to an evil licensing agreement by the SD Card Organization that would make them subject to legal action if they ever exposed the SD interfaces. (A word to prospective Linux handheld and convergence device manufacturers: avoid SD like the plague.)

As if the SD proprietary headaches weren’t enough, the Zaurus also used a proprietary connector sync module made by Japanese cable and parts vendor Hosiden that could only be ordered in lots of 10,000 at thousands of dollars a pop from its sole distributor in the US, making it exceeding difficult or impossible for third parties to design serial connector peripherals and USB cables for the device.

Leaving the SDK to Your Partners is a Bad Idea

On top of the internal developer community woes, there were external issues that affected the developers as well. Sharp had chosen the Trolltech/Lineo OpenPDA software stack as the basis of its platform, which was extremely problematic, as you couldn’t legally develop and sell commercial applications with it using the open source version of the developer kit. It was dual-licensed with the QPL and GPL.

Initially, if you wanted to sell commercial, non-open source applications, you had to pay several thousand dollars for Trolltech’s enterprise license or build your applications in PersonalJava 1.2 using the buggy and slow Insignia JVM on the machine. Only after two bad quarters of Zaurus sales did a $250 commercial Qtopia SDK become available from Trolltech, but this was too little and too late. Small software shops wanted a free solution for building commercial Zaurus applications. The general sentiment by the developer base on our mailing lists was that Palm and Microsoft’s PDA SDK’s were free, so why spend the extra money on the Zaurus?

It was no coincidence that most of the Zaurus’s applications were simple ports of existing open source applications ported by end users. Commercial developers ignored in the machine. Only one developer, TheKompany.com, emerged as a major vendor of commercial applications using the Qt embedded stack. The other major Zaurus commercial developer, Eon Games, used open source libraries and tools (such as SDL, which is LGPL licensed) to produce its software, which ran completely external of Qtopia.

As if SDK pricing issues weren’t enough to enrage developers, because both PIM sync engine options on the Zaurus for Windows and Linux were completely closed source (Qtopia Desktop from Trolltech for Linux and Windows, and Intellisync for Outlook users), developers couldn’t submit fixes or improvements to Sharp or Trolltech when the Zaurus was hampered early on with terrible sync problems with Outlook and Windows, ensuring complete commercial disaster in retail sales and initial product reviews.

Listen to Ben. He’s Smart.

My former developer support engineer at Sharp (now Mac OS X Qt and KDE super-hacker and Transformers collector extraordinaire) Benjamin Meyer has written a great “laundry list” of features for manufacturers looking to build Linux PDAs (http://www.csh.rit.edu/~benjamin/articles/linux_pda.php), as well as an inspired short essay-cum-bedtime story called “Bad Apples” (http://www.csh.rit.edu/~benjamin/articles/bad_apples.php) about how manufacturers should effectively manage and utilize open source and in-house developers to their best advantage. Both of these should be required reading for anyone looking to build a competitive and compelling device for the cut-throat and demanding consumer PDA market.

Ben’s most salient point is that developers should be given every opportunity possible to work on the software in your device or product. So wannabe Linux PDA manufacturers should listen to Benjamin and listen good:

“Developers want to work on Open Source software, your Open Source software! There is no excuse for not giving them every opportunity to do that. Providing timely updates, nightly builds, access to the bug tracking system, and mailing lists are all very easy things that can be provided in most cases. There are, of course, many more things, but those four are a very good start. If a project doesn’t even have those, then there are problems that this article can’t cover. … Helping developers work on your code is always a good thing.

The moral of the Sharp Zaurus story: Keep it Open, and Give the Developers What They Want.

Jason Perlow can be reached at perlow@linux-mag.com.

Comments are closed.