Truth is stranger than fiction. The connection that helped end HPC careers and companies in 2009
At various times, I start to write science fiction stories in my head. You know that place where all the books I have written reside. In any case, I wanted to write a year in review piece now that 2009 is winding down. And, not to worry I’ll have more SC09 video in the new year. Rather than a bullet list of highlights (or lowlights) I thought I would spin a tail that, as far as I know, is accurate and a bit more interesting than the standard year-end fare.
They say the world is connected in ways we do not know. It has even been noted that our actions influence those outside of our local sphere — more than one degree as it were. Quantum entanglement not withstanding, I believe simple Newtonian forces have created a Twilight Zone ending to a most pivotal year. I invite you to sit back, grab a cup of something, and ponder a tale that needs to be told.
I suspect 2009 will go down in history as the payback year. The payback for all the irrational exuberance for building a false economy. Of course, business failures are due to many reasons. The speed at which technology can make millionaires overnight can also bring down organizations almost as fast. This past year, however, we have seen the demise of many companies, contributors to the HPC market and community, through no large fault of their own. The economy dried up cash and put most customers on a spending freeze for the first part of the year. It is hard to make a living when the economy is passed out on the floor.
Perhaps the most notable event was the sale of Sun to Oracle. Of course, many people will say Sun was bleeding for a while and that the commodity steam roller was slowly crushing their propriety products. All that may be true, and Sun was also a well entrenched company, notably in both the academic and Wall Street sectors. They were a very “open company” and contributed a large amount of open software to the HPC community. It seems, they were not “to big to fail” and were eaten by Oracle. There are those that think “Oracle will do the right thing,” but in reality the Sun that was is gone.
Not long after the Sun/Oracle announcement, another bleeding UNIX company, SGI, is purchased in a fire sale by Rackable. Again, the name lives, but the old SGI is gone. Another stalwart of the HPC world has given in to the economy and the commodity avalanche that has been rolling over the HPC landscape.
Later in May, we learn SiCortex has closed. This news was different. SiCortex was not some established Silicon Valley company that was trying to adapt to a new market. No, SiCortex was a New England company with a new idea. And, it was growing, but not fast enough for the nervous venture capitalists it seems.
Rumors surface and were confirmed in June that interconnect vendor Quadrics was going away as well. Quadrics always seemed to be the Ferrari of interconnects. One could argue they were a victim of InfiniBand and you would probably be right. And yet, I have to wonder, were the economy not so far down could they have survived?
And finally, just last week news of Verari closing down seemed to remind us that it is not over. Their absence at SC09 was telling and there are reports that they may re-emerge as a new re-organized company, which in my experience means a somewhat protracted good-bye.
Before I move on to the plot twist, I do want to mention that 2009 was not all bad news. It turns out that even with the casualties, the HPC sector faired rather well compared to other sectors. IDC seems to think so anyway. Those that lost their jobs may beg to differ.
One of the bright spots for HPC was, of course, the release of the Nehalem from Intel. It was finally a true quad-core that did away with the memory bottleneck of the past. Another highlight was that AMD and Intel finally settled their differences and AMD got a much needed cash infusion as a result. AMD kept their place at the head of the multi-core parade by introducing a six core processor this year. Another highlight was the announced Fermi processor by NVidia. Fermi engineers seemed to have the HPC wish list on hand when they designed this new processor. The coming year could be very interesting for the GP-GPU market. AMD/ATI and NVidia are the only two real players now that Cell for HPC and Larrabee are gone. By the way, we really can’t miss Larrabee because it was never here.
The final milestone for 2009 was the introduction of what I call Cluster 3.0 (a full article is forthcoming). That is the use of dynamic provision to allow application driven HPC. This technology is going to open up the HPC market because it changes the way applications are delivered to end users.
Amid the market demise, I often thought how ironic it was that the cause of the economic mayhem was do to large HPC clusters calculating the “risk” associated with various financial instruments called derivatives. The term derivative come from the fact they are derived from other financial instruments that are at some point supposed to have a connection to some thing real like a mortgage. The very companies that sold the hardware to Wall Street may have indirectly contributed to their own demise. Those thousands of servers sales may have helped the bottom line in the past, but this was the first glimpse the hungry snake got of its own tail.
The story, however, gets a bit more sinister. At one point I happened upon this post about the
Intractability of Financial Derivatives that points to a paper called Computational Complexity and Information Asymmetry in Financial Products by Princeton computer scientists Sanjeev Arora, Boaz Barak, Markus Brunnermeier, and Rong Ge. I recommend reading the article and browsing the paper because it may be the reason you or someone you know is now unemployed. For brevity’s sake, I’ll give you the upshot. It is computational intractable (i.e. there is not enough computing power in the world) to determine the risk of most derivatives sold today. If you can’t determining the risk you have no idea what you are selling or buying. Sounds dangerous. Like it could possibly lead to a world-wide economic disaster.
This result begs several questions. Just what are the Wall Street firms calculating with their mountains of servers? According to the paper, they are solving problem that cannot be solved. If the Wall Street quants did not know that, then maybe they should be in the unemployment line with all those who worked at the companies I mentioned above. But, if they knew the problem was not solvable and sold derivatives anyway only because they could find a buyer than that sounds a bit like a sinister movie villain. Massive data centers set about calculating garbage so you can say to your customer “trust me I have the blinking lights.” And, on the other side customers are buying things that are computationally impossible to understand. And where does that leave us mere humans. Was it just exuberant stupidity or grievous fraud or a little of both?
If I were to write a story about the high-tech irony of selling machines so advanced that they stupidly contribute to their own demise, I might just get my first novel published. But, that story, it seems, has already been told.
Douglas Eadline is the Senior HPC Editor for Linux Magazine.