Eating Your Own Tail: HPC in 2009

Truth is stranger than fiction. The connection that helped end HPC careers and companies in 2009

At various times, I start to write science fiction stories in my head. You know that place where all the books I have written reside. In any case, I wanted to write a year in review piece now that 2009 is winding down. And, not to worry I’ll have more SC09 video in the new year. Rather than a bullet list of highlights (or lowlights) I thought I would spin a tail that, as far as I know, is accurate and a bit more interesting than the standard year-end fare.

They say the world is connected in ways we do not know. It has even been noted that our actions influence those outside of our local sphere — more than one degree as it were. Quantum entanglement not withstanding, I believe simple Newtonian forces have created a Twilight Zone ending to a most pivotal year. I invite you to sit back, grab a cup of something, and ponder a tale that needs to be told.

I suspect 2009 will go down in history as the payback year. The payback for all the irrational exuberance for building a false economy. Of course, business failures are due to many reasons. The speed at which technology can make millionaires overnight can also bring down organizations almost as fast. This past year, however, we have seen the demise of many companies, contributors to the HPC market and community, through no large fault of their own. The economy dried up cash and put most customers on a spending freeze for the first part of the year. It is hard to make a living when the economy is passed out on the floor.

Perhaps the most notable event was the sale of Sun to Oracle. Of course, many people will say Sun was bleeding for a while and that the commodity steam roller was slowly crushing their propriety products. All that may be true, and Sun was also a well entrenched company, notably in both the academic and Wall Street sectors. They were a very “open company” and contributed a large amount of open software to the HPC community. It seems, they were not “to big to fail” and were eaten by Oracle. There are those that think “Oracle will do the right thing,” but in reality the Sun that was is gone.

Not long after the Sun/Oracle announcement, another bleeding UNIX company, SGI, is purchased in a fire sale by Rackable. Again, the name lives, but the old SGI is gone. Another stalwart of the HPC world has given in to the economy and the commodity avalanche that has been rolling over the HPC landscape.

Later in May, we learn SiCortex has closed. This news was different. SiCortex was not some established Silicon Valley company that was trying to adapt to a new market. No, SiCortex was a New England company with a new idea. And, it was growing, but not fast enough for the nervous venture capitalists it seems.

Rumors surface and were confirmed in June that interconnect vendor Quadrics was going away as well. Quadrics always seemed to be the Ferrari of interconnects. One could argue they were a victim of InfiniBand and you would probably be right. And yet, I have to wonder, were the economy not so far down could they have survived?

And finally, just last week news of Verari closing down seemed to remind us that it is not over. Their absence at SC09 was telling and there are reports that they may re-emerge as a new re-organized company, which in my experience means a somewhat protracted good-bye.

Before I move on to the plot twist, I do want to mention that 2009 was not all bad news. It turns out that even with the casualties, the HPC sector faired rather well compared to other sectors. IDC seems to think so anyway. Those that lost their jobs may beg to differ.

One of the bright spots for HPC was, of course, the release of the Nehalem from Intel. It was finally a true quad-core that did away with the memory bottleneck of the past. Another highlight was that AMD and Intel finally settled their differences and AMD got a much needed cash infusion as a result. AMD kept their place at the head of the multi-core parade by introducing a six core processor this year. Another highlight was the announced Fermi processor by NVidia. Fermi engineers seemed to have the HPC wish list on hand when they designed this new processor. The coming year could be very interesting for the GP-GPU market. AMD/ATI and NVidia are the only two real players now that Cell for HPC and Larrabee are gone. By the way, we really can’t miss Larrabee because it was never here.

The final milestone for 2009 was the introduction of what I call Cluster 3.0 (a full article is forthcoming). That is the use of dynamic provision to allow application driven HPC. This technology is going to open up the HPC market because it changes the way applications are delivered to end users.

Amid the market demise, I often thought how ironic it was that the cause of the economic mayhem was do to large HPC clusters calculating the “risk” associated with various financial instruments called derivatives. The term derivative come from the fact they are derived from other financial instruments that are at some point supposed to have a connection to some thing real like a mortgage. The very companies that sold the hardware to Wall Street may have indirectly contributed to their own demise. Those thousands of servers sales may have helped the bottom line in the past, but this was the first glimpse the hungry snake got of its own tail.

The story, however, gets a bit more sinister. At one point I happened upon this post about the
Intractability of Financial Derivatives that points to a paper called Computational Complexity and Information Asymmetry in Financial Products by Princeton computer scientists Sanjeev Arora, Boaz Barak, Markus Brunnermeier, and Rong Ge. I recommend reading the article and browsing the paper because it may be the reason you or someone you know is now unemployed. For brevity’s sake, I’ll give you the upshot. It is computational intractable (i.e. there is not enough computing power in the world) to determine the risk of most derivatives sold today. If you can’t determining the risk you have no idea what you are selling or buying. Sounds dangerous. Like it could possibly lead to a world-wide economic disaster.

This result begs several questions. Just what are the Wall Street firms calculating with their mountains of servers? According to the paper, they are solving problem that cannot be solved. If the Wall Street quants did not know that, then maybe they should be in the unemployment line with all those who worked at the companies I mentioned above. But, if they knew the problem was not solvable and sold derivatives anyway only because they could find a buyer than that sounds a bit like a sinister movie villain. Massive data centers set about calculating garbage so you can say to your customer “trust me I have the blinking lights.” And, on the other side customers are buying things that are computationally impossible to understand. And where does that leave us mere humans. Was it just exuberant stupidity or grievous fraud or a little of both?

If I were to write a story about the high-tech irony of selling machines so advanced that they stupidly contribute to their own demise, I might just get my first novel published. But, that story, it seems, has already been told.

Comments on "Eating Your Own Tail: HPC in 2009"


One might suggest that the derivative valuation task is computationally intractable, and that it is the crux of the failure in derivative markets.

I\’d propose an alternate interpretation: The true goal of all derivative valuation programs is to predict the valuation attached to the derivative by OTHER valuation programs. That is, the entire industry is running models of each other\’s models. Derivatives are the reductio ad absurdum of the notion that the \”worth\” of a thing is what someone else is willing to pay for it. But in the case of derivatives, it seems that there has always been a moment when somebody asked the dangerous question: \”Are the models really even close to reality?\” In the case of the latest collapse, the answer eventually was \”no.\”

This wasn\’t a matter of computer systems creating their own disaster. The hedge fund industry would have gotten to the same place if they\’d had to run the Black-Scholes model on a soroban. If you run the model predicting model for long enough, it will wander away from the underlying reality. If, at the same time, the consequences of absurd risks are made irrelevant to the behavior of the people operating the machine, while the upside is apparent, then the system will fail. Over and over again.

(Has everybody forgotten LTCM? The Savings and Loan
collapse? This isn\’t ancient history.)


RE \”high-tech irony of selling machines so advanced that they stupidly contribute to their own demise,…\”

The Dune saga, with Omnius and the Butlerian Jihad?

speaking out on the great issues of our times…..


I hate to sound like someone\’s High School English teacher, but I had trouble paying attention to the content of this article \”do to\” all of the grammatical errors. Is there an editor on staff? With all due respect to the letters after Mr. Eadline\’s name, writing is not his strong suit.


The Models ARE actually to predict the true valuations. The exact algorithms _are_ intractable. There are quite a few problems in the real world the computation of which are Intractable, generally when you are searching for the optimal in a search space lager than 3 dimensions. However, these problems are solved by one of a _huge_ family of Machine Learning algorithms that provide an approximation of the optimal result. For example, one family of algorithms \”prunes\” the search space in each direction beyond which point it is improbable for the optimal solution to occur; the resulting limited space in tractably searchable. Another family randomly samples the search space, which process, after some time, gives a clear indication of where the optimal solutions are likely to occur in the search space; and then you search these localized areas exhaustively. An example is the Monte Carlo simulations often used for Financial Models. They give a \”probably approximately correct\” result. For example, VaR is the amount which, with 99% probability, you are likely to lose on the particular financial instrument the next day. This corresponds to an AA rating. if you use the value with 95% probability, which will take less time to estimate, the organization gets a B rating.

These huge families of algorithms are tractable, but take a long time to compute. These are amoung some of the problems for which HPC servers are used in Finance. The problems with the credit crunch is not a Computational or Computability problem. It is a problem in the underlying Maths and Statistical Models; the underlying assumptions of the models are inaccurate; those Models that are accurate predict values under \”normal\” conditions well, and they do not predict correctly under once in a lifetime \”extreme\” conditions.

So the Credit Crunch has nothing to do with HPC Computing.

Currently it sounds like WordPress is the top blogging platform available right now. (from what I’ve read) Is that what you’re using on your blog?

R66YMu uppxndcramqe, [url=http://jwcqfvocdrry.com/]jwcqfvocdrry[/url], [link=http://wjhgncgrynbe.com/]wjhgncgrynbe[/link], http://wynopbpmsltk.com/

Below you?ll come across the link to some web sites that we believe you need to visit.

Leave a Reply