Relax. It is just a catchy headline to draw your attention to some classic insights into the HPC cluster market.
True confession time. I stole that headline from Beowulf pioneer Thomas Sterling. As Thomas had originally intended, I am sure it sparked some reaction. Of course, sensational headlines generate pageviews, but they also help drive home important points. As a writer, there is no better reward than hungry readers devouring every word in search of some truth or insight. The only problem is it is not my headline nor my insight. Indeed, this headline servers to introduce a new feature here at Linux Magazine.
As some of you may know, at one point I was editor of ClusterWorld Magazine. One of my goals with the magazine was to get the movers and shakers of the growing HPC Cluster community/market to share their wisdom and experience with the readership. To accomplish this, I reserved the last page of every issue for what I called the Head Node column and asked an community/industry luminary to comment on an important topic. To my delight, the back page was a huge success and managed to pull some very fine words out out of some very bright people.
Recently, I was reviewing some of these columns and I thought, much of this “stuff” is still relevant and probably would be of interest to the current crop of Linux Magazine HPC readers (and to those ClusterWorld readers that were not paying attention back in the day). Then, in a flash of insight, I thought “Why not post the Head Node columns on the web as they were only published in the paper version!” Yes, I can still amaze myself with obvious conclusions.
Continuing with my category 1 brainstorm, I thought, “How about if I post the columns and provide some commentary on the opinions, ideas, and predictions shared by the writers”. Great idea, I told myself. It has been over 5 years since the columns were originally published and the community/market has changed quite a bit. The past columns will certainly make for good discussions and I can posting these every few weeks as part of my weekly column (i.e. when I cannot think of anything else to write about). It is, therefore, with great pride, I present the first installment in the new HPC Master Series: Beowulf in Chrysalis By Thomas Sterling.
Go ahead read it. I’ll wait here until your are done. Good stuff don’t you think? After I read it, I wonder if Thomas could have seen the eventual disruption cause by Beowulf. Back when he wrote the column (Fall 2003) clusters composed 42% of the Top500 List. Today’s Top500 List (Fall of 2008) shows clusters hitting the 82% mark. The entire HPC industry has been taken over by clusters. In the early days, anyone in the trenches could have told you that clusters would dominate, those on higher ground were not so sure. Of course the people in the trenches had their sleeves rolled up and were pushing at the edges of the technology. As Sterling states, we were witnessing the transition from Do-It-Yourself with Off-The-Shelf parts not intended for HPC to a thriving HPC cluster market complete with cluster specific hardware and software companies. Perhaps a more fitting metric is how HPC is fairing in the the current economy. The HPC market is expected to see modest declines, (5.4% in 2009) and rebound to a projected $11.7 Billion market with a healthy growth rate of 9.6% by 2012. (Courtesy of the recent IDC Predictions 2009 Webinar). Nothing dead or dying about that prediction.
In my opinion, the robustness of HPC due to the fact that it delivers results. Many companies now consider HPC essential to their competitiveness and when these benefits are coupled with a low cost barrier to entry, the decision to spend money becomes not one of convenience but of necessity.
Beowulf, or what I like to call the concept and practice of employing low-cost off-the-shelf commodity hardware to solve big (HPC) problems, has certainly emerged from the chrysalis. It has enabled a demand for HPC products and fueled the growth of what was once a somewhat staid market. And, perhaps most importantly it has put HPC in the hands of almost anyone. Alive and well seems more like it, but that is a rather boring headline.