The Top500 list is a valuable measure of HPC progress, but the race it has spawned maybe over for many organizations
In the United States, applying to college usually means taking the SAT test (Standard Aptitude Test). The goal of course, other than taking a bunch of money from every kid that wants to go to college, is to provide a common number to compare students from around the country. Presumably it is a predictor of how well a student will do in their first-year of college. Not everyone agrees, however.
Trying to reduce a person’s ability to one number is convenient and at the same time naive. If it were that simple, why ask any other questions. What if your college or job application asked for your name, SAT score, and IQ. A simple binary decision. “But, but…”, you plead, “there is more to it than that.” Of course there is because once you are in college the SAT score is useless, it does not help you further your education in any way. How well you perform in college and beyond is up to you. The same can be said for your IQ test (if you have ever had one).
Many people tend to agree with my analysis and at the same time are enamored with the Top500 List. As I have said before, “the list” is very useful and provides a good historical record and “snapshot” or the HPC community. I believe problems arise when the list is used for that which it was not intended. Quite often you will read “the X fastest computer in the world”, where X is a spot on the list. To which I always add, “running a single benchmark that has little bearing on many HPC applications. Results are voluntarily submitted and many machines are not on the list.” Such, arguments usually go unnoticed when writing the press release about the “new University supercomputer that is the X fastest computer in the world.” Which is why I am glad to hear that chasing the Top500 is maybe less important than has been in the past. A recent article in the Chronicle of Higher Education seems to support this notion.
In my opinion, this change is probably coming from several directions. First and flippant, the analyst’s expect the Chinese to own the list in very short order. Of course, there may be some competition for the top spot, but buy and large, the Chinese are dumping a lot of money into HPC. Why focus on a race you are going to loose.
Second and probably more to the point, results help funding for future grants while a good Top500 position does not. In a sense the “Top500 bubble” may be bursting. I consider this a good thing because there are plenty of other HPC issues that need addressing. For example, taking my SAT analogy even further, does a good Top500 placement predict how well the cluster will perform in the first year of service? Certainly if all the users are running linear algebra codes, but how well does the Top500 reflect other applications. What about utilization. Does a good list rank indicate users will get enough work done on the cluster? Lastly, current budget pressures or “doing more with less,” certainly cannot be discounted as a reason to not even consider getting your machine ranked.
To be fair, there are other methods to measure HPC performance. The HPC Challenge (HPCC) is a set of seven tests (the Top500 benchmark included) that give a much better picture of the “overall” machine performance. The HPCC tests are similar to a report card rather than a single test result. Of course, the old HPC adage that “your applications are the best benchmark” is still true and any sizable cluster purchase should include this type of testing.
One growing HPC sector is high data throughput machines. These machines are designed to allow processing of large data sets and therefore depend on high I/O rates as much as number crunching capability. Indeed, some data heavy problems would bring your typical Top500 beast to crawl. The Top500 is of little help assessing these machines.
In closing, Top500 performance is important, as is the HPCC, and the Green500 for that matter. Measuring complex systems from many angles helps us build better devices. My wish is that some of the effort and energy that goes into trying to build a faster HPC sports car goes into allowing more people to travel on the highway.
Douglas Eadline is the Senior HPC Editor for Linux Magazine.
Comments on "Chasing The Number"
Top500 is like Formula 1 – it may not have much real bearing on the cars we drive to work, but it pushes some of the boundaries of the technology, and it provides much media exposure. Media, and thus politicians and taxpayers are generally more likely to see and be impressed by a good Top500 result than by a dozen scientific papers resulting from the same machine’s “normal” work. That in turn helps in the race to get money allocated to science generally.
By the way, is it too much to ask a “Senior Editor” not to make 3 elementary language mistakes in one short paragraph? The plural of “analyst” is “analysts”, no apostrophe; the phrase is “by and large”, no “u”, and “lose” has one “o”.
It’s not too much to ask. I enjoy the content of these pieces but the numerous spelling and grammar errors, which are in every piece, detract from that comment. Use a copy editor!!
I’m glad I’m not the only one who noticed – I was going to comment myself.
Errors aside, I think this is an issue that plagues a lot of industries, not just HPC – the fact that benchmarks that a lot of people rely on use a highly limited number of metrics to determine rank. Sure, if a computer can run at 10 TFLOPS, that’s super – what if the I/O isn’t up to the task and it craters at 1.5? Millions of dollars down the drain because you didn’t look at the big picture. A 1000HP car that goes like a fish is no good if it steers like a cow. A hybrid car can shoot itself in the foot if it chooses a battery that causes as much pollution in its production as a 1970 Dodge V8. A 1000″ Plasma that maxes out at 1280×768 – what’s the point?
I know I’m going for some pretty extreme examples here, but the point is that if you only look at a limited number of metrics and don’t look at the big picture, it can really bite you in the backside when push comes to shove.
“The number” is merely a coarse indicator. What it means (and what it doesn’t) and how it is (mis)interpreted by folks is the problem – especially to those testosterone driven, overly competitive types.
As far as the linguistic mistakes, they are part of the long history of language. For example, “by and large” is a nautical term for sailing with and against the wind. It came from: “Thus you see the ship handled in fair weather and foul, by and learge.” – which has it’s own issues.
Read more: http://wiki.answers.com/Q/What_is_the_origin_of_the_phrase_By_and_large#ixzz1KXcXzv00
The article is fine, but the writing standards seem to be dropping. I agree with xembly above that reading poorly crafted prose detracts from the merit of the article. The word ‘maybe’ in the opening paragraph has a different meaning from the phrase ‘may be’ and is not interchangeable with it.
Grammatical mistakes of this nature are not ‘part of the long history of language’, as asserted by kalloyd, who compounds the matter by introducing a grammatical error of his own: using “it’s” when he clearly meant “its”.
I have never written anything novel, interesting or useful. However my spelling, grammar and punctuation are usually perfect. My colleague, on the other hand, cannot write a single sentence correctly in any language. His ideas, now patented, earn his employer over one million pounds a year in license fees, which is why he is the professor and I am not.
I have the first Linux magazine volume 1 number 1 would anyone be interested in buying it?