Digging Into The Top500

Who cares about the FLOPS. The real story is about who and how.

The Top500 is the favorite punching bag of many people in HPC. My beef is not with the Top500, it is with all those who make it out to be something it is not. In my opinion, it is good historical record of those machines that can run a single benchmark. There is of course certain bragging rights for landing on the Top500 and it may help people justify an expensive pile of hardware. I suppose getting your system listed on a web page is good thing™, but there are those who have real work to get done and can brag about minor things like application throughput, optimization, and utilization.

I have followed the list for the past several years. I think I can sum it as follows, more nodes, more cores, more InfiniBand, Blue Gene, bigger HPL number. Yawn. This year, however, I find the changes in the Top500 quite interesting. I’m not going to focus on top performance, but rather some trends in the list that seem to be changing.

There is a new champ. In and of itself this is often not very interesting, however, this time it was different in several ways. First, it was from China. The Tianhe-1A system at the National Supercomputer Center in Tianjin, achieved a performance level of 2.57 PFLOPS (i.e. it crunched more numbers than anyone else). China has been marching up in the Top500 ranking. This fall they had 41 total systems in the list. That is rather impressive considering two years earlier they had just 14 systems that made the cut. From what I hear, this is just the beginning.

The other notable difference is how they got to the top. Tianhe-1A is the first number one machine to use GPUs (NVidia). It also used a custom interconnect. Moreover, three of the the top systems used NVidia GPUs while a total of 28 systems on the list used GPU technology (NVidia, AMD, Cell). One other note about GPUs. This years Green500 had five GPU based systems (4 NVidia, 1 AMD). The Green500 ranks systems in MFLOPS/Watt.

I believe we will see more and more systems using GPUs — at least to get on the list. The GPU trend took a firm hold in 2010 and I expect it to continue in many verticals where “array processors” make sense. As I noted previously, I predict the GPU will migrate into the processor just like the co-processors of the past. GPUs for HPC computing are here to stay.

Moving on to other trends. Quad-core processors were used in 73% (365) of the systems, while 19% (95 systems) are already using processors with six or more cores. It is probably safe to say that the typical HPC node has at least 8 cores (2P motherboard with quad processors) and new systems will have at least twelve cores. Intel dominates the high-end processor market, with 79.6% (398) of all systems using Intel processors, although this is slightly down from six months ago (406 systems, 81.2%). AMD Opterons found their way into 57 systems (11.4%), up from 47 on the June list. AMD notes that 24 of the top 50 systems used their processors. IBM Power processor use is slowly declining and is now at 40 systems (8.0%), down from 42 previously.

Gigabit Ethernet (GigE) is still the most-used system interconnect technology (227 systems, down from 244 systems), due to its widespread use by industrial customers. The fact that GigE is “free” on the motherboard probably has something to do with this number. There were only seven systems sporting 10-GigE. As 10-GigE costs come down, it will be interesting to see if it gains in the list against InfiniBand (IB). Speaking of IB, 214 systems used InfiniBand on this list, up from 205 systems in the previous list. Interestingly, InfiniBand-based systems account for two and a half times as much performance (20.4 Pflop/s) than Gigabit Ethernet ones (8.7 Pflop/s).

In terms of operating systems, Linux continues to dominate. By my count, it was used on 449 systems. That is close to 90% of the list and probably the most dominate trend. I don’t expect this change in the near future. Windows HPC server 2008 was reported on five systems. Most notably, the Magic Cube Cluster at the Shanghai Supercomputer Center in China runs Windows. That system uses at least 2K nodes (possibly using using 4P motherboards). I think that is the largest Windows cluster I have seen to date. I’m not sure what advantage Windows brings to the party. Once you see a disk-less node boot in 20 seconds, you get the power of “open plumbing.”

In terms of application area, the leading sectors were “Not Specified” (34%) Research (16.4%), Finance (8.6%), Information Service (7.0%), and Geophysics (3.8%). Remember not all systems are on the Top500 and many are not specified in any case. There were even four clusters in the WWW category.

Finally, a question for the audience. There were 291 systems that had between 4K and 8K processors. That is a lot of processors. I am always curious how many of those are used at the same time by a single job. Surveys indicate that many applications don’t scale to more than 32 cores. Even the “big codes” may not exceed a thousand processors. The heroic codes can use thousands, but these applications are not very common. There were 9 systems with over 128K processors.

The question I always like to ask is; “Who uses 128K processors at once?” I usually don’t see that many hands go up in the air for that question. “How about 1000, or 100, or 50?” More hands start to go up. “Thank you. And now can you explain to me why you care about running Linpack on 128K processors?”

I never hear a good answer to that question.

Comments on "Digging Into The Top500"


here is an answer your question about who uses a large number of processors for one job:

Stochastic methods are very well adapted to using thousands of processors since an unlimited number tasks can be run independently and asynchronously. In addition, fault-tolerance is easy to achieve: if one task doesn’t transmit its output it affects only the error bar on the total result and not the average. These methods are quite frequent (namely in physics, chemistry, finance…) and will probably spread even more in the future due to its massively parallelizable property.

For example, this page shows an application of Quantum Monte Carlo
on the EGEE grid : http://qmcchem.ups-tlse.fr/index.php?title=Large-scale_quantum_Monte_Carlo_electronic_structure_calculations_on_the_EGEE_grid
In addition to grids, this code runs routinely on 512 cores (for a single calculation) with a nearly perfect speedup. It could run on much more if we could have access to more cores!



Nice summary about on the direction HPC clusters are taking.



I think Doug is referring to a single application running on 128,000 cores using MPI.



Boinc uses more then 128K cores. Their cluster just has some latency issues. :)


Im thankful for the article post.Thanks Again. Really Cool.


This is one awesome article.Really looking forward to read more. Really Cool.


My partner and I absolutely love your blog and find a lot of your post’s to be just what I’m looking for. Does one offer guest writers to write content for you personally? I wouldn’t mind writing a post or elaborating on a lot of the subjects you write regarding here. Again, awesome site!


The next time I read a blog, I hope that it doesnt disappoint me as a lot as this one. I imply, I know it was my choice to learn, however I really thought youd have something attention-grabbing to say. All I hear is a bunch of whining about one thing that you might fix for those who werent too busy on the lookout for attention.


The truth just shines thogruh your post


Leave a Reply

Your email address will not be published. Required fields are marked *


You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>