Building clusters and high-performance computing (HPC) systems is like building with Legos: oftentimes, your project screams for that perfect block you wish existed, but calling the Lego company and asking for eight custom pieces to finish your dream car, castle, or spaceship is just too expensive and time consuming.
Generally speaking, the same is true for clusters: the more custom the design, the more expensive the system, and the longer it takes to hit the market. Instead, manufacturers and researchers make do with off-the-shelf components.
So what “Cluster Lego” do we have now and what “blocks” will be available in the near future? Today’s basic building blocks are Intel and AMD servers and high-speed networks. Tomorrow’s toys? Let’s take a look.
Playing With Processors
Cluster Lego fall into a number of different categories, including processors, interconnects, and motherboards (and packaging and storage, but let’s ignore the latter two this month).
The first component chosen in a cluster is the processor. Both Intel and AMD CPUs are obvious candidates, but there are other possibilities. For example, even though Apple intends to switch to Intel processors, Linux on IBM’s POWER is still a viable option. And there are others.
Why would you choose anything other than the latest Opteron or Xeon? Because — just like the standard Lego piece that isn’t perfectly suitable — your requirements may demand a better solution.
For instance, if power consumption is a concern, you may want to look at the recent announcement about the VIA C7. The C7 supports the x86 instruction set, including SSE, SSE2, and SSE3, and features a full speed FPU, a hardware random number generator (RNG), 128 KB full-speed exclusive L2 cache with 32-way associativity, and a 2 GHz clock. While the specifications may sound rather weak compared to the big boys, consider that all this performance is achieved by consuming only 20 watts. Compare this to the 100+ watts required for more traditional solutions.
Five or six C7 processors consume as much power as a single Xeon or Opteron. Lower power consumption means lower electrical costs, lower cooling costs, and better reliability. In short, it is more cost effective. If your application shows good scalability and (or) if you need good random numbers (Monte Carlo methods), then the C7 may be something to consider.
Another alternate is the Cell processor from IBM. The Cell is a new, “dual use” processor that IBM, Sony, and Toshiba have developed. The Cell consists of a 64-bit POWER core and eight specialized co-processors. Using the co-processors, IBM reports that the Cell can deliver 256 single-precision gigalops from a single chip (and 26 gigaflops at double-precision). While the Cell’s main use will be in the video game market, that virtually guarantees large scale production of the new chip — a new and interesting Lego piece with which to build clusters.
Of course, dual core, 64-bit chips are now available, too. Chances are that the reign of single core, 32-bit processors in HPC is soon to end.
As any cluster veteran will tell you, the interconnect makes the cluster. But which interconnect? That depends on your needs.
Among commodity off-the-shelf interconnects, Ethernet rules. However, base prices for Gigabit Ethernet (GigE) network interface cards (NICs) and switches have fallen quite low, making GigE a superior choice if you can afford a little more gear. The threebest high performance Ethernet chipset families are from Intel, Broadcom, and Syskonnect, and all of these solutions have vendor- supported Linux drivers. (Remember, though, that performance of Ethernet chipsets may vary significantly based on application, implementation, and cost.
Many GigE cards come in two varieties: a 32-bit PCI card for clients and a 64-bit PCI card for servers. Don’t automatically assume you need the throughput and expense of the 64-bit cards.
If you need more throughput and lower latency than what Ethernet or GigE offers, prepare to pay more for your Lego pieces. There are plenty of choices, so carefully analyze your needs and count your pennies. There’s Infiniband (available from many vendors), Myrinet from Myricom, QsNet from Quadrics, InfiniPath from PathScale, and SCI from Dolphinics.
Again, what choice is right for you depends upon your application. High-performance interconnects can unleash the full potential of your processors for some applications.
So far, the choices have been important, but the selection hasn’t been very large. When considering motherboards, get ready for the 1,000-piece Lego expansion pack.
Motherboards are available from a a large number of companies. Many share the same chipset, which ultimately defines the features of the motherboard. Many motherboard vendors also add their own “extras” that sometimes help and sometimes hinder a cluster design.
For instance, if you need to move large amounts of data between nodes, pay close attention to the motherboard’s PCI implementation. There can be great differences between motherboards and you should measure the throughout of each motherboard you’re considering. After all, there’s no sense buying into an expensive interconnect only to find that the motherboard does not handle PCI traffic very well.
Another factor worth considering is the number of processors supported by each motherboard. The current trend favors dual-processor motherboards, but the advent of dual-core processors may change some of this thinking. Using dual core processors, quad-processor nodes (two dual cores processors) are possible, but not necessarily the most efficient. Again, it all depends on needs.
Motherboards can be quite large (say, for quad Opteron) or quite small (the VIA Mini-ITX). Like network adapters, motherboards target two main audiences, servers and desktops. Server boards are usually more expensive and are designed for small rackmount cases; on the other hand, desktop motherboards are designed for workstation users. However, don’t assume that a desktop motherboard is not suited for cluster use.
Some newer designs include PCI-Express connections for new video cards, but the motherboard doesn’t care if we use a PCI-Express Gigabit Ethernet adapter instead. This option nicely circumvents the 32-bit PCI bus found on most desktop motherboards.
SUBHEAD Putting it all together
The commodity market has provided plenty of interesting Lego pieces with which to build clusters. The choices may not be exactly what you need, but, at the moment, it’s what’s available in your toy chest.
Fortunately, with a little ingenuity and the use of open and flexible software, these pieces can usually provide a very workable solution. That’s a very good thing, because unlike Lego blocks, throwing a cluster against the wall because we cannot find just the right piece just doesn’t seem to have the same effect.
And remember: When evaluating hardware and software, the standard advice still applies: Benchmark your applications and design your cluster to suit your needs.
Douglas Eadline is the Senior HPC Editor for Linux Magazine.