dcsimg

The Case For A Case

The case for a personal cluster case. Can HPC growth be enhanced with only a few pieces of bent metal?

People that know me have learned that I tend to fret over some really strange things. My multi-core angst has been well documented. Another worry comes from a hobby of mine — low cost clustering. This indulgence does not cause me much distress as I like optimizing and engineering small systems. There is, however, something I wish for when I toss and turn at night. In building small low cost clusters, my goal is to measure how much real performance one can you get out of $2500 worth of commodity computing parts. And, by real performance I mean running the Top500 HPL program. I know HPL sucks, but is does have that long historical archive which helps put the performance of my little clusters that could into perspective.

I have not built a new system recently (hint to all you component vendors that want HPC glory) but my current 8 core machine can hit 53.4 HPL GFLOPS. The machine has four nodes in somewhat small, but easily stackable cases. I use inexpensive Micro-ATX motherboards with one Core2 Duo E6550 running at 2.3 GHz and 2 GB of RAM (The motherboards can take Core 2 quads and hold up to up to 8 GB of RAM.) Although the motherboard has an board GigE, I added Intel PCIe GigE desktop networked cards to each node for performance reasons. My design also used two 5-port GigE switches. I use the on-board GigE network for administration and NFS while compute traffic goes over the Intel GigE network. The stitches support Jumbo Packets which can help with performance in some cases. You can find pictures and a detailed hardware information here.

As an aside, my current performance translates to approximately $47 per GFLOPS. I have not done the pricing, but I’ll bet a quad-core version of my current machine might be possible with the recent price drops in desktop processors. Using quads would give me 16 cores next to my desk. There is even the possibility that I could have over 100 HPL GFLOPS at my disposal for about $2500. I do have concerns about whether GigE will be able to service four cores, but that is an engineering issue. Fortunately, Open MX just hit the 1.0 release.

Whenever I think about personal clustering, I think about a survey I did about five years ago. The survey simply asked what was the scalability of your application. The thing I remember most was the fact that 50% of the respondents could not scale over 64 processors. I did not have detailed data on why the application could not scale further, though I assume it ultimately come down to Amdahl’s Law. Aside from Amdahl’s law, I would assume that the only other thing hindering scalability is network performance, but let’s just assume that my original survey is within the ballpark.

If 64 or less cores is scalability threshold for most people, the desk side cluster may be a very interesting proposition. Power and cooling issues not withstanding, it is very possible to build a four node wire shelf cluster that could put 32 cores literally within arms reach (using dual socket motherboards and quad core processors). No need to wait in the queue for your programs to run. Astute readers may also note that a single 8 core workstation may work for many users as well. If it works for you, why not. In addition, a small system would make a great educational tool.

While calculating the local weather forecast under your desk, there is one other important use for your desk-side cluster — software development. In this case, both systems and application software could be developed without the need to carve out a piece of a working cluster. I am a firm believer that software gets written for the hardware that sits in front of people. If enough people have 16 or 32 cores next to their desk, then there is a good chance someone is going to try something new and different. That killer app is out there. And, finally a desk-side cluster has what we don’t like to talk about, but need just the same — reset switches.

At this point you have to be asking What is the problem Eadline? You seem to have all the cheapo FLOPS you need. Sigh. Indeed I do. But, what I don’t have is a really cool case for my cluster. I am still limited to wire shelves, cheap cases, plug strips, and cable ties. The end result is something that is bigger than it needs to be. I would prefer to have a personal cluster under my desk, but the contraption I have now is just not practical.

There are now gamer cases, server cases, desk-top cases, desk-side cases, mini-towers, mid-towers, full-towers, and the list goes on. In addition, there are those who modify cases in the some truly bazaar ways. What I have not found is a case that holds multiple ATX motherboards. Of course, there are some other issues like power supplies, wire management, switches, and cooling, but none of these are insurmountable problems. I would assume a cluster case would cost double or triple what a good single motherboard case would cost because the volumes would be low. A high price case would certainly skew my dollars per GFLOP ratio, but such is the cost for convenience.

I realize there are desk-side systems that pack server boards into a small area, but these usually have rather hefty price tag. Remember, I’m talking about the value based cluster (i.e. commodity cheap clusters). In my estimation there would be a market for such a cluster and case. Whether it would be used for education, software development, or real HPC computing a small desk-side cluster in a single case would have some real utility.

When discussing HPC the one topic that often comes up are the “hold backs” that limit HPC usage. Maybe all it takes is couple pieces steel bent the right way to launch HPC into the mainstream. What would you do with 16 cores under your desk. These are the kind of things I lose sleep over.

Comments on "The Case For A Case"

jhearns

Doug,
a company called Workstations UK (in the UK, naturally),
did preoduce cases just like that, and sold many Beowulf clusters built like this.
They’re still in business as far as I know, but no longer building clusters like this.

As you say, some sort of case which racks up motherboards would be cool. Off the top of my head, hobby electronics types used Veroboard cases where PCBs simply slotted into plastic guides on the side of the case. I don’t know if any of these are big enough to contain ATX motherboards.

Reply
bruno.santos

Well, here’s the thing: if you’re going to go DIY, you’ve got to go all the way! Think outside of the box… necessity is commonly known as the mother of inventions.
Ergo, if you want a DIY cluster, you’ve gotta make the case or do an overhaul to something else big, like this practical example!! And in this case, it uses multiple power supplies… but you can DIY the main power supply cables, as explained here.

But, if you don’t want to do it yourself, there is only two things that will drive the Multi-(mini-)ATX cases and gear into the mainstream: Games and Gamers!

But going back to DIY, at least with computers, you don’t need government approval to make your very own computer… when the same doesn’t apply to some other bigger things…

Reply
alvaroafernandez

There was a (now-defunct) company that sold a cluster made from Crusoe processors. Saw them in SC 06, I think. Wish I could remember their name. They had custom cases for their clusters. I think their motherboards were custom too. Maybe the case maker is still around, or kept the specs…?

Reply
dwilde1

Doug -

Your desire to do this on the cheap is admirable. I just bought a Dell Inspiron mini-tower with a 4-core and installed a second drive with FreeBSD 7 on it. Virtually linear speedup on my web and MySQL apps. 4 cores == (0.95%) 4x one 2.4G core. The Intel 4c is really 2x 2c, so it’s not quite clean, but damn close in the real world. The whole box with video and drives and 3G of DDR was $1000 incl display and UPS. I’m sure I could make a mini-cluster as you have and have a nice video wall to boot, not to mention a terabyte of drives.

You don’t really address why cheap is necessary. You get to play all day and toss and turn all night, but out here we live by value per GFLOP. If I put that much horsepower together, I better be making money from it. Yes, there were some pretty toys like yours at the Maker Faire in Austin this past week, but geeks with toys are just geeks with toys.

A Dell M1000e half full of blades will cost $50G – $100G fully decked out, but any real app for that much horsepower should pay back that much and more in less than two years, not to mention salary for a geek or three. It’ll still be running in five years, unlike your little homebrew. If you’re one of the geeks your app pays for, then you get to play for real in the big app called “business”.

Reply
jhearns

dwilde mentions a video wall.
Hmmmmm… I wonder if Chromium would work using open-mx?

Reply
samhdaniel

Doug,

I assume that you have seen this by now, but just in case:

http://helmer.sfe.se/

and

http://obscuredclarity.blogspot.com/2008/09/24-core-linux-cluster-in-2999-case-from.html

A google search should turn up two or three other variations on this theme. These may not be what you have in mind, but they are cost-effective and very workable.

Reply

I loved your post. Keep writing.

Reply

Great post. I was checking continuously this weblog and I am inspired! Very helpful info specifically the final part :) I deal with such info a lot. I was looking for this certain information for a long time. Thank you and good luck.

Reply

Please let me know if you’re looking for a article writer for your blog. You have some really great posts and I believe I would be a good asset. If you ever want to take some of the load off, I’d really like to write some material for your blog in exchange for a link back to mine. Please blast me an email if interested. Thank you!

Reply

We came across a cool web-site that you might love. Take a appear in the event you want.

Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>