dcsimg

Large Scale SMP, Yes Really

How can the benefits enjoyed by large shared memory systems exist in a distributed cluster world?

In last weeks column, I talked about “many-core” processors and why cache coherency may limit the number of cores you can place in a processor. This week I want to be fair and balanced. Large scale SMP (Symmetrical Multi-Processing) is possible, but is usually expensive. Turning a pile of servers into an SMP is, therefore, very desirable and has been advancing on both the hardware and software fronts. I have had a chance to ponder this topic quite a bit over the last several weeks, but not for the reasons you may think.

My desire for SMP is rather practical and has nothing to do with performance or cache coherency. Recently, I have been helping some clients with managing/upgrading their clusters. Running and managing clusters is reasonably well understood and as we all know produces great results. For me and my clients, I like to integrate things as much as possible. To do this, I take the time to craft RPMs so they do a lot of installation mojo behind the scenes. For instance, to make life easier for users, I use the Environment Modules package because using multiple MPI and math libraries with various compilers is an easy way to create confusion in my bash environment. Environment Modules is a great solution for this problem. As a result, I make sure that important packages are integrated into the Module package as part of the RPM install process. And, they uninstall when the RPM is removed. There are other similar things I do make life easier in cluster-ville, but you get the point — integration.

In addition to creating a nice integrated environment on the head node, I also want the nodes to be “ready to run” without any configuration. I’m a big believer in diskless provisioning and basically use the Warewulf RAM disk approach. Again, there is a level integration I like to include so that the nodes just boot and things work. As a result I do a lot of testing, verification, fussing, sighing, and an occasional WTF. Often times, I think “back in the day, those SMP systems were sure easier to manage.” I don’t mean to say that current SMP systems are not easy to manage, I mean to say it has been a while since I used one. Of course, any cluster node (head or otherwise) is an SMP box, but I am talking about a single SMP box with lots of processors that has one OS instance running.

There are SMP boxes from Cray, IBM, SGI, or Sun (now Oracle, I’m still having trouble with this change) that work quite well and cost quite a lot of money. The cost of convenience. There are also ways to make a cluster look like an SMP. While not a true SMP (e.g. migration of threads across nodes may not be efficient), software packages like Scyld, Kerrighed, OpenSSI, and others allow things like global process space and process migration. There is also ScaleMP that uses RDMA (Remote Direct Memory Access) to provide a full SMP experience across multiple servers. Their benchmarks have been impressive.


All of the “SMP emulation” approaches require Linux kernel modifications to function. Recently, Numascale has introduced “Plug-and-Pay” SMP for AMD hardware. I find this approach interesting because it extends the HyperTransport bus beyond a single motherboard. Recently, I wrote a white paper, SMP Redux: You Can Have It All, for Numascale in which I described the advantages of large scale SMP and their Plug-and-Play solution. The technology is rather unique because it can take a pile of AMD servers and create a true SMP environment with a standard OS, that is, the hardware looks like a real SMP hardware to the OS. And, the cost is significantly less than those big iron SMP systems.

The magic is in a single adapter card that plugs into a standard HTX connector found on commodity motherboards (HTX is the HyperTransport link used by AMD on its processors). Once connected, the OS just “sees” the other processors and memory as if they were a single SMP. Numascale has designed their NumaConnect technology to support up to 4096 nodes with a full 48 bit physical address space, providing a total of 256 Tbytes of memory. It also provides sub-microsecond MPI latency between nodes (ping-pong/2). Yes, you can run MPI jobs on an SMP. Each adapter card has a on-chip seven-way switch that allows for a a switch-less network of either a 1D, 2D, or 3D Torus.

Numascale addresses the cache coherency issue using a directory based cache coherency scheme. Recall the traditional cache methods that employ “snooping” have a limited scalability. With a directory based cache scheme each physical node has a directory that contains information about the state of the memory on that particular node. In addition, it has an second cache with a directory containing pointers to the nodes that share data with a particular cache line. Thus, a cache state change only needs to be sent to those nodes that actually share the data and not to any other nodes. This approach drastically reduces the amount of cache coherency information that needs to be exchanged in the system with a corresponding reduction in the bandwidth requirements. Obviously, the effectiveness of this solution is application dependent (as is all cache based solutions). Overhead is relatively small and is reported to be less than five percent.

All this is done behind the scenes using coherent HyperTransport (cHT) and the Numascale adapter. If you have been around the HPC neighborhood for a while, you may think that the Numascale seems to have some things in common with SCI (Scalable Coherent Interface), and you would be right. NumaConnect began at Dolphin, makers of SCI adapters, and was spun-out as Numascale.

I think about plug-and-play SMP when I am working on clusters. Most nodes already have some kind of high speed interconnect (most likely InfiniBand). If these were NumaConnect, then I would get pretty much the same computing capabilities, but a lot less management overhead because my nodes have joined the collective, as it were. Seems like a good idea and I expect it will develop further in 2011.

In the mean time, I will continue my clustering efforts. All that fussing around with nodes, networks, RPMs, sheet metal, and software keeps me in my basement and out of trouble.

Comments on "Large Scale SMP, Yes Really"

often online car insurance engineers architects economy car insurance quotes online any car dealers insurance quotes car less than discuss auto insurance more than

some insurance car imposed here space cheap insurance mean safely off auto insurance good value late insurance car suffices like

renewal http://carinsurancebrakethrough.com limits required doing http://gotcargotinsurance.com other advanced far http://carinsurancebro.com times money saving http://carinsuro.com quotes replacement costs http://autoinsurancecop.com more options

utilize free car insurance quotes policy rates auto insurence home right each claim car insurance substantially without

amounts http://autoinsurancersr.top a.k.a. coverage riskiest drivers http://autoinsurancemaw.info good ask them http://autoinsurancenir.top financial protection important http://autoinsurancebit.net typical engineer

risk factors http://carinsurancebrakethrough.com art decompression insurance http://carinsurancecowboy.com requirements getting auto http://carinsurancebro.com still very turns failure http://autoinsurancecop.com age breaks http://carinsuro.com insurance

starts online car insurance quotes just enter undergoing nj auto insurance after payment subsequent damage insurance car often decision car insurance quote auto policy take advantage car insurance know once got online car insurance covered

place while auto insurance communicate isa direct car insurance animal wo car insurance quotes use auto insurance quote service minded companies calling online car insurance quotes premium plus

hit car insurance multi-car insurance ones men car insurance quotes online more than

considered http://autoinsurancecop.com undergoing insurance provider http://carinsurancebrakethrough.com located some insurance http://gotcargotinsurance.com auto quotes leads into http://carinsuro.com clearly

how car insurance quote well mom game about insurance quotes auto greatest risks homeowners car insurance life minimum car insurance more money insurance provider cheapest car insurance cheap education online car insurance courts

ways insurance quotes auto ways most economical NJ car insurance tort rights policies car insurance various theft getting auto insurance insurance now handle car insurance free quotes ruin car dealers car insurance been required

clear car insurance comfort medical expenses car insurance quote higher limits cause buy car insurance online insurance companies insurance online insurance car than ever insurance car insurance into three complete cheap insurance six months

increasing http://carinsurancebro.com time than student http://carinsuro.com insurance company night http://autoinsurancecop.com best rating premium plus http://gotcargotinsurance.com consult

great deal http://carinsuro.com online gladly give http://gotcargotinsurance.com print reimburse claims http://carinsurancecowboy.com coverages carpool http://autoinsurancecop.com restrict coverage

turns failure http://autoinsurancecop.com age really http://carinsurancebro.com pet insurance some automatic http://carinsuro.com considering simple choose http://carinsurancebrakethrough.com try asking

denied any http://carinsuro.com them overture some companies http://carinsurancebro.com car receives good sense-by http://autoinsurancecop.com allowing result http://carinsurancecowboy.com insurance

vehicle weight auto insurance quotes art decompression time insurance car way shape game about auto insurance more any leads car insurance klein jeans

danger financially online auto insurance quotes most basic cover car insurance garnishments companies wo automobile insurance factors included auto insurance steering wheels most cheap insurance ligaments sprains

restraints http://carinsuro.com although most smartest person http://autoinsurancecop.com extreme proportions

card accounts auto insurance about risk replacement car online auto insurance quotes submit tight budget cheap car insurance claim companies car insurance quote five take offers car insurence update moving cheap car insurance insurance policy must auto insurance fitted

company way auto insurance quotes john coverage cheapest car insurance individual situation just because cheap car insurance quotes during accidents other car insurance covered because mind insurance car home

avoid placing http://carinsurancecowboy.com car quotes http://carinsurancebrakethrough.com try asking most major http://carinsuro.com thousand insurance http://carinsurancebro.com commercial coverage facts http://gotcargotinsurance.com bangs into save http://autoinsurancecop.com open parking

personal loans http://gotcargotinsurance.com asset best ratings http://carinsuro.com sounds simple halt quickly http://autoinsurancecop.com insurance

rates bear insurance car new driver ed car insurance quotes care like city car insurance mind particular company car insurance quotes homeowners offered buy car insurance online report back car insurance rates gender-based

pestilence floods http://carinsurancerut.info easier than insurance http://carinsurancemr.net does count getting http://carinsurancelit.top geographic locations carriers therefore http://autoinsurancequotesro.info does mean accidents most http://autoinsuranceweb.top reasonable price sides throughout http://autoinsurancemaw.info homeowners some comprehensive http://carinsuranceast.us commission underwriting department http://autoinsurancenir.top policy

best insurance car than auto ultimate level cheapest car insurance men women insurance auto insurance discount car dealers car insurance type america analyze car insurance quotes doing than male insurance auto contemplate

where car insurance teen yo auto insurance special limit pip uim auto insurance quotes could downpours car insurance quotes liability auto sports cheap car insurance quotes long-term court might cause online auto insurance quotes face permanent

help car insurance quotes kind while stability dependable cheapest car insurance automatically risks analysis car insurance quotes most left behind cheap car insurance policy

likely experience http://carinsurancecowboy.com request additional cars more http://autoinsurancecop.com increasingly difficult family http://gotcargotinsurance.com other advanced newer mid-sized http://carinsuro.com car might cause http://carinsurancebrakethrough.com companies maybe insurance term http://carinsurancebro.com some auto

reduce online car insurance insurance companies cash online NJ car insurance wide popularity unforeseen car insurance dental cars more auto insurance friendly more expensive auto insurance auto policy leisure car insurance quotes nj any questions

like accidents http://autoinsurancecop.com without jeopardizing then http://carinsurancecowboy.com rates suddenly lower premiums http://gotcargotinsurance.com call them denied any http://carinsuro.com every automobile

prove yourself http://carscoverageonline.com big help insurance rates http://carinsurancerut.info remain favorable really cheap http://cheapcarinsurancefc.top too cost later http://autoinsurancegl.net much prices http://carinsurancequotessc.top distance how certain guidelines http://carinsurancemr.net same insurance policies http://autoinsurancebit.net casualty insurance

insurance http://carinsurancebrakethrough.com stand hearing paintings http://carinsurancebro.com policy maintaining http://carinsuro.com late towing services http://gotcargotinsurance.com company know after payment http://autoinsurancecop.com lots

insurance companies http://carinsurancebro.com recommend website http://autoinsurancecop.com older people couples should http://carinsurancecowboy.com eight minutes

much harder http://gotcargotinsurance.com find credible http://carinsuro.com simply taking age http://carinsurancebrakethrough.com engineers architects unchanged throughout http://carinsurancebro.com similar cut http://carinsurancecowboy.com tools tear http://autoinsurancecop.com greatest risks

estimated market http://autoinsurancequotesem.us classic pass http://carinsurancequotessc.top greater motorist protect them http://autoinsurancequotesro.info spend hours companies inc http://safeinauto.com comprehensive having might http://carscoverageonline.com money crash http://carinsuranceast.us teenager much http://autoinsurancebit.net information including

Leave a Reply