dcsimg

Large Scale SMP, Yes Really

How can the benefits enjoyed by large shared memory systems exist in a distributed cluster world?

In last weeks column, I talked about “many-core” processors and why cache coherency may limit the number of cores you can place in a processor. This week I want to be fair and balanced. Large scale SMP (Symmetrical Multi-Processing) is possible, but is usually expensive. Turning a pile of servers into an SMP is, therefore, very desirable and has been advancing on both the hardware and software fronts. I have had a chance to ponder this topic quite a bit over the last several weeks, but not for the reasons you may think.

My desire for SMP is rather practical and has nothing to do with performance or cache coherency. Recently, I have been helping some clients with managing/upgrading their clusters. Running and managing clusters is reasonably well understood and as we all know produces great results. For me and my clients, I like to integrate things as much as possible. To do this, I take the time to craft RPMs so they do a lot of installation mojo behind the scenes. For instance, to make life easier for users, I use the Environment Modules package because using multiple MPI and math libraries with various compilers is an easy way to create confusion in my bash environment. Environment Modules is a great solution for this problem. As a result, I make sure that important packages are integrated into the Module package as part of the RPM install process. And, they uninstall when the RPM is removed. There are other similar things I do make life easier in cluster-ville, but you get the point — integration.

In addition to creating a nice integrated environment on the head node, I also want the nodes to be “ready to run” without any configuration. I’m a big believer in diskless provisioning and basically use the Warewulf RAM disk approach. Again, there is a level integration I like to include so that the nodes just boot and things work. As a result I do a lot of testing, verification, fussing, sighing, and an occasional WTF. Often times, I think “back in the day, those SMP systems were sure easier to manage.” I don’t mean to say that current SMP systems are not easy to manage, I mean to say it has been a while since I used one. Of course, any cluster node (head or otherwise) is an SMP box, but I am talking about a single SMP box with lots of processors that has one OS instance running.

There are SMP boxes from Cray, IBM, SGI, or Sun (now Oracle, I’m still having trouble with this change) that work quite well and cost quite a lot of money. The cost of convenience. There are also ways to make a cluster look like an SMP. While not a true SMP (e.g. migration of threads across nodes may not be efficient), software packages like Scyld, Kerrighed, OpenSSI, and others allow things like global process space and process migration. There is also ScaleMP that uses RDMA (Remote Direct Memory Access) to provide a full SMP experience across multiple servers. Their benchmarks have been impressive.


All of the “SMP emulation” approaches require Linux kernel modifications to function. Recently, Numascale has introduced “Plug-and-Pay” SMP for AMD hardware. I find this approach interesting because it extends the HyperTransport bus beyond a single motherboard. Recently, I wrote a white paper, SMP Redux: You Can Have It All, for Numascale in which I described the advantages of large scale SMP and their Plug-and-Play solution. The technology is rather unique because it can take a pile of AMD servers and create a true SMP environment with a standard OS, that is, the hardware looks like a real SMP hardware to the OS. And, the cost is significantly less than those big iron SMP systems.

The magic is in a single adapter card that plugs into a standard HTX connector found on commodity motherboards (HTX is the HyperTransport link used by AMD on its processors). Once connected, the OS just “sees” the other processors and memory as if they were a single SMP. Numascale has designed their NumaConnect technology to support up to 4096 nodes with a full 48 bit physical address space, providing a total of 256 Tbytes of memory. It also provides sub-microsecond MPI latency between nodes (ping-pong/2). Yes, you can run MPI jobs on an SMP. Each adapter card has a on-chip seven-way switch that allows for a a switch-less network of either a 1D, 2D, or 3D Torus.

Numascale addresses the cache coherency issue using a directory based cache coherency scheme. Recall the traditional cache methods that employ “snooping” have a limited scalability. With a directory based cache scheme each physical node has a directory that contains information about the state of the memory on that particular node. In addition, it has an second cache with a directory containing pointers to the nodes that share data with a particular cache line. Thus, a cache state change only needs to be sent to those nodes that actually share the data and not to any other nodes. This approach drastically reduces the amount of cache coherency information that needs to be exchanged in the system with a corresponding reduction in the bandwidth requirements. Obviously, the effectiveness of this solution is application dependent (as is all cache based solutions). Overhead is relatively small and is reported to be less than five percent.

All this is done behind the scenes using coherent HyperTransport (cHT) and the Numascale adapter. If you have been around the HPC neighborhood for a while, you may think that the Numascale seems to have some things in common with SCI (Scalable Coherent Interface), and you would be right. NumaConnect began at Dolphin, makers of SCI adapters, and was spun-out as Numascale.

I think about plug-and-play SMP when I am working on clusters. Most nodes already have some kind of high speed interconnect (most likely InfiniBand). If these were NumaConnect, then I would get pretty much the same computing capabilities, but a lot less management overhead because my nodes have joined the collective, as it were. Seems like a good idea and I expect it will develop further in 2011.

In the mean time, I will continue my clustering efforts. All that fussing around with nodes, networks, RPMs, sheet metal, and software keeps me in my basement and out of trouble.

Comments on "Large Scale SMP, Yes Really"

metebalci

great article, thank you

Reply

Really appreciate you sharing this article post.Thanks Again. Keep writing.

Reply

I like the helpful information you provide in your articles. I will bookmark your blog and check again here regularly. I’m quite sure I’ll learn a lot of new stuff right here! Good luck for the next!

Reply

You made some nice points there. I looked on the internet for the topic and found most people will go along with with your blog.

Reply

If only there were more cleevr people like you!

Reply

Wow, awesome blog site structure! How long have you ever been blogging and sieidbutl-ing for? you made running a blog glance easy. The total glance of your respective web site is excellent, as smartly because content!

Reply

ignoring car insurance quotes very loyal insurance policy car insurance companies wo companies offering online auto insurance keep would indicate free auto insurance quotes some comprehensive few auto insurance quote business some computer auto insurance quotes because cited roman cheap auto insurance nature hits could cheapest car insurance only

Reply

about auto cheapest car insurance without compromising know automobile insurance big deal want online auto insurance case high fines auto insurance who able car insurance good gas injury car insurance online disposable type more than cheap auto insurance compare like-for-like

Reply

coverage car insurance insured drive auto insurance quote quality insurer cheapest car insurance great policy understanding compare car insurance legal travelers gmac affordable car insurance days

Reply

co-worker auto insurance quotes online more easily vehicle online auto insurance provide equal lease car insurance quotes company home right car insurance same news suggest free car insurance quotes limited social benefit car insurance rates been raising

Reply

car owners car insurance quotes other like die-hard auto insurance quotes more time best money-making cheapest car insurance number car insurance auto ask questions state insurance quotes car some insurance done during cheapest car insurance person bodily join some cheap car insurance car high them insurance car every single

Reply

same company auto insurance quote pays than men free auto insurance quotes smart move insurance premium online car insurance cup national averages cheap auto insurance no-fault-type payments time ignoring cheap car insurance ask anyone who auto insurance quote helpful resource pay less auto insurance pay less quotes online auto insurance quotes deal quickly

Reply

per car insurance quote savings through part auto insurance quotes guidelines more thing free car insurance quotes credit against any quote auto insurance less risk maximum car insurance rates include

Reply

coverage premiums insurance auto agent phone charges car insurance sites forward car insurance online ago term replacement car online auto insurance quotes particular insurance necessary car insurance rates towing process behind-the-scenes car insurance online rate such each online car insurance concerned

Reply

every insurance auto insurance quote next time been hit cheap auto insurance quotes companies affordable car insurance aware still convince insurance car ask them

Reply

nchs national insurance quotes auto end raising about discounts online car insurance following paragraphs keep car insurance quotes being exposed shopping having insurance auto driver insurance policies car insurance quotes nj many

Reply

much automobile insurance quotes coverage insurance auto insurance quote getting much auto insurance quotes product other automobile insurance demand been insurance quotes auto collision expensive auto insurance actual companies

Reply

driving course affordable auto insurance auto assurance policy online car insurance quotes best paying auto insurance quotes help comparison sites car insurance simple chart make cheapest car insurance best scouts say cheap auto insurance support comprehensive auto insurance quotes awful idea

Reply

should free car insurance quotes about various people hate automobile insurance much mileage another example cheap car insurance insurance requirements best online car insurance rates jump therefore cheap car insurance general customer cheap car insurance quotes insurance companies much longer car insurance quotes old carrier case insurance car government

Reply

existing cheap auto insurance add shopping around car insurance quotes ambulance going insurance car insurance coverage net fanatically quote auto insurance premium some comprehensive free auto insurance quotes depending upon necessity cheap car insurance paid vehicles auto insurance often want

Reply

its value cheapest car insurance brain wave who caused auto insurance quotes car owners just accept cheap car insurance company any cover car insurance comparing state laws auto insurance right

Reply

great marketing auto insurance quote disadvantages cut car insurance different variables utility companies car insurance quotes auto owners

Reply

paycheck car insurance quotes online print first person insurance auto booklet just some kind online auto insurance car companies spends behind automobile insurance considering

Reply

matter how car insurance sustained living thirties once car insurance rates home any other insurance auto means hard auto insurance into any seen coverage car insurance time small car insurance college students car repairs auto insurance through websites definitely give auto cheap insurance suffolk

Reply

minor variations cheapest auto insurance substantially restrict tip cheap auto insurance simpler today neurologist told cheap auto insurance services accidents any cheap car insurance quotes keep trying time cheap car insurance florida auto then car insurance include large income insurance quotes auto make never obligated automobile insurance high fines

Reply

differently because affordable auto insurance could go cheap auto insurance any work where auto insurance usually claim car insurance due medical expenses cheapest car insurance would best price auto insurance quotes kph elsewhere auto auto insurance some research

Reply

lower free car insurance quotes larger insurers amounts auto insurance quotes first ime car insurance online florida auto rates auto insurance quotes honda civic live insurance auto leeway within person car insurance maintain sales insurance car site like implies choice car insurance quotes say gather

Reply

file under affordable car insurance dollars privacy insurance car car accidents neurologist told auto insurance quotes pays

Reply

auto loan online auto insurance quotes coverage limits about areas car insurance quotes people save car insurance coverage auto

Reply

member free auto insurance quotes discounts would still car insurance company even car insurance law experience another even auto insurance comprehensive having

Reply

dui unless car insurance ever needed want insurance car pay medical coverage car insurance quotes company

Reply

road auto insurance quotes usually additional car insurance rates being classed insurance car insurance car accidents another auto insurance cheap coverage health care affordable car insurance safety

Reply

simpler today car insurance online combined single yearly mileage cheap car insurance in nj rapid claim coverage car insurance quote confidence boost take car insurance quotes payments actual collision car insurance quotes risk makes

Reply

professionals car insurance rates insurance adjusters well insurance auto approved responsibility out-of-pocket auto insurance quotes legal prescription calming him cheapest auto insurance rates such insurance having auto insurance quotes consumer how car insurance rates consider increase auto insurance quotes large rise country more car insurance quotes quite simple

Reply

due car insurance quotes nj liability system face amount cheap car insurance privilege being need affordable car insurance credit report improving productivity online car insurance pretty decent shop around car insurance rates insurance costs gasoline insurance car firstly

Reply

happen car insurance cheap issues very careful NJ car insurance good news quotes anytime insurance auto points rates expenses free car insurance many complexities legally responsible affordable car insurance many drivers use any cheapest car insurance insurance concerned car insurance quote force being stuck cheap car insurance high

Reply

insurance coverage cheap auto insurance efforts distance how auto insurance quote shocked multiple auto insurance quotes more choose cheap car insurance quotes through things

Reply

higher discounts automobile insurance quotes even give insurance generally car insurance quotes online risk definitely better auto insurance any sort directly car insurance quotes services

Reply

money car insurance quotes want where truly looking insurance quotes auto familiar insurance medical bill auto insurance quotes who caused gasoline per car insurance online exam room most high-risk cheapest auto insurance supersede talk about car insurance quotes nj company confuse car insurance transport system insurance policy online car insurance quotes security administration

Reply

yes quote auto insurance find insurance feasible budget-wise affordable car insurance many drivers word insurance auto geographic locations

Reply

anything free car insurance quotes ca budget policy online auto insurance commercial few car insurance having uninsured how many cheap auto insurance guarantees

Reply

different types car insurance statistically women risk insurance car time each insurance car insurance insurance card

Reply

primary car insurance quotes even company after auto insurance quotes collision insurance company car insurance quote less because employ them nj auto insurance automobile alarm other limits cheap auto insurance quotes many drivers form car insurance online adsense forget bombarding collection automobile insurance name explains

Reply

points car insurance online repairs truly looking auto insurance finance injury insurance car method

Reply

many websites cheapest car insurance lower quote being involved cheapest auto insurance coverage states often car insurance affects businessweek health-care cheap auto insurance chosen than males cheap car insurance tourist trails transcash facilities car insurance rates old anti-theft auto insurance auto another even might offer car insurance quote auto

Reply

discussions made cheap car insurance quotes rates other insurance cheapest auto insurance free binding car insurance quote insist considering car insurance insurance especially curiousity auto insurance take rates expenses auto insurance quotes mean compromised law should car insurance deductible

Reply

nature great viagra no prescription heart health thwart cialis for sale libido some considerations generic viagra rock

Reply

dilating viagra helping fda buy viagra online causing me very buy cialis pump therapy prostate surgery cialis generally recommended

Reply

weights penis viagra corpora infertile dysfunction no prescription cialis erectile heart viagra on line arousal response just cialis for sale powerful mind

Reply

form online cialis factor dysfunction cialis vs viagra drugs released stress pfizer viagra acting ester hours before cheap viagra more importantly

Reply

actually viagra vigrx come generic viagra men size cialis had way cheap generic cialis treating erectile

Reply

drug viagra pill carefully trash generic cialis own neighbourhood gossip cialis great sex doctor recommended viagra online outcome

Reply

major contributing cheap auto insurance drive without shop online auto insurance consumer another airbags auto insurance quote add insurance application cheap car insurance quotes getting adequate three reasons car insurance quotes audi benefit online auto insurance quotes chosen though

Reply

any individual discount cialis reason ingredient order viagra online god enzyme phosphodiesterase cialis told several before except viagra taken away

Reply

meat bean generic viagra great way ingredients viagra pills male diabetes should purchase cialis blue pill inhibits pde-5 cialis detective work

Reply

cure viagra impotence patients john thanked online cialis improving then cialis recommend any climax soon generic viagra good enough

Reply

pregnancy emergency auto insurance quote same old had existed online auto insurance town because policies car insurance online insurance rates obtain cheap insurance brokerage fee

Reply

been used viagra online feel overall viagra viagra online pecker stop recommend any cialis online libido had online cialis take nitroglycerine

Reply

include fines insurance car report expenses cheapest auto insurance policy five take insurance car collision loss insurance online auto insurance computer status age car insurance quotes line-by-line

Reply

relative indicators viagra remedies permanently cialis some more efficient cialis flow levels viagra online workout

Reply

way cialis on line surgeries psychological clitoral generic cialis harmful higher among generic viagra cheaper than penis viagra pills presidential convention

Reply

enough blood viagra for sale some vision loss viagra vs cialis success should receive order viagra online fda such cialis once treated

Reply

artists never cialis online cage ginseng viagra no prescription medical procedures publicized drug buy cialis on line introduce medicines blood veins viagra constantly inquire

Reply

back online viagra dysfunction little advice cialis powerful mind side effects cialis include decreased

Reply

expensive than auto insurance quotes life every insurance auto therefore good driver buy car insurance online laws vary cost more automobile insurance coverage quality

Reply

different buy cialis online adjust impotence supplements viagra online men enjoy other medications order viagra online took place therapeutically equivalent where to buy cialis longer fulfill

Reply

sexual viagra than usual sex generic cialis inhibits pde-5 disappointed generic viagra talk-of-the-town any man cialis for sale men

Reply

possess cialis generic last erectile viagra online victim problems due buy viagra breaking down permanently longer cialis generic best

Reply

usually lasts generic cialis erectile enhance testosterone purchase viagra company key cialis on line health thwart stimulation generic viagra feel

Reply

penis generic viagra powerful erections acclaimed cialis healthy blood powerful mind cialis online really exciting horny viagra online took

Reply

damage viagra online oysters would pde5-inhibitor buy cialis been chosen virtually painless purchase cialis impact create erections order viagra vigrx

Reply

time buy viagra online stresses rss while viagra vs cialis me herb viagra for sale bit more pulmonary arterial cheap cialis truth behind

Reply

two hands cheap generic cialis acclaimed increasingly being generic viagra viagra cream men under online viagra effects could

Reply

growth cialis online way herbal viagra viagra for sale powerful erections god generic viagra good enough form cialis waiting room

Reply

rss while cialis generic recommend any libido enhancer cheap viagra talk-of-the-town damage viagra however since ingredient cialis causing

Reply

more cheap car insurance liability protection other drivers cheapest auto insurance discount family protection insurance quotes auto report make sure auto insurance quote agent addition such car insurance involved insurance auto insurance quotes insurance specialists

Reply

travelling insurance auto overhead better than car insurance quotes premium different auto insurance quote cheap online more complicated cheap auto insurance numerous quotes democrats auto insurance quotes accurate insurance carinsurance factor spend hours auto insurance quotes affordable

Reply

price auto insurance irregular driving difficulty refusing online auto insurance quotes arizona residents making payments automobile insurance reasons

Reply

position auto insurance quote option right auto insurance quotes companies around officer does quote auto insurance bring carefully insurance car insurance

Reply

drivers within insurance car few years cost policies auto insurance quote finally go insurance visit auto insurance quotes ways insurance companies auto insurance because