10GE is Ready for Your Cluster

The wait is over. Next generation Gigabit Ethernet has hit the ground running

Say “cluster” and try to keep your mind from images of massive, government-funded scientific applications or herds of caffeine-fueled grad students. Pretty tough. But in fact, the vast majority of high performance computing (HPC) clusters are nowhere near large enough to qualify as massive, are used in commercial environments, and run on Gigabit Ethernet interconnects. Even within the TOP500(r) Supercomputer Sites the number of clusters running Gigabit Ethernet is more than double the number of clusters running InfiniBand. Certainly, higher speed and lower latency would be nice for any installation. But the performance requirements for most applications just don’t merit the high cost and labor-intensive maintenance of InfiniBand.

What most Gigabit Ethernet HPC sites could really use is an upgrade to 10 Gigabit Ethernet (10GE)-if it could be done cost-effectively and reliably. Until now, that idea would generate hesitation and skepticism among knowledgeable decision-makers. But with Gigabit Ethernet already entrenched in the HPC market and providing a slew of advantages, only a few obstacles have prevented the widespread growth of 10GE. Those obstacles are quickly evaporating. With recent technology advances, pricing improvements, and proven vendors entering the market, the choice of 10GE for HPC clusters has become quite attractive.

Understanding 10GE

Understanding the environment for 10GE merits a little history. Although Ethernet has been around for three decades, the technology remains viable because it has evolved over time to meet changing industry requirements. Widespread Ethernet adoption began when the IEEE established the 10 Mbps Ethernet standard in 1983. That standard evolved to Fast Ethernet (100 Mbps), Gigabit Ethernet (1000 Mbps), and 10 Gigabit Ethernet, with 40 and 100 Gigabit standards coming soon. In fact, discussions have started about Terabit Ethernet-a million Mbps-a speed that was hard to imagine just a few years ago.

Despite this evolution, the basic Ethernet frame format and principles of operation have remained virtually unchanged. As a result, networks of mixed speeds (10/100/1000 Mbps) operate uniformly without the need for expensive or complex gateways. When Ethernet was first deployed it could easily be confused with true plumbing-it was coaxial tubing which required special tools even to bend it. As Ethernet evolved it absorbed advancements in cabling and optics, changed from shared to switched media, introduced the concept of virtualization via VLANs, and incorporated Jumbo Frames and many other improvements. Today Ethernet continues to evolve with sweeping changes such as support for block-level storage (Fibre Channel over Ethernet).

Ratified in 2002 as IEEE 802.3ae, today’s 10GE supports 10 Gigabits per second transmission over distances up to 80 km. In almost every respect, 10GE is fully compatible with previous versions of Ethernet. It uses the same frame format, Media Access Control (MAC) protocol, and frame size, and network managers can use familiar management tools and operational procedures.

Ethernet Advantages for HPC

The fact that more than half of the TOP500 Supercomputer Sites and almost all smaller clusters run Ethernet is no surprise when you look at the benefits this technology offers:

  • High Comfort Level: As a widely-used standard, Ethernet is a known environment for IT executives, network administrators, server vendors, and managed service providers around the world. They have the tools to manage it and the knowledge to maintain it. Broad vendor support is also a plus – almost all vendors support Ethernet.
  • Best Practices: High availability, failover, management, security, backup networks, and other best practices are well-established in Ethernet and their implementation is widely understood. This is another example of the wide acceptance and vendor support for Ethernet. (Good luck finding an InfiniBand firewall, for example!)
  • Single Infrastructure: Ethernet gives HPC administrators the advantage of a single infrastructure that supports the four major connectivity requirements: user access, server management, storage connectivity, and cluster interconnect. A single infrastructure is easier to manage and less expensive to purchase, power, and maintain than using a separate technology for storage or for the processor interconnect.
  • Lower Power Requirements: Power is one of the biggest expenses facing data center managers today. New environmental mandates combined with rising energy costs and demand are forcing administrators to focus on Green initiatives. Ethernet is an efficient option for power and cooling, especially when used in designs that reduce power consumption.
  • Lower cost: With new servers shipping 10G ports on the motherboard and 10G switch ports now priced below $500, 10GE has a compelling price/performance advantage over niche technologies such as InfiniBand.
  • Growth Path: Higher-speed Ethernet will capitalize on the large installed base of Gigabit Ethernet. New 40GE and 100GE products will become available soon, and will be supported by many silicon and equipment vendors.

For those applications that could benefit from higher speeds, 10GE offers even more benefits.

  • More Efficient Power Utilization: 10GE requires less power per gigabit than Gigabit Ethernet, so you get ten times the bandwidth without ten times the power.
  • Practical Performance: 10GE can obviously move data 10 times faster than Gigabit Ethernet, but due to the new generation of 10GE NICs it also can reduce latency between servers by about 8 times. This bandwidth and latency gain translates into higher application performance than you might imagine. For molecular dynamics (VASP running on a 64 core cluster) the application ran more than six times faster than Gigabit Ethernet and was nearly identical to InfiniBand DDR. In a mechanical simulation benchmark (PAM CRASH running on a 64 compute core cluster), 10GE completed tasks in about 70 percent less time than Gigabit Ethernet, and was equal to InfiniBand DDR. Similar results have been observed on common HPC cluster applications such as FLUENT and RADIOSS, and more test results are coming in with similar results.

These benchmarks are impressive. Vendors love talking about microseconds and gigabits per second. But the real advantage in commercial applications is the increase in user productivity, and that’s measured by the clock on the wall. If computations run 70 percent faster, users can be 70 percent more productive.

The advantages of 10GE have many cluster architects practically salivating at the prospect of upgrading to 10GE, and experts have been predicting rapid growth in the 10GE for cluster market for years. That hasn’t happened-yet.

Obstacles Eradicated

Until recently, 10GE was stuck in the starting gate because of a few – but arguably significant – problems involving pricing, stability, and standards. Those problems have now been overcome, and 10GE has taken off. Here’s what happened.

  • Network Interface Cards (NICs): Some early adopters of 10GE were discouraged by problems with the NICs, starting with the price. Until recently, the only NICs available for 10GE applications cost about $800 and many users prefer to use two of them per server. Now server vendors are starting to add an Ethernet chip to the motherboard-known as LAN-on-Motherboard (LOM) – instead of using a separate board. This advance drops the cost to well under $100 and removes the NIC price obstacle from 10GE. Standalone NIC prices are now as low as $500 and will continue to drop as LOM technology lets NIC vendors reach the high volumes they need to keep costs down.
  • Another NIC-related obstacle was the questionable reliability of some of the offerings. A few of these created a bad initial impression of 10GE, with immature software drivers that were prone to underperforming or even crashing. The industry has now grown past those problems, and strong players such as Chelsio, Intel and Broadcom are providing stable, reliable products.

  • Switch Prices: Like NICs, initial 10GE switch prices inhibited early adoption of the technology. The original 10GE switches cost as much as $20,000 per port, which was more than the price of a server. Now list prices for 10GE switches are lower than $500 per port, and street prices are even lower. And that pricing is available for embedded blade switches as well as the top of rack products.
  • Switch Scaling: A market inhibitor for large clusters was how to hook switches together to create a nonblocking cluster. Most clusters are small enough that this is not an issue. For larger clusters, CLOS technology for scaling Ethernet switches provides a solution, and is starting to become established in the market.
  • PHY Confusion: Rapid evolution of the different fiber optic transceiver standards was a stopper for customers. Standards defining the plug-in transceiver quickly changed from XENPAK to X2 to XFP to SFP+, with each bringing smaller size and lower cost. But because each type of transceiver has a different size and shape, a switch or NIC is only compatible with one option. Using multiple types of optics would increase data center complexity and add costs such as stockpiling additional spares. With visions of Blue-ray versus HD-DVD, VHS versus Betamax, and MS-DOS versus CP/M, users were unwilling to bet on a survivor and shunned the technology as they waited to see which way the market would move.
  • Eventually, the evolution culminated in SFP+. This technology is specified by the ANSI T11 Group for 8.5- and 10-Gbps Fibre Channel, as well as 10GE. The SFP+ module is small enough to fit 48 in a single rack-unit switch, just like the RHJ-45 connectors used in previous Ethernet generations. It also houses fewer electronics, thereby reducing the power and cost per port. SFP+ has been a boon to the 10GE industry, allowing switch vendors to pack more ports into smaller form factors, and lowering system costs through better integration of IC functions at the host card level. As a result, fewer sparks are flying in the format wars, and the industry is seeing a very rapid convergence onto SFP+.

  • Cabling: Many users have been holding out for 10GBase-T because it uses a common RJ45 connector and can give the market what it’s waiting for: simple, inexpensive 10GE. But the physics are different at 10GE. With current technology, the chips are expensive, power hungry, and require new cabling (Cat6A or Cat 7). 10GBase-T components also add 2.6 microseconds latency across each cable-exactly what you don’t want in a cluster interconnect. And as we wait for 10GBase-T, less expensive and less power-hungry technologies are being developed. 10GBASE-CX4 offers reliability and low latency, and is a proven solution that has become a mainstay technology for 10GE.

    Making the wait easier is new SFP+ Copper (Twinax) Direct Attach cables, which are thin, passive cables with SFP+ ends. With support for distances up to 10 meters, they are actually ideal for wiring inside a rack or between servers and switches that are in close proximity. At an initial cost of $40 to $50 and with an outlook for much lower pricing, Twinax provides a simpler and less expensive alternative to optical cables. With advances such as these, clarity is overcoming confusion in the market. The combination of SFP+ Direct Attach cables for short distances, familiar optical transceivers for longer runs, and 10GBASE-CX4 for the lowest latency, there are great choices today for wiring clusters.

Comments on "10GE is Ready for Your Cluster"

Hi there, I check your new stuff on a regular basis. Your writing style is awesome,
keep up the good work!

I think everything posted was very logical. However, what about this?suppose you were to write a awesome headline? I am not saying your content is not good., however suppose you added a post title that makes people want more? I mean 10GE is Ready for Your Cluster is kinda plain. You ought to look at Yahoo’s front page and watch how they write news headlines to grab viewers interested.You might add a related video or a related pic or two to grab readers interested about everything’ve written. Just my opinion, it could bring your posts a little livelier.my blog post :: cheap car insurance michigan

Here are some links to web sites that we link to since we consider they may be worth visiting.

Before choosing such a cat, think about the extra care and grooming that they require. When it comes to behavior, the Sphynx is nothing like the conventional domesticated cat that is known for lying around and sleeping most of the day. This doesn’t need to be a chore, many people enjoy grooming their cat while watching TV.my homepage … Ragdoll cats

This is a very good tip particularly to those fresh to the blogosphere.Short but very accurate information… Many thanks for sharing this one. A must read article!Check out my web site: colon cleanse [http://www.515600.org/]

Sites of interest we’ve a link to.

The people of different age group could easily rely on this dress alternative. This type involving Wedding Dresses is ideal for unbiased lady. You need to make sure that outstanding services is anticipated from the online wedding store exactly where you will be purchasing Bridesmaids Dresses Brisbane. In house, to feel more easiness it is worn in cotton material. They’ll also request you to provide the body measurements of your bridesmaids on the tool exactly where you will input the figures. It not only makes for an unforgettable day but also becomes a status symbol.

Phone Νumbers іnformation Extractor 2.zeroԌеt Yⲟur sigle * ɑt tһiѕ time cаll thе Vibeline ѕmall talk ⅼine and Ьegin speaking ѡith singles іn ʏοur space.urban mеn аnd women from all ߋνеr ɑгe аble tο vibe. Ꮐet уоur free test* in tɦе ρresent ɗay!inscribe Yοur area Code: * tests ɑге solely οut tɦere tо primmitive existence nameers ѡɦo are 18 years of dust ߋr ancient.Using spinster gossip family numbers and meeting оthers neҳt tto celphone ѕmall talk traces іѕ usually ɑ dedply enjoyable ɑnd entertaining ⲣast being.ѡe provide important ѕmall talk reign suggestions ɑnd resources neⲭt tо single house numbers tɦat ѡill һelp үοu connect νia 1000′ѕ оf singles.fjnd ɑ ɗate, honoring, romance, οr ƅе taught mre bʏ courting and ting safely.Start chatting ᴠia new individual іn ʏοur space, νia ѕimilar pursuits, proper presently! Dont continue modest, naje at present ɑnd meet one hundredpercent actual individuals. Ƅetter οf each οne, ѕtarted іѕ fast and easy, youu may Ье speaking new native mates іn a concern օff minutes іn ourr single chat гooms,սsing merely selecting uphill tthe phone.үоu’ll not shelter requested for аny bank card οr billing info.Aгᥱ ʏоu trying tο chkose tҺᥱ sexiest banter specialist,рrobably tҺe most skilled free chitchat family inside seattle workers, οr tɦᥱ bеst advantages?Ԝe ship excellent pride within thᥱ optimistic effects ԝе pull оff fⲟr our members. а ouple οf organizations provide ɑ person Ьу matchinsideg applications fߋr their purchasers, however spinster live chat strains іѕ tҺe most knowledgeable, top notch, respected, deep-priced greatest unattached chat rule numbers agency іmmediately.Ιt ⅾoes not situation if уоur ѡant іѕ sizable оr unheard օf, tҺе delightful attractive crew at siongle live ѕmall talk traces iѕ fully dedicated tο beneficial үⲟu to establish yur mind-bogglcontained bygg adult phone ѕmall talk trials try ɑ certainty. ѡе’гe more ⅼikely tto nearly alll oof ʏⲟur queries, move үοu through tһе venture at no matter ԝalk іѕ most luxurious ffor үօu, ɑnd aѕ ᴡell aѕ conquered any sort оf challenges іt’ѕ рossible ʏօu’ll effectively lunch.ʏou աill ɑctually ցеt pleasure from meeting іn tҺе air oon tҺіs gossiptsurrounded byg website. wе areunattached stay chitchat strains , and ᴡе’гe ϲontent material tο meet youur free ѕmall talk empire іn seattle objectives and expectations ƅoth аt thіѕ time іn аddition tⲟ օѵеr time tߋ return.Feel free tօ visit mу paghe :: PhoneSexDay

Wow, wonderful weblog layout! How long have you ever been running a blog for? you make blogging look easy. The full look of your web site is wonderful, as neatly as the content material!Also visit my site casino gambling (http://www.wikilogicfoundation.org/wiki/index.php?title=User:PenneyO287)

You cɑn aⅼѕߋ seek tɦе advice օf ʏοur fence contractor fοr all οther aspects of tɦе activity ᴡhich уοu arᥱ not fairly confident ԝith.Feel free tо surf to mү weblog; ogrodzenia plastikowe

Fantastic website. Plenty of useful info here.I’m sending it to several pals ans additionally sharing in delicious.And obviously, thank you on your effort!Here is my web-site :: very cheap car insurance

What’s up, I read your blog regularly. Your story-telling style is awesome, keep it up!prestamo

I blog often and I truly appreciate your information. This great article has really peaked mmy interest.I’m going to bookmark your website and keep checking for neww details about once per week.I subscribed to yoir RSS feed too.Here is my site; Cheap car insurance

This piece of writing will help tthe internet people for building up new web site or even a blog from start too end.Here is my webpage … cheap car insurance florida

Hello, everything is going nicely here and ofcourse every one is sharing information, that’s genuinely fine, ksep up writing.my blog; cheap car insurance in md

fantastic put up, very informative. I’m wondering why thhe opposite specialists of this sector don’t notice this. You must procxeed your writing.I’m confident, you’ve a huge readers’ base already!Feel free to surf to my webpage :: cheap car insurance

I like the valuable information youu provide in your articles.I’ll bookmark your blog and check agajn here frequently.I’m quite sure I will learn many new stuff right here! Good luck for the next!Here is my homepage :: Cheap Car Insurance

Now, if you like these two characteristics for your bridal dress, then you can definitely find the right one for you browsing through this category only. nonetheless; not every ladies take this into account. The little black dress is an eternal favorite to these occasions and looks adorably stunning. Furthermore, a boutique that both runs on the internet and in an actual shop surely signifies they’re stable enough to serve their own clients well. Lakota, Tlingit, Iroquois, DIne and Muscogee were considered the five most important Native American tribes.I will get the psychology cause of why we differ in terms of style.Feel free to visit my web-site … dresses

Hi, just wanted to mention, I liked this post. It was inspiring.Keep on posting!Have a look at my webpage; cheap car insurance in ohio

A person requested for the review of shopify ecommerce software program and this began the journey of reviewing ecommerce news india products.

Hi there, just wanted to say, I loved this blog post.It wwas practical. Keep on posting!Have a look at my web site cheap car insurance

When I originally commented I clicked the “Notify me when new comments are added” checkbox aand now each time a commentt is added I get several e-mails with the same comment. Is there any way you can remove people from that service?Thanks!Here iis mmy web blog – cheap liability car insurance

I read this piece of writing comoletely on the topic of the comparison of hottest and preceding technologies, it’s amazing article.my web blog: cheap car insurance in pa

Heya i’mfor the first time here. I found this board and I find It truly useful & it helped mee out a lot. I hope to give something bafk and aid others like yoou helped me.Also visitt my web site :: Cheap car insurance

Hey there! This is my first visit to your blog!We are a collection of volunteers and starting a new initiative iin a community in the same niche.Your blog provided us beneficial information to work on. You have done a marvellous job!My blog – cheap car insurance companies

A fascinating discussion is worth comment. I think that you should publish more about this subject matter,it might not be a taboo matter but usually folks don’t talk about such topics.To the next! Cheers!!Also visit my web-site – Car insurance companies

Just desired to stress I am just thrilled I stumbled in your web page.My web page: http://suplementy-na-rozrost-miesni.suplementy.bytom.pl

Really no matter if someone doesn’t know afterward its up tto other users that they will help, soo here it occurs.Feel ree to visit my web blog; Cheap Car Insurance

Hi my friend! I want to say that this article is awesome, great written and come with almost all significant infos.I’d like to peer more posts like thiks .Feell free to visit my page; cheap car insurance

If yߋu commence nailing rails and slats tօ tҺᥱ posts before tһе concrete ɦaѕ Һad sufficient time tо ѕеt thе posts will ƅᥱ knocked out οf level.Αlso vvisit mү site; Strona g?ówna

My family all tthe tiime say that I aam killing my time here att web, but I know I am getting knowledge all the time by reading such pleasant articles.Feel free to visit my web site; cheap car insurance quotes

heey there and thank you for your information – I have definitely picked up anything new from right here. I did however expertise some technical points using this site, since I experienced to reload the website mzny times previous to I could get it tto load properly.I had been wndering if your hosting is OK? Not that I’m complaining, bbut slow loading instances times will often affect your placeement in google andd can damage your high-quality score if adss andd marketing with Adwords. Anyway I’m adding this RSS too my email and could look out for a loot more of your respective intriguing content.Make sure you update this again very soon.Have a llook at mmy homepage :: Cheap car insurance

Best appetite suppressant pills Howdy are using WordPress for your blog platform? I’m new to tthe blog world but I’m trying to get started andd set up my own. Do you need any coding knwledge to make youjr own blog?Any help woulld be really appreciated!Review my website :: thhe best weight los piklls (credigo.ch)

I’m not that much of a online reader to be honest but your sites really nice, keep it up!I’ll go ahead and bookmark your site to come back later.CheersHere is my weblog; cheap car insurance in michigan

click here (Kristian) When somke one searches for his essential thing, thus he/she desires to be availabhle that in detail, thus that thing is maintained over here.

If some one desires to be updated with latest technologies then he must be go to see this site and be up to date every day.Here is my webpage … cheap car insurance in pa

Whats up this is somewhat of off topic but I was wanting to know if blogs use WYSIWYG editors or if you have to manually code with HTML. I’m starting a blog soon but have no coding experience so I wanted to get advice from someone with experience. Any help would be greatly appreciated!Visit my webpage – new hindi songs dhunde tujhko is the best hindi love songs 2016

I like the valuable information you provide in your articles.I’ll bookmark your weblog and check again here frequently.I am quite sure I’ll learn lots of new stuff right here! Good luck for the next!my blog post: postsjc.org (Dalene)

let’s just say that this will be your best red dress yet. These shoes go very well with the black dresses which involve lace and net. Let me introduce you to the colour red and show you how it can be just as versatile as black. There are so many ways we can get it wrong but do not worry there are so many ways that we can get it right.A government official presides over the civil part of the ceremony, which is the signing of the relevant documents. A black cocktail dress never fails to deliver its purpose.Are you looking for a fabulous Dress black cocktail.Also visit my website; red dress and black shoes for a party

I’m not sure if this is a formatting issue or something to do with browser compatibility but I figured I’d post to let you know.10GE is Ready for Your Cluster – firma curatenieMy blog – firma curatenie

If you have not merely the tip of the game will make you an ultimate picture spider-man unlimited hack of the game.Feel free to check into my illustrious web blog; spider-man unlimited hack with cydia

It is not my first time to pay a quick visit this web page, i am browsing this website dailly and get pleasant facts from here daily.my homepage; Natural Mango Cleanse review

Very few internet websites that take place to be detailed below, from our point of view are undoubtedly properly worth checking out.

For grtoup programs, carry a buddy and get 500 bht low cost on your course.Feel free to visit my website: e commerce classes nyc

Here are a few of the internet sites we advise for our visitors.

Here is an excellent Weblog You might Obtain Interesting that we encourage you to visit.

Usually posts some really exciting stuff like this. If you?re new to this site.

Leave a Reply