Storage Convergence: Fibre Channel over Ethernet

The ever-increasing demand for storage, network and compute resources for instant access of data and computational requirements lead to the continued growth of the variety of networks found in the data center. HPC is no exception to this trend.

DCB / CEE Introduction

An FCoE frame needs to be transported with the same lossless characteristics with which Fibre Channel is designed. To allow for this transport, the fundamental nature of Ethernet needs to change from a lossy, best effort delivery service to a lossless delivery service. To achieve this, the IEEE is defining a suite of protocols under the umbrella of DCB. These protocols are :

  • IEEE 802.1Qau Congestion Notification (CN)

  • IEEE 802.1Qbb Priority-based Flow Control (PFC)

  • IEEE 802.1Qaz Enhanced Transmission Selection (ETS)

  • IEEE 802.1 DCBX Data Center Bridging Exchange Protocol (DCBX)

Ethernet currently has a flow control mechanism defined in IEEE 802.3x PAUSE, but PAUSE only works at the link layer and has the undesired effects of spreading congestion and causing head-of-line blocking for servers that have not contributed towards the congesting. In an effort to solve this complex problem, IEEE 802.1Qau has been tasked with introducing end-to-end flow control in CEE networks.

The basic premise of congestion notification (CN) is that whenever a switch detects congested at a port—referred to as the congestion point—the switch samples frames going through that port and issues a CN message which is reflected back to the source of the sampled frame. In this way, the sources—called reaction points—are informed of the congestion in the network. The sources then reduce the amount of traffic they put out in the network and would keep reducing their transmission rates as long as they keep receiving the CN messages from the network. The switches stop reflecting the packets once the congestion is gone and sources respond to the lack of CN messages by increasing their transmission rates.

Since CN is an end-to-end flow control mechanism, most of its benefits are not realized until the whole, or a majority of, data center infrastructure is consolidated on CEE. Even when the majority of the infrastructure is CEE-enabled, congestion hotspots can be avoided by the careful implementation of provisioning policies and best practices. Organizations must carefully evaluate the regular operational overhead of CN and weigh them against the known provisioning practices.

Priority-based Flow Control (PFC)

PFC is an enhancement to the current link-level flow control mechanism defined in IEEE 802.3X (PAUSE). Current Ethernet protocols support the capability to assign different priorities to different applications, but the existing standard PAUSE mechanism ignores the priority information in the packet. This design results in a shutdown of the entire link for all applications—even if a single application causes the congestion. For FCoE, LAN, and IPC applications to effectively share the same link, it is important to ensure that congestion caused by any one of these applications does not cause disruption to the rest of the applications’ traffic or to network control traffic.

IEEE 802.1Qbb is tasked with enhancing the existing PAUSE protocol to include the priority information in the PAUSE messages about the priorities contributing to congestion. Based on the priority information collected through PAUSE, the server stops sending any traffic for that specific application while the other applications continue to make progress without disruption on the shared link.

Enhanced Transmission Selection (ETS)

ETS attempts to manage the traffic priorities between multiple applications by regulating flow and by assigning preset amounts of link bandwidth and relative priority to each application IEEE 802.1Qbb is tasked with defining scheduling mechanisms for managing traffic priorities on a converged link.

Market Opportunities

10 GbE-based solutions are evolving to address not only traditional storage and LAN market needs, but also commercial IPC and latency-sensitive HPC cluster interconnect segments. With CEE and FCoE enhancements to the plate the road for true convergence is possible.

While a lot of focus around FCoE and CEE deployment is focused on server edge convergence for traditional storage and networking connectivity, native 10 GbE and CEE present unique opportunities in the clustering arena.

With a large segment of the clustering market very familiar with Gigabit Ethernet, the transition to 10GbE seems compelling, especially for those applications requiring better latency characteristics and bandwidth. This also paves for incremental upgrade and selected transitions with investment protection in current deployments.

The price point of the 10GbE switch port and server port makes it a compelling offering against InfiniBand solutions. Though the latency characteristics of InfiniBand are better than those of 10 GbE, the Ethernet management paradigm and total cost of ownership are attractive.

The FCoE Paradigm

FCoE is based on established technology; both Ethernet and Fibre Channel technology. FCoE also provides the coexistence with native Ethernet networks and Fibre Channel fabrics. The primary components of the CNA server software include Ethernet and Storage (Fibre Channel) drivers, both derived from mature software.

FCoE continues to provide the same management paradigms as its underlying technologies – similar to Fibre Channel for Storage and Ethernet for Networking. This design enables easy adoption of the technology in the data center for the IT administrators.

With the ability to co-exist with both Fibre Channel and Ethernet, FCoE is uniquely positioned to enable heterogeneous deployments. This avoids the needs to “rip and replace” the existing hardware to deploy a new solution. FCoE hardware can be incrementally added to the data center infrastructure without significant changes to the manageability software or existing hardware.

10GbE provides 10x bandwidth improvement over traditional Ethernet and 1.2x over 8Gb Fibre channel deployments. Moreover, Ethernet as an underlying technology provides a clear path to 40Gb and 100 Gb. By sharing the interface, HPC storage, LAN, and IPC traffic can benefit from the lower cost and better reliability offered by FCoE (i.e. less cables, cards, and switches to fail). A welcome convergence for everyone.

Comments on "Storage Convergence: Fibre Channel over Ethernet"

I just want to say I am just beginner to blogging and certainly enjoyed this blog site. Almost certainly I’m going to bookmark your website . You actually come with awesome well written articles. Cheers for revealing your website.

I do agree with all the ideas you’ve presented to your post.
They are very convincing and will definitely work.

Nonetheless, the posts are very brief for novices.

May just you please prolong them a little from next
time? Thanks for the post.
creditos rapidos con asnef

I do not even know how I ended up here, but I thought this
post was good. I do not know who you are but definitely
you’re going to a famous blogger if you aren’t already
;) Cheers!
prestamos con asnef

mulberry purse Do you have any video of that? I ad like to find out more details.

Pretty! This has been an extremely wonderful article. Thank you for supplying this info.

Thanks for the blog article.Much thanks again. Want more.

Rock ‘n Roll pioneer Dale Hawkins dead in Little Rock. Obviously, to earn all 48 stars, you’ll need to play every mission on veteran or run it as fast as possible.
Some have suggested that mobile phones are already as great a threat to privacy as RFID.

Also visit my web page … Instrumental Music

Heya i am for the first time here. I found this board and I find It truly
useful & it helped me out a lot. I hope to give something back and aid others
like you helped me.

website; Brook,

Hi, i think that i noticed you visited my web site
thus i got here to return the desire?.I am attempting to find issues to
improve my web site!I guess its adequate to use a few of your ideas!!

Asking questions are genuinely fastidious thing if you are
not understanding something completely, except this post gives fastidious understanding even.

This information is worth everyone’s attention. When can I find out more?

site – Carol

Hello there! This post could not be written any better! Reading
this post reminds me of my old room mate! He
always kept talking about this. I will forward this write-up to him.
Fairly certain he will have a good read. Thank you
for sharing!
webpage (Benjamin)

Hi! Do you use Twitter? I’d like to follow you
if that would be okay. I’m absolutely enjoying your blog and look forward to new updates.

What a information of un-ambiguity and preserveness of precious knowledge concerning
unexpected emotions.

Hey there! I’ve been reading your site for
some time now and finally got the bravery to go ahead
and give you a shout out from Kingwood Tx! Just wanted to say keep up the
great job!
web page (Nichol)

??????? ????? ?????-????? ???? ??? – ????? ???? ???? – ????? ????? ???? – ????? iran.sc-????? ???? ????-????? ??????

Im obliged for the blog article.Really thank you! Really Great.

I really liked your article post.Much thanks again. Keep writing.

Thanks again for the blog post. Want more.

A big thank you for your post.Thanks Again. Will read on…

One of our guests not too long ago advised the following website.

Here are a few of the web pages we advocate for our visitors.

Here are a number of the web sites we advise for our visitors.

The details mentioned within the write-up are several of the most effective out there.

Leave a Reply