The ever-increasing demand for storage, network and compute resources for instant access of data and computational requirements lead to the continued growth of the variety of networks found in the data center. HPC is no exception to this trend.
DCB / CEE Introduction
An FCoE frame needs to be transported with the same lossless characteristics with which Fibre Channel is designed. To allow for this transport, the fundamental nature of Ethernet needs to change from a lossy, best effort delivery service to a lossless delivery service. To achieve this, the IEEE is defining a suite of protocols under the umbrella of DCB. These protocols are :
- IEEE 802.1Qau Congestion Notification (CN)
- IEEE 802.1Qbb Priority-based Flow Control (PFC)
- IEEE 802.1Qaz Enhanced Transmission Selection (ETS)
- IEEE 802.1 DCBX Data Center Bridging Exchange Protocol (DCBX)
Ethernet currently has a flow control mechanism defined in IEEE 802.3x PAUSE, but PAUSE only works at the link layer and has the undesired effects of spreading congestion and causing head-of-line blocking for servers that have not contributed towards the congesting. In an effort to solve this complex problem, IEEE 802.1Qau has been tasked with introducing end-to-end flow control in CEE networks.
The basic premise of congestion notification (CN) is that whenever a switch detects congested at a port—referred to as the congestion point—the switch samples frames going through that port and issues a CN message which is reflected back to the source of the sampled frame. In this way, the sources—called reaction points—are informed of the congestion in the network. The sources then reduce the amount of traffic they put out in the network and would keep reducing their transmission rates as long as they keep receiving the CN messages from the network. The switches stop reflecting the packets once the congestion is gone and sources respond to the lack of CN messages by increasing their transmission rates.
Since CN is an end-to-end flow control mechanism, most of its benefits are not realized until the whole, or a majority of, data center infrastructure is consolidated on CEE. Even when the majority of the infrastructure is CEE-enabled, congestion hotspots can be avoided by the careful implementation of provisioning policies and best practices. Organizations must carefully evaluate the regular operational overhead of CN and weigh them against the known provisioning practices.
Priority-based Flow Control (PFC)
PFC is an enhancement to the current link-level flow control mechanism defined in IEEE 802.3X (PAUSE). Current Ethernet protocols support the capability to assign different priorities to different applications, but the existing standard PAUSE mechanism ignores the priority information in the packet. This design results in a shutdown of the entire link for all applications—even if a single application causes the congestion. For FCoE, LAN, and IPC applications to effectively share the same link, it is important to ensure that congestion caused by any one of these applications does not cause disruption to the rest of the applications’ traffic or to network control traffic.
IEEE 802.1Qbb is tasked with enhancing the existing PAUSE protocol to include the priority information in the PAUSE messages about the priorities contributing to congestion. Based on the priority information collected through PAUSE, the server stops sending any traffic for that specific application while the other applications continue to make progress without disruption on the shared link.
Enhanced Transmission Selection (ETS)
ETS attempts to manage the traffic priorities between multiple applications by regulating flow and by assigning preset amounts of link bandwidth and relative priority to each application IEEE 802.1Qbb is tasked with defining scheduling mechanisms for managing traffic priorities on a converged link.
Market Opportunities
10 GbE-based solutions are evolving to address not only traditional storage and LAN market needs, but also commercial IPC and latency-sensitive HPC cluster interconnect segments. With CEE and FCoE enhancements to the plate the road for true convergence is possible.
While a lot of focus around FCoE and CEE deployment is focused on server edge convergence for traditional storage and networking connectivity, native 10 GbE and CEE present unique opportunities in the clustering arena.
With a large segment of the clustering market very familiar with Gigabit Ethernet, the transition to 10GbE seems compelling, especially for those applications requiring better latency characteristics and bandwidth. This also paves for incremental upgrade and selected transitions with investment protection in current deployments.
The price point of the 10GbE switch port and server port makes it a compelling offering against InfiniBand solutions. Though the latency characteristics of InfiniBand are better than those of 10 GbE, the Ethernet management paradigm and total cost of ownership are attractive.
The FCoE Paradigm
FCoE is based on established technology; both Ethernet and Fibre Channel technology. FCoE also provides the coexistence with native Ethernet networks and Fibre Channel fabrics. The primary components of the CNA server software include Ethernet and Storage (Fibre Channel) drivers, both derived from mature software.
FCoE continues to provide the same management paradigms as its underlying technologies – similar to Fibre Channel for Storage and Ethernet for Networking. This design enables easy adoption of the technology in the data center for the IT administrators.
With the ability to co-exist with both Fibre Channel and Ethernet, FCoE is uniquely positioned to enable heterogeneous deployments. This avoids the needs to “rip and replace” the existing hardware to deploy a new solution. FCoE hardware can be incrementally added to the data center infrastructure without significant changes to the manageability software or existing hardware.
10GbE provides 10x bandwidth improvement over traditional Ethernet and 1.2x over 8Gb Fibre channel deployments. Moreover, Ethernet as an underlying technology provides a clear path to 40Gb and 100 Gb. By sharing the interface, HPC storage, LAN, and IPC traffic can benefit from the lower cost and better reliability offered by FCoE (i.e. less cables, cards, and switches to fail). A welcome convergence for everyone.
Comments on "Storage Convergence: Fibre Channel over Ethernet"
Beware the vendor of a legacy technology which starts spouting catch phrases such as “based on established technology”, “derived from mature software”, and “uniquely positioned to enable heterogeneous deployments”. This is marketing blurb and needs to be recognized as such.
With 10GbE you can roll out iSCSI/TCP/IP now and have convergence without buying funky unproven switching technology. Alternatively if your company is heading down the Infiniband path then consider iSCSI/iSER/IB as your storage protocol (or SRP/IB if you prefer).
One of the Fibrechannel requirements is a reliable and predicable physical network. This was the whole reason to invent FC in the first place. Does ethernet technology (when augmented by the NEW features in the article) now provide this? What happens when limits are reached in these new switches which must deployed in order to allow the FCoE traffic to flow? How does this “mature software” react to the different characteristics of the ethernet fabric? How is your company’s financial officer going to react when you tell her these bleeding edge switches must be purchased (at what cost?) rather than submitting a request for more of the same switches that were purchased last time?
Another protocol, ATAoE, was very interesting. Simple enough to be implemented in hardware, it was an inventive solution that leveraged the strengths of every technology it involved. FCoE, in comparison, seems like it was levered (eg, with a prybar) to fit the involved technologies. What’s next, IB over WiFi? Really, I’m at a loss to understand what problem this is destined to solve.
If you have worked in the storage arena before you’d kow there are many challenges to be overcome when building out a new storage environment. Many of today’s protocols have limitations such as security and reliability in iSCSI. If you are running a mission critical data base you can’t afford to lose a packet due to congestion or latency in your switch fabric. Fiber channel prevents these problems from disturbing data flow but a significant cost in both hardware and expertise. As an installer of SAN equipment I can attest to the difficulty people have in configuring FC fabrics.
On the other had, when customers build iSCSI storage systems they are often too casual with the deployments and give up a significant level of performance and reliability by using generic ethernet adapters and shared switches that are usually busy doing other tasks.
With the advent of the new FcoE and the new CCE protocols we can reap the benefits of using a well understood infrastructure like ethernet while removing some of the limitations of existing protocols like iSCSI.
Quote: “… limitations such as security and reliability in iSCSI”.
Do you intend to support this ambit claim with specific examples? Are you concerned about the protocol or certain implementations?
Quote: “… I can attest to the difficulty people have in configuring FC fabrics.”
How does FCoE address these difficulties?
Quote: “… often too casual with the deployments …”
Why are customers going to be any less casual with deploying FCoE?
Quote: “If you are running a mission critical data base you can’t afford to lose a packet due to congestion or latency in your switch fabric”.
That’s plain fear-mongering. Packet loss (which happens significantly less frequently than some people try to suggest) is handled by different layers in different protocols. The reality is that unless there is catastrophic system failure the data transfer is delayed (either queued or retransmitted). Customers pay large money for HA solutions to protect data and to ensure continuity of services in the case of system failure.
Quote: “well understood infrastructure like ethernet”
Now your argument implies that ethernet is good enough to transport mission critical data. ;-) But it’s not ethernet; it’s Converged Enhanced Ethernet. It’s a new hack to an existing protocol layer which requires new network hardware to support it. How will non-CEE switches handle the accidental presence of the new (hence unrecognised) frame type? What will customers have to learn to configure this new infrastructure? How much will they have to pay?