The ever-increasing demand for storage, network and compute resources for instant access of data and computational requirements lead to the continued growth of the variety of networks found in the data center. HPC is no exception to this trend.
DCB / CEE Introduction
An FCoE frame needs to be transported with the same lossless characteristics with which Fibre Channel is designed. To allow for this transport, the fundamental nature of Ethernet needs to change from a lossy, best effort delivery service to a lossless delivery service. To achieve this, the IEEE is defining a suite of protocols under the umbrella of DCB. These protocols are :
- IEEE 802.1Qau Congestion Notification (CN)
- IEEE 802.1Qbb Priority-based Flow Control (PFC)
- IEEE 802.1Qaz Enhanced Transmission Selection (ETS)
- IEEE 802.1 DCBX Data Center Bridging Exchange Protocol (DCBX)
Ethernet currently has a flow control mechanism defined in IEEE 802.3x PAUSE, but PAUSE only works at the link layer and has the undesired effects of spreading congestion and causing head-of-line blocking for servers that have not contributed towards the congesting. In an effort to solve this complex problem, IEEE 802.1Qau has been tasked with introducing end-to-end flow control in CEE networks.
The basic premise of congestion notification (CN) is that whenever a switch detects congested at a portâ€”referred to as the congestion pointâ€”the switch samples frames going through that port and issues a CN message which is reflected back to the source of the sampled frame. In this way, the sourcesâ€”called reaction pointsâ€”are informed of the congestion in the network. The sources then reduce the amount of traffic they put out in the network and would keep reducing their transmission rates as long as they keep receiving the CN messages from the network. The switches stop reflecting the packets once the congestion is gone and sources respond to the lack of CN messages by increasing their transmission rates.
Since CN is an end-to-end flow control mechanism, most of its benefits are not realized until the whole, or a majority of, data center infrastructure is consolidated on CEE. Even when the majority of the infrastructure is CEE-enabled, congestion hotspots can be avoided by the careful implementation of provisioning policies and best practices. Organizations must carefully evaluate the regular operational overhead of CN and weigh them against the known provisioning practices.
Priority-based Flow Control (PFC)
PFC is an enhancement to the current link-level flow control mechanism defined in IEEE 802.3X (PAUSE). Current Ethernet protocols support the capability to assign different priorities to different applications, but the existing standard PAUSE mechanism ignores the priority information in the packet. This design results in a shutdown of the entire link for all applicationsâ€”even if a single application causes the congestion. For FCoE, LAN, and IPC applications to effectively share the same link, it is important to ensure that congestion caused by any one of these applications does not cause disruption to the rest of the applicationsâ€™ traffic or to network control traffic.
IEEE 802.1Qbb is tasked with enhancing the existing PAUSE protocol to include the priority information in the PAUSE messages about the priorities contributing to congestion. Based on the priority information collected through PAUSE, the server stops sending any traffic for that specific application while the other applications continue to make progress without disruption on the shared link.
Enhanced Transmission Selection (ETS)
ETS attempts to manage the traffic priorities between multiple applications by regulating flow and by assigning preset amounts of link bandwidth and relative priority to each application IEEE 802.1Qbb is tasked with defining scheduling mechanisms for managing traffic priorities on a converged link.
10 GbE-based solutions are evolving to address not only traditional storage and LAN market needs, but also commercial IPC and latency-sensitive HPC cluster interconnect segments. With CEE and FCoE enhancements to the plate the road for true convergence is possible.
While a lot of focus around FCoE and CEE deployment is focused on server edge convergence for traditional storage and networking connectivity, native 10 GbE and CEE present unique opportunities in the clustering arena.
With a large segment of the clustering market very familiar with Gigabit Ethernet, the transition to 10GbE seems compelling, especially for those applications requiring better latency characteristics and bandwidth. This also paves for incremental upgrade and selected transitions with investment protection in current deployments.
The price point of the 10GbE switch port and server port makes it a compelling offering against InfiniBand solutions. Though the latency characteristics of InfiniBand are better than those of 10 GbE, the Ethernet management paradigm and total cost of ownership are attractive.
The FCoE Paradigm
FCoE is based on established technology; both Ethernet and Fibre Channel technology. FCoE also provides the coexistence with native Ethernet networks and Fibre Channel fabrics. The primary components of the CNA server software include Ethernet and Storage (Fibre Channel) drivers, both derived from mature software.
FCoE continues to provide the same management paradigms as its underlying technologies â€“ similar to Fibre Channel for Storage and Ethernet for Networking. This design enables easy adoption of the technology in the data center for the IT administrators.
With the ability to co-exist with both Fibre Channel and Ethernet, FCoE is uniquely positioned to enable heterogeneous deployments. This avoids the needs to “rip and replace” the existing hardware to deploy a new solution. FCoE hardware can be incrementally added to the data center infrastructure without significant changes to the manageability software or existing hardware.
10GbE provides 10x bandwidth improvement over traditional Ethernet and 1.2x over 8Gb Fibre channel deployments. Moreover, Ethernet as an underlying technology provides a clear path to 40Gb and 100 Gb. By sharing the interface, HPC storage, LAN, and IPC traffic can benefit from the lower cost and better reliability offered by FCoE (i.e. less cables, cards, and switches to fail). A welcome convergence for everyone.