The ever-increasing demand for storage, network and compute resources for instant access of data and computational requirements lead to the continued growth of the variety of networks found in the data center. HPC is no exception to this trend.
Data centers can be found in multiple markets spanning multiple industries including, financial, government and education, manufacturing, automotive, health care, bio-technology, and high-tech. These data centers host multiple compute, storage, communication and infrastructural equipment for digital storage/retrieval, communications and management of information. The ever-increasing demand for storage, network and compute resources for instant access of data and computational requirements thereof, has in turn lead to the continued growth of the variety of networks found in the data center. HPC is no exception to this trend. Indeed, HPC often pushes the envelope for storage technologies.
The resulting network proliferation has increased IT costs through the need for multiple adapters that connect the servers found at the edge of the network through separate cables to the various networks. As networks proliferate demand is satisfied through the deployment of more servers and adapters. The growth in the number of servers and adapters directly lead to increased power and cooling demands placed on the data center. One of the largest consumers of data center costs is power and cooling, which for many data centers over the life of the equipment, can exceed the capital equipment expenditures.
The three most common types of networks found in the data center are:
- Storage Area Networks (SAN)
- Local Area Networks (LAN)
- Inter-processor Communication (IPC) Networks
As shown in Figure One, a single server may have some or all of these three networks. v
Figure One: Common Server Interconnects
Each network has evolved to solve a specific requirement delivering specific characteristics for its unique traffic type. The SAN network uses Fibre Channel technology to provide a guaranteed in-order, lossless transport for data to be sent to and from storage devices. The LAN network provides the traditional TCP/IP based Ethernet network for best effort data communications. The IPC network is typically used for High Performance Computing (HPC) clustered environments, where multiple servers communicate with each other using low latency messaging.
One of the biggest challenges for the IT administrator is how to reduce the data center costs while maintaining the Service Level Agreements (SLA) and performance that is expected of the network. Many technologies that have attempted to solve this problem have either been relegated to niche markets or have been found inadequate to solve the problem and thus have been discarded.
A promising solution is emerging to solve this problem. The various standards bodies — IEEE, ANSI and IETF — are collectively defining in the realms of their respective domains pieces of this solution that is expected to gain industry-wide adoption. A protocol and frame format called Fibre Channel over Ethernet (FCoE) is being defined by the ANSI T11. FCoE would need to be transported over an improved or enhanced Ethernet transport referred to as Converged Enhanced Ethernet (CEE). The Data Center Bridging (DCB) task force in the IEEE is working on a number of specifications for this protocol.
With this new protocol, comes a new class of adapter called a Converged Network Adapter (CNA). The basic premise of the CNA is to allow multiple networks to connect to a server over the same physical cable using a single adapter. A single physical connection from the CNA that can be used to transport SAN, LAN and IPC traffic can provide a lower Total Cost of Ownership through a single network infrastructure from the server edge to the access layer using common hardware and software management platforms. To achieve a converged I/O medium a widely adopted transport protocol is required.
Fibre Channel over Ethernet Introduction
Fibre Channel over Ethernet (FCoE) in essence is a mechanism to transport Fibre Channel over a DCB-enabled Ethernet infrastructure. A DCB-enabled Ethernet network differs from Ethernet networks widely deployed today for transporting TCP/IP. It avoids the complications of prior technologies which were designed to collapse infrastructures and be the ubiquitous network technology in that it does not require “fork-lifting” the network.
To transport an FCoE frame, a complete Fibre Channel frame is built in the same way as if it was being built for a traditional Fibre Channel network. FCoE layer encapsulates a fully-formed Fibre Channel frame and inserts a defined frame header before it is sent on the CEE link. Figure Two and Figure Three illustrate the FCoE Protocol Stack and the FCoE Frame Format respectively.
Figure Two: FCoE Protocol Stack. FC-2, FC-3, FC-4 are various Fibre Channel services.
FCoE encodes the Start of Frame (SOF) and includes it in the FCoE header prepended to the standard Fibre Channel frame. It then encodes End of Frame (EOF) at the end of the Fibre Channel frame before the frame is handed over to the CEE layer. A header is added and a Cyclic Redundancy Check (CRC) is completed before transmitting the unmodified Fibre Channel frame as data. The type of packet, FCoE or TCP/IP is defined in the type field.
Figure Three: FCoE Frame Format
As with any new protocol, an FCoE switch is required to expose the CEE and Fibre Channel ports, making forwarding decisions based on the Fibre Channel headers inside the CEE frame. It is similar to how a basic Fibre Channel switch would switch Fibre Channel frames. The only difference is that an FCoE switch supports communication between any ports on the switch, Fibre Channel or CEE. In addition an FCoE switch is also capable of receiving and processing basic TCP/IP traffic on the CEE ports since it is connected to a hostâ€”capable of acting as a standard IEEE 802.1Q bridge between all CEE ports in this Layer 2 mode. This operation is very similar to how a multi-layer Internet Protocol (IP) switch performs basic Ethernet switching at Layer 2 (L2) as well as IP routing at Layer 3 between its ports. An FCoE switch treats Fibre Channel traffic as a Layer 3 network protocol, and no LAN traffic is forwarded to Fibre Channel ports in the L2 mode.
An FCoE switch can be connected to other FCoE switches via the CEE ports, and it also can be connected to any existing Fibre Channel switch via the Fibre Channel ports.