Lippis Report 155: The Two-Tier High-End Data Center Ethernet Fabric Network Gains Steam
It hasn’t been since the mid 1990s that the networking industry was focused on multi-protocol integration or convergence. But the industry is gearing up for a major innovation and competitive cycle fueled by the multi-billion dollar addressable market for data center network fabrics. Over the last eighteen months, every major Ethernet infrastructure provider has been talking about two and three tier network fabrics for high-end data centers.
Companies such as Cisco, Arista Networks, HP/3Com, Force10, Voltaire, Extreme, Brocade, Juniper et al have announced network fabrics for data centers with five thousand and more servers with and without storage enablement. Juniper talks of a one-tier fabric through their Project Stratus work with IBM to be available some time in the future. Brocade recently introduced its Brocade One, which is a converged data center fabric. Extreme Networks launched its DirectAttachTM that eliminates virtual plus blade switch layers. HP has FlexFabric, a virtualized fabric for the data center. Cisco launched its FabricPath Switching System or FSS for the Nexus 7000 that enables massive scale of a two-tier fabric.
In this Lippis Report Research Note, we review the architectural attributes of two tier network fabrics.
BLADE Unified FabricArchitecture Delivers Economic & Data Center Network Design Advantages
The IT industry is at an inflection point as service delivery is becoming more and more centralized thanks to data center consolidation, virtualization, cloud and mobile computing. It is estimated that a third of all IT spend is concentrated in the data center, and this trend is only building thanks to favorable economics, motivating IT business leaders to centralize IT delivery.
The impact of this trend is more and more dense data centers made up of servers in the thousands to tens of thousands and higher. It is at the scale of 5,000 plus servers that a new network fabric is required for high-end data centers. High-end data center design is challenged with increasing complexity, the need for greater workload mobility and reduced energy consumption. Traffic patterns have also shifted significantly, from primarily client-server or as commonly referred to as north-to-south flows, to a combination of client-server and server-server or east-to-west plus north-to-south streams. These shifts have wreaked havoc on application response time and end user experience, since the network is not designed for these Brownian motion type flows.
Cisco Launches FabricPath Switching System For Scalable Data Center Ethernet Fabrics
The main requirements for high-end data center network fabric are low latency, large flat layer 2 domains to enable workload mobility, low power consumption, simplicity of design and significant bandwidth. Storage enablement, meaning consolidated I/O or virtualized I/O, is a growing priority and a new fabric that can support FiberChannel over Ethernet, iSCSI over Ethernet, iWARP over Ethernet or Infiniband over Ethernet, is a major plus. One salient observation is that it’s pretty clear that Ethernet is the network fabric of choice, as it is the only network protocol that enjoys continual innovation such as TRILL, Data Center Bridging, IEEE’s 802.1AQ, link aggregation, multi-pathing, and as recently ratified by the IEEE 40 Gbs and 100 Gbs speeds.
With the above requirements in mind, let us review data center network design options.
Three Tier Data Center Fabric
A three-tier network architecture is the dominant structure in data centers today and will likely continue as the optimal design for many networks. For most network architects and administrators, this type of design provides the best balance of asset utilization, layer 3 routing for segmentation, scaling and services, plus efficient physical design for cabling and fiber runs. By three tiers we mean, access switches/Top-of-Rack (ToR) switches, or modular/End-of-Row (EoR) switches that connect to servers and IP based storage. These access switches are connected via Ethernet to aggregation switches. The aggregation switches are connected into a set of core switches or routers that forward traffic flows from servers to an intranet and internet, and between the aggregation switches. It’s common in this structure to over-subscribe bandwidth in the access tier, and to a lesser degree, in the aggregation tier, which can increase latency and reduce performance. Inherent in this structure is the placement of layer 2 versus layer 3 forwarding that is Virtual Local Area Networking or VLANs and IP routing. Also common, is that VLANs are constructed within access and aggregation switches, while layer 3 capabilities in the aggregation or core switches route between them.
How Direct Attach from Extreme Networks Reduces Tiers in the Data Center
But within the high-end data center market, where the number of servers is in the thousands to tens of thousands plus and where north-south plus east-west traffic is significant, is where a new structure is needed. It is within these data centers where applications need a single layer 2 domain.
Two-tiers of network fabric
A two-tier fabric is designed with two kinds of switches: one that connects servers, and the second that connect switches creating a non-blocking, low latency fabric. In short, there are server facing and fabric facing switches. We use the terms ‘leaf’ switch to denote server facing or connecting switches and ‘spine’ to denote fabric facing or switches that connect leaf switches into the fabric. Together, leaf and spine switches create the fabric.
Cisco Threat Defense for Borderless Networks
Many IT leaders in Global 2000 firms will have deployed both two and three tier network structure, as different deployment models are used for different applications. For these leaders, a network equipment supplier that possesses product architecture flexibility, meaning an end-to-end product solution that accommodates tier two and three fabrics would be advantageous. This flexibility is found in product that supports layer 2 and layer 3 forwarding, as well as, a variety of line cards to offer design options.
A common network Operating System (OS) of products configured for two and three tier structure is important as IT operations gain efficiency to manage fabrics, as configuration and management are consistent. In addition, a common network OS offers rapid absorption of innovation to IT operations, as new OS features are available at the same time to all fabrics. The benefit of using a common product set to build tier two or three fabrics offers value around operational efficiency, training, sparing and ease of evolution between fabric deployments. In short, the network fabric needs to be simple and general purpose versus purpose built, which a common set of products creating tier two or three fabrics offer.
HP FlexFabric Virtualize network connections and capacity From the edge to the core An HP Converged Infrastructure innovation primer
A Unified/Converged Fabric
The concept of a unified fabric is to virtualize data center resources and connect them through a high bandwidth network that is very scalable, high performance and enables the convergence of multiple protocols onto a single physical network. These IT resources are compute, storage and applications, which are connected via a network fabric. In short, the network is the unified fabric and the network is Ethernet.
The industry tends to focus on storage transport over Ethernet as the main concept behind a unified/converged fabric with technologies such as Fiber Channel over Ethernet or FCoE, iSCSI over Ethernet, iWARP over Ethernet and even Infiniband over Ethernet. But this is a narrow view of a unified/converged fabric which is being expanded, thanks to continual innovation of Ethernet by the vendor community and standards organizations such as the IEEE and IETF.
Improved Network Security with IP and DNS Reputation
Ethernet innovations such as FCoE, Data Center Bridging or DCB, IETF’s Transparent Interconnection of Lots of Links or TRILL, CEE or Converged Enhanced Ethernet, link aggregation, IEEE’s 802.1AQ have enhanced Ethernet networking to support a wide range of new data center fabric design options. In addition to these protocol enhancements, the IEEE has ratified its work on defining 40Gb and 100Gb Ethernet, significantly increasing Ethernet’s ability to scale bandwidth. To demonstrate how Ethernet is evolving to be the unified fabric for high-end data centers, we explore Cisco’s new FabricPath Switching System innovation in this white paper.
The decision to implement a two or three tier network structure comes down to scale. For high-end data centers, a two-tier structure meets the requirements of low latency, movable workloads, scale, simplicity, etc. Many global 2000 concerns will have deployed both a two and three tier network fabric for their high end and less dense data centers.
When shopping for network equipment to construct two and three tier network fabrics, look for suppliers that support both rich Layer 3 routing services and scalable Layer 2 Ethernet capabilities to ensure choice and flexibility of three tier and scalable two tier fabric implementations. Such suppliers offer products that can be configured in multiple use cases and topologies where modules are inter-changeable, skills transferable and operations common between both fabric approaches.
But make no mistake about it, it’s a two-tier network fabric that IT business leaders and data center architects have gravitated toward for high performance computing, cloud scale data centers and just plain high end data centers of 5,000 and above servers.