Lippis Report 151: A Two or Three Tier High-End Data Center Ethernet Fabric Architecture?

nicklippis.jpgIt hasn’t been since the mid 1990s that the networking industry was focused on multi-protocol integration or convergence. The industry is gearing up for a major innovation and competitive cycle fueled by the multi-billion dollar addressable market for data center network fabrics. Over the last eighteen months, every major Ethernet infrastructure provider has been talking about two and three tier network fabrics for high-end data centers. Companies such as Cisco, Arista Networks, HP/3Com, Force10, Voltaire, Extreme, Brocade, Juniper et al have announced network fabrics for data centers with five thousand and more servers with and without storage enablement. Juniper talks of a one-tier fabric through their Project Stratus work with IBM to be available some time in the future. Brocade recently introduced its’ Brocade One, which is a converged data center fabric. Cisco just launched its’ FabricPath Switching System or FSS for the Nexus 7000 that enables massive scale of a two-tier fabric. In this Lippis Report Research Note, we review the architectural attributes of two and three tier network fabrics and review FSS and its accompanying F-Series 10GbE module.

The IT industry is at an inflection point as service delivery is becoming more and more centralized thanks to data center consolidation, virtualization, cloud and mobile computing. It is estimated that a third of all IT spend is concentrated in the data center and this trend is only building thanks to favorable economics, motivating IT business leaders to centralize IT delivery.

The impact of this trend is more and more dense data centers made up of servers in the thousands to tens of thousands and higher. It is at the scale of 5,000 plus servers that a new network fabric is required for high-end data centers. High-end data center design is challenged with increasing complexity, the need for greater workload mobility and reduced energy consumption. Traffic patterns have also shifted significantly, from primarily client-server or as commonly referred to as north-to-south flows, to a combination of client-server and server-server or east-to-west plus north-to-south streams. These shifts have wreaked havoc on application response time and end user experience, since the network is not designed for these Brownian motion type flows.

Cisco Launches FabricPath Switching System For Scalable Data Center Ethernet Fabrics

Listen to the Podcast

The main requirements for high-end data center network fabric are low latency, large flat layer 2 domains to enable workload mobility, low power consumption, simplicity of design and significant bandwidth. Storage enablement, meaning consolidated I/O or virtualized I/O, is a growing priority and a new fabric that can support FiberChannel over Ethernet, iSCSI over Ethernet, iWARP over Ethernet or Infiniband over Ethernet is a major plus. One salient observation is that it’s pretty clear that Ethernet is the network fabric of choice as it is the only network protocol that enjoys continual innovation such as TRILL, Data Center Bridging, link aggregation, multi-pathing, and soon, 40 Gbs and 100 Gbs speeds. With the above requirements in mind, let us review data center network design options.

A Modern Approach To FAX Management Via Unified Communications

Listen to the Podcast

Two and Three Tier Fabrics

A three-tier network architecture is the dominant structure in data centers today and will likely continue as the optimal design for many networks. For most network architects and administrators, this type of design provides the best balance of asset utilization, layer 3 routing for segmentation, scaling and services, plus efficient physical design for cabling and fiber runs. By three tiers, we mean access switches/Top-of-Rack (ToR) switches, or modular/End-of-Row (EoR) switches that connect to servers and IP based storage. These access switches are connected via Ethernet to aggregation switches. The aggregation switches are connected into a set of core switches or routers that forward traffic flows from servers to an intranet and internet, and between the aggregation switches. It’s common in this structure to over-subscribe bandwidth in the access tier, and to a lesser degree, in the aggregation tier, which can increase latency and reduce performance. Inherent in this structure is the placement of layer 2 versus layer 3 forwarding that is Virtual Local Area Networking or VLANs and IP routing. Also common, is that VLANs are constructed within access and aggregation switches, while layer 3 capabilities in the aggregation or core switches route between them.

But within the high-end data center market, where the number of servers is in the thousands to tens of thousands plus and east-west bandwidth is significant, is where a new structure is needed. It is within these data centers where applications need a single layer 2 domain.

Making Networks More Agile With Force10’s Open Automation

Listen to the Podcast

Two-tiers of network fabric

A two-tier fabric is designed with two kinds of switches: one that connects servers, and the second that connect switches creating a non-blocking, low latency fabric. In short, there are server facing and fabric facing switches. We use the terms ‘leaf’ switch to denote server facing or connecting switches and ‘spine’ to denote fabric facing or switches that connect leaf switches into the fabric. Together, a leaf and spine architecture create the fabric.

Many IT leaders in Global 2000 firms will have deployed both two and three tier network structure, as different deployment models are used for different applications. For these leaders, a network equipment supplier is needed that possesses product architecture flexibility, meaning an end-to-end product solution that accommodates tier two and three fabrics. This flexibility is found in product that supports layer 2 and layer 3 forwarding, as well as, a variety of line cards to offer design options.

A Simpler Data Center Fabric Emerges
For The Age of Massively Scalable Data Centers

Get the White Paper

A common network Operating System (OS) of products configured for two and three tier structure is important as IT operations gain efficiency to manage fabrics, as configuration and management are consistent. In addition, a common network OS offers rapid absorption of innovation to IT operations, as new OS features are available at the same time to all fabrics. The benefit of using a common product set to build tier two or three fabrics offers value around operational efficiency, training, sparing and ease of evolution between fabric deployments. In short, the network fabric needs to be simple and general purpose versus purpose built, which a common set of products creating tier two or three fabrics offer. This type of flexibility will enable IT leaders to address the challenges of scale outlined above.

In addition to product flexibility, some networking suppliers take a systems approach to their fabric design, meaning that a solution is built and pre-tested before it arrives on site. This ensures that IT does not have to perform system integration. With the increased concentration of computing and IT dollars into data centers, it’s only obvious that data centers are long-term corporate commitments. Therefore, it is only appropriate that the networking supplier of choice also has a proven long-term commitment to their product architecture.

Perhaps the best example of this is Cisco’s Catalyst 6000 switching architecture and its’ two-year-old Nexus product line. The Catalyst investment protection is well documented as it has been in operation for over a decade, which Cisco customers enjoy continued innovation and value added to this platform. Competitors view its’ longevity as a weakness. The Nexus product line has a similar investment protection philosophy with a fifteen-year plus lifespan expectation. Common to both Catalyst and Nexus is the fact that these products are built on silicon, developed at Cisco, affording investment protection from one generation of the hardware to the next.

Leading the Way to Borderless Networks

Get the White Paper

A Unified Fabric

The concept of a unified fabric is to virtualize data center resources and connect them through a high bandwidth network that is very scalable, high performance and enables the convergence of multiple protocols onto a single physical network. These IT resources are compute, storage and applications, which are connected via a network fabric. In short, the network is the unified fabric and the network is Ethernet.

The industry tends to focus on storage transport over Ethernet as the main concept behind a unified fabric with technologies such as Fiber Channel over Ethernet or FCoE, iSCSI over Ethernet, iWARP over Ethernet and even Infiniband over Ethernet. But this is a narrow view of a unified fabric, which is being expanded thanks to continual innovation of Ethernet by the vendor community and standards organizations such as the IEEE and IETF. Ethernet innovations such as FCoE, Data Center Bridging or DCB, link aggregation, Cisco’s VN-Link, FEX-Link and virtual PortChannel or vPC have enhanced Ethernet networking to support a wide range of new data center fabric design options. In addition to these protocol enhancements, the IEEE is scheduled to complete its’ work on defining 40Gb and 100Gb Ethernet during the summer of 2010, significantly increasing Ethernet’s ability to scale bandwidth. To demonstrate how Ethernet is evolving to be the unified fabric for high-end data centers, we explore Cisco’s new FabricPath Switching System innovation in this white paper.

Building Mission-Critical Data Center

Get the White Paper

The decision to implement a two or three tier network structure comes down to scale. For high-end data centers, a two-tier structure meets the requirements of low latency, movable workloads, scale, simplicity, etc. Many global 2000 concerns will have deployed both a two and three tier network fabric for their high end and less dense data centers.

When shopping for network equipment to construct two and three tier network fabrics, look for suppliers that support both rich Layer 3 routing services and scalable Layer 2 Ethernet capabilities to ensure choice and flexibility of three tier and scalable two tier fabric implementations. Such suppliers offer products that can be configured in multiple use cases and topologies where modules are inter-changeable, skills transferable and operations common between both fabric approaches.

2 Debates over Lippis Report 151: A Two or Three Tier High-End Data Center Ethernet Fabric Architecture?

  1. Nick Lippis said:

    Lippis Report 151: A Two or Three Tier High-End Data Center Ethernet Fabric Architecture?

  2. stu said:

    fyi – 40Gb & 100Gb Ethernet standards have been ratified