Lippis Report 121: A New Approach to Network Design When You’re In The Cloud

Nick LippisIn Lippis Report 120 we discussed how cloud computing is driving new networking requirements for both public and private cloud implementations. We focused on ethernet switch devices in that Research Note. But cloud computing may also require a new network design paradigm as well. The three-tier network model of edge or access, then aggregation or distribution and core have been the building blocks of modern computing networking for the past two decades and are still fundamental to classical enterprise network design. But in high performance data centers and in particular cloud computing a new two-tier model is being considered.

Lippis and Kerravala on Cisco’s Unified Computing, Nortel’s bankruptcy and more

Listen to the Podcast

Three-Tier Enterprise Network Design Stays Firm

The three-tier network model is based upon desktops, printers, servers and LAN-equipped devices being connected to access switches. These access switches are aggregated into “aggregation or distribution switches” to manage flows and building wiring. Aggregation switches connect to core routers/switches, which provide routing between aggregation switches and connectivity to wide area network services. It was the classical “dumbbell” model of big LAN and small WAN bandwidth that brought about the modern three-tier system as routing provides congestion management, traffic segmentation and a wide range of physical interfaces to the WAN.

Reduce Power Consumption through Integrated Services Delivery

Get the White Paper

Routing or layer 3 versus switching/bridging or layer 2 boundary placement decisions have ebbed and flowed with corporate IT network architects traditionally choosing layer 3 at the core and layer 2 mostly everywhere else. With increased traffic between aggregation switches many network architects place the layer 2/layer 3 boundary there to allow for inter-VLAN routing, client to server routing, stateful service integration and to avoid traffic flowing back and forth between core and aggregation. This is common for organizations with matrix traffic flows versus desktop-to-data center dominated flows. In building networks I expect the layer 2/layer 3 boundary to be increasingly at the aggregation tier to manage a huge increase in video, Web 2.0 streams, collaboration and corporate social networking mixed traffic streams.

Thriving in an Economic Downturn

Get the White Paper

A Transition To A New Set of Network Attached Devices

The access layer, or as some prefer to call it, the edge, is slowly disappearing in classical enterprise network design as the campus is transitioning to a different set of network attached devices from desktop to mobile personal utility accessories. This is most evident as wireless LAN connections outpace that of wired. But the need for PoE for iPhones, video surveillance cameras, WLAN access points, etc., will keep the access layer in place for a long time, regulating its pace of change. It’s not until fixed phones are replaced with virtualized soft UC phones and the majority of end-point connections are wireless that the access layer may be subsumed into a WLAN infrastructure. PoE support in the end will keep the access layer viable for years to come.

SMB Communications Pain Study: Uncovering the hidden cost of communications barriers and latency

Get the White Paper

For the vast majority of corporations the network core’s role of managing the large amount of LAN bandwidth competing to access the small amounts of WAN bandwidth will not change. Clearly layer 3 services are needed to define a logical layer, which provides and enforces forwarding policies, security, routing/forwarding, traffic aggregation, traffic segmentation, management, accounting, etc. But as service providers increasingly offer metro ethernet and other high-speed WAN services at 100Mbs to 10 GbE levels the LAN/WAN boundary will migrate to LAN interfaces for WAN access, but this will take some time; perhaps a decade or so as leased lines are the dominant form of router WAN connections today. In short the three-tier model will remain the network architecture in corporate campus networking, supporting classical applications such as file transfer/access, e-mail, UC, web 1.0 and 2.0, collaboration, etc., for the foreseeable future.

A Two-Tier Network Model Emerges In The Cloud

So what’s so different about the high performance data center and new cloud-computing environments that the three-tier model could be collapsed into two? In a word it’s “performance”. In two words it’s “consistent performance” under heavy load. Performance demand is more critical in this market with applications such as storage connect, high performance computing (HPC), video, extreme web 2.0 volumes, etc., requiring unique network attributes. Consider this: approximately 10 million servers are sold every year. In 2003 20% of servers were sold into HPC and large public facing Web sites according to IDC. In 2009 that number will increase to 50% of server units are sold into cloud and HPC environments. In short, high performance data center and new cloud-computing sites is becoming extremely server dense. Take server density on a scale we have not seen previously and add ultra application demand at load and you have the requirements for a new kind of networking.

To deliver performance at scale and under load of a cloud computing data center equipped with tens to hundreds of thousands of servers delivering applications to millions of users, network performance has to be non-blocking, highly reliable and faultless with low and predictable latency (sub-microsecond) for broadcast, multicast and unicast traffic types. In addition the cloud network needs to be aware of application flows rather than static addressing of devices so that changes in applications, servers and storage can occur without re-configuring the network. Ten-gigabit ethernet connections to servers, storage and between switches are the design direction now, which will scale up as the IEEE develops the 40 GbE and 100GbE standards, expected to be ratified in 2010.

Meeting these requirements offers scale and optimization of servers, applications and storage elements, which allow millions of applications to randomly spin up and down with demand much like atomic behavior described by Brownian motion. In short, traffic profiles in this high performance and dense application environment is unpredictable. This is a key design criterion; that is, networks need to anticipate wild matrix flows with overlapping peaks and valleys and move these flows without dropping packets at microsecond latency between server and storage over the network.

Access Layer Becomes a Virtual Layer

So how is networking design changing to address these high performance requirements? First, the access layer in virtualized data centers is changing dramatically and disappearing as it’s increasingly being subsumed into servers, either in the form of virtual switches and/or blade switches inside servers. A new wave of technology and intelligence is stretching the classic physical access layer into a new virtual access layer. In this new virtual access layer, switching takes place in a hypervisor virtual switching instance, and in other cases the network fabric is stretched to the rack level ensuring single point of management. Effectively the classic access model or end-of-row, top-of- rack and Blade Switching is evolving to a Distributed Access Fabric combining the advantages and benefits of EoR and ToR models.

Secondly, network traffic in clouds is a matrix of overlapping flows with web 2.0 and mash-ups driving massive server-server connections. Network latency becomes a fundamental limiting factor to application performance as the network becomes the bus connecting storage and computing. And as networking speeds increase to 40Gbs, 100Gbs and above the boundaries between storage, networking and computing are being redefined as virtualization is starting to show now.

Cloud Access and Cloud Core Made Up The Two-Tier Model

To accommodate these requirements a two-tier network model is being considered consisting of what I call a “Cloud Access” tier and “Cloud Core” tier. The Cloud Access tier connects servers while the Cloud Core consists of a series of non-blocking switches delivering mesh connectivity between non-blocking Cloud Access switches. The Cloud Core also connects storage and wide area services/routers to the cloud. Within both cloud tiers are switches that provide layer 2 and layer 3 services giving the cloud architect design options of deploying all layer 2, all layer 3 or a hybrid yielding choice as to where to place the layer 2/layer 3 boundary. We reviewed cloud switches in Lippis Report 120 Research Note.

For example, layer 3 services may only be in the Cloud Core or in both Cloud Access and Core which is important for web 2.0 and mash-up based traffic flows. In this model there is no third tier where traffic has to flow to accommodate server-to-server flows; traffic is either switched at Cloud Access or in the Cloud Core at less than 10 microseconds. Oversubscription needs to be carefully managed in a two-tier structure ranging from 1.5:1-to-10:1 Access: Core.

There are examples of a two-tier model in high performance data center applications. For example, the Infiniband architecture describes a leaf and spine structure, which is also championed by Arista Networks. What is important about this market segment is that ethernet switches based upon previous generation ASICS and network operating system technologies may not be up to the performance task. Only two firms, Cisco and Arista have developed new operating systems and hardware for this market.

While Cisco does not tout a two-tier architecture in its Data Center 3.0 program, its Nexus data center switches can clearly be configured in this form. For example, its high end Nexus 7000 would occupy the Cloud Core while its Nexus 5000/2000 occupies the Cloud Access tier. The Nexus 2000 provides GbE connections to servers while obtaining configuration and NX-OS services from the Nexus 5000 via 10GbE placed in end-of-row. The Nexus 2000 and 5000 may be two separate physical devices but they are logically one, making up the Cloud Access tier. In this scenario the Nexus 2000 is a line extender and I expect to see others introduce a similar approach as it delivers the cabling efficiency of top-of-rack and network management operational efficiency of end-of-row. The layer 2/layer 3 boundary resides in the Nexus 7000.

Arista Networks would deploy a series of its Arista 7148SX to construct the Cloud Core while having the option to deploy any of its three 10G switches in the Cloud Access, that being the 7148SX, 7184S, or 7124S. Arista’s Extensible OS (EOS) operating system is unique and purposely built for self-healing resilience and open extensibility designed specifically for cloud computing environments.

Over the next two quarters other networking companies will be announcing cloud-networking products, with most if not all based upon this two-tier model. Look for offerings from Force10, HP, Brocade and Juniper during 2009. Clearly there will be trailblazers and certain vertical market segments that will deploy the two-tier model sooner with a wider adoption after 2010 into 2015. Also note that the two- and three-tier models will co-exist with three-tier being the network architecture in building/campus networks and non-cloud/high performance data centers. But for the high-end cloud and high performance data centers, the two-tier model offers the attributes of low latency, cost and packet throughput required.

9 Debates over Lippis Report 121: A New Approach to Network Design When You’re In The Cloud

  1. Cynthia said:

    What would you consider to be a “problem” or “issue” that still requires some level of research to help advance Cloud Computing adoption?

  2. Nick Lippis said:

    We are so early in the adoption cycle of cloud computing that there are many area that require research. The economic model, scale, management at scale, how virtualization fits into cloud, what is the unified fabric, one, two or three tiers of networking, network access to cloud, inter-connecting private and public clouds, etc etc etc

  3. T Sridhar said:

    Enhancements/upgrades/alternatives to protocols like Spanning Tree Protocol (STP) are another data center networking trend. For example, the IEEE is working on Shortest Path Bridging (802.1AQ0 and the IETF is working on something similar called TRILL using Rbridges. Your thoughts?

  4. KIshore Seshadri said:

    Won’t the answer be different for different groups and different functions in a classic enterprise? I can see some apps that migrate to external clouds needing a fat pipe to an external cloud service provider but not needing the classic 3 tier architecture internally? You’re likely to see similar shifts for collaborative apps that migrate to SaaS models. Internal apps will likely continue to be built around the 3 tier model. I can however see networks inside cloud service providers migrating to a 2 tier model since server-to-data relationships and locations will not be known a priori. This will likely drive the need for full mesh topologies where any server may need to communicate with any other server and/or data storage. The topologies described by Arista and others for cloud networking would make sense inside such networks. If internal clouds do take off and get to large scale, I can see such islands of 2 tiering inside enterprises as well.

  5. Nick Lippis said:

    T Sridhar: Spanning Tree in cloud or other high end data centers and even in enterprise campus networks is fading as most seek to utilize trunk ports for moving traffic versus redundancy.

    KIshore Seshadri: I agree with your assessment. The bottom line is that there are conditions for 3 and 2 tier networking models. For now 2 tier in the cloud and potentially for private cloud implementations and 3 tier in traditional enterprise and data center implementations.

  6. mjmorris said:

    very good article on DC network design in cloud/virtualized world…worth the 5 minute read….

  7. ppgandhi said:

    If one continues to assume that hypervisor switch is the new access (tiers don’t need to be physical boxes), then isn’t there a possibility of 4 tiers in a data center / cloud with a blade server deployment?

    Tier-1 (edge): hypervisor switch
    Tier-2: blade switch (with 10G uplinks)
    Tier-3: 10g aggregation (L2/L3)
    Tier-4: Core (L3)


  8. thsridhar said:


    Your point is valid – the challenge would be tie in the management of these tiers to a unified view, thus reducing some of the configuration complexity. Nick’s outline above of the Cisco approach is relevant here:
    The Nexus 2000 and 5000 may be two separate physical devices but they are logically one, making up the Cloud Access tier. In this scenario the Nexus 2000 is a line extender….


  9. thsridhar said:


    Your point above:

    It’s not until fixed phones are replaced with virtualized soft UC phones and the majority of end-point connections are wireless that the access layer may be subsumed into a WLAN infrastructure. PoE support in the end will keep the access layer viable for years to come.

    PoE might still be used to power the APs in a WLAN infrastructure but there will, of course, be far fewer APs than desktop IP phones needing PoE support.