Tweet

In one of the most candid discussions with Jayshree Ullal, Arista Network’s CEO, she voices her view of Software-Defined Networking as not a market but a set of features. Jayshree views OpenFlow as but one in many APIs becoming available to program layer 2/3 networks. If you’re looking for SDN hype, then no need to listen to this podcast. But if you want a realistic view of this important industry trend, then listen up.

Listen to the Podcast

Tweet

By Arista Networks

Data center operators are expanding data centers to deliver virtualized and cloud-based services including business continuity and disaster recovery solutions. These additional service and availability requirements lead to increased bandwidth and greater distances between geographically dispersed data center sites. As new services are added, data center environments that were once fiber-rich can quickly run out of fiber and find the associated cost for adding more fiber to be prohibitively expensive. Cost-effective embedded DWDM and distance extension solutions can reduce complexity, operating costs and enable a rapid delivery of new services. Find out how by downloading this whitepaper.

Get the White Paper

Tweet

By Arista Networks

Advanced Event Management (AEM) is a powerful and flexible set of tools to automate tasks, customize the behavior of the system and associated operation of the switching infrastructure. Leveraging an open operating system, AEM allows operators to fully utilize the intelligence within EOS to respond to real-time events, automate routine tasks, and take local automated action based on changing network conditions. Simplifying the overall operations, AEM provides the tools to customize alerts and actions. This white paper describes Arista’s approach to Advanced Event Management.

Get the White Paper

Tweet

By Arista Networks

Software Defined Cloud Networking (SDCN) is a term often used when a controller external to the forwarding logic and the actual switch itself programs the network devices to alter or enhance the flow of traffic. SDCN can be achieved via standard APIs that several hardware and software architectures intend to support. This white paper describes Arista’s approach to Software Defined Cloud Networking

Get the White Paper

Tweet

Computer networking vendors have been increasing the speed and port density of their Ethernet switches while reducing power draw and price per port. But while Ethernet switching hardware marches on linearly, thanks to 10, 40 and 100GbE, networking software is taking a different historical path as the pace of compute and network technology evolution has diverged, with networking lagging. Highly virtualized server deployment has broken traditional networking approaches on multiple levels, for example. In response, the industry is now developing a “virtualized infrastructure” or “stack” to add network flexibility. To close the technology gap, Software-Defined Networking (SDN) is promoted as the new “organizing principle” to deliver network software and service value. While it will be, likely, years before SDN’s organizing principles take hold, I propose that these two industry activities are inexplicably linked and phased; here’s why…

Software-Defined Networking

There are multiple definitions of SDN. Making it even harder to pin down SDN, the definitions are evolving too. But this is common in a new breakout space for the computer networking industry that’s evolving fast. For this Lippis Report Research Note, we take the SDN definition that is based upon splitting the data plane or the forwarding hardware of an Ethernet switch from its control plane or the logic that controls how packets flow from ingress to egress. This split of data and control planes opens up an innovation injection point into networking that has not been previously available.

During 2011, a market has opened up for controllers. Currently Big Switch Networks, Nicira Networks and NEC are offering standalone centralized controllers. But limited controllers are also available in open source software, OpenStack and VMware’s vSphere/vCloud too. In addition Cisco’s IOS, Juniper’s Junos, Arista’s EOS, etc., are distributed controllers that may interoperate with centralized controllers in the future. In fact, Arista’s EOS already supports OpenFlow, OpenStack and vSphere/vCloud.

The link between the separated data and control plane is an open interface called OpenFlow. Now some end their SDN definition here, but this is just the beginning as the real promise of SDN are the applications that will reside upon the controller to address a wide range of networking issues and opportunities. In fact researchers at Princeton and Cornell are developing the Frenetic programming language that provides high-level network abstraction that gives programmers direct control over the network, allowing them to specify what they want the network to do without worrying about how to implement it.

One can imagine a wide range of applications residing upon a controller such as WAN optimization, traffic engineering optimization, load balancing, security services, etc. In essence, the control plan allows network services that are currently deployed as appliances to be virtualized appliances/applications much like applications that reside on top of a VM. It gets even more interesting, as a centralized control plane can be easily split in to many little control planes, each of which sees its own slice of the data plane topology. In traditional networking where control and data planes are one and the same and in each box, it is much harder to merge control planes and split data planes. It’s possible, but harder to keep complexity and stability in check over the long term. Splitting control plans can have huge value in public cloud multi-tenant or private cloud multi-team networking.

SDN and OpenFlow are at the early stages of its industry matriculation. But one thing is clear: SDN is an organizing principle whereas network software is developed by both network vendors and third parties, and network services are virtualized. SDN thus represents a new industry order and structure as to how value is added to networks. But I digress. The real issue today is solving network inflexibility in the face of highly virtualized data centers.

Enter the “Virtualized Stack” or Virtualized Infrastructure”

Virtualized server deployment has been propelled en masse, thanks to increased data center efficiency, by delivering the same or greater application workload with a reduced number of servers. While this is good, many IT business leaders are now realizing huge consequences to highly virtualized data centers that span from IP address change management to application management.

At the IP address level, networking has become extremely rigid within virtualized environments, slowing down process, limiting moves and changes as well as elongating the time to spin up an application that resides within a VM. Necessary network services to support the virtual cloud infrastructure, such as IP address assignment and management, are still performed largely with manual tools and processes, such as spreadsheets shuffled between various departments or operational groups, which can result in days of delay for something as simple as assigning an IP address to a VM. Contrast that with the virtual server administrator. Virtual instances of servers and machines can be dynamically provisioned, migrated and shut down by a virtual server administrator in minutes.

Moving up the stack, challenges are rooted in application management plus Layer 4-7 services such as WAN optimization, Application Delivery Controllers and security, especially in environments that include multiple hypervisors, a wide variety of workload types and shifting virtual machines.

For example, the new challenges of enterprise application management in virtualized data centers include: what type of and location of network intelligence is required when multiple hypervisors and various workloads exist and shift? Also how do operations groups maintain consistent security policy across both virtualized and non-virtualized environments consistently? And how do operations groups monitor and maintain application flow visibility?

Cisco

Cisco, for example, is addressing these issues via its Virtualization Stack and is now organizing its products around this initiative. Three components define Cisco’s virtualization stack, those being: 1) virtual networking, 2) virtual security and application networking services and 3) orchestration and provisioning. An important part of Cisco’s strategy is the virtualization of appliances such as its VSG or Virtual Security Gateway, the ASA 1000v, the support of VXLAN, the Nexus 1000v, etc.

Brocade, F5, Citrix

But F5, Citrix and Brocade are all virtualizing their appliances, moving away from physical single application appliances to an integrated virtualized suite. One can imagine that these virtualized applications will some time reside upon an SDN controller as their next stage of evolution. In addition each application delivery vendor has a way for programmers to control application network behavior. For example, Brocade recently launched OpenScript, a Perl-based scripting language used to modify the content of and control delivery of packets at Layer 4 through Layer 7 on its ServerIron ADX products. These scripting languages could be standardized and reside within an SDN controller.

Embrane

A good example of what the virtualized Layer 4-7 future may hold is that of a start-up firm called Embrane.

Embrane has virtualized server load balancing, firewalls and VPN termination and placed them upon a distributed software platform called heleos. Heleos runs on x86 servers and any hypervisor. It leverages a distributed virtual architecture that decouples network services functionality from the underlying physical infrastructure and hypervisor technology that it says provides high scalability, flexibility and performance.

IBM & NEC

IBM and NEC offer the best example of a commercial SDN offering with OpenFlow. NEC’s pFlow OpenFlow controller that resides within an IBM server manipulates IBM System Networking G8264 OpenFlow switch’s flow table. The link between the two is OpenFlow 1.0.0. The NEC pFlow controls traffic, discovers topology, gathers stats and other functions while the G8264 forwards traffic based upon these flow commands.

What’s impressive about the IBM/NEC SDN solution is that it has customers such as: Tervela validated the IBM and NEC OpenFlow solution ensures predictable performance of Big Data for complex and demanding business environments. Selerity’s IBM and NEC’s OpenFlow solution improved real-time
decision-making for global financial markets. Stanford’s IT Department chose IBM and NEC’s OpenFlow solution to deliver network capacity on-demand to its academic community. What’s important about these use cases is that IBM is communicating SDN via OpenFlow’s value in business terms, which will only increase as industry adoption accelerates.

In essence the SDN market has started, and as its technology underpinnings solidify, many of today’s network services will fall under the SDN umbrella. In fact, nearly all network vendors are launching SDN programs as a new way to communicate existing product value and their evolution into a SDN. Just like the Appian Way where all roads lead to Rome, all network services may very well lead to an SDN.

Tweet

An exclusive cross vendor comparative test report conducted at Ixia’s iSmiCity defined by the Lippis Report. This report provides detailed test information on the following products that have not been previously tested in public. The report details test results of the following products:

Alcatel-Lucent OmniSwitch 10K
Aclatel-Lucent OmniSwitch 6900-40X
Arista 7504 Series Data Center Switch
Arista 7124SX 10G SFP Data Center Switch
Arista 7050S-64 10/40G Data Center Switch.
IBM BNT RackSwitch G8124
IBM BNT RackSwitch G8264
Brocade VDXTM 6720-24 Data Center Switch
Brocade VDXTM 6730-32 Data Center Switch
Extreme Networks BlackDiamond® X8
Extreme Networks Summit® X670V
Dell/Force10 S-Series S4810
Hitachi Cable, Apresia15000-64XL-PSR
Juniper Network EX Series EX8200 Ethernet Switch
Mellanox/Voltaire® VantageTM 6048

This 125-page report is a must for those evaluating 10/40 GbE data center switching equipment for private or public cloud infrastructure. You don’t want to buy data center switching gear until you read this report.

Get the White Paper

Tweet

The Lippis Report has conducted three open industry test of 10 and 40GbE data center fabric switches at Ixia’s iSimCity. Michael Githens of Ixia interviews Nick Lippis of the Lippis Report to look back on what we have learned after testing eleven products from nine vendors including Alcatel-Lucent, Arista Networks, Brocade, Dell/Force10, Extreme Networks, Hitachi Data Systems, IBM, Juniper Networks and Mellanox/Voltaire. We then look forward as to what the industry will be serving up in 2012 for data center fabrics.

Download “Fall 2011 Open Industry Network Performance And Power Test Report” here.

Watch the Video

Tweet

By Arista Networks

VMWARE VXLAN is a new network technology developed by VMware that enables stateful VM mobility across traditional L3 routed boundaries. This enables more freedom and flexibility in matching workloads to computing power. By enabling a larger, and essentially flatter network while building on top of proven models for stable scaling of networks such as routing and equal-cost multipath forwarding, VXLAN enables any workload to be provisioned on any virtualized host, anywhere in the network that is IP reachable. No longer do routed topology decisions restrict workload mobility.

If you are a VMware and network administrator who is building virtualized networks with more than 250 VMs or want to stretch a virtual machine farm across two data centers or two or more routed domains with full workload portability, then you need to read this white paper.

Get the White Paper

Tweet

By Arista Networks

While servers and applications have fully embraced the concepts of automation, sadly the network infrastructure, on which they all rely, is still mired in legacy technologies. Current methodology, such as CLI, requires extensive hands-on provisioning and configuration by knowledgeable personnel. In modern cloud infrastructure, network managers must be able to centralize provisioning and configuration roles to improve reliability, minimize bring-up costs, and contain the expenses of creating a cloud data center service. Whether you are looking to maximize the efficiency and reliability of your existing operations, or you are looking to take advantage of a cloud-based infrastructure, fully automated provisioning is an essential capability, and Arista’s Zero Touch provisioning offers the first approach to automated network configuration.

Get the White Paper

Tweet

A cross vendor comparative test report conducted at Ixia’s iSmiCity defined by The Lippis Report. This report provides detailed test information on the following new products that have not been previously tested in public. The report details test results of the following products:

Alcatel-Lucent OmniSwitch 10K,
Arista 7504 Series Data Center Switch,
Arista 7124SX 10G SFP Data Center Switch,
Arista 7050S-64 10/40G Data Center Switch,
Brocade VDXTM 6720-24 Data Center Switch,
IBM BNT RackSwitch G8124,
IBM BNT RackSwitch G8264,
Force10 S-Series S4810,
Hitachi Cable, Apresia15000-64XL-PSR,
Juniper Network EX Series EX8216 Ethernet Switch,
Voltaire® VantageTM 6048.

This 93-page report is a must for those evaluating 10/40 GbE data center switching equipment for private or public cloud infrastructure. You don’t want to buy data center switching gear until you read this report.

To download the report click here.

Get the White Paper

Tweet

Andy Bechtolsheim, Founder, Chief Development Officer and Chairman of Arista Networks, joins me to discuss how the public and private data center cloud network market is emerging and the new type of networking it’s requiring.

Listen to the Podcast

Tweet

Ethernet networking is now the single most important data center technology to assure the new IT economic model of centralized application delivery. Yes that’s right—Ethernet as the data center fabric is the stability point in data center design that will dictate if a data center or cloud facility can scale to support huge application and storage traffic loads. And if you think that Ethernet switch performance is not important then you would be as right as the engineers who designed the Tacoma Narrows Bridge. In this Lippis Report Research Note, we explain why network performance of data center Ethernet switching products matter more now than ever.

Data centers are becoming IT black holes where no application can escape the gravity of its economic force. A few facts are in order:

Mobile Applications and Devices Soar: Mobile application use is expanding exponentially, thanks to the popularity of the iPhone and increasingly Android smartphones. Most important about this is the traffic load these applications are placing on data center Ethernet fabrics. The vast majority of mobile applications are hosted in data centers and/or public cloud facilities. The application model of mobile devices is not to load them up with thick applications like Microsoft Word, PowerPoint, Excel, etc, but to load them with thin clients that access their application and data in data centers, private and/or public cloud facilities. As of this writing, there are some 205,000 plus smartphone applications.

A New Tier of Computing Emerges: A new and rapidly growing tier of computing has emerged in 2010. This tier is the Android tablet and iPad. According to the Wall Street Journal, sales of tablet devices (Android plus iPad) are expected to hit 19.5 million units in 2010 and 54.8 million in 2011. In contrast, Garter predicts that PC shipments will be 352 million units in 2010. In just six short months, tablets now represent some 6% of PC shipments and are expected to displace nearly 10% of PC shipments by 2014!

What is important about this new tier of computing is its application model, which is nearly the same as smartphones. That is these 10s of millions and growing numbers of tables are relying on data centers plus private/public cloud facilities for their applications placing further traffic load on Ethernet data center fabrics.

Virtualized Desktops: 2011 will be the year of the virtualized desktop. Frustrated with Microsoft’s enterprise application licensing, plus desktop support model, IT business leaders will turn toward virtualizing desktops at increasing numbers in 2011. The application model of virtualized desktops is to deliver a wide range of corporate applications hosted in data centers and/or private/public clouds over the enterprise network. While there are no estimates to the traffic load this will place on campus and data center Ethernet networking, one can only assume it will be huge.

Storage Traffic over Ethernet Fabric: Converged I/O or unified networking where storage and network traffic flow over a single Ethernet network will increasingly be adopted in 2011. A single converged network adaptor or CNA plugged into a server provides the conduit for storage and application traffic flows to traverse over an Ethernet fabric. The number of suppliers offering CNAs has grown significantly, including Intel, HP, Emulex, IBM, ServierEngines, QLogic, Cisco, Brocade, etc. In addition, the IEEE opened up the door for mass deployment as it has ratified the key Ethernet standards for lossless Ethernet. What will drive converged I/O is the reduced cost of cabling, NIC and switching hardware.

The above trends are just starting to take hold. Over the next five years, a sea change in IT delivery will occur. It’s clear that the number of mobile smartphones and tablets will only increase as will their reliance data center hosted applications. Virtualized desktops too will force an increase in centralized application delivery while storage traffic increasingly flows over Ethernet fabrics. Corporate application portfolios will change dramatically as will their application traffic profiles with loads being ever more unpredictable. There will be surprises or unforeseen changes that may very well accelerate these trends.

From a data center design point of view, IT architects discovered over three years ago that they can scale compute resources to nearly unlimited dimension thanks to multi-core processors, virtualization and cloud spec design. And with centralization comes huge corporate advantage that being centralized complexity to manage IT more effectively. But more importantly is the fact that IT represents on average only 2% of corporate revenue but has a profound impact on the other 98% of corporate operational spend and competitiveness. With application centralization, IT business leaders can more easily control IT and target it toward reducing corporate operational spend through streamlined business processes or launch new services to respond to market dynamics.

At the center of this massive application centralization transition is networking as it ties compute, storage and internet access together. Ethernet networking, in particular, is now the single most important data center technology to assure the new IT economic model of centralized application delivery. Now most corporations and cloud providers are scaling up their data center bandwidth with 10GbE. In fact, over the last quarter, many networking companies have reported greater than 60% shipment growth in their layer 2 and layer 3 fixed and modular Ethernet switches. So the above trends are driving network demand.

But IT architects and business decision makers need to understand the underlying performance and power consumption metrics of the switches they deploy. The only way to be assured that the Ethernet fabric that is being deployed now in the data center will scale to support increasing application load and storage traffic is to review public, independent, credible and repeatable network throughput and latency performance numbers across multiple vendors.

During the mid 1990s, Scott Bradner of Harvard University and Nick Lippis of the Lippis Report offered independent comparative Ethernet switch performance test evaluations to guide IT business leaders with their purchase decisions. But network purchase decisions have much greater weight to them now as over 80% of IT budgets are spent in the data center. Further, HP wouldn’t have purchased 3Com or IBM wouldn’t have purchased BLADE if they didn’t realize how critically important networking has become to successful data center and cloud computing design.

It’s for the above reasons the Lippis Report has teamed with Ixia to deliver an open data center fabric evaluation of 10GbE switches. Several network equipment manufacturers will participate in this industry-first evaluation, including Alcatel-Lucent, Apresia, Arista, Blade, Juniper Networks and Voltaire. The testing, which is taking place at Ixia’s iSimCity location in Santa Clara, will use Ixia’s Xcellon-Flex load modules to evaluate the performance of the participating vendors’ top-of-the-line 10 GE data center devices.

We’ll publish a report on our findings in mid January so stay tuned.

Tweet

Major IT Delivery Transitions IT Business Leaders Are Managing
Application owners and developers have been deploying and writing applications as if networks had no boundaries or were borderless. By “application owners” I mean IT departments chartered with IT application delivery and management. By “application developers” I mean in-house corporate software developers, independent software vendors (or ISVs) and software companies. There has always been a disconnect between applications and network architects where developers write applications to run over a network as long as there is connectivity. In addition, service-oriented architecture (SOA) based applications call for greater application componentization, which increases messaging between application components, resulting in the network having a direct impact on application performance. In essence, application owners, developers and application standard bodies assume that networks are borderless as the industry is organized around the OSI model where knowledge and skills at one layer, e.g., the network is not necessarily taken into account at another layer, i.e., the application. Therefore, the normal state of affairs is that network designers have been tasked to optimize applications to improve user experience especially when the application was not written to run over a particular kind of network. This status quo does not scale and needs to be re-thought.

Business Drives Applications that Drive Computing that Drive Networking

Every cycle of computing has brought with it this discontinuity between applications and networks with the possible exception of mainframe computing and SNA. Minicomputer applications designed for local ASCII terminal connections were extended over the Wide Area Network (WAN) and via virtual terminals. Client-server computing applications designed to run over Local Area Networks (LANs) were extended over the WAN. At first the internet was text based until the mid 1990s when the web was developed, bringing graphics, audio and video to a network that needed a massive upgrade to support new media rich applications.

IT today is no different. Application developers are writing mobile applications at a frenzied pace thanks to Apple’s iPhone and iPad, Google’s Android, RIM’s Blackberry and now Cisco’s CIUS plus Avaya’s Flare, etc. Legacy enterprise applications are being extended to mobile platforms too with the assumption of a suitable network for delivery. At the same time, applications are being increasingly centralized into consolidated data centers creating greater distance between users and their applications plus data. Some estimate that over 80% of enterprises have undergone a data center consolidation process, which is significant, but we are just at the beginning of the centralization trend.

Thanks to the economics and performance offered by server virtualization, much more consolidation will occur with associated challenges. For example, IT leaders require application tracking as applications are moved from Virtual Machine (VM) to VM as they tune/optimize their virtual infrastructure or respond to peak loads as well as manage VM failovers. In addition to virtualization, massive data centers we call cloud-computing facilities are being built to host applications at scale plus offer infrastructure, platform and other IT services. According to the Yankee Group, 56% of IT business leaders seek to take advantage of cloud-computing technology and build their own private cloud center while 24% seek a fully-managed cloud-computing facility. In the same study, 32% of IT business leaders will seek a hybrid cloud approach that is, connect their private cloud to a service provider’s public cloud. While these market numbers are impressive, they could be much higher as IT leaders express that their top three concerns as they consider cloud services is application performance issues, according to IDC.

In addition to increased mobile and cloud-computing trends, video communications, both on-demand and real-time, have become the largest percentage of internet traffic type. In fact, Cisco Systems recently predicted that by 2014 video traffic will be greater than 94% of all global internet traffic!

This disconnect between applications and network architects will more than likely continue as application owners/developers/standards continue to view networks without borders and boundaries. However, for most network architects, there is no single network, but a wired network, wireless, campus, wide area, data center, branch office network, telecommuting network, mobile network, etc. In fact, most enterprises have a diverse infrastructure in which they are tasked to delivery applications over and for those applications to perform at high standards. The good news is that network designers and architects are starting to build borderless networks that anticipate unforeseen application changes, are equipped with a portfolio of application performance features and simplify deployment and management of IT services…more on this below.

Application Performance Challenges

From the above discussion, it’s clear that enterprise-computing applications are being demanded and stretched over increasingly borderless networks. Consider that the number of small or remote offices and mobile employees are increasing significantly. It’s impossible to argue the mobile computing surge with over 3.3 million iPads shipped in the first three months of its launch, and new entrants such as Cisco and Avaya offering CIUS and Flare tablets, respectively, for business users. In addition, data centers are being consolidated with cloud computing, offering further consolidation and centralization of applications. Applications are changing too as developers add rich media features, and video becomes a dominate application type. Employees, customers, partners and suppliers will be accessing applications over ever-larger distances, via a plethora of endpoints and different networks.

To assure applications perform their task and deliver an excellent user experience, network architects and designers will be increasingly challenged with network capacity being taxed as a wider application portfolio competes for network resources. Today’s model of application performance optimization is to implement appliances within remote sites and data centers, which increases certain application performance, but at the high capital and operational expense of increased network complexity. In addition to network capacity and complexity issues, latency or application transaction delay and how to efficiently utilize data center resources are challenges faced by network architects as they seek to maintain high application performance over a borderless network. Relating specific application transaction problems to network behavior to ascertain if a correlation exists is yet another challenge.

Application Performance Creates Corporate Value

At the center of application performance is corporate performance. The ability of IT leaders to respond to executive management directives is directly linked to corporate performance. Executive management may be challenged with a competitive threat or a new market opportunity, etc., requiring fast corporate response. IT leaders who can execute directives quickly have built an agile business capable of changing when markets or customers shift under them, placing their corporation in a better competitive position to serve its customers and prospects. For example, consider a retail store under competitive pricing pressure where executive management decides to respond with an alternative offer. IT may be able to display the new offer via digital signage quickly allowing the business to respond.

Key to business agility is the IT attribute of rapid innovation absorption–that is, the capability to deploy new applications and technologies at the speed of business opportunity. Most IT infrastructures consist of innovation and features which are already in place, but IT organizations require knowledge, skills and tools to put them to work when needed.

A borderless network that is capable of application performance delivers these attributes of innovation absorption and business agility. In addition, IT resource utilization can be optimized, and most important to users is that they gain an excellent IT experience independent of geographic location, endpoint device or application, which in the end improves productivity.

As an example of optimal resource utilization, consider Cisco’s ISR G2 branch office router that integrates unified communications, wide area application optimization, network security, LAN/WAN networking plus supports its AXP (or Application eXtension Platform), which run applications at the branch office router. In one branch office, an IT manager can deliver networking, security, voice and video communications and host applications while gaining visibility to applications. This type of resource utilization not only saves on capital cost and energy spend, but offers IT operational efficiency, rapid application deployment and innovation absorption.

To gain the full value of corporate applications, their performance must deliver excellent user experience. An excellent experience should not only occur while working in the office or at home, but anywhere in between, even while talking on a mobile endpoint. Independent of geographic location, a user accessing his/her business services and/or personal services should be the same seamless experience. Application performance is key to excellent experience and should be consistently good whether sitting at a desktop watching a video or engaged in a Web conference, and then immediately transitioning to an iPhone for example. The user should have an excellent experience at the highest level afforded by his/her endpoint. To deliver this seamless user experience, application performance technology needs to be incorporated in corporate IT infrastructure, endpoint devices or a combination of both.

That is, networking silos need to become an integrated network without borders. For applications to offer the best possible user experience, then the use of application acceleration technology as appliances or an overlay needs to be integrated into the network fabric and into network operating systems. This technology, which has improved application delivery for specific applications, needs to become systemic and fully distributed throughout the network fabric. The integration or pervasiveness of application acceleration technology within networks and endpoints is its natural evolutionary next step. Over the next few months we’ll see vendors such as Cisco, HP Networking, Juniper, Riverbed, Citrix, Blue Coat, et al, start to deliver on this vision.

Tweet

By Arista Networks

VM Tracer offers the tightest integration between virtual machines and network infrastructure. Visibility in to virtualized infrastructure is key for management and operations of mobile and virtual environments.

Find out how by downloading this white paper.

Get the White Paper

Tweet

By Arista Networks

The increase in utilization of virtual machines and increasingly virtualized resources in the data center has caused a reduction in network visibility into the virtual infrastructure. VM Tracer provides the visibility necessary for the network team to support virtual environments and the automation necessary for server administrators to be effective and efficient.

Find out how by downloading this paper.

Get the White Paper

Tweet

Networking is entering a new phase or era. During the 1990s, new networking markets opened up, creating multi-billion dollar opportunities for the vendor community and corporate cost savings for IT business leaders. First, it was shared LANs and routing, then switched LANs, then Frame Relay to speed up WANs, then SNA over IP, then remote access via dial-up and VPN, then MPLS, then IP telephony, then Wireless LANs etc… and now, it’s video and cloud networking. You get the picture. But what we didn’t realize as we build these networks is that they are silos with disparate management systems and unique access methods resulting in operational cost overlap and, most importantly, user frustration as they transition application use from desktop, to mobile end point, to remote endpoint. In short, we built boundaries around applications in the form of networks and it is the dismantling of these borders that vendors are now starting to deliver and differentiate upon. It’s not just Cisco that communicates borderless networks, but HP Networking, Juniper, Brocade, Extreme, Avaya, Force10 and others too. Why is the industry entering a new age of borderless networking and what’s in it for IT business leaders, is explained in this Lippis Report Research Note.

As each new wave of computing entered corporate IT departments, a new set of networking requirements arose. To connect remote 3270 terminals via SNA to mainframes, IT implemented an analog multipoint wide area network or WAN. To connect remote ANSI terminals to minicomputers, IT departments implemented pools of dial-up modems and private line WANs. To connect personal computers (PCs) via Client-Server computing, IT departments implemented Local Area Networks or LANs via LAN switches, which we now call wired connections. To connect multiprotocol LANs over the corporate WAN, IT departments implemented routed networks. To gain access to LAN based applications while remote, IT departments implemented Virtual Private Networks or VPNs. And, as computing and applications go mobile, IT has been implementing Wireless Local Area Networks or WLANs. In short, each network was deployed to service a certain computing style and application set. These networks are silos, and with advances in technology, IT business leaders can now design one borderless network to provide a broad array of common access methods to support a plethora of endpoints and applications.

Siloed networking frustrates users, as each access network performs differently depending upon its access method. Siloed networking also frustrates IT, as each siloed network has its own management system creating inefficient IT operations. In addition, siloed networking does not meet today’s IT “any access” requirements.

There are boundaries or silos that need to be broken down in many places of the network. In today’s modern IT world, applications are being extended over multiple networks e.g., wired, wireless, cellular, remote, virtual, etc where users need to shift their application access back and forth between these different network access methods and expect the same or consistent experience. In short, networks need to be borderless so that applications can be accessed independent upon network entry point and IT operations efficient. This “any access” trend is accelerating as IT business leaders seek to connect not only traditional desktops and laptops, but smartphones, notebooks, tablets, iPads, cameras and building control systems into a common general purpose network that support multiple logical network topologies.

Crossing purpose-built silos is difficult for applications, as bandwidth and quality of service issues limit application portability thus their usefulness. These different access methods offer limited consistency resulting in user frustration when they shift application access from desktop to mobile smartphone to VPN and back again.

And this shifting of application access between different networks and endpoints is only going to increase. Apple sold over 3.3 million iPads in its first 3 months, the highest uptake of any endpoint device. Google activates 100,000 Android based phones a day. Cisco recently announced its CIUS android-based table for business use with tight links to its unified communications (UC) and videoconference systems. Every major UC provider will be offering similar devices while traditional computer vendors serve up android-based tablets over the next few quarters. The iPad and Android tablet is a new tier of computing which will drive users to access their applications over mobile and wireless networks in addition to their desktop and VPN networks.

If IT business leaders are unable to get ahead of this curve and think of network access from an architected and unified design point of view, than unfortunately, their users and IT cost will be more frustrated and expensive, respectively, than others. Siloed networks are friction points as they create boundaries between network access types degrading user experience, which results in decreased productivity and increased IT operational cost. The result is a high total cost of ownership and less then optimal user experience, and thus decreased corporate productivity. The status quo of siloed networking is about to change.

Cisco’s Borderless Network Architecture

From a design point of view, borderless networking requires three core attributes: 1) reliability, 2) security and 3) seamlessness. Cisco was the first to articulate a vision for borderless networks, which has resonated with IT business leaders as it represents a solution to their pain. For example, Cisco’s borderless network architecture is built upon five services: 1) mobility or users in motion, 2) Energy efficiency called EnergyWise, 3) integrated network security via its TrustSec architecture, 4) application performance and 5) video management, control and distribution via its MediaNet. These borderless network services are built within switching, routing, security, wireless and wide area application services or WAAS infrastructure products. It’s the integration of these services into existing network infrastructure and their control via policy and management that enable a borderless experience to occur.

Juniper’s New Network

But Cisco is not the only supplier to grasp the problem siloed networks create. Juniper Networks is working to a similar end, albeit it hasn’t articulated it well. It provides VPN, LAN Switching, mobile security through its acquisition of SMobile and is working toward a flat cloud Ethernet fabric through its project Stratus and New Network initiatives. For example, Juniper plans to integrate SMobile security into its JUNOS Pulse endpoint software for network connectivity and acceleration breaking down the boundary between LAN based and mobile network access.

HP Networking’s Converged Infrastructure

When HP Networking launched its comprehensive network portfolio in April of this year it emphasized the elimination of network silos. The HP Networking portfolio strives to eliminate redundant equipment by integrating wired and wireless environments with security from edge to core. From an IT operations perspective, this translates into a “single pane of glass” for management, configuration, deployment and monitoring these networks as if one. HP Networking hopes to implement a common policy management to reduce human error of network operations while creating a consistent user experience across access mediums.

Brocade One

Brocade has jumped on the borderless bandwagon also in June of this year with the introduction of its “Brocade One”. Brocade One emphasizes the convergence of wired, wireless and cellular networking to offer a seamless user experience. In addition, Brocade One describes its view of a simplified virtualized data center network fabric that scales to cloud spec. In essence, Brocade One is about eliminating the boundaries around wired, wireless and data center networking.

Arista Network’s VM Tracer

Arista Networks doesn’t use the terminology of borderless networking either, but its recent VM Tracer strives to eliminate the boundaries between physical and virtual networking environments. VM Tracer does this by being integrated into Arista’s EOS linking Arista switches to VMware’s vCenter. This linkage creates an adaptive infrastructure in which the network responds to changes in the VM network while also providing complete visibility into the virtual machine network.

Extreme’s DirectAttach

Extreme Networks has focused on removing two network boundaries; the wired and wireless boundary and the physical to virtual network boundary. For the latter, Extreme has introduced its Direct Attach approach to data center networking that eliminates the virtual switch layer, simplifying the network and improving performance.

Force10’s Open Automation

Force10’s focus in eliminating boundaries is in the data center between physical and virtual networks. Force 10′s Open Automation initiative seeks to align dynamic data center changes with network configuration and policies, a huge barrier to virtualized data center management and scale.

While each of the above suppliers are at different points in their borderless network initiatives, the direction is clear. The boundaries between siloed networking are coming down be it in the data center, campus, branch office or home. For IT business leaders this means simplified operations and management as a key attribute is the “single pane of glass” approach to network management for siloed networks. The big surprise and delight will be found in enhanced user experience, as borderless networking strives to deliver a common access method for all networking types while enabling applications to be extended across a plethora of different endpoints, depending upon endpoint capabilities and network resources.

In essence, borderless networking’s value proposition is that it enables a corporation to be more adaptive or agile while increasing user experience and reducing operational cost. With the majority of IT business leaders trading off reductions in operational spend for an increase in capital expenditure, borderless networking is the right solution at the right time.

Tweet

It hasn’t been since the mid 1990s that the networking industry was focused on multi-protocol integration or convergence. But the industry is gearing up for a major innovation and competitive cycle fueled by the multi-billion dollar addressable market for data center network fabrics. Over the last eighteen months, every major Ethernet infrastructure provider has been talking about two and three tier network fabrics for high-end data centers.

Companies such as Cisco, Arista Networks, HP/3Com, Force10, Voltaire, Extreme, Brocade, Juniper et al have announced network fabrics for data centers with five thousand and more servers with and without storage enablement. Juniper talks of a one-tier fabric through their Project Stratus work with IBM to be available some time in the future. Brocade recently introduced its Brocade One, which is a converged data center fabric. Extreme Networks launched its DirectAttachTM that eliminates virtual plus blade switch layers. HP has FlexFabric, a virtualized fabric for the data center. Cisco launched its FabricPath Switching System or FSS for the Nexus 7000 that enables massive scale of a two-tier fabric.

In this Lippis Report Research Note, we review the architectural attributes of two tier network fabrics.

The IT industry is at an inflection point as service delivery is becoming more and more centralized thanks to data center consolidation, virtualization, cloud and mobile computing. It is estimated that a third of all IT spend is concentrated in the data center, and this trend is only building thanks to favorable economics, motivating IT business leaders to centralize IT delivery.

The impact of this trend is more and more dense data centers made up of servers in the thousands to tens of thousands and higher. It is at the scale of 5,000 plus servers that a new network fabric is required for high-end data centers. High-end data center design is challenged with increasing complexity, the need for greater workload mobility and reduced energy consumption. Traffic patterns have also shifted significantly, from primarily client-server or as commonly referred to as north-to-south flows, to a combination of client-server and server-server or east-to-west plus north-to-south streams. These shifts have wreaked havoc on application response time and end user experience, since the network is not designed for these Brownian motion type flows.

The main requirements for high-end data center network fabric are low latency, large flat layer 2 domains to enable workload mobility, low power consumption, simplicity of design and significant bandwidth. Storage enablement, meaning consolidated I/O or virtualized I/O, is a growing priority and a new fabric that can support FiberChannel over Ethernet, iSCSI over Ethernet, iWARP over Ethernet or Infiniband over Ethernet, is a major plus. One salient observation is that it’s pretty clear that Ethernet is the network fabric of choice, as it is the only network protocol that enjoys continual innovation such as TRILL, Data Center Bridging, IEEE’s 802.1AQ, link aggregation, multi-pathing, and as recently ratified by the IEEE 40 Gbs and 100 Gbs speeds.

With the above requirements in mind, let us review data center network design options.

Three Tier Data Center Fabric

A three-tier network architecture is the dominant structure in data centers today and will likely continue as the optimal design for many networks. For most network architects and administrators, this type of design provides the best balance of asset utilization, layer 3 routing for segmentation, scaling and services, plus efficient physical design for cabling and fiber runs. By three tiers we mean, access switches/Top-of-Rack (ToR) switches, or modular/End-of-Row (EoR) switches that connect to servers and IP based storage. These access switches are connected via Ethernet to aggregation switches. The aggregation switches are connected into a set of core switches or routers that forward traffic flows from servers to an intranet and internet, and between the aggregation switches. It’s common in this structure to over-subscribe bandwidth in the access tier, and to a lesser degree, in the aggregation tier, which can increase latency and reduce performance. Inherent in this structure is the placement of layer 2 versus layer 3 forwarding that is Virtual Local Area Networking or VLANs and IP routing. Also common, is that VLANs are constructed within access and aggregation switches, while layer 3 capabilities in the aggregation or core switches route between them.

But within the high-end data center market, where the number of servers is in the thousands to tens of thousands plus and where north-south plus east-west traffic is significant, is where a new structure is needed. It is within these data centers where applications need a single layer 2 domain.

Two-tiers of network fabric

A two-tier fabric is designed with two kinds of switches: one that connects servers, and the second that connect switches creating a non-blocking, low latency fabric. In short, there are server facing and fabric facing switches. We use the terms ‘leaf’ switch to denote server facing or connecting switches and ‘spine’ to denote fabric facing or switches that connect leaf switches into the fabric. Together, leaf and spine switches create the fabric.

Many IT leaders in Global 2000 firms will have deployed both two and three tier network structure, as different deployment models are used for different applications. For these leaders, a network equipment supplier that possesses product architecture flexibility, meaning an end-to-end product solution that accommodates tier two and three fabrics would be advantageous. This flexibility is found in product that supports layer 2 and layer 3 forwarding, as well as, a variety of line cards to offer design options.

A common network Operating System (OS) of products configured for two and three tier structure is important as IT operations gain efficiency to manage fabrics, as configuration and management are consistent. In addition, a common network OS offers rapid absorption of innovation to IT operations, as new OS features are available at the same time to all fabrics. The benefit of using a common product set to build tier two or three fabrics offers value around operational efficiency, training, sparing and ease of evolution between fabric deployments. In short, the network fabric needs to be simple and general purpose versus purpose built, which a common set of products creating tier two or three fabrics offer.

A Unified/Converged Fabric

The concept of a unified fabric is to virtualize data center resources and connect them through a high bandwidth network that is very scalable, high performance and enables the convergence of multiple protocols onto a single physical network. These IT resources are compute, storage and applications, which are connected via a network fabric. In short, the network is the unified fabric and the network is Ethernet.

The industry tends to focus on storage transport over Ethernet as the main concept behind a unified/converged fabric with technologies such as Fiber Channel over Ethernet or FCoE, iSCSI over Ethernet, iWARP over Ethernet and even Infiniband over Ethernet. But this is a narrow view of a unified/converged fabric which is being expanded, thanks to continual innovation of Ethernet by the vendor community and standards organizations such as the IEEE and IETF.

Ethernet innovations such as FCoE, Data Center Bridging or DCB, IETF’s Transparent Interconnection of Lots of Links or TRILL, CEE or Converged Enhanced Ethernet, link aggregation, IEEE’s 802.1AQ have enhanced Ethernet networking to support a wide range of new data center fabric design options. In addition to these protocol enhancements, the IEEE has ratified its work on defining 40Gb and 100Gb Ethernet, significantly increasing Ethernet’s ability to scale bandwidth. To demonstrate how Ethernet is evolving to be the unified fabric for high-end data centers, we explore Cisco’s new FabricPath Switching System innovation in this white paper.

The decision to implement a two or three tier network structure comes down to scale. For high-end data centers, a two-tier structure meets the requirements of low latency, movable workloads, scale, simplicity, etc. Many global 2000 concerns will have deployed both a two and three tier network fabric for their high end and less dense data centers.

When shopping for network equipment to construct two and three tier network fabrics, look for suppliers that support both rich Layer 3 routing services and scalable Layer 2 Ethernet capabilities to ensure choice and flexibility of three tier and scalable two tier fabric implementations. Such suppliers offer products that can be configured in multiple use cases and topologies where modules are inter-changeable, skills transferable and operations common between both fabric approaches.

But make no mistake about it, it’s a two-tier network fabric that IT business leaders and data center architects have gravitated toward for high performance computing, cloud scale data centers and just plain high end data centers of 5,000 and above servers.

Tweet

Networking has become “rigid”. Yes I know it’s almost absurd to attribute inflexibility or rigidity to networking. Look what TCP/IP has done for us. There are nearly 2 billion people connected to the internet and according to the Internet World Stats internet user growth rate increased by 380% between 2000-2009. With 2 billion people and growing online, accessing a plethora of applications via a wide range of end-points there is no doubt that the internet and TCP/IP has been a much bigger success than anyone would have imagined back in the early ’90s. But there’s always a give and take between computing and networking where one drives and changes the other. Right now we are in a compute innovation cycle that’s driving a fundamental change in networking which screams out the need for more flexibility.

Sure networking has increased from a bandwidth point of view and the IETF has added new protocols and network services, but it hasn’t kept up with compute innovation. As data centers pack more compute power and operating systems (OS) per physical server, thanks to virtualization, the need to move containers of OS plus applications and data around have sky rocked. In addition, traffic patterns have shifted tremendously as client-server or north-south flows are layered on top of server-server or east-west flows. And yes, there are new networking approaches being offered by vendors and standard organizations such as Cisco’s FlexPath, Juniper’s Stratus, Brocades VCS, Extreme’s Direct Attach, Force 10’s Open Automation, Arista’s Multi-Chassis Link Aggregation, BLADE’s Unified FabricArchitecture, the IETF’s TRILL and LISP and IEEE’s 802.1AQ, but these may be short term solutions to a much bigger networking problem.

Computing has always driven network design as mainframes drove SNA and analog multi-point wide area networks (WANs) during the ’70s. Mini-computers drove peer-to-peer networking protocols like DecNet, OSI and TCP/IP in the ’80s. Client-Server computing drove LANs and TCP into the mainstream in the early ’90s. The Web drove the internet in the 2000s and now server virtualization and cloud computing is once again changing fundamental networking requirements to make them more flexible.

The rigid label is a powerful one as it creates frustration by not addressing or enabling new business processes. Every time a network protocol or architecture was labeled as too rigid it was replaced and in the process a new market emerged on the scale of tens of billions of dollars. SNA was labeled as too rigid to support peer-to-peer networking. The T1 multiplexer market of the late ’80s and early ’90s was too rigid to support data traffic and thus routing replaced it. The PSTN and TDM were too rigid as they doled out bandwidth in 56Kbs chunks and were unable to support internet and VoIP traffic. The national entertainment network is rigid too as it doesn’t support two-way communications and it also will be replaced slowly but surely.

So where is networking not flexible enough? It’s in virtualized data centers. Some analyst groups estimate that 30% of workloads are virtualized and increasing. Since virtualization or a VM is the new atomic layer of data centers, networking is falling short in public as well as private clouds. Ideally, all resources (compute, storage, and networking) would be pooled, with services dynamically drawing from the pools to meet demand. Virtualization techniques have succeeded in enabling processes to be moved between machines, but constraints in the data center network continue to create barriers that prevent agility, for example, VLANs, ACLs, broadcast domains, Load Balancers, Firewall/IPS Security settings and service-specific network engineering.

The well understood problem is that when a VM is moved from one physical machine to another the network, load balancers, firewalls/IPS, broadcast domains, etc., have to be reconfigured. There is no automation in place, meaning that the network is not flexible or agile enough to make the changes required. Now this problem has scale to it as it’s a growing requirement of both IT executives managing corporate IT assets and service/cloud providers.

There are market solutions available today and more are coming that address “network automation” which enable the network to reconfigure itself as a VM and/or workload is moved within a data center. Cisco’s Nexus 1000V, HP Network Automation software and its Virtual Connect approach, Force 10’s Open Automation, Blade Network Technologies VMReady Network Virtualization, Arista Network’s Virtualized Extensible Operating System or vEOS and others are addressing the problem of network agility or lack thereof in virtualized environments.

But the problem gets bigger and more complex when distance and cloud provider entities become engaged. None of the solutions above address moving a VM from one physical server to another over large distance, be it around town, across state lines, across the country or the globe. Some are using IF-MAP as a registry, sort of like facebook for computers that publish their resources and use this information to automate network configuration to support large distance VM moves.

The problem gets larger yet when workloads move from a private cloud to a public cloud. (Definition note: There is no single definition of a workload, so for my purpose here I assume a container including a VM and associated applications and data that can be moved as simply as drag and drop or some other string of instructions). In short, all the software that is needed to compile and run an application for a set of users is a workload. The network inflexibility problem grows even larger when moving workloads between public clouds.

Now is this a real problem? You bet it is. Consider the value also of portable or mobile workloads to Enterprise and service providers. Workload mobility means capacity on demand, business continuance, and disaster recovery, etc. In addition, as IT leaders explore public and private cloud alternatives, they will want to move workloads from their data center to a provider’s and move the workload back when and if required. For reasons of security and trust, IT business leaders will demand mobility. For example, if your cloud provider goes bankrupt, then you will want to move your workload out quickly. If your cloud provider’s performance drops again then you could move your workload out. If your cloud provider is the target of a terrorist attack or is turned into a large botnet then you can move your workload out.

In addition to security and piece of mind, mobile workloads will fundamentally change IT delivery, capital structure and most importantly business models and processes. Once IT can move workload anywhere in their data center, across their data centers or to a provider they have tiered with, the question becomes when and how fast does IT move workload? If IT can perform all the provisioning in software and enable workload moves to occur transparently and safely with address, identity, security preservation, enabled trust, control and interoperability across providers, then the question is when does IT need to move workload? This level of mobility is an industry-wide initiative as it offers significant and material business value. Business value is created as IT could move workload in a follow- the-sun model, following the lowest cost per kilowatt-hour model; workload could move to avoid a disaster, or for capacity on demand, or for lowest cost of workload execution, etc.

So how can data center networks become more flexible? A key element of the solution is agility or the ability to dynamically grow and shrink resources to meet demand and to draw those resources from the most optimal location. Today, the network stands as a barrier to agility and increases the fragmentation of resources, which leads to low server utilization and prevents portable or mobile workloads.

Tweet

In Lippis Report 151: A Two or Three Tier High-End Data Center Ethernet Fabric Architecture? we detailed the new two tier data center Ethernet fabric that is becoming conventional wisdom amongst business leaders of high end data centers and cloud computing service providers. The networking industry is headed for a major innovation and competitive cycle fueled by a multi-billion dollar addressable market for data center network fabrics. Over the last eighteen months, every major Ethernet infrastructure provider has announced or taken a position on two tier network fabrics for high-end data centers. Companies such as Cisco, Arista Networks, Force10, Voltaire, HP/3Com, Juniper, Extreme, Brocade, BLADE Network Technology, et al have announced network fabrics for data centers with two thousand and more servers that either support storage enablement or not. In this Lippis Report Research Note, we review why it is Ethernet that will be the network fabric of high performance computing or HPC and cloud computing deployments.

For high-end data centers, HPC plus private and public cloud computing networks connecting thousands of servers, a new set of requirements have emerged. Low latency and high performance are the two driving requirements. Yes, there are more, especially when the fabric needs to enable converged storage, but let’s focus on latency and performance for now. Traditional three tier (server access, distribution and core) fabrics designed primarily for north-south traffic flows, that is client-server computing utilized spanning tree protocol (STP) and slower speed Ethernet (100Mbs to 1Gbs). Thanks to web 2.0, mash-ups and social networking sites east-to-west or server-server traffic flows have spiked requiring networks to support both north-south and east-west flows.

As most network engineers know, STP was designed to avoid loops that confused Ethernet as it was designed as a bus topology. STP shuts down redundant links between common switches to maintain the bus. Therefore, connecting access switches to distribution switches utilizing STP would require that network engineers over-subscribe the links between switches as only half of the bandwidth could be used. Oversubscription would also create blocking of packets between points too. To avoid this design, nearly every major switch manufacturer offered link aggregation that is the ability to shut off STP and aggregate links between switches. While this was and is a benefit, the down side has been that vendors only offered the ability to aggregate two links, which still drove oversubscription and blocking.

Recently, industry players such as Cisco and Arista Networks have offered the ability to scale up aggregation of links from 16 to 32, while at the same time delivering multipathing that allows packets to be forwarded across multiple links to arrive at its intended destination. Switch-processing capacity to support these massive inter-switch links have been increased too. These design changes, along with Ethernet’s innovation march, has ushered in the two-tier network design fabric option.

A two-tier fabric is designed with two kinds of switches; one that connects servers and the second that connect switches creating a non-blocking, low latency fabric. We use the terms ‘leaf’ switch to denote server connecting switches and ‘spine’ to denote switches that connect leaf switches. Together a leaf and spin architecture create the network fabric.

In late June 2010, Cisco announced its’ FabricPatch Switching System or FSS and its’ F-Series modules that support 32 ports of 10GbE of auto-sensing 1/10GbE and is essentially for server access and aggregation. FabricPath provides a new level of bandwidth scale to connect Nexus switches and delivers a new fabric design option with unique attributes for IT architects and designers. FabricPath is a NX-OS innovation, meaning that its’ capabilities are embedded within the NX-OS network OS for the data center. FabricPath essentially is multipath Ethernet; a scheme that provides high-throughput, reduced and more deterministic latency, and greater resiliency compared to traditional Ethernet.

FabricPath combines today’s layer 2 or Ethernet networking attributes and enhances it with layer 3 capabilities. In short, FabricPath brings some of the capabilities available in routing into a traditional switching context. For example, FabricPath offers the benefits of layer 2 switching such as low cost, easy configuration and workload flexibility. What this means is that when IT needs to move VMs and/or applications around the data center to different physical locations, it can do so in a simple and straightforward manner without requiring VLAN, IP address and other network reconfiguration. In essence, FabricPath delivers plug and play capability, which has been an early design attribute of Ethernet. Further, large broadcast domains and storms inherent in layer 2 networks that occurred during the mid 1990s have been mitigated with technologies such as VLAN pruning, Reverse Path Forwarding, Time-to-Live, etc.

The layer 3 capabilities added to FabricPath deliver scalable bandwidth allowing IT architects to build much larger layer 2 networks with very high cross-sectional bandwidth eliminating the need for oversubscription. In addition, FabricPath affords high availability as it eliminates STP, which only allows one path and blocks all others, and replaces it with multiple paths between endpoints within the data center. This offers increased redundancy as traffic has multiple paths in which to reach its final destination.

FabricPath employs routing techniques such as building a route table of different nodes in a network. It possesses a routing protocol, which calculates paths that packets can traverse through the network. What is being added to FabricPath is the ability for the control plane or the routing protocols to know the topology of the network and choose different routes for traffic to flow. Not only can FabricPath choose different routes, it can use multiple routes simultaneously so traffic can span across multiple routes at once. These layer 3 features enable FabricPath to use all links between switches to pass traffic as STP is no longer used and would shut down redundant links to eliminate loops. Therefore, this would yield incremental levels of resiliency and bandwidth capacity, which is paramount as compute and virtualization density continue to raise driving scale requirements up.

Designing A 160 Tbps Data Center Fabric

As an example to how multi link aggregation, the elimination of STP, high switching capacity and 10GbE connections create a highly scalable two-tier layer 2 Ethernet fabric, we use Cisco’s FSS and its’ F-Series module in the Nexus 7000. The following details the design of a 160 Tbps switching fabric with FabricPath and the F-Series module for high performance data centers using Cisco’s Nexus 7000 switches. This architecture can support over 8,000 servers connected at 10GbE or 4,000 servers dual homed at 10GbE with attributes of being non-blocking, low latency (5 microseconds), high bandwidth, reliability, plus simplicity of workload movement.

To build a 160 Tbps two-tier fabric, thirty-two Nexus 7018 switches populated with F-Series 10GbE modules would connect servers. These thirty switches are leaf switches. Each leaf chassis provides 256 10GbE ports to connect servers and another 256 10GbE ports to connect into spine switches. Therefore, each leaf is directly connected to each spine with sixteen FabricPath ports at 10GbE equaling a total of 256 10GbE ports for each leaf switch. There are sixteen spine switches each accepting 512 10GbE FabricPath ports. A single leaf chassis connects 256 10GbE ports into a spine equaling approximately 2.5Tbs. Multiplying each thirty-two leaf’s contribution into the fabric yields 80Tbs. As Ethernet is full-duplex, the total fabric switching capacity is 160
Tbps. Therefore, 160Tbps of switching fabric is available across all thirty-two leaf chassis. As 256 10GbE equals 2.5 Tbs, which also equals 16 FabricPath links to each one of sixteen spine switches, yields 2.5 Tbs, the fabric is non-blocking.

As for layer 2 and layer 3 forwarding, the job of the spine is to forward packets from leaf switches at layer 2, creating a single tier fabric. A key attribute of this architecture is that each 16-way FabricPath links are Equal Cost Multipathing or ECMP. What 16-way FabricPath ECMP provides are two benefits: 1) It delivers more paths for traffic to flow, which increases available bandwidth in the fabric and 2) as they’re distributed across all switches, diversity of routes is enabled to distribute packet forwarding. In essence what 16-way FabricPath ECMP provides is a very low latency, high bandwidth approach to supporting both north-to-south and east-to-west traffic flows simultaneously.

While the above is a Cisco deployment example Arista’s new 7500 series of Ethernet switches support 6 Billion packets per second at wire speed. The 7500s can be configured into a massive two-tier network fabric thanks to it support of 32 port MLAG (Multi-Chassis Link Aggregation) affording the connection of 18,000 to 30,000 servers.

Ethernet continues to evolve. The IEEE recently ratified the 40 and 100 GbE standard with vendors such as Force 10, Cisco, Arista, Extreme, BLADE, Brocade, Voltaire, HP et al announcing support and scheduling product delivery. While the above two-tier network example provides the perspective from the large switch provider, below is BLADE Network Technologies perspective, a company focused on server connectivity.

BLADE Network Technologies believes that as Ethernet delivers new levels of speed and intelligence, it will be the dominant two-tier network fabric for high-end next-generation data centers.
For many applications, low latency is a key requirement, and latency is an area where two-tier networks excel. Studies of stock trading exchanges have shown that tens of milliseconds of delay in data delivery can represent a ten percent drop in revenues, and delays of even five microseconds per trade can cost hundreds of thousands of dollars. Industry-specific requirements for uncompressed data and end-to-end deterministic latency within tens of microseconds make attaining such performance even more difficult. These factors have combined to make raw switching speed a top priority, and today’s best-of-breed 10 Gigabit Ethernet switches achieve can operate with under 700 nanoseconds of port-to-port latency while consuming a miniscule amount of power equivalent to that of standard light bulbs.

As next-generation networks get flatter – driven by latency and bandwidth requirements – emerging Layer 2 technologies such as the IETF’s Transparent Interconnection of Lots of Links or TRILL, enable this trend. The idea behind TRILL is to replace spanning tree as a mechanism to find loop free trees within Layer 2 broadcast domains. Using a routing protocol to build forwarding trees within a Layer 2 broadcast domain enables the flexibility and efficiency to route Layer 2 traffic, just like one would Layer 3 traffic, without the overhead associated with Layer 3 packet processing. TRILL will offer important features, such as support for both broadcast and multicast, load splitting along multiples paths, support for multiple points of attachment, and no tangible delay in service after attachment.

In the data center, bottlenecks are moving from the CPU and memory access to the I/O of the servers. Today’s multi-core servers are now able to sustain a great amount of traffic, requiring fast, flat networks, especially now that virtualization is widely deployed. Analysts have predicted that the 10G market will double year-to-year in 2010 and 2011. More servers using 10G increases the requirement for 40G and 100G in upstream networks. With 10G widely available and 40G coming online, Ethernet networks can enable data and storage traffic to use a single wire, using FCoE or iSCSI for example, and provide the raw speed that makes Ethernet with its economies of scale, to supplant InfiniBand for HPC requirements.

The reason Ethernet will be the network fabric for high-end data center networks is that the vendor community continues to innovate and build upon this protocol. Ethernet innovations are many and are beyond bandwidth increases from 10Mbs, 100Mbs, 1Gbs, 10Gbs, 40Gbs and 100Gbs, which are obvious. Link aggregation, multi-pathing and so much more propel Ethernet’s relevance and suitability to new challenging networking requirements.

Tweet

In Lippis Report 148 we reviewed the major drivers and trends that are propelling the high-end data center Ethernet switch market to well over a $1B annual run rate. In this Lippis Report Research Note, we review the major suppliers of these switches. We review Cisco, Arista Networks Force10 Networks, BLADE Network Technologies, HP/3Com/H3C, Voltaire, Avaya, Brocade, and Juniper and identify their unique positions and offerings to participants in the burgeoning market. Our focus is the high-end, high density 10GbE switches that are enabling virtualized cloud computing data centers thanks to Terabits per second of back plane switching capacity, billions of packets per second of layer 2/3 forwarding, hundreds of 10GbE port connectivity per chassis, a new two-tier architecture, microsecond level latency, low power consumption, non-stop operation and software hooks that eliminate network barriers to large scale server virtualization. The engineering in these switches should be celebrated, as they represent the state-of-the-art in computer and network design. In short, they represent the fundamental building block of a new generation of IT delivery based upon cloud computing and virtualization. This Research Note is a must read for any IT executive designing a data center.

After finishing this Research Note, it became evident that this market needs a set of industry neural 10GbE switch test to independently verify vendor claims. We hope to make such a contribution this Fall.

Cisco Systems Nexus Family of Switches

Cisco’s approach to data center Ethernet switching is rooted in its Data Center 3.0 strategy which seeks to scale server virtualization while introducing a platform to enable a unified fabric or converged network and storage running on one physical Ethernet network. Cisco’s data center Ethernet switch portfolio is primarily the Nexus family of switches including the 7000, 5000, 2000 and 1000v. NX-OS is a purpose built data center operating system that runs across the entire Nexus family. NX-OS integrates a number of higher system availability functionalities such as virtual port- channel (vPC), and the capability to upgrade software without disrupting traffic. The Nexus 1000v is a softswitch that resides in a VM hypervisor. The Nexus 1000v’s main job is to eliminate network configuration barriers that exist when moving a VM from one physical machine to another. To accomplish this, the 1000v creates a port profile including VLAN, ACL, policy, security, etc. with persistence, which moves with a VM as a virtualization administer moves a VM from one physical machine to another.

The Nexus 2000 family of Fabric Extenders (FEX) introduces the concept of a remote line card of the parent Nexus 5000 switches and sits on the top-of-rack connecting servers to the switch fabric. The extender concept allows the 2000 and 5000 to be managed as one switch. This configuration reduces cabling requirements and offers an economical approach to server connection, thus providing the benefits of both end-of-row and top-of-rack deployments. The Nexus 5000 Series is 10 Gb Ethernet and Unified Fabric capable switches, connecting Nexus 2000s and servers directly at 100/1/10GbE/FCoE, while providing layer 2 forwarding. Providing layer 3 forwarding, dense 1/10GbE connectivity is the Nexus 7000 Series. The Nexus 7000 Series is available in a 10 and 18 slot chassis and is Cisco’s flagship data center Ethernet switch series. As a point of reference, the Nexus 7000 is now on an annualized run rate of $1B for Cisco, which is more than 10 times greater than any other switch supplier in the data center switch market. The high end 7000 connects 512 10GbE ports with 128 line-rate 10 Gigabit Ethernet ports. The Nexus 7000 Series switches can be segmented into virtual devices, delivering true segmentation of network traffic, context-level fault isolation, and management through the creation of independent hardware and software partitions. Overlay Virtualization Transport (OTV) provides customers a simplified DCI solution by extending layer 2 VLANs over existing IP networks. We have profiled the Nexus 7000 when first released and is available here. The Nexus switches can create a two-tier architecture with the 2000/5000, providing server connectivity and layer 2 forwarding between servers. The Nexus 7000 connects the 2000/5000 to each other and the internet/intranet with high density, high reliability layer 2/3 forwarding.

Arista Networks 7500 Family of Modular Switches

Arista Networks is a new comer to the data center Ethernet market, but its management team is seasoned and customer base growing. It provides six fixed 10GbE switches; five 1/10GbE 7100 and the 1GbE 7048 along with the new Best of Interop awarding winning 7500 modular switch. The 7100/7048 switches connect servers in a Top-of-Rack configuration while the 7500 aggregates these switches and connects them to the internet and intranet. This is a two-tier, “leaf-spine” architecture. The 7500 boasts ultra high performance layer 2/3 1/10 Gb Ethernet switching for high performance computing and cloud computing data centers. The 7500 supports 384 10GbE ports, 5.7Bpps at layer 2 or 3, high packet buffers 18GB deep, ultra low port-port latency of 4.5 microseconds and 10Terabit loss less switch fabric connecting modules.

The 7500 is 10GbE port dense, compact, cloud spec fast, green and prepared for 40 and 100GbE, with a price tag 50% below competitive offerings, according to Arista. While the 7500’s hardware architecture is impressive, its operating system EOS, Extensible Operating System, offers another set of uniqueness. For example, all Arista switches run the same binary image of EOS, easing administration while hastening switch feature upgrades. EOS is a modular OS that allows partners to run their software in the Arista switch, consolidating the number of management and network appliances required, thus increasing performance while reducing energy consumption and physical space. Arista’s EOS modularity was designed as a unique state sharing architecture that separates switch state from protocol processing and application logic. EOS is built on top of a standard Linux kernel. All EOS processes run in their own protected memory space and exchange state through an in-memory database. This multi-process state sharing architecture provides the foundation for in-service-software updates and self-healing resiliency. You can listen to a podcast interview with Douglas Gourlay, VP Marketing and Anshul Sadana, VP Customer & Systems Engineering from Arista on the introduction of the 7500 Series of Ethernet switches here

HP/3Com/H3C’s A12500 Core Data Center Switches

HP has spent 25 years building and selling networking products to its worldwide client base and is currently #2 in the market, with a 21% port count share and the fastest growing networking company in the industry. The combined HP/3COM acquisition brings core switching products, the #1 market share position in China, TippingPoint Intrusion Prevention System and ProCurve edge switches, representing a new choice for clients who are frustrated by today’s current offerings. HP will combine these two entities and operate under the banner of “HP Networking.”

The HP Converged Infrastructure Architecture and FlexFabric blueprint approach the modern data center with a vision that places networking at the center of an integrated data center solution and accelerates deployment of enterprise services and applications. It is designed to drive simplicity through streamlined network designs and centralized management, enhance agility with high performance security, and accelerated provisioning, and reduce cost with energy efficiency and low total cost of ownership. Central to HP FlexFabric is policy-driven network provisioning tightly integrated with server and storage management in an end-to-end data center converged infrastructure.

HP data center solutions are purpose built, using the latest advanced systems and ASIC technologies. “A” family data center networking platforms leverage a common operating system, Comware™ and are managed with a single-pane manager, Intelligent Management Center (IMC). HP switches make use of an HP-developed technology – Intelligent Resilient Framework (IRF) – to create a resilient virtual switching fabric. IRF delivers geographic independence, distributed high-availability, resiliency and millisecond re-convergence across layer 2 and layer 3 protocols. These innovations allow customers to build a simplified, high performing, highly resilient and flat (two-tier) data center network design. They overcome the limitations of low performance/scale, high cost/latency inherent in legacy solutions, which rely on multi-tier network designs, disjointed platform operating systems and complex resiliency protocols.

A key enabler of this transformational design flexibly is the HP next-generation data center switching architecture. This starts with the flagship HP A12500 core data center switch – which is based on a 100G design that uses a multi-level, multi-plane, non-blocking switching architecture to provide high performance and scalability. The A12500 supports 6.66 Tbps of high-performance switching capacity (future support for 13.32 Tbps) and scales to 2.2 billion packets per second of forwarding performance. The A12518 supports 512 10 Gigabit Ethernet or 864 Gigabit Ethernet ports in a single chassis. Its future-proof design accommodates 40/100 Gigabit Ethernet and emerging unified network requirements such as end-to-end FCoE/Data Center Ethernet.

Force10 Networks ExaScale E Series

Force10 Networks was one, if not the first company to offer 1 and 10Gb switching solutions for high-performance computing and data center markets in Fortune 100 companies, Internet portals, global carriers, leading research laboratories and government organizations. It offers a wide range of Ethernet switching and routing products that deliver high port density and resiliency to help customers deploy a high-availability, agile and standards-based GbE and 10 GbE network fabric, while reducing power and cooling costs. Its Ethernet switching products are designed to leverage virtualized data center environments and automate Ethernet networking. For example, its VirtualScale enables management of virtual chassis. Its VirtualControl enables virtualizing logical switching and routing boundaries. For automation, Force10 has developed an architecture, which automates network resource allocation as applications and services spin up and down. This architecture is built upon its HyperLink and SwitchLink technology, two new software features implemented within its Force10 Operating System (FTOS). HyperLink provides real-time communication between Force10 switches and hypervisors or virtual switches to enable automatic provisioning of one or many virtual LANs (VLANs) across multiple switches simultaneously. The SwitchLink feature provides real-time communication with middleware orchestration tools to enable automatic provisioning and management of virtual devices anywhere in the network.

Force10’s modular Ethernet switch data center product portfolio includes the ExaScale E-Series, optimized for core deployments in large-scale, high-performance 10GbE data centers, and the C-Series, optimized for mid-range data centers. Both the E-Series and C-Series come in multiple form factors, run FTOS and are dense high performance switching platforms equipped with redundancy, availability, fault-tolerant operations and many line card options. In addition, Force10 offers the fixed configuration S-Series product line for GbE and 10 GbE ToR configurations. Force10 promotes a vision of simplified data center topologies, using integrated switching and routing in the core, using chassis based E-Series or C-Series products, and fixed configuration ToR access products allowing both 1 tier and 2 tier designs. One tier can be achieved with high density E-Series platform for server aggregation, switching at the server edge, and routing off the same platform to the Internet / WAN. The two-tier architecture can be achieved leveraging ToR switching for server aggregation along with Force10’s chassis based systems in the core. In addition to a large direct sales force, IBM OEM’s Force10’s ExaScale platform as part of IBM’s iDataPlex clustering solution. You can listen to a podcast interview with Steve Garrison, VP Marketing of Force10 on their 40 GbE offering here.

BLADE Network Technologies RackSwitch Family of Ethernet Switches

BLADE Network Technologies (BNT) has been working in the data center switch market since 2006 with much success providing 1/10Gb Ethernet switches for blade servers and top-of-rack configurations. BLADE was launched from Nortel and made up of the successful Alteon Networks group. Their success stems from their ability to identify the top-of-rack and blade switch market in ’06, along with an OEM go to market strategy that included all of the top tier blade server providers such as HP, IBM and NEC. The result is that BLADE has shipped over 8m ports, achieved 25% growth from 2008 to 2009 (in a down economy), owns 50+ % of the blade switch market, is number 3 in the Fixed 10GbE market according to Dell’Oro Group, and has demonstrated scale with at least one customer installing over 16,000 of its switches.

BLADE offers the RackSwitch family of Ethernet switches, which are ToR, 1U high switches. They include the 24-port 360ns latency RackSwitch G8100 10GbE, 48-port RackSwitch G8000 1/10 GbE aggregation and the 24-port 700ns latency RackSwitch G8124 10GbE. Over a year ago, BLADE released its virtualization software called VMready that automates network settings for VM movement ensuring that network settings migrate when a VM is moved from one physical server to another. VMready scales to a 1000 virtual port switch, is based on standards and works with most popular hypervisors.

In addition to VMready, RackSwitch’s unique attributes are found in the fact that they were designed for the data center versus being a wiring closet switch re-formatted for the data center. For example, the RackSwitch BLADEOS supports CEE for unified fabrics, uplink failure detection, virtualization, dual homing for servers, low (80-170Watts) power consumption, back-to-front or front-to-back airflow and very low latency in the 700-360 nanosecond range.

Voltaire’s Vantage 8500

Voltaire has a long history in high performance computing and data center networking as it is one of the key leaders in the InfiniBand market. Voltaire enjoys distribution relationships with HP and IBM, as well as Bull, Fujitsu, NEC, SGI and Oracle. The result is a 100% + year over year revenue growth for Q1 as reported on May 5th. Last October, Voltaire entered the 10 GbE market with the introduction of its Vantage 8500 Ethernet layer 2-core switch. The Vantage 8500 boasts less than 1 microsecond of latency, a low 10 watts per port power consumption and 288 wire speed 10GbE ports in a 15U high chassis. The Vantage 8500’s unique industry contribution is that it’s based on converged enhanced Ethernet (CEE) technology providing InfiniBand-like capabilities to the Ethernet data center. In fact, Voltaire has ported many of InfiniBand’s key characteristics to the Vantage 8500 such as a lossless switching fabric, multi-pathing, virtualization, fabric-wide congestion management and QoS.

From a network design point of view, Voltaire supports a two tier network architecture that enables a simplified, ‘flat’ data center network and puts an end to the era of the over-provisioned network. Voltaire’s design centered on the Vantage 8500 is to support a two-tier data center network that scales from hundreds to a few thousand core ports, which requires high capacity, non-blocking 10 Gigabit Ethernet core switches. By clustering up to twelve Vantage 8500 switches together, IT business leaders can expand their data center to many thousands of servers while preserving the efficiency and price-per-port, without degrading performance or latency which occurs in traditional hierarchical network designs. To support ToR implementations, Voltaire and BLADE Network Technologies announced recently a partnership where BLADE ToR RackSwitches are aggregated by Voltaire’s Vantage 8500, rounding out the two-tier data center Ethernet network architecture.

The Vantage 8500 also features software-based capabilities to address virtualized and converged data center environments. Voltaire’s Unified Fabric Manager™ (UFM) software, application acceleration software and management OS (VT-OS) provide management and performance enhancement tools. These tools were developed and optimized in InfiniBand environments and are now available for Ethernet-based data centers. Voltaire’s recently introduced Unified Fabric Manager™ (UFM™) 3.0 software orchestrates physical and virtual switches delivering guaranteed levels of service per application. It’s the first and only Ethernet fabric management software that dynamically orchestrates end-to-end virtual machine connectivity for multi-vendor, scale-out data center networks.

Avaya’s VSP 9000

During the April 2009 Las Vegas Interop trade show, Nortel committed to the data center Ethernet market with the announcement of its Virtual Services Platform or VSP 9000 switch, which supports up to 27 Terabits per second (Tbps) of backplane switching and 240 10GbE ports per chassis at first release. Avaya announced their commitment to the VSP 9000 and said that it will be generally available in the second half of 2010 while already in controlled availability. The VSP 9000 is built upon the Ethernet Routing Switch 8600/8800 software providing a proven software foundation, mid-plane architecture, a fully programmable network processor unit for flexible data forwarding and carrier-grade Linux.

The VSP 9000 is designed to deliver high-density 10GbE, 40GbE and 100GbE. Its design center is rooted in highly dense connectivity environments that are all mission critical, by definition. Early testing validation of the VSP 9000 promises to provide ultra-high reliability and availability delivering below 50ms failover support, which is critical to eliminate application disruption thanks to its patented hardware failure detection differentiation. The VSP 9000 switch fabrics are lossless Ethernet capable and therefore well positioned to support the next generation Data Center requirements for convergence of storage onto the Ethernet infrastructure.

The VSP 9000’s unique network architecture is found in its ability to cluster four switches together, in that the total architecture exceeds 100 Tbs, with the number of 10GbE ports per rack being up to 720. Avaya continues to invest in Switch Clustering technology (Active/Active resiliency model) such as SMLT (split multi-link trunking) and RSMLT (routed-SMLT), which provides link, switch and router redundancy mechanisms. Three modules are being introduced in the first VSP 9000 release, a 24 port SFP+ for 1 GbE and 10 GbE connectivity, a 48-port of SFP module in addition to a 48-port 10/100/1000 TX module. Future plans include 40GbE and 100GbE interfaces, and even higher-capacity Switch Fabric modules.

Juniper Networks’s EX8200 & EX4500

In January of 2008, Juniper Networks launched its much-anticipated entry into the enterprise Ethernet switch market. Juniper’s focus is on the enterprise data center, campus and branch, as well as the service provider market. Juniper provides a suite of Ethernet switch products, including the EX4200 with Virtual Chassis technology for GbE Top-of-Rack (ToR) and End-of-Row (EoR) data center access, the EX2500 24-port and new EX4500 48-port 10GbE ToR switches, and the EX8200 high-density, high-performance line of modular Ethernet switches.

According to Juniper, it simplifies customer enterprise LAN architectures and advances the economics of networking via its most recently launched initiative called the “new network” for data centers. Juniper’s “new network” promises critical innovations in automation, virtualization and fabric technologies. These innovations are to reduce time to operation by up to 50 percent and eliminate up to 35 percent of data center networking capital expenditures. One aspect of the “new network” is a simplified two-tier network architecture, which may be reduced to one when “Project Stratus” is completed with IBM. The reduction of a three-tier architecture to two is accomplished by utilizing Juniper’s Virtual Chassis fabric technology in the access layer, in conjunction with its high-density, high-performance platforms such as EX8200 and EX4500 in the LAN core, thus eliminating the aggregation or distribution layer. According to Juniper, collapsing the distribution layer reduces complexity in the data center as well as campus networks by reducing the number of managed devices by up to 89%, providing up to 39% savings in space, 44% savings in power and reducing the number of switch interactions by up to 99% compared to three-layer networks. According to Juniper, this approach improves application performance by also reducing latency up to 77% compared to three-layer networks. Note that these claims and numbers are Juniper’s and not mine.

At the core of Juniper’s data center Ethernet product family is the EX8200 line of modular switches. The EX8208 and EX8216 are eight and sixteen-slot modular switches. The EX8216 sports a maximum of 640 10GbE ports and 1.92Bpps and 6.2Tbps backplane speed. The EX8200 is said to support 40GbE and 100GbE interfaces in the future. The EX8200s connect either EX4200 GbE or EX2500 and EX4500 10GbE ToR switches together while providing access to internet/intranet. All Juniper switches run Junos, the network operating system that provides reliability and availability features, developed for the high-performance enterprise and service provider market.

Brocade’s NetIron MLX Series of Switches

In July of 2008, Brocade had purchased Foundry Networks, catapulting them into the Ethernet switch market as one of the top five Ethernet switch/router vendors by revenue. Brocade, with its long history of data center storage, saw that converged I/O was going to happen and prepared the company to participate in this market. At the high end of Brocade’s data center Ethernet switch products is the NetIron MLX-4, MLX-8, MLX-16 and MLX-32 routers, which support 4, 8, 16 and 32 I/O module slots, respectively. We’ll focus on the high end NetIron MLX-32 here, which has been in production since August 2006.

The NetIron MLX-32 boasts a total of fully redundant non-blocking 7.68 Tbps switch fabric capacity. Brocade says that the MLX-32 can forward some 2.284 Bpps of Layer 2/3 packets and support 1,536 and 256 non-blocking 1 GbE and 10 GbE ports, respectively. Note that the new high density 10 GbE was announced the same day as this Research Note was made public. All four NetIron MLX systems are designed for non-stop operation, supporting 1:1 management module redundancy, N+1 switch module redundancy, M+N power module redundancy and N+1 fan redundancy. The NetIron MLX architecture is an adaptive self-routing Clos switch fabric with a virtual output queue (VOQ) design. This non-blocking architecture is optimized for maximum throughput and low latency for all packet sizes.

Tweet

By Arista Networks

For many years, networks have been oversubscribed. Such oversubscription was tolerable in enterprise networks as the applications were not bandwidth intensive, but not anymore. In a datacenter, the primary reason for this oversubscription is insufficient uplink bandwidth from each rack. Each top-of-rack switch is typically connected to two aggregation switches for redundancy. However, half the uplinks are blocked by spanning tree to avoid loops in the network; this reduces the available bandwidth between the rack and aggregation layer of the network by 50%. Arista’s Multi-Chassis Link Aggregation (MLAG) feature removes this bottleneck and allows the utilization of all interconnects.

To learn how, download this paper from Arista Networks.

Get the White Paper

Tweet

During last week’s Cisco Q3 FY10 quarterly financial conference call, John Chambers, Cisco’s CEO, said something that impressed and shocked me. The company has been quiet about the growth rates for its Nexus line of data center switches until this call. What shocked me was that the Nexus 7000 is now on an annualized run rate of $1B, yes that’s Billion with a B! I remember being interviewed by John Markoff of the NY Times in Jan ’08 about the Cisco’s Nexus and Juniper’s yet to be announced Ethernet switches. In just 27 short months, the Nexus product line including the 7000, 5000 and 2000 represents a $1.4 B run rate of revenue to Cisco. Another insight gained from this ramp up is that the data center networking trends that we’ve discussed here in various Lippis Report Research Notes are powerful demand drivers for Cisco and other companies participating in this lucrative emerging market and its just starting! Companies such as Arista Networks, Force10 Networks, Blade Network Technologies, HP/3Com/H3C, Voltaire, Avaya, Brocade, Juniper, et al, have unique positions and offerings to participants in the burgeoning market. In this Lippis Report Research Note, we review the mega trends driving high market growth. We save a product review of each of the suppliers for our next Lippis Report Research Note.

In addition to the run rate numbers above, Cisco also posted a milestone of 1 million 10 GbE ports shipped, providing a strong indicator that the 10GbE market is nearing a tipping point to high volume, as pricing drops and its use accelerates. The following are mega trends driving this tremendous market growth. Traffic demand drives bandwidth and that’s the first mega trend.

Traffic Profile Changes: Gone are the days when data center networks primarily shuffle asymmetric email messages and low bandwidth client-server computing applications between endpoints and servers. Best effort data delivery, where latency was secondary to delivering data accurately, has changed to being a paramount design element where 10 milliseconds means the difference between losing a customer or capturing revenue. Traffic is now highly mixed, moving around a data center in near Brownian motion between servers, storage, internet and intranet thanks to a plethora of old and new applications such as mash-ups, VoIP, search, backups, storage access, emerging converged I/O etc. In addition to Brownian motion traffic flows and low latency requirements, the volume of traffic continues to skyrocket and shows no sign of abating. Remember when the Dow dropped by 1000 points in early May of this year? Financial services firms saw an average of 40 times the amount of traffic in their data centers as traders responded to the drop. There is no better driver for traffic volume as financial markets in turmoil. The traditional model of over subscribing data center bandwidth by as much as 80:1 is the norm, and IT business leaders are looking for a more efficient model.

Workload Mobility: With the advent of server virtualization IT leaders are able to decouple an operating system from its underlying server hardware and increase the number of instances an operating system can be replicated on a single server. Server virtualization reduced the number of physical servers needed and in the process reduced energy and cooling requirements. Now that an operating system only needs to know which hypervisor it’s running on, that operating system instance and the applications it services can be moved from one physical server to another in near real-time with the click of a mouse, thus providing workload mobility or portability as well as a rapid application procurement tool.

So what does all of this have to do with networking? A lot, first moving these workloads around a data center consumes huge bandwidth and has low latency requirements to driving raw bandwidth requirements. Secondary, and most importantly to the industry, is that networking or should I say the rigid structure of IP addressing/VLANs, etc are impeding the automation of these workload moves. In short, the data center network needs to be reconfigured when VMs are moved from one physical server to the next in the same data center and it simply does not work if a VM is moved between data centers separated over distance, between a data center and a cloud provider and between cloud providers. This is the area of the infrastructure 2.0 working group.

Doug Goulay said it best in his recent Network World post.

“When moving VMs between machines there is a caveat:  if you want your TCP connections and IP addressing to stay intact the receiving physical host must be capable of supporting the same IP address that the VM moving to it is actively using.  This means that both physical hosts have to be in the same subnet or in the same VLAN depending which layer of the network you are looking at.  Since the largest number of physical servers that can be supported doing this is around 64 it doesn’t change the addressing architecture too much, unless the servers are in different data centers, or are connected to different access layer switches that talk to different aggregation layer switches.  If this is the case the network architecture all of a sudden starts dramatically impeding the movement of VMs:  either VM mobility is impeded, or the network is redesigned. 

Some people often ask me, “can’t I do this with DNS?’  In short, no.  DNS is cached at many client sites, ignoring your TTL.  Additionally, DNS is cached on many PCs for the life of an application session.  If you try to change the IP address of your backup server while you are in the middle of a 2GB backup do not expect the connection to continue.  TCP doesn’t work this way.”

Increased Density: It’s no secret that data centers are bursting from the seams as the economic down turn kicked large IT capital outlays down the road until economic conditions improved. Business leaders have been postponing increasing data centers space, that is square footage, while power density has grown exponentially, until very recently, as cooling requirements increase unabated. Power and cooling capacity are the primary constraints to data center expansion. To deal with these realities, IT business leaders are left with only one option, appropriate capital to either upgrade power and cooling systems or build a new data center. The impact of high energy densities is that server hardware is no longer the primary cost component of a data center. The purchase price of a new (1U) server is now exceeded by the capital cost of power and cooling infrastructure to support that server and will soon be exceeded by the lifetime energy costs alone for that server. In short, energy costs are on their way to dominate data center economics.

To help mitigate these trends, the new data center switches offer increased server connection density at lower energy consumption levels. In addition, their own energy consumption to shuffle packets around has been reduced, for some by as much as 50%. To connect an every increasing dense set of servers, new generation of data center switches boast a two tier network architecture to support thousands to tens of thousands to hundreds of thousands of servers. To deal with high server density connectivity, server access is via a leaf switch, while leaf switches and storage connect to a modular spine switch. The two-tier approach offers efficient connectivity density, low latency albeit this depends highly upon the internal switch design, and is ready to support consolidated I/O.

Consolidated I/O while early in its adoption cycle will go a long way in reducing power consumption of servers as they will have a single network interface for both storage and networking. In addition, consolidated I/O promises to reduce the need for a separate storage switch too again reducing capital, energy and cooling cost.

Back to server density. Server density will only get, well, more dense. If the industry trajectory of cloud computing is realized any where near what the conventional wisdom dictates, then there will be more and more highly dense cloud computing sites supporting an ever increasing number of enterprise, government and consumer applications. How many cloud computing sites does the US need to support all IT applications? With nearly 16 million servers installed nation wide, according to IDC, and with each cloud computing site supporting hundreds of thousands of servers, then perhaps the number of cloud computing sites would be in the hundreds. While its unrealistic that all US enterprises and governments will be hollowed out of their data centers and applications via cloud computing with today’s technology and business control believes; the trend line is clear, there will be a smaller number of very large cloud providers delivering applications to a wide range of customers. Almost like a supernova transforms into a black hole, applications will not be able to escape the gravitational pull of the scale and economics of cloud computing if the industry gets anywhere near this size scale.

The networking industry has been busy adapting to these powerful trends with new internal switching architectures, data center network architecture and automation. Internal switching architectures are being designed with high internal switching capacity in the terabit rage, lower energy consumption in the 10W/port range, low latency and of course high port density. The data center network architecture most are progressing toward is a two –tier leaf-spin approach mentioned above. These switches possess the highest levels of reliability, serviceability and redundancy, as networking is at the center of this massive server connectivity density.

Network automation is another area of investment where VMs can be moved within and between data centers, as well as between data centers and cloud providers, plus between cloud providers. A few companies are addressing network automation, but this is a huge issue that the industry needs to wrap its arms around and provide a scalable solution.

In the next Lippis Report Reseach note, we’ll review Cisco, Arista Networks, Force10 Networks, Blade Network Technologies, HP/3Com/H3C, Voltaire, Avaya, Brocade, Juniper, et al, and highlight their unique positions and offerings to participants in the burgeoning market.

Tweet

By Arista Networks

Extensibility is a system design principle where the implementation of the operating system takes into consideration future growth. It is a systemic measure of the ability to extend the operating system and the level of effort required to implement the extension. Extensions can involve the addition of new functionality or the modification of existing functionality.

Find out how Arista’s Network Operating System EoS improves system uptime and delivers rapid service restoration in the event of failure by downloading this whitepaper.

Get the White Paper

Tweet

This past Interop in Las Vegas was one of the best I have attended, since even before the economy took a noise dive in 2008. The tone and level of excitement of the industry’s growth potential was refreshingly up beat from the hundreds of IT and vendor executives I talked with. While the size of Interop is a small fraction of what it was in the late 1990s, (70k attendees with over 600 exhibitors to ~ 15K attendees with ~ 200 exhibitors) it still provides a pulse of the networking industry. In fact, Interop has come full circle, back to being a networking event even though it has added other topics. You have to give Dan Lynch credit for creating such a long lasting venue for our industry. Congratulations to Cisco, Arista Networks, HP/3Com, Mallonx for winning best of show in their respective categories and for Arista for winning Best of Interop. In this Lippis Report Research Note I provide the key industry themes that were evident at Interop this year.

The following are my observations of Interop 2010 in LV.

Network Infrastructure Takes Center Stage: Even though Interop provided attendees with thirteen educational content areas including cloud computing, IT security, Enterprise 2.0, etc., it’s the changes taking place in the network infrastructure business that was front and center, loud and clear. The following was the topic of conversations throughout Interop:

• Cisco’s introduction of its Best of Show winning Aironet 3500 Series Access Point with CleanAir technology,
• Arista Networks’ introduction of and winning Best of Show and Best of Interop for its Arista 7500 10Gb modular Ethernet cloud computing switch,
• HP’s closing of its acquisition of 3Com and winning Best of Show for its TippingPoint Virtual Controller,
• HP’s planned acquisition of Palm,
• Avaya’s reassertion in the network business with the introduction of its Ethernet Routing Switch 8800, WLAN 8100 and Advanced Gateway 2330,
• Voltaire’s new Vantage™ 8500, 10 GbE Layer 2 core Ethernet switch,
• Force10’s open network automation demonstrations and 40GbE module

With the above announcements and accomplishments, two thoughts come to mind. First is that Interop is finally back to core networking issues, and second, the above announcements provide a window into the huge changes that are taking place in our industry.

New Industry Structure Emerges: The networking industry has been consolidating for some time now and will only continue. Corporations have some $2T in cash and equivalents on their books, which will be put to work acquiring companies and investing in growth markets. The big growth market in our industry is the fundamental change IT is starting to progress through. HP’s actions last week provided a preview of what’s to come.

HP stole the headlines last week with their shorter then expected closing of their 3Com acquisition, in addition to their intent to purchase Palm. HP realizes that the IT industry is structurally changing away from fixed desktop computing accessing corporate applications hosted in data centers, to mobile computing accessing applications hosted in corporate data centers and cloud computing facilities. The big winner in this transition is networking, as without it, cloud and mobile computing will not happen. Palm gives HP a smartphone platform to participate in the mobile computing market while 3Com expands its corporate networking portfolio significantly.

HP vs Cisco: The buzz at Interop around HP was how it will compete with Cisco. The HP executives and booth personnel were the most energized I have ever seen. HP views their competitive advantage along the lines of innovation, open network architecture and economics. Thinking it through however, HP’s focus will be more on supply chain efficiencies to drive down their cost of producing networking gear close to server economics while leveraging their massive and productive channel to gain market share.

The supply chain efficiency is a great idea, but will take at least a year if not more to deliver. The thinking here is that a 40 Watt power supply is the same, independent of its final designation, as long as it powers a server, router, etc. So can HP redesign their product lines for common components where they gain huge cost efficiency thanks to volume purchasing? Perhaps, but this will take time. Their channel strength should deliver results in the short term. If HP executives are correct and that the market wants a strong number two networking provider, then its channel should produce fairly quickly. If it doesn’t, then this premise is questionable. HP networking is about $5B now; if it doesn’t grow faster then the industry by a significant amount next year, then something is wrong.

Remember HP is competing with a $40B powerhouse that is Cisco Systems, which has a massive and productive channel too that are energized to sell, not only networking gear, but also unified communications, Cisco’s new server platform UCS and video equipment. As for innovation, HP is a great operational company therefore expect them to take cost out of their products. Nevertheless, Cisco is the innovation king, thanks to its systemic incorporation of innovation in product development, plus its ability to integrate acquisitions quickly and materially. Cisco does not only innovate in its products, but around them, offering architected solutions. Examples of this are everywhere, including its borderless network architecture, EnergyWise, UCS, the new 3000 series stackables, Power over Ethernet Plus, its’ ISR G2, the Nexus line of data center switches, its’ approach to integrated network security, etc.

Here’s an example of the power of innovation. A client and Lippis Report subscriber has funded a new $20M data center. During their due diligence, they visited Dell, HP, IBM and Cisco. This CIO will go with Cisco’s UCS. The reason is that during the customer visit, Cisco first described the major direction and trends in data center virtualization and cloud computing in such a way that my client said “Cisco looked into the future and designed UCS to exploit these changes while all the other vendors were selling their old blade systems”. Now this is significant, as this CIO only purchased equipment from market share leaders, that is, he would buy from HP for servers, Dell for desktop systems, Cisco for networking, Avaya for communications etc. Cisco’s innovation in UCS changed his long-standing principal of buying only from market share leaders and will buy UCS for this new data center. So the basis of competition between Cisco and HP will fall into three categories; innovation, supply chain management and channel productivity.

A Mobile and Cloud Computing IT Model Is Disrupting The Status Quo

The Interop announcements above were aligned with this new world order of IT. For example, Arista Networks delivers a massively powerful 10GE switch for cloud spec data centers and high performance data center environments. Clearly investment in cloud infrastructure is a growth market which motivated Voltaire to enter the Ethernet market and leverage its Infiniband experience to deliver converged I/O for both Infiniband and Fiber Channel Over Ethernet (FCoE). As computing is in a rapid technology innovation stage thanks to server virtualization, networking has lagged in its ability to automate network changes brought on by VM moves. This has motivated Force10, F5 and Infoblox to demonstrate innovative approaches to automating network changes so that network administrators do not have to be involved in the process of VM moves and/or the provisioning of new IT services as demand is increased and/or decreased.

It’s clear that HP networking products has gained awareness and will receive consideration. As HP opens the consideration door, Avaya wishes to enter too with its refreshed and new data networking products. Avaya is now lead by experienced IP networking executives that understand voice and data. The Nortel channel also understands voice and data. Ever since Avaya closed its acquisition of Nortel, those channel partners that put selling Nortel gear on hold, have started to come back. They are comfortable now as stability, R&D funding and a strong financially viable company has emerged.

The networking industry is an upside down pyramid with Cisco at the top followed by a few others in the billion-dollar range. Then there are a number of $100M sized firms followed by a few start-ups. The successful firms will be the ones that embrace the new world order of IT that is being brought on as IT leaders de-emphasizes desktop computing and invest in mobile plus cloud computing.

Tweet

By Arista Networks

Can Ethernet compete with Infiniband in the low-latency trading and high performance computing markets? Administrators and IT professionals face a choice when deciding whether to invest in Infiniband or Ethernet for their low-latency networks. This paper addresses many of the characteristics of Infiniband that have made their way into Ethernet.

To learn more, download the Arista Networks whitepaper.

Get the White Paper

Tweet

Our industry is in a compute innovation cycle thanks to virtualization and cloud computing and it’s changing fundamental networking requirements and design. These changes are beyond increased packet processing performance and ultra low latency. They extend to network design and product features that enable increased server virtualization scale, workload mobility and cloud computing. I discuss a new model for networking born out of data center virtualization and cloud computing with Doug Gourlay, Vice President Marketing at Arista Networks. This is Arista Networks’ first podcast; sure to be a classic.

Enjoy, Nick

Listen to the Podcast

Tweet

Networks today require predictable performance and are much more aware of application flows than traditional networks with static addressing of devices. Enterprise networks in the past were designed for specific applications while new cloud designs in the data center can address a multitude of applications. This is clearly a radical departure from today’s oversubscribed networks in which delays and high transit latency are inherent.

To learn more, download the Arista Networks whitepaper.

Get the White Paper

Tweet

Arista’s new 7500 Series of Ethernet switches is touted as the fastest Ethernet switch on the planet. Arista Networks has been delivering ultra high performance fixed 1/10 Gb Ethernet switches for high performance computing and cloud computing data centers. But this week Arista shocked the industry by introducing a massively powerful Ethernet switch platform that is 10 GbE port dense, compact, cloud spec fast, amazingly green plus smart and prepared for 40 and 100GbE with a price tag 50% below competitive offerings. I talk with Douglas Gourlay, Vice President Marketing and Anshul Sadana Vice President, Customer & Systems Engineering both from Arista Networks about a new age of network design in the cloud-computing era.

Listen to the Podcast

Tweet

Power over Ethernet (PoE) powers IT endpoints like IP phones and WLAN Access points. But newer devices like IP video phones, IP surveillance cameras, thin client display, next Generation IP phones, 802.11n WLAN access points and Pan Tilt Zoom (PTZ) Surveillance cameras require more than 15W of power so the IEEE recently standardized a new 30W PoE standard called PoE Plus. Soon most, if not all, corporations will have a mix of end points that require old PoE and new PoE plus power. IT leaders can meet these requirements by leveraging the enhanced PoE and PoE Plus features offered by Cisco’s Catalyst 4500 E-Series product line. Moreover Cisco has announced two new Catalyst 4500 Series Line Cards with readiness for PoE Plus and inline power up to 30W per port. I talk with Sachin Gupta, Director and Product Manager at Cisco Systems about these new PoE design features and what new options are available to IT architects as they build out corporate collaborative solutions.

Listen to the Podcast

Tweet

By Arista Networks

What is Arista’s definition of scalability of a data center network? What are the critical requirements and what issues must be solved to address data center scalability? Scalability of the data center network is the ability for network technologies to accept increased traffic or new devices without impacting the contribution margin. In this white paper Arista Networks discuses network design best practices for scaling up Data Center Networks.

Find out how by downloading this paper.

Get the White Paper

Tweet

By Arista

The standard for 10 Gigabit Ethernet (IEEE802.3ae) was ratified in 2002. While 10GbE deployments have grown every year since then, the technology has primarily been used to interconnect switches and routers. Almost all of the server connections in data centers have remained at 1 Gbps, limiting the amount of network throughput available to each server. With recent enhancements in CPU performance, system I/O, and storage I/O the gigabit network has increasingly become the application and workload performance bottleneck.
The primary reason for staying with Gigabit Ethernet has been cost-performance. Until recently it has been more cost-effective to have multiple GbE connections rather than a single 10 GbE port. In addition, most installed servers typically cannot utilize the full bandwidth of a 10 GbE connection. However both of these factors are changing, which are leading to widespread adoption of 10 GbE for server connectivity over the next few years.

Download this white paper for an overview of the factors that are driving the growth for 10 GbE in the data center.

Get the White Paper