Tweet

The data center switching market is heating up. To address the scale issues posed by mobile and cloud computing nearly every network vendor is launching its own version of a 10/40/100 GbE fabric to connect servers and storage to the internet. At the heart of this fabric is a two-tier (Fat-Tree) network made up of leaf/ToR and spine/Core switches. Here leafs connect servers and spines connect leafs while also being interconnected in a logical mesh. The protocols to create this logical mesh are based upon IS-IS link state routing, but each vendor is taking a unique approach with Cisco using its FastPath, Alcatel-Lucent and Avaya using SPB (802.1aq Shortest Path Bridging) while Brocade VDX is based upon TRILL (Transparent Interconnection of Lots of Links). Juniper recently announced QFabric but has not detailed what it’s using for logical meshing. At the center of new data center design are leaf and spine switches. In Lippis Report Research Note 166, we detailed the latest ToR switches. In this Lippis Report Research Note 167, we dive into performance and power consumption measurements plus the use of SPB of Alcatel-Lucent’s OmniSwitch 10K, a new entry into spine/core data center switching market.

During December 6-10, 2010, the Lippis Report and Ixia conducted the industry’s first 10GbE data center switching evaluation of Top-of-Rack and Core Ethernet switches at the modern iSimCity lab in Santa Clara, CA. We evaluated Alcatel-Lucent’s OmniSwitch 10K, Arista’s 7504 Series Data Center Switch, BLADE Network Technologies’, an IBM Company, IBM BNT RackSwitch G8124 and IBM BNT RackSwitch G8264, Force10 Network’s S-Series S4810, Hitachi Cable’s Apresia 15000-64XL-PSR, Juniper Network’s EX Series EX8216 Ethernet Switch and Voltaire®’s Vantage™ 6048. We are conducting a second round of test scheduled for the week of April 4-8 at iSimCity, and it is open to all suppliers of 10 and 40 GbE data center switching.

There were three Core/Spine Switches evaluated for performance and power consumption in the Lippis/Ixia test. These participating vendors were:

Alcatel-Lucent OmniSwitch 10K
Arista 7504 Series Data Center Switch
Juniper Network EX Series EX8216 Ethernet Switch

These switches represent the state-of-the-art of computer network hardware and software engineering, and are central to private/public data center cloud computing infrastructure.

If not for this category of Ethernet switching, cloud computing would not exist. The Lippis/Ixia public test was the first evaluation for every Core switch tested. Each supplier’s Core switch was evaluated for its fundamental performance and power consumption features. The Lippis/Ixia test results demonstrate that these new Core switches provide state-of-the-art performance at efficient power consumption levels not seen before. The port density tested for these Core switches ranged from 128 10GbE ports to a high of 256 10GbE.

IT business leaders are responding favorably to Core switches equipped with a value proposition of high performance, high port density, competitive acquisition cost, virtualization aware services, high reliability and low power consumption. These Core switches currently are in high demand with quarterly revenues for mid-size firms in the $20 to $40M plus range. The combined market run rate for both ToR and Core 10GbE switching is measured in the multibillion-dollar range. Further, Core switch price points on a 10GbE per port basis are a low of $1,200 to a high of $6,093.

Their list price varies from $230,000 to $780,000 with an average order usually being in the million plus dollar range. While there is a large difference in list price as well as price per port between vendors, the reason is found in the number of network services supported by the various suppliers and 10GbE port density.

We compare each of the above firms in terms of their ability to forward packets: quickly (i.e., latency), without loss or their throughput at full line rate, when ports are oversubscribed with network traffic by 150%, in IP multicast mode and in cloud simulation. We also measure their power consumption.

Alcatel-Lucent launched its new entry into the enterprise data center market on December 17, 2010, with the OmniSwitch ™ 10K. The OmniSwitch was the most densely-populated device tested with 256 ports of 10GbE. The test numbers below represent the first public performance and power consumption measurements for the OmniSwitch™ 10K running software version 7.1.1.R01.1638. The Alcatel-Lucent OmniSwitch™ 10K Modular Ethernet LAN Chassis is the first of a new generation of network adaptable LAN switches. It exemplifies Alcatel-Lucent’s approach to enabling what it calls Application Fluent Networks, which are designed to deliver a high-quality user experience while optimizing the performance of legacy, real-time and multimedia applications. So how did the OmniSwitch 10K do?

RFC 2544 Layer 2 and 3 Latency Test

The OmniSwitch 10K was tested across all 256 ports of 10GbE. Its average latency ranged from a low of 20,561 ns or 20 μs to a high of 36,823 ns or 36 μs at jumbo size 9216 Byte size frames for layer 2 traffic. Its average delay variation ranged between 5 and 10 ns, providing consistent latency across all packet sizes at full line rate. What this means is that the OmniSwitch 10K can be counted on to forward packets at these latencies without much variation which is extremely important for predictable performance.

For layer 3 traffic, the OmniSwitch 10K’s measured average latency ranged from a low of 20,128 ns or 20μs at 64Bytes to a high of 45,933 ns or 45μs at jumbo size 9216 Byte size frames. Its average delay variation for layer 3 traffic ranged between 4 and 10 ns, providing consistent latency across all packet sizes at full line rate.

RFC 2544 Layer 2 and 3 Throughput Test

The OmniSwitch 10K demonstrated 100% throughput as a percentage of line rate across all 256 10GbE ports. In other words, not a single packet was dropped while the OmniSwitch 10K was presented with enough traffic to populate all of its 256 10GbE ports at line rate simultaneously for both L2 and L3 traffic flows. Not a single packet was dropped while 2.5 Tbps of traffic passed through its line cards and backplane.

RFC 2889 Congestion Test

The OmniSwitch 10K demonstrated nearly 80% of aggregated forwarding rate as percentage of line rate during congestion conditions. A single
10GbE port was flooded at 150% of line rate. The OmniSwitch did not use HOL blocking, which means that as the 10GbE port on the OmniSwitch became congested, it did not impact the performance of other ports. There was no back pressure detected as the Ixia test gear did not receive flow control frames. This was not the same for the Arista 7504. See the full test report here.

RFC 3918 IP Multicast

The OmniSwitch 10K demonstrated 100% aggregated throughput for IP multicast traffic with latencies ranging from 9,596 ns at 64 Byte size packets to 28,059 ns at 9216 Byte size packets. The OmniSwitch 10K demonstrated the lowest multicast latencies of all vendors.

Cloud Simulation Test

The one test that was not RFC based is a cloud simulation that was developed by the Lippis Report and Ixia. This test determines the traffic delivery performance of the DUT (device under test) in forwarding a variety of north-south and east-west traffic in cloud-computing applications. This test measures the throughput, latency, jitter and loss on a per application traffic type basis across M sets of 8-port topologies. The following traffic types are used: web (HTTP), database-server, server-database, iSCSI storage-server, iSCSI server-storage, client-server plus server-client. The north-south client-server traffic simulates Internet browsing; the database traffic simulates server-server lookup and data retrieval, while the storage traffic simulates IP-based storage requests and retrieval. When all traffic is transmitted, the throughput, latency, jitter and loss performance are measured on a per traffic type basis.

The OmniSwitch 10K performed extremely well under cloud simulation conditions by delivering 100% aggregated throughput while processing a large combination of east-west and north-south traffic flows. Zero packet loss was observed as its latency stayed under 28μs.

Power Consumption Test

The OmniSwitch 10K represents a new breed of cloud network spine switches with power efficiency being a core value. The OmniSwitch consumes 13.3 Watts/10GbE port with a TEER (Telecommunications Energy Efficiency Ratio) value of 71. TEER is a measure of network-element efficiency quantifying a network component’s ratio of “work performed” to energy consumed. Larger TEER values are better and the OmniSwitch is second only to Arista in TEER value while Juniper’s EX8216 measured a 44 TEER. You can download the OmniSwitch 10K test report here:

The OmniSwitch 10K power cost per 10GbE is estimated at $16.26 per year. The three-year cost to power the OmniSwitch is estimated at $12,485.46 and represents less than 3% of its list price. Keeping with data center best practices, its cooling fans flow air front to back, which is the norm except for Juniper’s EX8216 which pushes air from side to side unless a third party cabinet from vendors, such as Chatsworth, enclose the EX8216 to support hot-aisle and cold-aisle deployments.

Discussion:

The OmniSwitch™ 10K seeks to improve application performance and user experience with deep packet buffers, lossless virtual output queuing (VOQ) fabric and extensive traffic management capabilities. This architecture proved its value during the RFC2889 layer 2 and layer 3 congestion test with a 78% aggregated forwarding rate when a single 10GbE port was oversubscribed at 150% of line rate. The OmniSwitch™ 10K did not use HOL blocking, back pressure or signal back to the Ixia test equipment with Aggregated Flow Control Frames to slow down traffic flow. Not tested but notable features are its security and high availability design for uninterrupted uptime. The OmniSwitch™ 10K was found to have low power consumption, front-to-back cooling, front-accessible components and a compact form factor. The OmniSwitch™ 10K is designed to meet the requirements for mid- to large-sized enterprises data centers.

To demonstrate how the OmniSwitch™ 10K operates as a lossless fabric plus its ability to deliver carrier class quality of service (QoS), Alcatel-Lucent conducted two separate sets of tests; its data is available here. The lossless fabric test configured 256 x 10GbE ports connected with fully meshed traffic running at wire-speed via Ixia test equipment. The objective of this test was to demonstrate that as fabric and management modules were pulled and inserted into the OmniSwitch™ 10K chassis, zero loss at 100% load would result, and the fabric would be lossless. With fully meshed traffic running through all 256 GbE ports, the following modules were changed.

1. Fabric module was pulled out.
2. Fabric module was inserted back.
3. Management module (a fabric resides on this module) was pulled out.
4. Management module was inserted back.
5. Management module was pulled out causing a management failover in addition to fabric failover.
6. Management module was inserted back.

The result of the lossless fabric was that the fabric was lossless as the above modules were pulled and inserted.

The carrier class QoS objective was to demonstrate no packet loss at wire-speed with P0-P7 (priority) traffic running in fully meshed scenario as in the test above. The carrier class QoS test configured 256 x 10GbE ports connected with fully meshed traffic, priority 0 to 7, running at wire-speed via Ixia test equipment. In this scenario, the OmniSwitch™ 10K delivered zero loss with consistent store-and-forward average latency in range of 132,357 ns to 139,448 ns. See this test report details.

Cloud Network Architecture

There are three approaches to connect spine switches together to create a network fabric. MC-LAG (Multi-Chassis Link Aggregation Group), which allows one or more links to be aggregated together to form a Link Aggregation Group, TRILL and/or SPB are emerging standards that design a solution for shortest path frame routing in multi-hop Ethernet networks with arbitrary topologies, using an existing link-state routing protocol technology.

While there is debate over which approach is best, SPB has the following advantages. SPB deployments are planned for 2011 and offer increased scalability than TRILL. Further, SPB will interoperate with carrier infrastructure to allow private-public or private-private or public-public data center-to-data center connections. For network architects/designers and operations, there is a quick learning curve as SPB uses the existing IS-IS protocol, and for service providers, SPB is already available through OAM (Operations, Administration and Maintenance), enabling it to be managed through existing management services.

Paramount in the two-tier leaf-spine architecture is high-spine switch performance, which collapses the aggregation layer in the traditional three-tier network connecting spine switches together. The above captures the major trends and demands that IT business leaders are requiring from the networking industry. The underpinnings of private and public data center cloud network fabric are 10GbE switching with 40GbE and 100GbE ports/modules. 40GbE and 100GbE are in limited availability now but will be increasingly offered and adopted during 2011. Network performance including throughput performance and latency are fundamental switch attributes to understand and review across suppliers, because if the 10GbE switches an IT leader selects cannot scale performance to support increasing traffic volume plus shifts in traffic profile, not only will the network fail to be a fabric unable to support converge storage traffic, but business processes, application performance and user experience will suffer too.

During 2011, an increasing number of servers will be equipped with 10GbE LAN on Motherboard (LOM) driving 10GbE network requirements, and in 2012, high-end servers will be equipped with 40GbE LOM starting 40GbE’s growth curve. In addition, with nearly 80% of IT spend being consumed in data center infrastructure with all IT assets eventually running over 10GbE switching, the stakes could not be higher to select the right product upon which to build this fundamental corporate asset. Further, data center network equipment has the longest life span of all IT equipment; therefore, networking is a long-term investment and vendor commitment.

We review the Alcatel-Lucent OmniSwitch 10K from a perspective of performance and power measurement, mesh protocol support and key product features. Alcatel-Lucent has entered the data center switching market with a very competitive Core/spine switch. Clearly there are differences between Core switch vendors, and it’s advised to conduct a detailed review. For starters Click here for a copy of Alcatel-Lucent’s OmniSwitch 10K plus cross-vendor test results report.

Tweet

During December 6-10, 2010, the Lippis Report and Ixia conducted the industry’s first 10GbE data center switching evaluation of Top-of-Rack and Core Ethernet switches at the modern iSimCity lab in Santa Clara, CA. We evaluated Alcatel-Lucent’s OmniSwitch 10K, Arista’s 7504 Series Data Center Switch, BLADE Network Technologies’, an IBM Company, IBM BNT RackSwitch G8124 and IBM BNT RackSwitch G8264, Force10 Network’s S-Series S4810, Hitachi Cable’s Apresia 15000-64XL-PSR, Juniper Network’s EX Series EX8216 Ethernet Switch and Voltaire®’s Vantage™ 6048. We are conducting a second round of test scheduled for the week of April 4-8 at iSimCity, and it is open to all suppliers of 10GbE data center switching. We learned a lot about these products, both in the lab and out. In this Lippis Report Research Note, we dive into the Top-of-Rack 10GbE switches we tested as they represent a new generation of products that exhibit low power consumption, low latency, high performance and are all based upon new single chip designs from Broadcom, Marvell or Fulcrum Micro.

The Top-of-Rack (ToR) switches tested at iSimCity were the:

BLADE Network Technologies, an IBM Company, IBM BNT RackSwitch G8124 and IBM BNT RackSwitch G8264;
Force10 Network’s S-Series S4810;
Hitachi Cable’s Apresia 15000-64XL-PSR;
Voltaire®’s Vantage™ 6048.

All of these ToR switches utilize a new single chip design, but mostly from different silicon suppliers. With a single chip provided by chip manufacturers Broadcom, Marvell or Fulcrum Micro, vendors are free to invest resources other than ASIC development, which can consume much of a company’s engineering and financial resources. With merchant silicon providing a forwarding engine for their switches, these vendors are free to choose where to innovate, be it in buffer architecture, network services such as virtualization support, 40GbE uplink or fan-in support, etc.

The Lippis/Ixia test results demonstrate that these new chip designs provide state-of-the-art performance at efficient power consumption levels not seen before. In addition, price points on a 10GbE per port basis for ToR switches are a low of $351 to a high of $520.

IT business leaders are responding favorably to ToR switches equipped with a value proposition of high performance, low acquisition price and low power consumption. These ToR switches currently are the hot boxes in the industry, with quarterly revenues for mid-size firms in the $10 to $15M plus. We compared each of the above firms in terms of their ability to forward packets: quickly (i.e., latency) without loss or their throughput at full line rate, when ports are oversubscribed with network traffic by 150 percent, in IP multicast mode and in cloud simulation. We also measured their power consumption. Click Here for a copy of BLADE’s G8124 and G8264 plus cross-vendor test results report and Click Here for a copy of Force10’s S4810 plus cross-vendor specific report.

Latency Measurement Anomalies

When evaluating five products from four companies, there are bound to be anomalies. One anomaly was found during latency measurement. As both BLADE and Force 10 use the same Broadcom chip in their G8264 and S4810 ToR switches, respectively, one would expect their latency measurements would be close, but the S4810 showed lower latency values. As it turns out, the Broadcom chip allows switches to forward in cut-through and/or store-and-forward mode. The G8264 was configured and tested in cut-through mode while the S4810 and all other switches were configured and tested in store-and-forward. Test equipment, such as Ixia and others, measure latency very differently in these two forwarding modes.

During store-and-forward testing, test equipment subtract packet transmission latency, decreasing actual latency measurements by the time it takes to transmit a packet from input to output port. This makes comparisons between the two-latency measurement testing methodologies difficult. Also other potential device specific factors can impact latency too. But looking at the bigger picture, latency is being measured in the hundreds to thousands of nanoseconds across various packet sizes, making these switches the fastest forwarding engines in the market.

One of the biggest surprises was Voltaire’s Vantage 6048 ToR latency results, which were the highest of the group by nearly a factor of 2. Voltaire, now owned by Mellanox, used the Marvell 10GbE single chip code named Lion. The Hitachi Apresia 15000-64XL-PSR showed low latency results but it had other difficulties. For example, the largest frame size supported is 9044, excluding it from the 9216 byte size packet tests. Further, there is no latency data for the Apresia 15000-64XL-PSR at 64 bytes due to configuration difficulties during testing. The 15000-64XL-PSR could not be configured to maintain a VLAN at 64 bytes which eliminated packet signature to measure latency at this packet size.

A big surprise and delight found was how low the average delay variation was for all suppliers. Average delay variation was in the 5 to 10ns range, meaning that all of the above ToR switches deliver their latency results reliably.

Throughput

The results of RFC 2544 throughput testing should be boring with all ToR switches showing 100% throughput at line rate. The only anomaly here was the Apresia 15000-64XL-PSR during layer 2 forwarding, dropping packets at between 128 to 2176 packet sizes.

Congestion Testing

RFC 2889 congestion testing was telling too. Here the IBM BNT RackSwitch G8124 and IBM BNT RackSwitch G8264, Force10 Network’s S-Series S4810 and Voltaire®’s Vantage™ 6048 performed as expected, that is, offering 100% line rate under congestion conditions without head of line blocking and using back pressure or pause messages to control the flow of traffic. Here again, Hitachi Cable’s Apresia 15000-64XL-PSR showed head of line blocking and low throughput especially at the higher packet sizes of 2176 bytes.

IP Multicast

For RFC 3918 IP Multicast Throughput No Drop Rate testing, the IBM BNT RackSwitch G8124 and IBM BNT RackSwitch G8264 and Force10 Network’s S-Series S4810 performed flawlessly, exhibiting 100% line rate throughput and nanosecond latency with the G8124’s average latency 700ns and below. The IBM BNT RackSwitch G8264 and Force10 Network’s S-Series S4810 IP multicast performed as expected as they are both based upon the same Broadcom chip. The G8264 demonstrated a slight advantage of 100ns at the higher packet sizes while Force10 showed approximately 100ns advantage at the lower packet sizes. The Apresia 15000-64XL-PSR and Vantage 6048 do not support IP Multicast at this time.

Cloud Simulation

The one test that was not RFC based is a cloud simulation that was developed by the Lippis Report and Ixia. This test determines the traffic delivery performance of the DUT (device under test) in forwarding a variety of north-south and east-west traffic in cloud-computing applications. This test measures the throughput, latency, jitter and loss on a per application traffic type basis across M sets of 8-port topologies. The following traffic types are used: web (HTTP), database-server, server-database, iSCSI storage-server, iSCSI server-storage, client-server plus server-client. The north-south client-server traffic simulates Internet browsing, the database traffic simulates server-server lookup and data retrieval, while the storage traffic simulates IP-based storage requests and retrieval. When all traffic is transmitted, the throughput, latency, jitter and loss performance are measured on a per traffic type basis.

This test is telling too as it’s designed to be a simulation of real-world cloud-computing traffic. The results here show that the IBM BNT RackSwitch G8124 and G8264 delivered the lowest latency consistently across all protocol types. The Apresia 15000-64XL-PSR performed very well in this test too, followed by Force10’s S4810 followed by Voltaire’s Vantage 6048. Anomalistically, both the Force10 S4810 and Vantage 6048 spiked in terms of latency for east-west database-server, HTTP and iSCSI-Storage traffic flows. Both IBM BNT RackSwitches and Force10’s S4810 tested in cut-through mode.

Power Consumption

Power consumption or energy efficiency has become a paramount concern in data centers as the cost of power and cooling start to dominate TCO (total cost of ownership) over a three-year period. The ToR switches tested offer the lowest power consumption of switching products evaluated in public industry test. Their power consumption measured in WATTS per 10GbE via ATIS methodology ranged from 3.6 to 5.5. We then projected annual cost per 10GbE to be between $4.36 to $6.70, with the Apresia 15000-64XL-PSR offering the lowest power consumption. The IBM BNT RackSwitch G8264 and Force10’s S4810 were very close at $4.78 and $4.91, respectively, with the G8264 having a slight advantage. Of the 48-port 10GbE ToR switches, Voltaire’s Vantage 6048 consumed the most energy at 5.5Watts/10GbE.

While not confirmed, the IBM BNT RackSwitch G8124 may be based upon the Fulcrum single chip set, code named Bali, as well as Arista’s 7124 and Force10’s S2410. The Apresia 15000-64XL-PSR may be based upon the Broadcom Trident single chip. There are rumors in the industry too that large networking firms may start to utilize merchant silicon rather than build their own, as these chips offer a quicker path to market and are delivering solid performance, latency and power efficiency results.

While I detail ten recommendations in the test report, here I’ll focus on one. 10GbE ToR switches are ready for mass deployment, delivering full line rate
throughput at zero packet loss and nanosecond latency plus single- to double-digit delay variation. In addition, these ToR switches offer low power consumption with energy cost over a three-year period estimated between 3 and 4% of acquisition cost. Clearly there are differences between vendors, and it’s advised to conduct a detailed review. For starters Click here for a copy of BLADE’s G8124 and G8264 plus cross-vendor test results report and Click Here for a copy of Force10’s S4810 plus cross-vendor specific report.

Tweet

By Voltaire

The Data Center Bridging Group of the IEEE worked to enhance Ethernet to support class isolation, low latency, I/O and switch virtualization, lossless traffic flows, congestion control, multi-path L2 routing, and L2 discovery and capability exchange. These new technologies are referred to as Converged Enhanced Ethernet (CEE) or Data Center Ethernet (DCE). Voltaire’s CEE switches and software offer performance requirements thanks to its scalable Ethernet fabric that lowers overall fabric costs, lowers power consumption, has greater efficiencies, and simplifies management.

This document describes the challenges inherent in traditional Ethernet solutions and how Voltaire’s scale-out Ethernet architecture effectively addresses those challenges.

Get the White Paper

Tweet

A cross vendor comparative test report conducted at Ixia’s iSmiCity defined by The Lippis Report. This report provides detailed test information on the following new products that have not been previously tested in public. The report details test results of the following products:

Alcatel-Lucent OmniSwitch 10K,
Arista 7504 Series Data Center Switch,
Arista 7124SX 10G SFP Data Center Switch,
Arista 7050S-64 10/40G Data Center Switch,
Brocade VDXTM 6720-24 Data Center Switch,
IBM BNT RackSwitch G8124,
IBM BNT RackSwitch G8264,
Force10 S-Series S4810,
Hitachi Cable, Apresia15000-64XL-PSR,
Juniper Network EX Series EX8216 Ethernet Switch,
Voltaire® VantageTM 6048.

This 93-page report is a must for those evaluating 10/40 GbE data center switching equipment for private or public cloud infrastructure. You don’t want to buy data center switching gear until you read this report.

To download the report click here.

Get the White Paper

Tweet

Ronnie Kenneth, CEO, and Asaf Somekh, VP Marketing of Voltaire, a leader in data center networking, are my guests as we dive into Ethernet data center fabric design options. Data Center vendors are discussed too, and you have to hear what they had to say about Cisco. Enjoy, Nick

Visit the Link

Tweet

Networking has become “rigid”. Yes I know it’s almost absurd to attribute inflexibility or rigidity to networking. Look what TCP/IP has done for us. There are nearly 2 billion people connected to the internet and according to the Internet World Stats internet user growth rate increased by 380% between 2000-2009. With 2 billion people and growing online, accessing a plethora of applications via a wide range of end-points there is no doubt that the internet and TCP/IP has been a much bigger success than anyone would have imagined back in the early ’90s. But there’s always a give and take between computing and networking where one drives and changes the other. Right now we are in a compute innovation cycle that’s driving a fundamental change in networking which screams out the need for more flexibility.

Sure networking has increased from a bandwidth point of view and the IETF has added new protocols and network services, but it hasn’t kept up with compute innovation. As data centers pack more compute power and operating systems (OS) per physical server, thanks to virtualization, the need to move containers of OS plus applications and data around have sky rocked. In addition, traffic patterns have shifted tremendously as client-server or north-south flows are layered on top of server-server or east-west flows. And yes, there are new networking approaches being offered by vendors and standard organizations such as Cisco’s FlexPath, Juniper’s Stratus, Brocades VCS, Extreme’s Direct Attach, Force 10’s Open Automation, Arista’s Multi-Chassis Link Aggregation, BLADE’s Unified FabricArchitecture, the IETF’s TRILL and LISP and IEEE’s 802.1AQ, but these may be short term solutions to a much bigger networking problem.

Computing has always driven network design as mainframes drove SNA and analog multi-point wide area networks (WANs) during the ’70s. Mini-computers drove peer-to-peer networking protocols like DecNet, OSI and TCP/IP in the ’80s. Client-Server computing drove LANs and TCP into the mainstream in the early ’90s. The Web drove the internet in the 2000s and now server virtualization and cloud computing is once again changing fundamental networking requirements to make them more flexible.

The rigid label is a powerful one as it creates frustration by not addressing or enabling new business processes. Every time a network protocol or architecture was labeled as too rigid it was replaced and in the process a new market emerged on the scale of tens of billions of dollars. SNA was labeled as too rigid to support peer-to-peer networking. The T1 multiplexer market of the late ’80s and early ’90s was too rigid to support data traffic and thus routing replaced it. The PSTN and TDM were too rigid as they doled out bandwidth in 56Kbs chunks and were unable to support internet and VoIP traffic. The national entertainment network is rigid too as it doesn’t support two-way communications and it also will be replaced slowly but surely.

So where is networking not flexible enough? It’s in virtualized data centers. Some analyst groups estimate that 30% of workloads are virtualized and increasing. Since virtualization or a VM is the new atomic layer of data centers, networking is falling short in public as well as private clouds. Ideally, all resources (compute, storage, and networking) would be pooled, with services dynamically drawing from the pools to meet demand. Virtualization techniques have succeeded in enabling processes to be moved between machines, but constraints in the data center network continue to create barriers that prevent agility, for example, VLANs, ACLs, broadcast domains, Load Balancers, Firewall/IPS Security settings and service-specific network engineering.

The well understood problem is that when a VM is moved from one physical machine to another the network, load balancers, firewalls/IPS, broadcast domains, etc., have to be reconfigured. There is no automation in place, meaning that the network is not flexible or agile enough to make the changes required. Now this problem has scale to it as it’s a growing requirement of both IT executives managing corporate IT assets and service/cloud providers.

There are market solutions available today and more are coming that address “network automation” which enable the network to reconfigure itself as a VM and/or workload is moved within a data center. Cisco’s Nexus 1000V, HP Network Automation software and its Virtual Connect approach, Force 10’s Open Automation, Blade Network Technologies VMReady Network Virtualization, Arista Network’s Virtualized Extensible Operating System or vEOS and others are addressing the problem of network agility or lack thereof in virtualized environments.

But the problem gets bigger and more complex when distance and cloud provider entities become engaged. None of the solutions above address moving a VM from one physical server to another over large distance, be it around town, across state lines, across the country or the globe. Some are using IF-MAP as a registry, sort of like facebook for computers that publish their resources and use this information to automate network configuration to support large distance VM moves.

The problem gets larger yet when workloads move from a private cloud to a public cloud. (Definition note: There is no single definition of a workload, so for my purpose here I assume a container including a VM and associated applications and data that can be moved as simply as drag and drop or some other string of instructions). In short, all the software that is needed to compile and run an application for a set of users is a workload. The network inflexibility problem grows even larger when moving workloads between public clouds.

Now is this a real problem? You bet it is. Consider the value also of portable or mobile workloads to Enterprise and service providers. Workload mobility means capacity on demand, business continuance, and disaster recovery, etc. In addition, as IT leaders explore public and private cloud alternatives, they will want to move workloads from their data center to a provider’s and move the workload back when and if required. For reasons of security and trust, IT business leaders will demand mobility. For example, if your cloud provider goes bankrupt, then you will want to move your workload out quickly. If your cloud provider’s performance drops again then you could move your workload out. If your cloud provider is the target of a terrorist attack or is turned into a large botnet then you can move your workload out.

In addition to security and piece of mind, mobile workloads will fundamentally change IT delivery, capital structure and most importantly business models and processes. Once IT can move workload anywhere in their data center, across their data centers or to a provider they have tiered with, the question becomes when and how fast does IT move workload? If IT can perform all the provisioning in software and enable workload moves to occur transparently and safely with address, identity, security preservation, enabled trust, control and interoperability across providers, then the question is when does IT need to move workload? This level of mobility is an industry-wide initiative as it offers significant and material business value. Business value is created as IT could move workload in a follow- the-sun model, following the lowest cost per kilowatt-hour model; workload could move to avoid a disaster, or for capacity on demand, or for lowest cost of workload execution, etc.

So how can data center networks become more flexible? A key element of the solution is agility or the ability to dynamically grow and shrink resources to meet demand and to draw those resources from the most optimal location. Today, the network stands as a barrier to agility and increases the fragmentation of resources, which leads to low server utilization and prevents portable or mobile workloads.

Tweet

In Lippis Report 151: A Two or Three Tier High-End Data Center Ethernet Fabric Architecture? we detailed the new two tier data center Ethernet fabric that is becoming conventional wisdom amongst business leaders of high end data centers and cloud computing service providers. The networking industry is headed for a major innovation and competitive cycle fueled by a multi-billion dollar addressable market for data center network fabrics. Over the last eighteen months, every major Ethernet infrastructure provider has announced or taken a position on two tier network fabrics for high-end data centers. Companies such as Cisco, Arista Networks, Force10, Voltaire, HP/3Com, Juniper, Extreme, Brocade, BLADE Network Technology, et al have announced network fabrics for data centers with two thousand and more servers that either support storage enablement or not. In this Lippis Report Research Note, we review why it is Ethernet that will be the network fabric of high performance computing or HPC and cloud computing deployments.

For high-end data centers, HPC plus private and public cloud computing networks connecting thousands of servers, a new set of requirements have emerged. Low latency and high performance are the two driving requirements. Yes, there are more, especially when the fabric needs to enable converged storage, but let’s focus on latency and performance for now. Traditional three tier (server access, distribution and core) fabrics designed primarily for north-south traffic flows, that is client-server computing utilized spanning tree protocol (STP) and slower speed Ethernet (100Mbs to 1Gbs). Thanks to web 2.0, mash-ups and social networking sites east-to-west or server-server traffic flows have spiked requiring networks to support both north-south and east-west flows.

As most network engineers know, STP was designed to avoid loops that confused Ethernet as it was designed as a bus topology. STP shuts down redundant links between common switches to maintain the bus. Therefore, connecting access switches to distribution switches utilizing STP would require that network engineers over-subscribe the links between switches as only half of the bandwidth could be used. Oversubscription would also create blocking of packets between points too. To avoid this design, nearly every major switch manufacturer offered link aggregation that is the ability to shut off STP and aggregate links between switches. While this was and is a benefit, the down side has been that vendors only offered the ability to aggregate two links, which still drove oversubscription and blocking.

Recently, industry players such as Cisco and Arista Networks have offered the ability to scale up aggregation of links from 16 to 32, while at the same time delivering multipathing that allows packets to be forwarded across multiple links to arrive at its intended destination. Switch-processing capacity to support these massive inter-switch links have been increased too. These design changes, along with Ethernet’s innovation march, has ushered in the two-tier network design fabric option.

A two-tier fabric is designed with two kinds of switches; one that connects servers and the second that connect switches creating a non-blocking, low latency fabric. We use the terms ‘leaf’ switch to denote server connecting switches and ‘spine’ to denote switches that connect leaf switches. Together a leaf and spin architecture create the network fabric.

In late June 2010, Cisco announced its’ FabricPatch Switching System or FSS and its’ F-Series modules that support 32 ports of 10GbE of auto-sensing 1/10GbE and is essentially for server access and aggregation. FabricPath provides a new level of bandwidth scale to connect Nexus switches and delivers a new fabric design option with unique attributes for IT architects and designers. FabricPath is a NX-OS innovation, meaning that its’ capabilities are embedded within the NX-OS network OS for the data center. FabricPath essentially is multipath Ethernet; a scheme that provides high-throughput, reduced and more deterministic latency, and greater resiliency compared to traditional Ethernet.

FabricPath combines today’s layer 2 or Ethernet networking attributes and enhances it with layer 3 capabilities. In short, FabricPath brings some of the capabilities available in routing into a traditional switching context. For example, FabricPath offers the benefits of layer 2 switching such as low cost, easy configuration and workload flexibility. What this means is that when IT needs to move VMs and/or applications around the data center to different physical locations, it can do so in a simple and straightforward manner without requiring VLAN, IP address and other network reconfiguration. In essence, FabricPath delivers plug and play capability, which has been an early design attribute of Ethernet. Further, large broadcast domains and storms inherent in layer 2 networks that occurred during the mid 1990s have been mitigated with technologies such as VLAN pruning, Reverse Path Forwarding, Time-to-Live, etc.

The layer 3 capabilities added to FabricPath deliver scalable bandwidth allowing IT architects to build much larger layer 2 networks with very high cross-sectional bandwidth eliminating the need for oversubscription. In addition, FabricPath affords high availability as it eliminates STP, which only allows one path and blocks all others, and replaces it with multiple paths between endpoints within the data center. This offers increased redundancy as traffic has multiple paths in which to reach its final destination.

FabricPath employs routing techniques such as building a route table of different nodes in a network. It possesses a routing protocol, which calculates paths that packets can traverse through the network. What is being added to FabricPath is the ability for the control plane or the routing protocols to know the topology of the network and choose different routes for traffic to flow. Not only can FabricPath choose different routes, it can use multiple routes simultaneously so traffic can span across multiple routes at once. These layer 3 features enable FabricPath to use all links between switches to pass traffic as STP is no longer used and would shut down redundant links to eliminate loops. Therefore, this would yield incremental levels of resiliency and bandwidth capacity, which is paramount as compute and virtualization density continue to raise driving scale requirements up.

Designing A 160 Tbps Data Center Fabric

As an example to how multi link aggregation, the elimination of STP, high switching capacity and 10GbE connections create a highly scalable two-tier layer 2 Ethernet fabric, we use Cisco’s FSS and its’ F-Series module in the Nexus 7000. The following details the design of a 160 Tbps switching fabric with FabricPath and the F-Series module for high performance data centers using Cisco’s Nexus 7000 switches. This architecture can support over 8,000 servers connected at 10GbE or 4,000 servers dual homed at 10GbE with attributes of being non-blocking, low latency (5 microseconds), high bandwidth, reliability, plus simplicity of workload movement.

To build a 160 Tbps two-tier fabric, thirty-two Nexus 7018 switches populated with F-Series 10GbE modules would connect servers. These thirty switches are leaf switches. Each leaf chassis provides 256 10GbE ports to connect servers and another 256 10GbE ports to connect into spine switches. Therefore, each leaf is directly connected to each spine with sixteen FabricPath ports at 10GbE equaling a total of 256 10GbE ports for each leaf switch. There are sixteen spine switches each accepting 512 10GbE FabricPath ports. A single leaf chassis connects 256 10GbE ports into a spine equaling approximately 2.5Tbs. Multiplying each thirty-two leaf’s contribution into the fabric yields 80Tbs. As Ethernet is full-duplex, the total fabric switching capacity is 160
Tbps. Therefore, 160Tbps of switching fabric is available across all thirty-two leaf chassis. As 256 10GbE equals 2.5 Tbs, which also equals 16 FabricPath links to each one of sixteen spine switches, yields 2.5 Tbs, the fabric is non-blocking.

As for layer 2 and layer 3 forwarding, the job of the spine is to forward packets from leaf switches at layer 2, creating a single tier fabric. A key attribute of this architecture is that each 16-way FabricPath links are Equal Cost Multipathing or ECMP. What 16-way FabricPath ECMP provides are two benefits: 1) It delivers more paths for traffic to flow, which increases available bandwidth in the fabric and 2) as they’re distributed across all switches, diversity of routes is enabled to distribute packet forwarding. In essence what 16-way FabricPath ECMP provides is a very low latency, high bandwidth approach to supporting both north-to-south and east-to-west traffic flows simultaneously.

While the above is a Cisco deployment example Arista’s new 7500 series of Ethernet switches support 6 Billion packets per second at wire speed. The 7500s can be configured into a massive two-tier network fabric thanks to it support of 32 port MLAG (Multi-Chassis Link Aggregation) affording the connection of 18,000 to 30,000 servers.

Ethernet continues to evolve. The IEEE recently ratified the 40 and 100 GbE standard with vendors such as Force 10, Cisco, Arista, Extreme, BLADE, Brocade, Voltaire, HP et al announcing support and scheduling product delivery. While the above two-tier network example provides the perspective from the large switch provider, below is BLADE Network Technologies perspective, a company focused on server connectivity.

BLADE Network Technologies believes that as Ethernet delivers new levels of speed and intelligence, it will be the dominant two-tier network fabric for high-end next-generation data centers.
For many applications, low latency is a key requirement, and latency is an area where two-tier networks excel. Studies of stock trading exchanges have shown that tens of milliseconds of delay in data delivery can represent a ten percent drop in revenues, and delays of even five microseconds per trade can cost hundreds of thousands of dollars. Industry-specific requirements for uncompressed data and end-to-end deterministic latency within tens of microseconds make attaining such performance even more difficult. These factors have combined to make raw switching speed a top priority, and today’s best-of-breed 10 Gigabit Ethernet switches achieve can operate with under 700 nanoseconds of port-to-port latency while consuming a miniscule amount of power equivalent to that of standard light bulbs.

As next-generation networks get flatter – driven by latency and bandwidth requirements – emerging Layer 2 technologies such as the IETF’s Transparent Interconnection of Lots of Links or TRILL, enable this trend. The idea behind TRILL is to replace spanning tree as a mechanism to find loop free trees within Layer 2 broadcast domains. Using a routing protocol to build forwarding trees within a Layer 2 broadcast domain enables the flexibility and efficiency to route Layer 2 traffic, just like one would Layer 3 traffic, without the overhead associated with Layer 3 packet processing. TRILL will offer important features, such as support for both broadcast and multicast, load splitting along multiples paths, support for multiple points of attachment, and no tangible delay in service after attachment.

In the data center, bottlenecks are moving from the CPU and memory access to the I/O of the servers. Today’s multi-core servers are now able to sustain a great amount of traffic, requiring fast, flat networks, especially now that virtualization is widely deployed. Analysts have predicted that the 10G market will double year-to-year in 2010 and 2011. More servers using 10G increases the requirement for 40G and 100G in upstream networks. With 10G widely available and 40G coming online, Ethernet networks can enable data and storage traffic to use a single wire, using FCoE or iSCSI for example, and provide the raw speed that makes Ethernet with its economies of scale, to supplant InfiniBand for HPC requirements.

The reason Ethernet will be the network fabric for high-end data center networks is that the vendor community continues to innovate and build upon this protocol. Ethernet innovations are many and are beyond bandwidth increases from 10Mbs, 100Mbs, 1Gbs, 10Gbs, 40Gbs and 100Gbs, which are obvious. Link aggregation, multi-pathing and so much more propel Ethernet’s relevance and suitability to new challenging networking requirements.

Tweet

In Lippis Report 148 we reviewed the major drivers and trends that are propelling the high-end data center Ethernet switch market to well over a $1B annual run rate. In this Lippis Report Research Note, we review the major suppliers of these switches. We review Cisco, Arista Networks Force10 Networks, BLADE Network Technologies, HP/3Com/H3C, Voltaire, Avaya, Brocade, and Juniper and identify their unique positions and offerings to participants in the burgeoning market. Our focus is the high-end, high density 10GbE switches that are enabling virtualized cloud computing data centers thanks to Terabits per second of back plane switching capacity, billions of packets per second of layer 2/3 forwarding, hundreds of 10GbE port connectivity per chassis, a new two-tier architecture, microsecond level latency, low power consumption, non-stop operation and software hooks that eliminate network barriers to large scale server virtualization. The engineering in these switches should be celebrated, as they represent the state-of-the-art in computer and network design. In short, they represent the fundamental building block of a new generation of IT delivery based upon cloud computing and virtualization. This Research Note is a must read for any IT executive designing a data center.

After finishing this Research Note, it became evident that this market needs a set of industry neural 10GbE switch test to independently verify vendor claims. We hope to make such a contribution this Fall.

Cisco Systems Nexus Family of Switches

Cisco’s approach to data center Ethernet switching is rooted in its Data Center 3.0 strategy which seeks to scale server virtualization while introducing a platform to enable a unified fabric or converged network and storage running on one physical Ethernet network. Cisco’s data center Ethernet switch portfolio is primarily the Nexus family of switches including the 7000, 5000, 2000 and 1000v. NX-OS is a purpose built data center operating system that runs across the entire Nexus family. NX-OS integrates a number of higher system availability functionalities such as virtual port- channel (vPC), and the capability to upgrade software without disrupting traffic. The Nexus 1000v is a softswitch that resides in a VM hypervisor. The Nexus 1000v’s main job is to eliminate network configuration barriers that exist when moving a VM from one physical machine to another. To accomplish this, the 1000v creates a port profile including VLAN, ACL, policy, security, etc. with persistence, which moves with a VM as a virtualization administer moves a VM from one physical machine to another.

The Nexus 2000 family of Fabric Extenders (FEX) introduces the concept of a remote line card of the parent Nexus 5000 switches and sits on the top-of-rack connecting servers to the switch fabric. The extender concept allows the 2000 and 5000 to be managed as one switch. This configuration reduces cabling requirements and offers an economical approach to server connection, thus providing the benefits of both end-of-row and top-of-rack deployments. The Nexus 5000 Series is 10 Gb Ethernet and Unified Fabric capable switches, connecting Nexus 2000s and servers directly at 100/1/10GbE/FCoE, while providing layer 2 forwarding. Providing layer 3 forwarding, dense 1/10GbE connectivity is the Nexus 7000 Series. The Nexus 7000 Series is available in a 10 and 18 slot chassis and is Cisco’s flagship data center Ethernet switch series. As a point of reference, the Nexus 7000 is now on an annualized run rate of $1B for Cisco, which is more than 10 times greater than any other switch supplier in the data center switch market. The high end 7000 connects 512 10GbE ports with 128 line-rate 10 Gigabit Ethernet ports. The Nexus 7000 Series switches can be segmented into virtual devices, delivering true segmentation of network traffic, context-level fault isolation, and management through the creation of independent hardware and software partitions. Overlay Virtualization Transport (OTV) provides customers a simplified DCI solution by extending layer 2 VLANs over existing IP networks. We have profiled the Nexus 7000 when first released and is available here. The Nexus switches can create a two-tier architecture with the 2000/5000, providing server connectivity and layer 2 forwarding between servers. The Nexus 7000 connects the 2000/5000 to each other and the internet/intranet with high density, high reliability layer 2/3 forwarding.

Arista Networks 7500 Family of Modular Switches

Arista Networks is a new comer to the data center Ethernet market, but its management team is seasoned and customer base growing. It provides six fixed 10GbE switches; five 1/10GbE 7100 and the 1GbE 7048 along with the new Best of Interop awarding winning 7500 modular switch. The 7100/7048 switches connect servers in a Top-of-Rack configuration while the 7500 aggregates these switches and connects them to the internet and intranet. This is a two-tier, “leaf-spine” architecture. The 7500 boasts ultra high performance layer 2/3 1/10 Gb Ethernet switching for high performance computing and cloud computing data centers. The 7500 supports 384 10GbE ports, 5.7Bpps at layer 2 or 3, high packet buffers 18GB deep, ultra low port-port latency of 4.5 microseconds and 10Terabit loss less switch fabric connecting modules.

The 7500 is 10GbE port dense, compact, cloud spec fast, green and prepared for 40 and 100GbE, with a price tag 50% below competitive offerings, according to Arista. While the 7500’s hardware architecture is impressive, its operating system EOS, Extensible Operating System, offers another set of uniqueness. For example, all Arista switches run the same binary image of EOS, easing administration while hastening switch feature upgrades. EOS is a modular OS that allows partners to run their software in the Arista switch, consolidating the number of management and network appliances required, thus increasing performance while reducing energy consumption and physical space. Arista’s EOS modularity was designed as a unique state sharing architecture that separates switch state from protocol processing and application logic. EOS is built on top of a standard Linux kernel. All EOS processes run in their own protected memory space and exchange state through an in-memory database. This multi-process state sharing architecture provides the foundation for in-service-software updates and self-healing resiliency. You can listen to a podcast interview with Douglas Gourlay, VP Marketing and Anshul Sadana, VP Customer & Systems Engineering from Arista on the introduction of the 7500 Series of Ethernet switches here

HP/3Com/H3C’s A12500 Core Data Center Switches

HP has spent 25 years building and selling networking products to its worldwide client base and is currently #2 in the market, with a 21% port count share and the fastest growing networking company in the industry. The combined HP/3COM acquisition brings core switching products, the #1 market share position in China, TippingPoint Intrusion Prevention System and ProCurve edge switches, representing a new choice for clients who are frustrated by today’s current offerings. HP will combine these two entities and operate under the banner of “HP Networking.”

The HP Converged Infrastructure Architecture and FlexFabric blueprint approach the modern data center with a vision that places networking at the center of an integrated data center solution and accelerates deployment of enterprise services and applications. It is designed to drive simplicity through streamlined network designs and centralized management, enhance agility with high performance security, and accelerated provisioning, and reduce cost with energy efficiency and low total cost of ownership. Central to HP FlexFabric is policy-driven network provisioning tightly integrated with server and storage management in an end-to-end data center converged infrastructure.

HP data center solutions are purpose built, using the latest advanced systems and ASIC technologies. “A” family data center networking platforms leverage a common operating system, Comware™ and are managed with a single-pane manager, Intelligent Management Center (IMC). HP switches make use of an HP-developed technology – Intelligent Resilient Framework (IRF) – to create a resilient virtual switching fabric. IRF delivers geographic independence, distributed high-availability, resiliency and millisecond re-convergence across layer 2 and layer 3 protocols. These innovations allow customers to build a simplified, high performing, highly resilient and flat (two-tier) data center network design. They overcome the limitations of low performance/scale, high cost/latency inherent in legacy solutions, which rely on multi-tier network designs, disjointed platform operating systems and complex resiliency protocols.

A key enabler of this transformational design flexibly is the HP next-generation data center switching architecture. This starts with the flagship HP A12500 core data center switch – which is based on a 100G design that uses a multi-level, multi-plane, non-blocking switching architecture to provide high performance and scalability. The A12500 supports 6.66 Tbps of high-performance switching capacity (future support for 13.32 Tbps) and scales to 2.2 billion packets per second of forwarding performance. The A12518 supports 512 10 Gigabit Ethernet or 864 Gigabit Ethernet ports in a single chassis. Its future-proof design accommodates 40/100 Gigabit Ethernet and emerging unified network requirements such as end-to-end FCoE/Data Center Ethernet.

Force10 Networks ExaScale E Series

Force10 Networks was one, if not the first company to offer 1 and 10Gb switching solutions for high-performance computing and data center markets in Fortune 100 companies, Internet portals, global carriers, leading research laboratories and government organizations. It offers a wide range of Ethernet switching and routing products that deliver high port density and resiliency to help customers deploy a high-availability, agile and standards-based GbE and 10 GbE network fabric, while reducing power and cooling costs. Its Ethernet switching products are designed to leverage virtualized data center environments and automate Ethernet networking. For example, its VirtualScale enables management of virtual chassis. Its VirtualControl enables virtualizing logical switching and routing boundaries. For automation, Force10 has developed an architecture, which automates network resource allocation as applications and services spin up and down. This architecture is built upon its HyperLink and SwitchLink technology, two new software features implemented within its Force10 Operating System (FTOS). HyperLink provides real-time communication between Force10 switches and hypervisors or virtual switches to enable automatic provisioning of one or many virtual LANs (VLANs) across multiple switches simultaneously. The SwitchLink feature provides real-time communication with middleware orchestration tools to enable automatic provisioning and management of virtual devices anywhere in the network.

Force10’s modular Ethernet switch data center product portfolio includes the ExaScale E-Series, optimized for core deployments in large-scale, high-performance 10GbE data centers, and the C-Series, optimized for mid-range data centers. Both the E-Series and C-Series come in multiple form factors, run FTOS and are dense high performance switching platforms equipped with redundancy, availability, fault-tolerant operations and many line card options. In addition, Force10 offers the fixed configuration S-Series product line for GbE and 10 GbE ToR configurations. Force10 promotes a vision of simplified data center topologies, using integrated switching and routing in the core, using chassis based E-Series or C-Series products, and fixed configuration ToR access products allowing both 1 tier and 2 tier designs. One tier can be achieved with high density E-Series platform for server aggregation, switching at the server edge, and routing off the same platform to the Internet / WAN. The two-tier architecture can be achieved leveraging ToR switching for server aggregation along with Force10’s chassis based systems in the core. In addition to a large direct sales force, IBM OEM’s Force10’s ExaScale platform as part of IBM’s iDataPlex clustering solution. You can listen to a podcast interview with Steve Garrison, VP Marketing of Force10 on their 40 GbE offering here.

BLADE Network Technologies RackSwitch Family of Ethernet Switches

BLADE Network Technologies (BNT) has been working in the data center switch market since 2006 with much success providing 1/10Gb Ethernet switches for blade servers and top-of-rack configurations. BLADE was launched from Nortel and made up of the successful Alteon Networks group. Their success stems from their ability to identify the top-of-rack and blade switch market in ’06, along with an OEM go to market strategy that included all of the top tier blade server providers such as HP, IBM and NEC. The result is that BLADE has shipped over 8m ports, achieved 25% growth from 2008 to 2009 (in a down economy), owns 50+ % of the blade switch market, is number 3 in the Fixed 10GbE market according to Dell’Oro Group, and has demonstrated scale with at least one customer installing over 16,000 of its switches.

BLADE offers the RackSwitch family of Ethernet switches, which are ToR, 1U high switches. They include the 24-port 360ns latency RackSwitch G8100 10GbE, 48-port RackSwitch G8000 1/10 GbE aggregation and the 24-port 700ns latency RackSwitch G8124 10GbE. Over a year ago, BLADE released its virtualization software called VMready that automates network settings for VM movement ensuring that network settings migrate when a VM is moved from one physical server to another. VMready scales to a 1000 virtual port switch, is based on standards and works with most popular hypervisors.

In addition to VMready, RackSwitch’s unique attributes are found in the fact that they were designed for the data center versus being a wiring closet switch re-formatted for the data center. For example, the RackSwitch BLADEOS supports CEE for unified fabrics, uplink failure detection, virtualization, dual homing for servers, low (80-170Watts) power consumption, back-to-front or front-to-back airflow and very low latency in the 700-360 nanosecond range.

Voltaire’s Vantage 8500

Voltaire has a long history in high performance computing and data center networking as it is one of the key leaders in the InfiniBand market. Voltaire enjoys distribution relationships with HP and IBM, as well as Bull, Fujitsu, NEC, SGI and Oracle. The result is a 100% + year over year revenue growth for Q1 as reported on May 5th. Last October, Voltaire entered the 10 GbE market with the introduction of its Vantage 8500 Ethernet layer 2-core switch. The Vantage 8500 boasts less than 1 microsecond of latency, a low 10 watts per port power consumption and 288 wire speed 10GbE ports in a 15U high chassis. The Vantage 8500’s unique industry contribution is that it’s based on converged enhanced Ethernet (CEE) technology providing InfiniBand-like capabilities to the Ethernet data center. In fact, Voltaire has ported many of InfiniBand’s key characteristics to the Vantage 8500 such as a lossless switching fabric, multi-pathing, virtualization, fabric-wide congestion management and QoS.

From a network design point of view, Voltaire supports a two tier network architecture that enables a simplified, ‘flat’ data center network and puts an end to the era of the over-provisioned network. Voltaire’s design centered on the Vantage 8500 is to support a two-tier data center network that scales from hundreds to a few thousand core ports, which requires high capacity, non-blocking 10 Gigabit Ethernet core switches. By clustering up to twelve Vantage 8500 switches together, IT business leaders can expand their data center to many thousands of servers while preserving the efficiency and price-per-port, without degrading performance or latency which occurs in traditional hierarchical network designs. To support ToR implementations, Voltaire and BLADE Network Technologies announced recently a partnership where BLADE ToR RackSwitches are aggregated by Voltaire’s Vantage 8500, rounding out the two-tier data center Ethernet network architecture.

The Vantage 8500 also features software-based capabilities to address virtualized and converged data center environments. Voltaire’s Unified Fabric Manager™ (UFM) software, application acceleration software and management OS (VT-OS) provide management and performance enhancement tools. These tools were developed and optimized in InfiniBand environments and are now available for Ethernet-based data centers. Voltaire’s recently introduced Unified Fabric Manager™ (UFM™) 3.0 software orchestrates physical and virtual switches delivering guaranteed levels of service per application. It’s the first and only Ethernet fabric management software that dynamically orchestrates end-to-end virtual machine connectivity for multi-vendor, scale-out data center networks.

Avaya’s VSP 9000

During the April 2009 Las Vegas Interop trade show, Nortel committed to the data center Ethernet market with the announcement of its Virtual Services Platform or VSP 9000 switch, which supports up to 27 Terabits per second (Tbps) of backplane switching and 240 10GbE ports per chassis at first release. Avaya announced their commitment to the VSP 9000 and said that it will be generally available in the second half of 2010 while already in controlled availability. The VSP 9000 is built upon the Ethernet Routing Switch 8600/8800 software providing a proven software foundation, mid-plane architecture, a fully programmable network processor unit for flexible data forwarding and carrier-grade Linux.

The VSP 9000 is designed to deliver high-density 10GbE, 40GbE and 100GbE. Its design center is rooted in highly dense connectivity environments that are all mission critical, by definition. Early testing validation of the VSP 9000 promises to provide ultra-high reliability and availability delivering below 50ms failover support, which is critical to eliminate application disruption thanks to its patented hardware failure detection differentiation. The VSP 9000 switch fabrics are lossless Ethernet capable and therefore well positioned to support the next generation Data Center requirements for convergence of storage onto the Ethernet infrastructure.

The VSP 9000’s unique network architecture is found in its ability to cluster four switches together, in that the total architecture exceeds 100 Tbs, with the number of 10GbE ports per rack being up to 720. Avaya continues to invest in Switch Clustering technology (Active/Active resiliency model) such as SMLT (split multi-link trunking) and RSMLT (routed-SMLT), which provides link, switch and router redundancy mechanisms. Three modules are being introduced in the first VSP 9000 release, a 24 port SFP+ for 1 GbE and 10 GbE connectivity, a 48-port of SFP module in addition to a 48-port 10/100/1000 TX module. Future plans include 40GbE and 100GbE interfaces, and even higher-capacity Switch Fabric modules.

Juniper Networks’s EX8200 & EX4500

In January of 2008, Juniper Networks launched its much-anticipated entry into the enterprise Ethernet switch market. Juniper’s focus is on the enterprise data center, campus and branch, as well as the service provider market. Juniper provides a suite of Ethernet switch products, including the EX4200 with Virtual Chassis technology for GbE Top-of-Rack (ToR) and End-of-Row (EoR) data center access, the EX2500 24-port and new EX4500 48-port 10GbE ToR switches, and the EX8200 high-density, high-performance line of modular Ethernet switches.

According to Juniper, it simplifies customer enterprise LAN architectures and advances the economics of networking via its most recently launched initiative called the “new network” for data centers. Juniper’s “new network” promises critical innovations in automation, virtualization and fabric technologies. These innovations are to reduce time to operation by up to 50 percent and eliminate up to 35 percent of data center networking capital expenditures. One aspect of the “new network” is a simplified two-tier network architecture, which may be reduced to one when “Project Stratus” is completed with IBM. The reduction of a three-tier architecture to two is accomplished by utilizing Juniper’s Virtual Chassis fabric technology in the access layer, in conjunction with its high-density, high-performance platforms such as EX8200 and EX4500 in the LAN core, thus eliminating the aggregation or distribution layer. According to Juniper, collapsing the distribution layer reduces complexity in the data center as well as campus networks by reducing the number of managed devices by up to 89%, providing up to 39% savings in space, 44% savings in power and reducing the number of switch interactions by up to 99% compared to three-layer networks. According to Juniper, this approach improves application performance by also reducing latency up to 77% compared to three-layer networks. Note that these claims and numbers are Juniper’s and not mine.

At the core of Juniper’s data center Ethernet product family is the EX8200 line of modular switches. The EX8208 and EX8216 are eight and sixteen-slot modular switches. The EX8216 sports a maximum of 640 10GbE ports and 1.92Bpps and 6.2Tbps backplane speed. The EX8200 is said to support 40GbE and 100GbE interfaces in the future. The EX8200s connect either EX4200 GbE or EX2500 and EX4500 10GbE ToR switches together while providing access to internet/intranet. All Juniper switches run Junos, the network operating system that provides reliability and availability features, developed for the high-performance enterprise and service provider market.

Brocade’s NetIron MLX Series of Switches

In July of 2008, Brocade had purchased Foundry Networks, catapulting them into the Ethernet switch market as one of the top five Ethernet switch/router vendors by revenue. Brocade, with its long history of data center storage, saw that converged I/O was going to happen and prepared the company to participate in this market. At the high end of Brocade’s data center Ethernet switch products is the NetIron MLX-4, MLX-8, MLX-16 and MLX-32 routers, which support 4, 8, 16 and 32 I/O module slots, respectively. We’ll focus on the high end NetIron MLX-32 here, which has been in production since August 2006.

The NetIron MLX-32 boasts a total of fully redundant non-blocking 7.68 Tbps switch fabric capacity. Brocade says that the MLX-32 can forward some 2.284 Bpps of Layer 2/3 packets and support 1,536 and 256 non-blocking 1 GbE and 10 GbE ports, respectively. Note that the new high density 10 GbE was announced the same day as this Research Note was made public. All four NetIron MLX systems are designed for non-stop operation, supporting 1:1 management module redundancy, N+1 switch module redundancy, M+N power module redundancy and N+1 fan redundancy. The NetIron MLX architecture is an adaptive self-routing Clos switch fabric with a virtual output queue (VOQ) design. This non-blocking architecture is optimized for maximum throughput and low latency for all packet sizes.

Tweet

During last week’s Cisco Q3 FY10 quarterly financial conference call, John Chambers, Cisco’s CEO, said something that impressed and shocked me. The company has been quiet about the growth rates for its Nexus line of data center switches until this call. What shocked me was that the Nexus 7000 is now on an annualized run rate of $1B, yes that’s Billion with a B! I remember being interviewed by John Markoff of the NY Times in Jan ’08 about the Cisco’s Nexus and Juniper’s yet to be announced Ethernet switches. In just 27 short months, the Nexus product line including the 7000, 5000 and 2000 represents a $1.4 B run rate of revenue to Cisco. Another insight gained from this ramp up is that the data center networking trends that we’ve discussed here in various Lippis Report Research Notes are powerful demand drivers for Cisco and other companies participating in this lucrative emerging market and its just starting! Companies such as Arista Networks, Force10 Networks, Blade Network Technologies, HP/3Com/H3C, Voltaire, Avaya, Brocade, Juniper, et al, have unique positions and offerings to participants in the burgeoning market. In this Lippis Report Research Note, we review the mega trends driving high market growth. We save a product review of each of the suppliers for our next Lippis Report Research Note.

In addition to the run rate numbers above, Cisco also posted a milestone of 1 million 10 GbE ports shipped, providing a strong indicator that the 10GbE market is nearing a tipping point to high volume, as pricing drops and its use accelerates. The following are mega trends driving this tremendous market growth. Traffic demand drives bandwidth and that’s the first mega trend.

Traffic Profile Changes: Gone are the days when data center networks primarily shuffle asymmetric email messages and low bandwidth client-server computing applications between endpoints and servers. Best effort data delivery, where latency was secondary to delivering data accurately, has changed to being a paramount design element where 10 milliseconds means the difference between losing a customer or capturing revenue. Traffic is now highly mixed, moving around a data center in near Brownian motion between servers, storage, internet and intranet thanks to a plethora of old and new applications such as mash-ups, VoIP, search, backups, storage access, emerging converged I/O etc. In addition to Brownian motion traffic flows and low latency requirements, the volume of traffic continues to skyrocket and shows no sign of abating. Remember when the Dow dropped by 1000 points in early May of this year? Financial services firms saw an average of 40 times the amount of traffic in their data centers as traders responded to the drop. There is no better driver for traffic volume as financial markets in turmoil. The traditional model of over subscribing data center bandwidth by as much as 80:1 is the norm, and IT business leaders are looking for a more efficient model.

Workload Mobility: With the advent of server virtualization IT leaders are able to decouple an operating system from its underlying server hardware and increase the number of instances an operating system can be replicated on a single server. Server virtualization reduced the number of physical servers needed and in the process reduced energy and cooling requirements. Now that an operating system only needs to know which hypervisor it’s running on, that operating system instance and the applications it services can be moved from one physical server to another in near real-time with the click of a mouse, thus providing workload mobility or portability as well as a rapid application procurement tool.

So what does all of this have to do with networking? A lot, first moving these workloads around a data center consumes huge bandwidth and has low latency requirements to driving raw bandwidth requirements. Secondary, and most importantly to the industry, is that networking or should I say the rigid structure of IP addressing/VLANs, etc are impeding the automation of these workload moves. In short, the data center network needs to be reconfigured when VMs are moved from one physical server to the next in the same data center and it simply does not work if a VM is moved between data centers separated over distance, between a data center and a cloud provider and between cloud providers. This is the area of the infrastructure 2.0 working group.

Doug Goulay said it best in his recent Network World post.

“When moving VMs between machines there is a caveat:  if you want your TCP connections and IP addressing to stay intact the receiving physical host must be capable of supporting the same IP address that the VM moving to it is actively using.  This means that both physical hosts have to be in the same subnet or in the same VLAN depending which layer of the network you are looking at.  Since the largest number of physical servers that can be supported doing this is around 64 it doesn’t change the addressing architecture too much, unless the servers are in different data centers, or are connected to different access layer switches that talk to different aggregation layer switches.  If this is the case the network architecture all of a sudden starts dramatically impeding the movement of VMs:  either VM mobility is impeded, or the network is redesigned. 

Some people often ask me, “can’t I do this with DNS?’  In short, no.  DNS is cached at many client sites, ignoring your TTL.  Additionally, DNS is cached on many PCs for the life of an application session.  If you try to change the IP address of your backup server while you are in the middle of a 2GB backup do not expect the connection to continue.  TCP doesn’t work this way.”

Increased Density: It’s no secret that data centers are bursting from the seams as the economic down turn kicked large IT capital outlays down the road until economic conditions improved. Business leaders have been postponing increasing data centers space, that is square footage, while power density has grown exponentially, until very recently, as cooling requirements increase unabated. Power and cooling capacity are the primary constraints to data center expansion. To deal with these realities, IT business leaders are left with only one option, appropriate capital to either upgrade power and cooling systems or build a new data center. The impact of high energy densities is that server hardware is no longer the primary cost component of a data center. The purchase price of a new (1U) server is now exceeded by the capital cost of power and cooling infrastructure to support that server and will soon be exceeded by the lifetime energy costs alone for that server. In short, energy costs are on their way to dominate data center economics.

To help mitigate these trends, the new data center switches offer increased server connection density at lower energy consumption levels. In addition, their own energy consumption to shuffle packets around has been reduced, for some by as much as 50%. To connect an every increasing dense set of servers, new generation of data center switches boast a two tier network architecture to support thousands to tens of thousands to hundreds of thousands of servers. To deal with high server density connectivity, server access is via a leaf switch, while leaf switches and storage connect to a modular spine switch. The two-tier approach offers efficient connectivity density, low latency albeit this depends highly upon the internal switch design, and is ready to support consolidated I/O.

Consolidated I/O while early in its adoption cycle will go a long way in reducing power consumption of servers as they will have a single network interface for both storage and networking. In addition, consolidated I/O promises to reduce the need for a separate storage switch too again reducing capital, energy and cooling cost.

Back to server density. Server density will only get, well, more dense. If the industry trajectory of cloud computing is realized any where near what the conventional wisdom dictates, then there will be more and more highly dense cloud computing sites supporting an ever increasing number of enterprise, government and consumer applications. How many cloud computing sites does the US need to support all IT applications? With nearly 16 million servers installed nation wide, according to IDC, and with each cloud computing site supporting hundreds of thousands of servers, then perhaps the number of cloud computing sites would be in the hundreds. While its unrealistic that all US enterprises and governments will be hollowed out of their data centers and applications via cloud computing with today’s technology and business control believes; the trend line is clear, there will be a smaller number of very large cloud providers delivering applications to a wide range of customers. Almost like a supernova transforms into a black hole, applications will not be able to escape the gravitational pull of the scale and economics of cloud computing if the industry gets anywhere near this size scale.

The networking industry has been busy adapting to these powerful trends with new internal switching architectures, data center network architecture and automation. Internal switching architectures are being designed with high internal switching capacity in the terabit rage, lower energy consumption in the 10W/port range, low latency and of course high port density. The data center network architecture most are progressing toward is a two –tier leaf-spin approach mentioned above. These switches possess the highest levels of reliability, serviceability and redundancy, as networking is at the center of this massive server connectivity density.

Network automation is another area of investment where VMs can be moved within and between data centers, as well as between data centers and cloud providers, plus between cloud providers. A few companies are addressing network automation, but this is a huge issue that the industry needs to wrap its arms around and provide a scalable solution.

In the next Lippis Report Reseach note, we’ll review Cisco, Arista Networks, Force10 Networks, Blade Network Technologies, HP/3Com/H3C, Voltaire, Avaya, Brocade, Juniper, et al, and highlight their unique positions and offerings to participants in the burgeoning market.

Tweet

This past Interop in Las Vegas was one of the best I have attended, since even before the economy took a noise dive in 2008. The tone and level of excitement of the industry’s growth potential was refreshingly up beat from the hundreds of IT and vendor executives I talked with. While the size of Interop is a small fraction of what it was in the late 1990s, (70k attendees with over 600 exhibitors to ~ 15K attendees with ~ 200 exhibitors) it still provides a pulse of the networking industry. In fact, Interop has come full circle, back to being a networking event even though it has added other topics. You have to give Dan Lynch credit for creating such a long lasting venue for our industry. Congratulations to Cisco, Arista Networks, HP/3Com, Mallonx for winning best of show in their respective categories and for Arista for winning Best of Interop. In this Lippis Report Research Note I provide the key industry themes that were evident at Interop this year.

The following are my observations of Interop 2010 in LV.

Network Infrastructure Takes Center Stage: Even though Interop provided attendees with thirteen educational content areas including cloud computing, IT security, Enterprise 2.0, etc., it’s the changes taking place in the network infrastructure business that was front and center, loud and clear. The following was the topic of conversations throughout Interop:

• Cisco’s introduction of its Best of Show winning Aironet 3500 Series Access Point with CleanAir technology,
• Arista Networks’ introduction of and winning Best of Show and Best of Interop for its Arista 7500 10Gb modular Ethernet cloud computing switch,
• HP’s closing of its acquisition of 3Com and winning Best of Show for its TippingPoint Virtual Controller,
• HP’s planned acquisition of Palm,
• Avaya’s reassertion in the network business with the introduction of its Ethernet Routing Switch 8800, WLAN 8100 and Advanced Gateway 2330,
• Voltaire’s new Vantage™ 8500, 10 GbE Layer 2 core Ethernet switch,
• Force10’s open network automation demonstrations and 40GbE module

With the above announcements and accomplishments, two thoughts come to mind. First is that Interop is finally back to core networking issues, and second, the above announcements provide a window into the huge changes that are taking place in our industry.

New Industry Structure Emerges: The networking industry has been consolidating for some time now and will only continue. Corporations have some $2T in cash and equivalents on their books, which will be put to work acquiring companies and investing in growth markets. The big growth market in our industry is the fundamental change IT is starting to progress through. HP’s actions last week provided a preview of what’s to come.

HP stole the headlines last week with their shorter then expected closing of their 3Com acquisition, in addition to their intent to purchase Palm. HP realizes that the IT industry is structurally changing away from fixed desktop computing accessing corporate applications hosted in data centers, to mobile computing accessing applications hosted in corporate data centers and cloud computing facilities. The big winner in this transition is networking, as without it, cloud and mobile computing will not happen. Palm gives HP a smartphone platform to participate in the mobile computing market while 3Com expands its corporate networking portfolio significantly.

HP vs Cisco: The buzz at Interop around HP was how it will compete with Cisco. The HP executives and booth personnel were the most energized I have ever seen. HP views their competitive advantage along the lines of innovation, open network architecture and economics. Thinking it through however, HP’s focus will be more on supply chain efficiencies to drive down their cost of producing networking gear close to server economics while leveraging their massive and productive channel to gain market share.

The supply chain efficiency is a great idea, but will take at least a year if not more to deliver. The thinking here is that a 40 Watt power supply is the same, independent of its final designation, as long as it powers a server, router, etc. So can HP redesign their product lines for common components where they gain huge cost efficiency thanks to volume purchasing? Perhaps, but this will take time. Their channel strength should deliver results in the short term. If HP executives are correct and that the market wants a strong number two networking provider, then its channel should produce fairly quickly. If it doesn’t, then this premise is questionable. HP networking is about $5B now; if it doesn’t grow faster then the industry by a significant amount next year, then something is wrong.

Remember HP is competing with a $40B powerhouse that is Cisco Systems, which has a massive and productive channel too that are energized to sell, not only networking gear, but also unified communications, Cisco’s new server platform UCS and video equipment. As for innovation, HP is a great operational company therefore expect them to take cost out of their products. Nevertheless, Cisco is the innovation king, thanks to its systemic incorporation of innovation in product development, plus its ability to integrate acquisitions quickly and materially. Cisco does not only innovate in its products, but around them, offering architected solutions. Examples of this are everywhere, including its borderless network architecture, EnergyWise, UCS, the new 3000 series stackables, Power over Ethernet Plus, its’ ISR G2, the Nexus line of data center switches, its’ approach to integrated network security, etc.

Here’s an example of the power of innovation. A client and Lippis Report subscriber has funded a new $20M data center. During their due diligence, they visited Dell, HP, IBM and Cisco. This CIO will go with Cisco’s UCS. The reason is that during the customer visit, Cisco first described the major direction and trends in data center virtualization and cloud computing in such a way that my client said “Cisco looked into the future and designed UCS to exploit these changes while all the other vendors were selling their old blade systems”. Now this is significant, as this CIO only purchased equipment from market share leaders, that is, he would buy from HP for servers, Dell for desktop systems, Cisco for networking, Avaya for communications etc. Cisco’s innovation in UCS changed his long-standing principal of buying only from market share leaders and will buy UCS for this new data center. So the basis of competition between Cisco and HP will fall into three categories; innovation, supply chain management and channel productivity.

A Mobile and Cloud Computing IT Model Is Disrupting The Status Quo

The Interop announcements above were aligned with this new world order of IT. For example, Arista Networks delivers a massively powerful 10GE switch for cloud spec data centers and high performance data center environments. Clearly investment in cloud infrastructure is a growth market which motivated Voltaire to enter the Ethernet market and leverage its Infiniband experience to deliver converged I/O for both Infiniband and Fiber Channel Over Ethernet (FCoE). As computing is in a rapid technology innovation stage thanks to server virtualization, networking has lagged in its ability to automate network changes brought on by VM moves. This has motivated Force10, F5 and Infoblox to demonstrate innovative approaches to automating network changes so that network administrators do not have to be involved in the process of VM moves and/or the provisioning of new IT services as demand is increased and/or decreased.

It’s clear that HP networking products has gained awareness and will receive consideration. As HP opens the consideration door, Avaya wishes to enter too with its refreshed and new data networking products. Avaya is now lead by experienced IP networking executives that understand voice and data. The Nortel channel also understands voice and data. Ever since Avaya closed its acquisition of Nortel, those channel partners that put selling Nortel gear on hold, have started to come back. They are comfortable now as stability, R&D funding and a strong financially viable company has emerged.

The networking industry is an upside down pyramid with Cisco at the top followed by a few others in the billion-dollar range. Then there are a number of $100M sized firms followed by a few start-ups. The successful firms will be the ones that embrace the new world order of IT that is being brought on as IT leaders de-emphasizes desktop computing and invest in mobile plus cloud computing.

Tweet

By Voltaire

This paper describes the challenges inherent in existing Ethernet solutions and how Voltaire’s new scale-out Ethernet architecture effectively addresses those challenges. Voltaire’s InfiniBand products and fabric management solutions have been addressing Data center architecture changes for years. Today, Voltaire delivers the fabric for the world’s largest supercomputers, the world’s fastest financial trading platforms, and the world’s most scalable database machines. Soon, Voltaire will extend its product offering to include Converged Enhanced Ethernet CEE switches and software, allowing end users with less demanding performance requirements to benefit from a far more scalable Ethernet fabric that also lowers overall fabric costs, lowers power consumption, has greater efficiencies, and simplifies management.

Find out how by downloading this white paper.

Get the White Paper