Lippis Report 182: Top 10 Findings: The Cloud Network Industry Test of 10/40GbE Fabrics
The Fall 2011 Open Industry Network Performance and Power Test Report is now available. Since our Spring 2011 test, we added four products from three vendors to the 11 products from eight vendors already tested. We now have data on 15 data center switching products from nine vendors in the new report to be released after Thanksgiving. Our cloud networking test of 10 and 40GbE is now the industry benchmark for cloud networking. In fact, only those companies that are sure of their product(s) enter the test at Ixia’s iSimCity. We found that 40GbE is hard, and thus you have to give credit to the vendors that go through the testing—in this test, those vendors are Extreme Networks, Brocade and Alcatel-Lucent. These firms have high performance data center switching product that is Enterprise and Cloud service provider ready. In this Lippis Report Research Note, we share our the top 10 findings from this round of testing. Lippis Report subscribers can download the 125-page report here, free of charge.
What We Have Learned From Eighteen Months of Testing
To assist IT business leaders with the design and procurement of their private or public data center cloud fabric, the Lippis Report and Ixia have conducted an open industry evaluation of 10GbE and 40GbE data center switches. These test were conducted at the Ixia iSimCity Santa Clara, CA, laboratories.
The Lippis Report test, based on independent validation, communicates credibility, competence, openness and trust to potential buyers of 10GbE and 40GbE data center switching equipment as the tests are open to all suppliers and are fair, thanks to RFC and custom-based tests that are repeatable. The private/public data center cloud 10GbE and 40GbE fabric test was free for vendors to participate and open to all industry suppliers of 10GbE and 40GbE switching equipment, both modular and fixed configurations.
Catalyst 6500 Upgrade From Sup720 to Sup2T
While Lippis Report subscribers can download the full report here, below are our top ten findings from conducting these three rounds of testing. The Fall Lippis/Ixia test proved to show that the industry is advancing at a breakneck pace. And we do expect to see more products being submitted for test in the March 26th Spring 2012 test. Based upon three series of industry test, the following top ten findings have become evident.
Securing and Gaining Deep Visibility To What Applications Are Flowing Over Your Enterprise Network with NetFlow
1) 10GbE Top of Rack (ToR) and Core Switches: 10GbE ToR and core switches are ready for mass deployment. There have been 15 new switches since Interop 2011, and there will be 15 more launched during 2012.
2) Fastest Ethernet Switches under the Milky Way: We are in the 500 ns ToR and 2 us core switch era. For core switches, the Extreme x8 is two to nine times faster than any other core switch we have tested. ToR switch latency will decline to 100s of ns within two years, thanks to better merchant silicon plus Phy-less designs. Core switch latency will decline to ns area with 40 & 100GbE speeds plus the next generation of merchant silicon.
Fundamental Changes In Data Center Networking Afoot
3) Merchant Silicon Proves Its Value: Most switches entering the Lippis Report test are based upon a new generation of merchant silicon. They are based upon a single chip design from Broadcom, Fulcrum or Marvell. Broadcom currently leads this space and is becoming the Intel of the networking industry.
4) Switch Vendors Differentiate Products Mostly on Software: There are differences between suppliers at both the box and system level. At the box level, we find latency, congestion, power and software differences. We also find differences in how these vendors propose building cloud networks. There is difference in cloud network architecture approach, such as support for TRILL, SBP, MLAG and/or ECMP. There are differences in network services and virtualization aware support.
The Policy-Governed Network
5) Ability to Support Storage Engagement: Most core and ToR switches demonstrated throughput performance without loss and low latency variability to support storage enablement. Most switching firms will be offering a range of convergence options during 2012, including ToR switches with direct fiber channel connections and/or FCoE, ATA over Ethernet and iSCSI over Ethernet support.
A Comprehensive Testing of Cisco Systems Catalyst 6500 Sup2T
6) 40GbE Is Ready: 40GbE support as a downlink from ToR to End of Row (EoR) and in the core at density is here, ready and performs as advertised. In addition 40GbE cost is approximately 3 to 4 times that of 10GbE, making 40GbE favorable from a pricing point of view too. There are plenty of ToR switches that support multiple 40GbE options such as Alcatel-Lucent OmniSwitch 6900-X40, IBM BNT RackSwitch G8264, Arista 7504 Series Data Center Switch, Dell/Force10 S-Series S4810, etc. In the core switch market, there is only one company with high-density 40GbE, and that’s Extreme BlackDiamond X8 with 192-40GbE. But we expect at least four more high-density 40GbE core switches to be launched in 2012. Note, at times, we did some observe difficulty with preamble and equalization at the physical QSFP+ level causing packet loss, but this we mitigated through software control.
Cloud-Ready Network Architecture
7) Low Power Consumption: Power consumption in networking devices is dropping precipitously. All ToR and core switches offer low power consumption with energy cost over three-years estimated between 1.3% and 4% of acquisition cost. Two of the most impressive results we observed was that of Extreme Networks’ BlackDiamond X8 Core switch and Brocade’s VDXTM 6730-32 Data Center ToR Switch. The Extreme X8 consumed a low 5.2W/10GbE; that’s nearly as low as a Christmas bulb. Brocade’s VDXTM 6730-32 Data Center ToR switch consumed a low 1.5W/10GbE; that’s about 20% of the power a Christmas bulb consumes! In addition to power consumption all switches support front-rear or rear-front airflow in support of hot/cold aisle designs.
Accelerating Unified Communications with an Enterprise-Wide Architecture
8) Virtualization Scale Support: All switches in this test are able to support far greater numbers of VMs and physical servers than their physical ports allow; that is, their logical networking scales to support very large virtualized data center infrastructure.
9) 10/40GbE Recommended as Cloud Network Fabric: From server connections to ToR to core switching plus storage enablement and virtualization aware software 10GbE is recommended as the fabric for cloud networking environments. We recommend that IT business leaders take full advantage of server I/O at 10Gbps bandwidth and low latency as it will provide the highest performance and greatest data center design options moving forward. With 10GbE ToR switch cost per port in the $350 to $670 range, core switch cost per 10GbE port in the $1.2K to $6K range plus 40GbE cost per port in the 3 to 4 times of 10GbE, Ethernet technology is well segmented for data center needs. 10GbE and 40GbE switches have the logical networking to support highly virtualized infrastructure with dense VM:physical server ratios of 30:1 to 60:1. With ToR and core switch latencies in the 500 ns to 2 microsecond range, the industry’s 10GbE switches possess the raw performance and capacity to support storage enablement, albeit this area is evolving.
The Hitchhiker’s Guide to the New Data Center Network
10) Software-Defined Networking/OpenFlow: While not tested during these past three rounds of test, software-defined networking or SDN and OpenFlow will be increasingly important during 2012 as companies seek to differentiate their high performance switch products with increased features and functionality. SDN with OpenFlow promises to offer such added value.
The next Lippis Report test at iSimCity is scheduled for the week of March 26, 2012. We expect more 40GbE products in the Spring.