Lippis Report 181: Early Results of the Lippis Report Open Industry Cloud Network Evaluation of 10/40Gbps Ethernet Fabrics at Ixia’s iSimCity
During the weeks of October 10 and October 31, 2011, at Ixia’s iSimCity, the Lippis Report conducted its third industry test of cloud networking data center switches operating at 10 and 40GbE. In just six short months, the industry has moved forward by breaking all previous records of data center switch speed, power consumption, port density and bandwidth. We added four products from three vendors to the eleven products from eight vendors already tested. We now have data on fifteen data center switching products from nine vendors in the new report to be released after Thanksgiving. During May 2011 Interop, we had eleven vendors provide verbal commitment to participate in this Fall industry test (remember it is free for vendors to submit products to test). As the deadline for signed agreements came, this field of eleven dropped to three because their products were simply not ready. 40GbE is hard, and thus you have to give credit to the vendors that go through the testing—in this test, those vendors are Extreme Networks, Brocade and Alcatel-Lucent. These firms have high performance data center switching product that is Enterprise and Cloud service provider ready. In this Lippis Report Research Note, we share our insights gained from testing all these products and provide the topic cloud networking industry trends taking shape now.
Arista Network’s Ken Duda Explains VxLAN, the first Virtual Networking Protocol
To assist IT business leaders with the design and procurement of their private or public data center cloud fabric, the Lippis Report and Ixia have conducted an open industry evaluation of 10GbE and 40GbE data center switches. These test were conducted at the Ixia iSimCity Santa Clara, CA, laboratories. The resources available for this test at Ixia’s iSimCity are out of reach for nearly all corporate IT departments with test equipment on the order of $9.5M, devices under test on the order of $2M, plus costs associated with housing, power and cooling the lab plus 22 or so engineers from around the industry. It’s our hope that this industry effort will remove performance, power consumption and latency concern from the purchase decision, allowing IT architects and IT business leaders to focus on other vendor selection criteria, such as post sales support, platform investment, vision, company financials, etc.
Next-Generation Networks: Business Value for Today and Tomorrow
The Lippis Report test, based on independent validation at Ixia’s iSimCity, communicates credibility, competence, openness and trust to potential buyers of 10GbE and 40GbE data center switching equipment as the tests are open to all suppliers and are fair, thanks to RFC and custom-based tests that are repeatable. The private/public data center cloud 10GbE and 40GbE fabric test was free for vendors to participate and open to all industry suppliers of 10GbE and 40GbE switching equipment, both modular and fixed configurations.
40 Gb and 100 Gb Ethernet
Ixia supplied all test equipment needed to conduct the tests while Leviton provided optical SPF+ connectors and optical cabling, and Siemon provided copper and fiber optic QSFP+ cables and transceivers for 40GbE connections. Each 10GbE supplier was allocated lab time to run the test with the assistance of an Ixia engineer. Each switch vendor configured its equipment while Ixia engineers ran the test and logged the resulting data.
Network Virtualization using Shortest Path Bridging and IP/SPB
While we can’t just yet release data on the latest round of testing, we can share with some of the records that were broken. We measured for the first time core switch latency in single digit microseconds and single digit Watts/10GbE power consumption. Also for the first time, we measured power consumption in top of rack switches power consumption in very low single digits. We measured how fast core switches can forward packets at very high density being 256 0GbE plus 24 40 GbE ports, and this was only a third of this switch’s port density. We measured congestion, IP Multicast, cloud simulation, latency and throughout for 24 40GbE, a first in this series of industry test.
Building Cloud-Scale Networks Using Open Fabric Architectures
In just six short months, data center Ethernet core switching has increased in speed by nearly a factor of 10, its power consumption dropped by nearly 50% while port density increased by nearly 3 times. In ToR switching, power consumption is down by over 50% while these products add 40GbE uplines and storage enablement such as direct Fiber Channel and/or Fiber Channel over Ethernet connections. With all of these advances, the one thing that is holding steady is pricing as the industry serves up more features for the same or slighly more dollars.
The Fall Lippis/Ixia test proved to show that the industry is advancing at a breakneck pace. And we do expect to see more products being submitted for test in the Spring 2012 test. Based upon three series of industry test, the following trends have become evident.
Faster Forwarding: While the Fall test showed new records in latency measurements—that is, how fast a switch can forward packets at zero packet loss or 100% wire speed throughput—switching products will get even faster. While it’s anticipated that the Fall core switch latency records will not be broken in 2012, ToR switches will show significant improvement getting into the range of 100ns with 100Mbs Etherent uplinks.
Hybrid Cut-Through and Store and Forward Switching: To make switches faster, merchant silicon vendors have taken a new look at packet forwarding. It used to be that Ethernet switches were either cut-through (CT)—where packets were not stored for processing—or store and forward (S&F)—where packets were stored, processed then forwarded. Now switches use both forwarding techniques, where the first few hundred packets are forwarded via S&F and the rest, CT.
IP Multicast Rises in Importance. With the huge increase in video traffic, IP Multicast performance and, in particular, how switch replicator chips perform will be increasingly scrutinized. We tested the lowest latency of IP Multicast during the Fall test, indicating that switch speed of forwarding IP Multicast is becoming an important product selection criteria.
40GbE Arrives in 2012: Due to 40GbE component shortages in Asia, most vendors could not participate in the Fall test. These shortages will abate over the next quarter, creating a wave of new 40GbE modules and products during 2012. With 40GbE being 3 to 4 times the cost of 10GbE, look for a quick ramp up in ToR uplink and core switching modules.
The Rise of Merchant Silicon: Merchant silicon from Broadcom, Fulcrum MicroSystems and Marvell manufacture low-cost chips for network switches that have lowered the risks for new entrants into the hot data center Ethernet fabric market. In the last few months alone, 10 companies announced new products based upon one of the above merchant silicon 10 and 40 Gbps Ethernet chips. We expect to see enhancements to network virtualization, support for software-defined networking and a focus on buffer architecture.
New Set of Best of Breed Products: With merchant silicon competing with custom ASICS, a new class of best of breed products has emerged with more to follow during 2012. These products will be pushing the envelope on packet forwarding speed, power consumption, port density, storage enablement and network virtualization, thanks to VXLAN/NVGRE support and software-defined networking.
Software-Defined Cloud Networking: As best of breed Ethernet data center switches get more powerful while consuming less power, these products will need to tap into a growing software base to add value to these networking products. Software-Defined Cloud Networking or SDCN promises to ignite a cycle of innovation that shifts competitiveness to network software that enables firms like Cisco, HP, Extreme Networks, IBM, Arista Networks, Force10/Dell, Avaya, Huawei, Brocade, Juniper Networks, Alcatel-Lucent, Enterasys and others to compete by rapidly adding software features to low-cost merchant silicon-based network products. There are two approaches to SDCN: 1) OpenFlow based that defines an open interface between switches and a controller or 2) hypervisor virtual network controllers that plug directly into switches.
The next Lippis Report test at iSimCity is scheduled for the Spring of 2012. We expect more 40GbE products plus the observation and measurement of the above trends.