The last week of January 2008 kicked off a year in which Cisco delivered a new switching architecture for the data center and Juniper finally made public their LAN switching products. The announcements were a day apart and many in the industry thought that these products would be head to head competitors, like the companies themselves. When you look into the two announcements you find that Cisco’s Nexus is a generation above Juniper’s EX-series of switches as Cisco’s vision of Nexus is beyond server connectivity and looks to redefine how all data center servers, storage and networks are wired. Juniper’s EX-series of LAN switches can have a role in the data center, but they’re designed for enterprise desktop and server connectivity. In short Juniper does not offer anything more compelling than what is already available from Cisco’s Catalyst 6500 and 4500E switches, Foundry, Extreme, ProCurve and Force 10. I am disappointed that Juniper didn’t take this opportunity to innovate and offer solutions to today’s pressing enterprise infrastructure problems. Below is an assessment of the Cisco and Juniper product announcements.

You can tell that Cisco has spent a lot of time with customers while in the design stage of Nexus. It possesses so many innovations that could only come from customer interactions which guide smart engineers. Cisco spent time understanding customer best practices so that Nexus’ system default behaviors align with these best practices. The result is a unique switch with attributes tailor- made for the data center.

Cisco only seems to build new networking platforms when there’s a bandwidth/speed and service shift in the market. When the enterprise market started to transition from 10/100 Mbs shared Ethernet to switched Ethernet with VLANS, that triggered a combined bandwidth and service change, which forced the need for a new platform. As the market adopted 1 GbE, QOS and layer three forwarding, Cisco introduced the Catalyst 6500 platform. The Catalyst 6K was able to absorb next generation network technology to supporting 10 GbE. But now extreme high-density 10GbE is needed while the market is poised to adopt high densities of 40 GbE and 100 GbE as Ethernet becomes lossless, fiber channel over Ethernet capabilities emerge ushering in an era of unified data center fabric. In the data center to support the new service of server virtualization increased bandwidth and density per server is driving the need for a new networking platform.

Cisco analyzed these market changes and decided, years ago, that it was time to introduce a new platform for data center networking as there had been a bifurcation between enterprise campus and data center requirements. For example, Power over Ethernet is not needed in the data center while reliability and performance requirements are different in the metro, campus, branch, et al., areas of the network. It is this bifurcation that requires Cisco to continue the Catalyst 6500 although targeting it more at traditional enterprise networking.

Enter Nexus

Cisco’s new switching platform, the Nexus Family, is squarely focused on critical infrastructure for the data center. Its value proposition is rooted in four key areas.

1. Unified Fabric and IO that enables storage, Ethernet IP, High Performance Computing (HPC) to converge into a single network infrastructure. A unified fabric will lower total data center power requirements by approximately 8%. Given that networks consume 7% of overall data center power draw, Nexus neutralizes its own power consumption impact. Power efficiency is gained by device design and the reduction of data center devices needed such as the number of PCI interfaces on servers, the number of network devices, and driving increased levels of storage consolidation.
2. A New Switching Platform. Cisco started the software design of Nexus with SAN-OS to meet base line availability requirements of storage networks and then built a platform on top of this that delivers unprecedented availability to the point that it will not drop a single packet during upgrades or when service requires lossless transmission, e.g. storage.
3. The Nexus operating system, called NX-OS supports fiber channel, Ethernet, and IP all in one product line and one operating system. NX-OS, described in detail below, provides a virtualized control plane, which essentially looks like VMware for switches. NX-OS can run multiple virtual switches concurrently on one hardware product, splitting views and responsibilities across different administrative domains. This allows a lot of flexibility such as running a lab network on top of a production network, or to model a configuration change or determine a policy change on top of a network to understand its behavior and then cut over to production when comfortable. Overlapping administration is now possible, e.g., a SAN administrator and a LAN administrator are provided their respective views and controls without interfering with each other.
4. Data Center Network Management. Cisco added a full XML API so it could provide a data center network management platform built on top of Fabric Manager that shows L2, L3, fiber channel and unified fabric topology discovery, visualization, management, and Craft interface all on one screen. From Data Center Network Management ops can see EIGRP topologies, OSPF topologies, BGP in the future, etc., on an end-to-end system visibility basis.

How Nexus Unified Fabric Works

Consider a group of servers and fiber channel arrays all connected via a group of Nexus switches. The way the unified fabric works is it reduces the number of IO interfaces coming out of servers unifying IO interfaces down to one or two; two for redundancy. For example, a server may boot from a fiber channel array located at one end of the data center while executing a block based file copy to another fiber channel array is in process; these two arrays may be doing a synchronous replication and then FTP-ing a file to another server in the data center. In short, the unified fabric supports all data center functions between servers, storage and networks. The unified fabric is gatewayless, seamless, there is no performance degradation, there is no control plane or NPU used in the data path. Nexus is implementing unified fabric in multi-chassis, multi-topology environments, eliminating spanning tree, making all data paths active. Eliminating spanning tree not only improves the survivability and availability of the network, but cross sectional bandwidth goes up by a minimum of 2X. Lastly the way that Cisco implemented the unified fabric is that intelligent fabric applications such as storage media encryption, encryption data at rest, data migration mobility, synchronized replication, etc., is not lost, as there is no performance degradation traversing between MDS (Multilayer DataCenter Switch) Nexus and back to MDS.

On top of the unified fabric is Cisco’s TrustSec. Nexus is the execution platform for TrustSec. Nexus wire rate encrypts every link with AES128 with no performance degradation. Cisco also enabled security group tags so it can flatten the address space out in the data center which decouples an addressing scheme allowing simple logical topology schemes to be put back in place.

Business Benefits

Some of the business benefits associated with Nexus are first and foremost lower power draw; second is the increased utility gained in the data center. For example, data center ops only have to wire a server once, then never again. Nexus software allows data center ops to control the personality associated with every link in the data center. Every server will be connected to all storage! Most enterprises data centers servers are 15-20% SAN attached. When Nexus wires up servers and connects via FCoE (Fiber Channel over Ethernet) to MDS fiber channel attached storage, storage, 100% of servers become SAN attached immediately. The opportunities for storage consolidation are huge.

Cisco filed over 1500 patents, invested over $1 Billion a year in R&D, wrote 6 million lines of code for NX-OS and 2 million more for DCNM (Data Center Network Manager) to deliver unified fabric and IO for the data center. Nexus will consist of a modular rack, chassis and blade switching form factors representing multiple products all of which can reside in the same data center running the same operating system: NX-OS. What follows are some of the unique characteristics of Nexus.

Data Center Class Reliability

The Nexus possesses a zero service disruption design which stems from line card insertion and removal to software upgrades. Every aspect of Nexus has been designed to reduce the impact of faults. Its supervisor engine does not have uplinks, a forwarding engine, and does not posses a switch fabric; this eliminates the potential of failure of those system components in the event of a control plane loss. Management of Nexus plays an important role in reliability, too. Data center operators can SSH into Nexus, type reload and watch the entire box boot up and never be disconnected. If there is a corrupted boot flash, data center ops can restore the system in approximately one to two minutes versus the 23 hours it currently takes by a terminal server to copy that image improving time to recover by some 99.95%.

A Lossless Fabric Architecture

Nexus has a lossless switching fabric architecture meaning that it does not drop frames on the backplane ever, being similar to storage device design. Nexus is designed to support dense 40 GbE and 100 GbE systemically throughout the architecture including the switching fabric, chassis, backplane, control plane, etc. Customers will not have to buy or replace or upgrade common equipment to support 40 and 100 gigabit Ethernet in Nexus as these protocols are developed in the standard bodies. Not only will customers not have to upgrade Nexus to support higher speed and greater density Ethernet and fiber channel, but Nexus does not need to be taken out of service to adopt these new technologies. In short, Nexus can absorb new technologies in a non-disruptive, always on, way. The Nexus platform is designed to deliver capabilities for aggregation of data center technologies. Its ability to virtualize its control and data plane while offering a staggering performance of fifteen terabits of switching capacity offers a lot of headroom for new technology integration along with flexibility to support multiple management models.

Industrial Design

Nexus’ physical layout was designed for ease of management, cable layout organization and reduced power consumption thanks to a well thought through air flow and cooling plan. Physical redundancy increases availability by not disrupting operations if a fan tray is disconnected or part of the fabric or cables are pulled. The lockable front doors open up and expose an integrated cable management design that offers the option to dress cables all to the left or all to the right or an equalized fan out. Up to 384 Cat6A cables on the 10-slot Nexus 7000 are supported.

The power supply design is fault tolerant, meaning that the Nexus can sustain grid failure; it can lose power grid A or lose a PDU (Power Distribution Unit) and it can lose three out of six power inputs and still operate. It can lose a power supply knocking out another of six inputs and the Nexus 7000 is still manageable. While Nexus has many power inputs its actual draw is less than 40% of power input allowing it to sustain grid failures, power supply failures, etc.

Nexus port densities are the highest in the industry. The initial two line cards available are a 32 port 10 GbE board and a 48 port 10/100/1000 MbE board; both cards support wire rate encryption and role base security i.e., Cisco’s TrustSec. Nexus will support 512 10 GbE in its 18 slot chassis and 256 10GbE on its 10 slot chassis. All line cards connect to all fabrics and backplanes. There are no fabric packet drops. Latency on Nexus is deterministic. The fabric design optimizes frame size for the fabric as it approaches a congested state. Cisco pushed queuing to the inputs, which allows the fabric to self optimize frame size under load allowing latency to stay deterministic even under extreme load.

The ins and outs of packets

The key aspect of Nexus is its unified fabric design, which means that it directly connects servers, storage and IO. How are different framing formats accommodated as they enter and leave Nexus? Are all frames normalizing to a common format as data enters Nexus or are packets and frames switched in native formats? First, native formatting stays native on the wire destined to Nexus; there is no propriety format on the wire. Once frames or packets are inside the system the port ASIC determines the protocol type, buffers incoming data while executing a lookup on the forwarding engines to determine egress interface before enqueuing and forwarding across the fabric. As packets enter the backplane Cisco utilizes a well-understood and stable process for making forwarding decisions. The fabric chip makes a forwarding decision on what Cisco calls a FPOE (Fabric Port of Exit) header. The FPOE header is neutral to packet format. FPOE can be put on a fiber channel frame or an Ethernet frame and the frame is going to be forwarded across the backplane. Therefore packet formats are forwarded in their native frame format.

To eliminate congestion and increase reliability, Cisco has implemented an arbitration mechanism, which queues on the ingress line card and every egress port too in Nexus so that no frames are dropped in the fabric. So before a packet is placed on the fabric Nexus knows if the egress line card has the capability of receiving it. With the egress line card available to service the packet, Nexus then writes the header, the FPOE headers, and sends it across a given switch fabric. If Nexus is delivered a jumbo frame, it chops it up into four single frames. If Nexus receives a set of 64 byte frames queuing up it will put one fabric port of exit header on the concatenated or now super frame and forward that set of 64 byte frames to the egress where it takes the header off and forwards the individual frames to its destination. Note that no packets are ever forwarded on the wire out-of-order or fragmented.

In short, the Nexus ingress line card performs the destination look up, determines the physical port it needs to exit on, and then utilizes the FPOE header to move packets across its back layers. The EARL Forwarding Engine and Fabric writes a fabric port of exit header, which instructs Nexus how to forward the packet across the switch fabric of what line card and what egress port it’s going out. The egress port then performs any serialization to the frame as it’s placed on the wire.

Future Proofing:

The Nexus internal architecture is both media and format independent supporting any kind of storage, IO and networking media. It possesses performance to support dense numbers of 40 and 100 gigabit Ethernet overtime too, especially as 10GbE will become the standard for storage interconnect. Nexus has been designed to be protocol agnostic with tremendous headroom allowing the data center market to move in a number of directions, none of which would make obsolete investments made.

Every time a switch fabric module is installed in Nexus it adds 46 gbs per switch fabric, per slot with the first generation fabric. With five slots at 46 Gbs each, Nexus delivers a full duplex 230 Gbs or 230 Gbs in and 230 Gbs out of every slot concurrently in this system. This 460 Gbs of switching capacity is multiplied by 8 pay load slots in the 10 slot chassis or 16 pay load slots in the 18 slot chassis to derive at its overall switching capacity. Note that there is note there is 115Gbps of capacity to each Supervisor Engine too.

The Nexus 10- and 18-slot chassis will accommodate its next generation switching fabric. Cisco believes that it can double performance and over a product life cycle potentially quadruple performance capability thanks in part to the signal integrity design. The line cards and IO modules insert vertically while fabric modules are horizontal making them perpendicular to each other eliminating a backplane and its awkward design which limits backward compatibility. In short, by eliminating a backplane, Nexus does not have complex cable routing issues, which drive up signal to noise ratio, preserving signal integrity at high ultra high speeds. The distance between fabric, supervisor and line cards is approximately a 1/4 inch of copper, easing the engineering to deliver 100 GbE.

Cisco is already building its second-generation of line cards, which will work with the first generation fabric supervisors, chassis and other common equipment. So customers can buy Nexus today, buy something else a year and a half or two years from now and not have to upgrade common equipment or experience an outage or disruption in service to perform the upgrade.

A Data Center Built Operating System

To deliver on the unified fabric, reliability and performance required in the data center Cisco looked toward a new OS (Operating System) for Nexus. It built upon SAN-OS and IOS. Cisco started with SAN-OS as the foundation of Nexus OS (NX-OS). It then analyzed IOS taking out the routing protocols, command interrupter, etc. It made some modifications to configuration roll back to IOS then combined this with SAN-OS to build NX-OS. Cisco’s acquisition of Procket’s assets contributed to Layer-3 software with a modular, multi-process, endian independent code-base.

Six million lines of code later and NX-OS is Cisco’s strategic OS for the data center. Over time any member of the Nexus family will run NX-OS. There will be convergence of Cisco’s other data center platforms into NX-OS over the next several years. NX-OS is multi-threaded, multi-processed with every process being virtualized. Every process is restartable; every process has separate protected memory space, and has a separate memory table manager so it can be statefully restarted with no service disruptions. Nexus can restart OSPF faster than a switch can send a hello packet!

A zero service disruption design enables Nexus to unify fabric and IO by converging front-end networks, storage networks, back-end networks and backup networks into one unified fabric. NX-OS delivers a virtualized context of management. A data center can have up to eight virtualized contexts that provide views of what looks like mini switches inside a single device. Two data center op teams could have separate command line interfaces. One ops manager can type restart BGP and it will not restart another ops manager’s BGP session or vice a versa. NX-OS possesses stateful process restart, which heals faster than networks converge (100s of ms) and a concept called graceful system operations.

Graceful system operations links all protocol layers in an effort to simplify ops. For example, if ops commands Nexus to reload at the same time critical traffic or flows are passing through it then Nexus should not reload and drop or disrupt that traffic. To avoid this scenario, Cisco decided to make sure a Nexus switch says goodbye before it leaves the network.

If a Nexus is commanded to reload, it will first signal to other Nexus’ that it’s reloading. It sends HSRP (Hot Standby Router Protocol) leaves, it freezes spanning tree state that stops sending BPDUs (Bridge Protocol Data Units), it sends LACP (Link Aggregation Control Protocol) marker frames and sets all route metrics to infinity. Every one of these signals is a standard way of signaling that a device is leaving the network. Nexus makes all these protocols work together so that the network pre-converges around the pending administrative outage. Traffic will not be disrupted during administrative outages, as it will be re-routed to assure reliability.

Cisco also looked at the way NX-OS boots to assure that it undergoes a graceful stateful boot process that’s not disruptive either. Nexus does not advertise an all 0’s route until it has its BGP route update complete. This is a nice change in the booting process as it never made sense for a router to advertise that it’s the default gateway, if it hasn’t received its routing table yet. This graceful system operation is carried out all the way down to the line card level. The operational procedure for removing a line card is to push the release pin, watch as the LED blinks, when the LED turns off, take the line card out. Nexus pre-converged the line card to its safe to remove it status without disrupting service.

From an operational point of view a data center network manager will have a single pane of glass view of the complete data center infrastructure with visibility of routing topologies, devices, reports, etc. But beyond device management, Cisco’s vision is to take data center assurance testing which it has been running and link it to the data center network manager. What Cisco is planning to do is provide data as it updates testing on new versions, or as it finds field notices or defects or combinatorial problems in switches and certain code revs and certain deployment characteristics. A customer will be able to see this data overlaid on top of their infrastructure and know what devices they should inspect and which parts of their infrastructure need additional attention.

Juniper EX-Series and Nexus Do Not Compete

The Cisco Nexus is a big deal for a couple of reasons but the single item is that Nexus will offer business and IT leaders a new approach to data center design that will reduce the number of data center devices needed, increase performance, simplify/streamline operations and materially reduce power draw ushering in a new era of green networking. This is an engineering achievement on the scale of a Boeing 777. Cisco is the only networking concern that can accomplish this scale of project. IBM, HP, EMC, Microsoft et al., have the financial resources but clearly not the talent. No networking concern could pull this off. Nexus is off the charts on vision, scope and completeness.

Juniper’s EX-Series of Ethernet switches includes fixed configuration EX 3200 and 4200 switches plus its chassis based EX 8200. They are promoted as high performance, reliable and secure. Juniper offers 1 and 10 GbE port density, a high performance platform that it says will scale to support 100 GbE. The EX is targeted at traditional enterprise configurations such as branch offices, campus and headquarters facilities.

Juniper’s EX 8200 is focused on competing with Foundry Networks BigIron RX Series, Force 10s E1200 and Cisco’s Catalyst 6500 series of switches. Juniper does not have the enterprise channel to move these switches and unfortunately there isn’t a large compelling set of features that would dislodge an incumbent. With Foundry and Extreme at a run rate of some $672M and $400 M respectively and Force 10 at approx $100M, it seems like it would be difficult for Juniper to generate $100 M in the next 18 months with this product line. Cisco’s Nexus is a next generation product while Juniper’s switches offer few advances over exiting LAN switch architectures.

Posted 11 February 2008 at 4:54 pm under Lippis Report, Unified Communications & Comm.-Enabled Business. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.