data center transition

Traditional Data Center | Cisco ACI

Traditionally, we have built our networks based on a hierarchical design. This is often referred to as the traditional data center with a three-tier design with an access layer, an aggregation layer, and a core layer. Historically, this design enabled a substantial amount of predictability because aggregation switch blocks simplified the spanning-tree topology. In addition, the need for scalability often pushed this design into modularity, which increased predictability. However, although we increased predictability, the main challenge inherent in the three-tier models is that it was difficult to scale. As the number of endpoints increases and the need to move between segments, we need to span layer 2.

The traditional data center design, often leads to poor network design and human error. You don’t want to have layer 2 segment between the data center unless you have the proper controls. Although modularization is still desired in networks today, the general trend has been to move away from this design type that evolves around spanning tree to a more flexible and scalable solution with VXLAN and other similar Layer 3 overlay technologies. In addition, the Layer 3 overlay technologies bring a lot of network agility which is vital to business success.

The word agility refers to making changes, deploying services, and supporting the business at the speed it desires. This means different things to different organizations. For example, a network team can be considered agile if it can deploy network services in a matter of weeks. In others, it could mean that business units in a company should be able to get applications to production or scale core services on demand through automation with Ansible CLI or Ansible Tower. Regardless of how you define agility, there is little disagreement with the idea that network agility is vital to business success. The problem we have is that network agility has traditionally been hard to achieve until now with the Cisco ACI. Let’s recap some of the main data center transitions to understand fully.


Traditional Data Center Transformation

Diagram: Traditional data center transformation.


Traditional Data Center:

  • Layer 2 to the Core

The traditional data center has gone through several transitions. Firstly, we had Layer 2 to the core. Then, from the access to the core, we had Layer 2 and not Layer 3. A design like this would, for example, trunk all VLANs to the core. For redundancy, you would manually prune VLANs from the different trunk links. Our challenge with this approach of having Layer 2 to the core relies on Spanning Tree Protocol. Therefore redundant links are blocked. As a result, we don’t have the full bandwidth, leading to performance degradation and simply wasting resources. Another challenge is to rely on topology changes to fix the topology. Spanning Tree Protocol does have timers to limit the convergence and can be tuned for better performance. Still, we rely on the convergence from Spanning Tree Protocol to fix the topology but Spanning Tree Protocol was never meant to be a routing protocol. Compared to other protocols operating higher up in the stack are designed to be more optimized to react to changes in the topology. But STP is not an optimized control plane protocol which is a big hinder to the traditional data center. You could relate this to how VLANs have transitioned to become a security feature. However, their purpose was originally for performance reasons.


  • Routing to Access Layer

To overcome these challenges to building stable data center networks, the Layer 3 boundary gets pushed further and further to the network’s edge. Layer 3 networks can use the advances in routing protocols that can handle failures and link redundancy much more efficiently. Alot more efficnerty that Spanning Tree Protocol that should never have been there in the first place. Then we had routing at the access. With this design, we can eliminate Spanning Tree Protocol to the core and then run Equal Cost MultiPath (ECMP) from the access to the core. We can run ECMP as we are now Layer 3 routing from the access to the core layer instead of running STP that blocks redundant links.  However, equal cost multipath (ECMP) routes offer a simple way to share the network load by distributing traffic onto other paths. ECMP is therefore typically applied only to entire flows or sets of flows. A flow in this respect may be characterized by destination address, source address, transport level ports, payload protocol.


A Key Point: Equal Cost MultiPath (ECMP)

Equal Cost MultiPath (ECMP) brings many advantages; firstly, ECMP gives us full bandwidth with equal costs links. As we are routing, we no longer have to block redundant links to prevent loops at Layer 2. However, we still have Layer 2 in the network design, and we still have Layer 2 on the access layer; therefore, parts of the network will still rely on Spanning Tree Protocol, and it converges times when there is a change in the topology. So we may have Layer 3 from the access to the core, but we still have Layer 2 connections at the edge and rely on STP to block redundant links to prevent loops. Another potential drawback is that having smaller Layer 2 domains can limit where the application can reside in the data center network. Which drives more of a need to to transition from the traditional data center design.


data center network design

Diagram: Data center network design: Equal cost multi path.


The Layer 2 domain that the applications may use could be limited to a single server rack connected to one ToR or two ToR for redundancy with a layer 2 interlink between the two ToR switches to pass the Layer 2 traffic. These designs are not optimal as you have to specify where you want your applications to be set. Therefore, putting the breaks on agility. As a result, there was another key data center transition, and this was the introduction to the overlay data center designs.


  • The Rise of Virtualization

Virtualization is creating a virtual — rather than actual — version of something, such as an operating system (OS), a server, a storage device, or network resources. Virtualization uses software that simulates hardware functionality to create a virtual system. It is creating a virtual version of something like computer hardware. It was initially developed during the mainframe era. With virtualization, the virtual machine could exist on any host. As a result, Layer 2 had to be extended to every switch. This was problematic for Larger networks as the core switch had to learn every MAC address for every flow that traversed it. To overcome this and take advantage of the convergence and stability of layer 3 networks, overlay networks became the choice for data center networking. 

VXLAN is an encapsulation protocol that provides data center connectivity using tunneling to stretch Layer 2 connections over an underlying Layer 3 network. In data centers, VXLAN is the most commonly used protocol to create overlay networks that sit on top of the physical network, enabling the use of virtual networks. The VXLAN protocol supports the virtualization of the data center network while addressing the needs of multi-tenant data centers by providing the necessary segmentation on a large scale.

Here we are encapsulating traffic into a VXLAN header and forwarding between VXLAN tunnel endpoints, known as the VTEPs. With overlay networking, we have the overlay and the underlay concept. By encapsulating the traffic into the overlay VXLAN, we now use the underlay, which in the ACI is provided by IS-IS, to provide the Layer 3 stability and redundant paths using Equal Cost Multipathing (ECMP) along with the fast convergence of routing protocols.



  • The Cisco Data Center Transition

The Cisco data center has gone through several stages when you think about it. First, we started with Spanning Tree, moved to Spanning Tree with vPCs, and then replaced the Spanning Tree with FabricPath. FabricPath is what is known as a MAC-in-MAC Encapsulation. Then we replaced Spanning Tree with VXLAN, which is a MAC-in-IP Encapsulation. Today in the data center, VXLAN is the de facto overlay protocol for data center networking. The Cisco ACI uses an enhanced version of VXLAN to implement both Layer 2 and Layer 3 forwarding with a unified control plane. Replacing SpanningTree with VXLAN, where we have a MAC-in-IP encapsulation, was a welcomed milestone for data center networking.


Introduction to the ACI

The Cisco Cisco Application Centric Infrastructure Fabric (ACI) is the Cisco SDN solution for the data center. Cisco has taken a different approach from the centralized control plane SDN approach with other vendors and has created a scalable data center solution that can be extended to multiple on-premises, public, and private cloud locations. The ACI fabric has many components that include Cisco Nexus 9000 Series switches with the APIC Controller running in the leaf/spine ACI fabric mode. These components form the building blocks of the ACI, supporting a dynamic integrated physical and virtual infrastructure.


A key point. The Cisco ACI version.

Before Cisco ACI 4.1, the Cisco ACI fabric allowed only a two-tier (spine-and-leaf switch) topology, in which each leaf switch is connected to every spine switch in the network with no interconnection between leaf switches or spine switches. Starting from Cisco ACI 4.1, the Cisco ACI fabric allows a multitier (three-tiers) fabric and two tiers of leaf switches, which provides the capability for vertical expansion of the Cisco ACI fabric. This is useful to migrate a traditional three-tier architecture of core aggregation access that has been a common design model for many enterprise networks and is still required today.


A key point. The APIC Controller.

The network is driven by the database consisting of the Cisco Application Policy Infrastructure Controller ( APIC) working in a cluster from the management perspective. The APIC is the centralized point of control and everything you want to configure you can do in the APIC. Consider the APIC to be the brains of the ACI fabric and server as the single source of truth for configuration within the fabric. The APIC controller is a policy engine and holds the defined policy, which essentially tells the other elements in the ACI fabric what to do. This database allows you to manage the network as a single entity. 

In summary, the APIC is the the infrastructure controller is the main architectural component of the Cisco ACI solution. It is the unified point of automation and management for the Cisco ACI fabric, policy enforcement, and health monitoring. The APIC is not involved in data plane forwarding.

data center layout

Diagram: Data center layout: The Cisco APIC controller.


The APIC represents the management plane which allows the system to maintain the control and data plane in the network. The APIC is not the control plane device, nor does it sit in the data traffic path. Remember that the APIC controller can crash, and you still have forwarded in the fabric. The ACI solution is not an SDN centralized control plane approach. The ACI is a distributed fabric with independent control planes on all fabric switches. 


Modular data center design: The Leaf and Spine 

Leaf-spine is a two-layer data center network topology that’s useful for data centers that experience more east-west network traffic than north-south traffic. The topology comprises leaf switches (servers and storage connect) and spine switches (to which leaf switches connect). In this two-tier Clos architecture, every lower-tier switch (leaf layer) is connected to each top-tier switch (spine layer) in a full-mesh topology. The leaf layer consists of access switches that connect to devices such as servers. The spine layer is the network’s backbone and is responsible for interconnecting all leaf switches. Every leaf switch connects to every spine switch in the fabric. The path is randomly chosen, so the traffic load is evenly distributed among the top-tier switches. Therefore, if one of the top-tier switches fails, it would only slightly degrade performance throughout the data center.

Unlike the traditional data center, the ACI operates with a Leaf and Spine architecture. Now traffic comes in through a device sent from an end host. In the ACI, this is known as a Leaf device. We also have the Spine devices that are Layer 3 routers with no special hardware dependencies.  In a basic leaf and spine fabric, every Leaf is connected to every Spine, and any endpoint in the fabric is always the same distance in terms of hops and latency from every other endpoint that is internal to the fabric. The ACI Spine switches are Clos intermediary switches with many key functions. Firstly, they exchange routing updates with leaf switches via Intermediate System-to-Intermediate System (IS-IS) and rapidly forward packets between leaf switches. They provide endpoint lookup services to leaf switches through the Council of Oracle Protocol (COOP). They also handle route reflection to the leaf switches using Multiprotocol BGP (MP-BGP).


Cisco ACI Overview

Diagram: Cisco ACI Overview.


The Leaf switches are the ingress/egress points for traffic into and out of the ACI fabric. In addition, they are the connectivity points for the variety of endpoints that the Cisco ACI supports. The leaf switches provide end-host connectivity. The spines act as a fast, non-blocking Layer 3 forwarding plane that supports Equal Cost Multipathing (ECMP) between any two endpoints in the fabric and uses overlay protocols such as VXLAN under the hood. VXLAN enables any workloads to exist anywhere in the fabric. By using VXLAN, we can now have workloads exist anywhere in the fabric without introducing too much complexity.


      • A Key Point:

This is a big improvement to data center networking as now we can have workloads, physical or virtual, in the same logical layer 2 domain, even when we are running Layer 3 down to each ToR switch. The ACI is a scalable solution as the underlay is specifically built to be scalable as more links are added to the topology. Along with being resilient when links in the fabric are brought down due to, for example, maintenance or failure. 


  • The Normalization event

XLAN is an industry-standard protocol that extends Layer 2 segments over Layer 3 infrastructure to build Layer 2 overlay logical networks. The ACI infrastructure Layer 2 domains reside in the overlay, with isolated broadcast and failure bridge domains. This approach allows the data center network to grow without the risk of creating too large a failure domain. All traffic in the ACI fabric is normalized as VXLAN packets. At the ingress, ACI encapsulates external VLAN, VXLAN, and NVGRE packets in a VXLAN packet. This is known as ACI encapsulation normalization. As a result, the forwarding in the ACI fabric is not limited to or constrained by the encapsulation type or encapsulation overlay network. If need be, the ACI bridge domain forwarding policy can be defined to provide standard VLAN behavior where required.

When traffic hits the Leaf, there is a normalization event. The normalization takes traffic sent from the servers to the ACI and makes it ACI compatible. Essentially, we are giving traffic that is sent from the servers a VXLAN ID so it can be sent across the ACI fabric. Traffic is normalized and then encapsulated with a VXLAN header and routed across the ACI fabric to the destination leaf where the destination endpoint is. This is, in a nutshell, how the ACI leaf and Spine work. We have a set of leaf switches that connect to the workloads and the spines that connect to the Leaf. VXLAN is the overlay protocol that carries data traffic across the ACI fabric.  A key point to this type of architecture is that the Layer 3 boundary is moved to the Leaf. This brings a lot of value and benefits to data center design. This boundary makes more sense as we have to route and encapsulate at this layer without going up to the core layer.