What is OpenFlow

September 25, 2014

by Matt Conran Blog

What is OpenFlow?

In today's rapidly evolving digital landscape, network management and data flow control have become critical for businesses of all sizes. OpenFlow is one technology that has gained significant attention and is transforming how networks are managed. In this blog post, we will delve into the concept of OpenFlow, its advantages, and its implications for network control.

OpenFlow is an open-standard communications protocol that separates the control and data planes in a network architecture. It allows network administrators to have direct control over the behavior of network devices, such as switches and routers, by utilizing a centralized controller.

Traditional network architectures follow a closed model, where network devices make independent decisions on forwarding packets. On the other hand, OpenFlow introduces a centralized control plane that provides a global view of the network and allows administrators to define network policies and rules from a centralized location.

OpenFlow operates by establishing a secure channel between the centralized controller and the network switches. The controller is responsible for managing the flow tables within the switches, defining how traffic should be forwarded based on predefined rules and policies. This separation of control and data planes allows for dynamic network management and facilitates the implementation of innovative network protocols.

One of the key advantages of OpenFlow is its ability to simplify network management. By centralizing control, administrators can easily configure and manage the entire network from a single point of control. This reduces complexity and enhances the scalability of network infrastructure. Additionally, OpenFlow enables network programmability, allowing for the development of custom networking applications and services tailored to specific requirements.

OpenFlow plays a crucial role in network virtualization, as it allows for the creation and management of virtual networks on top of physical infrastructure. By abstracting the underlying network, OpenFlow empowers organizations to optimize resource utilization, improve security, and enhance network performance. It opens doors to dynamic provisioning, isolation, and efficient utilization of network resources.

Matt Conran

Highlights: What is OpenFlow?

How does OpenFlow work?

1: OpenFlow allows network controllers to determine the path of network packets in a network of switches. There is a difference between switches and controllers. With separate control and forwarding, traffic management can be more sophisticated than access control lists (ACLs) and routing protocols.

2: An OpenFlow protocol allows switches from different vendors, often with proprietary interfaces and scripting languages, to be managed remotely. Software-defined networking (SDN) is considered to be enabled by OpenFlow by its inventors.

3: With OpenFlow, Layer 3 switches can add, modify, and remove packet-matching rules and actions remotely. By doing so, routing decisions can be made periodically or ad hoc by the controller and translated into rules and actions with a configurable lifespan, which are then deployed to the switch’s flow table, where packets are forwarded at wire speed for the duration of the rule.

The Role of OpenFlow Controllers

If the switch cannot match packets, they can be sent to the controller. The controller can modify existing flow table rules or deploy new rules to prevent a structural traffic flow. It may even forward the traffic itself if the switch is instructed to forward packets rather than just their headers.

OpenFlow uses Transport Layer Security (TLS) over Transmission Control Protocol (TCP). Switches wishing to connect should listen on TCP port 6653. In earlier versions of OpenFlow, port 6633 was unofficially used. The protocol is mainly used between switches and controllers.

### Origins of OpenFlow

The inception of OpenFlow can be traced back to the early 2000s when researchers at Stanford University sought to create a more versatile and programmable network architecture. Traditional networking relied heavily on static and proprietary hardware configurations, which limited innovation and adaptability. OpenFlow emerged as a solution to these challenges, offering a standardized protocol that decouples the network control plane from the data plane. This separation allows for centralized control and dynamic adjustment of network traffic, fostering innovation and agility in network design.

### How OpenFlow Works

At its core, the OpenFlow protocol facilitates communication between network devices and a centralized SDN controller. It does this by using a series of flow tables within network switches, which are programmed by the controller. These flow tables dictate how packets should be handled, whether they are forwarded to a destination, dropped, or modified in some way. By leveraging OpenFlow, network administrators can deploy updates, optimize performance, and troubleshoot issues with unprecedented speed and precision, all from a single point of control.

Introducing SDN

Recent changes and requirements have driven networks and network services to become more flexible, virtualization-aware, and API-driven. One major trend affecting the future of networking is software-defined networking ( SDN ). The software-defined architecture aims to extract the entire network into a single switch.

Software-defined networking (SDN) is an evolving technology defined by the Open Networking Foundation ( ONF ). It involves the physical separation of the network control plane from the forwarding plane, where a control plane controls several devices. This differs significantly from traditional IP forwarding that you may have used in the past.

The Core Concepts of SDN

At its heart, Software-Defined Networking decouples the network control plane from the data plane, allowing network administrators to manage network services through abstraction of lower-level functionality. This separation enables centralized network control, which simplifies the management of complex networks. The control plane makes decisions about where traffic is sent, while the data plane forwards traffic to the selected destination. This approach allows for a more flexible, adaptable network infrastructure.

The activities around OpenFlow

Even though OpenFlow has received a lot of industry attention, programmable networks and decoupled control planes (control logic) from data planes have been around for many years. To enhance ATM, Internet, and mobile networks’ openness, extensibility, and programmability, the Open Signaling (OPENING) working group held workshops in 1995. A working group within the Internet Engineering Task Force (IETF) developed GSMP to control label switches based on these ideas. June 2002 marked the official end of this group, and GSMPv3 was published.

Data and control plane

Therefore, SDN separates the data and control plane. The main driving body behind software-defined networking (SDN) is the Open Networking Foundation ( ONF ). Introduced in 2008, the ONF is a non-profit organization that wants to provide an alternative to proprietary solutions that limit flexibility and create vendor lock-in.

The insertion of the ONF allowed its members to run proof of concepts on heterogeneous networking devices without requiring vendors to expose their software’s internal code. This creates a path for an open-source approach to networking and policy-based controllers.

Knowledge Check: Data & Control Plane

### Data Plane: The Highway for Your Data

The data plane, often referred to as the forwarding plane, is responsible for the actual movement of packets of data from source to destination. Imagine it as a network’s highway, where data travels at high speed. This component operates at the speed of light, handling massive amounts of data with minimal delay. It’s designed to process packets quickly, ensuring that the information arrives where it needs to be without interruption.

### Control Plane: The Brain Behind the Operation

While the data plane acts as the highway, the control plane is the brain that orchestrates the flow of traffic. It makes decisions about routing, managing the network topology, and controlling the data plane’s operations. The control plane uses protocols to determine the best paths for data and to update routing tables as needed. It’s responsible for ensuring that the network operates efficiently, adapting to changes, and maintaining optimal performance.

### Interplay Between Data and Control Planes

The synergy between the data and control planes is what enables modern networks to function effectively. The control plane provides the intelligence and decision-making necessary to guide the data plane. This interaction ensures that data packets take the best possible paths, reducing latency and maximizing throughput. As networks evolve, the lines between these planes may blur, but their distinct roles remain pivotal.

### Real-World Applications and Innovations

The concepts of data and control planes are not just theoretical—they have practical applications in technologies such as Software-Defined Networking (SDN) and Network Function Virtualization (NFV). These innovations allow for greater flexibility and scalability in managing network resources, offering businesses the agility needed to adapt to changing demands.

Building blocks: SDN Environment

As a fundamental building block of an SDN deployment, the controller, the SDN switch (for example, an OpenFlow switch), and the interfaces are present in the controller to communicate with forwarding devices, generally the southbound interface (OpenFlow) and the northbound interface (the network application interface).

In an SDN, switches function as basic forwarding hardware, accessible via an open interface, with the control logic and algorithms offloaded to controllers. Hybrid (OpenFlow-enabled) and pure (OpenFlow-only) OpenFlow switches are available.

OpenFlow switches rely entirely on a controller for forwarding decisions, without legacy features or onboard control. Hybrid switches support OpenFlow as well, in addition to traditional operation and protocols. Today, hybrid switches are the most common type of commercial switch. A flow table performs packet lookup and forwarding in an OpenFlow switch.

### The Role of the OpenFlow Controller

The OpenFlow controller is the brain of the SDN, orchestrating the flow of data across the network. It communicates with the network devices using the OpenFlow protocol to dictate how packets should be handled. The controller’s primary function is to make decisions on the path data packets should take, ensuring optimal network performance and resource utilization. This centralization of control allows for dynamic network configuration, paving the way for innovative applications and services.

### OpenFlow Switches: The Workhorses of the Network

While the controller is the brain, OpenFlow switches are the workhorses, executing the instructions they receive. These switches operate at the data plane, where they forward packets based on the rules set by the controller. Each switch maintains a flow table that matches incoming packets to particular actions, such as forwarding or dropping the packet. This separation of control and data planes is what sets SDN apart from traditional networking, offering unparalleled flexibility and control.

### Flow Tables: The Heart of OpenFlow Switches

Flow tables are the core component of OpenFlow switches, dictating how packets are handled. Each entry in a flow table consists of match fields, counters, and a set of instructions. Match fields identify the packets that should be affected by the rule, counters track the number of packets and bytes that match the entry, and instructions define the actions to be taken. This modular approach allows for precise traffic management and is essential for implementing advanced network policies.

OpenFlow Table & Routing Tables

### What is an OpenFlow Flow Table?

OpenFlow is a protocol that allows network controllers to interact with the forwarding plane of network devices like switches and routers. At the heart of OpenFlow is the flow table. This table contains a set of flow entries, each specifying actions to take on packets that match a particular pattern. Unlike traditional routing tables, which rely on predefined paths and protocols, OpenFlow flow tables provide flexibility and programmability, allowing dynamic changes in how packets are handled based on real-time network conditions.

### Understanding Routing Tables

Routing tables are a staple of traditional networking, used by routers to determine the best path for forwarding packets to their final destinations. These tables consist of a list of network destinations, with associated metrics that help in selecting the most efficient route. Routing protocols such as OSPF, BGP, and RIP are employed to maintain and update these tables, ensuring that data flows smoothly across the interconnected web of networks. While reliable, routing tables are less flexible compared to OpenFlow flow tables, as changes in network traffic patterns require updates to routing protocols and configurations.

Example of a Routing Table

### Key Differences Between OpenFlow Flow Table and Routing Table

The primary distinction between OpenFlow flow tables and routing tables lies in their approach to network management. OpenFlow flow tables are dynamic and programmable, allowing for real-time adjustments and fine-grained control over traffic flows. This makes them ideal for environments where network agility and customization are paramount. Conversely, routing tables offer a more static and predictable method of packet forwarding, which can be beneficial in stable networks where consistency and reliability are prioritized.

### Use Cases: When to Use Each

OpenFlow flow tables are particularly advantageous in software-defined networking (SDN) environments, where network administrators need to quickly adapt to changing conditions and optimize traffic flows. They are well-suited for data centers, virtualized networks, and scenarios requiring high levels of automation and scalability. On the other hand, traditional routing tables are best used in established networks with predictable traffic patterns, such as those found in enterprise or service provider settings, where reliability and stability are key.

You may find the following useful for pre-information:

What is OpenFlow?

OpenFlow was the first protocol of the Software-Defined Networking (SDN) trend and is the only protocol that allows the decoupling of a network device’s control plane from the data plane. In most straightforward terms, the control plane can be thought of as the brains of a network device. On the other hand, the data plane can be considered hardware or application-specific integrated circuits (ASICs) that perform packet forwarding.

Numerous devices also support running OpenFlow in a hybrid mode, meaning OpenFlow can be deployed on a given port, virtual local area network (VLAN), or even within a regular packet-forwarding pipeline such that if there is not a match in the OpenFlow table, then the existing forwarding tables (MAC, Routing, etc.) are used, making it more analogous to Policy Based Routing (PBR).

What is SDN?

Despite various modifications to the underlying architecture and devices (such as switches, routers, and firewalls), traditional network technologies have existed since the inception of networking. Using a similar approach, frames and packets have been forwarded and routed in a limited manner, resulting in low efficiency and high maintenance costs. Consequently, the architecture and operation of networks need to evolve, resulting in SDN.

By enabling network programmability, SDN promises to simplify network control and management and allow innovation in computer networking. Network engineers configure policies to respond to various network events and application scenarios. They can achieve the desired results by manually converting high-level policies into low-level configuration commands.

Often, minimal tools are available to accomplish these very complex tasks. Controlling network performance and tuning network management are challenging and error-prone tasks.

A modern network architecture consists of a control plane, a data plane, and a management plane; the control and data planes are merged into a machine called Inside the Box. To overcome these limitations, programmable networks have emerged.

How OpenFlow Works:

At the core of OpenFlow is the concept of a flow table, which resides in each OpenFlow-enabled switch. The flow table contains match-action rules defining how incoming packets should be processed and forwarded. The centralized controller determines these rules and communicates using the OpenFlow protocol with the switches.

When a packet arrives at an OpenFlow-enabled switch, it is first matched against the rules in the flow table. If a match is found, the corresponding action is executed, including forwarding the packet, dropping it, or sending it to the controller for further processing. This decoupling of the control and data planes allows for flexible and programmable network management.

What is OpenFlow SDN?

The main goal of SDN is to separate the control and data planes and transfer network intelligence and state to the control plane. These concepts have been exploited by technologies like Routing Control Platform (RCP), Secure Architecture for Network Enterprise (SANE), and, more recently, Ethane.

In addition, there is often a connection between SDN and OpenFlow. The Open Networking Foundation (ONF) is responsible for advancing SDN and standardizing OpenFlow, whose latest version is 1.5.0.

An SDN deployment starts with these building blocks.

For communication with forwarding devices, the controller has the SDN switch (for example, an OpenFlow switch), the SDN controller, and the interfaces. An SDN deployment is based on two basic building blocks: a southbound interface (OpenFlow) and a northbound interface (the network application interface).

As the control logic and algorithms are offloaded to a controller, switches in SDNs may be represented as basic forwarding hardware. Switches that support OpenFlow come in two varieties: pure (OpenFlow-only) and hybrid (OpenFlow-enabled).

Pure OpenFlow switches do not have legacy features or onboard control for forwarding decisions. A hybrid switch can operate with both traditional protocols and OpenFlow. Hybrid switches make up the majority of commercial switches available today. In an OpenFlow switch, a flow table performs packet lookup and forwarding.

OpenFlow reference switch

The OpenFlow protocol and interface allow OpenFlow switches to be accessed as essential forwarding elements. A flow-based SDN architecture like OpenFlow simplifies switching hardware. Still, it may require additional forwarding tables, buffer space, and statistical counters that are difficult to implement in traditional switches with integrated circuits tailored to specific applications.

There are two types of switches in an OpenFlow network: hybrids (which enable OpenFlow) and pores (which only support OpenFlow). OpenFlow is supported by hybrid switches and traditional protocols (L2/L3). OpenFlow switches rely entirely on a controller for forwarding decisions and do not have legacy features or onboard control.

Hybrid switches are the majority of the switches currently available on the market. This link must remain active and secure because OpenFlow switches are controlled over an open interface (through a TCP-based TLS session). OpenFlow is a messaging protocol that defines communication between OpenFlow switches and controllers, which can be viewed as an implementation of SDN-based controller-switch interactions.

Openflow switch — Diagram: OpenFlow switch. The source is cable solution.

Identify the Benefits of OpenFlow

Application-driven routing. Users can control the network paths.	The networks paths.A way to enhance link utilization.
An open solution for VM mobility. No VLAN reliability.	A means to traffic engineer without MPLS.
A solution to build very large Layer 2 networks.	A way to scale Firewalls and Load Balancers.
A way to configure an entire network as a whole as opposed to individual entities.	A way to build your own encryption solution. Off-the-box encryption.
A way to distribute policies from a central controller.	Customized flow forwarding. Based on a variety of bit patterns.
A solution to get a global view of the network and its state. End-to-end visibility.	A solution to use commodity switches in the network. Massive cost savings.

The following table lists the Software Networking ( SDN ) benefits and the problems encountered with existing control plane architecture:

Identify the benefits of OpenFlow and SDN	Problems with the existing approach
Faster software deployment.	Large scale provisioning and orchestration.
Programmable network elements.	Limited traffic engineering ( MPLS TE is cumbersome )
Faster provisioning.	Synchronized distribution policies.
Centralized intelligence with centralized controllers.	Routing of large elephant flows.
Decisions are based on end-to-end visibility.	Qos and load based forwarding models.
Granular control of flows.	Ability to scale with VLANs.
Decreases the dependence on network appliances like load balancers.

A key point: The lack of a session layer in the TCP/IP stack

Regardless of the hype and benefits of SDN, neither OpenFlow nor other SDN technologies address the real problems of the lack of a session layer in the TCP/IP protocol stack. The problem is that the client’s application ( Layer 7 ) connects to the server’s IP address ( Layer 3 ), and if you want to have persistent sessions, the server’s IP address must remain reachable.

This session’s persistence and the ability to connect to multiple Layer 3 addresses to reach the same device is the job of the OSI session layer. The session layer provides the services for opening, closing, and managing a session between end-user applications. In addition, it allows information from different sources to be correctly combined and synchronized.

The problem is the TCP/IP reference module does not consider a session layer, and there is none in the TCP/IP protocol stack. SDN does not solve this; it gives you different tools to implement today’s kludges.

Control and data plane

When we identify the benefits of OpenFlow, let us first examine traditional networking operations. Traditional networking devices have a control and forwarding plane, depicted in the diagram below. The control plane is responsible for setting up the necessary protocols and controls so the data plane can forward packets, resulting in end-to-end connectivity. These roles are shared on a single device, and the fast packet forwarding ( data path ) and the high-level routing decisions ( control path ) occur on the same device.

What is OpenFlow | SDN separates the data and control plane?

Control plane

The control plane is part of the router architecture and is responsible for drawing the network map in routing. When we mention control planes, you usually think about routing protocols, such as OSPF or BGP. But in reality, the control plane protocols perform numerous other functions, including:

Connectivity management ( BFD, CFM )	Interface state management ( PPP, LACP )
Service provisioning ( RSVP for InServ or MPLS TE)	Topology and reachability information exchange ( IP routing protocols, IS-IS in TRILL/SPB )
Adjacent device discovery via HELLO mechanism	ICMP

Control plane protocols run over data plane interfaces to ensure “shared fate” – if the packet forwarding fails, the control plane protocol fails as well.

Most control plane protocols ( BGP, OSPF, BFD ) are not data-driven. A BGP or BFD packet is never sent as a direct response to a data packet. There is a question mark over the validity of ICMP as a control plane protocol. The debate is whether it should be classed in the control or data plane category.

Some ICMP packets are sent as replies to other ICMP packets, and others are triggered by data plane packets, i.e., data-driven. My view is that ICMP is a control plane protocol that is triggered by data plane activity. After all, the “C” is ICMP does stand for “Control.”

Data plane

The data path is part of the routing architecture that decides what to do when a packet is received on its inbound interface. It is primarily focused on forwarding packets but also includes the following functions:

ACL logging	Netflow accounting
NAT session creation	NAT table maintenance

The data forwarding is usually performed in dedicated hardware, while the additional functions ( ACL logging, Netflow accounting ) typically happen on the device CPU, commonly known as “punting.” The data plane for an OpenFlow-enabled network can take a few forms.

However, the most common, even in the commercial offering, is the Open vSwitch, often called the OVS. The Open vSwitch is an open-source implementation of a distributed virtual multilayer switch. It enables a switching stack for virtualization environments while supporting multiple protocols and standards.

Software-defined networking changes the control and data plane architecture.

The concept of SDN separates these two planes, i.e., the control and forwarding planes are decoupled. This allows the networking devices in the forwarding path to focus solely on packet forwarding. An out-of-band network uses a separate controller ( orchestration system ) to set up the policies and controls. Hence, the forwarding plane has the correct information to forward packets efficiently.

In addition, it allows the network control plane to be moved to a centralized controller on a server instead of residing on the same box carrying out the forwarding. Moving the intelligence ( control plane ) of the data plane network devices to a controller enables companies to use low-cost, commodity hardware in the forwarding path. A significant benefit is that SDN separates the data and control plane, enabling new use cases.

A centralized computation and management plane makes more sense than a centralized control plane.

The controller maintains a view of the entire network and communicates with Openflow ( or, in some cases, BGP with BGP SDN ) with the different types of OpenFlow-enabled network boxes. The data path portion remains on the switch, such as the OVS bridge, while the high-level decisions are moved to a separate controller. The data path presents a clean flow table abstraction, and each flow table entry contains a set of packet fields to match, resulting in specific actions ( drop, redirect, send-out-port ).

When an OpenFlow switch receives a packet it has never seen before and doesn’t have a matching flow entry, it sends the packet to the controller for processing. The controller then decides what to do with the packet.

Applications could then be developed on top of this controller, performing security scrubbing, load balancing, traffic engineering, or customized packet forwarding. The centralized view of the network simplifies problems that were harder to overcome with traditional control plane protocols.

A single controller could potentially manage all OpenFlow-enabled switches. Instead of individually configuring each switch, the controller can push down policies to multiple switches simultaneously—a compelling example of many-to-one virtualization.

Now that SDN separates the data and control plane, the operator uses the centralized controller to choose the correct forwarding information per-flow basis. This allows better load balancing and traffic separation on the data plane. In addition, there is no need to enforce traffic separation based on VLANs, as the controller would have a set of policies and rules that would only allow traffic from one “VLAN” to be forwarded to other devices within that same “VLAN.”

The advent of VXLAN

With the advent of VXLAN, which allows up to 16 million logical entities, the benefits of SDN should not be purely associated with overcoming VLAN scaling issues. VXLAN already does an excellent job with this. It does make sense to deploy a centralized control plane in smaller independent islands; in my view, it should be at the edge of the network for security and policy enforcement roles. Using Openflow on one or more remote devices is easy to implement and scale.

It also decreases the impact of controller failure. If a controller fails and its sole job is implementing packet filters when a new user connects to the network, the only affecting element is that the new user cannot connect. If the controller is responsible for core changes, you may have interesting results with a failure. New users not being able to connect is bad, but losing your entire fabric is not as bad.

Spanning tree VXLAN — Diagram: Loop prevention. Source is Cisco

A traditional networking device runs all the control and data plane functions. The control plane, usually implemented in the central CPU or the supervisor module, downloads the forwarding instructions into the data plane structures. Every vendor needs communications protocols to bind the two planes together to download forward instructions.

Therefore, all distributed architects need a protocol between control and data plane elements. The protocol binding this communication path for traditional vendor devices is not open-source, and every vendor uses its proprietary protocol (Cisco uses IPC—InterProcess Communication ).

Openflow tries to define a standard protocol between the control plane and the associated data plane. When you think of Openflow, you should relate it to the communication protocol between the traditional supervisors and the line cards. OpenFlow is just a low-level tool.

OpenFlow is a control plane ( controller ) to data plane ( OpenFlow enabled device ) protocol that allows the control plane to modify forwarding entries in the data plane. It enables SDN to separate the data and control planes.

Proactive versus reactive flow setup

OpenFlow operations have two types of flow setups: Proactive and Reactive.

With Proactive, the controller can populate the flow tables ahead of time, similar to a typical routing. However, the packet-in event never occurs by pre-defining your flows and actions ahead of time in the switch’s flow tables. The result is all packets are forwarded at line rate. With Reactive, the network devices react to traffic, consult the OpenFlow controller, and create a rule in the flow table based on the instruction. The problem with this approach is that there can be many CPU hits.

The following table outlines the critical points for each type of flow setup:

Proactive flow setup	Reactive flow setup
Works well when the controller is emulating BGP or OSPF.	Used when no one can predict when and where a new MAC address will appear.
The controller must first discover the entire topology.	Punts unknown packets to the controller. Many CPU hits.
Discover endpoints ( MAC addresses, IP addresses, and IP subnets )	Compute forwarding paths on demand. Not off the box computation.
Compute off the box optimal forwarding.	Install flow entries based on actual traffic.
Download flow entries to the data plane switches.	Has many scalability concerns such as packet punting rate.
No data plane controller involvement with the exceptions of ARP and MAC learning. Line-rate performance.	Not a recommended setup.

Hop-by-hop versus path-based forwarding

The following table illustrates the key points for the two types of forwarding methods used by OpenFlow: hop-by-hop forwarding and path-based forwarding:

Hop-by-hop Forwarding	Path-based Forwarding
Similar to traditional IP Forwarding.	Similar to MPLS.
Installs identical flows on each switch on the data path.	Map flows to paths on ingress switches and assigns user traffic to paths at the edge node
Scalability concerns relating to flow updates after a change in topology.	Compute paths across the network and installs end-to-end path-forwarding entries.
Significant overhead in large-scale networks.	Works better than hop-by-hop forwarding in large-scale networks.
FIB update challenges. Convergence time.	Core switches don’t have to support the same granular functionality as edge switches.

Obviously, with any controller, the controller is a lucrative target for attack. Anyone who knows you are using a controller-based network will try to attack the controller and its control plane. The attacker may attempt to intercept the controller-to-switch communication and replace it with its commands, essentially attacking the control plane with whatever means they like.

An attacker may also try to insert a malformed packet or some other type of unknown packet into the controller ( fuzzing attack ), exploiting bugs in the controller and causing the controller to crash.

Fuzzing attacks can be carried out with application scanning software such as Burp Suite. It attempts to manipulate data in a particular way, breaking the application.

The best way to tighten security is to encrypt switch-to-controller communications with SSL and self-signed certificates to authenticate the switch and controller. It would also be best to minimize interaction with the data plane, except for ARP and MAC learning.

To prevent denial-of-service attacks on the controller, you can use Control Plane Policing ( CoPP ) on Ingress to avoid overloading the switch and the controller. Currently, NEC is the only vendor implementing CoPP.

The Hybrid deployment model is helpful from a security perspective. For example, you can group specific ports or VLANs to OpenFlow and other ports or VLANs to traditional forwarding, then use traditional forwarding to communicate with the OpenFlow controller.

Software-defined networking or traditional routing protocols?

The move to a Software-Defined Networking architecture has clear advantages. It’s agile and can react quickly to business needs, such as new product development. For businesses to succeed, they must have software that continues evolving.

Otherwise, your customers and staff may lose interest in your product and service. The following table displays the advantages and disadvantages of the existing routing protocol control architecture.

+Reliable and well known.	-Non-standard Forwarding models. Destination-only and not load-aware metrics**
+Proven with 20 plus years field experience.	-Loosely coupled.
+Deterministic and predictable.	-Lacks end-to-end transactional consistency and visibility.
+Self-Healing. Traffic can reroute around a failed node or link.	-Limited Topology discovery and extraction. Basic neighbor and topology tables.
+Autonomous.	-Lacks the ability to change existing control plane protocol behavior.
+Scalable.	-Lacks the ability to introduce new control plane protocols.
+Plenty of learning and reading materials.

** Basic EIGRP IETF originally proposed an Energy-Aware Control Plane, but the IETF later removed this.

Software-Defined Networking: Use Cases

Edge Security policy enforcement at the network edge.	Authenticate users or VMs and deploy per-user ACL before connecting a user to the network.
Custom routing and online TE.	The ability to route on a variety of business metrics aka routing for dollars. Allowing you to override the default routing behavior.
Custom traffic processing.	For analytics and encryption.
Programmable SPAN ports	Use Openflow entries to mirror selected traffic to the SPAN port.
DoS traffic blackholing & distributed DoS prevention.	Block DoS traffic as close to the source as possible with more selective traffic targeting than the original RTBH approach**. The traffic blocking is implemented in OpenFlow switches. Higher performance with significantly lower costs.
Traffic redirection and service insertion.	Redirect a subset of traffic to network appliances and install redirection flow entries wherever needed.
Network Monitoring.	The controller is the authoritative source of information on network topology and Forwarding paths.
Scale-Out Load Balancing.	Punt new flows to the Openflow controller and install per-session entries throughout the network.
IPS Scale-Out.	OpenFlow is used to distribute the load to multiple IDS appliances.

**Remote-Triggered Black Hole: RTBH refers to installing a host route to a bogus IP address ( RTBH address ) pointing to NULL interfaces on all routers. BGP is used to advertise the host routes to other BGP peers of the attacked hosts, with the next hop pointing to the RTBH address, and it is mainly automated in ISP environments.

SDN deployment models

Guidelines:

Start with small deployments away from the mission-critical production path, i.e., the Core. Ideally, start with device or service provisioning systems.
Start at the Edge and slowly integrate with the Core. Minimize the risk and blast radius. Start with packet filters at the Edge and tasks that can be easily automated ( VLANs ).
Integrate new technology with the existing network.
Gradually increase scale and gain trust. Experience is key.
Have the controller in a protected out-of-band network with SSL connectivity to the switches.

There are 4 different models for OpenFlow deployment, and the following sections list the key points of each model.

Native OpenFlow

They are commonly used for Greenfield deployments.
The controller performs all the intelligent functions.
The forwarding plane switches have little intelligence and solely perform packet forwarding.
The white box switches need IP connectivity to the controller for the OpenFlow control sessions. If you are forced to use an in-band network for this communication path using an isolated VLAN with STP, you should use an out-of-band network.
Fast convergence techniques such as BFD may be challenging to use with a central controller.
Many people believe that this approach does not work for a regular company. Companies implementing native OpenFlow, such as Google, have the time and resources to reinvent the wheel when implementing a new control-plane protocol ( OpenFlow ).

Native OpenFlow with Extensions

Some control plane functions are handled from the centralized controller to the forwarding plane switches. For example, the OpenFlow-enabled switches could load balancing across multiple links without the controller’s previous decision. You could also run STP, LACP, or ARP locally on the switch without interaction with the controller. This approach is helpful if you lose connectivity to the controller. If the low-level switches perform certain controller functions, packet forwarding will continue in the event of failure.
The local switches should support the specific OpenFlow extensions that let them perform functions on the controller’s behalf.

Hybrid ( Ships in the night )

This approach is used where OpenFlow runs in parallel with the production network.
The same network box is controlled by existing on-box and off-box control planes ( OpenFlow).
Suitable for pilot deployment models as switches still run traditional control plane protocols.
The Openflow controller manages only specific VLANs or ports on the network.
The big challenge is determining and investigating the conflict-free sharing of forwarding plane resources across multiple control planes.

Integrated OpenFlow

OpenFlow classifiers and forwarding entries are integrated with the existing control plane. For example, Juniper’s OpenFlow model follows this mode of operation where OpenFlow static routes can be redistributed into the other routing protocols.
No need for a new control plane.
No need to replace all forwarding hardware
It is the most practical approach as long as the vendor supports it.

Closing Points on OpenFlow

OpenFlow is a communication protocol that provides access to the forwarding plane of a network switch or router over the network. It was initially developed at Stanford University and has since been embraced by the Open Networking Foundation (ONF) as a core component of SDN. OpenFlow allows network administrators to program the control plane, enabling them to direct how packets are forwarded through the network. This decoupling of the control and data planes is what empowers SDN to offer more dynamic and flexible network management.

The architecture of OpenFlow is quite straightforward yet powerful. It consists of three main components: the controller, the switch, and the protocol itself. The controller is the brains of the operation, managing network traffic by sending instructions to OpenFlow-enabled switches. These switches, in turn, execute the instructions received, altering the flow of network data accordingly. This setup allows for centralized network control, offering unprecedented levels of automation and agility.

**Advantages of OpenFlow**

OpenFlow brings several critical advantages to network management and control:

1. Flexibility and Programmability: With OpenFlow, network administrators can dynamically reconfigure the behavior of network devices, allowing for greater adaptability to changing network requirements.

2. Centralized Control: By centralizing control in a single controller, network administrators gain a holistic view of the network, simplifying management and troubleshooting processes.

3. Innovation and Experimentation: OpenFlow enables researchers and developers to experiment with new network protocols and applications, fostering innovation in the networking industry.

4. Scalability: OpenFlow’s centralized control architecture provides the scalability needed to manage large-scale networks efficiently.

**Implications for Network Control**

OpenFlow has significant implications for network control, paving the way for new possibilities in network management:

1. Software-Defined Networking (SDN): OpenFlow is a critical component of the broader concept of SDN, which aims to decouple network control from the underlying hardware, providing a more flexible and programmable infrastructure.

2. Network Virtualization: OpenFlow facilitates network virtualization, allowing multiple virtual networks to coexist on a single physical infrastructure.

3. Traffic Engineering: By controlling the flow of packets at a granular level, OpenFlow enables advanced traffic engineering techniques, optimizing network performance and resource utilization.

OpenFlow represents a paradigm shift in network control, offering a more flexible, scalable, and programmable approach to managing networks. By separating the control and data planes, OpenFlow empowers network administrators to have fine-grained control over network behavior, improving efficiency, innovation, and adaptability. As the networking industry continues to evolve, OpenFlow and its related technologies will undoubtedly play a crucial role in shaping the future of network management.

Summary: What is OpenFlow?

In the rapidly evolving world of networking, OpenFlow has emerged as a game-changer. This revolutionary technology has transformed the way networks are managed, offering unprecedented flexibility, control, and efficiency. In this blog post, we will delve into the depths of OpenFlow, exploring its definition, key features, and benefits.

What is OpenFlow?

OpenFlow can be best described as an open standard communications protocol that enables the separation of the control plane and the data plane in network devices. It allows centralized control over a network’s forwarding elements, making it possible to program and manage network traffic dynamically. By decoupling the intelligence of the network from the underlying hardware, OpenFlow provides a flexible and programmable infrastructure for network administrators.

Key Features of OpenFlow

a) Centralized Control: One of the core features of OpenFlow is its ability to centralize network control, allowing administrators to define and implement policies from a single point of control. This centralized control improves network visibility and simplifies management tasks.

b) Programmability: OpenFlow’s programmability empowers network administrators to define how network traffic should be handled based on their specific requirements. Through the use of flow tables and match-action rules, administrators can dynamically control the behavior of network switches and routers.

c) Software-Defined Networking (SDN) Integration: OpenFlow plays a crucial role in the broader concept of Software-Defined Networking. It provides a standardized interface for SDN controllers to communicate with network devices, enabling dynamic and automated network provisioning.

Benefits of OpenFlow

a) Enhanced Network Flexibility: With OpenFlow, network administrators can easily adapt and customize their networks to suit evolving business needs. The ability to modify network behavior on the fly allows for efficient resource allocation and improved network performance.

b) Simplified Network Management: By centralizing network control, OpenFlow simplifies the management of complex network architectures. Policies and configurations can be applied uniformly across the network, reducing administrative overhead and minimizing the chances of configuration errors.

c) Innovation and Experimentation: OpenFlow fosters innovation by providing a platform for the development and deployment of new network protocols and applications. Researchers and developers can experiment with novel networking concepts, paving the way for future advancements in the field.

Conclusion

OpenFlow has ushered in a new era of network management, offering unparalleled flexibility and control. Its ability to separate the control plane from the data plane, coupled with centralized control and programmability, has opened up endless possibilities in network architecture design. As organizations strive for more agile and efficient networks, embracing OpenFlow and its associated technologies will undoubtedly be a wise choice.

Advances of IP routing and Cloud

September 16, 2014

by Matt Conran Blog

With the introduction and hype around Software Defined Networking ( SDN ) and Cloud Computing, one would assume that there has been little or no work with the advances in IP routing. You could say that the cloud has clouded the mind of the market. Regardless of the hype around this transformation, routing is still very much alive and makes up a vital part of the main internet statistics you can read. Packets still need to get to their destinations.

Advanced in IP Routing

The Internet Engineering Task Force (IETF) develops and promotes voluntary internet standards, particularly those that comprise the Internet Protocol Suite (TCP/IP). The IETF shapes what comes next, and this is where all the routing takes place. It focuses on anything between the physical layer and the application layer. It doesn’t focus on the application itself, but on the technologies used to transport it, for example, HTTP.

In the IETF, no one is in charge, anyone can contribute, and everyone can benefit. As you can see from the chart below, that routing ( RTG ) has over 250 active drafts and is the most popular working group within the IETF.

IP routinng — Diagram: IETF Work Distribution.

The routing area is responsible for ensuring the continuous operation of the Internet routing system by maintaining the scalability and stability characteristics of the existing routing protocols and developing new protocols, extensions, and bug fixes promptly

The following table illustrates the subgroups of the RTG working group:

Bidirectional Forwarding Detection (BFD)	Open Shortest Path First IGP (OSPF)
Forwarding and Control Element Separation (forces)	Routing Over Low power and Lossy networks (roll)
Interface to the Routing System (i2rs)	Routing Area Working Group (RTGW)
Inter-Domain Routing (IDR)	Secure Inter-Domain Routing (SCIDR)
IS-IS for IP Internets (isis)	Source Packet Routing in Networking (spring)
Keying and Authentication for Routing Protocols (Karp)
Mobile Ad-hoc Networks (manet)

The chart below displays the number of drafts per subgroup of the routing area. There has been a big increase in the subgroup “roll,” which is second to BGP. “Roll” relates to “Routing Over Low power and Lossy networks” and is driven by the Internet of Everything and Machine-to-Machine communication.

OSPF Enhancements

OSPF is a link-state protocol that uses a common database to determine the shortest path to any destination.

Two main areas of interest in the Open Shortest Path First IGP (OSPF) subgroups are OSPFv3 LSA Extendibility and Remote Loop-Free Alternatives ( LFAs ). One benefit IS-IS has over OSPF is its ability to easily introduce new features with the inclusion of Type Length Values ( TLVs ) and sub-TLVs. The IETF draft-IETF-OSPF-ospfv3-lsa-extend extends the LSA format by allowing the optional inclusion of TLVs, making OSPF more flexible and extensible. For example, OSPFv3 uses a new TLV to support intra-area Traffic Engineering ( TE ), while OSPFv2 uses an opaque LSA.

Another shortcoming of OSPF is that it does not install a backup route in the routing table by default. Having a pre-installed backup up path greatly improves convergence time. With pre-calculated backup routes already installed in the routing table, the router process does not need to go through the convergence process’s LSA propagation and SPF calculation steps.

Loop-free alternatives (LFA)

Loop-Free Alternatives ( LFA ), known as Feasible Successors in EIGRP, are local router decisions to pre-install a backup path.
In the diagram below:

-Router A has a primary ( A-C) and secondary ( A-B-C) path to 10.1.1.0/24
-Link State allows Router A to know the entire topology
-Router A should know that Router B is an alternative path. Router B is a Loop-Free Alternate for destination 10.1.1.0/24

This is not done with any tunneling, and the backup route needs to exist for it to be used by the RIB. If the second path doesn’t exist in the first place, the OSPF process cannot install a Loop-Free Alternative. The LFA process does not create backup routes if they don’t already exist. An LFA is simply an alternative loop-free route calculated at any network router.

A drawback of LFA is that it cannot work in all topologies. This is most notable in RING topologies. The answer is to tunnel and to get the traffic past the point where it will loop. This effectively makes the RING topology a MESH topology. For example, the diagram below recognizes that we must tunnel traffic from A to C. The tunnel type doesn’t matter – it could be a GRE tunnel, an MPLS tunnel, an IP-in-IP tunnel, or just about any other encapsulation.

In this network:

-Router A’s best path through E
-Routers C’s best path is through D
-Router A must forward traffic directly to C to prevent packets from looping back.

In the preceding example, we will look at “Remote LFA,” which leverages an MPLS network and Label Distribution Protocol ( LDP ) for label distribution. If you use Traffic Engineering ( TE ), it’s called “TE Fast ReRoute” and not “Remote LFA.” There is also a hybrid model combining Remote LFA and TE Fast ReRoute, and is used only when the above cannot work efficiently due to a complex topology or corner case scenario.

Remote LFAs extend the LFA space to “tunneled neighbors”.

– Router A runs a constrained SPF and finds C is a valid LFA

– Since C is not directly connected, Router A must tunnel to C

a) Router A uses LDP to configure an MPLS path to C

b) Installs this alternate path as an LFA in the CEF table

– If the A->E link fails.

a) Router A begins forwarding traffic along the LDP path

The total time for convergence usually takes 10ms.

Remote LFA has some topology constraints. For example, they cannot be calculated across a flooding domain boundary, i.e., an ABR in OSPF or L1/L2 boundary is IS-IS. However, they work in about 80% of all possible topologies and 90% of production topologies.

BGP Enhancements

BGP is a scalable distance vector protocol that runs on top of TCP. It uses a path algorithm to determine the best path to install in the IP routing table and for IP forwarding.

Recap BGP route advertisement:

RR client can send to a RR client.
RR client can send to a non RR client.
A non-RR client cannot send to a non-RR client.

One drawback to the default BGP behavior is that it only advertises the best route. When a BGP Route Reflector receives multiple paths to the same destination, it will advertise only one of those routes to its neighbors.

This can limit the visibility in the network and affect the best path selection used for hot potato routing when you want traffic to leave your AS as quickly as possible. In addition, all paths to exit an AS are not advertised to all peers, basically hiding ( not advertising ) some paths to exit the network.

The diagram below displays default BGP behavior; the RR receives two routes from PE2 and PE3 about destination Z; due to the BGP best path mechanism, it only advertises one of those paths to PE1.

Route Reflector - Default — Diagram: Route Reflector – Default.

In certain designs, you could advertise the destination CE with different Route Distinguishers (RDs), creating two instances for the same destination prefix. This would allow the RR to send two paths to PE.

Diverse BGP path distribution

Another new feature is diverse BGP Path distribution, where you can create a shadow BGP session to the RR. It is easy to deploy, and the diverse iBGP session will announce the 2nd best path. Shadow BGP sessions are especially useful in virtualized deployments, where you can create another BGP session to a VM acting as a Route-Reflector. The VM can then be scaled out in a virtualized environment creating numerous BGP sessions. You are allowing the advertisements of multiple paths for each destination prefix.

Route Reflector - Shadow Sessions — Diagram: Route Reflector – Shadow Sessions.

BGP Add-path

A special extension to BGP known as “Add Paths” allows BGP speakers to propagate and accept multiple paths for the same prefix. The BGP Add-Path feature will signal diverse paths, so you don’t need to create shadow BGP sessions. There is a requirement that all Add-Path receiver BGP routers must support the Add-Path capability.

There are two flavors of the Add-Path capability, Add-n-path, and Add-all-path. The “Add-n-path” will add 2 or 3 paths depending on the IOS version. With “Add-all-path,” the route reflector will do the primary best path computation (only on the first path) and then send all paths to the BR/PE. This is useful for large ECMP load balancing, where you need multiple existing paths for hot potato routing.

Source packet routing

Another interesting draft the IETF is working on is Source Packet Routing ( spring ). Source Packet Routing is the ability of a node to specify a forwarding path. As the packet arrives in the network, the edge device looks at the application, determines what it needs, and predicts its path throughout the network. Segment routing leverages the MPLS data plane, i.e., push, swap, and pop controls, without needing LDP or RSVP-TE. This avoids millions of labels in the LDP database or TE LSPs in the networks.

Diagram: Application Controls – Network Delivers

The complexity and state are now isolated to the network’s edges, and the middle nodes are only swapping labels. The source routing is based on the notion of a 32-bit segment that can represent any instructions, such as service, context, or IGP-based forwarding construct. This results in an ordered chain of topological and service instructions where the ingress node pushes the segment list on the packet.

Prefix Hijacking in BGP

BGP hijacking revolves around locating an ISP that is not filtering advertisements, or its misconfiguration makes it susceptible to a man-in-the-middle attack. Once located, an attacker can advertise any prefix they want, causing some or all traffic to be diverted from the real source towards the attacker.

In February 2008, a large portion of YouTube’s address space was redirected to Pakistan when the Pakistan Telecommunication Authority ( PTA ) decided to restrict access to YouTube.com inside the country but accidentally blackholed the route in the global BGP table.

These events and others have led the Secure-Inter Domain Routing Group ( SIDR ) to address the following two vulnerabilities in BGP:

-Is the AS authorized to originate an IP prefix?

-Is the AS-Path represented in the route the same as the path through which the NLRI traveled?

This lockdown of BGP has three solution components:

RPKI Infrastructure	Offline repository of verifiable secure objects based on public-key cryptography
	Follows resources (IPv4/v6 + ASN) allocation hierarchy to provide “right of use”
BGP Secure Origin AS	You only validate the Origin AS of a BGP UPDATE
	Solves most frequent incidents (*)
	No changes to BGP nor the router’s hardware impact
	Standardization is almost finished and running code
BGP PATH Validation	BGPSEC proposal under development at IETF
	Requires forward signing AS-PATH attribute
	Changes in BGP and possible routers

The roll-out and implementation should be gradual and create islands of trust worldwide. These islands of trust will eventually interconnect together, making BGP more secure.

The table below displays the RPKI Deployment State;

RIR	Total	Valid	Invalid	Unknown	Accuracy	RPKI Adoption Rate
AFRINIC	100%	.44%	.42%	99.14%	51.49%	.86%
APNIC	100%	.22%	.24%	99.5%	48.32%	.46%
ARIN	100%	.4%	.14%	99.46%	74.65%	.54%
LACNIC	100%	17.84%	2.01%	80.15%	89.87%	19.85%
RIPE NCC	100%	6.7%	0.59%	92.71%	91.92%	7.29%

Cloud Enhancements – The Intercloud

Today’s clouds have crossed well beyond the initial hype, and applications are now offered as on-demand services ( anything-as-a-service [XaaS] ). These services are making significant cost savings, and the cloud transition is shaping up to be as powerful as the previous one – the Internet. The Intercloud and the Internet of Things are the two new big clouds of the future.

Currently, the cloud market is driven by two spaces – the public cloud ( off-premise ) and the private cloud (on-premise). The intercloud takes the concept of cloud much further and attempts to connect multiple public clouds. A single application that could integrate services and workloads from ten or more clouds would create opportunities and potentially alter the cloud market landscape significantly. Hence, it is important to know and understand the need for cloud migration and its related problems.

We are already beginning to see signs of this in the current market. Various applications, such as Spotify and Google Maps, authenticate unregistered users with their Facebook credentials. Another use case is a cloud IaaS provider could divert incoming workload to another provider if it doesn’t have the resources to serve the incoming requests, essentially cloud bursting from provider to provider. It would also make economic sense to move workload and services between cloud providers based on cooling costs ( follow the moon ). Or maybe dynamically move workloads between providers, so they are closest to the active user base ( follow the sun )

The following diagram displays a Dynamic Workload Migration between two Cloud companies.

A: Cloud 1 finds Cloud 2	-Naming, Presence
B: Cloud 1 Trusts Cloud 2	-Certificates, Trustsec
C: Both Cloud 1 and 2 negotiate	-Security, Policy
D: Cloud 1 sets up Cloud 2	-Placement, Deployment
E: Cloud 1 sends to Cloud 2	-VM runs in cloud-Addressing, configurations

The concept of Intercloud was difficult to achieve with the previous version of vSphere based on the restriction of latency for VMotion to operate efficiently. Now vSphere v6 can tolerate 100 msec of RTT.

InterCloud is still a conceptual framework, and the following questions must be addressed before it can be moved from concept to production.

1) Intercloud security

2) Intercloud SLA management

3) Interoperability across cloud providers.

Cisco’s One Platform Kit (onePK)

The One Platform Kit is Cisco’s answer to Software Defined Networking. It aims to provide simplicity and agility to a programmatic network. It’s a set of APIs driven by programming languages, such as C and Java, that are used to program the network. We currently have existing ways to program the network with EEM applets but lack an automation tool that can program multiple devices simultaneously. It’s the same with Performance Routing ( PfR ). PfR can program and traffic engineer the network by remotely changing metrics, but the decisions are still local and not controller-based.

Traffic engineering

One useful element of Cisco’s One Platform Kit is its ability to perform “Off box” traffic engineering, i.e., the computation is made outside the local routing device. It allows you to create route paths throughout the network without relying on default routing protocol behavior. For example, the cost is the default metric for route selection for equal-length routes in OSPF. This cannot be changed, which makes the routing decisions very static. In addition, Cisco’s One Platform Kit (onePK) allows you to calculate routes using different variables you set, giving you complete path control.

Cisco Switch Virtualization Nexus 1000v

September 11, 2014

by Matt Conran Blog

Cisco Switch Virtualization Nexus 1000v

Virtualization has become integral to modern data centers in today's digital landscape. With the increasing demand for agility, flexibility, and scalability, organizations are turning to virtual networking solutions to meet their evolving needs. One such solution is the Nexus 1000v, a virtual network switch offering comprehensive features and functionalities. In this blog post, we will delve into the world of the Nexus 1000v, exploring its key features, benefits, and use cases.

The Nexus 1000v is a distributed virtual switch that operates at the hypervisor level, providing advanced networking capabilities for virtual machines (VMs). It is designed to integrate seamlessly with VMware vSphere, offering enhanced network visibility, control, and security.

Cisco Switch Virtualization is a revolutionary concept that allows network administrators to create multiple virtual switches on a single physical switch. By abstracting the network functions from the hardware, it provides enhanced flexibility, scalability, and efficiency. With Cisco Switch Virtualization, businesses can maximize resource utilization and simplify network management.

At the forefront of Cisco's Switch Virtualization portfolio is the Nexus 1000v. This powerful platform brings the benefits of virtualization to the data center, enabling seamless integration between virtual and physical networks. By extending Cisco's renowned networking capabilities into the virtual environment, Nexus 1000v empowers organizations to achieve consistent policy enforcement, enhanced security, and simplified operations.

The Nexus 1000v boasts a wide range of features that make it a compelling choice for network administrators. From advanced network segmentation and traffic isolation to granular policy control and deep visibility, this platform has it all. By leveraging the power of Cisco's Virtual Network Services (VNS), organizations can optimize their network infrastructure, streamline operations, and deliver superior performance.

Deploying Cisco Switch Virtualization, specifically the Nexus 1000v, requires careful planning and consideration. Organizations must evaluate their network requirements, ensure compatibility with existing infrastructure, and adhere to best practices. From designing a scalable architecture to implementing proper security measures, attention to detail is crucial to achieve a successful deployment.

To truly understand the impact of Cisco Switch Virtualization, it's essential to explore real-world use cases and success stories. From large enterprises to service providers, organizations across various industries have leveraged the power of Nexus 1000v to transform their networks. This section will highlight a few compelling examples, showcasing the versatility and value that Cisco Switch Virtualization brings to the table.

Matt Conran

Highlights: Cisco Switch Virtualization Nexus 1000v

Hypervisor and vSphere Introduction

An operating system can run multiple operating systems on a single hardware host using a hypervisor, also known as a virtual machine manager. Operating systems use the host’s processor, memory, and other resources. Hypervisors control the host processor, memory, and other resources and allocate what each operating system needs. Hypervisors run guest operating systems or virtual machines on top of them.

Designed specifically for integration with VMware vSphere environments, the Cisco Nexus 1000V Series Switch runs Cisco NX-OS software. Enterprise-class performance, scalability, and scalability are delivered by VMware vSphere 2.0 across multiple platforms. Within the VMware ESX hypervisor, the Nexus 1000V runs. With the Cisco Nexus 1000V Series, you can take advantage of Cisco VN-Link server virtualization technology

• Policy-based virtual machine (VM) connectivity

• Mobile VM security

• Network policy

• Nondisruptive operational model for your server virtualization and networking teams

As with physical servers, virtual servers can be configured with the same network configuration, security policy, diagnostic tools, and operational models as physical servers. The Cisco Nexus 1000V Series is also compatible with VMware vSphere, vCenter, ESX, and ESXi.

A brief overview of the Nexus 1000V system

There are two primary components of the Cisco Nexus 1000V Series switch:

VEM (Virtual Ethernet Module): Executes inside hypervisors
VSM (External Virtual Supervisor Module): Manages VEMs

Nexus 1000v implements a generic concept of Cisco Distributed Virtual Switch (DVS). VMware ESX or ESXi executes the Cisco Nexus 1000V Virtual Ethernet Module (VEM). The VEM’s application programming interface (API) is VMware vNetwork Distributed Switch (vDS).

By integrating the API with VMware VMotion and Distributed Resource Scheduler (DRS), advanced networking capabilities can be provided to virtual machines. In the VEM, Layer 2 switching and advanced networking functions are performed based on configuration information from the VSM:

Virtual routing and forwarding

Virtual routing and forwarding form the basis of this stack. Firstly, network virtualization comes with two primary methods: 1) One too many and 2) Many to one. The “one too many” network virtualization method means you segment one physical network into multiple logical segments. Conversely, the “many to one” network virtualization method consolidates numerous physical devices into one logical entity. By definition, they seem to be opposites, but they fall under the same umbrella in network virtualization.

Before you proceed, you may find the following posts helpful:

Network virtualization

Before we get stuck in Cisco virtualization, let us address some basics. For example, if you have multiple virtual endpoints share a physical network. Still, different virtual endpoints belong to various customers, and the communication between these endpoints also needs to be isolated. In other words, the network is a resource, too, and network virtualization is the technology that enables the sharing of a standard physical network infrastructure.

Virtualization uses software to simulate traditional hardware platforms and create virtual software-based systems. For example, virtualization allows specialists to construct a single virtual network or partition a physical network into multiple virtual networks.

Cisco Switch Virtualization: Logical segmentation: One too many

We have one-to-many network virtualization for the Cisco switch virtualization design; a single physical network is logically segmented into multiple virtual networks. For example, each virtual network could correspond to a user group or a specific security function.

End-to-end path isolation requires the virtualization of networking devices and their interconnecting links. VLANs have been traditionally used, and hosts from one user group are mapped to a single VLAN. To extend the path across multiple switches at Layer 2, VLAN tagging (802.1Q) can carry VLAN information between switches. These VLAN trunks were created to transport multiple VLANs over a single Ethernet interface.

The diagram below displays two independent VLANs, VLAN201 and VLAN101. These VLANs can share one physical wire to provide L2 reachability between hosts connected to Switch B and Switch A via Switch C, but they remain separate entities.

VLANs are sufficient for small Layer 2 segments. However, today’s networks will likely have a mix of Layer 2 and 3 routed networks. In this case, Layer 2 VLANs alone are insufficient because you must extend the Layer 2 isolation over a Layer 3 device. This can be achieved by using Virtual Routing and Forwarding ( VRF ), the next step in the Cisco switch virtualization. A virtual routing and forwarding instance logically carves a Layer 3 device into several isolated independent L3 devices. The virtual routing and forwarding configured locally cannot communicate directly.

The diagram below displays one physical Layer 3 router with three VRFs: VRF Yellow, VRF Red, and VRF Blue. These virtual routing and forwarding instances are completely separated; without explicit configuration, routes in one virtual routing and forwarding instance cannot be leaked to another.

Virtual Routing and Forwarding

The virtualization of the interconnecting links depends on how the virtual routers are connected. If they are physically ( directly ) connected, you could use a technology known as VRF-lite to separate traffic and 802.1Q to label the data plane. This is known as hop-by-hop virtualization.

However, it’s possible to run into scalability issues when the number of devices grows. This design is typically used when you connect virtual routing and forwarding back to back, i.e., no more than two devices.

When the virtual routers are connected over multiple hops through an IP cloud, you can use generic routing encapsulation ( GRE ) or Multiprotocol Label Switching ( MPLS ) virtual private networks.

GRE is probably the simpler of the Layer 3 methods, and it can work over any IP core. GRE can encapsulate the contents and transport them over a network with the network unaware of the packet contents. Instead, the core will see the GRE header, virtualizing the network path.

Cisco Switch Virtualization: The additional overhead

When designing Cisco switch virtualization, you need to consider the additional overhead. There are a further 24 bytes overhead for the GRE header, so it may be the case that the forwarding router may break the datagram into two fragments, so the packet may not be larger than the outgoing interface MTU. To resolve the fragmentation issue, you can correctly configure MTU, MSS, and Path MTU parameters on the outgoing and intermediate routers.

The GRE standard is typically static. You only need to configure tunnel endpoints, and the tunnel will be up as long as you can reach those endpoints. However, recent designs can establish a dynamic GRE tunnel.

MPLS/VPN, on the other hand, is a different beast. It requires signaling to distribute labels and build an end-to-end Label Switched Path ( LSP ). The label distribution can be done with BGP+label, LDP, and RSVP. Unlike GRE tunnels, MPLS VPNs do not have to manage multiple point-to-point tunnels to provide a full mesh of connectivity. Instead, they are used for connectivity, and packets’ labels provide traffic separation.

Cisco switch virtualization: Many to one

Many-to-one network consolidation refers to grouping two or more physical devices into one. Examples of this Cisco switch virtualization technology include a Virtual Switching System ( VSS ), Stackable switches, and Nexus VPC. Combining many physicals into one logical entity allows STP to view the logical group as one, allowing all ports to be active. By default, STP will block the redundant path.

Software-defined networking takes this concept further; it completely abstracts the entire network into a single virtual switch. The control and data planes are on the same device on traditional routers, yet they are decoupled with SDN. The control plan is now on a policy-driven controller, and the data plane is local on the OpenFlow-enabled switch.

Network Virtualization

Server and network virtualization presented the challenge of multiple VMs sharing a single network physical port, such as a network interface controller ( NIC ). The question then arises: how do I link multiple VMs to the same uplink? How do I provide path separation? Today’s networks need to virtualize the physical port and allow the configuration of policies per port.

NIC-per-VM design

One way to do this is to have a NIC-per-VM design where each VM is assigned a single physical NIC, and the NIC is not shared with any other VM. The hypervisor, aka virtualization layer, would be bypassed, and the VM would access the I/O device directly. This is known as VMDirectPath.

This direct path or pass-through can improve performance for hosts that utilize high-speed I/O devices, such as 10 Gigabit Ethernet. However, the lack of flexibility and the ability to move VMs offset higher performance benefits.

Virtual-NIC-per-VM in Cisco UCS (Adapter FEX)

Another way to do this is to create multiple logical NICs on the same physical NIC, such as Virtual-NIC-per-VM in Cisco UCS (Adapter FEX). These logical NICs are assigned directly to VMs, and traffic gets marked with a vNIC-specific tag on the hardware (VN-Tag/802.1ah).

The actual VN-Tag tagging is implemented in the server NICs so that you can clone the physical NIC in the server to multiple virtual NICs. This technology provides faster switching and enables you to apply a rich set of management features to local and remote traffic.

Software Virtual Switch

The third option is to implement a virtual software switch in the hypervisor. For example, VMware introduced virtual switching compatibility with its vSphere ( ESXi ) hypervisor, called vSphere Distributed Switch ( VDS ). Initially, they introduced a local L2 software switch, which was soon phased out due to a lack of distributed architecture.

Data physically moves between the servers through the external network, but the control plane abstracts this movement to look like one large distributed switch spanning multiple servers. This approach has a single management and configuration point, similar to stackable switches – one control plane with many physical data forwarding paths.

The data does not move through a parent partition but logically connects directly to the network interface through local vNICs associated with each VM.

Network virtualization and Nexus 1000v ( Nexus 1000 )

The VDS introduced by VMware lacked any good networking features, which led Cisco to introduce the Nexus 1000V software-based switch. The Nexus 1000v is a multi-cloud, multi-hypervisor, and multi-services distributed virtual switch. Its function is to enable communication between VMs.

Nexus 1000 components: VEM and VSM

The Nexus 1000v has two essential components:

The Virtual Supervisor Module ( VSM )
The Virtual Ethernet Module ( VEM ).

Compared to a physical switch, the VSM could be viewed as the supervisor, setting up the control plane functions for the data plane to forward efficiently, and the VEM as the physical line cards that do all the packet forwarding. The VEM is the software component that runs within the hypervisor kernel. It handles all VM traffic, including inter-VM frames and Ethernet traffic between a VM and external resources.

The VSM runs its NX-OS code and controls the control and management planes, which integrate into a cloud manager, such as a VMware vCenter. You can have two VSMs for redundancy. Both modules remain constantly synchronized with unicast VSM-to-VSM heartbeats to provide stateful failover in the event of an active VSM failure.

The two available communication options for VSM to VEM are:

Layer 2 control mode: The VSM control interface shares the same VLAN with the VEM.
Layer 3 control mode: The VEM and the VSM are in different IP subnets.

The VSM also uses heartbeat messages to detect a loss of connectivity between it and the VEM. However, the VEM does not depend on connectivity to the VSM to perform its data plane functions and will continue forwarding packets if the VSM fails.

With Layer 3 control mode, the heartbeat messages are encapsulated in a GRE envelope.

Nexus 1000 and VSM best practices

L2 control is recommended for new installations.
Use MAC pinning instead of LACP.
Packet, Control, and Management in the same VLAN.
Do not use VLAN 1 for Control and Packet.
Use 2 x VSM for redundancy.

The max latency between VSM and VEM is ten milliseconds. Therefore, a VSM can be placed outside the data center if you have a high-quality DCI link, and the VEM can still be controlled.

Nexus 1000v InterCloud – Cisco switch virtualization

A vital element of the Nexus 1000 is its use case for hybrid cloud deployments and its ability to place workloads in private and public environments via a single pane of glass. In addition, the Nexus 1000v interCloud addresses the main challenges with hybrid cloud deployments, such as security concerns and control/visibility challenges within the public cloud.

The Nexus 1000 interCloud works with Cisco Prime Service Controller to create a secure L2 extension between the private data center and the public cloud.

This L2 extension is based on Datagram Transport Layer Security ( DTLS ) protocol and allows you to securely transfer VMs and Network services over a public IP backbone. DTLS derives the SSL protocol and provides communications privacy for datagram protocols, so all data in motion is cryptographically isolated and encrypted.

Nexus 1000v Hybrid Cloud Components

Cisco Prime Network Service Controller for InterCloud **	A VM that provides a single pane of glass to manage all functions of the inter clouds
InterCloud VSM	Manage port profiles for VMs in the InterCloud infrastructure
InterCloud Extender	Provides secure connectivity to the InterCloud Switch in the provider cloud. Install in the private data center.
InterCloud Switch	Virtual Machine in the provider data center has secure connectivity to the InterCloud Extender in the enterprise cloud and secure connectivity to the Virtual Machines in the provider cloud.
Cloud Virtual Machines	VMs in the public cloud running workloads.

Prerequisites

Port 80	HTTP access from PNSC for AWS calls and communicating with InterCloud VMs in the provider cloud
Port 443	HTTPS access from PNSC for AWS calls and communicating with InterCloud VMs in the provider cloud
Port 22	SSH from PNSC to InterCloud VMs in the provider cloud
UDP 6644	DTLS data tunnel
TCP 6644	DTLS control tunnel

VXLAN – Virtual Extensible LAN

The requirement for applications on demand has led to an increased number of required VLANs for cloud providers. The standard 12-bit identifier, which provided 4000 VLANs, proved to be a limiting factor in multi-tier, multi-tenant environments, and engineers started to run out of isolation options.

This has introduced a 24-bit VXLAN identifier, offering 16 million logical networks. Now, we can cross Layer 3 boundaries. The MAC in UDP encapsulation uses switch hashing to analyze UDP packets and efficiently distribute all packets in a port channel.

VXLAN works like a layer 2 bridge ( Flood and Learn ); the VEM learn does all the heavy lifting, learns all the VM source MAC and Host VXLAN IPs, and encapsulates the traffic according to the port profile to which the VM belongs. Broadcast, Multicast, and unknown unicast traffic are sent as Multicast.

At the same time, unicast traffic is encapsulated and shipped directly to the destination host’s VXLAN IP, aka destination VEM. Enhanced VXLAN offers VXLAN MAC distribution and ARP termination, making it more optional.

VXLAN Mode Packet Functions

Packet	VXLAN(multicast mode)	Enhanced VXLAN(unicast mode)	Enhanced VXLANMAC Distribution	Enhanced VXLANARP Termination
Broadcast /Multicast	Multicast Encapsulation	Replication plus Unicast Encap	Replication plus Unicast Encap	Replication plus Unicast Encap
Unknown Unicast	Multicast Encapsulation	Replication plus Unicast Encap	Drop	Drop
Known Unicast	Unicast Encapsulation	Unicast Encap	Unicast Encap	Unicast Encap
ARP	Multicast Encapsulation	Replication plus Unicast Encap	Replication plus Unicast Encap	VEM ARP Reply

vPath – Service chaining

Intelligent Policy-based traffic steering through multiple network services.

vPath allows you to intelligently traffic steer VM traffic to virtualized devices. It intercepts and redirects the initial traffic to the service node. Once the service node performs its policy function, the result is cached, and the local virtual switch treats the subsequent packets accordingly. In addition, it enables you to tie services together to push the VM through each service as required. Previously, if you wanted to tie services together in a data center, you needed to stitch the VLANs together, which was limited by design and scale.

Cisco virtualization — Nexus and service chaining

vPath 3.0 is now submitted to the IETF for standardization, allowing service chaining with vPath and non-vpath network services. It enables you to use vpath service chaining between multiple physical devices and supporting multiple hypervisors.

License Options

Nexus 1000 Essential Edition	Nexus 1000 Advanced Edition
Full Layer-2 Feature Set	All Features of Essential Edition
Security, QoS Policies	VSG firewall
VXLAN virtual overlays	VXLAN Gateway
vPath enabled Virtual Services	TrustSec SGA
Full monitoring and management capabilities	A platform for other Cisco DC Extensions in the Future
Free	$695 per CPU MSRP

Nexus 1000 features and benefits

Switching	L2 Switching, 802.1Q Tagging, VLAN, Rate Limiting (TX), VXLAN IGMP Snooping, QoS Marking (COS & DSCP), Class-based WFQ
Security	Policy Mobility, Private VLANs w/ local PVLAN Enforcement Access Control Lists, Port Security, Cisco TrustSec Support Dynamic ARP inspection, IP Source Guard, DHCP Snooping
Network Services	Virtual Services Datapath (vPath) support for traffic steering & fast-path off-load[leveraged by Virtual Security Gateway (VSG), vWAAS, ASA1000V]
Provisioning	Port Profiles, Integration with vC, vCD, SCVMM*, BMC CLM Optimized NIC Teaming with Virtual Port Channel – Host Mode
Visibility	VM Migration Tracking, VC Plugin, NetFlow v.9 w/ NDE, CDP v.2 VM-Level Interface Statistics, vTrackerSPAN & ERSPAN (policy-based)
Management	Virtual Centre VM Provisioning, vCenter Plugin, Cisco LMS, DCNM Cisco CLI, Radius, TACACs, Syslog, SNMP (v.1, 2, 3) Hitless upgrade, SW Installer

Advantages and disadvantages of the Nexus 1000

Advantages	Disadvantages
The Standard edition is FREE; you can upgrade to an enhanced version when needed.	VEM and VSM internal communication is very sensitive to latency. Due to their chatty nature, they may not be good for inter-DC deployments.
Easy and Quick to deploy	VSM – VEM, VSM (active) – VSM (standby) heartbeat time of 6 seconds makes it sensitive to network failures and congestion.
It offers you many rich network features unavailable on other distributed software switches.	VEM over-dependency on VSM reduces resiliency.
Hypervisor agnostic	VSM is required for vSphere HA, FT, and VMotion to work.
Hybrid Cloud functionality

Closing Points on Cisco Nexus 1000v

Virtual Ethernet Module (VEM):

The Nexus 1000v employs the Virtual Ethernet Module (VEM), which runs as a module inside the hypervisor. This allows for efficient and direct communication between VMs, bypassing the traditional reliance on the hypervisor networking stack.

Virtual Supervisor Module (VSM):

The Virtual Supervisor Module (VSM) serves as the control plane for the Nexus 1000v, providing centralized management and configuration. It enables network administrators to define policies, manage virtual ports, and monitor network traffic.

Policy-Based Virtual Network Management:

With the Nexus 1000v, administrators can define policies to manage virtual networks. These policies ensure consistent network configurations across multiple hosts, simplifying network management and reducing the risk of misconfigurations.

Advanced Security and Monitoring Capabilities:

The Nexus 1000v offers granular security controls, including access control lists (ACLs), port security, and dynamic host configuration protocol (DHCP) snooping. Additionally, it provides comprehensive visibility into network traffic, enabling administrators to monitor and troubleshoot network issues effectively.

Benefits of the Nexus 1000v:

Enhanced Network Performance:

By offloading network processing to the VEM, the Nexus 1000v minimizes the impact on the hypervisor, resulting in improved network performance and reduced latency.

Increased Scalability:

The distributed architecture of the Nexus 1000v allows for seamless scalability, ensuring that organizations can meet the growing demands of their virtualized environments.

Simplified Network Management:

With its policy-based approach, the Nexus 1000v simplifies network management tasks, enabling administrators to provision and manage virtual networks more efficiently.

Use Cases:

Data Centers:

The Nexus 1000v is particularly beneficial in data center environments where virtualization is prevalent. It provides a robust and scalable networking solution, ensuring optimal performance and security for virtualized workloads.

Cloud Service Providers:

Cloud service providers can leverage the Nexus 1000v to enhance their network virtualization capabilities, offering customers more flexibility and control over their virtual networks.

The Nexus 1000v is a powerful virtual network switch that provides advanced networking capabilities for virtualized environments. Its rich features, policy-based management approach, and seamless integration with VMware vSphere allow organizations to achieve enhanced network performance, scalability, and management efficiency. As virtualization continues to shape the future of data centers, the Nexus 1000v remains a valuable tool for optimizing virtual network infrastructures.

Summary: Cisco Switch Virtualization Nexus 1000v

Welcome to our blog post, where we dive into the world of Cisco Switch Virtualization, explicitly focusing on the Nexus 1000v. In this article, we will unravel the complexities surrounding switch virtualization, explore the key features of the Nexus 1000v, and understand its significance in modern networking environments.

Understanding Switch Virtualization

Switch virtualization is a technique that allows for creating multiple virtual switches on a single physical switch, enabling greater flexibility and efficiency in network management. Organizations can consolidate their infrastructure, reduce costs, and streamline network operations by virtualizing switches.

Introducing the Nexus 1000v

The Cisco Nexus 1000v is a powerful switch virtualization solution that extends the functionality of VMware environments. Unlike traditional virtual switches, it provides a more comprehensive set of features and advanced network control. It seamlessly integrates with VMware vSphere, offering enhanced visibility, security, and policy management.

Key Features of the Nexus 1000v

– Distributed Virtual Switch: The Nexus 1000v operates as a distributed virtual switch, distributing network intelligence across all hosts in the virtualized environment. This ensures consistent policies, simplified troubleshooting, and improved performance.

– Virtual Port Profiles: With virtual port profiles, administrators can define consistent network policies for virtual machines, irrespective of their physical location. This simplifies network provisioning and reduces the chances of misconfigurations.

– Network Analysis Module (NAM): The Nexus 1000v incorporates NAM, a robust monitoring and analysis tool that provides deep visibility into virtual network traffic. This enables administrators to identify and resolve network issues, ensuring optimal performance quickly.

Deployment Considerations

When planning to deploy the Nexus 1000v, it is essential to consider factors such as network architecture, compatibility with existing infrastructure, and scalability requirements. It is advisable to consult with Cisco experts or certified partners to ensure a smooth and successful implementation.

Conclusion:

In conclusion, the Cisco Nexus 1000v is a game-changer in switch virtualization. Its advanced features, seamless integration with VMware environments, and extensive network control make it an ideal choice for organizations seeking to optimize their network infrastructure. By understanding the fundamentals of switch virtualization and exploring Nexus 1000v’s capabilities, network administrators can unlock a world of possibilities in network management and performance.

Spine Leaf Architecture

September 4, 2014

by Matt Conran Blog

Network Architecture · Data Center Fabrics

Spine–Leaf Architecture: Designing Scalable Data Center Fabrics

Modern data centers must move increasing volumes of traffic between distributed applications, virtual machines, containers and storage systems. Traditional hierarchical networks can struggle to deliver the bandwidth, predictable latency and flexibility these workloads demand. Spine–leaf architecture addresses these challenges through a scalable, multi-path network fabric.

Also known as a folded-Clos architecture, a conventional two-tier spine–leaf fabric consists of leaf switches that connect servers and other endpoints, and spine switches that interconnect the leaf layer. Each leaf connects to every spine, creating multiple equal-cost paths between endpoints attached to different leaves. Routing protocols and Equal-Cost Multipath (ECMP) forwarding allow eligible traffic flows to use these alternative paths.

This design is particularly suited to east–west traffic: communication between application servers, distributed services, virtualised workloads and storage systems inside the data center. Rather than forcing traffic through multiple hierarchical aggregation layers, the fabric provides a consistent path structure that can improve capacity planning, simplify expansion and make network behaviour easier to reason about.

The engineering principles behind the fabric

Predictable paths Inter-leaf traffic normally traverses one spine, providing a consistent hop count across the fabric.

Multipath forwarding ECMP distributes eligible flows across available equal-cost routes instead of relying on a single uplink.

Scale-out capacity Add leaf switches for endpoint capacity or additional spines when the design and port budgets permit.

Failure resilience Redundant paths can preserve connectivity after failures, subject to topology, routing and remaining capacity.

However, a spine–leaf diagram alone does not guarantee a non-blocking network. Switch forwarding capacity, link speeds, oversubscription, ECMP hashing, routing convergence and application traffic patterns all influence actual performance. Resilient designs must account for both normal operating conditions and the capacity remaining after a component fails.

What this guide explores

Move from topology fundamentals to real engineering decisions: compare traditional and spine–leaf designs, calculate oversubscription, trace ECMP paths, investigate failures, examine routed underlays and VXLAN/EVPN overlays, and understand how telemetry helps engineers diagnose and optimise a data center fabric.

Interactive Reading Guide

7 SECTIONS · ONE FABRIC

Explore the Spine–Leaf Architecture Guide

Move from the fundamentals of fabric design to capacity planning, resilience, overlays and real-world operations. Select any section to jump directly to its explanation.

FUNDAMENTALS → IMPLEMENTATION

Section 01 · Architecture Fundamentals

Traditional Three-Tier vs. Spine–Leaf Architecture

Data center architecture determines how traffic moves between servers, how bandwidth is distributed, and how the network responds as workloads grow. Understanding the difference between traditional hierarchical designs and spine–leaf fabrics is the starting point for making informed capacity, resilience and routing decisions.

The traditional three-tier model

Traditional data centers commonly use three logical network layers: access, aggregation and core. Access switches connect servers and endpoints. Aggregation switches collect access-layer traffic and may provide Layer 3 routing, policy enforcement and connectivity between network segments. Core switches provide high-speed transport between aggregation blocks and other network domains.

This hierarchy can work well for traffic patterns that move predominantly between access networks and external destinations. However, the aggregation and core layers can become capacity constraints when large numbers of distributed applications exchange traffic internally. Scaling may require additional uplinks, larger chassis or redesigns of existing aggregation blocks.

Traditional design

Three-tier hierarchy

Core

↓ ↑

Aggregation / Distribution

↓ ↑

Access switches

↓ ↑

Servers and endpoints

Multiple hierarchical layers can introduce additional hops and concentrate traffic on aggregation uplinks.

Modern fabric

Two-tier spine–leaf

Spine switches

↕ Multiple routed paths

Leaf switches

↕

Servers and endpoints

Each leaf connects to every spine, providing equal-cost paths between different leaves in a conventional full fabric.

Why the design approach changed

Virtualisation, cloud platforms, distributed databases and containerised applications have increased the importance of east–west traffic—communication between workloads inside the data center. An application request may trigger multiple exchanges between services, storage systems and compute nodes before a response is returned.

Spine–leaf architecture addresses this pattern through a regular, repeatable topology. Endpoints connect to leaf switches, and each leaf connects to every spine. Traffic between endpoints attached to different leaves normally travels from the source leaf to a spine and then to the destination leaf. In a conventional two-tier fabric, this provides a predictable inter-leaf hop count.

Multiple routed paths allow Equal-Cost Multipath (ECMP) to forward eligible flows across available links. This can improve aggregate bandwidth utilisation and provide alternative paths when links or spine devices fail. The actual result depends on link capacity, routing convergence, traffic distribution and the fabric's remaining resources.

Key architectural differences

Design consideration	Traditional three-tier	Spine–leaf fabric
Topology	Hierarchical access, aggregation and core	Leaf and spine switching tiers
Inter-leaf path	May traverse aggregation and core layers	Normally one spine between different leaves
Multipath forwarding	Depends on the design and supported protocols	Commonly uses ECMP over routed links
Scaling	May require uplink or hierarchical redesign	Add leaves or spines within design limits
Capacity planning	Focuses on hierarchical uplinks and aggregation	Focuses on leaf uplinks, spine capacity and oversubscription
Failure behaviour	Depends on redundancy and available alternate paths	Remaining ECMP paths can maintain connectivity if reachable

What engineers must still consider

Spine–leaf is not automatically non-blocking, congestion-free or fault-tolerant in every deployment. The architect must calculate uplink capacity, understand oversubscription, verify the forwarding capabilities of the switches and test the consequences of losing a spine or a leaf uplink. Redundant server connections also require an appropriate design, such as supported LACP or multichassis link aggregation where needed.

Engineering takeaway

The key difference is not simply two tiers instead of three. Spine–leaf creates a regular multipath fabric that makes inter-leaf connectivity and scale-out capacity more predictable. Good engineering still depends on the routing design, available bandwidth, failure strategy and workload requirements.

Interactive Architecture Explorer · 01

Compare the Network Architectures

Switch architectures, select network devices to inspect their roles, and animate an illustrative server-to-server path. This simplified diagram explains the design concepts rather than representing a vendor-specific production configuration.

Network devices Connections Servers

Selected device / concept

Leaf switches

Leaf switches connect servers and provide uplinks into the spine layer.

Architecture insight

2 tiers

A common spine–leaf fabric has leaf and spine layers. Actual topology and capacity depend on the design.

Traffic animation is illustrative. Actual forwarding paths depend on routing, ECMP hashing, link state, topology and the configured network.

Section 02 · Fabric and Routing

Spine–Leaf Fabric Architecture: Routing, ECMP and Traffic Flow

The defining feature of a spine–leaf network is its regular connectivity pattern. Every leaf switch connects to every spine switch, while servers and other endpoints attach to the leaf layer. This creates a predictable physical fabric in which routing and multipath forwarding determine how traffic reaches its destination.

Understanding the role of each layer

Leaf switches provide endpoint connectivity. They commonly connect to physical servers, virtualisation hosts, storage systems, firewalls and external network services. Depending on the design, leaf switches may also host Layer 3 gateway functions and VXLAN tunnel endpoints (VTEPs).

Spine switches provide the transit fabric between leaves. In a routed Layer 3 design, each leaf-to-spine connection is normally a point-to-point routed link. Spine switches forward traffic between the relevant leaf-facing interfaces without requiring the spine tier to act as a shared Layer 2 segment.

Tier 01

Leaf layer

Connects endpoints, advertises reachable routes and forwards traffic towards the destination leaf or an upstream service.

Tier 02

Spine layer

Interconnects the leaf switches and provides multiple transit paths across the fabric.

Routing

Underlay control plane

Protocols such as eBGP or OSPF establish Layer 3 reachability. The protocol and route design depend on the chosen fabric.

How a packet crosses the fabric

Consider Server A connected to Leaf 1 and Server B connected to Leaf 2. When Server A sends traffic to Server B, Leaf 1 performs its forwarding lookup and selects an appropriate next hop. In a conventional routed fabric, the packet travels across one of the available leaf-to-spine links. The selected spine then forwards it across its connection to Leaf 2, which delivers the packet to Server B.

Illustrative inter-leaf forwarding path

Server A

→

Leaf 1

→

Spine

→

Leaf 2

→

Server B

The diagram illustrates the forwarding sequence, not a complete physical topology. A full fabric normally has multiple spines and a connection from each leaf to every spine.

Endpoints attached to the same leaf may communicate directly through that switch when the destination is locally reachable. Traffic between different leaves normally traverses a spine. This regular structure provides a consistent inter-leaf hop count, although the actual latency still depends on link speeds, congestion, forwarding performance and workload behaviour.

Equal-Cost Multipath (ECMP)

ECMP allows a router to install multiple next hops for a destination when the routing protocol identifies paths with equal routing cost. In a spine–leaf fabric, a leaf can commonly reach a destination prefix through multiple spine switches. Traffic can therefore use several available paths instead of depending on a single uplink.

Many implementations use a hash based on packet or flow fields to select a next hop. This typically keeps packets from a flow on the same path, helping preserve packet ordering. Consequently, ECMP does not guarantee that every link receives exactly the same amount of traffic: hash distribution, flow sizes and traffic patterns can create uneven utilisation.

Engineering factor	What it means
Equal routing cost	Multiple next hops are eligible for forwarding to a destination according to the routing implementation.
Flow hashing	Traffic is mapped to paths using selected header fields; one large flow may dominate an individual link.
Available path count	The number of usable paths depends on topology, routing policy, installed routes and hardware limits.
Failure convergence	Following a failure, routes and forwarding state must update before traffic can reliably use alternate paths.

Underlay and overlay: two different responsibilities

The underlay provides IP reachability across the physical fabric. In a VXLAN/EVPN design, leaf switches commonly advertise VTEP loopback reachability through the underlay. The overlay carries tenant or workload traffic using VXLAN encapsulation between VTEPs, while EVPN can distribute endpoint and reachability information.

Keeping these functions distinct helps engineers troubleshoot connectivity. A workload communication problem may arise from underlay routing, overlay control-plane information, endpoint learning, policy enforcement or the physical links themselves. Verifying each layer separately narrows the fault domain.

Engineering takeaway

Spine–leaf provides a repeatable physical topology; routing establishes reachability, and ECMP enables multipath forwarding. Predictable performance still requires careful capacity planning, balanced traffic distribution and validation of failure behaviour.

Interactive Fabric Lab · 02

Trace a Route Through the Spine–Leaf Fabric

Follow traffic from Server A to Server B. Compare two illustrative equal-cost paths, inspect the role of each device, and step through forwarding. The example assumes a routed underlay with ECMP-capable paths between the leaf switches.

Path A Path B Other links

Step 1 of 4 · Traffic enters Leaf 1

Forwarding stage

Leaf 1 receives traffic

Leaf 1 receives the packet from Server A and performs a forwarding lookup. The route and next-hop information determine how it proceeds.

Fabric principle

ECMP

Equal-cost multipath can provide more than one eligible next hop to the destination.

Important: ECMP makes eligible equal-cost next hops available; it does not guarantee that every flow or link carries exactly the same amount of traffic. The forwarding choice depends on the routing table, hashing algorithm, flow characteristics and current topology.

Section 03 · Capacity Engineering

Spine–Leaf Capacity Planning: Bandwidth, Oversubscription and Growth

A spine–leaf fabric can provide predictable paths and substantial aggregate bandwidth, but its real capacity depends on link speeds, the number of uplinks, traffic distribution and the amount of bandwidth required during failures. Capacity planning must account for both normal operation and degraded conditions.

What oversubscription actually means

Oversubscription compares the bandwidth offered by a group of downstream interfaces with the bandwidth available through its upstream links. It is a useful design indicator, not a prediction that every link will be congested. Actual utilisation depends on workload demand, traffic locality, flow sizes and how traffic is distributed across available paths.

Core design formula

Oversubscription ratio = Downstream bandwidth ÷ Upstream bandwidth

Calculate both sides using comparable bandwidth units and clearly define which interfaces and traffic direction are included.

1:1

Nominally non-oversubscribed

The defined downstream capacity equals the available upstream capacity. This does not guarantee zero contention in every traffic pattern or at every point in the network.

3:1

Oversubscribed fabric

The defined downstream capacity is three times the upstream capacity. The design relies on demand not requiring every downstream interface to transmit at full rate simultaneously.

Workload-led

Capacity matched to demand

The target ratio should reflect measured traffic, service objectives, growth expectations and the cost of additional links.

Worked example: calculating leaf uplink capacity

Consider a leaf switch with 48 server-facing ports operating at 25 Gbps and four uplinks to the spine layer, each operating at 100 Gbps. This simplified example assumes all stated ports can transmit at their nominal line rate and compares aggregate downstream port capacity with aggregate uplink capacity.

Example fabric capacity calculation

Server-facing capacity

48 × 25 Gbps = 1,200 Gbps

Aggregate spine uplinks

4 × 100 Gbps = 400 Gbps

Nominal oversubscription ratio

1,200 ÷ 400 = 3:1

Illustrative uplink capacity per leaf

400 Gbps

The resulting 3:1 ratio means the aggregate server-facing port capacity is three times the aggregate uplink capacity. It does not mean the leaf will always experience congestion. If most traffic remains local to the leaf, less bandwidth may need to cross the fabric. If many servers simultaneously send large flows to endpoints on other leaves, uplinks can become a bottleneck.

This example also excludes protocol overhead, port breakout constraints, oversubscription elsewhere in the network and implementation-specific forwarding limits. Production designs should use the actual hardware configuration and expected traffic patterns rather than relying on this ratio alone.

Plan for failures, not just normal operation

Aggregate capacity changes when a link or spine fails. If a leaf has four 100 Gbps uplinks, losing one leaves three uplinks, or 300 Gbps of nominal uplink bandwidth, assuming the remaining links and paths are operational and usable. The example's downstream-to-uplink ratio therefore rises from 3:1 to 4:1.

The actual impact depends on the failure location, routing convergence, ECMP next-hop changes and the distribution of traffic across the surviving links. A resilient design must consider whether the remaining fabric can support critical workloads while operating in this reduced-capacity state.

Design scenario	Capacity question	What to validate
Normal operation	Can the fabric support expected peak demand?	Utilisation, flow distribution, latency and congestion.
Single uplink failure	Can remaining leaf uplinks carry priority traffic?	Surviving bandwidth, routing convergence and ECMP rebalancing.
Spine failure	Can the remaining spines handle redistributed traffic?	Remaining path capacity, failure-domain impact and link utilisation.
Growth phase	Will new endpoints exceed the planned fabric capacity?	Port availability, uplink capacity, spine capacity and expansion limits.

Capacity planning is an ongoing engineering task

Design decisions should be revisited using interface counters, telemetry, flow records and application performance measurements. Monitor sustained utilisation as well as short-lived bursts, packet drops, queue behaviour and latency. A fabric can show moderate average utilisation while still experiencing congestion during brief high-demand periods.

Growth planning should also include the physical limits of the chosen platform: available ports, supported speeds, optics, cabling, power, cooling and the number of endpoints that can be added without redesigning the fabric.

Engineering takeaway

Calculate oversubscription explicitly, validate the ratio against real workload behaviour, and repeat the capacity calculation after relevant failures. A well-designed spine–leaf fabric is not simply one with many links; it is one whose available bandwidth meets service requirements under both normal and degraded conditions.

Interactive Capacity Planner · 03

Calculate Leaf Oversubscription

Change the number and speed of server-facing ports and the uplinks connecting a leaf to the spine layer. See the nominal bandwidth ratio, then simulate the loss of one uplink to understand how redundancy affects available capacity.

1. Configure the leaf switch

Start with a typical example, then adjust the values to explore different port-density and uplink designs.

Server-facing port count 48 ports

8 ports64 ports

Server port speed 25 Gb/s

10 Gb/s100 Gb/s

2. Configure the fabric uplinks

Uplink capacity is the sum of the configured uplink line rates. These controls assume all configured links are available before failure.

Uplink count 4 links

1 link8 links

Speed per uplink 100 Gb/s

25 Gb/s400 Gb/s

Worked example

48 × 25 Gb/s server ports provide 1,200 Gb/s of nominal server-facing line rate. Four × 100 Gb/s uplinks provide 400 Gb/s of nominal uplink line rate, giving a 3:1 ratio.

3. Review the capacity results

Values are nominal line-rate calculations per direction, not a prediction of real application throughput.

Server-facing capacity

1,200

Gb/s aggregate

Uplink capacity

400

Gb/s aggregate

Uplink capacity per server port

8.33

Gb/s average capacity share

Uplink links available

4 / 4

Before failure simulation

Downlink-to-uplink ratio

Server-facing line rate ÷ uplink line rate

3:1

Uplink capacity compared with downlink capacity 33.3%

Illustrative 3:1 oversubscription

The aggregate server-facing line rate is three times the nominal uplink line rate. Actual contention depends on the traffic pattern and active demand.

4. Simulate one uplink failure

This calculation removes one uplink's capacity. It assumes the remaining links can continue forwarding; real convergence and link distribution depend on the platform and routing design.

Remaining uplink capacity 300 Gb/s

Remaining uplinks 3 of 4

Ratio after failure 4:1

Uplink capacity reduction 25%

Engineering note: Ethernet is full-duplex, so line rate is normally stated per direction; do not double these values when calculating a one-direction oversubscription ratio. The ratio alone does not establish a safe or unsafe design. Validate expected traffic, burst behaviour, ECMP distribution, failure scenarios, buffer behaviour and application requirements.

Section 04 · Resilience Engineering

Failure Domains, Resilience and High Availability

A resilient spine–leaf fabric is designed to keep essential communication working when links or devices fail. Its regular topology provides multiple potential paths between leaves, but redundancy alone does not guarantee uninterrupted service. Routing convergence, remaining capacity, device dependencies and the location of the failure all influence the outcome.

Understand the main failure domains

A failure domain is the part of a network whose operation may be affected by a particular fault. A single link failure may have a limited impact, while a failed leaf can disconnect the servers attached to it. A spine failure may affect traffic across multiple leaves, even when the fabric has alternative spines available.

Link failure

One path is lost

If other usable paths remain, routing can select alternate next hops. Available bandwidth falls, and traffic may need to redistribute across the surviving links.

Spine failure

Fabric capacity is reduced

Leaves may continue communicating through the remaining spines. The remaining fabric must have sufficient capacity to carry the traffic that is redirected onto it.

Leaf failure

Attached endpoints are at risk

Servers connected only to the failed leaf can lose network connectivity. Dual-homing or redundant attachment may help, but only when the complete design supports it.

How ECMP helps the fabric recover

Equal-Cost Multipath (ECMP) allows a router to install multiple eligible next hops with the same routing cost. Traffic can then be distributed across those paths, commonly using a hash of packet or flow fields. If a next hop becomes unavailable and the routing system detects the change, the forwarding table can be updated to use the remaining eligible paths.

ECMP does not make every flow move instantly or guarantee perfectly even link utilisation. The result depends on the routing protocol, failure detection, convergence behaviour, hash inputs, flow sizes and the forwarding hardware. A small number of large flows can produce uneven utilisation even when several paths are available.

Illustrative failure sequence: one spine becomes unavailable

Normal

Multiple paths are available Leaves have routes through several spines. Eligible traffic is distributed across the installed equal-cost next hops.

Failure

A spine or its path fails Failure detection and routing updates determine when the affected next hop is removed or becomes unusable.

Recompute

Forwarding uses surviving paths The routing and forwarding systems update their usable next hops. Some packets may be lost or delayed during detection and convergence.

Validate

Service continues if the design permits Confirm that critical flows recover and that surviving links and spines can handle the redistributed traffic.

Redundancy must include enough surviving capacity

Consider a leaf with four 100 Gbps uplinks to separate spines. Before a failure, the nominal aggregate uplink capacity is 400 Gbps. If one uplink becomes unavailable, the remaining three links provide 300 Gbps of nominal aggregate capacity, assuming each remaining link is usable at its stated rate.

That reduction matters even if routing converges successfully. If traffic demand exceeds the capacity available after the failure, congestion or packet loss may occur. Capacity planning should therefore evaluate expected demand with a link or spine unavailable, not just the normal operating state.

Failure	Potential consequence	Design and test focus
Leaf-to-spine link	Fewer uplinks and reduced capacity from the affected leaf.	Confirm alternate routes, detection time and surviving uplink utilisation.
Entire spine	Multiple leaves lose one set of paths through the fabric.	Check remaining spine capacity and traffic redistribution across the fabric.
Leaf switch	Endpoints attached only to that leaf may become unreachable.	Validate server attachment redundancy, gateway placement and application recovery.
Power or shared dependency	Several devices or links may fail together.	Review power feeds, cabling paths, rack placement and common failure dependencies.

Design for independent failure domains

Two links do not necessarily provide independent protection if they share a physical path, power source or intermediate device. Likewise, connecting a server to two switches does not automatically create active-active resilience. The server NIC configuration, switching design, gateway behaviour and routing or multi-chassis technology must work together.

Depending on the architecture, server redundancy may use routed connectivity, Layer 2 link aggregation, a multi-chassis link aggregation mechanism or another supported host-attachment design. Choose the method according to platform capabilities and the required failure behaviour rather than assuming that one method fits every fabric.

Test recovery instead of assuming it

A practical resilience test should deliberately remove a link or device in a controlled environment, observe routing and forwarding changes, and measure the effect on representative application traffic. Check packet loss, recovery time, latency, route changes and the utilisation of surviving links. Repeat the test for the failures that matter to the service.

Engineering takeaway

A resilient spine–leaf fabric needs alternate paths, working failure detection, correct routing behaviour and sufficient surviving capacity. Validate the whole system—including server attachment and shared physical dependencies—under realistic failure conditions. Redundant topology is the starting point; tested recovery is the outcome that matters.

Interactive Resilience Lab · 04

What Happens When a Fabric Component Fails?

Select a failure scenario to see how a simplified routed spine–leaf fabric is affected. You can also select an individual spine or leaf to inspect its role. Green links represent available connections; red links indicate failed or affected connections.

Available connection Failed / affected Other connection

Available spine paths between Leaf 1 and Leaf 2

Illustrative paths through operational spines

Fabric connectivity

Available

For the selected source and destination in this model

Nominal transit capacity

100%

Simplified relative capacity, not measured throughput

Normal operation

Both spine paths are available in this simplified topology. Actual traffic distribution depends on routing state, ECMP hashing and link capacity.

Component inspection

Spine–leaf resilience

Select a device in the diagram to inspect its role. Use the failure controls to explore how losing a component changes the available paths.

This is a conceptual failure model, not a live network simulation. It assumes a two-spine, two-leaf routed fabric with independent links and enough remaining capacity to carry traffic after a single spine or uplink failure. Real outcomes depend on topology, routing convergence, link speeds, platform limits, endpoint attachment and traffic demand. A failed leaf can disconnect endpoints attached only to that leaf.

Section 05 · Fabric Architecture

Underlay, VXLAN/EVPN and Cisco ACI

Spine–leaf describes a physical and logical fabric topology, but it does not prescribe a single way to provide network services. A common design uses a routed Layer 3 underlay to connect the switches and an overlay to provide logical Layer 2 or Layer 3 connectivity across that fabric. Cisco ACI is another approach, combining a spine–leaf fabric with a policy-driven architecture and centralised controller-based management.

Separate transport from network services

The underlay and overlay solve different problems. The underlay provides IP reachability between the relevant fabric endpoints, while an overlay can create logical network segments that are independent of the physical location of a workload. Keeping these roles distinct makes the architecture easier to reason about, operate and troubleshoot.

Physical fabric

Leaf and spine switches, server-facing interfaces, fabric links, transceivers, cabling and physical capacity. This layer determines the available ports, speeds and physical paths.

Routed underlay

IP addressing, routing adjacencies and routes that provide reachability across the fabric. Common designs use an IGP or BGP, depending on the operational model and requirements.

Overlay and segmentation

VXLAN can encapsulate tenant traffic across the IP fabric. EVPN commonly distributes MAC and IP reachability information and supports control-plane functions for VXLAN overlays.

Policy and operations

Network policy, tenant segmentation, automation, telemetry and troubleshooting workflows define how the fabric is configured, monitored and managed.

What VXLAN and EVPN contribute

VXLAN provides an encapsulation mechanism that carries an inner Ethernet frame across an IP network. The outer IP packet travels through the underlay, allowing logical Layer 2 segments to span multiple switches without requiring the same physical Layer 2 segment to extend across the entire fabric.

EVPN is a control-plane mechanism commonly used with VXLAN to advertise endpoint reachability and support MAC learning and IP-related information. It can help the fabric make forwarding decisions without relying solely on data-plane flooding and learning. Exact behaviour depends on the EVPN design, device capabilities and configuration.

Illustrative packet journey between two logical segments

1 · Source workload

A server sends traffic into its attached leaf.

2 · Overlay processing

The leaf applies the configured forwarding and encapsulation behaviour.

3 · Underlay forwarding

The outer IP packet is routed across the leaf–spine fabric.

4 · Destination leaf

The destination tunnel endpoint processes the encapsulated traffic.

5 · Destination workload

The decapsulated frame is forwarded according to the destination network and policy.

This is an illustrative forwarding sequence, not a claim that every packet must cross a spine. Traffic between endpoints on the same leaf may be forwarded locally, depending on the network and forwarding configuration. Traffic between different leaves typically traverses the routed fabric.

Why the underlay still matters

An overlay does not remove the need for a reliable underlay. Tunnel endpoints must have the IP reachability required to communicate, and the physical fabric must provide sufficient bandwidth and predictable forwarding. If underlay routes, interfaces or MTU settings are incorrect, overlay services can fail even when their logical configuration appears valid.

Reachability

Routing and next hops

Verify interface addressing, routing adjacencies, route installation and reachability between relevant tunnel endpoints.

Encapsulation

MTU and packet handling

Account for encapsulation overhead and verify that the path supports the required packet sizes without unintended drops.

Forwarding

ECMP and capacity

Check usable next hops, traffic distribution, link utilisation and whether surviving paths can support the workload.

Where Cisco ACI fits

Cisco Application Centric Infrastructure (ACI) uses a spine–leaf fabric with a policy-driven operational model. In a typical ACI deployment, the Application Policy Infrastructure Controller (APIC) provides centralised policy and fabric management. The fabric switches enforce the configured policy and perform the forwarding functions supported by the platform.

ACI is not simply another name for VXLAN/EVPN. It is a broader architecture with its own policy model, management components and control mechanisms. By contrast, a conventional spine–leaf network can be assembled using routed switching and separately selected overlay and automation technologies.

Design area	Routed spine–leaf with VXLAN/EVPN	Cisco ACI
Architecture	A routed fabric with an overlay and control-plane choices selected for the deployment.	An integrated fabric architecture with a policy-driven model.
Configuration model	Commonly uses switch configuration, automation and selected management or controller tools.	Uses ACI constructs and APIC-based policy and fabric management.
Segmentation	Implemented through the selected overlay, routing and policy design.	Implemented through ACI policy constructs and supported fabric features.
Operational considerations	Requires a consistent design and operational model across the selected protocols and tools.	Requires understanding of ACI policy, fabric operation, controller health and platform-specific behaviour.

Neither approach is universally better. Selection depends on operational skills, policy requirements, interoperability, automation, platform constraints, lifecycle costs and the organisation's preferred way of managing change.

Engineering takeaway

Keep the roles clear: the physical fabric provides connectivity, the routed underlay provides IP reachability, and an overlay such as VXLAN/EVPN can provide logical network services. Cisco ACI packages fabric operation and policy into a broader architecture. Whichever model you choose, validate routing, encapsulation, MTU, segmentation and failure behaviour as one end-to-end system.

Interactive Overlay Lab · 05

Trace a Packet Through Underlay and Overlay

A modern data-centre fabric can separate IP transport from tenant connectivity. Explore the routed underlay, the VXLAN data plane and the EVPN control plane, then step through an illustrative packet journey between endpoints connected to different leaf switches.

Routed underlay VXLAN overlay EVPN control plane

Transport

Routed underlay

Provides IP reachability between VTEP addresses across the leaf–spine fabric. Routing and ECMP select available transport paths.

Data plane

VXLAN overlay

Encapsulates an inner Ethernet frame in an outer IP/UDP/VXLAN packet, carrying a VNI to identify the overlay segment.

Reachability

EVPN control plane

Uses BGP EVPN routes to distribute MAC and, where applicable, IP reachability information so VTEPs can learn remote endpoints.

Step 1 of 4 · Packet journey

VXLAN data plane

Encapsulate at the source VTEP

Leaf 1 receives Endpoint A's Ethernet frame, identifies the tenant segment and encapsulates the frame in VXLAN for the destination VTEP on Leaf 2.

Inner source: 10.10.1.10

Inner destination: 10.10.1.20

VNI: 10010

Outer destination: 192.0.2.12

Conceptual example only. The IP addresses and VNI are illustrative. Exact EVPN route types, gateway placement, ARP/ND handling, multicast or ingress-replication behaviour and policy depend on the implementation. Cisco ACI is a distinct policy-driven fabric architecture; it should not be treated as simply another name for a generic VXLAN/EVPN deployment.

Section 06 · Operations and Observability

Telemetry, Analytics and Troubleshooting a Spine–Leaf Fabric

A spine–leaf fabric can have many equal-cost paths, high aggregate bandwidth and redundant links, yet applications can still suffer from packet loss, latency or intermittent connectivity. Effective operations combine physical interface health, routing state, forwarding behaviour and application-level evidence to locate the actual fault rather than treating every symptom as a switch problem.

Build visibility across the fabric

No single monitoring source explains every network problem. Interface counters can reveal physical errors and congestion; routing information shows which paths are eligible; flow telemetry helps explain where traffic is going; and application measurements show whether the network is meeting service requirements. Correlating these sources provides a more reliable picture than examining isolated counters or alerts.

Physical layer

Interfaces and optics

Monitor link state, CRC errors, discards, interface resets, transceiver alarms and changes in error counters over time.

Control plane

Routing and neighbours

Check routing adjacencies, installed routes, next-hop changes, ECMP membership and evidence of repeated convergence.

Traffic behaviour

Flows and utilisation

Identify heavy flows, asymmetric traffic patterns, sustained utilisation, bursts, queue drops and uneven path loading.

Service layer

Latency and reachability

Correlate network events with application latency, packet loss, connection failures and the timing of user-visible incidents.

Know what each monitoring source can tell you

Source	Useful evidence	Important limitation
SNMP or interface polling	Interface status, byte and packet counters, errors and discards.	Polling intervals may miss short bursts; counter changes need context.
Streaming telemetry	Frequent structured measurements of supported interface, queue, routing or device state.	Available metrics and sampling behaviour vary by platform and configuration.
Flow records	Traffic conversations, source and destination patterns, protocol usage and traffic-volume estimates.	Sampling and export configuration affect what is visible; flow records do not necessarily explain every packet drop.
Routing and forwarding tables	Selected routes, next hops, ECMP paths and forwarding entries.	A valid route does not prove that the physical path is healthy or that application traffic is performing well.
Active probes and application monitoring	Reachability, round-trip time, loss and application response behaviour.	Results depend on probe placement, test traffic and the path taken by the measured flow.

A systematic troubleshooting workflow

Start with the observed symptom and narrow the scope. Determine which endpoints, applications, leaves and paths are affected. Establish when the issue began, compare current state with a known-good baseline, and look for changes that match the incident timeline.

Define the impact

Identify affected source and destination endpoints, the application or service, whether the issue is constant or intermittent, and whether it affects one leaf or multiple leaves.

Check physical health

Inspect interface state, error counters, discards, optics, cabling and queue statistics along the likely path. Compare counter changes over a consistent time interval.

Verify routing and ECMP

Confirm reachability and installed routes. Check whether the expected next hops are present, whether adjacencies are stable and whether a recent failure changed path selection.

Examine traffic and congestion

Compare utilisation across relevant uplinks, inspect available queue and drop metrics, and look for large flows or uneven ECMP distribution. High average utilisation is not the only possible cause of congestion.

Correlate with application evidence

Compare network events with application latency, retransmissions, connection errors and probe results. Check endpoint, host and service health before concluding that the fabric is responsible.

Verify the fix

Repeat the same tests, confirm counters stop increasing abnormally, validate the intended paths and check application performance. Record the cause and the evidence that demonstrated recovery.

Example: diagnosing intermittent application latency

Imagine that an application communicates between servers connected to different leaf switches. Users report occasional delays, but the links remain up and routing adjacencies appear stable. The correct response is to gather evidence across several layers rather than assuming that the fabric has sufficient capacity because no interface is continuously saturated.

Observation	Possible interpretation	Next check
Interface errors increase	A physical link, optic, cable or interface issue may be present.	Identify the affected interface and inspect physical diagnostics and error-counter changes.
Queue drops rise during bursts	Short periods of congestion may be affecting latency or packet delivery.	Correlate queue statistics with traffic bursts and the affected flows.
One uplink is much busier than others	ECMP hashing or a small number of large flows may create uneven utilisation.	Inspect available flow data, hash behaviour and per-link counters.
Routes change near the incident time	A link event or routing instability may have altered the forwarding path.	Review logs, adjacency events, route changes and convergence timing.
Fabric indicators appear normal	The cause may lie at the host, virtual switch, application or another dependency.	Compare endpoint metrics, service logs, packet captures and independent probes.

Build useful baselines and alerts

Alerting is more effective when it combines thresholds with trends and context. A brief utilisation spike may be expected, while a sustained rise in discards or recurring routing changes may deserve immediate investigation. Establish baselines for normal and peak periods, and tune alert thresholds to the platform and service requirements.

Useful operational dashboards bring together topology, interface health, link utilisation, routing state, device events and service performance. Time synchronisation across devices and monitoring systems is essential when correlating an incident. Retain enough historical data to compare the period before, during and after the fault.

Engineering takeaway

Troubleshoot from evidence, not assumptions. Correlate physical health, routing and ECMP state, traffic distribution, congestion indicators and application symptoms. The aim is to identify the fault domain, demonstrate the cause with observable data, and verify that the corrective action restores the required service.

Interactive Troubleshooting Lab · 06

Diagnose a Spine–Leaf Fabric Incident

Choose a symptom, inspect the simulated telemetry and work through the diagnostic sequence. The objective is to identify evidence that supports a hypothesis before changing the network.

Select an incident scenario

Incident summary

Investigation required

Application response times rise during busy periods. Determine whether congestion, physical errors or another fault is supported by evidence.

Simulated telemetry snapshot

Utilisation

92%

Busy uplink

CRC errors

No increase observed

Path state

Routes present

Normal Warning / investigate Suspected fault Verified / recovered

Diagnostic workflow · Select a step

Investigation focus

Establish the scope before changing anything

Identify which applications, source and destination endpoints, paths and time windows are affected. Compare the symptoms with a known-good baseline.

01Record the first observed failure and its duration.

02Check whether the issue affects one flow or many.

Training simulation: all metrics and fault indicators are illustrative, not live device telemetry. High utilisation alone does not prove congestion; correlate it with drops, queue behaviour, latency and the affected traffic. A route being present does not prove the complete data path is healthy.

Section 07 · Design and Implementation

Spine–Leaf Design Decisions and Implementation Checklist

A successful spine–leaf deployment starts with workload requirements, not a switch count or a preferred topology diagram. The design should establish the required connectivity, performance, resilience and operational model before hardware, routing protocols and overlays are selected. Each decision should be validated against the services the fabric must support.

Start with requirements, not assumptions

Document how many endpoints must connect, where workloads communicate, which applications are sensitive to latency, and how much traffic is expected to cross between leaves. Include planned growth, maintenance requirements, failure scenarios and operational constraints. These inputs determine the appropriate topology, link speeds, oversubscription target and expansion strategy.

Connectivity

Endpoint and port needs

Estimate server counts, port speeds, dual-homing requirements, network segments and the number of leaf switches needed.

Performance

Traffic and latency

Understand east–west traffic, peak demand, workload locality, latency targets and expected flow distribution.

Resilience

Failure requirements

Define which failures the fabric must tolerate and what performance is required while operating in a degraded state.

Operations

Management and skills

Plan configuration, automation, monitoring, troubleshooting, maintenance windows and team expertise.

Work through the design in a controlled sequence

Choose the physical topology

Determine the number of leaves and spines, required endpoint density, cabling paths and supported link speeds. Check platform port limits and expansion constraints before finalising the topology.

Define the routed underlay

Select the addressing plan, routing protocol, link addressing, route policy and failure-detection approach. Keep the design consistent and document how routes should behave during faults.

Set capacity and oversubscription targets

Calculate aggregate server-facing bandwidth and uplink capacity. Test expected peak traffic, account for ECMP behaviour, and verify remaining capacity after relevant link or spine failures.

Select segmentation and overlay services

Decide whether routed connectivity alone meets requirements or whether VXLAN/EVPN, another overlay or a policy-driven fabric architecture is needed. Document Layer 2 and Layer 3 service boundaries and security controls.

Plan endpoint and gateway redundancy

Define server attachment, gateway placement, link aggregation or other supported redundancy mechanisms. Validate the complete design rather than assuming two connections guarantee continuity.

Build observability into the deployment

Establish telemetry, interface and queue monitoring, routing event collection, flow visibility, time synchronisation and application-level performance baselines.

Validate before production rollout

Test connectivity, expected forwarding paths, policy enforcement, failure recovery, degraded capacity and operational procedures. Record the results and resolve gaps before expanding the deployment.

Implementation readiness checklist

Use the following checklist during design review and pre-production validation. Each item should have supporting evidence, such as a calculation, configuration review, test result or documented decision.

✓

Topology and physical capacity Leaf and spine counts, port density, link speeds, cabling, power and expansion limits have been reviewed.

✓

Routing and reachability Addressing, routing adjacencies, route selection, ECMP membership and intended forwarding behaviour are documented.

✓

Capacity and degraded operation Oversubscription calculations reflect expected workloads, and surviving capacity has been evaluated for relevant failures.

✓

Segmentation and security Network boundaries, routing policy, access controls and overlay behaviour meet the intended service requirements.

✓

Endpoint and gateway resilience Host connectivity, gateway availability and any link-aggregation or multi-chassis mechanisms have been tested for the intended failures.

✓

Monitoring and alerting Interface health, routing changes, traffic distribution, congestion and application symptoms can be correlated during an incident.

✓

Failure and recovery tests Link and device failures have been tested in a controlled environment, including convergence, packet loss and service recovery.

✓

Operational ownership Configuration standards, change control, escalation paths, maintenance procedures and recovery documentation are available.

Common design mistakes to avoid

Assumption	Why it can be misleading	Better approach
More spines always solve capacity problems	Endpoint links, leaf uplinks, hardware limits or traffic distribution may remain the bottleneck.	Identify the actual constraint and recalculate end-to-end capacity.
ECMP guarantees equal utilisation	Hashing and flow-size differences can create uneven traffic distribution.	Inspect per-link counters and available flow telemetry under realistic loads.
Redundancy guarantees uninterrupted service	Convergence, shared dependencies, endpoint design and reduced capacity can still interrupt or degrade applications.	Define failure requirements and measure recovery through controlled tests.
An overlay removes underlay complexity	Tunnel endpoints still depend on routing, MTU, physical connectivity and correct forwarding state.	Troubleshoot underlay and overlay as separate but connected layers.
Average utilisation tells the whole story	Short bursts, queue congestion, packet drops or a few large flows can affect service even when averages look acceptable.	Combine time-series metrics, queue data, flow evidence and application measurements.

Plan for the full lifecycle

The fabric should be designed for more than its initial deployment. Establish how new leaves will be added, how capacity will be reassessed, how software and hardware will be maintained, and how configuration changes will be tested. Define measurable service targets and periodically review whether the fabric still meets them as workload patterns and business requirements change.

A scalable architecture is one that can grow predictably without introducing unmanaged complexity. That requires consistent configuration, clear ownership, repeatable validation and operational evidence—not just a topology that looks symmetrical.

Engineering takeaway

Design from workload requirements, calculate capacity explicitly, separate underlay and overlay responsibilities, and test failure behaviour before production. A robust spine–leaf fabric balances performance, resilience, operational simplicity and the cost of future growth.

Interactive Design Workshop · 07

Build a Spine–Leaf Design Readiness Plan

Configure a sample fabric, review its nominal bandwidth and consider what happens when an uplink fails. Then complete the readiness checklist to assess whether the design has been specified, tested and documented.

1. Configure the fabric

Server-facing ports per leaf 48

Server port speed 25 Gb/s

Uplinks per leaf 4

Uplink speed 100 Gb/s

Spines in the fabric 2

The drawing shows up to five representative spines. The configured spine count is used in the design inputs, not as proof of available switch capacity.

Traffic profile

2. Review the design

Aggregate downlink

1.20 Tb/s

Per leaf, nominal line rate

Aggregate uplink

400 Gb/s

Per leaf, nominal line rate

Oversubscription

3.00:1

Downlink capacity : uplink capacity

After one uplink fails

4.00:1

Remaining uplink line rate: 300 Gb/s

Uplink capacity relative to downlinks 33%

3. Implementation-readiness checklist

0 of 8 complete

Traffic profile, workload growth and performance targets are documented. Oversubscription is assessed against expected traffic and busy-hour demand. Link, switch and port capacity are verified for the intended topology. Failure scenarios are tested with enough surviving capacity. Underlay routing, ECMP and convergence behaviour are validated. Segmentation, gateway redundancy and endpoint attachment are defined. Telemetry, alert thresholds and troubleshooting access are prepared. Rollback, acceptance tests and operational documentation are ready.

Design review result

Design review not yet complete

Use the capacity calculation as an initial check, complete the readiness items and validate the design against workload and failure requirements.

Capacity model assumptions: nominal port line rates are compared in one direction and are not doubled for full duplex. The model does not account for protocol overhead, internal switch fabric limits, traffic locality or real flow distributions. After one uplink fails, one uplink's capacity is removed. The topology is illustrative and does not prove that the selected switches have sufficient ports, bandwidth or forwarding capacity. Validate the physical design and required failure scenarios before deployment.

```

☕

Support Network Insight

Help support independent engineering guides, interactive networking tools, and future learning labs.

Support Network Insight ```

What is FabricPath

August 31, 2014

by Matt Conran Blog

What is Fabric Path

In today's digital era, businesses rely heavily on networking infrastructure to ensure seamless communication and efficient data transfer. Cisco FabricPath is a cutting-edge technology that provides a scalable and resilient solution for modern network architectures. In this blog post, we will delve into the intricacies of Cisco FabricPath, exploring its features, benefits, and use cases.

Cisco FabricPath is a comprehensive network virtualization technology designed to address the limitations of traditional Ethernet networks. It offers a flexible and scalable approach for building large-scale networks that can handle the increasing demands of modern data centers. By combining the benefits of Layer 2 simplicity with Layer 3 scalability, Cisco FabricPath provides a robust and efficient solution for building high-performance networks.

Cisco Fabric Path is a networking technology that provides a scalable and flexible solution for data center networks. It leverages the benefits of both Layer 2 and Layer 3 protocols, combining the best of both worlds to create a robust and efficient network infrastructure.

Simplified Network Design: One of the key advantages of Cisco Fabric Path is its ability to simplify network design and reduce complexity. By utilizing a loop-free topology and eliminating the need for complex Spanning Tree Protocol (STP) configurations, Fabric Path streamlines the network architecture and improves overall efficiency.

Increased Scalability: Scalability is a crucial aspect of any modern network infrastructure, and Cisco Fabric Path excels in this area. With its support for large Layer 2 domains and the ability to accommodate thousands of VLANs, Fabric Path provides organizations with the flexibility to scale their networks as per their evolving needs.

Enhanced Traffic Load Balancing: Cisco Fabric Path incorporates Equal-Cost Multipath (ECMP) routing, which enables efficient distribution of traffic across multiple paths. This results in improved performance, reduced congestion, and enhanced load balancing capabilities within the network.

Converged Traffic and Virtualization: Fabric Path allows for the convergence of both Ethernet and Fibre Channel traffic onto a single infrastructure, simplifying management and reducing costs. Additionally, it seamlessly integrates with virtualization technologies, enabling organizations to leverage the benefits of virtual environments without compromising on performance or security.

Matt Conran

Highlights: What is Fabric Path

Nexus OS Software Release

Introduced by Cisco in Nexus OS software Release 5.1(3), FabricPath Nexus allows architects to design highly scalable actual Layer 2 fabrics. Similar to the spanning tree, it provides an almost plug-and-play deployment model with the benefits of Layer 3 routing, allowing FabricPath networks to scale at an unprecedented level.

In addition to its simplicity, Fabric Path enables faster, simpler, and flatter data center networks. Cisco FabricPath uses routing principles to allow Layer 2 scaling, bringing the stability of Layer 3 routing to Layer 2. Fabric path traffic is no longer forwarded along a spanning tree design. As a result, we now have a more scalable design that is not limited by bisectional bandwidth.

**Understanding the Core Concepts of FabricPath**

At its core, FabricPath combines the best aspects of Layer 2 and Layer 3 networking, creating a unified fabric that simplifies data center operations. Unlike traditional Spanning Tree Protocol (STP), FabricPath employs a routing protocol that ensures optimal path selection and loop prevention. This results in a more efficient and robust network design, capable of handling the demands of modern applications. Key to this is the use of Intermediate System to Intermediate System (IS-IS) protocol, which facilitates dynamic path calculation and enhances network resilience.

**Advantages of Implementing FabricPath**

Cisco FabricPath offers a multitude of benefits that address common network challenges. One of its primary advantages is the elimination of STP-related issues, such as blocked links and inefficient bandwidth utilization. With FabricPath, all available paths are active, leading to increased throughput and reduced latency. Additionally, the protocol supports seamless scalability, allowing networks to grow without significant reconfiguration. This makes it an ideal choice for businesses looking to future-proof their infrastructure.

**Deploying FabricPath in Your Network**

Deploying Cisco FabricPath in your network involves several considerations to ensure optimal performance. First, it’s crucial to assess your current network architecture and identify areas that would benefit from FabricPath integration. Once identified, the implementation process involves configuring FabricPath on compatible Cisco Nexus switches and ensuring proper interoperability with existing network protocols. Proper planning and execution are essential to reap the full benefits of this innovative technology.

Data Analytics and FabricPath

Data analytics is the visibility engine that reveals how FabricPath behaves as a dynamic Layer‑2 routing system rather than a traditional spanning‑tree‑based topology. FabricPath generates rich telemetry at every stage of its operation: IS‑IS control‑plane updates, switch‑ID reachability, multidestination tree selection, load‑balancing behavior across equal‑cost paths, and forwarding‑plane transitions during link or node failures. Analytics platforms ingest this telemetry to expose how FabricPath actually performs under real workloads.

They highlight issues that classic monitoring often misses—imbalanced traffic across FabricPath ECMP paths, overloaded multidestination trees, noisy switch‑IDs causing control‑plane churn, or slow reconvergence triggered by unstable adjacencies. By correlating FabricPath behavior with application performance and underlay health, analytics transforms the FabricPath fabric from a conceptual Layer‑2 routing domain into a fully observable, measurable system.

When analytics is integrated directly with the FabricPath control plane, the fabric becomes adaptive rather than static. Real‑time analytics feeds controllers and operations systems with precise state: IS‑IS SPF pressure, switch‑ID stability, tree‑root load, and per‑path forwarding performance. With this intelligence, the network can rebalance traffic across FabricPath ECMP paths, adjust tree‑root selection, dampen unstable nodes, or pre‑emptively reroute flows away from degraded segments of the fabric.

Analytics also validates multidestination tree consistency, detects asymmetric forwarding between FabricPath domains, and predicts when control‑plane churn is about to trigger instability. The result is a FabricPath environment that learns from telemetry, optimizes itself continuously, and maintains predictable performance even during failures or heavy east‑west traffic bursts. Data analytics transforms FabricPath into a self‑optimizing Layer‑2 routing fabric.

FabricPatch as the Unified Fabric Abstraction Layer

FabricPatch is a modern abstraction layer that models the entire data‑center fabric — underlay, overlay, control‑plane, and service‑plane — as a single programmable patch. Instead of treating routing, switching, VXLAN, EVPN, and service chaining as separate systems, FabricPatch unifies them into a single logical fabric object. Each patch represents a slice of the network: its nodes, links, tunnels, VNIs, EVPN routes, routing adjacencies, and service chains. Your simulator models FabricPatch as structured JavaScript objects describing patch boundaries, patch health, patch convergence, patch services, and patch dependencies. This transforms FabricPatch from a conceptual SDN idea into a dynamic, testable system that reacts to failures, load, mobility, and control‑plane churn in real time.

The FabricPatch lab uses a multi‑engine pipeline that evaluates every dimension of the fabric. A Patch‑Topology engine validates node membership, link consistency, and patch boundaries. A Patch‑Routing engine measures SPF/BGP/IS‑IS convergence inside the patch. A Patch‑Overlay engine evaluates VXLAN/EVPN behavior, VTEP health, VNI segmentation, and MAC mobility. A Patch‑Service engine checks service‑chain correctness (firewalls, load balancers, telemetry nodes). A Patch‑Flow engine analyzes ECMP distribution, tunnel load, and per‑patch traffic flow. A Patch‑Stability engine evaluates flap frequency, dampening, and control‑plane churn. Each engine produces severity, timing, and root‑cause signals that feed into your multi‑page cinematic CLI, creating a complete visual of how the fabric behaves under stress.

To make FabricPatch intuitive, the lab uses every visual technique from your previous labs — and adds new ones. The topology page shows FabricPatch maps, where each patch is drawn as a shaded ASCII block containing nodes, tunnels, and service chains. The engines page uses bordered tables with shaded rows to display patch‑routing, patch‑overlay, patch‑service, and patch‑stability decisions. The failures page animates patch isolation, patch split‑brain, EVPN withdraw storms, service‑chain breaks, and underlay reconvergence using timestamped timelines. The traffic page shows PatchFlow diagrams, where VXLAN encapsulated flows move across patch boundaries with animated load bars. The dashboard page displays color‑coded widgets for patch health, patch convergence time, patch service‑chain latency, patch mobility events, and patch loss percentage.

Key FabricPath Notes:

– Understanding the Basics: Fabric Path is a layer 2 network technology that provides the advantages of traditional Ethernet and routing protocols. By leveraging multiprotocol label switching (MPLS) techniques, it enables the creation of scalable, efficient, and resilient networks. Fabric Path ensures optimal traffic flow and minimizes congestion by utilizing a loop-free topology and distributing forwarding information across the network.

– Key Features and Benefits: Fabric Path offers several compelling features, making it a highly desirable solution for modern network architectures. Firstly, its ability to support large Layer 2 domains without the limitations of Spanning Tree Protocol (STP) enables efficient utilization of network resources.

Additionally, Fabric Path provides increased bandwidth and redundancy, leading to enhanced performance and reliability. With its support for Equal-Cost Multipath (ECMP) load balancing, the technology allows for efficient distribution of traffic across multiple paths, further optimizing network utilization.

– Seamless Migration and Interoperability: One of the significant advantages of Fabric Path is its seamless integration with existing network infrastructures. It enables organizations to migrate from conventional Ethernet-based networks to Fabric Path gradually, without disrupting their ongoing operations. This interoperability ensures a smooth transition and allows businesses to leverage the benefits of Fabric Path without compromising their existing investments.

– Real-World Applications: Fabric Path has found extensive applications in various industries and network environments. In data centers, it provides a highly scalable and flexible solution for building large Layer 2 domains, enabling seamless virtual machine mobility and workload distribution.

Similarly, Fabric Path offers simplified network design, reduced complexity, and improved performance in campus networks. Furthermore, it has proven to be an effective solution for service providers, facilitating the delivery of bandwidth-intensive services with high availability and resiliency.

The need for layer 2

In the past, data centers were designed primarily to provide high availability. Layer 2 is crucial for modern data centers. Today’s networks must be agile and flexible, just like the organizations they serve since switching allows devices to be moved and infrastructure to be modified transparently, expanding the Layer 2 domain would satisfy this additional requirement. On the other hand, existing switching technologies rely on inefficient forwarding schemes based on spanning trees that cannot be extended to the entire network. The flexibility of Layer 2 compromises the scalability of Layer 3.

Routing concepts at layer 2

Cisco FabricPath: Expanding Routing Concepts to Layer 2 Cisco® FabricPath extends routing stability and scalability to Layer 2 of Cisco NX-OS. Workloads can be moved across the entire data center by eliminating the need to segment the switched domain. As a result, the bisectional bandwidth of the network is no longer limited by a spanning tree, allowing for massive scalability.

An entirely new Layer 2 data plane is created with Cisco FabricPath, where frames enter the fabric with routable source and destination addresses. The source address of a frame is its receiving switch’s address, and its destination address is its destination switch’s address. As soon as the frame reaches the remote switch, it is de-encapsulated and delivered in its original Ethernet format.

The role of Fabric Path

In large data centers, virtualization of physical servers began a few years ago. Due to server virtualization and economies of scale, “mega data centers” containing tens of thousands of servers emerged due to server virtualization.

As a result, distributed applications had to be supported on a large scale and provisioned in different data center zones. A scalable and resilient Layer 2 fabric was required to enable any-to-any communication. FabricPath was developed by Cisco to meet these new demands. Providing scalability, resilience, and flexibility, FabricPath is a highly scalable Layer 2 fabric.

Fabric Path Requirements

– Massive Scalable Data Centers (MSDCs) and virtualization technologies have led to the development of large Layer 2 domains in data centers with more than 1000 servers and a design for scalability. Due to the limitations of Spanning Tree Protocol (STP), Layer 2 switching has evolved into technologies such as TRILL and FabricPath. To understand FabricPath’s limitations, you need to consider the limitations of current Layer 2 networks based on STP:

– By blocking redundant paths, STP creates loop-free topologies in Layer 2 networks. STP uses the root selection process to accomplish this. To build shortest paths to the root switch, all the other switches block the other ports while building shortest paths to the root switch. The result is a Layer 2 network topology that is loop-free.

– This blocks layer 2 networks because all redundant paths are blocked. PVST, which enables per-VLAN load balancing, also has limitations in multipathing support, although some enhancements were made using the per-VLAN Spanning Tree Protocol (PVSTP).

– The root bridge is selected based on the shortest path, which results in inefficient path selection between switches. So, selecting a path between switches doesn’t necessarily mean choosing the shortest path. Take two access switches as an example connected to distribution and each other. If the distribution switch serves as the root bridge for STP, the link between the two access switches is blocked. All traffic flows through the distribution switch.

– Unavailability of Time-To-Live (TTL): The Layer 2 packet header doesn’t have a TTL field. This can lead to network meltdowns in switched networks. This is because a forwarding loop can cause a broadcast packet to duplicate, consuming excessive network resources exponentially.

MAC address scalability: Nonhierarchical flat addressing of Ethernet MAC addresses leads to limited scalability since MAC address summarization is impossible. Additionally, all the MAC addresses are essentially assigned to every switch in the Layer 2 network, increasing the size of Layer 2 tables.

As a result, Layer 3 routing protocols provide multipathing and efficient shortest paths between all nodes in the network, which resolve the shortcomings of Layer 2 networks. Layer 3 solves these issues, but the network design becomes static. Static network design limits Layer 2 domain size, so virtualization cannot be used. Thanks to FabricPath’s combination of the two technologies, a Layer 2 network can be flexible and scaled with Layer 3 networks.

Fabric Path Benefits

With FabricPath, data center architects and administrators can design and implement scalable Layer 2 fabrics. Benefits of FabricPath include:

Maintains the plug-and-play features of classical Ethernet: Due to the minimal configuration requirements and the fact that the administrator must include the FabricPath core network interfaces, configuration effort is significantly reduced.

Unicast forwarding, multicast forwarding, and VLAN pruning are also controlled by a single protocol (IS-IS). FabricPath operations, administration, and management (OAM) now support ping and trace routes, allowing network administrators to troubleshoot Layer 2 FabricPath networks similarly to Layer 3 networks.

Multipathing

Multipathing allows data center network architects to build large, scalable networks using N-way (more than one path) multipathing. A network administrator can also incrementally add new devices as needed to the existing topology. Using flat topologies, MSDC networks can be connected by only one hop between nodes. A single node failure in N-way multipathing results in a reduction in fabric bandwidth of 1/Nth.

With the enhanced Layer 2 network capabilities combined with Layer 3 capabilities, multiple paths can be created between endpoints instead of just one, replacing STP. It allows network administrators to increase bandwidth as bandwidth requirements increase incrementally.The FabricPath protocol enables traffic to be forwarded over the shortest path to the destination, reducing network latency. This is more efficient than Layer 2 forwarding based on STP.

With FabricPath, MAC addresses are learned selectively based on active flows with conversational MAC learning. As a result, the need for large MAC tables is reduced.

Related: Before you proceed, you may find the following posts helpful:

What is Fabric Path

Scalable Layer 2

We must support distributed applications at a considerable scale and have the flexibility to provision them in different zones of data center topologies. This necessitated creating a scalable and resilient Layer 2 fabric enabling any-to-any communication without workload placement restrictions—Cisco developed FabricPath to meet these new demands.

FabricPath is a powerful network technology from Cisco Systems that provides a unified, programmable fabric to connect, manage, and optimize data center networks. It is based on a distributed Layer 2 network protocol that enables the creation of multi-tenant, multi-domain, and multi-site networks with a single, unified control plane. FabricPath operates on a flat, non-hierarchical topology designed to simplify network virtualization and automation.

FabricPath delivers a highly scalable Layer 2 fabric

FabricPath delivers a highly scalable Layer 2 fabric. It uses a single control protocol (IS-IS) for unicast forwarding, multicast forwarding, and VLAN pruning. FabricPath also enables traffic to be forwarded across the shortest path to the destination, thus reducing latency in the Layer 2 network. This is more efficient than Layer 2 forwarding based on the STP.

FabricPath includes several features that make it ideal for large enterprise networks and data centers. It uses a distributed control plane to provide a unified view of the network and reduce network complexity. In addition, FabricPath supports virtualization, allowing the creation of multiple virtual networks within the same physical infrastructure. It also allows the creation of multiple forwarding instances and provides fast convergence times.

The Challenges Of Inefficient Forwarding Schemes

1: The challenge is that existing switching technologies have inefficient IP forwarding schemes based on spanning trees and cannot be extended to the network. Therefore, current designs compromise the flexibility of Layer 2 and the scaling offered by Layer 3. On the other hand, Fabric Path introduces a new method of forwarding.

2: The data design can stay the same as a leaf and spine. Still, we have a new Layer 2 data plane with fabric paths that encapsulate the frames entering the fabric with a header consisting of routable source and destination addresses.

3: These addresses are the address of the switch on which the frame was received and the address of the destination switch to which the frame is heading. From there, the frame is routed until it reaches the remote switch, where it is de-encapsulated and delivered in its original Ethernet format. FabricPath Nexus also uses a Shortest Path First (SPF) routing protocol to determine reachability and path selection in the FabricPath domain.

4: With Fabric Path, we have a simple and flexible behavior of Layer 2 while using the routing mechanisms that make IP reliable and scalable. So you may ask, what about the Layer 2 and 3 boundaries? The Layer 2 and 3 boundary still exists in a data center based on Cisco FabricPath. However, there is little difference in how traffic is forwarded in those two distinct areas of the network. The following sections discuss the drivers for FabricPath and what you may opt for in its design.

Why Cisco Fabricpath?

1) No Multipathing support at Layer 2: Spanning Tree Protocol ( STP ) lacks any good Layer 2 multipathing features for large data centers. The protocol has been enhanced with PVST per VLAN load balancing, but this feature can only load balance on VLANs.

2) MAC address scalability: Layer 2 end hosts are discovered by their MAC address, and this type of host addressing cannot be hierarchical and summarized. For example, one MAC address cannot represent a stub of networks. Traditional Layer 3 networks overcome this by introducing ABRs in OSPF or summarization/filtering in EIGRP. Also, in the Layer 2 network, all the MAC addresses are populated in ALL switches, leading to large requirements in the Layer 2 table sizes.

3) Instability of Layer 2 networks: Layer 3 networks have an eight-bit Time to Live ( TTL ) field that prevents datagrams from persisting (e.g., going in circles ) on the internet. However, compared to Layer 3 headers, the Layer 2 packet header does not have a TTL field. The lack of a TTL field will cause Layer 2 packets to loop infinitely, causing a network meltdown.

4) Incompetent path selection: The shortest path for a Layer 2 network depends on the placement of the Root switch. Depending on costs and port priorities, you can influence the root port selection ( forwarding port ), but the root switch’s placement is how the forwarding path is built.

For example, in the diagram below, the most optimum traffic for the server-to-server flows would be via the inter-switch link, but as you can see, spanning tree blocks, this port, and traffic takes the sub-optimal path through the distribution switch.

Issues with Spanning Tree: Vendors’ responses.

A Spanning Tree allows only one path to be active between any two nodes and blocks the rest, making it unsuitable for low-latency data centers and cloud environments. Every vendor addressing the data center market proposes augmenting or replacing a Spanning Tree with a link-state protocol.

For example, Brocade uses TRILL in the data plane, while the control plane is based on Fabric Shortest Path First, an ANSI standard used by all Fibre Channel SAN fabrics as the link-state routing protocol.

On the other hand, Juniper implemented a tagging mechanism in the Broadcom silicon in its QFabric switches rather than a link-state protocol. Cisco FabricPath is considered a “superset” of TRILL, bringing scale to the data center and improving application performance.

Fabric Path typical use cases

Fabric Path can support any new protocol that can be done elegantly in IS-IS by adding new extensions without modifying the base infrastructure. Each IS-IS Intermediate router advertises one or more IS-IS Link State Protocol Data Units (LSPs) with routing information.

The LSP comprises a fixed header and several tuples, each consisting of a Type, a Length, and a Value. Such tuples are commonly known as TLVs and are a good way of encoding information in a flexible and extensible format. These make IS-IS a very extensible routing protocol, and FabricPath takes advantage of this extensibility.

This allows FarbicPath to support the following prominent use cases.

Large flat data centers that need Layer 2 multipathing and equidistant endpoints.
DC requires a reduction of Layer 2 table sizes ( done via MAC conversational learning ).

Cisco FabricPath control plane

FabricPath is a Layer 2 overlay network with an IS-IS control plane. Using FabricPath IS-IS, the switches build their forwarding tables, similar to building the forwarding table in Layer 3 networks. The extensions used in IS-IS to support Fabricpath allow this Layer 2 overlay to take advantage of all the scalable and load balancing ( ECMP, up to 16 routes ) benefits of a Layer 3 network while retaining the benefits of a plug-and-play Layer 2 network.

The FabricPath Header

The FabricPath header has a hop count in one of the fields, which mitigates temporary loops in FabricPath networks. This header uses locally assigned hierarchical MAC addresses for forwarding frames within the network. The original Layer 2 frames are encapsulated with a FabricPath header, and a new CRC is appended to the existing packet. One of the main elements of the FabricPath header is the SwitchID, and the core switches forward Fabricpath traffic by examining this field. The switch ID is the field used in the FabricPath domain to forward packets to the correct destination switch.

Why use IS-IS as the FabricPath Nexus control plane?

We touched on this just a moment ago. Its control protocol is built on top of the Intermediate System–to Intermediate System (IS-IS) routing protocol, which provides fast convergence and has been proven to scale up to the largest service provider environments.

IS-IS is flexible and can be extended to support other functions with new type-length values (TLVs).
TLV is also known as tag-length value and encodes optional information.IS-IS runs directly over the link layer, thereby preventing the need for any underlying Layer 3 protocol like IP to work.

Virtual PortChannel

Fabricpath Nexus uses Virtual PortChannel. Now, we have multiple active link capabilities, resulting in active-active forwarding paths. The vPC allows a more granular design over standard port channeling, which only allows you to terminate on one switch. In addition, Cisco vPC enables a more flexible triangular design. Both aggregation technologies can use LACP for the control plane to negotiate the links.

Virtual Device Context

Fabricpath Nexus also uses Virtual Device Contexts (VDC), which allows each FabricPath control-plane protocol and functional block to run in its own protected memory space as individual processes for stability and fault isolation. A VDC design enables modular building blocks to improve security and performance.

FabricPath Nexus and conversational MAC learning

FabricPath Nexus performs conversational MAC learning, enabling a switch to learn only those MACs involved in active bidirectional communication. Similar to a three-way handshake, this new technique leads to the population of only the interested host’s MAC addresses rather than all MAC addresses in the domain. This dramatically reduces the need for large table sizes as each switch only learns the MAC addresses that the hosts under its interface are actively communicating with. As a result, edge nodes only know the MAC addresses of local nodes or nodes that want to speak with local nodes directly.

FabricPath Nexus benefits and drawbacks

Benefits	Drawbacks
Plug-and-play features like Classical Ethernet	Cisco proprietary
The single control plane for ALL types of traffic and good troubleshooting features to debug problems at Layer 2	Fabric interfaces carry only FabricPath encapsulated traffic
High performance and high availability using multipathing**	Useful as a DCI solution only over short distances
Easy to add new devices to an existing FabricPath domain	NA
Small Layer 2 table sizes result in better performance	NA

** This enables the MSDC networks to have flat topologies, separating the nodes by a single hop.

Although IS-IS forms the basis of Cisco FabricPath, you don’t need to be an IS-IS expert. You can enable FabricPath interfaces and begin forwarding FabricPath encapsulated frames in the same way they can activate Spanning Tree and interconnect switches.

The only necessary configuration is distinguishing the core ports, which link the switches, from the edge ports, where end devices are attached. No other parameters need to be tuned to achieve an optimal configuration, and the switch addresses are assigned automatically for you.

Closing Points on FabricPath

Cisco FabricPath is an innovative technology that combines the best of both Layer 2 and Layer 3 architectures. By leveraging the Intermediate System to Intermediate System (IS-IS) protocol, it simplifies network topology and enhances scalability. Unlike traditional spanning tree protocols, FabricPath eliminates loop issues and provides multipath capabilities, ensuring optimal data flow across the network. This section will explore how FabricPath works and why it’s a game-changer for modern networks.

One of the standout features of Cisco FabricPath is its ability to create a flat, scalable network fabric that supports large-scale data center environments. It offers seamless mobility, allowing for the efficient movement of virtual machines across the network without impacting performance. Additionally, FabricPath supports equal-cost multipathing (ECMP), which optimizes bandwidth usage and enhances network resiliency. This section will break down these features and explain how they contribute to a more efficient and reliable networking environment.

Successfully deploying Cisco FabricPath requires a strategic approach. This section will provide best practices for implementation, including considerations for network design, hardware compatibility, and performance optimization. We’ll also discuss common challenges and how to overcome them, ensuring a smooth transition to a FabricPath-enabled network. By following these guidelines, organizations can maximize the benefits of FabricPath and future-proof their network infrastructure.

Summary: What is Fabric Path

Key Features of Cisco FabricPath:

1. MAC-in-MAC Encapsulation: Cisco FabricPath utilizes MAC-in-MAC encapsulation to overcome the traditional Spanning Tree Protocol (STP) limitations. By encapsulating Layer 2 frames within another Layer 2 frame, FabricPath enables efficient forwarding and eliminates the need for STP.

2. Loop-Free Topology: Unlike STP-based networks, Cisco FabricPath employs a loop-free topology, ensuring optimal forwarding paths and maximizing network utilization. This feature enhances network resilience and eliminates the risk of network outages caused by loops.

3. Scalability: Cisco FabricPath supports up to 16 million virtual ports, enabling organizations to scale their networks without compromising performance. This scalability makes it ideal for large data centers and enterprises with growing network demands.

4. Traffic Optimization: Cisco FabricPath optimizes traffic flows using Equal-Cost Multipath (ECMP) routing. ECMP distributes traffic across multiple paths, allowing for efficient load balancing and improved network performance.

Benefits of Cisco FabricPath:

1. Simplified Network Design: Cisco FabricPath simplifies network design by eliminating the need for complex STP configurations. With its loop-free architecture, FabricPath reduces network complexity and improves overall network stability.

2. Enhanced Network Resilience: Cisco FabricPath ensures high availability and resilience by utilizing multiple paths and load-balancing techniques. In the event of a link failure, traffic is automatically rerouted, minimizing downtime and enhancing network reliability.

3. Increased Performance: With its scalable design and traffic optimization capabilities, Cisco FabricPath delivers superior network performance. By distributing traffic across multiple paths, FabricPath minimizes bottlenecks and improves overall network throughput.

Use Cases of Cisco FabricPath:

1. Data Center Networks: Cisco FabricPath is widely used in data center environments, providing a scalable and resilient networking solution. Its ability to handle high traffic volumes and optimize data flows makes it an ideal choice for modern data centers.

2. Virtualized Environments: Cisco FabricPath is particularly beneficial in virtualized environments. It simplifies network provisioning and enhances virtual machine mobility. Its scalability and flexibility enable seamless communication between virtualized resources.

Conclusion: Cisco FabricPath is a powerful networking solution that offers numerous benefits for organizations seeking scalable and resilient network architectures. With its loop-free topology, MAC-in-MAC encapsulation, and traffic optimization capabilities, Cisco FabricPath simplifies network design, enhances network resilience, and boosts overall performance. By implementing Cisco FabricPath, businesses can build robust and efficient networks that meet the demands of today’s digital landscape

What is VXLAN

August 28, 2014

by Matt Conran Blog

What is VXLAN

In the rapidly evolving networking world, virtualization has become critical for businesses seeking to optimize their IT infrastructure. One key technology that has emerged is VXLAN (Virtual Extensible LAN), which enables the creation of virtual networks independent of physical network infrastructure. In this blog post, we will delve into the concept of VXLAN, its benefits, and its role in network virtualization.

VXLAN is an encapsulation protocol designed to extend Layer 2 (Ethernet) networks over Layer 3 (IP) networks. It provides a scalable and flexible solution for creating virtualized networks, enabling seamless communication between virtual machines (VMs) and physical servers across different data centers or geographic regions.

VXLAN is a technology that creates virtual networks within an existing physical network. A Layer 2 overlay network runs on top of the current Layer 2 network. VXLAN utilizes UDP as the transport protocol, providing a secure, efficient, and reliable way to create a virtual network.

VXLAN encapsulates the original Layer 2 Ethernet frames within UDP packets, using a 24-bit VXLAN Network Identifier (VNI) to distinguish between different virtual networks. The encapsulated packets are then transmitted over the underlying IP network, enabling the creation of virtualized Layer 2 networks across Layer 3 boundaries.

Scalability: VXLAN solves the limitations of traditional VLANs by providing a much larger network identifier space, accommodating up to 16 million virtual networks. This scalability allows for the efficient isolation and segmentation of network traffic in highly virtualized environments.

VXLAN enables the decoupling of virtual and physical networks, providing the flexibility to move virtual machines across different physical hosts or even data centers without the need for reconfiguration. This flexibility greatly simplifies workload mobility and enhances overall network agility.

Multitenancy: With VXLAN, multiple tenants can securely share the same physical infrastructure while maintaining isolation between their virtual networks. This is achieved by assigning unique VNIs to each tenant, ensuring their traffic remains separate and secure.

Underlay Network: VXLAN relies on an IP underlay network, which must provide sufficient bandwidth, low latency, and optimal routing. Careful planning and design of the underlay network are crucial to ensure optimal VXLAN performance.

Network Virtualization Gateway: To enable communication between VXLAN-based virtual networks and traditional VLAN-based networks, a network virtualization gateway, such as a VXLAN Gateway or an overlay-to-underlay gateway, is required. These gateways bridge the gap between virtual and physical networks, facilitating seamless connectivity.

Matt Conran

Highlights: What is VXLAN

Understanding VXLAN Basics

It is essential to grasp VXLAN’s fundamental concepts to comprehend it. VXLAN enables the creation of virtualized Layer 2 networks over an existing Layer 3 infrastructure. It uses encapsulation techniques to extend Layer 2 segments over long distances, enabling flexible deployment of virtual machines across physical hosts and data centers.

VXLAN Encapsulation: One of the key components of VXLAN is encapsulation. When a virtual machine sends data across the network, VXLAN encapsulates the original Ethernet frame within a new UDP/IP packet. This encapsulated packet is then transmitted over the underlying Layer 3 network, allowing for seamless communication between virtual machines regardless of their physical location.

VXLAN Tunneling: VXLAN employs tunneling to transport the encapsulated packets between VXLAN-enabled devices. These devices, known as VXLAN Tunnel Endpoints (VTEPs), establish tunnels to carry VXLAN traffic. By leveraging tunneling protocols like Generic Routing Encapsulation (GRE) or Virtual Extensible LAN (VXLAN-GPE), VTEPs ensure the delivery of encapsulated packets across the network.

Benefits of VXLAN

VXLAN brings numerous benefits to modern network architectures. It enables network virtualization and multi-tenancy, allowing for the efficient and secure isolation of network segments. VXLAN also provides scalability, as it can support a significantly higher number of virtual networks than traditional VLAN-based networks. Additionally, VXLAN facilitates workload mobility and disaster recovery, making it an ideal choice for cloud environments.

Implementing VXLAN

VXLAN Implementation Considerations: While VXLAN offers immense advantages, there are a few considerations to consider when implementing it. VXLAN requires network devices that support the technology, including VTEPs and VXLAN-aware switches. It is also crucial to properly configure and manage the VXLAN overlay network to ensure optimal performance and security.

Data Analytics with VXLAN

Data analytics is the intelligence layer that exposes how VXLAN behaves across both the overlay and underlay, turning a complex encapsulation system into a measurable, predictable fabric. VXLAN generates rich telemetry at every stage of its operation: VTEP tunnel health, VNI segmentation behavior, MAC learning rates, EVPN route‑type propagation (RT‑2, RT‑3, RT‑5), underlay ECMP path usage, and encapsulation/decapsulation latency. Analytics platforms ingest this telemetry to reveal how VXLAN actually performs under real workloads. They highlight issues that traditional monitoring misses—imbalanced ECMP hashing across VXLAN tunnels, MAC mobility spikes, noisy VTEPs causing churn, or underlay reconvergence events that silently shift overlay paths. By correlating overlay behavior with underlay routing and application performance, analytics transforms VXLAN from a conceptual overlay into a fully observable system.

When analytics is integrated directly with the VXLAN control plane—especially EVPN—it enables the overlay to become adaptive rather than static. Real‑time analytics feeds controllers and orchestration systems with precise state: tunnel latency, EVPN update storms, MAC‑to‑VTEP mapping volatility, and underlay path health. With this intelligence, the network can rebalance traffic across VXLAN tunnels, dampen unstable MAC entries, adjust VTEP placement, or pre‑emptively reroute flows away from degraded underlay paths. In EVPN fabrics, analytics validates route‑type consistency, detects asymmetric propagation between peers, and predicts when MAC mobility is about to trigger instability. The result is a VXLAN environment that learns from telemetry, optimizes itself continuously, and maintains predictable performance even during failures or heavy east‑west traffic bursts. Data analytics transforms VXLAN into a self‑optimizing overlay fabric.

VXLAN as the Scalable Overlay for Modern Fabrics

VXLAN transforms the data center by decoupling tenant networks from physical topology, allowing millions of isolated segments to coexist across a routed underlay. Instead of stretching VLANs or relying on fragile Layer‑2 domains, VXLAN encapsulates Ethernet frames inside UDP packets and transports them across ECMP‑rich Layer‑3 fabrics. In your simulator, each VXLAN domain is represented as structured objects describing VTEPs, VNIs, tunnels, EVPN route types, MAC learning behavior, and underlay path diversity. This turns VXLAN from a conceptual overlay into a dynamic, testable system that reacts to topology changes, control‑plane events, and underlay convergence in real time.

The VXLAN lab uses a multi‑engine pipeline to evaluate how well the overlay behaves under load, churn, and failure. A VTEP engine measures tunnel health and encapsulation latency, a VNI engine evaluates segmentation and MAC learning rate, and an EVPN engine analyzes RT‑2/RT‑3 propagation, MAC mobility, and control‑plane convergence. The underlay engine measures ECMP diversity, routing stability, and packet loss during convergence, while the flood‑and‑learn engine tracks unknown unicast and broadcast pressure. Each engine produces severity, timing, and root‑cause signals that feed into your multi‑page CLI, creating a cinematic, step‑by‑step visualization of VXLAN behavior across both overlay and underlay layers.

To make VXLAN intuitive and screenshot‑ready, the lab uses every visual technique you’ve developed: ASCII topology diagrams showing VTEPs and tunnels, shaded engine tables, multi‑page tabs, dashboard widgets, animated traffic‑flow bars, and timestamped failure timelines. The topology page displays leaf‑spine underlay paths and VXLAN tunnels; the engines page shows VTEP, VNI, EVPN, underlay, and flood‑and‑learn decisions; the failures page animates tunnel drops, EVPN withdraws, and MAC mobility events; and the traffic page shows VXLAN encapsulated flows shifting across ECMP paths during underlay reconvergence. Together, these visuals turn VXLAN into a living overlay system that engineers can explore, analyze, and optimize.

Data centers evolution

In recent years, data centers have seen a significant evolution. This evolution has brought popular technologies such as virtualization, cloud computing (private, public, and hybrid), and software-defined networking (SDN). Mobile-first and cloud-native data centers must scale, be agile, secure, consolidate, and integrate with compute/storage orchestrators. As well as visibility, automation, ease of management, operability, troubleshooting, and advanced analytics, today’s data center solutions are expected to include many other features.

A more service-centric approach is replacing device-by-device management. Most requests for proposals (RFPs) specify open application programming interfaces (APIs) and standards-based protocols to prevent vendor lock-in. A Cisco Virtual Extensible LAN (VXLAN)-based fabric using Nexus switches2 and NX-OS controllers form Cisco Virtual Extensible LAN (VXLAN).

what is spine and leaf architecture — Diagram: What is spine and leaf architecture. 2-Tier Spine Leaf Design

Issues with STP

When a switch receives redundant paths, the spanning tree protocol must designate one of those paths as blocked to prevent loops. While this mechanism is necessary, it can lead to suboptimal network performance. Blocked ports limit bandwidth utilization, which can be particularly problematic in environments with heavy data traffic.

One significant concern with the spanning tree protocol is its slow convergence time. When a network topology changes, the protocol takes time to recompute the spanning tree and reestablish connectivity. During this convergence period, network downtime can occur, disrupting critical operations and causing frustration for users.

What is VXLAN?

The Internet Engineering Task Force (IETF) developed VXLAN, or Virtual eXtensible Local-Area Network, as a network virtualization technology standard. Multi-tenant networks allow multiple organizations to share a physical network without accessing each other’s traffic.

The VXLAN can be compared to individual apartment apartments: each apartment is a separate, private dwelling within a shared physical structure, just as each VXLAN is a discrete, private network segment within a shared physical infrastructure.

With VXLANs, physical networks can be segmented into 16 million logical networks. To encapsulate Layer 2 Ethernet frames, User Datagram Protocol (UDP) packets with a VXLAN header are used. Combining VXLAN with Ethernet virtual private networks (EVPNs), which transport Ethernet traffic over WAN protocols, allows Layer 2 networks to be extended across Layer 3 IP or MPLS networks.

Benefits of VXLAN:

– Scalability: VXLAN allows creating up to 16 million logical networks, providing the scalability required for large-scale virtualized environments.

– Network Segmentation: By leveraging VXLAN, organizations can segment their networks into virtual segments, enhancing security and isolating traffic between applications or user groups.

– Flexibility and Mobility: VXLAN enables the movement of VMs across physical servers and data centers without the need to reconfigure network settings. This flexibility is crucial for workload mobility in dynamic environments.

– Interoperability: VXLAN is an industry-standard protocol supported by various networking vendors, ensuring compatibility across different network devices and platforms.

Use Cases for VXLAN

– Data Center Interconnect (DCI): VXLAN allows organizations to interconnect multiple data centers, enabling seamless workload migration, disaster recovery, and workload balancing across different locations.

– Multi-Tenant Environments: VXLAN enables service providers to offer virtualized network services to multiple tenants securely and isolatedly. This is particularly useful in cloud computing environments.

– Network Virtualization: VXLAN plays a crucial role in network virtualization, allowing organizations to create virtual networks independent of the underlying physical infrastructure. This enables greater flexibility and agility in managing network resources.

VXLAN vs. GRE

VXLAN, an overlay network technology, is designed to address the limitations of traditional VLANs. It enables the creation of virtual networks over an existing Layer 3 infrastructure, allowing for more flexible and scalable network deployments. VXLAN operates by encapsulating Layer 2 Ethernet frames within UDP packets, extending Layer 2 domains across Layer 3 boundaries.

GRE, on the other hand, is a simple IP packet encapsulation protocol. It provides a mechanism for encapsulating arbitrary protocols over an IP network and is widely used for creating point-to-point tunnels. GRE encapsulates the payload packets within IP packets, making it a versatile option for connecting remote networks securely.

Point-to-point GRE networks serve as a foundational element in modern networking. They allow for encapsulation and efficient transmission of various protocols over an IP network. Point-to-point GRE networks enable seamless communication and data transfer by establishing a direct virtual link between two endpoints.

Understanding mGRE

mGRE serves as the foundation for building DMVPN networks. It allows multiple sites to communicate with each other over a shared public network infrastructure while maintaining security and scalability. By utilizing a single mGRE tunnel interface on a central hub router, multiple spoke routers can dynamically establish and tear down tunnels, enabling seamless communication across the network.

The utilization of mGRE within DMVPN offers several key advantages. First, it simplifies network configuration by eliminating the need for point-to-point tunnels between each spoke router. Second, mGRE provides scalability, allowing for the dynamic addition or removal of spoke routers without impacting the overall network infrastructure. Third, mGRE enhances network resiliency by supporting multiple paths and providing load-balancing mechanisms.

Key VXLAN advantages

Because VXLANs are encapsulated inside UDP packets, they can run on any network that can send UDP packets. No matter how physically or geographically far a VTEP is from the decapsulating VTEP, it must forward UDP datagrams.

VXLAN and EVPN enable operators to create virtual networks from physical ports on any Layer 3 network switch supporting the standard. Connecting a port on switch A to two ports on switch B and another port on switch C creates a virtual network that appears to all connected devices as one physical network. Devices in this virtual network cannot see VXLANs or the underlying network fabric.

Problems that VXLAN solves

In the same way, as server virtualization has increased agility and flexibility, decoupling virtual networks from physical infrastructure has done the same. Therefore, network operators can scale their infrastructure rapidly and economically to meet growing demand while securely sharing a single physical network. For privacy and security reasons, networks are segmented to prevent one tenant from seeing or accessing the traffic of another.

In a similar way to traditional virtual LANs (VLANs), VXLANs enable operators to overcome the scaling limitations associated with VLANs.

Up to 16 million VXLANs can be created in an administrative domain, compared to 4094 traditional VLANs. Cloud and service providers can segment networks using VXLANs to support many tenants.
By using a VXLAN, you can create network segments between different data centers. In traditional VLAN networks, broadcast domains are created by segmenting traffic by VLAN tags, but once a packet containing VLAN tags reaches a router, the VLAN information is removed. There is no limit to the distance VLANs can travel within a Layer 2 network. Layer 3 boundaries, such as virtual machine migration, are generally avoided in certain use cases. Segmenting networks based on VXLAN encapsulates packets as UDP packets, while segmenting networks based on VXLAN encapsulates packets as IP packets. A virtual overlay network can extend as far as the physical Layer 3 routed network can reach when all switches and routers in the path support VXLAN without the applications running on the overlay network having to cross any Layer 3 boundaries. Servers connected to the network are still part of the Layer 2 network, even though UDP packets may have transited one or more routers.
Using Layer 2 segmentation on top of an underlying Layer 3 network allows one to segment a Layer 2 network over an underlying Layer 3 network and support many network segments. By providing Layer 2 segmentation on top of an underlying Layer 3 network, Layer 2 networks can remain small even if they are distant. Smaller Layer 2 networks can prevent MAC table overflows on switches.

Primary VXLAN applications

A service provider or cloud provider deploys VXLAN for apparent reasons: they have many tenants or customers, and they must separate the traffic of one customer from another due to legal, privacy, and ethical considerations.

Users, departments, or other groups of network-segmented devices may be tenants in enterprise environments for security reasons. Isolating IoT network traffic from production network applications is a good security practice for Internet of Things (IoT) devices such as data center environmental sensors.

VXLAN has been widely adopted and is now used in many large enterprise networks for virtualization and cloud computing. It provides:

A secure and efficient way to create virtual networks.
Allowing for the creation of multi-tenant segmentation.
Efficient routing.
Hardware-agnostic capabilities.

With its widespread adoption, VXLAN has become an essential technology for network virtualization.

Example: VXLAN Flood and Learn

Understanding VXLAN Flood and Learn

VXLAN flood and learn handles unknown unicast, multicast, and broadcast traffic in VXLAN networks. It allows the network to learn and forward traffic to the appropriate destination without relying on traditional flooding techniques. By leveraging multicast, VXLAN flood and learn improves efficiency and reduces the network’s reliance on flooding every unknown packet.

Proper multicast group management is essential to implementing VXLAN flood and learning with multicast. VXLAN uses multicast groups to distribute unknown traffic efficiently within the network.

VXLAN flood and learn with multicast offers several benefits for data center networks. Firstly, it reduces the flooding of unknown traffic, which helps minimize network congestion and improves overall performance. Additionally, it allows for better scalability by avoiding the need to flood every unknown packet to all VTEPs (VXLAN Tunnel Endpoint). This results in more efficient network utilization and reduced processing overhead.

Related: Before you proceed, you may find the following posts helpful for pre-information:

What is VXLAN

Traditional layer two networks have issues because of the following reasons:

Spanning tree: Restricts links.
Limited amount of VLANs: Restricts scalability;
Large MAC address tables: Restricts scalability and mobility

Spanning-tree avoids loops by blocking redundant links. By blocking connections, we create a loop-free topology and pay for links we can’t use. Although we could switch to a layer three network, some technologies require layer two networking.

VLAN IDs are 12 bits long, so we can create 4094 VLANs (0 and 4095 are reserved). Data centers may need help with only 4094 available VLANs. Let’s say we have a service provider with 500 customers. There are 4094 available VLANs, so each customer can only have eight.

The Role of Server Virtualization

Server virtualization has exponentially increased the number of addresses in our switches’ MAC addresses. Before server virtualization, there was only one MAC address per switch port. With server virtualization, we can run many virtual machines (VMs) or containers on a single physical server. Virtual NICs and virtual MAC addresses are assigned to each virtual machine. One switch port must learn many MAC addresses.

A data center could connect 24 or 48 physical servers to a top-of-rack (ToR) switch. Since there may be many racks in a data center, each switch must store the MAC addresses of all VMs that communicate. Networks without server virtualization require much larger MAC address tables.

Guide: VXLAN

In the following lab, I created a Layer 2 overlay with VXLAN over a Layer 3 core. A bridge domain VNI of 6001 must match both sides of the overlay tunnel. What Is a VNI? The VLAN ID field in an Ethernet frame has only 12 bits, so VLAN cannot meet isolation requirements on data center networks. The emergence of VNI specifically solves this problem.

Note: The VNI

A VNI is a user identifier similar to a VLAN ID. A VNI identifies a tenant. VMs with different VNIs cannot communicate at Layer 2. During VXLAN packet encapsulation, a 24-bit VNI is added to a VXLAN packet, enabling VXLAN to isolate many tenants.

In the screenshot below, you will notice that I can ping from desktop 0 to desktop one even though the IP addresses are not in the routing table of the core devices, simulating a Layer 2 overlay. Consider VXLAN to be the overlay and the routing Layer 3 core to be the underlay.

In the following screenshot, notice that the VNI has been changed. The VNI needs to be changed in two places in the configuration, as illustrated below. Once changed, the Peers are down; however, the NVE interface remains up. The VXLAN layer two overlay is not operational.

How does VXLAN work?

VXLAN uses tunneling to encapsulate Layer 2 Ethernet frames within IP packets. Each VXLAN network is identified by a unique 24-bit segment ID, the VXLAN Network Identifier (VNI). The source VM encapsulates the original Ethernet frame with a VXLAN header, including the VNI. The encapsulated packet is then sent over the physical IP network to the destination VM and decapsulated to retrieve the original Ethernet frame.

Analysis:

Notice below that it is running a ping from desktop 0 to desktop 1. The IP addresses assigned to this host are 10.0.0.1 and 10.0.0.2. First, notice that the ping is booming. When I do a packet capture on the links Gi1 connected to Leaf A, we see the encapsulation of the ICMP echo request and reply.

Everything is encapsulated into UDP port 1024. In my configurations of Leaf A and Leaf B, I explicitly set the VXLAN port to 1024.

XLAN and Network Virtualization.

VXLAN and network virtualization

VXLAN is a form of network virtualization. Network virtualization cuts a single physical network into many virtual networks, often called network overlays. Virtualizing a resource allows it to be shared by multiple users. Virtualization provides the illusion that each user is on his or her resources.

In the case of virtual networks, each user is under the misconception that there are no other users of the network. To preserve the illusion, virtual networks are separated from one another. Packets cannot leak from one virtual network to another.

VXLAN Loop Detection and Prevention

So, before we dive into the benefits of VXLAN, let us address the basics of loop detection and prevention, which is a significant driver for using network overlays such as VLXAN. The challenge is that data frames can exist indefinitely when loops occur, disrupting network stability and degrading performance.

In addition, loops introduce broadcast radiation, increasing CPU and network bandwidth utilization, which degrades user application access experience. Finally, in multi-site networks, a loop can span multiple data centers, causing disruptions that are difficult to pinpoint. Overlay networking can solve much of this.

VXLAN vs VLAN

However, first-generation Layer-2 Ethernet networks could not natively detect or mitigate looped topologies, while modern Layer-2 overlays implicitly build loop-free topologies. Therefore, overlays do not need loop detection and mitigation as long as no first-gen Layer-2 network is attached. Essentially, there is no need for a VXLAN spanning tree.

So, one of the differences between VXLAN vs VLAN is that the VLAN has a 12-bit VID while VXLAN has a 24-bit VID network identifier, allowing you to create up to 16 million segments. VXLAN has tremendous scale and stable loop-free networking and is a foundation technology in the ACI Cisco.

VXLAN and Data Center Interconnect

VXLAN has revolutionized data center interconnect by providing a scalable, flexible, and efficient solution for extending Layer 2 networks. Its ability to enable network segmentation, multi-tenancy support, and seamless mobility makes it a valuable technology for modern businesses.

However, careful planning, consideration of network infrastructure, and security measures are essential for successful implementation. By harnessing the power of VXLAN, organizations can achieve a more agile, scalable, and interconnected data center environment.

VXLAN vs VLAN: The VXLAN Benefits Drive Adoption

Introduced by Cisco and VMware and now heavily used in open networking, VXLAN stands for Virtual eXtensible Local Area Network. It is perhaps the most popular overlay technology for IP-based SDN data centers and is used extensively with ACI networks.

VXLAN was explicitly designed for Layer 2 over Layer 3 tunneling. Its early competition from NVGRE and STT is fading away, and VXLAN is becoming the industry standard. VLXAN brings many advantages, especially in loop prevention, as there is no need for a VXLAN spanning tree.

VXLAN Benefits: Scale and loop-free networks.

Today, overlays such as VXLAN almost eliminate the dependency on loop prevention protocols. However, even though virtualized overlay networks such as VXLAN are loop-free, having a failsafe loop detection and mitigation method is still desirable because loops can be introduced by topologies connected to the overlay network.

Loop prevention traditionally started with Spanning Tree Protocols (STP) to counteract the loop problem in first-gen Layer-2 Ethernet networks. Over time, other approaches evolved by moving networks from “looped topologies” to “loop-free topologies.

While LAG and MLAG were used, other approaches for building loop-free topologies arose using ECMP at the MAC or IP layers. For example, FabricPath or TRILL is a MAC layer ECMP approach that emerged in the last decade. More recently, network virtualization overlays that build loop-free topologies on top of IP layer ECMP became state-of-the-art.

VXLAN vs VLAN: Why Introduce VXLAN?

STP issues and scalability constraints: STP is undesirable on a large scale and lacks a proper load-balancing mechanism. A solution was needed to leverage the ECMP capabilities of an IP network while offering extended VLANs across an IP core, i.e., virtual segments across the network core. There is no VXLAN spanning tree.
Multi-tenancy: Layer 2 networks are capped at 4000 VLANs, restricting multi-tenancy design—a big difference in the VXLAN vs VLAN debates.
ToR table scalability: Every ToR switch may need to support several virtual servers, and each virtual server requires several NICs and MAC addresses. This pushes the limits on the ToR switch’s table sizes. In addition, after the ToR tables become full, Layer 2 traffic will be treated as unknown unicast traffic, which will be flooded across the network, causing instability to a previously stable core.

Diagram: STP Blocking. Source Cisco Press free chapter.

VXLAN use cases

Use Case	VXLAN Details
Use Case 1	Multi-tenant IaaS Clouds where you need a large number of segments
Use Case 2	Link Virtual to Physical Servers. This is done via software or hardware VXLAN to VLAN gateway
Use Case 3	HA Clusters across failure domains/availability zones
Use Case 4	VXLAN works well over fabrics that have equidistant endpoints
Use Case 5	VXLAN-encapsulated VLAN traffic across availability zones must be rate-limited to prevent broadcast storm propagation across multiple availability zones

What is VXLAN? The operations

When discussing VXLAN vs VLAN, VXLAN employs a MAC over IP/UDP overlay scheme and extends the traditional VLAN boundary of 4000 VLANs. The 12-bit VLAN identifier in traditional VLANs capped scalability within the SDN data center and proved cumbersome if you wanted a VLAN per application segment model. VXLAN scales the 12-bit to a 24-bit identifier and allows for 16 million logical endpoints, with each endpoint potentially offering another 4,000 VLANs.

While tunneling does provide Layer 2 adjacency between these logical endpoints and allows VMs to move across boundaries, the main driver for its insertion was to overcome the challenge of having only 4000 VLAN.

Typically, an application segment has multiple segments; between each segment, you will have firewalling and load-balancing services, and each segment requires a different VLAN. The Layer 2 VLAN segment transfers non-routable heartbeats or state information that can’t cross an L3 boundary. You will soon reach the 4000k VLAN limit if you are a cloud provider.

vxlan vs vlan — Multiple segments are required per application stack.

The control plane

The control plane is very similar to the spanning tree control plane. If a switch receives a packet destined for an unknown address, the switch will forward the packet to an IP address that floods the packet to all the other switches.

This IP address is, in turn, mapped to a multicast group across the network. VXLAN doesn’t explicitly have a control plane and requires an IP multicast running in the core for forwarding traffic and host discovery.

Best practices for enabling IP Multicast in the core

IP Multicast	In the Core
Bidirectional PIM or PIM Sparse Mode
Redundant Rendezvous Points (RP)
Shared trees (reduce the amount of IP multicast state)
Always check the IP multicast table sizes on core and ToR switches
Single IP multicast address for multiple VXLAN segments is OK

The requirement for IP multicast in the core made VXLAN undesirable from an operation point of view. For example, creating the tunnel endpoints is simple, but introducing a protocol like IP multicast to a core just for the tunnel control plane was considered undesirable. As a result, some of the more recent versions of VXLAN support IP unicast.

VXLAN uses a MAC over IP/UDP solution to eliminate the need for a spanning tree. There is no VXLAN spanning tree. This enables the core to be IP and not run a spanning tree. Many people ask why VXLAN uses UDP. The reason is that the UDP port numbers cause VXLAN to inherit Layer 3 ECMP features. The entropy that enables load balancing across multiple paths is embedded into the UDP source port of the overlay header.

2nd Lab Guide: Multicast VLXAN

In this lab guide, we will look at a VXLAN multicast mode. The multicat mode requires both unicast and multicast connectivity between sites. Similar to the previous one, this configuration guide uses OSPF to provide unicast connectivity, and now we have an additional bidirectional Protocol Independent Multicast (PIM) to provide multicast connectivity.

This does not mean that you don’t have a multicast-enabled core. It would be best if you still had multicast enabled on the core.

So we are not tunneling multicast over an IPv4 core without having multicast enabled on the core. I have multicast on all Layer 3 interfaces, and the mroute table is populated on all Layer 3 routers. With the command: Show ip mroute, we are tunneling the multicast traffic, and with the command: Show nve vni, we have multicast group 239.0.0.10 and a state of UP.

VXLAN benefits and stability

The underlying control plan network impacts the stability of VXLAN and the applications running within it. For example, if the underlying IP network cannot converge quickly enough, VLXAN packets may be dropped, and an application cache timeout may be triggered.

The rate of change in the underlying network significantly impacts the stability of the tunnels, yet the rate and change of the tunnels do not affect the underlying control plane. This is similar to how the strength of an MPLS / VPN overlay is affected by the core’s IGP.

VXLAN Points	VXLAN benefits	VXLAN drawbacks
Point 1	Runs over IP Transport	No control plane
Point 2	Offers a large number of logical endpoints	Needs IP Multicast***
Point 3	Reduced flooding scope	No IGMP snooping ( yet )
Point 4	Eliminates STP	No Pvlan support
Point 5	Easily integrated over existing Core	Requires Jumbo frames in the core ( 50 bytes)
Point 6	Minimal host-to-network integration	No built-in security features **
Point 7		Not a DCI solution ( no arp reduction, first-hop gateway localization, no inbound traffic steering i.e, LISP )

** VXLAN has no built-in security features. Anyone who gains access to the core network can insert traffic into segments. The VXLAN transport network must be secure, as no existing firewall or intrusion prevention system (IPS) equipment can be seen in the VXLAN traffic.

*** Recent versions have Unicast VXLAN. Nexus 1000V release 4.2(1)SV2(2.1)

Updated: VXLAN enhancements

MAC distribution mode is an enhancement to VXLAN that prevents unknown unicast flooding and eliminates data plane MAC address learning. Traditionally, this was done by flooding to locate an unknown end host, but it has now been replaced with a control plane solution.

During VM startup, the VSM ( control plane ) collects the list of MAC addresses and distributes the MAC-to-VTEP mappings to all VEMs participating in a VXLAN segment. This technique makes VXLAN more optimal by unicasting more intelligently, similar to Nicira and VMware NVP.

ARP termination works by giving the VSM controller all the ARP and MAC information. This enables the VSM to proxy and respond locally to ARP requests without sending a broadcast. Because 90% of broadcast traffic is ARP requests ( ARP reply is unicast ), this significantly reduces broadcast traffic on the network.

Closing Points on VXLAN

VXLAN is a network virtualization technology that encapsulates Ethernet frames within UDP packets, allowing the creation of a virtual network overlay. This overlay extends beyond traditional VLANs, providing more address space and enabling communication across different physical networks. With a 24-bit segment ID, VXLAN supports up to 16 million unique network segments, compared to the 4096 segments possible with VLANs.

At its core, VXLAN uses tunneling to encapsulate Layer 2 frames within Layer 3 packets. This process involves two key components: the VXLAN Tunnel Endpoint (VTEP) and the VXLAN header. VTEPs are responsible for encapsulating and decapsulating the Ethernet frames. They also handle the communication between different network segments, ensuring seamless data transmission across the network.

The adoption of VXLAN brings several advantages to network management:

1. **Scalability**: VXLAN’s ability to support millions of network segments allows for unprecedented scalability, making it ideal for large-scale data centers.

2. **Flexibility**: By decoupling the network overlay from the physical infrastructure, VXLAN enables flexible network designs that can adapt to changing business needs.

3. **Improved Resource Utilization**: VXLAN optimizes the use of network resources, enhancing performance and reducing congestion by spreading traffic across multiple paths.

VXLAN is particularly beneficial in scenarios where network scalability and flexibility are paramount. Common use cases include:

– **Data Center Interconnect**: Connecting multiple data centers over a single, unified network fabric.

– **Multi-Tenant Environments**: Providing isolated network segments for different tenants in cloud environments.

– **Disaster Recovery**: Facilitating seamless failover and recovery by extending network segments across geographically dispersed locations.

Summary: What is VXLAN

VXLAN, short for Virtual Extensible LAN, is a network virtualization technology that has recently gained significant popularity. In this blog post, we will examine VXLAN’s definition, workings, and benefits. So, let’s dive into the world of VXLAN!

Understanding VXLAN Basics

VXLAN is an encapsulation protocol that enables the creation of virtual networks over existing Layer 3 infrastructures. It extends Layer 2 segments over Layer 3 networks, allowing for greater flexibility and scalability. By encapsulating Layer 2 frames within Layer 3 packets, VXLAN enables efficient communication between virtual machines (VMs) across physical hosts or data centers.

VXLAN Operation and Encapsulation

To understand how VXLAN works, we must look at its operation and encapsulation process. When a VM sends a Layer 2 frame, it is encapsulated into a VXLAN packet by adding a VXLAN header. This header includes information such as the VXLAN network identifier (VNI), which helps identify the virtual network to which the packet belongs. The VXLAN packet is then transported over the underlying Layer 3 network to the destination physical host, encapsulated, and delivered to the appropriate VM.

Benefits and Use Cases of VXLAN

VXLAN offers several benefits that make it an attractive choice for network virtualization. Firstly, it enables the creation of large-scale virtual networks, allowing for seamless VM mobility and workload placement flexibility. VXLAN also helps overcome the limitations of traditional VLANs by providing a much larger address space, accommodating the ever-growing number of virtual machines in modern data centers. Additionally, VXLAN facilitates network virtualization across geographically dispersed data centers, making it ideal for multi-site deployments and disaster recovery scenarios.

VXLAN vs. Other Network Virtualization Technologies

While VXLAN is widely used, it is essential to understand its key differences and advantages compared to other network virtualization technologies. For instance, VXLAN offers better scalability and flexibility than traditional VLANs. It also provides better isolation and segmentation of virtual networks, making it an ideal choice for multi-tenant environments. Additionally, VXLAN is agnostic to the physical network infrastructure, allowing it to be easily deployed in existing networks without requiring significant changes.

Conclusion:

In conclusion, VXLAN is a powerful network virtualization technology that has revolutionized how virtual networks are created and managed. Its ability to extend Layer 2 networks over Layer 3 infrastructures, scalability, flexibility, and ease of deployment make VXLAN a go-to solution for modern data centers. Whether for workload mobility, multi-site implementations, or overcoming VLAN limitations, VXLAN offers a robust and efficient solution. Embracing VXLAN can unlock new possibilities in network virtualization, enabling organizations to build agile, scalable, and resilient virtual networks.

Concept set of mobile network, wireless Internet connection technology. Wifi illustration. People use device to connect Internet network Modern colorful flat vector illustration for poster, banner.

Routing Convergence

August 25, 2014

by Matt Conran Blog

Network Engineering Guide

Routing Convergence: From Failure Detection to Forwarding Recovery

Routing convergence determines how a network responds when its topology changes. When a link fails, a router becomes unreachable, or a route is withdrawn, routing protocols must react, select usable paths, and update forwarding state. Understanding each stage helps engineers reduce disruption and diagnose why traffic does not recover as quickly as expected.

A network is not resilient simply because it has redundant links or multiple routing protocols. Resilience depends on how effectively the network detects a failure, communicates the change, calculates an alternative path, and installs the resulting forwarding decision. Until those operations complete, packets may be lost, delayed, or sent along a path that is no longer usable.

Different protocols approach these tasks in different ways. OSPF and IS-IS distribute link-state information and calculate paths using SPF algorithms. EIGRP can use a feasible successor when its loop-free conditions are satisfied. BGP processes path attributes and routing updates, with its behaviour influenced by policy, session state, and route propagation. Their convergence characteristics therefore depend on more than the protocol name alone.

This engineering guide breaks convergence into measurable stages and connects routing-protocol behaviour to the forwarding plane. Through interactive experiments, you will investigate failure detection, topology changes, route selection, alternate paths, and the installation of updated forwarding entries.

The key engineering principle

Convergence time is an outcome of several processes, not one timer. Separate failure detection, control-plane processing, route calculation, and forwarding recovery when measuring performance or troubleshooting an outage.

STAGE 01 Detect Identify the failed link, neighbour, or forwarding path.

STAGE 02 Propagate Communicate the topology or reachability change.

STAGE 03 Calculate Re-evaluate routes and select a usable alternative.

STAGE 04 Install & Forward Update forwarding state and verify traffic recovery.

Network Insight · Engineering Guide

Routing Convergence: Table of Contents

Follow the complete convergence process, from initial failure detection to forwarding recovery and operational verification. Select a section to jump directly to its introduction.

Section 1 · Fundamentals

Understanding Routing Convergence

Before troubleshooting a routing failure, it is important to understand what convergence actually means. A network must detect a change, process routing information, select usable paths, and update forwarding behaviour. These activities are related, but they are not the same event.

In this section: Control plane versus forwarding plane, the four stages of convergence, and the factors that influence recovery time.

Section 01 · Foundations

Understanding Routing Convergence

Routing convergence is the process through which routing systems respond to a network change and establish usable, consistent routing and forwarding state. It connects topology information, route calculation, and packet delivery—and determines how much disruption users may experience when the network changes.

What does it mean for a network to converge?

Routers build their view of reachable destinations from directly connected networks, static configuration, and information learned through routing protocols. When a link fails or a neighbour becomes unavailable, some of that information may no longer describe the current network. Routing systems must detect the change, process the resulting information, and recalculate affected routes.

A network is considered converged for a particular routing domain when the relevant protocol state has settled and routers have processed the changes needed to make their routing decisions. This does not necessarily mean every router has an identical routing table. Different routers can have different perspectives, policies, and next hops while still operating correctly. The important requirement is that their routing and forwarding behaviour is consistent with the intended network design.

Control plane versus forwarding plane

Understanding convergence starts with separating two related functions. The control plane determines reachability and selects routes. The forwarding plane uses installed forwarding entries to move packets toward their destinations. A routing protocol can detect a failure and calculate an alternative route before every relevant forwarding entry has been updated.

Control plane

Determine the route

Maintains protocol state, exchanges routing information, evaluates route preferences, and selects paths. Examples include OSPF SPF calculations, EIGRP route selection, and BGP best-path processing.

Forwarding plane

Move the packet

Looks up packet destinations in forwarding state, selects an outgoing interface or next hop, and transmits traffic. Hardware and software implementation details vary by platform.

The Routing Information Base (RIB) contains selected routing information. The Forwarding Information Base (FIB) contains destination-to-forwarding information used by the forwarding process. The FIB is derived from routing information and may be programmed into hardware, software, or a combination of both. It is therefore more accurate to describe the FIB as derived forwarding state than simply as a copy of the RIB.

The four stages of convergence

A useful troubleshooting model divides a convergence event into four stages. The stages are closely related and may overlap; they are not always four strictly sequential operations.

01 · DETECT Recognise failure

Identify a failed link, lost neighbour, or unreachable next hop.

02 · PROPAGATE Share the change

Notify relevant routing processes and distribute updated information.

03 · CALCULATE Select a path

Re-evaluate available routes and choose a valid replacement.

04 · INSTALL Restore forwarding

Update forwarding state and verify that traffic can use the path.

Why the distinction matters

Fast failure detection does not guarantee fast application recovery. Route propagation, route calculation, forwarding-table programming, and the availability of a genuinely independent alternate path can all affect the result.

Convergence is more than a protocol timer

Convergence performance depends on the topology, routing information being processed, protocol configuration, implementation, processor load, and forwarding hardware. A precomputed backup path may allow local repair before the entire routing domain has completed its response. Conversely, a route that is theoretically available may not provide a usable repair if it violates routing policy or loop-free requirements.

For this reason, engineers should avoid treating a single advertised convergence time as a guarantee. Measurements must specify the event being timed, the start and end points, the traffic being tested, and the conditions under which the result was obtained.

What you should be able to explain

Why routing convergence and forwarding recovery are related but distinct.
How a topology change affects routing information and selected paths.
Why the RIB and FIB serve different purposes.
Why failure detection, route calculation, and forwarding updates should be measured separately.

Engineering takeaway: treat convergence as a chain of observable events. Establish what failed, when the routing system recognised it, when a replacement route became available, and when traffic successfully resumed.

Network Insight · Interactive Lab 01

Routing Convergence: Follow a Failure to Recovery

Inject a link failure and follow the sequence from detection to route recalculation and forwarding recovery. Step through the events to see why convergence is more than a single timer.

PRIMARY PATH ACTIVE Destination prefix: 10.40.40.0/24

Network topology All links operational

Primary path Alternate path Failed link

Convergence event Stage 0 / 7

READY

Primary route is forwarding

R1 forwards traffic towards R4 through R2. The alternate path through R3 is available in this conceptual topology.

Selected next hopR2

Forwarding statePrimary path

Failure detectionNot triggered

Route statusStable

This is a staged teaching model, not a vendor-specific timer or a real packet-forwarding test. Event order and convergence time vary by protocol and implementation.

Route and forwarding state R1 · destination prefix

State	Next hop	Interpretation
Selected route	R2	Primary route selected
Forwarding entry	R2	Traffic uses primary path
Alternate route	R3	Available as the backup path

Event timeline 0 / 7 events

Failure occursPrimary R2–R4 link goes down

Failure detectedR2 recognises the adjacency failure

Routing information propagatesR1 learns that the primary route is affected

Route recalculationR1 selects the eligible alternate via R3

RIB selection changesThe selected route points to R3

FIB updateForwarding state is updated

Forwarding verifiedTraffic can use R1 → R3 → R4

Engineering takeaway: Routing convergence includes failure detection, propagation, route selection and forwarding-state update. A changed route in the control plane does not, by itself, prove that traffic is already forwarding over the new path.

Section 2 · Failure Detection

Failure Detection and Propagation

Routing recovery begins when a router recognises that something has changed. Physical interface events, protocol timers, and mechanisms such as Bidirectional Forwarding Detection (BFD) can help identify failures. The routing protocol must then communicate the resulting change to the routers that need to react.

In this section: Failure-detection mechanisms, protocol notifications, convergence timelines, and the trade-offs involved in timer tuning.

Section 02 · Failure Mechanics

Failure Detection and Propagation

Before a routing protocol can select a replacement path, the network must recognise that something has changed. How quickly a failure is detected—and how reliably that information reaches the routing processes that need it—strongly influences convergence and traffic disruption.

Failure detection is the starting point

A network failure is not always immediately obvious to the routing protocol. A physical interface going down can generate a direct local event. A remote router becoming unreachable, a silent packet drop, or a one-way communication failure may require additional detection mechanisms before the system can declare a neighbour or path unavailable.

The detection method depends on the failure being considered. Physical carrier state, routing-protocol hello or keepalive messages, hold timers, and Bidirectional Forwarding Detection (BFD) can all contribute to failure detection. They are not interchangeable, and each has different assumptions and operational trade-offs.

Local signal

Interface or link state

A router may learn immediately that its local interface has lost carrier or been disabled. Detection is less direct when the underlying fault does not change the interface state.

Protocol signal

Neighbour messages

Routing protocols use periodic messages and associated timers to establish or maintain adjacency and session state. A missed message is meaningful only in the context of the protocol's detection rules.

Dedicated mechanism

Bidirectional Forwarding Detection

BFD can provide rapid detection of failures along a monitored forwarding path. A routing protocol or other registered client can use the resulting state change when the platform and configuration support it.

Layer 2 example

Unidirectional link detection

Mechanisms such as UDLD on supported Ethernet platforms can help identify certain one-way link conditions. Their behaviour and configuration are platform-dependent.

Detection, notification, and recovery are different events

Engineers often describe a network as having a particular failure-detection time. That is only one part of the recovery timeline. A failure must be detected, the relevant state change must be acted upon, routing information must be processed, and usable forwarding state must become available.

Failure occurs
A link, device, interface, or forwarding path stops behaving as expected. The fault may or may not be visible as an immediate interface-down event.
The fault is detected
A physical event, protocol timer, BFD session, or another supported mechanism identifies the condition.
Routing state changes
The routing process or registered client reacts to the event and updates neighbour, adjacency, or next-hop state as appropriate.
Updated information is processed
Relevant routing information is flooded, advertised, withdrawn, or otherwise propagated according to the protocol and its rules.
Traffic recovers
A valid path is selected and forwarding state is updated—or a previously installed repair path is activated where supported.

Important measurement distinction

A BFD session going down is evidence that the monitored session detected a problem. It does not, by itself, prove that the routing table, forwarding hardware, or application traffic has recovered.

How routing protocols propagate a failure

Once a failure is recognised, the control plane must determine which routes and neighbours are affected. Different protocol families communicate and recalculate reachability in different ways.

Protocol or mechanism	Typical response	Engineering consideration
OSPF / IS-IS	Topology information is updated and distributed; affected routers calculate routes from their resulting link-state database.	Flooding, SPF scheduling, topology size, and implementation behaviour influence recovery.
EIGRP	Neighbour and route state is updated; feasible alternatives may be used where available and eligible.	An eligible feasible successor can reduce recovery work, but an alternative is not guaranteed for every destination.
BGP	Session or next-hop changes can trigger route withdrawals, updates, and new best-path decisions.	Policy, path selection, route scale, session behaviour, and propagation all affect the outcome.
BFD	Signals the state of a monitored forwarding path to configured clients.	BFD is a detection mechanism, not a routing protocol or a complete recovery solution by itself.

These are typical behaviours, not fixed performance guarantees. Configuration, software release, platform capabilities, topology, and the precise failure scenario can change the observed response.

Timer tuning: faster is not always safer

Shorter detection intervals can reduce the time needed to identify certain failures. However, aggressive settings also reduce the margin for packet loss, scheduling delays, congestion, or temporary processing spikes. If a healthy neighbour is declared down incorrectly, the resulting adjacency reset or route withdrawal can create avoidable disruption.

Timer values should therefore be selected in line with the network design, supported platform limits, expected path behaviour, and operational testing. The objective is not simply to make every timer as small as possible; it is to achieve dependable detection with an acceptable recovery time.

Validate the failure type: determine whether the issue is a physical outage, a silent forwarding failure, or a control-plane/session problem.
Check both ends: compare interface state, neighbour state, counters, and logs where available.
Verify dependencies: confirm that routing protocols are configured to act on the relevant detection mechanism.
Test under realistic conditions: include packet loss, load, and recovery behaviour rather than relying only on configured timer values.

What to observe during a failure investigation

A useful troubleshooting record distinguishes the time of the original fault from the time each router detected it, the time routing state changed, and the time traffic recovered. Comparing these events helps identify whether the delay comes from detection, propagation, route calculation, forwarding installation, or a missing alternate path.

Engineering takeaway: failure detection starts convergence, but it does not finish it. Diagnose each stage independently, and confirm recovery by observing the resulting forwarding behaviour—not merely by checking that a neighbour has gone down.

Network Insight · Interactive Lab 02

Failure Detection: How Fast Does the Network Notice?

Compare an immediate interface-down event with timer-based detection. Observe the difference between a failure occurring, a router detecting it and routing information being propagated.

NETWORK HEALTHY Test prefix: 10.40.40.0/24

Failure detection topologyLink operational

Operational primary linkFailed test linkDestination

Detection controlsStep 0 / 4

Detection model

READY

Waiting for a failure

Choose a detection model, then inject the failure. This simulation uses ordered stages rather than claiming a universal detection time.

Failure stateNone

Detection stateNot started

Propagation stateNot started

Primary adjacencyUp

The timer option represents a detection interval conceptually. It does not run a real protocol timer or emulate vendor-specific BFD, OSPF, IS-IS, EIGRP or BGP behaviour.

Detection sequence0 / 4 events

Failure occursThe R2–R4 link becomes unavailable

Local detectionR2 recognises the link or adjacency failure

Routing process reactsThe protocol processes the changed neighbour or link state

Information propagatesOther routers can begin their own recalculation

Event logConceptual sequence

READYWaiting for a failure injection.

Engineering takeaway: Failure detection identifies that a neighbour, interface or path is no longer usable. Propagation and route recalculation follow; they are separate stages.

Section 3 · Route Computation

Route Calculation and Protocol Behaviour

Once a routing change is known, the router must determine which paths remain usable. Link-state protocols calculate routes from topology information, EIGRP uses its DUAL algorithm, and BGP evaluates eligible paths using its selection process and routing policies. Each protocol responds differently to a change.

In this section: SPF recalculation, EIGRP feasible successors, BGP best-path selection, and why an alternate route is not always immediately usable.

Section 03 · Routing Intelligence

Route Calculation and Protocol Behaviour

After a failure is detected and the relevant routing information changes, the network must determine which paths remain usable. Routing protocols solve this problem in different ways: some calculate routes from a shared topology, while others compare advertised paths, metrics, policies, and protocol-specific conditions.

Why route calculation matters

Detecting a failure does not automatically identify the best replacement route. A router must process the new information, determine which destinations are affected, and decide whether an alternate path is available. The amount of work depends on the protocol, the network design, the number of affected routes, and the platform's implementation.

Some changes have a limited scope. Others alter a large part of the topology or affect many advertised routes. The resulting convergence behaviour can therefore vary significantly even within the same routing protocol.

Stage 1 Receive changed information

Stage 2 Evaluate affected routes

Stage 3 Select a usable path

Stage 4 Prepare forwarding update

Link-state protocols: topology and SPF

OSPF and IS-IS distribute link-state information within their respective flooding scopes. Routers build and maintain a link-state database representing the topology they have learned. When relevant information changes, a router can run a Shortest Path First (SPF) calculation to determine routes based on that topology and the protocol's metric rules.

A topology change does not necessarily require every router to perform identical work at exactly the same time. Routers may receive updates at different times, and implementations can schedule, pace, or optimise SPF calculations. The effect depends on the change, the topology, and the implementation.

Topology view

Link-state database

Represents the link-state information learned within the relevant area or level. Accurate and current topology information is a prerequisite for a meaningful SPF result.

Route computation

Shortest Path First

Calculates shortest paths according to the protocol's metric model. The resulting routes are candidates for installation, subject to route selection and platform processing.

Distance-vector behaviour: EIGRP example

EIGRP uses neighbour information and route metrics to maintain reachability. Its Diffusing Update Algorithm (DUAL) supports loop-free route computation. When a primary route becomes unavailable, a router may be able to use a feasible successor—a qualifying alternate route already known to satisfy the feasibility condition.

If a suitable feasible successor is available, the router can often change to that route without conducting a full diffusing computation for the destination. If no feasible successor exists, additional query-and-reply processing may be needed. The outcome depends on the available topology and route state.

Key distinction

A backup path is not automatically a feasible successor. The feasibility condition matters because it helps EIGRP preserve loop-free behaviour during route changes.

Path-vector behaviour: BGP best-path selection

BGP exchanges reachability information between peers using UPDATE messages. When a route is withdrawn, a session changes state, or a new path is advertised, the router evaluates the affected routes using its import policies, available path attributes, and best-path decision process.

BGP selection can involve attributes such as local preference, AS path, origin, MED, and implementation-specific tie-breakers. The precise decision sequence varies with vendor implementation and configuration. A route that appears shorter or has a lower MED is not necessarily selected if an earlier decision criterion favours another path.

After the best path changes, BGP may advertise a changed route to eligible peers according to export policy and protocol rules. Route propagation and the decisions made by other routers can extend the overall recovery process beyond the first router's local calculation.

Protocol family	How it evaluates change	What can affect convergence
OSPF / IS-IS	Updates link-state information and calculates routes from the resulting topology.	Flooding scope, topology size, SPF scheduling, route installation, and implementation.
EIGRP	Evaluates successor and alternate routes using its metric and feasibility rules.	Availability of a feasible successor, query scope, and route state.
BGP	Evaluates candidate paths using policy and path attributes, then propagates eligible changes.	Policy, peer state, path diversity, route scale, and propagation behaviour.

Why a replacement path may not be available

A routing protocol can only select from the paths it knows about and is permitted to use. A physically connected alternate may not be advertised, may fail a protocol's eligibility rules, may be rejected by policy, or may lead to a destination that is no longer reachable.

Reachability: does the candidate path still lead to the destination?
Protocol rules: is the path eligible under the protocol's selection and loop-prevention mechanisms?
Policy: has an import, export, filtering, or preference rule changed which path can be used?
Next-hop resolution: can the selected route's next hop be resolved and used by the forwarding system?
Installation: can the resulting forwarding state be programmed successfully?

Fast convergence is not simply fast computation

A route calculation may finish quickly while traffic remains disrupted because the required update has not propagated, the selected next hop is unresolved, or forwarding hardware has not yet been updated. Conversely, a precomputed repair path can sometimes restore forwarding before the full control-plane process has completed.

For this reason, convergence investigations should distinguish the time when a protocol receives changed information, the time when it reaches a new route decision, and the time when the forwarding plane can successfully use that decision.

Engineering takeaway: protocols use different algorithms and selection rules to respond to topology and reachability changes. Understand the protocol's decision process, verify that a valid alternate path exists, and measure the forwarding outcome rather than assuming route calculation alone equals recovery.

Network Insight · Interactive Lab 03

Route Calculation: Which Path Wins?

Break the primary path, compare the remaining candidates and follow the distinction between route calculation, route selection and installation. The protocol selector changes the explanation, not a full implementation of each routing protocol.

PRIMARY ROUTE SELECTED Destination: 10.40.40.0/24

Candidate path topologyPrimary path available

Primary candidateAlternate candidateFailed link

Decision engineStage 0 / 5

Routing protocol perspective

NORMAL OPERATION

Primary route is selected

R1 currently uses R2 as the next hop to reach 10.40.40.0/24. The route via R3 is an alternate candidate in this teaching topology.

Selected next hopR2

Route calculationStable

Alternate routeAvailable

Installation stateInstalled

Protocol behaviour is represented conceptually. The topology does not run SPF, DUAL, BGP best-path rules or real routing updates.

Candidate route evaluationR1 · 10.40.40.0/24

Candidate A · via R2Primary path is reachable and selected.

Selected

Candidate B · via R3Alternate path is available but not selected.

Standby

Selection principle: A router can use only routes it has learned or calculated and considers eligible. The selected route depends on protocol-specific rules and configuration.

Calculation timeline0 / 5 stages

Failure recognisedThe primary path is no longer usable

Routing information updatedThe protocol processes the changed topology or route

Alternatives evaluatedEligible remaining candidates are considered

New route selectedR1 selects the route via R3 in this model

Route installedThe selected route is ready for forwarding installation

Engineering takeaway: Route calculation, best-path selection and route installation are related but distinct. A protocol's method of finding or choosing a route depends on its design.

Section 4 · Forwarding Recovery

RIB, FIB and Fast Reroute

A router can calculate a replacement route without immediately restoring traffic. The selected route must be reflected in forwarding state, and packets must be able to use the resulting next hop. Techniques such as Equal-Cost Multipath (ECMP) and Loop-Free Alternates (LFA) can help maintain or restore forwarding, depending on the topology and platform.

In this section: How the RIB and FIB differ, forwarding installation, alternate paths, and fast-reroute techniques.

Section 04 · Forwarding Recovery

RIB, FIB and Fast Reroute

A routing protocol can calculate a replacement path, but packets recover only when the forwarding system can use a valid next hop. Understanding the Routing Information Base (RIB), Forwarding Information Base (FIB), and fast-reroute mechanisms explains why routing convergence and traffic recovery are related—but not identical—events.

From route information to packet forwarding

Routers maintain control-plane information about reachable destinations and use it to determine which routes should be selected. The selected routing information is then used to program the forwarding system. On many platforms, this involves software processes coordinating with hardware tables or other forwarding resources.

The RIB and FIB serve different purposes. The RIB stores routing information and supports route selection. The FIB contains the forwarding entries used to make packet-forwarding decisions, often with resolved next-hop and adjacency information appropriate to the platform.

01 · Learn Receive routes and topology information

02 · Select Choose eligible routes for the RIB

03 · Program Resolve and install forwarding entries

04 · Forward Send packets using the active FIB

Control plane

Routing Information Base

Contains routing information learned from connected networks, static configuration, and routing protocols. Route selection determines which route or routes are eligible to contribute to forwarding.

Forwarding plane

Forwarding Information Base

Supports destination lookups for packet forwarding. Entries may reference resolved next hops, interfaces, adjacency information, and other platform-specific forwarding structures.

Why the RIB and FIB can differ in timing

A new best route in the control plane does not prove that every forwarding resource has already been updated. The system may need to resolve the next hop, reconcile competing routes, allocate forwarding resources, and program software or hardware tables. The timing and architecture differ across platforms.

During a failure, the old forwarding entry may point to an unusable next hop until the forwarding state changes. Alternatively, a platform may already have an eligible alternate or repair entry that can be activated quickly. Observing both control-plane route state and data-plane behaviour helps establish whether recovery is complete.

Operational rule

Do not treat “the route is present in the routing table” as conclusive evidence that packets are successfully reaching the destination. Verify next-hop resolution, forwarding entries, interface state, and end-to-end traffic where possible.

Equal-Cost Multi-Path forwarding (ECMP)

ECMP allows a router to forward traffic over multiple eligible paths that have equal cost according to the applicable routing decision. The forwarding system commonly uses a hash of packet fields to select a path, helping keep flows on a consistent path rather than distributing each packet independently.

If one ECMP next hop fails, the router may remove or avoid that next hop and continue using the remaining eligible paths. This can reduce disruption, but the exact effect depends on the implementation, hash behaviour, flow distribution, and whether the surviving paths have sufficient capacity.

Path availability: the remaining next hops must still be valid and reachable.
Flow impact: traffic may be rehashed when the set of next hops changes, potentially affecting packet ordering for some flows.
Capacity: surviving links must be able to handle the shifted traffic load.
Installation: the forwarding plane must apply the updated next-hop set successfully.

Fast reroute and precomputed repair paths

Fast reroute techniques aim to reduce traffic loss by making a repair path available before a failure occurs. Rather than waiting for every step of the normal control-plane reconvergence process, a router may be able to redirect affected traffic onto a precomputed alternate when the protected failure is detected.

Examples include Loop-Free Alternates (LFA) and, where supported and configured, remote LFA or Segment Routing-based repair mechanisms. Their applicability depends on topology, protocol support, platform capabilities, and the failure being protected against.

Mechanism	Primary purpose	Important limitation
ECMP	Use multiple equal-cost forwarding paths.	Does not guarantee that every failure has a surviving path or that traffic remains evenly distributed.
LFA	Provide an eligible loop-free alternate next hop for protected destinations.	Topology may not provide a valid LFA for every destination and failure.
Remote LFA / Segment Routing repair	Extend repair options where supported, including cases not covered by a direct LFA.	Requires suitable topology, configuration, protocol support, and forwarding capabilities.
Normal reconvergence	Calculate and install the new forwarding state after the routing system processes the change.	Traffic disruption may persist while detection, propagation, calculation, and installation occur.

Forwarding recovery and control-plane convergence

A repair path can restore traffic before the routing protocol has completed all of its normal calculations and advertisements. In that case, the data plane may be forwarding successfully while the control plane continues to converge. The reverse can also happen: a new route may appear in the RIB while traffic still fails because of unresolved next hops, stale forwarding state, or a separate downstream fault.

These are not necessarily contradictions. They are signs that different components have different responsibilities and update timelines. Reliable testing should record both routing-state changes and packet-level results.

How to verify forwarding recovery

Inspect the selected route and its next-hop resolution.
Check the relevant FIB entry and forwarding adjacency where the platform exposes them.
Confirm that failed interfaces or next hops are no longer used for affected traffic.
Run suitable reachability or application tests across the affected path.
Compare the observed traffic recovery time with the routing and forwarding event timeline.

Engineering takeaway: the RIB represents selected routing information, while the FIB supports actual packet forwarding. ECMP and fast reroute can reduce disruption when valid alternatives exist, but successful recovery must be verified in the forwarding plane—not inferred from a route calculation alone.

Network Insight · Interactive Lab 04

RIB to FIB: When Does Forwarding Recover?

Fail the primary link and step through route selection and forwarding installation. Compare normal convergence with a conceptual pre-installed fast-reroute repair path.

PRIMARY FORWARDING ACTIVE Destination: 10.40.40.0/24

Forwarding topologyPrimary path active

Primary routeAlternate routeFailed link

Recovery controlsStage 0 / 5

Forwarding strategy

Normal convergenceRecalculate, select the alternate route, then update forwarding state. Pre-installed repair pathConceptually use an eligible precomputed local repair while broader convergence continues.

NORMAL OPERATION

Primary forwarding entry is active

R1 forwards the destination prefix through R2. The alternate path through R3 exists in the topology but is not currently selected.

RIB next hopR2

FIB next hopR2

Forwarding statusPrimary active

Repair stateStandby

Fast reroute is represented conceptually. Real repair behaviour depends on the technology, topology, precomputed repair availability and platform support.

Routing and forwarding stateR1 · 10.40.40.0/24

Plane / entry	Current next hop	What it tells you
Selected route (RIB)	R2	Primary route selected
Forwarding entry (FIB)	R2	Packets use primary path
Repair candidate	R3	Available as alternate path

Recovery sequence0 / 5 stages

Primary link failsThe R2–R4 path becomes unavailable

Failure response beginsNormal convergence or a repair mechanism reacts

RIB selection changesThe control plane selects the route via R3

FIB is updatedForwarding state reflects the selected next hop

Forwarding verifiedCheck packet delivery over the alternate path

Engineering takeaway: The RIB represents selected routing information; the FIB is the forwarding structure used to send packets. Verify both, and test traffic, before declaring recovery.

Section 5 · Operational Verification

Convergence Measurement and Troubleshooting

Reliable convergence analysis depends on evidence, not assumptions. A routing log may show when a change was detected, while protocol state, RIB and FIB inspection, packet tests, and application monitoring reveal different parts of the recovery process. A meaningful result requires a clearly defined start event and end condition.

In this section: Convergence metrics, repeatable failure tests, route flapping, telemetry, and a structured troubleshooting workflow.

Section 05 · Operational Evidence

Convergence Measurement and Troubleshooting

A network can appear healthy in a routing table while traffic is still being lost—or it can restore forwarding before every control-plane process has settled. Reliable convergence analysis requires a clear measurement boundary, timestamped evidence, and a methodical way to separate detection, route calculation, forwarding recovery, and application impact.

Define what convergence means for the test

A convergence measurement is meaningful only when its start and end points are explicit. Measuring from the moment a cable is disconnected to the moment a monitoring probe succeeds is different from measuring the interval between a routing adjacency failure and a new route appearing in the RIB.

Before testing, identify the failure being introduced, the destination or traffic flow being observed, and the event that will count as recovery. Use the same definitions across repeated tests so results can be compared fairly.

Control-plane metric

Routing convergence

Measures a defined routing-state transition, such as the time from a topology event to a stable, expected route selection. Specify exactly which event marks the beginning and end.

Forwarding metric

Traffic recovery

Measures when the tested packets or flows can successfully use the recovered path. This may occur before or after the routing system reaches its final stable state.

Service metric

Application recovery

Measures when the application meets its service criteria again, such as successful transactions or acceptable latency. Application recovery can include dependencies beyond routing.

Stability metric

Time to stable state

Checks whether the route and traffic remain healthy after recovery, rather than counting a brief successful probe as proof that the incident is over.

Do not mix measurement boundaries

A result such as “convergence took 200 ms” is incomplete unless the failure trigger, observation method, destination, and recovery criterion are stated. Different test methods can produce different—and individually valid—measurements.

Build an evidence timeline

The most useful troubleshooting timeline combines control-plane events with forwarding and traffic observations. Synchronised clocks make it easier to correlate logs across routers, although timestamp precision and collection delays should also be considered.

Record the trigger
Capture when the link, interface, session, or simulated fault was introduced. Record the test conditions and affected destination or flow.
Capture detection
Record interface transitions, neighbour-state changes, BFD events, or protocol timer expirations as applicable.
Capture route processing
Observe topology updates, route withdrawals or advertisements, SPF or other route calculations, and best-path changes where available.
Verify forwarding state
Check the selected route, next-hop resolution, FIB or adjacency changes, and any active fast-reroute behaviour.
Confirm traffic recovery
Use repeated probes or representative application tests to determine when service returns and whether it remains stable.

Evidence to collect during an investigation

Evidence source	What it helps establish	What it cannot prove alone
Interface counters and logs	Physical state transitions, errors, drops, and link-related events.	That all routing and forwarding state has recovered.
Neighbour and session state	Whether the relevant protocol adjacency or session is established.	That every destination has a usable route.
RIB and route details	Which route is selected and why a candidate path may have been preferred or rejected.	That the forwarding hardware is already using that path.
FIB and next-hop details	Which forwarding entries and next hops are programmed, where observable.	That end-to-end traffic is successful beyond the local forwarding point.
Packet probes and application telemetry	Observed loss, latency, reachability, and service recovery for the tested flow.	The exact internal routing event responsible for every symptom.

Recognise route flapping and unstable recovery

Route flapping occurs when route availability or attributes repeatedly change. Repeated interface transitions, unstable sessions, intermittent reachability, or changing path attributes can prevent the network from settling into a stable state.

Flapping can increase control-plane work and cause repeated forwarding changes. In BGP, route-flap damping may be available in some environments, but its use requires care: suppressing a route can extend unreachability, and the behaviour depends on configuration and operational policy.

Correlate repeated events: compare interface logs, neighbour resets, route changes, and traffic-loss windows.
Look for a shared cause: investigate common links, power, optics, congestion, software events, or unstable downstream dependencies.
Inspect policy and attributes: for BGP, determine whether changing advertisements or policy inputs are altering best-path decisions.
Check the final state: confirm the selected route and forwarding path remain stable after the last observed event.

Use repeatable tests instead of assumptions

A single successful run may hide timing variation. Repeat controlled tests, keep the topology and traffic conditions consistent, and record the observed recovery interval for each run. Where the test is intended to represent production behaviour, include realistic traffic load and relevant failure types.

Report the measurement method alongside the results. Depending on the objective, useful summaries may include minimum, median, maximum, and a high percentile of recovery times. Preserve individual run data as well: aggregate values can conceal an occasional long outage.

A practical troubleshooting sequence

Confirm the fault and its scope before changing configuration.
Identify whether detection was delayed or incorrect.
Determine whether routing information propagated as expected.
Inspect route selection and verify that a usable alternative exists.
Check FIB installation, next-hop resolution, and fast-reroute state where supported.
Test end-to-end traffic and application behaviour.
Repeat the test after remediation and compare evidence against the original baseline.

Engineering takeaway: measure convergence from a defined event to a defined recovery condition. Correlate logs, routing state, forwarding state, and traffic evidence to locate the delay, then repeat the test to verify that the fix improves both recovery time and stability.

Network Insight · Interactive Lab 05

Convergence Incident: Measure, Diagnose, Verify

Investigate a simulated routing incident from first failure evidence to verified forwarding recovery. Step through the event timeline, inspect the evidence and complete the engineering checklist.

INCIDENT READY Incident RC-05 · Prefix 10.50.50.0/24

Incident scenarioStage 0 / 5

Reported symptom: intermittent packet loss

A primary path has failed. The operations team needs to establish when failure was detected, when routing state changed, when forwarding recovered and whether traffic actually resumed.

INITIAL STATE

Begin the incident investigation

Start the incident to simulate a link failure. Then step through detection, route recalculation, forwarding installation and verification. The sequence is illustrative, not a live router measurement.

Failure evidenceNot observed

Control-plane stateStable

Forwarding statePrimary path

Traffic verificationNot tested

No real timers or millisecond values are generated. Use actual event timestamps and defined start/end criteria when measuring a production network.

Event evidenceIllustrative records

InterfaceAwaiting failure event

RoutingNo route change observed yet

ForwardingPrimary path currently selected

TrafficEnd-to-end test not run

Evidence rule: Do not infer the entire convergence interval from a single log. Correlate interface, protocol, route, forwarding and traffic observations.

Convergence timeline0 / 5 events

Failure occursPhysical or logical path event

Failure is detectedInterface state, timer, BFD or other evidence

Route selection changesControl plane selects an eligible alternative

Forwarding state updatesFIB and relevant forwarding hardware are checked

Traffic recovery is verifiedProbe or application test confirms delivery

Engineer’s verification checklist0 / 5 checked

Confirm the failure eventCheck interface state, physical alarms or the relevant failure signal. Check detection and protocol responseReview neighbour state, timers, BFD or protocol logs where applicable. Inspect the selected routeConfirm the new next hop, route source and alternate-path eligibility. Verify forwarding installationInspect the FIB or platform forwarding entry, not only the routing table. Test actual trafficUse suitable probes, counters or application checks to validate recovery.

Complete the checklist as you investigate. A checked item records your assessment; it does not query a live network.

Measurement principle: Define the start event and recovery endpoint before calculating convergence time. Control-plane convergence, forwarding recovery and application recovery are related but distinct outcomes.

Conclusion · Engineering Practice

Routing Convergence: From Failure to Recovery

Routing convergence is the process through which a network detects a change, distributes the relevant information, recalculates usable paths, and updates forwarding behaviour. Understanding each stage helps engineers distinguish a routing-protocol problem from a forwarding, detection, or service-recovery problem.

Detect

Recognise a failure or topology change through interface events, protocol timers, or mechanisms such as BFD.

Propagate

Distribute the changed reachability or topology information to the routers that need to react.

Calculate

Run the relevant protocol logic and determine which routes or alternate paths remain eligible.

Forward

Install the resulting forwarding state and verify that traffic can use the intended path.

The key engineering principle

A routing-table change does not, by itself, prove that an application has recovered. Control-plane processing, forwarding installation, packet delivery, and service recovery must be measured separately.

Routing convergence troubleshooting checklist

When investigating a slow or incomplete recovery, work through the following questions in sequence:

When did the failure occur, and when was it first detected?
Did the relevant routing protocol propagate the change correctly?
Was the expected replacement route eligible and selected?
Did the RIB and FIB update as expected?
Did packets actually traverse the replacement path?
Did application traffic recover, and was the result repeatable?

Explore · Trace · Diagnose

Put routing convergence into practice

Move from theory to an interactive BGP environment. Explore route propagation, inspect best-path decisions, follow routing-state changes, and investigate how network behaviour changes when a link or route becomes unavailable.

Launch the BGP Core Simulator

Opens the existing Network Insight simulator in a new tab.

Continue learning with Network Insight

Use these learning resources to connect routing theory with practical forwarding and fault investigation.

BGP Learning Centre

Explore BGP architecture, route propagation, best-path selection, and fault investigation with the BGP Learning Centre and simulator .

IP Forwarding

Follow how routers use routing and forwarding information to make packet-by-packet forwarding decisions. Use the IP forwarding guide alongside this article.

BGP Route Selection and Propagation

Extend your understanding of path attributes, route eligibility, route advertisement, and how routing changes affect reachability.

Failure Detection and Fast Reroute

Study the difference between detecting a failure, selecting an alternate path, installing forwarding state, and restoring traffic.

☕

Support Network Insight

Help support the development of interactive networking labs, BGP simulations, and educational content.

Support the Project

Physical diagram - Fabric Extenders and Parent switches

How-to: Fabric Extenders & VPC

August 22, 2014

by Matt Conran Blog

Topology Diagram

The topology diagram depicts two Nexus 5K acting as parent switches with physical connections to two downstream Nexus 2k (FEX) acting as the 10G physical termination points for the connected server.

Part1. Connecting the FEX to the Parent switch:

The FEX and the parent switch use Satellite Discovery Protocol (SDP) periodic messages to discovery and register with one another.

When you initially log on to the Nexus 5K you can see that the OS does not recognise the FEX even though there are two FEXs that are cabled correctly to parent switch. As the FEX is recognised as a remote line card you would expect to see it with a “show module” command.

N5K3# sh module
Mod Ports Module-Type Model Status
— —– ——————————– ———————- ————
1 40 40x10GE/Supervisor N5K-C5020P-BF-SUP active *
2 8 8×1/2/4G FC Module N5K-M1008 ok
Mod Sw Hw World-Wide-Name(s) (WWN)
— ————– —— ————————————————–
1 5.1(3)N2(1c) 1.3 —

2 5.1(3)N2(1c) 1.0 93:59:41:08:5a:0c:08:08 to 00:00:00:00:00:00:00:00

Mod MAC-Address(es) Serial-Num

— ————————————– ———-

1 0005.9b1e.82c8 to 0005.9b1e.82ef JAF1419BLMA

2 0005.9b1e.82f0 to 0005.9b1e.82f7 JAF1411AQBJ

We issue the “feature fex” command we observe the FEX sending SDP messages to the parent switch i.e. RX but we don’t see the parent switch sending SDP messages to the FEX i.e. TX.

Notice in the output below there is only “fex:Sdp-Rx” messages.

N5K3# debug fex pkt-trace
N5K3# 2014 Aug 21 09:51:57.410701 fex: Sdp-Rx: Interface: Eth1/11, Fex Id: 0, Ctrl Vntag: -1, Ctrl Vlan: 1
2014 Aug 21 09:51:57.410729 fex: Sdp-Rx: Refresh Intvl: 3000ms, Uid: 0x4000ff2929f0, device: Fex, Remote link: 0x20000080
2014 Aug 21 09:51:57.410742 fex: Sdp-Rx: Vendor: Cisco Systems Model: N2K-C2232PP-10GE Serial: FOC17100NHX
2014 Aug 21 09:51:57.821776 fex: Sdp-Rx: Interface: Eth1/10, Fex Id: 0, Ctrl Vntag: -1, Ctrl Vlan: 1
2014 Aug 21 09:51:57.821804 fex: Sdp-Rx: Refresh Intvl: 3000ms, Uid: 0x2ff2929f0, device: Fex, Remote link: 0x20000080
2014 Aug 21 09:51:57.821817 fex: Sdp-Rx: Vendor: Cisco Systems Model: N2K-C2232PP-10GE Serial: FOC17100NHU

The FEX appears as “DISCOVERED” but no additional FEX host interfaces appear when you issue a “show interface brief“.

Command: show fex [chassid_id [detail]]: Displays information about a specific Fabric Extender Chassis ID

Command: show interface brief: Display interface information and connection status for each interface.

N5K3# sh fex
FEX FEX FEX FEX
Number Description State Model Serial
————————————————————————
— ——– Discovered N2K-C2232PP-10GE SSI16510AWF
— ——– Discovered N2K-C2232PP-10GE SSI165204YC
N5K3#
N5K3# show interface brief

——————————————————————————–

Ethernet VLAN Type Mode Status Reason Speed Port

Interface Ch #

——————————————————————————–

Eth1/1 1 eth access down SFP validation failed 10G(D) —

Eth1/2 1 eth access down SFP validation failed 10G(D) —

Eth1/3 1 eth access up none 10G(D) —

Eth1/4 1 eth access up none 10G(D) —

Eth1/5 1 eth access up none 10G(D) —

Eth1/6 1 eth access down Link not connected 10G(D) —

Eth1/7 1 eth access down Link not connected 10G(D) —

Eth1/8 1 eth access down Link not connected 10G(D) —

Eth1/9 1 eth access down Link not connected 10G(D) —

Eth1/10 1 eth fabric down FEX not configured 10G(D) —

Eth1/11 1 eth fabric down FEX not configured 10G(D) —

Eth1/12 1 eth access down Link not connected 10G(D) —

snippet removed

The Fabric interface Ethernet1/10 show as DOWN with a “FEX not configured” statement.

N5K3# sh int Ethernet1/10
Ethernet1/10 is down (FEX not configured)
Hardware: 1000/10000 Ethernet, address: 0005.9b1e.82d1 (bia 0005.9b1e.82d1)
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA
Port mode is fex-fabric
auto-duplex, 10 Gb/s, media type is 10G

Beacon is turned off

Input flow-control is off, output flow-control is off

Rate mode is dedicated

Switchport monitor is off

EtherType is 0x8100

snippet removed

To enable the parent switch to fully discover the FEX we need to issue the “switchport mode fex-fabric” under the connected interface. As you can see we are still not sending any SDP messages but we are discovering the FEX.

The next step is to enable the FEX logical numbering under the interface so we can start to configure the FEX host interfaces. Once this is complete we run the “debug fex pkt-trace” and we are not sending TX and receiving RX SDP messages.

Command:”fex associate chassis_id“: Associates a Fabric Extender (FEX) to a fabric interface. To disassociate the Fabric Extender, use the “no” form of this command.

From the “debug fexpkt-race” you can see the parent switch is now sending TX SDP messages to the fully discovered FEX.

N5K3(config)# int Ethernet1/10
N5K3(config-if)# fex associate 101
N5K3# debug fex pkt-trace
N5K3# 2014 Aug 21 10:00:33.674605 fex: Sdp-Tx: Interface: Eth1/10, Fex Id: 101, Ctrl Vntag: 0, Ctrl Vlan: 4042
2014 Aug 21 10:00:33.674633 fex: Sdp-Tx: Refresh Intvl: 3000ms, Uid: 0xc0821e9b0500, device: Switch, Remote link: 0x1a009000
2014 Aug 21 10:00:33.674646 fex: Sdp-Tx: Vendor: Model: Serial: ———-

2014 Aug 21 10:00:33.674718 fex: Sdp-Rx: Interface: Eth1/10, Fex Id: 0, Ctrl Vntag: 0, Ctrl Vlan: 4042

2014 Aug 21 10:00:33.674733 fex: Sdp-Rx: Refresh Intvl: 3000ms, Uid: 0x2ff2929f0, device: Fex, Remote link: 0x20000080

2014 Aug 21 10:00:33.674746 fex: Sdp-Rx: Vendor: Cisco Systems Model: N2K-C2232PP-10GE Serial: FOC17100NHU

2014 Aug 21 10:00:33.836774 fex: Sdp-Rx: Interface: Eth1/11, Fex Id: 0, Ctrl Vntag: -1, Ctrl Vlan: 1

2014 Aug 21 10:00:33.836803 fex: Sdp-Rx: Refresh Intvl: 3000ms, Uid: 0x4000ff2929f0, device: Fex, Remote link: 0x20000080

2014 Aug 21 10:00:33.836816 fex: Sdp-Rx: Vendor: Cisco Systems Model: N2K-C2232PP-10GE Serial: FOC17100NHX

2014 Aug 21 10:00:36.678624 fex: Sdp-Tx: Interface: Eth1/10, Fex Id: 101, Ctrl Vntag: 0, Ctrl Vlan: 4042

2014 Aug 21 10:00:36.678664 fex: Sdp-Tx: Refresh Intvl: 3000ms, Uid: 0xc0821e9b0500, device: Switch, Remote snippet removed

Now the 101 FEX status changes from “DISCOVERED” to “ONLINE”. You may also see an additional FEX with serial number SSI165204YC as “DISCOVERED” and not “ONLINE”. This is due to the fact that we have not explicitly configured it under the other Fabric interface.

N5K3# sh fex
FEX FEX FEX FEX
Number Description State Model Serial
————————————————————————
101 FEX0101 Online N2K-C2232PP-10GE SSI16510AWF
— ——– Discovered N2K-C2232PP-10GE SSI165204YC
N5K3#
N5K3# show module fex 101

FEX Mod Ports Card Type Model Status.

— — —– ———————————- —————— ———–

101 1 32 Fabric Extender 32x10GE + 8x10G Module N2K-C2232PP-10GE present

FEX Mod Sw Hw World-Wide-Name(s) (WWN)

— — ————– —— ———————————————–

101 1 5.1(3)N2(1c) 4.4 —

FEX Mod MAC-Address(es) Serial-Num

— — ————————————– ———-

101 1 f029.29ff.0200 to f029.29ff.021f SSI16510AWF

Issuing the “show interface brief” we see new interfaces, specifically host interfaces for the FEX. The syntax below shows that only one interface is up; interface labelled Eth101/1/1. Reason for this is that only one end host (server) is connected to the FEX

N5K3# show interface brief
——————————————————————————–
Ethernet VLAN Type Mode Status Reason Speed Port
Interface Ch #
——————————————————————————–
Eth1/1 1 eth access down SFP validation failed 10G(D) —
Eth1/2 1 eth access down SFP validation failed 10G(D) —
snipped removed

——————————————————————————–

Port VRF Status IP Address Speed MTU

——————————————————————————–

mgmt0 — up 192.168.0.53 100 1500

——————————————————————————–

Ethernet VLAN Type Mode Status Reason Speed Port

Interface Ch #

——————————————————————————–

Eth101/1/1 1 eth access up none 10G(D) —

Eth101/1/2 1 eth access down SFP not inserted 10G(D) —

Eth101/1/3 1 eth access down SFP not inserted 10G(D) —

Eth101/1/4 1 eth access down SFP not inserted 10G(D) —

Eth101/1/5 1 eth access down SFP not inserted 10G(D) —

Eth101/1/6 1 eth access down SFP not inserted 10G(D) —

snipped removed

N5K3# sh run int eth1/10
interface Ethernet1/10
switchport mode fex-fabric
fex associate 101

The Fabric Interfaces do not run a Spanning tree instance while the host interfaces do run BPDU guard and BPDU filter by default. The reason why the fabric interfaces do not run spanning tree is because they are backplane point to point interfaces.

By default, the FEX interfaces will send out a couple of BPDU’s on start-up.

N5K3# sh spanning-tree interface Ethernet1/10
No spanning tree information available for Ethernet1/10
N5K3#
N5K3#
N5K3# sh spanning-tree interface Eth101/1/1Vlan Role Sts Cost Prio.Nbr Type
—————- —- — ——— ——– ——————————–
VLAN0001 Desg FWD 2 128.1153 Edge P2p

N5K3#

N5K3# sh spanning-tree interface Eth101/1/1 detail

Port 1153 (Ethernet101/1/1) of VLAN0001 is designated forwarding

Port path cost 2, Port priority 128, Port Identifier 128.1153

Designated root has priority 32769, address 0005.9b1e.82fc

Designated bridge has priority 32769, address 0005.9b1e.82fc

Designated port id is 128.1153, designated path cost 0

Timers: message age 0, forward delay 0, hold 0

Number of transitions to forwarding state: 1

The port type is edge

Link type is point-to-point by default

Bpdu guard is enabled

Bpdu filter is enabled by default

BPDU: sent 11, received 0

N5K3# sh spanning-tree interface Ethernet1/10
No spanning tree information available for Ethernet1/10
N5K3#
N5K3#
N5K3# sh spanning-tree interface Eth101/1/1Vlan Role Sts Cost Prio.Nbr Type—————- —- — ——— ——– ——————————–
VLAN0001 Desg FWD 2 128.1153 Edge P2p

N5K3#

N5K3# sh spanning-tree interface Eth101/1/1 detail

Port 1153 (Ethernet101/1/1) of VLAN0001 is designated forwarding

Port path cost 2, Port priority 128, Port Identifier 128.1153

Designated root has priority 32769, address 0005.9b1e.82fc

Designated bridge has priority 32769, address 0005.9b1e.82fc

Designated port id is 128.1153, designated path cost 0

Timers: message age 0, forward delay 0, hold 0

Number of transitions to forwarding state: 1

The port type is edge

Link type is point-to-point by default

Bpdu guard is enabled

Bpdu filter is enabled by default

BPDU: sent 11, received 0

Issue the commands below to determine the transceiver type for the fabric ports and also the hosts ports for each fabric interface.

Command: “show interface fex-fabric“: displays all the Fabric Extender interfaces

Command: “show fex detail“: Shows detailed information about all FEXs. Including more recent log messages related to the FEX.

N5K3# show interface fex-fabric
Fabric Fabric Fex FEX
Fex Port Port State Uplink Model Serial
—————————————————————
101 Eth1/10 Active 3 N2K-C2232PP-10GE SSI16510AWF
— Eth1/11 Discovered 3 N2K-C2232PP-10GE SSI165204YC
N5K3#
N5K3#

N5K3# show interface Ethernet1/10 fex-intf

Fabric FEX

Interface Interfaces

—————————————————

Eth1/10 Eth101/1/1

N5K3#

N5K3# show interface Ethernet1/10 transceiver

Ethernet1/10

transceiver is present

type is SFP-H10GB-CU3M

name is CISCO-TYCO

part number is 1-2053783-2

revision is N

serial number is TED1530B11W

nominal bitrate is 10300 MBit/sec

Link length supported for copper is 3 m

cisco id is —

cisco extended id number is 4

N5K3# show fex detail

FEX: 101 Description: FEX0101 state: Online

FEX version: 5.1(3)N2(1c) [Switch version: 5.1(3)N2(1c)]

FEX Interim version: 5.1(3)N2(1c)

Switch Interim version: 5.1(3)N2(1c)

Extender Serial: SSI16510AWF

Extender Model: N2K-C2232PP-10GE, Part No: 73-12533-05

Card Id: 82, Mac Addr: f0:29:29:ff:02:02, Num Macs: 64

Module Sw Gen: 12594 [Switch Sw Gen: 21]

post level: complete

pinning-mode: static Max-links: 1

Fabric port for control traffic: Eth1/10

FCoE Admin: false

FCoE Oper: true

FCoE FEX AA Configured: false

Fabric interface state:

Eth1/10 – Interface Up. State: Active

Fex Port State Fabric Port

Eth101/1/1 Up Eth1/10

Eth101/1/2 Down None

Eth101/1/3 Down None

Eth101/1/4 Down None

snippet removed

Logs:

08/21/2014 10:00:06.107783: Module register received

08/21/2014 10:00:06.109935: Registration response sent

08/21/2014 10:00:06.239466: Module Online S

Now we quickly enable the second FEX connected to fabric interface E1/11.

N5K3(config)# int et1/11
N5K3(config-if)# switchport mode fex-fabric
N5K3(config-if)# fex associate 102
N5K3(config-if)# end
N5K3# sh fex
FEX FEX FEX FEX
Number Description State Model Serial

————————————————————————

101 FEX0101 Online N2K-C2232PP-10GE SSI16510AWF

102 FEX0102 Online N2K-C2232PP-10GE SSI165204YC

N5K3# show fex detail

FEX: 101 Description: FEX0101 state: Online

FEX version: 5.1(3)N2(1c) [Switch version: 5.1(3)N2(1c)]

FEX Interim version: 5.1(3)N2(1c)

Switch Interim version: 5.1(3)N2(1c)

Extender Serial: SSI16510AWF

Extender Model: N2K-C2232PP-10GE, Part No: 73-12533-05

Card Id: 82, Mac Addr: f0:29:29:ff:02:02, Num Macs: 64

Module Sw Gen: 12594 [Switch Sw Gen: 21]

post level: complete

pinning-mode: static Max-links: 1

Fabric port for control traffic: Eth1/10

FCoE Admin: false

FCoE Oper: true

FCoE FEX AA Configured: false

Fabric interface state:

Eth1/10 – Interface Up. State: Active

Fex Port State Fabric Port

Eth101/1/1 Up Eth1/10

Eth101/1/2 Down None

Eth101/1/3 Down None

Eth101/1/4 Down None

Eth101/1/5 Down None

Eth101/1/6 Down None

snippet removed

Logs:

08/21/2014 10:00:06.107783: Module register received

08/21/2014 10:00:06.109935: Registration response sent

08/21/2014 10:00:06.239466: Module Online Sequence

08/21/2014 10:00:09.621772: Module Online

FEX: 102 Description: FEX0102 state: Online

FEX version: 5.1(3)N2(1c) [Switch version: 5.1(3)N2(1c)]

FEX Interim version: 5.1(3)N2(1c)

Switch Interim version: 5.1(3)N2(1c)

Extender Serial: SSI165204YC

Extender Model: N2K-C2232PP-10GE, Part No: 73-12533-05

Card Id: 82, Mac Addr: f0:29:29:ff:00:42, Num Macs: 64

Module Sw Gen: 12594 [Switch Sw Gen: 21]

post level: complete

pinning-mode: static Max-links: 1

Fabric port for control traffic: Eth1/11

FCoE Admin: false

FCoE Oper: true

FCoE FEX AA Configured: false

Fabric interface state:

Eth1/11 – Interface Up. State: Active

Fex Port State Fabric Port

Eth102/1/1 Up Eth1/11

Eth102/1/2 Down None

Eth102/1/3 Down None

Eth102/1/4 Down None

Eth102/1/5 Down None

snippet removed

Logs:

08/21/2014 10:12:13.281018: Module register received

08/21/2014 10:12:13.283215: Registration response sent

08/21/2014 10:12:13.421037: Module Online Sequence

08/21/2014 10:12:16.665624: Module Online

Part 2. Fabric Interfaces redundancy

Static Pinning is when you pin a number of host ports to a fabric port. If the fabric port goes down so do the host ports that are pinned to it. This is useful when you want no oversubscription in the network.

Once the host port shut down due to a fabric port down event, the server if configured correctly should revert to the secondary NIC.

The “pinning max-link” divides the number specified in the command by the number of host interfaces to determine how many host interfaces go down if there is a fabric interface failure.

Now we shut down fabric interface E1/10, you can see that Eth101/1/1 has changed its operation mode to DOWN. The FEX has additional connectivity with E1/11 which remains up.

Enter configuration commands, one per line. End with CNTL/Z.
N5K3(config)# int et1/10
N5K3(config-if)# shu
N5K3(config-if)#
N5K3(config-if)# end
N5K3# sh fex detail
FEX: 101 Description: FEX0101 state: Offline
FEX version: 5.1(3)N2(1c) [Switch version: 5.1(3)N2(1c)]

FEX Interim version: 5.1(3)N2(1c)

Switch Interim version: 5.1(3)N2(1c)

Extender Serial: SSI16510AWF

Extender Model: N2K-C2232PP-10GE, Part No: 73-12533-05

Card Id: 82, Mac Addr: f0:29:29:ff:02:02, Num Macs: 64

Module Sw Gen: 12594 [Switch Sw Gen: 21]

post level: complete

pinning-mode: static Max-links: 1

Fabric port for control traffic:

FCoE Admin: false

FCoE Oper: true

FCoE FEX AA Configured: false

Fabric interface state:

Eth1/10 – Interface Down. State: Configured

Fex Port State Fabric Port

Eth101/1/1 Down Eth1/10

Eth101/1/2 Down None

Eth101/1/3 Down None

snippet removed

Logs:

08/21/2014 10:00:06.107783: Module register received

08/21/2014 10:00:06.109935: Registration response sent

08/21/2014 10:00:06.239466: Module Online Sequence

08/21/2014 10:00:09.621772: Module Online

08/21/2014 10:13:20.50921: Deleting route to FEX

08/21/2014 10:13:20.58158: Module disconnected

08/21/2014 10:13:20.61591: Offlining Module

08/21/2014 10:13:20.62686: Module Offline Sequence

08/21/2014 10:13:20.797908: Module Offline

FEX: 102 Description: FEX0102 state: Online

FEX version: 5.1(3)N2(1c) [Switch version: 5.1(3)N2(1c)]

FEX Interim version: 5.1(3)N2(1c)

Switch Interim version: 5.1(3)N2(1c)

Extender Serial: SSI165204YC

Extender Model: N2K-C2232PP-10GE, Part No: 73-12533-05

Card Id: 82, Mac Addr: f0:29:29:ff:00:42, Num Macs: 64

Module Sw Gen: 12594 [Switch Sw Gen: 21]

post level: complete

pinning-mode: static Max-links: 1

Fabric port for control traffic: Eth1/11

FCoE Admin: false

FCoE Oper: true

FCoE FEX AA Configured: false

Fabric interface state:

Eth1/11 – Interface Up. State: Active

Fex Port State Fabric Port

Eth102/1/1 Up Eth1/11

Eth102/1/2 Down None

Eth102/1/3 Down None

Eth102/1/4 Down None

snippet removed

Logs:

08/21/2014 10:12:13.281018: Module register received

08/21/2014 10:12:13.283215: Registration response sent

08/21/2014 10:12:13.421037: Module Online Sequence

08/21/2014 10:12:16.665624: Module Online

Port Channels can be used instead of static pinning between parent switch and FEX so in the event of a fabric interface failure all hosts ports remain active. However, the remaining bandwidth on the parent switch will be shared by all the host ports resulting in an increase in oversubscription.

Part 3. Fabric Extender Topologies

Straight-Through: The FEX is connected to a single parent switch. The servers connecting to the FEX can leverage active-active data plane by using host vPC.

Shutting down the Peer link results in ALL vPC member ports in the secondary peer become disabled. For this reason it is better to use a Dual Homed.

Dual Homed: Connecting a single FEX to two parent switches.

In active – active, a single parent switch failure does not affect the host’s interfaces because both vpc peers have separate control planes and manage the FEX separately.
For the remainder of the post we are going to look at Dual Homed FEX connectivity with host Vpc.

Full configuration:

N5K1:
feature lacp
feature vpc
feature fex
!
vlan 10
!

vpc domain 1

peer-keepalive destination 192.168.0.52

interface port-channel1

switchport mode trunk

spanning-tree port type network

vpc peer-link

interface port-channel10

switchport access vlan 10

vpc 10

interface Ethernet1/1

switchport access vlan 10

spanning-tree port type edge

speed 1000

interface Ethernet1/3 – 5

switchport mode trunk

spanning-tree port type network

channel-group 1 mode active

interface Ethernet1/10

switchport mode fex-fabric

fex associate 101

interface Ethernet101/1/1

switchport access vlan 10

channel-group 10 mode on

N5K2:

feature lacp

feature vpc

feature fex

vlan 10

vpc domain 1

peer-keepalive destination 192.168.0.51

interface port-channel1

switchport mode trunk

spanning-tree port type network

vpc peer-link

interface port-channel10

switchport access vlan 10

vpc 10

interface Ethernet1/2

switchport access vlan 10

spanning-tree port type edge

speed 1000

interface Ethernet1/3 – 5

switchport mode trunk

spanning-tree port type network

channel-group 1 mode active

interface Ethernet1/11

switchport mode fex-fabric

fex associate 102

interface Ethernet102/1/1

switchport access vlan 10

channel-group 10 mode on

The FEX do not support LACP so configure the port-channel mode to “ON”

The first step is to check the VPC peer link and general VPC parameters.

Command: “show vpc brief“. Displays the vPC domain ID, the peer-link status, the keepalive message status, whether the configuration consistency is successful, and whether peer-link formed or the failure to form

Command: “show vpc peer-keepalive” Displays the destination IP of the peer keepalive message for the vPC. The command also displays the send and receives status as well as the last update from the peer in seconds and milliseconds

N5K3# sh vpc brief
Legend:
(*) – local vPC is down, forwarding via vPC peer-link
vPC domain id : 1
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive

Configuration consistency status: success

Per-vlan consistency status : success

Type-2 consistency status : success

vPC role : primary

Number of vPCs configured : 1

Peer Gateway : Disabled

Dual-active excluded VLANs : –

Graceful Consistency Check : Enabled

vPC Peer-link status

———————————————————————

id Port Status Active vlans

— —- —— ————————————————–

1 Po1 up 1,10

vPC status

—————————————————————————-

id Port Status Consistency Reason Active vlans

—— ———– —— ———– ————————– ———–

10 Po10 up success success 10

N5K3# show vpc peer-keepalive

vPC keep-alive status : peer is alive

–Peer is alive for : (1753) seconds, (536) msec

–Send status : Success

–Last send at : 2014.08.21 10:52:30 130 ms

–Sent on interface : mgmt0

–Receive status : Success

–Last receive at : 2014.08.21 10:52:29 925 ms

–Received on interface : mgmt0

–Last update from peer : (0) seconds, (485) msec

vPC Keep-alive parameters

–Destination : 192.168.0.54

–Keepalive interval : 1000 msec

–Keepalive timeout : 5 seconds

–Keepalive hold timeout : 3 seconds

–Keepalive vrf : management

–Keepalive udp port : 3200

–Keepalive tos : 192

The trunk interface should be forwarding on the peer link and VLAN 10 must be forwarding and active on the trunk link. Take note if any vlans are on err-disable mode on the trunk.

N5K3# sh interface trunk
——————————————————————————-
Port Native Status Port
Vlan Channel
——————————————————————————–
Eth1/3 1 trnk-bndl Po1
Eth1/4 1 trnk-bndl Po1
Eth1/5 1 trnk-bndl Po1

Po1 1 trunking —

——————————————————————————–

Port Vlans Allowed on Trunk

——————————————————————————–

Eth1/3 1-3967,4048-4093

Eth1/4 1-3967,4048-4093

Eth1/5 1-3967,4048-4093

Po1 1-3967,4048-4093

——————————————————————————–

Port Vlans Err-disabled on Trunk

——————————————————————————–

Eth1/3 none

Eth1/4 none

Eth1/5 none

Po1 none

——————————————————————————–

Port STP Forwarding

——————————————————————————–

Eth1/3 none

Eth1/4 none

Eth1/5 none

Po1 1,10

——————————————————————————–

Port Vlans in spanning tree forwarding state and not pruned

——————————————————————————–

Eth1/3 —

Eth1/4 —

Eth1/5 —

Po1 —

——————————————————————————–

Port Vlans Forwarding on FabricPath

——————————————————————————–

N5K3# sh spanning-tree vlan 10

VLAN0010

Spanning tree enabled protocol rstp

Root ID Priority 32778

Address 0005.9b1e.82fc

This bridge is the root

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32778 (priority 32768 sys-id-ext 10)

Address 0005.9b1e.82fc

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type

—————- —- — ——— ——– ——————————–

Po1 Desg FWD 1 128.4096 (vPC peer-link) Network P2p

Po10 Desg FWD 1 128.4105 (vPC) Edge P2p

Eth1/1 Desg FWD 4 128.129 Edge P2p

Check the Port Channel database and determine the status of the port channel

N5K3# show port-channel database
port-channel1
Last membership update is successful
3 ports in total, 3 ports up
First operational port is Ethernet1/3
Age of the port-channel is 0d:00h:13m:22s
Time since last bundle is 0d:00h:13m:18s
Last bundled member is Ethernet1/5

Ports: Ethernet1/3 [active ] [up] *

Ethernet1/4 [active ] [up]

Ethernet1/5 [active ] [up]

port-channel10

Last membership update is successful

1 ports in total, 1 ports up

First operational port is Ethernet101/1/1

Age of the port-channel is 0d:00h:13m:20s

Time since last bundle is 0d:00h:02m:42s

Last bundled member is Ethernet101/1/1

Time since last unbundle is 0d:00h:02m:46s

Last unbundled member is Ethernet101/1/1

Ports: Ethernet101/1/1 [on] [up] *

To execute reachability tests, create an SVI on the first parent switch and run ping tests. You must first enable the feature set “feature interface-vlan”. The reason we create an SVI in VLAN 10 is because we need an interfaces to source our pings.

N5K3# conf t
Enter configuration commands, one per line. End with CNTL/Z.
N5K3(config)# fea
feature feature-set
N5K3(config)# feature interface-vlan
N5K3(config)# int vlan 10
N5K3(config-if)# ip address 10.0.0.3 255.255.255.0

N5K3(config-if)# no shu

N5K3(config-if)#

N5K3(config-if)# end

N5K3# ping 10.0.0.3

PING 10.0.0.3 (10.0.0.3): 56 data bytes

64 bytes from 10.0.0.3: icmp_seq=0 ttl=255 time=0.776 ms

64 bytes from 10.0.0.3: icmp_seq=1 ttl=255 time=0.504 ms

64 bytes from 10.0.0.3: icmp_seq=2 ttl=255 time=0.471 ms

64 bytes from 10.0.0.3: icmp_seq=3 ttl=255 time=0.473 ms

64 bytes from 10.0.0.3: icmp_seq=4 ttl=255 time=0.467 ms

— 10.0.0.3 ping statistics —

5 packets transmitted, 5 packets received, 0.00% packet loss

round-trip min/avg/max = 0.467/0.538/0.776 ms

N5K3# ping 10.0.0.10

PING 10.0.0.10 (10.0.0.10): 56 data bytes

Request 0 timed out

64 bytes from 10.0.0.10: icmp_seq=1 ttl=127 time=1.874 ms

64 bytes from 10.0.0.10: icmp_seq=2 ttl=127 time=0.896 ms

64 bytes from 10.0.0.10: icmp_seq=3 ttl=127 time=1.023 ms

64 bytes from 10.0.0.10: icmp_seq=4 ttl=127 time=0.786 ms

— 10.0.0.10 ping statistics —

5 packets transmitted, 4 packets received, 20.00% packet loss

round-trip min/avg/max = 0.786/1.144/1.874 ms

N5K3#

Do the same tests on the second Nexus 5K.

N5K4(config)# int vlan 10
N5K4(config-if)# ip address 10.0.0.4 255.255.255.0
N5K4(config-if)# no shu
N5K4(config-if)# end
N5K4# ping 10.0.0.10
PING 10.0.0.10 (10.0.0.10): 56 data bytes
Request 0 timed out

64 bytes from 10.0.0.10: icmp_seq=1 ttl=127 time=1.49 ms

64 bytes from 10.0.0.10: icmp_seq=2 ttl=127 time=1.036 ms

64 bytes from 10.0.0.10: icmp_seq=3 ttl=127 time=0.904 ms

64 bytes from 10.0.0.10: icmp_seq=4 ttl=127 time=0.889 ms

— 10.0.0.10 ping statistics —

5 packets transmitted, 4 packets received, 20.00% packet loss

round-trip min/avg/max = 0.889/1.079/1.49 ms

N5K4# ping 10.0.0.13

PING 10.0.0.13 (10.0.0.13): 56 data bytes

Request 0 timed out

Request 1 timed out

Request 2 timed out

Request 3 timed out

Request 4 timed out

— 10.0.0.13 ping statistics —

5 packets transmitted, 0 packets received, 100.00% packet loss

N5K4# ping 10.0.0.3

PING 10.0.0.3 (10.0.0.3): 56 data bytes

Request 0 timed out

64 bytes from 10.0.0.3: icmp_seq=1 ttl=254 time=1.647 ms

64 bytes from 10.0.0.3: icmp_seq=2 ttl=254 time=1.298 ms

64 bytes from 10.0.0.3: icmp_seq=3 ttl=254 time=1.332 ms

64 bytes from 10.0.0.3: icmp_seq=4 ttl=254 time=1.24 ms

— 10.0.0.3 ping statistics —

5 packets transmitted, 4 packets received, 20.00% packet loss

round-trip min/avg/max = 1.24/1.379/1.647 ms

Shut down one of the FEX links to the parent and you see that the FEX is still reachable via the other link that is in the port channel bundle.

N5K3# conf t
Enter configuration commands, one per line. End with CNTL/Z.
N5K3(config)# int Eth101/1/1
N5K3(config-if)# shu
N5K3(config-if)# end
N5K3#
N5K3#

N5K3# ping 10.0.0.3

PING 10.0.0.3 (10.0.0.3): 56 data bytes

64 bytes from 10.0.0.3: icmp_seq=0 ttl=255 time=0.659 ms

64 bytes from 10.0.0.3: icmp_seq=1 ttl=255 time=0.515 ms

64 bytes from 10.0.0.3: icmp_seq=2 ttl=255 time=0.471 ms

64 bytes from 10.0.0.3: icmp_seq=3 ttl=255 time=0.466 ms

64 bytes from 10.0.0.3: icmp_seq=4 ttl=255 time=0.465 ms

— 10.0.0.3 ping statistics —

5 packets transmitted, 5 packets received, 0.00% packet loss

round-trip min/avg/max = 0.465/0.515/0.659 ms

If you would like to futher your knowlege on VPC and how it relates to Data Center toplogies and more specifically, Cisco’s Application Centric Infrastructure (ACI), you can check out my following training courses on Cisco ACI. Course 1: Design and Architect Cisco ACI, Course 2: Implement Cisco ACI, and Course 3: Troubleshooting Cisco ACI,

Redundant links with Virtual PortChannels

August 18, 2014

by Matt Conran Blog

Data Centre Networking · Resilience · Routing

Redundant Links with Virtual PortChannels

Modern data centres need more than a backup connection. They need resilient paths, effective bandwidth utilisation and predictable behaviour when links or switches fail. Virtual PortChannels (vPC) help achieve active-active Layer 2 connectivity by allowing a downstream device to form a port-channel across two compatible Cisco Nexus switches.

Traditional designs can leave redundant paths blocked by Spanning Tree Protocol (STP), while a poorly planned aggregation design can create uneven traffic distribution or unexpected failure conditions. Port channels, LACP and multi-chassis link aggregation provide different mechanisms for building resilient connectivity—but each has its own operational requirements and failure modes.

This guide examines how these mechanisms work, how to verify them, and how to investigate what happens when a member link, peer-link or switch becomes unavailable. It also connects Layer 2 redundancy to Layer 3 forwarding and BGP, where routing decisions and equal-cost multipath (ECMP) can provide additional resilience across a data-centre fabric.

What you will learn

01 / FOUNDATIONS

Port-channels and LACP

Understand physical member links, logical interfaces, negotiation modes and link-level redundancy.

02 / ARCHITECTURE

vPC operation

Explore peer-links, keepalive connectivity, consistency checks, primary and secondary roles, and orphan ports.

03 / TRAFFIC

Hashing and load balancing

Learn why multiple active links do not guarantee perfectly even traffic distribution and how to identify polarization.

04 / ROUTING

BGP and ECMP

Connect resilient Layer 2 access with routed leaf-spine fabrics, BGP path selection, next-hop reachability and multipath forwarding.

05 / OPERATIONS

Failure investigation

Trace link failures, peer-link loss, routing changes and recovery using a structured troubleshooting workflow.

06 / PRACTICE

Design and verification labs

Apply the concepts to topology walkthroughs, failure scenarios, configuration checks and design decisions.

Key design principle: vPC and BGP solve different problems. vPC provides multi-chassis Layer 2 link aggregation; BGP exchanges routing information and can support multipath forwarding in routed designs. A resilient architecture must account for both the data path and the control-plane behaviour during failures.

Cisco vPC EtherChannel LACP STP BGP ECMP Leaf-Spine Fabrics Network Troubleshooting

Network Insight · Article Guide

Explore the vPC Learning Guide

Navigate from EtherChannel fundamentals to vPC architecture, BGP and ECMP integration, failure investigation and practical labs. Select any section to jump directly to its content.

YOUR READING PATH · 9 SECTIONS Ready to explore

Section 01 · Foundations

Redundant Links, EtherChannel and vPC Fundamentals

Redundancy is a core requirement in data-centre networking. Applications, servers and network services need connectivity to continue when an individual cable, interface or switch fails. However, adding physical links alone does not guarantee that all available capacity will be used efficiently or that every failure will be handled correctly.

A robust design must combine physical connectivity with mechanisms that control forwarding, distribute traffic and respond to failures. EtherChannel groups multiple Ethernet links into one logical interface, while Cisco Virtual PortChannel (vPC) extends the port-channel concept across a pair of supported switches.

Why a single link is not enough

A single connection between a server and a switch creates a potential single point of failure. If the cable, interface or switch port becomes unavailable, traffic using that connection may be interrupted until connectivity is restored or an alternative path is selected.

Multiple links can improve capacity and resilience, but their behaviour depends on how they are configured. Independent parallel Layer 2 links can create loops, while designs relying on Spanning Tree Protocol (STP) may place some redundant links into a non-forwarding state for a given VLAN. Link aggregation provides another approach by treating compatible physical links as one logical connection.

Physical redundancy

Multiple cables and interfaces reduce dependence on a single physical connection. Diversity of line cards, paths and power domains can provide further protection when designed appropriately.

Logical aggregation

A port-channel presents a bundle of compatible member links as one logical interface to supported protocols and connected devices.

Multi-switch resilience

vPC allows a supported downstream device to connect its port-channel to two separate Cisco Nexus peers, subject to the platform's design and configuration requirements.

EtherChannel versus vPC

These terms are related, but they describe different parts of the design. A conventional port-channel aggregates links between two endpoints. A vPC allows the two endpoints of a downstream logical port-channel to reside on separate switches within a configured vPC domain.

Technology	Primary purpose	Design consideration
EtherChannel / port-channel	Aggregate multiple physical links into one logical interface.	Member links must meet compatibility requirements and share a supported aggregation configuration.
LACP	Negotiate and maintain a link aggregation group.	Active/passive modes and configuration compatibility affect whether the bundle forms.
vPC	Provide a multi-switch port-channel to a connected device.	Peer-link health, keepalive, consistency and failure handling must be designed and verified.
STP	Prevent Layer 2 forwarding loops in bridged networks.	It remains relevant to the broader topology; vPC does not remove every STP requirement.

How a port-channel uses multiple links

When a port-channel is operational, the switch selects a physical member for each eligible traffic flow using a platform-dependent hashing algorithm. Depending on the configured inputs, the hash may use source and destination MAC addresses, IP addresses or transport-layer ports. Different flows can therefore use different members.

Configure the member interfaces. Ensure the physical interfaces have compatible settings and are eligible to join the same aggregation group.
Establish the logical bundle. Configure the port-channel and, where used, compatible LACP settings. Verify that the expected members become active.
Distribute traffic. The switch hashes eligible flows onto member links. This is normally flow-based distribution, not a guarantee that every packet takes the next available link.
Handle a member failure. Traffic can use surviving active members if the bundle remains operational, although capacity is reduced and some flows may be temporarily disrupted.

Engineering insight: Aggregating four links does not mean every individual flow receives four times the bandwidth. A particular flow is typically assigned to one member at a time, and actual throughput depends on hashing, traffic patterns, link speeds and the capabilities of the platform.

Where BGP enters the picture

Port-channels and vPC primarily address link aggregation and Layer 2 connectivity. In routed data-centre fabrics, BGP can provide a separate layer of resilience by exchanging reachability information between routers and supporting equal-cost multipath (ECMP) where the design, configuration and platform permit it.

This distinction matters during failures. A physical member failure may be handled within a port-channel without the routing protocol seeing a change to the logical interface. By contrast, a routed link or next-hop failure may trigger routing updates and a new forwarding decision. The recovery process depends on which component failed and how the control and data planes are configured.

Key takeaway: Redundant links, EtherChannel, LACP, vPC and BGP are complementary tools—not interchangeable technologies. Start by identifying the failure domain you need to protect, then select the appropriate mechanism and verify its actual failure behaviour.

Interactive learning · Module 01

From Redundant Links to vPC

Explore three ways to connect a server or downstream switch to the network. Compare a standalone physical link, a conventional EtherChannel and a virtual PortChannel across two switches.

Switch attachment Two switches

Logical link One coordinated vPC

Resilience model Multi-switch

vPC: a multi-switch port-channel

A downstream device can form a port-channel across two vPC peers. The switches coordinate through the vPC architecture, allowing supported topologies to use links to both switches without treating them as unrelated standalone connections.

Important: The design still depends on peer-link health, keepalive behavior, configuration consistency, and the platform's documented failure handling.

Knowledge check

What is the key architectural difference between a conventional EtherChannel and a vPC?

Educational illustration only. The link-failure control changes the diagram, not a real network device. Exact behavior depends on the vendor platform, topology and configuration.

Section 02 · Architecture

Understanding the vPC Peer-Link, Keepalive and Consistency

A vPC domain consists of two cooperating switches that maintain separate control planes while presenting a coordinated multi-chassis port-channel to connected devices. This arrangement enables active forwarding across supported links, but it also introduces important synchronization and failure-handling requirements.

Three concepts are central to understanding vPC operation: the peer-link, the peer-keepalive link and consistency checks. They have different purposes. The peer-link carries required inter-peer traffic and synchronization; the keepalive helps peers determine whether the other switch is alive; and consistency checks identify configuration differences that could compromise correct operation.

1. The main components of a vPC domain

Data and synchronization

Peer-link

A logical link between the two vPC peers. It carries synchronization and selected traffic required by vPC operation. Its capacity, redundancy and failure behaviour must be planned carefully.

Peer liveness

Peer-keepalive

A separate heartbeat path used to help distinguish a peer failure from a peer-link failure. It should be designed to remain available independently of the peer-link where practical.

Configuration safety

Consistency checks

Checks compare relevant configuration parameters across peers. Incompatible settings can prevent a vPC from coming up or cause a member to be suspended, depending on the parameter and platform.

2. How the peer-link supports the domain

The peer-link is not simply another downstream traffic connection. It supports communication between the vPC peers and provides a path for certain traffic when the normal forwarding path is unavailable. The network should not depend on it as the routine path for all end-host traffic.

vPC Peer A Independent control plane

Peer-link Synchronization and selected inter-peer traffic

vPC Peer B Independent control plane

Conceptual representation only. Actual peer-link requirements and forwarding behaviour depend on the Cisco Nexus platform, software release and configuration.

Under normal conditions, traffic from a downstream device should use its active vPC member links. If a peer loses its local member links for a particular vPC, the surviving peer and peer-link may be involved in forwarding traffic, subject to the applicable vPC forwarding rules. This is one reason peer-link capacity and failure scenarios matter.

Design principle: Do not treat the peer-link as a substitute for correctly designed downstream and routed connectivity. Understand which traffic can cross it, how much capacity is required, and what the platform does when peer-link members fail.

3. Why the peer-keepalive is separate

The keepalive mechanism helps the peers determine whether the other switch is reachable when peer-link communication is interrupted. That distinction is important: losing the peer-link does not automatically mean that the other switch has failed.

Observed condition	What it may indicate	Engineering response
Peer-link up, keepalive reachable	Normal peer communication, assuming the domain is otherwise healthy.	Verify peer-link members, consistency status and vPC state.
Peer-link down, keepalive reachable	The peers may still be alive, but their main synchronization path is unavailable.	Investigate peer-link interfaces, port-channel state and the platform's documented failure response.
Peer-link down, keepalive also unavailable	The peers may be unable to distinguish a peer failure from a communication partition.	Check peer reachability, keepalive path independence and the documented dual-active protection behaviour.
One peer is genuinely down	The surviving peer may continue forwarding supported traffic, depending on the failure and configuration.	Verify vPC state, downstream connectivity and recovery after the peer returns.

These are diagnostic patterns, not a universal state machine. The exact response to peer-link and keepalive failures is platform- and configuration-dependent, so operational decisions must be checked against the relevant vendor documentation.

4. Consistency checks: preventing incompatible configurations

Because both switches participate in a coordinated logical topology, important configuration must be compatible. Differences in VLAN membership, port-channel parameters, trunk settings or other vPC-relevant configuration can affect operation. Not every parameter is treated identically: some inconsistencies are more serious than others, and the resulting action depends on the specific check.

Confirm both peers belong to the intended vPC domain.
Verify peer-link status and its member interfaces.
Check vPC consistency output for global and per-vPC issues.
Compare VLANs, allowed VLAN lists, MTU and relevant interface settings.
Confirm port-channel membership and LACP state on both peers.
Review platform-specific requirements before changing a live domain.

5. Primary and secondary roles

vPC peers have primary and secondary roles for particular operational decisions. These roles do not mean that the secondary peer is simply an idle standby switch: both peers can forward traffic over their supported active vPC links. Role election, role changes and behavior during failure must be understood in the context of the platform and release.

The design should document the intended role priorities and the expected behavior during reloads, peer-link loss, keepalive loss and recovery. Avoid assuming that a role label alone predicts which traffic will be forwarded after every possible failure.

6. Orphan ports and single-homed devices

An orphan port is typically a port connected to a device through only one vPC peer rather than through a vPC to both peers. Such devices deserve special attention because a peer or peer-link failure can leave the device with different connectivity from a dual-homed server.

Identify orphan-connected devices, determine which peer they depend on, and test their reachability under the failure conditions relevant to the design. Where necessary, use supported platform-specific protection mechanisms and documented operational procedures.

Key takeaway: Reliable vPC operation depends on more than bringing up a port-channel. The peer-link, keepalive, consistency state, role behavior and orphan-port exposure must all be understood together. These are the foundations for the failure investigations covered later in this guide.

Interactive learning · Module 02

Inside the vPC Peer Relationship

The peer-link and peer-keepalive serve different purposes. Explore how their states differ and why you must investigate both before drawing conclusions about a vPC failure.

Peer-link Operational

Keepalive Operational

Investigation priority Routine monitoring

Normal operation

The peer-link supports the vPC peer relationship, while the keepalive path provides a separate means of checking peer liveness. Confirm that both paths and the vPC consistency status are healthy.

What to investigate: Peer status, peer-link members, keepalive reachability and consistency checks.

Investigation checklist

Use these checks to guide your diagnosis for the selected condition.

Confirm the vPC domain and peer state. Verify peer-link interfaces and port-channel status. Confirm peer-keepalive reachability. Review consistency status, logs and affected member ports.

Example verification commands

For a Cisco NX-OS-style environment, relevant commands may include:

show vpc
show port-channel summary
show interface port-channel 10
show logging

Command availability and output vary by platform and release. Use the appropriate vendor documentation for peer-keepalive and consistency verification.

Knowledge check

Why is it important to check the peer-keepalive path separately from the peer-link?

This diagram is a conceptual teaching aid, not a device-state simulator. A real peer-link or keepalive failure can have platform-specific consequences. Never infer the exact forwarding outcome from a diagram alone.

Section 03 · Traffic Engineering

Load Balancing, Hashing and Traffic Polarization

A port-channel can provide multiple active physical links, but those links will not necessarily carry equal amounts of traffic. Switches commonly use a hashing algorithm to select a member link for each eligible flow. The result depends on the traffic pattern, the selected hash inputs and the implementation supported by the platform.

Understanding this distinction is essential when designing redundant data-centre connections. A bundle may be operational and have all its members up, yet one link can become heavily utilized while other members remain comparatively idle. This uneven distribution is often described as traffic polarization or hash imbalance.

1. How port-channel hashing works

When a frame or packet is eligible for forwarding through an aggregated interface, the switch evaluates selected header fields. It uses those values to calculate a hash and map the result to one of the available physical members.

Incoming flow Traffic arrives at the switch

Hash inputs Selected MAC, IP or transport fields

Hash result Algorithm maps the flow to a member

Physical link Selected member forwards the traffic

Simplified conceptual model. Exact hash algorithms, fields, seed values and member-selection behavior vary by platform and software release.

2. Common hash inputs

Depending on the platform and configuration, the hashing algorithm can use Layer 2, Layer 3 or Layer 4 information. A richer set of inputs can help distinguish traffic flows that share the same source or destination address.

Hash input	What it identifies	Potential limitation
Source MAC	The originating Layer 2 address.	Many flows from one source can share the same value.
Destination MAC	The destination Layer 2 address.	Traffic directed to one next-hop MAC can concentrate on a subset of members.
Source and destination IP	The Layer 3 endpoints of a flow.	A workload with a small number of IP endpoint pairs may not spread evenly.
TCP/UDP ports	Transport-layer conversations between endpoints.	Availability and use of these fields depend on the platform, packet type and configured hashing policy.
Combined fields	A broader flow identity using multiple header values.	More inputs can improve distribution but cannot guarantee perfectly balanced utilization.

3. Why traffic polarization occurs

Hashing is not the same as sending each new packet to whichever link is least busy. It is generally designed to keep a flow on a consistent member, helping preserve packet ordering. As a result, the number of flows, their sizes and the distribution of their hash values influence the load on each member.

Uneven flow sizes

A few high-volume transfers can consume most of one member's capacity, even when the total number of flows appears balanced.

Limited hash diversity

If many flows share the same relevant header fields, they may map to the same member or a narrow subset of members.

Multiple hashing stages

Traffic crossing several aggregation or ECMP stages can experience correlated distribution effects, leaving some paths busier than others.

Important: A four-member port-channel does not guarantee that every flow receives one quarter of the bandwidth. Nor does a 25% share of the flow count guarantee a 25% share of bytes. Always evaluate actual traffic volume and member utilization.

4. Choosing and validating a hashing policy

Hashing configuration should reflect the traffic that the network actually carries. A policy suitable for many server-to-server flows may perform poorly when most traffic is concentrated on a small number of destinations. The available choices and their effects must be verified against the switch platform's documentation.

Identify the main traffic sources, destinations and application flow patterns.
Check which hash inputs and load-balancing options the platform supports.
Measure traffic rates and errors on individual physical members, not only the logical port-channel.
Compare flow distribution with byte counts, packet counts and interface utilization.
Test whether the selected policy improves distribution without creating unexpected forwarding behavior.
Recheck the distribution after link failures, workload changes or configuration updates.

5. The relationship between port-channel hashing and BGP ECMP

Port-channel hashing and BGP ECMP both distribute traffic across multiple forwarding choices, but they operate at different levels. Port-channel hashing chooses a physical member of an aggregated link. ECMP chooses between eligible equal-cost next hops in the routing and forwarding system.

In a routed leaf-spine fabric, a packet might be assigned to an ECMP next hop and then traverse an aggregated physical connection. Each stage may apply its own hashing behavior. Good distribution at one stage does not guarantee good distribution end to end.

Mechanism	Selection being made	What to investigate
Port-channel hashing	Which physical member carries a flow.	Member utilization, hash inputs, link capacity and traffic skew.
BGP ECMP	Which eligible equal-cost next hop is used.	Installed routes, eligible paths, next-hop resolution and forwarding-table programming.
End-to-end forwarding	The combined path through the network.	Traffic distribution at each stage, congestion points and the effect of failures.

6. Diagnosing an imbalanced bundle

Begin by confirming that the port-channel and its expected members are operational. Then examine per-member traffic counters over the same measurement interval. Look for sustained utilization differences, errors, drops or unexpected changes after a member or path failure.

If the physical links are healthy but the load remains uneven, compare the observed distribution with the workload's flow sizes and the configured hashing inputs. Where the design also uses routed ECMP, inspect route and next-hop state separately rather than assuming that the port-channel explains every imbalance.

Key takeaway: Multiple active links create forwarding choices, not automatic load equality. Effective load balancing comes from understanding hash behavior, measuring real traffic, and verifying distribution at both the link-aggregation and routing layers.

Interactive learning · Module 03

Explore Port-Channel Hashing

A port-channel usually selects an egress member using a hash of configured packet fields. The goal is to distribute flows across available links while keeping packets in a flow on a consistent path. Change the inputs and observe this illustrative model.

Hash input profile

Traffic profile

Active members 4 / 4

Flow distribution Relatively spread

Model status Normal

Illustrative load by member

The bars represent a modelled share of traffic, not measured link utilisation. A real switch uses platform-specific hash algorithms and forwarding details.

Why hashing matters

Diverse flows can spread across multiple members. The distribution is not guaranteed to be perfectly even, and a single flow is generally not split packet-by-packet across every member.

Knowledge check

Why can a port-channel show uneven utilisation even when all member links are up?

Educational model only: member selection and load percentages are illustrative, not a reproduction of a specific vendor's hashing algorithm. Hash fields, resilient hashing, member selection and failure behavior depend on platform and configuration.

Section 04 · BGP and Routed Resilience

BGP and Layer 3 Routing in the Data Centre

Virtual PortChannels provide resilient Layer 2 connectivity by allowing a device to form a logical port-channel across two cooperating switches. BGP addresses a different requirement: exchanging routing information between routers and selecting eligible paths through a Layer 3 network. In modern data centres, both technologies may be used in the same architecture, but they solve different problems.

Understanding the boundary between Layer 2 redundancy and Layer 3 routing is important when designing leaf-spine fabrics, connecting server networks to routed infrastructure, or investigating a failure that affects only part of the forwarding path.

1. A leaf-spine fabric using BGP

In a common routed leaf-spine design, leaf switches connect to servers, storage or other endpoint networks. Spine switches provide routed connectivity between the leaves. BGP can exchange reachability information across these links and support multiple eligible paths through the fabric.

Server group A Endpoint networks

Server group B Endpoint networks

↓ Access connections

Leaf 1 Server-facing and routed interfaces

Leaf 2 Server-facing and routed interfaces

↕ Routed leaf-to-spine links · BGP sessions

Spine 1 Routed transit

Spine 2 Routed transit

Conceptual topology only. A production fabric normally has routed links from each leaf to each spine. The diagram does not represent every physical connection.

With multiple spine switches, a leaf may learn multiple eligible routes to a destination. If the routing policy and platform permit equal-cost multipath forwarding, the switch can install multiple next hops and distribute traffic across them.

2. BGP, ECMP and next-hop selection

BGP route exchange

BGP advertises network reachability and path attributes between peers. Policy and best-path processing determine which paths are selected or made eligible for multipath forwarding.

ECMP forwarding

Equal-Cost Multipath can use multiple eligible next hops for a destination when the routing system and forwarding platform support and install those paths.

Next-hop resolution

A route is useful only when its next hop can be resolved and programmed into the forwarding system. Routing and interface state must be checked together.

Important distinction: BGP does not automatically mean that all available paths will carry traffic. Route policy, path attributes, next-hop reachability, multipath configuration and platform capabilities all influence the forwarding result.

3. vPC and BGP solve different problems

A vPC pair presents a multi-chassis Layer 2 aggregation capability to a connected device. BGP exchanges Layer 3 routes between routing peers. One technology does not replace the other, and their relationship depends on the overall network design.

Design concern	vPC / port-channel	BGP / ECMP
Primary purpose	Redundant Layer 2 connectivity and aggregated links.	Route exchange and Layer 3 path selection.
Forwarding choice	Selects a physical member within a logical link.	Selects among eligible routed next hops.
Failure investigation	Check member state, peer-link state, consistency and port-channel counters.	Check BGP sessions, routes, next hops, route policy and forwarding-table state.
Design trade-off	Requires careful multi-chassis coordination and failure handling.	Requires sound routing policy, path design and convergence planning.

For example, a server may connect redundantly to a vPC pair while those switches participate in a separate routed fabric. Alternatively, a design may use routed server-facing links and ECMP without relying on a vPC pair for that connection. The appropriate choice depends on the desired failure domains, operational model and platform support.

4. What happens when a link or path fails?

Resilience depends on which component fails and how that failure is detected. A physical member failure inside a port-channel is different from the loss of a routed leaf-to-spine link, a BGP session failure, or the loss of an entire switch.

Failure event	What may happen	What to verify
One port-channel member fails	Traffic may continue over remaining eligible members, with reduced aggregate capacity.	Member state, errors, capacity and traffic redistribution.
One routed fabric link fails	Routing and forwarding may use another eligible path if one is available and installed.	Interface state, route changes, next-hop programming and packet loss.
A BGP session fails	Routes learned through that session may be withdrawn or otherwise changed according to protocol state and policy.	Session state, route withdrawals, alternate paths and convergence time.
A switch fails	Recovery depends on the topology, remaining paths, gateway design and failure-detection mechanisms.	Reachability, redundant paths, gateway behavior and application impact.

These are general outcomes, not guarantees. Actual behavior depends on the topology, platform, configuration, failure-detection timers and the availability of alternate forwarding paths.

5. A practical BGP and ECMP verification workflow

When traffic does not follow the expected path, separate the investigation into routing control-plane checks and forwarding data-plane checks. A BGP session can be established while a particular destination is missing, rejected by policy or not installed for multipath forwarding.

Check adjacency: Confirm the expected BGP peers are established and inspect recent session changes.
Check route reception: Verify that the expected prefix is received and accepted by policy.
Check best-path and multipath eligibility: Determine why paths are selected, rejected or excluded from ECMP.
Check next-hop reachability: Confirm that each intended next hop is resolvable.
Check forwarding state: Verify which next hops are actually installed in the routing or forwarding table.
Measure traffic: Inspect per-interface counters, drops, utilization and the effects of a controlled failure test.

Engineering takeaway: vPC provides one form of Layer 2 resilience; BGP and ECMP provide routed reachability and multipath forwarding. Reliable data-centre design requires engineers to understand each layer, test its failure behavior and verify the complete forwarding path.

Interactive learning · Module 04

BGP Route Propagation and ECMP

Follow an illustrative route advertisement through a data-centre fabric. Then withdraw one route and inspect how the available paths change. BGP selects routes; ECMP can allow multiple eligible next hops to participate in forwarding.

Prefix 10.10.10.0/24

Available paths 2 illustrative paths

Current stage Route origin

Step 1 — A prefix is originated

A network advertises 10.10.10.0/24 into the routing system. The receiving router must learn the route before it can evaluate the path and install an eligible next hop.

Engineering focus: Confirm the route exists at the origin and that the intended BGP session is established.

Path status

Path A · via Spine 1Available in this illustrative scenario

Path B · via Spine 2Available in this illustrative scenario

What to verify on a real network

Use commands appropriate to your router OS and design:

show ip bgp summary
show ip bgp 10.10.10.0/24
show ip route 10.10.10.0/24
show ip cef 10.10.10.0/24

These are illustrative Cisco IOS-style examples; exact syntax and ECMP inspection commands vary by platform.

Knowledge check

What is the key difference between BGP path selection and ECMP forwarding?

This is a conceptual learning model, not a live BGP implementation. ECMP depends on the routes accepted, next-hop resolution, routing policy, platform capabilities and configuration. The diagram does not imply that every displayed link is a BGP adjacency in every design.

Section 05 · Resilience Engineering

Failure Scenarios, Redundancy and Network Convergence

Redundancy is only valuable when the network can respond correctly to a failure. A vPC design may provide multiple physical connections, while BGP and ECMP provide alternative routed paths. However, each mechanism detects different events and follows its own recovery behavior. Engineers must understand how those mechanisms interact during a real outage.

The goal is not simply to keep every link green. It is to preserve reachability, maintain acceptable application performance and restore the intended forwarding state within an acceptable period.

1. Understand the failure domain

Before troubleshooting, identify which component has failed. A single physical member, an entire port-channel, a vPC peer-link, a peer switch or a routed BGP adjacency represents a different failure domain. The observed symptom alone may not identify the root cause.

Physical-link failure

One member stops forwarding. If the logical port-channel remains operational, traffic may continue over the remaining active members, but total capacity is reduced.

vPC peer-link failure

The two peers lose their inter-peer connection. Peer-keepalive status and platform-specific protection behavior become important when determining whether forwarding should continue.

Routed-path failure

A routed interface, next hop or BGP session may become unavailable. Traffic can use an alternate path only if one is eligible, reachable and installed for forwarding.

2. What happens during common failures?

Failure scenario	Expected consideration	Engineering checks
One member link fails	Remaining port-channel members may continue carrying traffic, with less available bandwidth.	Member state, interface errors, utilization and port-channel status.
All members of a logical link fail	The logical connection may go down, triggering another forwarding or routing response depending on the topology.	Logical interface state, alternate paths, route changes and endpoint reachability.
vPC peer-link fails	Peer behavior depends on keepalive reachability, peer roles, configuration and platform protection mechanisms.	Peer-link state, keepalive status, consistency state and relevant system logs.
One vPC peer fails	Surviving connectivity depends on the remaining peer, attached links, gateway design and failure handling.	Peer status, server-facing links, gateway reachability and application traffic.
BGP adjacency fails	Routes learned through that session may be withdrawn or changed; alternate forwarding depends on available paths and convergence.	BGP session state, route table, next hops and forwarding-table entries.
Uneven traffic after recovery	Traffic may redistribute differently because of changed members, paths or flow hashes.	Per-member counters, ECMP next hops, drops and traffic distribution.

Design warning: Do not assume that every failure triggers immediate, lossless failover. Detection timers, protocol behavior, hardware programming, topology and configuration all influence recovery. Validate the behavior on the actual platform.

3. Control-plane convergence versus data-plane forwarding

The control plane detects and communicates changes in network state. The data plane forwards packets using the currently programmed forwarding information. These processes are related, but they are not identical and may not complete at exactly the same time.

Failure detection

A physical interface, peer relationship or routing session detects a change through its configured detection mechanism.

State and route updates

The affected protocol or subsystem updates its state. A routing change may cause paths to be withdrawn, recalculated or selected.

Forwarding adjustment

Hardware or software forwarding state is updated to use the remaining eligible links or next hops.

Traffic and application recovery

Traffic resumes along the available path, but packet loss, latency or reduced capacity may remain during or after the transition.

This sequence is a conceptual troubleshooting model. Some systems react locally before routing convergence, and implementations can overlap or reorder these stages.

4. Why alternate paths do not guarantee zero packet loss

A redundant path may exist without being immediately usable. The failure must be detected, the relevant forwarding decision must change, and the alternate path must be capable of carrying the traffic. A routing protocol can also have a valid alternate route that is not eligible for installation under the current policy or multipath rules.

Capacity: Remaining links must be able to accommodate the shifted traffic load.
Reachability: Alternate next hops and destinations must remain reachable.
Convergence: Detection and forwarding updates must complete within the application's tolerance.
Consistency: vPC peers and connected devices must maintain a valid forwarding arrangement.
Asymmetric paths: Forward and return traffic may follow different paths, which can affect stateful devices or troubleshooting.
Failure correlation: Redundant links should not share an avoidable single point of failure, such as one cable path or power domain.

5. A structured failure-testing workflow

Controlled testing is more reliable than assuming redundancy works because the configuration appears correct. Establish a baseline, introduce one planned failure at a time, measure the impact and restore the original state before testing the next scenario.

Capture the baseline

Record interface state, port-channel membership, vPC health, BGP sessions, installed routes, traffic counters and application reachability.

Introduce a controlled failure

In an approved test window or lab, disable one selected link or simulate a planned adjacency failure. Avoid changing multiple components at once.

Measure the impact

Track packet loss, interruption duration, latency, route changes, member utilization and any application errors.

Restore and compare

Confirm that the failed component recovers, the intended paths return, and traffic distribution returns to an acceptable state.

6. Failure testing checklist

Confirm the intended redundant links and alternate routes are present before testing.
Verify that the failure is detected by the expected mechanism.
Measure both control-plane changes and actual data-plane traffic.
Check that remaining links do not become overloaded.
Confirm gateway and endpoint reachability throughout recovery.
Review logs and counters for errors that remain after the failure is cleared.
Document the measured recovery time and compare it with the service requirement.

Key takeaway: Resilience must be demonstrated, not assumed. A robust design combines independent failure domains, valid alternate forwarding paths, predictable convergence and repeatable tests that measure the impact on real traffic.

Interactive learning · Module 05

Failure Investigation and Convergence

Redundant paths help a network survive failures, but they do not guarantee zero packet loss. Explore common failures, distinguish physical and routing symptoms, and follow a structured investigation before deciding whether service has recovered.

Selected fault Member link

Expected concern Traffic shift

Recovery status Needs verification

Member-link failure

One physical member of a port-channel fails. If other members remain operational, the port-channel may continue forwarding, but capacity falls and some flows may be disrupted during detection and remapping.

First investigation: Check physical interface state, port-channel membership, errors and traffic counters.

Investigation checklist

Work through these checks for the selected failure. Tick them off as you investigate.

Confirm the reported fault and its scope. Inspect interface, port-channel or routing state. Confirm alternate paths and next-hop availability. Verify traffic recovery with counters and end-to-end tests.

Example verification commands

show interface status
show port-channel summary
show vpc
show interface counters errors
show ip bgp summary
show ip route

These are illustrative commands; syntax and output vary by platform and operating-system version.

Knowledge check

After a redundant path fails, what is the strongest evidence that service has recovered?

Failure behavior depends on the platform, topology, configuration, failure-detection timers and routing design. This teaching widget illustrates investigation priorities; it does not predict exact convergence time or guarantee a particular forwarding outcome.

Section 06 · Architecture Decisions

vPC vs Layer 3 Multipathing and BGP ECMP

Both Virtual PortChannels and routed multipathing can improve network resilience, but they achieve it in different ways. vPC allows a connected device to use a logical Layer 2 port-channel across two cooperating switches. Layer 3 multipathing uses multiple eligible routed next hops to forward traffic through the network.

The design decision is not simply which technology is better. It is where the network needs Layer 2 adjacency, where routing boundaries should exist, and how the design should respond when links, switches or routing adjacencies fail.

1. Two different approaches to redundancy

Approach A — vPC

A connected device forms one logical port-channel using physical links to two cooperating switches. The vPC peers coordinate relevant forwarding and consistency behavior.

Approach B — Layer 3 ECMP

A routed device has multiple eligible next hops for a destination. The forwarding system distributes flows across installed paths according to its multipath and hashing implementation.

Server Dual-connected endpoint

Two physical links · One logical port-channel

vPC Peer A Coordinated Layer 2 forwarding

vPC Peer B Coordinated Layer 2 forwarding

Alternative model: routed links and multiple eligible next hops

Leaf 1 Routed next hop

Leaf 2 Routed next hop

Conceptual comparison, not a single combined topology. Actual connectivity and routing behavior depend on the chosen design.

2. Compare the forwarding models

Design dimension	vPC / Layer 2 aggregation	Layer 3 multipathing / ECMP
Forwarding model	Aggregated Layer 2 connectivity across two cooperating switches.	Routing and forwarding across multiple eligible next hops.
Primary dependency	vPC peer coordination, peer-link health, consistency and member state.	Route availability, next-hop reachability and multipath eligibility.
Failure handling	Depends on port-channel membership and vPC-specific protection behavior.	Depends on failure detection, route changes and the availability of alternate installed paths.
Traffic distribution	Hashing selects a physical port-channel member.	Hashing selects among eligible routed next hops.
Layer 2 scope	Can support a Layer 2 segment spanning the vPC-connected infrastructure.	Routed boundaries limit the need to extend a Layer 2 segment across the fabric.
Operational focus	Peer state, port-channel health and Layer 2 forwarding consistency.	BGP state, route policy, next-hop resolution and forwarding-table programming.

3. Where BGP ECMP fits in a leaf-spine fabric

In a routed leaf-spine architecture, each leaf commonly has routed connections to multiple spine switches. BGP can exchange reachability information over those connections. Where multiple paths are eligible and ECMP is supported and configured, the forwarding system can use multiple next hops to reach a destination.

Path diversity

Multiple routed paths can provide alternatives when a link or spine becomes unavailable, provided the remaining topology offers valid reachability.

Routing boundaries

Routed links can reduce dependence on extending the same Layer 2 segment throughout the fabric.

Independent verification

Engineers can inspect BGP routes, installed next hops and interface counters to identify where forwarding behavior differs from the intended design.

Remember: BGP is the routing protocol; ECMP is a forwarding capability that may use multiple eligible equal-cost paths. Running BGP alone does not guarantee that multiple paths will be installed or used.

4. When might each approach be appropriate?

vPC may be appropriate when a server, appliance or downstream switch needs redundant Layer 2 connectivity to a pair of switches. Routed multipathing may be attractive when the network can use Layer 3 links and routing to provide path diversity across a fabric.

Requirement	Design direction to evaluate	Key question
Endpoint requires Layer 2 adjacency across two switches	Evaluate vPC or another supported multi-chassis aggregation design.	Can the peer relationship and failure behavior meet the requirement?
Fabric can use routed point-to-point links	Evaluate a routed leaf-spine architecture with BGP and ECMP.	Are alternate paths available, eligible and correctly installed?
Application requires high availability	Evaluate the complete path, including server NICs, switches, routing and upstream services.	Does the design survive the relevant failure without unacceptable impact?
Uneven traffic utilization	Inspect port-channel hashing and routed ECMP distribution separately.	At which forwarding stage is traffic concentrating?

5. Avoid treating vPC and ECMP as interchangeable

A vPC port-channel and routed ECMP may both distribute traffic, but they operate at different forwarding layers. They also have different state, failure mechanisms and troubleshooting commands. A design can use both technologies in different parts of the same network, provided their interaction is understood and supported by the platforms.

Document the Layer 2 and Layer 3 boundaries.
Identify which component detects each expected failure.
Verify the alternate forwarding path exists before testing a failure.
Measure convergence, packet loss and remaining link capacity.
Check port-channel members and ECMP next hops independently.
Validate the design against the vendor's platform and software documentation.

Key takeaway: Choose vPC when multi-chassis Layer 2 aggregation is required; evaluate routed BGP ECMP when the design calls for multiple Layer 3 paths. The strongest architecture is the one whose failure domains, forwarding behavior and operational requirements have been deliberately designed and tested.

Interactive learning · Module 06

Choose the Right Multipathing Model

vPC and BGP ECMP solve different networking problems. Compare a Layer 2 multi-switch attachment with a Layer 3 routed fabric, then simulate a path failure to see which part of the design must be investigated.

Primary purpose Dual-switch Layer 2 attachment

Forwarding context EtherChannel / vPC

Failure to investigate Member and peer state

Layer 2 attachment with vPC

A supported downstream device can form a port-channel across two vPC peers, presenting a logical attachment while using links to both switches. Peer-link health, consistency and platform-specific failure behavior remain important.

Engineering check: Verify vPC state, peer-link, member interfaces, consistency and traffic counters.

vPC · Layer 2 attachment

Provides a multi-switch port-channel attachment for supported devices.
Depends on peer coordination and platform-specific vPC behavior.
Does not itself provide BGP route selection or Layer 3 ECMP.

BGP ECMP · Layer 3 paths

Uses multiple eligible equal-cost next hops when supported and configured.
Depends on routing policy, route selection, next-hop reachability and forwarding installation.
Does not require a vPC peer-link between the routed next hops.

Verification command examples

show vpc
show port-channel summary
show ip bgp summary
show ip route
show interface counters errors

These are illustrative Cisco-style commands. Availability and syntax vary by platform and software version.

Knowledge check

Which statement best describes the relationship between vPC and BGP ECMP?

These designs can coexist in a larger network, but one does not automatically configure or guarantee the other. The drawings are conceptual and do not represent a vendor-specific production topology.

Section 07 · Operational Troubleshooting

vPC Troubleshooting and Verification

A resilient design needs more than redundant links and correctly configured switches. Engineers must be able to verify the operational state of the vPC domain, identify inconsistencies, isolate failures and confirm that traffic is using the expected forwarding paths.

The most effective troubleshooting approach is to work from the underlying physical interfaces towards the logical port-channel, vPC peer relationship and end-to-end forwarding path. This helps distinguish a physical connectivity problem from a peer coordination, Layer 2 or routing issue.

1. Begin with the health of the vPC domain

Start by confirming that both peer switches are available and that their peer relationship is functioning as expected. Check the peer-link, peer-keepalive mechanism, vPC member interfaces and consistency status. A single healthy interface does not prove that the complete vPC arrangement is healthy.

Peer relationship

Verify that the peers recognize each other and that the expected vPC domain state is present.

Peer-link

Check physical members, logical interface state, errors and capacity. Confirm that the peer-link is operating as designed.

Keepalive

Verify the configured keepalive path and inspect logs for reachability changes or missed messages.

Consistency

Review vPC consistency results and any reported type-1 or type-2 configuration mismatches.

Safety first: Peer-link and keepalive failures can affect vPC forwarding and peer protection behavior. Do not troubleshoot by blindly shutting interfaces or changing peer roles. Understand the current state and follow the platform's documented recovery procedure.

2. Verify the physical members and logical port-channel

A port-channel can have a logical interface that appears configured while one or more expected physical members are down, suspended or not bundled correctly. Compare the configured membership with the operational state.

On Cisco Nexus platforms, the following NX-OS commands are common starting points. Exact command availability and output vary by platform and software release.

show port-channel summary show interface port-channel 10 show interface ethernet 1/1 show vpc

Replace the example interface and port-channel identifiers with those used in your environment. Check whether the physical members are bundled, whether the logical interface is up, and whether counters indicate errors or drops.

3. Troubleshooting symptoms and likely causes

Observed symptom	Possible causes	Next checks
Port-channel is down	Member links are down, members are not bundled, or configuration does not match.	Physical state, member configuration, LACP state and port-channel summary.
Member is suspended	Negotiation, compatibility or configuration problems.	LACP details, interface logs, speed, duplex and channel-group settings.
vPC consistency warning	Relevant configuration differs between peers.	Consistency output, VLAN membership, port-channel settings and platform guidance.
Peer-link is down	Physical failure, member bundle failure or a configuration issue.	Peer-link members, interface errors, cabling and peer state.
Traffic is unevenly distributed	Hash distribution, flow size differences or reduced available capacity.	Per-member traffic counters, hashing policy and workload patterns.
Connectivity fails after a routing change	Route withdrawal, missing next hop, policy issue or unavailable alternate path.	BGP session state, route table, next-hop reachability and forwarding entries.

4. Investigate vPC consistency issues

Consistency checks help identify configuration differences between peers that could affect forwarding. The exact checks and the impact of a mismatch depend on the platform, interface type and software version.

Do not assume that every warning has the same severity. Some inconsistencies can suspend a vPC or prevent it from operating as intended; others may generate warnings without immediately disabling forwarding.

Compare VLAN membership and allowed VLAN lists.
Check port-channel and LACP configuration on both peers.
Verify relevant trunk, access and spanning-tree settings.
Review vPC-specific consistency output and system logs.
Make corrective changes using the platform's documented procedure, then verify the resulting operational state.

5. Use counters to investigate traffic problems

When connectivity works but performance is poor, examine counters rather than relying only on link status. A physical link can remain up while experiencing errors, congestion or uneven utilization.

show interface counters errors show interface ethernet 1/1 show port-channel summary

These are illustrative NX-OS commands. Verify the supported syntax for your specific device and software release.

Compare traffic rates across all members of the same port-channel.
Look for CRC errors, input errors, output drops and unexpected counter changes.
Check whether one large flow is dominating a member link.
Confirm that the remaining members have enough capacity during a failure.
Compare the observed behavior with the configured hash policy.

6. Follow the traffic across the Layer 3 boundary

If the vPC domain appears healthy but remote destinations are unreachable, continue the investigation into the routed network. A working port-channel does not guarantee that a route exists or that the destination is reachable through the expected next hop.

show ip route show ip bgp summary

These examples are intended for supported Cisco routing contexts; syntax and command availability vary. Check the routing table, BGP adjacency state and next-hop reachability before concluding that vPC is responsible for the outage.

7. A repeatable troubleshooting workflow

Define the impact

Determine which endpoints, VLANs, links or destinations are affected and whether the issue is total loss, intermittent connectivity or reduced performance.

Check physical and logical state

Verify interface status, port-channel membership, LACP state and relevant error counters.

Check the vPC domain

Inspect peer status, peer-link, keepalive and consistency results. Review logs for state transitions around the time of the problem.

Verify the forwarding path

Confirm VLAN and Layer 2 reachability where applicable, then inspect routing, next hops and forwarding entries for remote destinations.

Validate the correction

Confirm that the original symptom has cleared, counters are stable and the network has returned to its intended state. Document the root cause and evidence.

Key takeaway: Troubleshoot from the physical interface through the port-channel and vPC domain, then follow the traffic into the routed network where necessary. Use operational state, consistency results, counters and routing evidence together rather than relying on a single command or status indicator.

Interactive troubleshooting · Module 07

Diagnose the Fault Before Changing the Network

Choose a symptom to build a focused investigation plan. Work from the observed fault toward the relevant control-plane and data-plane evidence. The checks below are a learning aid, not live device output.

1. Select the symptom

Selected investigation

Port-channel member investigation

Determine whether the port-channel is operational, which members are active, and whether traffic loss is linked to a physical interface, a flow-hashing pattern or a downstream fault.

2. Work through the checks

Check port-channel and member state. Confirm operational members and any suspended or down interfaces.

0 of 4 checks completed

Start by establishing the current operational state before changing configuration.

3. Example verification commands

Use commands appropriate to the device and software release. Compare outputs before and after a controlled test.

show port-channel summary
show interface port-channel 10
show interface counters errors

Interpretation: a member being up does not alone prove that every flow is distributed evenly or that the end-to-end path is healthy.

Knowledge check

You observe that a BGP session is established, but the expected prefix is missing from the routing table. What is the best next step?

Command syntax and available fields vary by platform and software version. Follow the vendor's documentation and your operational procedures. Record the baseline, make one controlled change at a time, and verify both recovery and end-to-end traffic.

Section 08 · Design and Deployment

vPC Design Best Practices and Deployment Checklist

A successful vPC deployment starts with a clear redundancy objective, a supported topology and a documented failure model. Link aggregation alone does not guarantee end-to-end resilience. The peer relationship, physical infrastructure, Layer 2 configuration, routing design and operational procedures must work together.

Use the following principles to review a proposed design or assess an existing data-centre deployment. Platform-specific implementation details must always be checked against the relevant vendor documentation.

1. Build redundancy into the physical design

Separate failure domains

Where practical, diversify cable paths, line cards, power sources and upstream connectivity so that one local incident does not remove every redundant path.

Plan peer-link capacity

Size the peer-link for its expected control and data traffic, including relevant failure scenarios. Do not assume normal utilization represents the worst-case requirement.

Protect the keepalive path

Design peer-keepalive connectivity according to platform guidance. It should provide useful peer-liveness information independently of the peer-link failure it helps diagnose.

Preserve usable capacity

Confirm that the surviving links and switches can carry the expected load after a failure, rather than merely keeping endpoints connected.

2. Keep the configuration consistent

vPC peers need compatible configurations for the features and interfaces involved in the design. Consistency checking can detect important differences, but it should not replace a deliberate configuration review.

Verify the vPC domain, peer-link and peer-keepalive configuration.
Review VLAN membership, trunk settings and spanning-tree requirements.
Confirm port-channel and LACP settings match the intended design.
Check relevant MTU, speed, interface and feature compatibility.
Document the treatment of orphan ports and single-homed devices.
Review supported software versions, known limitations and upgrade procedures.

Design principle: Do not copy a configuration between peers without validating which settings must match and which settings are intentionally different. Follow the consistency requirements for the specific platform and release.

3. Decide where Layer 2 should end

One of the most important architectural decisions is determining which networks need Layer 2 adjacency and where Layer 3 routing should begin. Extending Layer 2 can be necessary for some workloads, but it also increases the scope of the Layer 2 failure domain.

Design question	What to evaluate
Does the endpoint need Layer 2 adjacency?	Determine whether vPC-based connectivity is required or whether a routed attachment is suitable.
Can the fabric use routed links?	Evaluate a Layer 3 leaf-spine design with routing and ECMP where multiple paths are appropriate.
Where is the gateway located?	Document gateway redundancy, first-hop behavior and the forwarding path during peer or link failure.
How are failures contained?	Identify which devices, VLANs, routes and applications are affected by each failure domain.

4. Integrate BGP and ECMP deliberately

Where the data centre uses BGP, review the routing architecture independently from the vPC configuration. BGP route exchange, best-path selection and ECMP forwarding have their own operational requirements. A healthy vPC domain does not prove that the routed network has a valid path to every destination.

Document BGP peers, address families, route policies and intended advertisements.
Verify next-hop reachability and which routes are installed for forwarding.
Confirm that multiple paths are eligible for ECMP where multipath forwarding is expected.
Inspect both ECMP next-hop distribution and port-channel member utilization.
Test how link, peer and BGP-session failures affect traffic and convergence.

Remember: vPC hashing chooses a physical member of a logical link. ECMP chooses among eligible routed next hops. If traffic is unevenly distributed, investigate each forwarding stage rather than assuming one mechanism explains the entire path.

5. Interactive deployment readiness checklist

Use this checklist during a design review or before a controlled deployment. Tick each item only after the relevant configuration, documentation or test evidence has been verified.

Checklist progress is held in this page view only. It is not saved to an account or network device. Completing the checklist is not a substitute for platform-specific validation or an approved change plan.

6. Final design review

Before declaring the design ready, confirm that the team can explain how traffic flows during normal operation, which links carry it, what happens when each important component fails, and how the resulting behavior will be verified. A design that can be tested, monitored and recovered consistently is easier to operate than one that relies on assumptions about redundancy.

Key takeaway: Build vPC resilience around supported peer behavior, consistent configuration, adequate capacity and independent failure domains. Where BGP and ECMP are also used, validate the routing and forwarding layers separately, then test the complete end-to-end design.

Section 09 · Practical Learning

Interactive vPC and BGP Labs

Understanding redundancy means more than knowing which links exist. You need to predict what happens when a component fails, identify which control-plane or forwarding state changes, and verify whether traffic can continue to reach its destination.

Explore these five guided scenarios. Each lab gives you a problem, an investigation sequence, verification commands and the observations you should look for. Select a lab to switch between scenarios.

Lab safety: These are illustrative exercises, not live-device simulations. Commands and outputs depend on platform and software version. Perform failure tests only in an authorized lab or maintenance window with a documented recovery plan.

LAYER 2 ETHERCHANNEL

Lab 1: Member-link failure

Scenario

A server connects through a port-channel whose physical members provide redundant bandwidth. One member link fails, but the logical port-channel may remain operational.

Your investigation

Record the original port-channel state and member status.
Identify which physical interface has failed.
Check whether the port-channel remains up and has fewer active members.
Inspect interface errors, utilization and application connectivity.

Example verification commands

show port-channel summary
show interface port-channel 10
show interface ethernet 1/1
show interface counters errors

Expected observation If a surviving member remains active and the port-channel stays up, traffic may continue over the remaining capacity. Existing flows can be redistributed or affected depending on platform behavior and the hashing implementation.

Engineering question: Does the remaining capacity support the expected load?

Test your reasoning

Reveal each answer after considering the scenario. These questions check the design principles behind the labs.

1. A single port-channel member fails, but the logical port-channel remains up. Must BGP necessarily reset?

No. If the routed interface and relevant adjacency remain operational, a member failure may be handled below the routing layer. The actual result depends on where the failed link sits and how the platform reports and handles the failure.

2. Does a healthy vPC peer relationship prove that every BGP route is valid?

No. vPC health and BGP route validity are separate checks. Inspect routing adjacencies, route selection, next-hop reachability and forwarding state independently.

3. If a route has two eligible ECMP next hops, will every flow use both paths equally?

Not necessarily. Hashing, flow count, flow sizes and traffic patterns influence distribution. Two installed next hops do not guarantee equal utilization.

How to use these labs in a real environment

Write down the expected behavior before making a change.
Capture baseline interface, vPC, routing and traffic information.
Change only one controlled failure condition at a time.
Compare the observed result with the prediction and record any packet loss or convergence delay.
Restore the original state and verify end-to-end connectivity before proceeding.

Key takeaway: Troubleshoot from the physical interface through the port-channel and vPC state, then inspect Layer 3 routing and end-to-end forwarding. This layered method helps isolate whether a fault affects link redundancy, peer behavior, route selection or the final traffic path.

Commands shown are examples for Cisco NX-OS-style environments. Syntax and availability vary by platform and release. The interactive panels illustrate learning scenarios and do not execute commands, change device state or persist lab progress.

Hands-on learning · Module 09

vPC and BGP Troubleshooting Lab Console

Choose a scenario, examine the conceptual topology and work through a guided investigation. Each lab focuses on evidence gathering, interpreting network state and identifying what to verify next. These are simulated learning exercises, not live device sessions.

Layer 2 · Port-channel

Lab 01 — Member-link failure

A port-channel has lost one physical member. Determine whether the bundle remains operational and whether the remaining capacity is sufficient for the observed traffic.

Investigation progress

Begin by confirming the operational state of the port-channel and its physical members.

Expected observations will appear here after the scenario is simulated.

Lab objective

Confirm member state, assess remaining capacity and verify end-to-end forwarding.

Commands to investigate

Use these examples as a starting point and adapt them to the platform.

show port-channel summary
show interface port-channel 10
show interface counters errors

What good evidence looks like

Confirm that the port-channel is up, identify active members and compare traffic and error counters before and after the event.

Lab knowledge check

One physical port-channel member fails. What should you verify first?

All topology changes and observations in this widget are illustrative. They do not emulate a particular switch operating system, predict exact convergence time or guarantee lossless recovery. Follow platform documentation and approved change procedures when working on real devices.

```

☕

Support Network Insight

Help support independent engineering guides, interactive networking tools, and future learning labs.

Support Network Insight ```

Data Center Topologies

August 14, 2014

by Matt Conran Blog

Network Architecture / Data Center Design

Data Center Topology: From Three-Tier Networks to Leaf-Spine Fabrics

Data center topology defines how switches, servers, storage systems and external networks connect—and ultimately how application traffic moves through the infrastructure. The architecture you choose influences latency, available bandwidth, resilience, operational complexity and the ability to scale as workloads grow.

Traditional hierarchical networks remain relevant in many environments, but modern cloud platforms and distributed applications increasingly rely on scalable Layer 3 fabrics, leaf-spine architectures and equal-cost multipath routing. Understanding these designs means looking beyond the network diagram to examine routing behaviour, traffic patterns, failure domains and the control plane that keeps the fabric connected.

Design principle: build around application requirements

A good topology is not simply the one with the most links or the most switches. It is the one that delivers the required capacity, predictable forwarding, appropriate redundancy and manageable failure behaviour at an acceptable cost.

01 / STRUCTURE

Topology and Fabric Design

Compare three-tier, centralized, top-of-rack, end-of-row and leaf-spine architectures.

02 / ROUTING

BGP and ECMP

Explore routing adjacencies, route selection, multipath forwarding and convergence during failures.

03 / OVERLAYS

VXLAN and EVPN

Understand how Layer 3 underlays support virtual networks and distributed endpoint reachability.

04 / OPERATIONS

Capacity and Resilience

Evaluate oversubscription, traffic distribution, observability and failure-domain behaviour.

In this guide: Work from fundamental topology choices to modern data center fabrics, BGP-based routing, traffic engineering considerations and practical troubleshooting.

NETWORK INSIGHT · FIELD GUIDE

Explore Data Center Topology

Navigate the architecture, routing, traffic flows and resilience principles behind modern data center networks.

06 sections

01 / ARCHITECTURE FUNDAMENTALS

Understanding Data Center Architectures

Data center topology defines how servers, storage systems, switches and external networks connect. The architecture determines how traffic moves through the infrastructure and influences available bandwidth, latency, redundancy, scalability and operational complexity.

This section introduces traditional three-tier networks, centralized switching, top-of-rack (ToR), end-of-row (EoR) and leaf-spine architectures. Compare their connectivity models and trade-offs to understand which designs best match different workload and infrastructure requirements.

Engineering focus Choose topology according to traffic patterns, growth expectations, capacity, failure domains and the cost of operating the network.

Section 01 / Architecture Fundamentals

The Main Data Center Architectures

Data center networks can be built using hierarchical switching layers, centralized switching platforms, distributed access switches or fabric-based designs. Each approach creates a different balance between cabling, port utilization, bandwidth, failure isolation and operational complexity.

The right starting point is the workload. A smaller environment with limited server-to-server traffic may not need the same architecture as a cloud platform supporting thousands of virtual machines, containers or data-intensive applications. Topology selection should reflect traffic patterns, growth expectations, availability targets and the cost of operating the network over time.

Common architectural approaches

Hierarchical

Traditional Three-Tier Architecture

A traditional enterprise design separates access, aggregation or distribution, and core functions. Access switches connect servers, the aggregation layer concentrates traffic and may enforce policy, and the core provides high-speed connectivity between network blocks.

This model remains useful where existing infrastructure, established operational practices or clear policy boundaries make a hierarchical design appropriate. However, multiple switching tiers can increase path length and create bottlenecks when east-west traffic grows.

Centralized

Centralized Switching

In a centralized design, servers or server racks connect back to a central switching location. Concentrating switching resources can simplify management and make port allocation efficient in smaller environments.

As the facility grows, cable lengths, pathway congestion, centralized capacity limits and the consequences of equipment failure must be evaluated carefully.

Distributed Access

Top-of-Rack and End-of-Row

Top-of-rack (ToR) places access switches in individual server racks. End-of-row (EoR) places switching equipment in a dedicated rack serving a row of servers. Both approaches can provide redundant connectivity when designed with suitable independent paths and devices.

ToR generally shortens server cabling and can isolate access failures to a rack. EoR can consolidate switching resources, but may require more structured cabling. Neither approach automatically determines whether the wider network uses a three-tier or leaf-spine architecture.

Fabric-Based

Leaf-Spine Architecture

A leaf-spine fabric connects each leaf switch to multiple spine switches. Servers and other endpoints connect to the leaf layer, while the spine layer provides paths between different leaves. A Layer 3 underlay commonly uses equal-cost multipath routing to distribute eligible traffic across available paths.

This architecture provides predictable path structure and supports incremental expansion. Its capacity and resilience depend on link speeds, the number of spine switches, oversubscription, routing design and failure behaviour.

Important distinction: ToR and EoR describe where access switching is placed. Three-tier and leaf-spine describe broader network architectures. A data center can use ToR switches as its leaf layer in a leaf-spine fabric.

Compare the design trade-offs

Design consideration	Three-tier / centralized	Leaf-spine fabric
Traffic paths	May traverse several hierarchical layers.	Typically uses a consistent leaf-to-spine-to-leaf path.
Scaling	Expansion depends on available tier capacity and design.	Can expand by adding leafs and, when needed, spine capacity.
Multipath routing	Depends on the routing and switching design.	Layer 3 ECMP is commonly used between leafs and spines.
Operational considerations	May suit established environments and existing investments.	Requires deliberate underlay, routing, capacity and failure planning.

Before selecting a topology

Traffic patterns

Determine whether workloads primarily communicate with external networks or generate substantial east-west traffic between servers.

Capacity and growth

Estimate endpoint growth, uplink bandwidth, oversubscription limits and the cost of adding future capacity.

Failure domains

Identify the impact of losing a switch, link, rack or entire network block, including the capacity that remains.

Operations and cost

Balance equipment and cabling costs against monitoring, troubleshooting, automation and lifecycle requirements.

NETWORK INSIGHT · INTERACTIVE ARCHITECTURE LAB

Explore Data Center Architectures

Compare four common ways to connect compute, access and switching. Select an architecture, then click a device in the topology to inspect its role in the network.

Three-tier architecture Illustrative topology

Selectable device Network connection Highlighted path

Selected device

Core switch

The core provides high-speed connectivity between distribution blocks and upstream networks in a traditional three-tier design.

Architecture insight

Separate layers make roles easy to understand, but traffic may traverse multiple tiers and scaling can increase operational complexity.

Layered Design structure

Multi-hop Typical path shape

Layer by layer Common scaling approach

Design strengths

Clear separation of network roles
Familiar operational model
Can suit hierarchical traffic patterns

Design considerations

Potentially longer paths between servers
Capacity planning across tiers
Additional layers to operate and troubleshoot

These diagrams illustrate common design patterns, not vendor-specific configurations. Actual forwarding paths, redundancy and oversubscription depend on the implementation. ToR means Top of Rack; EoR means End of Row; Clos describes a multistage network fabric.

02 / FABRIC ARCHITECTURE

Leaf-Spine and Clos Fabrics

Leaf-spine architecture connects each leaf switch to multiple spine switches, creating a consistent path structure between different server racks. It is widely used in modern data centers where distributed applications, virtualization and cloud workloads generate substantial east-west traffic.

Explore how leaf and spine layers work together, how two-tier fabrics relate to Clos network designs, and how adding switches can expand connectivity. The section also examines oversubscription, available path diversity and the limits of relying on topology alone to guarantee performance.

Engineering focus Understand fabric stages, leaf-to-spine connectivity, expansion options and the capacity that remains when a device or link fails.

Section 02 / Fabric Architecture

Leaf-Spine and Clos Fabrics

Leaf-spine architecture is a common foundation for modern data center networks. Instead of sending traffic through a succession of hierarchical switching layers, it connects access-layer leaf switches to a set of spine switches. This creates a fabric with consistent paths between leaf switches and provides a repeatable way to expand connectivity as workloads grow.

The architecture is particularly useful for environments with substantial east-west traffic, including virtualized platforms, container infrastructure, distributed storage and cloud-native applications. Its performance depends on the actual link speeds, number of paths, traffic distribution and capacity provisioned throughout the fabric.

The physical fabric

Spine layer

Spine 1

Spine 2

Spine 3

Every leaf connects to every spine

Leaf layer

Leaf 1

Leaf 2

Leaf 3

Leaf 4

Server and workload connectivity

Servers

Storage

Compute

Services

Conceptual topology, not a physical cabling diagram. Each leaf has links to each spine; server-facing connections are shown separately. The simplified graphic illustrates connectivity rather than every individual link.

1. The role of leaf switches

Leaf switches provide connectivity for servers, storage, firewalls, load balancers and other data center endpoints. They may also perform Layer 3 routing, enforce network policy and act as VXLAN tunnel endpoints when an overlay is deployed.

In a Layer 3 fabric, endpoints in different subnets can communicate through routed paths. When VXLAN EVPN is used, the leaf layer can additionally provide overlay connectivity and distribute endpoint reachability through the EVPN control plane.

2. The role of spine switches

Spine switches interconnect the leaf layer and provide transit paths between leaf switches. In a conventional two-tier leaf-spine fabric, spine switches do not connect directly to servers, and leaf switches do not connect directly to other leaf switches.

Adding spine capacity can increase the number of available paths and aggregate fabric bandwidth, provided the leaf uplinks, interfaces, routing configuration and downstream capacity are designed to use it. A spine failure can reduce available capacity even when connectivity survives.

Why path predictability matters: Traffic between endpoints attached to different leaf switches normally traverses one spine switch. This creates a consistent switching-hop structure, although actual latency still depends on link speeds, device processing, congestion and the forwarding path.

How the architecture scales

01 / ADD LEAF CAPACITY

Expand endpoint connectivity

Add leaf switches with uplinks to the fabric's spine layer. Confirm that the existing spine and uplink capacity can support the additional traffic.

02 / ADD SPINE CAPACITY

Increase fabric bandwidth

Additional spine switches can create more equal-cost paths. Validate routing limits, optics, cabling, interface capacity and platform support before expanding.

03 / MANAGE OVERSUBSCRIPTION

Plan for real traffic demand

Compare aggregate server-facing bandwidth with available uplink bandwidth. Size the fabric according to workload behaviour and acceptable contention.

04 / CONTROL FAILURE IMPACT

Test the remaining capacity

Evaluate whether surviving spines and links can carry traffic after a failure, not simply whether alternate paths exist.

Two-tier and multi-tier Clos designs

Leaf-spine is a practical implementation of a two-tier Clos-style fabric. Larger environments may introduce a super-spine or additional fabric stages to interconnect multiple pods, rooms or data halls. These designs extend the architecture while requiring careful consideration of path diversity, capacity and failure boundaries.

Design	Typical purpose	Key consideration
Two-tier leaf-spine	Connect endpoints across a single fabric.	Spine capacity, leaf uplinks and ECMP behaviour.
Multi-pod fabric	Connect modular groups of racks or infrastructure.	Inter-pod connectivity and failure isolation.
Super-spine / multi-stage Clos	Scale connectivity across larger fabrics or multiple data halls.	Additional stages, path design and end-to-end capacity.

NETWORK INSIGHT · FABRIC EXPLORER

Explore Leaf-Spine and Clos Fabrics

Investigate how leaf and spine switches connect, trace an illustrative server-to-server path, simulate a spine-link failure, and see why uplink capacity and oversubscription matter.

Explore:

Two-tier leaf-spine fabric All illustrated links up

Network device Available link Illustrative selected path Failed link

Topology focus

Leaf switches

Leaf switches connect servers and provide uplinks into the spine layer. In a conventional two-tier fabric, each leaf connects to every spine.

Fabric principle

A full-mesh leaf-to-spine design provides multiple potential paths between different leaves. Routing and ECMP determine which next hops are installed and used.

Trace a flow

Source leaf

Destination leaf

2 Illustrated spine paths between selected leaves

8 Leaf-to-spine links currently available

Healthy Illustrative fabric state

Explore leaf uplink oversubscription 3:1 example

1:1 · equal nominal capacity 5:1 · more aggregate server capacity

A 3:1 illustrative oversubscription ratio means aggregate nominal server-facing bandwidth is three times the uplink bandwidth. Actual throughput depends on traffic patterns, link rates and congestion.

Why use a Clos-style fabric?

A multistage design can provide scalable connectivity and multiple paths without requiring every leaf to connect directly to every other leaf.

What should engineers verify?

Check spine and leaf capacity, uplink speed, ECMP behavior, failure convergence and the bandwidth remaining after a link or device failure.

This is an educational model, not a live network or packet-level simulator. The four-leaf, two-spine diagram is illustrative. ECMP distributes eligible flows across available next hops; it does not guarantee that each physical link carries exactly the same load.

03 / TRAFFIC ENGINEERING

Data Center Traffic Patterns

A data center network must accommodate the traffic generated by applications, storage systems, virtual machines and distributed services. Understanding where traffic originates and where it travels is essential for designing a fabric with sufficient capacity and predictable performance.

This section distinguishes east-west traffic between internal workloads from north-south traffic entering or leaving the data center. It explores how flow size, concurrent demand, ECMP hashing, uplink capacity and congestion influence the actual behaviour of a leaf-spine network.

Engineering focus Size the fabric around real workload demand, identify potential bottlenecks and validate performance during normal operation and failure conditions.

Section 03 / Traffic Engineering

Data Center Traffic Patterns

A data center topology must be designed around the traffic it carries, not simply the number of switches and links it contains. Application placement, storage access, replication, service dependencies and external connectivity all influence where traffic flows and where congestion can occur.

Leaf-spine fabrics are particularly suited to distributed east-west communication, but north-south traffic, large data transfers and uneven workload demand can still create bottlenecks. Understanding these patterns helps engineers size uplinks, evaluate redundancy and choose appropriate routing and forwarding designs.

Two fundamental traffic directions

01 / EAST-WEST

Workload-to-workload traffic

Traffic between servers, virtual machines, containers, storage systems and distributed application components. In modern environments, this can represent a substantial share of total network demand.

02 / NORTH-SOUTH

Traffic entering or leaving the fabric

Traffic between internal workloads and external networks, users, WANs, internet services or other data centers. Its path may include border leaf switches, firewalls, routers and load balancers.

Example / east-west traffic path

Communication between two server racks

Server A
Rack 1

Leaf 1

Spine

Leaf 2

Server B
Rack 2

Simplified logical path. In a conventional two-tier leaf-spine fabric, inter-leaf traffic normally traverses a spine. The forwarding plane selects an available path according to the installed routes and forwarding configuration.

Why east-west traffic affects fabric design

Distributed applications frequently exchange data across multiple servers. Examples include database replication, distributed storage, service-to-service requests, cluster coordination and virtual machine communication. If these flows cross leaf switches, the spine layer and leaf uplinks must provide enough capacity for their combined demand.

Traffic demand is rarely perfectly uniform. Some flows are short and latency-sensitive, while others transfer large volumes of data. Multiple simultaneous flows may also compete for the same links. Consequently, aggregate bandwidth alone does not reveal every congestion risk.

Important distinction: A fabric can have several equal-cost paths and still experience congestion. ECMP provides a mechanism for distributing eligible flows across paths; it does not guarantee that every flow uses a different link or that traffic is perfectly balanced.

North-south traffic and border connectivity

External traffic commonly enters or leaves through designated border leaf switches or dedicated edge devices. These connect the fabric to WAN routers, internet edge infrastructure, firewalls, load balancers and other networks.

The design must consider both bandwidth and resilience. Concentrating external connectivity on a small number of devices may simplify policy enforcement, but it can also create capacity constraints or larger failure domains. Distributing border functions can improve scale, provided routing policy and security controls remain consistent.

Traffic patterns and engineering decisions

Traffic pattern	Potential pressure point	Design response
Server-to-server	Leaf uplinks and spine capacity.	Assess aggregate demand, path diversity and ECMP.
Storage replication	Sustained high-throughput flows.	Measure peak demand and validate link utilization.
Internet or WAN access	Border links, security appliances and external routing.	Plan edge capacity, policy and redundant paths.
Large data transfers	Flow imbalance and transient congestion.	Evaluate hashing behaviour, flow sizes and telemetry.
Failure or maintenance	Reduced capacity on surviving paths.	Validate traffic behaviour after losing a link or device.

Measure the traffic, not just the topology

Traffic engineering should use interface counters, utilization trends, flow telemetry, packet drops, latency measurements and application performance data. These observations help distinguish a routing issue from a capacity problem or an application-generated traffic burst.

Engineers should also examine behaviour during failure conditions. A fabric that operates comfortably during normal traffic may become congested when a spine or uplink is unavailable and flows are redistributed across fewer remaining paths.

NETWORK INSIGHT · TRAFFIC EXPLORER

Explore Data Center Traffic Patterns

Follow traffic between servers, across the fabric and toward external networks. Adjust the illustrative load to understand why workload patterns, path capacity and congestion matter when designing a data center.

East-west: server to server Illustrative traffic flow

Network link East-west flow North-south flow Potential congestion

Traffic focus

East-west traffic

Traffic between workloads inside the data center. In a leaf-spine fabric, communication between servers on different leaves normally traverses a spine.

Engineering insight

Distributed applications, storage replication and service-to-service calls can generate substantial east-west traffic. Design for workload demand rather than assuming all traffic exits the data center.

Example flow

East-West Selected traffic direction

Leaf → Spine → Leaf Illustrative forwarding path

Moderate Illustrative link demand

Simulate traffic demand 45% illustrative load

Low demand High demand

At this illustrative load, the selected links have moderate demand. Real congestion depends on offered traffic, available link capacity, buffering, queueing and the distribution of flows.

Why traffic direction matters

East-west traffic can stay inside the fabric and use leaf uplinks and spine capacity. North-south traffic crosses the boundary toward external networks or services.

What engineers should measure

Compare interface utilization, drops, queue depth, flow records and application latency. A high average utilization can hide short bursts that cause packet loss or delay.

This widget is a conceptual teaching model, not a live traffic generator or telemetry system. The load slider represents illustrative demand, not measured utilization. Actual paths and capacity depend on topology, routing, link rates, hashing and workload behavior.

04 / ROUTING CONTROL PLANE

BGP in Data Center Fabrics

Border Gateway Protocol (BGP) provides a scalable way to distribute network reachability information across modern data center fabrics. In a routed leaf-spine design, BGP can establish the routes that allow traffic to move between endpoints while supporting controlled route propagation and multipath forwarding.

Explore eBGP and iBGP design approaches, route advertisements, best-path selection, ECMP and convergence. The section also introduces the relationship between a routed underlay and a VXLAN EVPN overlay, showing why control-plane state must be checked alongside the actual forwarding table.

Engineering focus Trace route propagation, validate next-hop reachability, understand multipath installation and troubleshoot routing changes without confusing BGP state with end-to-end connectivity.

Section 04 / Routing Control Plane

BGP in Data Center Fabrics

Border Gateway Protocol (BGP) is widely used to exchange reachability information in modern data center networks. In a routed leaf-spine fabric, it can provide scalable route distribution, controlled reachability, multipath forwarding and predictable responses to topology changes.

BGP does not replace the physical fabric. It supplies control-plane information that routers use to decide which network destinations are reachable and which paths can be installed for forwarding. The resulting behaviour depends on the routing design, policy, platform capabilities and operational configuration.

Keep the layers distinct: the physical topology defines available links; the routing control plane distributes reachability; the forwarding plane moves packets using installed forwarding entries. A healthy BGP session alone does not prove that the intended traffic path is working.

01 / Choosing a BGP fabric design

DESIGN A

eBGP leaf-spine

Leaf and spine switches use different autonomous system numbers (ASNs), and BGP sessions between them operate as external BGP. This is a common design for routed Clos fabrics because it can keep routing relationships explicit and support repeatable deployment patterns.

DESIGN B

iBGP-based fabric

Routers exchange BGP routes within the same autonomous system. Larger designs may use route reflectors to distribute routes without requiring a full mesh of iBGP sessions between every participating router.

DESIGN C

Hybrid routing

A fabric may combine an IGP or static routing for underlay reachability with BGP for other routing functions. The exact division depends on the design; there is no requirement that every data center use one universal routing model.

DESIGN D

Border and external routing

Border leaf switches or edge routers exchange routes with external networks. Import and export policies determine which prefixes can enter or leave the fabric and how they are represented to other routing domains.

Example: eBGP in a leaf-spine fabric

In this simplified example, each leaf has an eBGP session to each spine. Leaf-to-spine routing sessions exchange the reachability needed to forward traffic across the fabric. The example illustrates the relationship rather than prescribing a specific ASN plan or configuration.

Illustrative two-tier routed fabric

SPINE LAYER

Spine 1 AS 65101

Spine 2 AS 65102

eBGP sessions between each leaf and both spines

LEAF LAYER

Leaf 1 AS 65001

Leaf 2 AS 65002

Conceptual relationship only; individual physical links and separate BGP sessions are not drawn. ASNs are illustrative. This design assumes the chosen BGP configuration permits the required route exchange between the relevant devices.

Design caveat: eBGP does not automatically guarantee a working fabric or multipath forwarding. ASN allocation, next-hop reachability, route policy, prefix limits, supported BGP features and ECMP configuration must all be validated.

02 / Underlay and overlay routing

The BGP underlay

The underlay is the routed IP infrastructure that connects the switches and provides IP reachability across the fabric. In a BGP-routed underlay, BGP can advertise the infrastructure prefixes needed to reach loopbacks and other routed destinations. These routes provide the foundation for control-plane sessions and overlay tunnel endpoints where applicable.

The VXLAN EVPN overlay

In a VXLAN EVPN design, VXLAN encapsulates tenant traffic across the IP fabric, while EVPN uses BGP to distribute overlay reachability information. This can include information about MAC addresses, IP addresses and tenant segments, depending on the EVPN route type and deployment.

Component	Primary responsibility	Example question
Underlay routing	IP reachability between routed fabric endpoints.	Can this VTEP reach the remote VTEP address?
BGP EVPN control plane	Advertises overlay reachability and related information.	Has the expected MAC/IP or prefix route been learned?
VXLAN data plane	Encapsulates tenant packets across the IP underlay.	Can the encapsulated packet traverse the fabric?
Forwarding and policy	Applies installed forwarding entries and applicable policies.	Does traffic reach the correct destination and tenant?

These functions are related but not interchangeable. A healthy underlay does not prove that EVPN routes are correct, and receiving an EVPN route does not by itself prove that end-to-end forwarding succeeds.

03 / How BGP selects routes

From route advertisement to forwarding

Receive: a BGP neighbor advertises a prefix and its associated path attributes. The receiving router evaluates the update against session state, import policy and protocol rules.

Validate: the route must be acceptable under policy and protocol requirements. Its next hop must also be resolvable for the route to be usable for forwarding.

Select: the router evaluates eligible paths using its BGP decision process and implementation-specific policy. Attributes such as local preference, AS path, origin and MED can influence the outcome; their exact effect depends on the decision context and configuration.

Install: the selected route or eligible set of multipaths may be installed into the routing table and forwarding hardware, subject to platform limits, configuration and interactions with other routing sources.

Forward: packets are sent using the installed forwarding entries. The actual path depends on the forwarding implementation, ECMP eligibility, hashing and current topology.

Best path is not the same as every installed path. BGP implementations can select a best path while also installing multiple eligible paths for multipath forwarding. Multipath rules and supported attributes vary by platform and configuration.

Why BGP policy matters

BGP exchanges reachability, but policy determines which routes are accepted, preferred, modified or advertised onward. In a data center, an incorrect policy can suppress a required prefix, permit an unintended route or cause traffic to prefer an unexpected exit.

Engineers should explicitly document intended import and export behaviour, allowed prefixes, ASN expectations, next-hop handling and any route attributes used to influence path selection. Reusable policy templates can reduce configuration drift across large fabrics.

04 / ECMP and BGP multipath

How equal-cost paths support fabric capacity

Equal-Cost Multipath (ECMP) allows a router to forward traffic across multiple eligible next hops for a destination. BGP can supply candidate paths, but the routing implementation and configuration determine whether multiple paths are considered equivalent and installed.

Once multiple next hops are installed, forwarding hardware commonly uses a hash of packet or flow fields to select a path. This helps distribute traffic while typically preserving packet ordering within a flow. It does not mean every individual packet is sent down a different path, nor does it guarantee perfectly even utilization.

Mechanism	Role	Common misconception
BGP best-path selection	Chooses a preferred route according to policy and decision rules.	It does not necessarily install every received path.
BGP multipath	Allows multiple eligible BGP paths to be installed.	It is not automatic on every platform or configuration.
ECMP forwarding	Distributes eligible flows across equal-cost next hops.	It does not guarantee perfect load balance.
LACP	Negotiates and manages a Layer 2 link aggregation group.	It is not the same mechanism as Layer 3 ECMP.

ECMP and LACP can both use hashing in forwarding or distribution decisions, but they operate at different levels. ECMP selects among Layer 3 next hops; LACP negotiates membership in a Layer 2 aggregated link. A design may use one or both, depending on its physical and logical architecture.

05 / Convergence and failure handling

What happens when a path fails?

When a link or device fails, the fabric must detect the event, update the relevant routing state and adjust forwarding. BGP may withdraw affected routes or advertise revised reachability. The forwarding plane then changes its installed entries as the control plane and hardware respond.

Convergence time depends on failure detection, session behaviour, routing policy, protocol timers, implementation, hardware programming and the scope of the change. BFD may help detect some failures faster when supported and correctly configured, but it is not a universal guarantee of a particular recovery time.

FAILURE 01

Single link loss

If alternate paths remain available and the routing design recognizes the failure, traffic may be redirected through surviving paths. Check both route changes and actual forwarding.

FAILURE 02

Spine or peer loss

Routes or next hops associated with the failed device may be removed. Surviving paths carry the remaining demand, potentially increasing utilization and exposing oversubscription.

FAILURE 03

Policy or route withdrawal

A prefix may disappear even though physical links and BGP sessions remain up. Investigate advertisements, filters, next-hop resolution and the routing table.

FAILURE 04

Session instability

Repeated BGP resets can cause route churn and traffic disruption. Check interface errors, reachability, timers, logs and control-plane resource conditions.

Resilience must be tested under load. Redundant links are useful only if surviving paths have sufficient capacity and forwarding converges as intended. Test representative link and device failures, and measure packet loss, recovery time and post-failure utilization.

06 / Practical BGP troubleshooting

Work from session state to packet forwarding

When a prefix is missing or traffic takes an unexpected path, avoid starting with a broad configuration change. Follow the route through the control plane and forwarding plane, checking each stage before moving to the next.

Verify the session. Confirm the intended BGP neighbor is Established, and inspect logs, reset reasons, interface status and reachability between peer addresses.

Check received routes. Confirm the expected prefix is being advertised by the peer and received locally. Use the platform's received-route or neighbor-route commands where supported.

Inspect policy. Review prefix lists, route maps, import/export policies, community handling, maximum-prefix controls and any policy that could reject or modify the route.

Inspect best path and next hop. Identify why a candidate route was selected or rejected. Confirm that the selected next hop is resolvable and reachable.

Verify multipath installation. If several paths are expected, check the routing table for multiple installed next hops and confirm that multipath settings and platform capabilities permit them.

Validate forwarding. Inspect the forwarding table, interface counters, flow telemetry and packet path. For VXLAN EVPN, also verify underlay reachability, relevant EVPN routes, VTEP reachability and tenant forwarding state.

Example diagnostic commands

These are illustrative Cisco IOS/IOS XE-style commands. Exact syntax and available detail vary by operating system and platform; use the equivalent commands for your switches.

show ip bgp summary
show ip bgp
show ip route
show ip cef
show running-config | section router bgp

Start with the neighbor summary, then inspect the target prefix and its candidate paths. Check the routing table to determine which route was selected, and inspect the forwarding table to verify the next hop used for packet forwarding. Review the BGP configuration and policies when the observed result differs from the intended design.

The relevant BGP neighbor is Established.
The expected prefix is received and passes import policy.
The selected route has a valid, reachable next hop.
Expected ECMP next hops are installed when multipath is intended.
Export policy allows the required route to reach downstream peers.
Forwarding state and traffic telemetry agree with the expected path.
Failure tests confirm acceptable convergence and surviving capacity.

Continue with the BGP Learning Centre

Explore route propagation, best-path selection and fault investigation through the Network Insight BGP Learning Centre. It complements this architecture section with a practical, interactive way to examine BGP behaviour.

Open the BGP Learning Centre: Explore. Trace. Diagnose. →

NETWORK INSIGHT · BGP FABRIC EXPLORER

Explore BGP in a Leaf-Spine Fabric

Select a router, inspect the illustrative BGP sessions, trace how a prefix is learned, and simulate a path failure. See how routing information and forwarding paths work together in a data center network.

Illustrative BGP fabric Sessions established

iBGP eBGP Failed path

Selected element

Spine 1

A spine switch provides fabric transit. Depending on the design, it may establish eBGP sessions with leaf switches and exchange underlay reachability.

Engineering insight

BGP distributes reachability information. The forwarding plane uses the installed routes to move packets; a BGP session being established does not alone prove that an application path is working.

Interactive controls

6Illustrative BGP sessions

2Illustrative available paths

EstablishedSelected scenario state

Follow a route advertisement 10.10.10.0/24

1Prefix learnedA peer advertises reachability to the prefix.

2BGP evaluatesAttributes and policy influence route selection.

3Route installedAn eligible route is installed for forwarding.

4Packet forwardedThe forwarding table directs traffic to a next hop.

Underlay reachability

The underlay provides IP reachability between fabric devices and their loopbacks or next hops. eBGP is often used between leaf and spine switches in routed fabrics, though designs vary.

Route selection and forwarding

BGP applies policy and path-selection rules to choose eligible routes. ECMP may install multiple equal-cost next hops when supported by the routing design and platform.

Educational model only. This topology and its route progression are illustrative, not a live BGP implementation. Real session counts, ASNs, route attributes, ECMP eligibility, convergence and forwarding behavior depend on the configured network and platform.

05 / Observe

Network Visibility, Traffic Intelligence and BGP

You cannot reliably protect a network if you cannot establish what is happening within it. Flow records, packet captures, encrypted-traffic metadata and security logs reveal different aspects of network behaviour and provide evidence for troubleshooting and threat detection.

This section connects traffic analysis with BGP route visibility, helping engineers investigate unexpected communication patterns, routing changes and reachability problems using evidence from multiple layers.

05 / Observe

Network Visibility: From Traffic Flows to BGP Intelligence

Security controls determine what should be allowed. Observability helps engineers establish what is actually happening. By correlating flow records, packet-level evidence, encrypted traffic metadata and routing information, network teams can identify suspicious behaviour, investigate connectivity failures and understand how an incident moves across the network.

A firewall log may show that a connection was permitted, but it rarely explains the entire conversation. A packet capture may reveal a protocol exchange, yet only represent a short interval or one observation point. Flow telemetry can expose communication patterns across a longer period, while routing telemetry can explain which paths were available and why traffic may have taken an unexpected route.

Engineering principle: No single telemetry source provides the complete picture. Strong network investigations correlate evidence from multiple layers rather than treating an alert, a flow record or a routing change as conclusive proof on its own.

5.1 — Choose the right observation method

Flow telemetry

NetFlow and IPFIX

Summarise conversations using fields such as source and destination addresses, ports, protocol, packet counts, byte counts and timestamps. IPFIX provides an extensible framework for exporting structured flow information.

Sampling

sFlow

Uses statistical sampling of packets and interface counters to provide scalable visibility into traffic patterns, particularly in large switched and routed networks. Sampling can miss individual events.

Packet evidence

SPAN and packet capture

SPAN mirrors selected switch traffic to an observation port. Packet captures can reveal protocol headers, retransmissions, handshakes and other details that summaries cannot show. Mirror oversubscription can cause packet loss.

Encrypted traffic

Metadata and behaviour

Connection timing, volume, direction, destination reputation, DNS context where available, TLS handshake metadata and endpoint telemetry can help detect anomalies without assuming that encrypted payloads are readable.

Method	Best used for	Important limitation
NetFlow / IPFIX	Traffic accounting, unusual communication patterns and historical investigation	Typically lacks full packet payload and detailed packet-by-packet sequence
sFlow	High-volume traffic trends and scalable network sampling	Sampling may omit short-lived or low-volume activity
SPAN / packet capture	Protocol troubleshooting and detailed session analysis	Limited observation point, capture duration and possible mirror-port loss
Endpoint and security logs	Process attribution, identity, host events and security outcomes	Coverage and reliability depend on instrumentation and log integrity
BGP and routing telemetry	Route changes, path selection, reachability and control-plane anomalies	A valid route does not prove that the application or forwarding path works

5.2 — Build an evidence chain

Consider a server that suddenly transfers much more data than usual. A single bandwidth graph might identify the spike, but it cannot establish whether the cause is a legitimate backup, a compromised service or an unexpected change in traffic routing. A structured investigation combines several sources.

DetectBaseline deviation

→

CorrelateFlows and identities

→

InspectPackets and logs

→

ValidateRoutes and forwarding

First, establish when the anomaly began. Next, identify the communicating endpoints, destination ports, volume and duration. Compare the behaviour against the normal baseline. Then inspect available DNS, endpoint, firewall and packet evidence. Finally, check whether a route change, next-hop failure or policy update coincided with the event.

Interactive investigation explorer

Select an observation to see what it can establish and what should be checked next.

INVESTIGATION VIEW Flow records: establish who communicated

Compare source and destination addresses, ports, protocol, bytes, packets and timestamps. Identify unusual destinations, repeated connections or a sudden change in traffic volume. Then correlate the flow with endpoint and security logs.

5.3 — Encrypted traffic: visibility without payload inspection

Encryption protects confidentiality, but it does not make every aspect of a connection invisible. Depending on the protocol, configuration and observation point, engineers may still have access to connection duration, byte counts, packet sizes, timing, DNS information, destination addresses and some handshake metadata.

Behavioural baselines

Detect changes in connection frequency, data volume, session duration and communication direction compared with a service's established pattern.

Identity correlation

Combine network observations with endpoint, workload and authentication records to associate a connection with a device, user or application where the evidence supports it.

Protocol-aware analysis

Use available TLS, DNS and transport metadata carefully. Encrypted Client Hello and encrypted DNS can limit visibility into some fields, while QUIC changes what traditional passive inspection can observe.

Important distinction: A large encrypted transfer to an unfamiliar destination is an investigation lead, not proof of data exfiltration. Validate the destination, expected application behaviour, endpoint context and any applicable organisational policy before drawing conclusions.

5.4 — BGP visibility: understand the route behind the traffic

Border Gateway Protocol (BGP) exchanges reachability information between autonomous systems and distributes routes within an autonomous system through iBGP. Its control plane influences the paths that IP packets may follow. For network security and troubleshooting, this means traffic anomalies should sometimes be investigated alongside BGP updates, routing-table changes, next-hop resolution and forwarding state.

Routing intelligence

Three autonomous systems, one destination prefix

Illustrative topology for correlating route propagation with observed traffic.

AS65002 External BGP peer

AS65001 Internal BGP and routing policy

AS65003 External BGP peer

Example destination: 10.10.10.0/24

# Illustrative route observation Prefix: 10.10.10.0/24 Protocol: BGP AS_PATH: 65002 64496 NEXT_HOP: 192.0.2.2 Observation: Route present in BGP table Validation: Confirm best path, next-hop reachability and FIB

The values above are illustrative. They do not describe a live network or establish that this route is selected, reachable or safe.

What should engineers observe?

UPDATE activity: identify announcements and withdrawals, including the prefix affected and the time of the change.
AS_PATH changes: investigate unexpected path changes, unusual origins and route attributes that may affect selection.
Best-path selection: compare competing routes, local preference, AS_PATH length and other attributes in the context of the vendor's decision process and policy.
Next-hop resolution: verify that the selected next hop is reachable through the relevant routing information.
Forwarding state: confirm that the selected route is programmed into the forwarding information base (FIB) and that traffic can actually traverse the path.
Session health: correlate BGP neighbour state transitions, resets, timers and logs with the time the traffic anomaly occurred.

Security note: BGP route leaks and route hijacks can affect reachability or redirect traffic. Monitoring is only one defence. Apply appropriate prefix filtering, route-policy controls, peer validation and, where supported and operationally suitable, RPKI-based Route Origin Validation. ROV validates route-origin authorisation; it does not by itself validate the entire AS_PATH or prevent every routing attack.

5.5 — BGP and traffic telemetry must be correlated

A route appearing in the BGP table is not the same as a successful application connection. BGP provides control-plane reachability information, while the forwarding plane handles packets. An investigation must distinguish a route that has been received from one that has been selected, installed and successfully used.

Observation	Possible interpretation	Next validation step
Traffic volume drops suddenly	Application failure, link problem, filtering or a route change	Correlate interface counters, flow records, route events and endpoint health
AS_PATH changes unexpectedly	Legitimate policy change, upstream change or suspicious announcement	Check the source peer, prefix origin, route policy and trusted routing evidence
BGP route exists, but traffic fails	Unresolved next hop, FIB inconsistency, ACL, MTU issue or downstream failure	Check route selection, next-hop recursion, FIB, interface state and packet path
Repeated BGP session resets	Transport instability, peer configuration, resource pressure or timer expiry	Inspect neighbour logs, TCP connectivity, interface errors and control-plane load
Unexpected outbound traffic	New workload behaviour, misconfiguration or potentially malicious activity	Correlate destination, workload identity, DNS, security events and route history

5.6 — Operational requirements for reliable visibility

Observability systems are only as useful as the quality and consistency of their data. Time synchronisation is essential when correlating events across routers, firewalls, switches, hosts and collectors. Retention policies should preserve enough history to establish a baseline and investigate delayed reports. Telemetry collectors should be monitored for gaps, overload and export failures.

Synchronise timestamps across network devices, endpoints and telemetry platforms.
Document flow sampling rates, SPAN limitations and which interfaces are actually observed.
Monitor telemetry loss, collector health, storage capacity and data-retention windows.
Restrict access to sensitive flow records, packet captures and routing logs.
Protect management and telemetry channels against unauthorised access or tampering.
Establish normal traffic and routing baselines before treating deviations as incidents.

Section takeaway: Effective visibility connects traffic behaviour, security events and routing decisions. Flow records show communication patterns; packet captures provide deeper protocol evidence; encrypted-traffic metadata supports behavioural analysis; and BGP telemetry helps explain changes in network reachability. Correlation turns those separate signals into a defensible engineering investigation.

NETWORK INSIGHT · VISIBILITY WORKBENCH

Investigate the Data Center with Network Telemetry

Explore how flow records, interface counters, packet captures and BGP telemetry reveal different parts of a network problem. Select a diagnostic view to correlate traffic behavior with routing and forwarding state.

Traffic and telemetry sources Illustrative telemetry

Network connection Selected diagnostic path Warning signal Potential fault

Selected source

Flow telemetry

Flow records summarize conversations between endpoints. They help identify major talkers, traffic direction and volume, but do not normally contain every packet's full contents.

Engineering insight

No single telemetry source explains every fault. Correlating traffic records, interface counters, packet evidence and routing state can narrow the cause much faster.

Diagnostic actions

Flow recordsSelected evidence source

Traffic patternWhat it can reveal

PartialEvidence coverage

Illustrative diagnostic evidence

Signal	Example observation	Interpretation	Status
Flow records	Large app-to-database flow	Identify traffic source and volume	Useful
Interface counters	Output drops increasing	Investigate congestion or queue pressure	Check
Packet capture	Retransmissions observed	Correlate loss, delay and endpoint behavior	Check
BGP telemetry	Next-hop route changed	Verify route selection and forwarding state	Useful

What the data tells you

Flow telemetry helps locate large or unexpected conversations. It is valuable for traffic engineering and capacity planning, but its detail depends on the exporter, sampling and collection interval.

Correlate before concluding

Compare timestamps and affected paths across telemetry sources. A counter increase, a packet symptom and a routing change are stronger evidence when they align in time and relate to the same flow.

Educational demonstration only. The evidence values are illustrative examples, not live measurements. Available telemetry, sampling, counter definitions, packet visibility and BGP monitoring capabilities vary by platform and deployment.

06 / Respond

Incident Response, Threat Detection and Network Forensics

When suspicious activity is detected, the priority shifts from observation to controlled investigation and response. Engineers must validate the evidence, determine the scope of the incident, preserve relevant records and contain the threat without creating unnecessary operational disruption.

Explore the incident response lifecycle, practical forensic techniques, containment decisions and BGP incident investigation, from the first alert through recovery and lessons learned.

06 / Respond

Incident Response, Threat Detection and Network Forensics

Detecting suspicious activity is only the beginning. Effective incident response requires engineers to validate the alert, establish the scope of the incident, preserve evidence, contain the threat and restore services safely. Network telemetry, endpoint records, security controls and routing information provide complementary evidence that helps teams understand what happened and determine the appropriate response.

A security alert does not automatically establish that a system has been compromised. A sudden increase in outbound traffic, an unfamiliar destination, repeated authentication failures or an unexpected BGP route change may indicate a security incident, a configuration error or legitimate operational activity. The objective is to investigate systematically rather than act on an assumption.

Core principle: Incident response must balance containment, evidence preservation, service availability and business risk. Every action should have a clear purpose, an authorised scope and a way to verify its outcome.

6.1 — The incident response lifecycle

A practical response process moves from preparation through detection, analysis, containment, eradication and recovery. Lessons learned then feed back into monitoring, configuration standards and security controls. In a serious incident, these activities may overlap, and responders may need to revisit earlier stages as new evidence emerges.

PrepareTools, roles and playbooks

→

DetectAlert and validate

→

ContainLimit further impact

→

EradicateRemove root cause

→

RecoverRestore and monitor

Phase 1

Preparation

Define incident ownership, escalation paths, approved containment actions, logging requirements, backup procedures and communication channels before an incident occurs.

Phase 2

Detection and analysis

Validate the alert, establish a timeline, identify affected assets and correlate evidence from network, endpoint, identity and security systems.

Phase 3

Containment

Reduce the opportunity for further harm using proportionate, authorised controls. Consider dependencies and the operational consequences of isolating a device or blocking a communication path.

Phase 4

Eradication and recovery

Remove the confirmed cause, remediate affected systems, restore trusted configurations and verify that services are operating correctly before returning to normal operations.

6.2 — Detect and validate suspicious network activity

Detection systems compare observed activity against rules, threat intelligence, behavioural baselines and known indicators. The resulting alerts must be evaluated in context. An IP address associated with a threat feed, for example, is a useful investigation lead but may be outdated, shared by multiple services or unrelated to the activity under examination.

Signal	Potential concern	Validation approach
Unusual outbound data volume	Possible data transfer or unexpected application behaviour	Check flow records, destination, workload identity, scheduled jobs and endpoint events.
Repeated authentication failures	Possible credential guessing or account misuse	Correlate identity logs, source addresses, successful logins and account activity.
Repeated DNS queries	Possible misconfiguration, automated activity or suspicious domain generation	Inspect query patterns, resolver logs, domain context and the originating endpoint.
Unexpected east-west traffic	Possible lateral movement or a newly introduced dependency	Compare with workload communication baselines, segmentation policy and process telemetry.
Unexpected BGP announcement	Possible route leak, misconfiguration or unauthorised route propagation	Verify the peer, prefix, origin, policy, route history and expected maintenance activity.
Repeated connection resets	Possible service instability, filtering or hostile connection activity	Correlate packet captures, firewall decisions, server logs and interface counters.

6.3 — Interactive incident investigation

Select a scenario to explore the evidence an engineer should collect before choosing a response. These are illustrative investigation paths, not automated diagnoses.

Investigation workbench

Select a scenario to reveal its evidence requirements and recommended next steps.

INVESTIGATION SCENARIO Unusual outbound data transfer

Establish which host initiated the transfer, the destination, data volume, timing and expected application behaviour. Correlate flow records with endpoint activity, DNS context and available security logs before deciding whether containment is necessary.

6.4 — Network forensics and evidence preservation

Network forensics reconstructs relevant events from available evidence. Depending on the environment, that evidence may include packet captures, NetFlow or IPFIX records, DNS logs, firewall events, authentication records, endpoint telemetry, configuration changes and BGP update history.

Packet captures

Preserve the relevant capture interval and observation point. Record any capture filters, truncation settings, packet loss indicators and limitations affecting interpretation.

Centralised logs

Preserve original records where possible, document collection methods and correlate events using consistent timestamps and asset identifiers.

Configuration history

Compare approved and current firewall rules, ACLs, routing policies, interface configuration and relevant changes made before or during the incident.

Evidence handling: Follow organisational policy and applicable legal requirements. Restrict access to sensitive records, preserve provenance, document collection and handling, and use cryptographic hashes where appropriate to help demonstrate that collected files have not changed.

# Example: calculate a SHA-256 hash of a collected evidence file # Run on an authorised system with access to the evidence file. shasum -a 256 incident-capture.pcap # Record the hash, filename, collection time, collector and storage location. # A hash supports integrity verification; it does not prove the original # capture was complete or that its contents are accurate.

6.5 — Containment: reduce risk without creating another outage

Containment aims to limit the impact of an incident while preserving enough operational capability to investigate and recover. The right action depends on confidence in the evidence, the criticality of the affected service, the likely threat and the available alternatives.

Containment option	Potential benefit	Risk to consider
Isolate an endpoint	Limits some network communications from a suspected host	May interrupt critical services or remove access needed for investigation.
Apply a targeted firewall rule	Restricts a verified malicious flow or exposure	An overly broad rule may block legitimate dependencies.
Disable compromised credentials	Reduces the opportunity for further unauthorised access	Can interrupt automated workloads or legitimate administrative activity.
Change routing policy	May mitigate an established routing threat or isolate an affected path	Incorrect changes can cause route withdrawal, traffic blackholing or wider reachability loss.
Increase targeted monitoring	Improves evidence collection when immediate isolation is not justified	Monitoring alone may be insufficient when active harm is continuing.

Operational safeguard: High-impact containment actions should follow the incident response plan and appropriate authorisation. In a critical network, coordinate changes with service owners, assess failover behaviour and maintain a documented rollback plan wherever feasible.

6.6 — BGP incidents: distinguish control-plane anomalies from forwarding failures

BGP-related incidents require particular care because an incorrect response can affect reachability well beyond a single router. Unexpected route advertisements, route leaks, session instability and policy errors may alter the paths available to traffic. However, a route change alone does not establish malicious intent.

# Illustrative BGP incident triage checklist 1. Identify the affected prefix and first observed change. 2. Determine which peer advertised or withdrew the route. 3. Compare AS_PATH, origin and relevant route attributes. 4. Verify import/export policy and expected maintenance changes. 5. Check the selected route and next-hop resolution. 6. Confirm the FIB and test the affected forwarding path. 7. Apply an authorised mitigation and verify the result.

In an interdomain routing incident, useful safeguards may include carefully maintained prefix filters, explicit import and export policies, maximum-prefix controls, appropriate peer authentication and RPKI Route Origin Validation where supported. ROV helps validate whether an origin AS is authorised to originate a prefix; it does not validate every aspect of the route or prevent every possible BGP attack.

Within an autonomous system, engineers should also check iBGP propagation, route-reflector behaviour, next-hop reachability, IGP dependencies and forwarding state. A route may be visible in one routing table but absent from another, or selected by BGP without being usable for forwarding.

6.7 — Recovery, verification and lessons learned

Recovery is more than removing a block or restarting a service. Before returning affected systems to normal operation, teams should verify that the original cause has been addressed, required security controls are working and relevant traffic is following expected paths.

Confirm the affected systems, accounts, prefixes or services have been identified.
Remove the confirmed cause and remediate the relevant configuration or vulnerability.
Restore systems from trusted sources where required and verify their security posture.
Validate routing, next-hop reachability, forwarding state and application connectivity.
Maintain heightened monitoring for recurrence and unexpected changes.
Document the timeline, decisions, evidence, impact, recovery steps and outstanding risks.
Update detection rules, runbooks, access controls and training based on lessons learned.

6.8 — Measure incident response effectiveness

Response metrics help organisations identify gaps and improve operational readiness. They should be interpreted alongside incident severity, service impact and investigation quality rather than treated as isolated performance targets.

Detection

Mean time to detect

Measures the elapsed time between a defined incident starting point and detection. The starting point must be consistently defined for comparisons to be meaningful.

Response

Mean time to contain

Measures how long it takes to implement an effective containment measure after the relevant response clock starts.

Quality

Recurrence and false positives

Review recurring incident causes, unnecessary escalations, detection coverage and whether mitigations actually reduce risk without unacceptable disruption.

Section takeaway: Effective incident response joins detection, evidence preservation, containment, remediation and recovery into a controlled engineering process. Network telemetry and BGP visibility provide important context, but reliable decisions depend on corroborating evidence, understanding operational dependencies and verifying that corrective actions work.

NETWORK INSIGHT · RESILIENCE LAB

What Happens When a Data Center Fabric Fails?

Simulate illustrative link and spine failures in a leaf-spine fabric. Follow the surviving paths, inspect the remaining connectivity and identify what engineers should verify before declaring the fabric resilient.

Leaf-spine fabric connectivity Normal operation

Available connection Traffic path Failed component or link Illustrative alternate path

Scenario analysis

Normal operation

Both spine switches are available. Each leaf has two uplinks, allowing traffic to use multiple fabric paths when routing and forwarding are configured accordingly.

Engineering lesson

Redundancy is only useful when alternate paths are operational, routing converges correctly and the remaining links can carry the offered load.

Test controls

8 / 8Available leaf-to-spine links

2 / 2Available spine switches

2Illustrative paths between selected leaves

Failure investigation checklist

Check	Question	Why it matters	Status
Physical state	Are the links and devices operational?	Confirms the failure boundary	Baseline
Routing	Have paths been withdrawn or recalculated?	Control-plane state influences path selection	Baseline
Forwarding	Are alternate next hops installed?	Confirms traffic has a usable path	Baseline
Capacity	Can surviving links carry the load?	Prevents congestion after failover	Baseline

Resilience is more than redundancy

Multiple links and switches create alternate paths, but they do not guarantee uninterrupted service. Detection time, routing convergence, forwarding updates, hashing and remaining capacity all affect the outcome.

Validate the recovery

After a failure, verify reachability and application behavior, inspect routing and forwarding state, compare interface counters and confirm that the surviving fabric is not overloaded.

Educational model only. This is a simplified two-spine, four-leaf fabric. The diagram illustrates connectivity and possible alternate paths; it does not calculate live routing convergence, ECMP hashing, throughput, packet loss or application availability. Actual behavior depends on topology, routing policy, hardware and configuration.

Network Insight · Architecture Briefing

Designing Data Center Networks for Scale, Visibility and Resilience

Data center topology is more than a diagram of switches and links. It determines how workloads communicate, how capacity scales, how routing adapts to change and how engineers isolate faults when the network behaves unexpectedly.

Six ideas to take away

From architectural foundations to operational troubleshooting, each design decision affects the next.

Choose the right architecture

Three-tier designs separate access, aggregation and core functions. Leaf-spine fabrics provide a more consistent path structure for many modern, east-west-heavy workloads.

Understand the fabric

Leaf switches connect endpoints; spine switches interconnect the leaves. Clos-style designs can scale horizontally, subject to port count, link speed and oversubscription choices.

Design for real traffic

North-south and east-west flows create different demands. Capacity planning must consider traffic concentration, ECMP distribution, burstiness and workload communication patterns.

Make routing part of the design

BGP can provide scalable underlay routing in a leaf-spine fabric. Route policy, next-hop reachability, ECMP and convergence behavior all influence the forwarding outcome.

Make the network observable

Flow telemetry, interface counters, packet captures and routing telemetry reveal different parts of the story. Correlating them helps connect symptoms to causes.

Prove resilience under failure

Redundant links are only the beginning. Test failover, verify routing and forwarding state, check surviving capacity and confirm that affected applications recover.

A simplified fabric mental model

SPINE LAYER Path diversity

Leaf 1Leaf 2Leaf 3Leaf 4

ComputeStorageApplications

Think in layers, paths and failure domains.

A useful fabric design connects physical topology, routing behavior and application requirements. Each layer answers a different engineering question: what is connected, where can traffic travel and what happens when a component is lost?

✓Topology: Are there enough paths between the required endpoints?

✓Routing: Are the intended next hops available and installed?

✓Capacity: Can the remaining fabric carry the expected load?

✓Operations: Can telemetry identify the fault and verify recovery?

The engineering takeaway

A resilient data center is not simply one with redundant links. It is one whose behavior can be understood, measured and tested.

Build around the workload, understand the trade-offs in the chosen topology, validate routing and forwarding, and exercise failure scenarios before they occur in production. The goal is not just connectivity—it is predictable performance and recoverable service.

```

☕

Support Network Insight

Help support independent engineering guides, interactive networking tools, and future learning labs.

Support Network Insight ```

Green data center with eco friendly electricity usage tiny person concept. Database server technology for file storage hosting with ecological and carbon neutral power source vector illustration.

Data Center Design with Active Active design

August 12, 2014

by Matt Conran Blog

Network Insight · High Availability Architecture

Active-Active Data Center Design: Building for Continuous Availability

What happens when one data center can no longer serve its workloads? An active-active architecture aims to keep multiple sites operational at the same time, distributing traffic and capacity while preparing services to withstand component, network and site failures.

Modern applications rarely depend on a single network, server or storage system. They operate across interconnected infrastructure, distributed services and, increasingly, multiple locations. Active-active design brings these components together into an architecture where more than one site can actively serve workloads. But operating two sites simultaneously is not simply a matter of duplicating hardware. Routing, application behavior, data consistency, traffic distribution and failure detection must work together.

Data Center A

Primary active site in this illustration

Applications Compute Storage

Serving workloads

Data Center B

Second active site in this illustration

Applications Compute Storage

Serving workloads

01 · Connectivity

Route around failures

Use diverse network paths and appropriate routing policies to preserve reachability when links or devices fail.

02 · Consistency

Keep application state safe

Design replication, quorum and application behavior around the consistency requirements of the service.

03 · Recovery

Prove service continuity

Test site isolation, traffic shifts and recovery behavior instead of assuming that redundancy guarantees availability.

The essential distinction: both sites can be active without every application being able to survive every failure. True service resilience depends on the complete design—including data consistency, health detection, routing, capacity and application recovery behavior.

Multi-site networking BGP and DCI Data consistency High availability Failure engineering

Network Insight · Technical guide

Explore the Architecture

Navigate the six engineering layers of an active-active design, from site architecture and BGP to application resilience, recovery and cloud connectivity.

06 Sections

01 Understanding Active-Active Design Foundations and availability
02 Multi-Site Architecture and BGP Routing and traffic engineering
03 Layer 2, Overlays and Mobility VXLAN/EVPN and workload placement
04 Application and State Consistency Replication, sessions and data
05 Failure and Recovery Observability and validation
06 Cloud Connectivity and Design Choices Hybrid networking and resilience

Architecture→Resilience→Recovery 6 topics

Section 01 · Foundations

Understanding Active-Active Data Center Design

Explore how two or more data center sites can serve workloads concurrently, how this differs from active-passive resilience, and why application behaviour, dependency design and failure handling matter as much as network redundancy.

Active vs. passiveAvailabilityFailure domains

Section 01 · Architecture Fundamentals

Understanding Active-Active Data Center Design

Active-active design allows two or more data center sites to serve production workloads concurrently. The objective is to use distributed capacity while maintaining service continuity when a component, network path or site becomes unavailable.

Traditional active-passive designs generally keep one site serving traffic while another is prepared to take over during a failure. In an active-active design, both sites perform useful production work during normal operation. Traffic may be distributed through global load balancing, application-aware routing, anycast services or other mechanisms appropriate to the application and network architecture.

This can improve resource utilisation and provide additional capacity options. It also introduces design questions that cannot be solved by routing alone: where application state lives, how data is replicated, how clients reach healthy services, and what happens when the sites lose contact with one another.

Active-active operating model Conceptual architecture · Not a deployment blueprint

Data Center A

Serving production traffic
Local compute and services

Active

Data Center B

Serving production traffic
Local compute and services

Active

Both sites are active under normal conditions. The actual traffic split, replication method, routing policy and failover behaviour depend on the application and implementation.

Three principles that shape the design

Concurrent workload

Both sites contribute to normal service delivery. Capacity planning must account for traffic distribution and whether surviving infrastructure can absorb additional demand after a failure.

Consistent service behaviour

Applications must handle sessions, shared state, data replication and concurrent updates according to their consistency and availability requirements.

Designed failure response

Health checks, routing policies, load balancers and application recovery logic must work together to direct clients towards services that can actually respond.

Important distinction: two active sites do not automatically provide application resilience. If both sites depend on the same database, identity service, WAN path or control-plane component, that shared dependency may still become a single point of failure.

Questions to answer before choosing active-active

Can the application safely run across multiple sites at the same time?
How much data loss, if any, is acceptable following a site failure?
What recovery time is required, and how will client traffic move to healthy services?
Can one surviving site handle the required workload during a major failure?
How will split-brain conditions, network partitions and inconsistent state be detected and handled?

Engineering takeaway

Active-active is an end-to-end service architecture, not simply a pair of interconnected data centers. Reliable outcomes depend on coordinated network design, application behaviour, data consistency, capacity planning and tested recovery procedures.

Interactive Lab 01 · Architecture explorer

What Happens When a Data Center Fails?

Switch the deployment model or simulate a site failure. Observe the illustrative traffic paths, site states and engineering implications. This is a conceptual model, not a live network simulator.

Choose a scenario

Data Center A

Active · serving traffic

Data Center B

Active · serving traffic

SCENARIO: ACTIVE-ACTIVE

Both sites serve production traffic

Workloads at both sites are available to serve requests. Traffic distribution depends on application placement, routing, load balancing and health checks. Shared dependencies and data consistency still need deliberate design.

Engineering checks

Confirm application and data consistency.
Verify each site's independent capacity.
Test routing and service health during failure.

Section 02 · Routing architecture

Multi-Site Network Architecture and BGP

Follow traffic between data centers and external networks. Understand how BGP exchanges reachability, how routing policy and multipath influence path selection, and how engineers avoid asymmetric traffic, route leaks and unexpected failover behaviour.

DCI designBGP policyTraffic engineering

Section 02 · Network Architecture

Multi-Site Network Architecture and BGP

An active-active design needs more than two functioning data centers. It needs predictable routing, resilient inter-site connectivity and a clear policy for how users and applications reach services when paths or entire sites fail.

The network provides the connectivity between data centers, users, applications and external services. In a multi-site architecture, Data Center Interconnect (DCI) links connect the sites, while routing and traffic-management systems determine which paths are available and where traffic should be sent.

Border Gateway Protocol (BGP) is commonly used to exchange reachability information between sites, WAN networks and external networks. Its policy controls can influence path selection and traffic distribution. However, BGP does not by itself verify that an application is healthy or that its data is consistent.

Two-site network architecture Illustrative topology · Not to scale

Data Center A

Applications and local services

Leaf Spine Border

Local routing · BGP policy

Data Center B

Applications and local services

Leaf Spine Border

Local routing · BGP policy

Diverse links only provide meaningful resilience when they avoid relevant shared risks, such as the same conduit, carrier equipment, power source or upstream failure domain.

The three network design layers

Layer 01

Data Center Interconnect

DCI may use routed IP connectivity, an optical transport service or an overlay architecture. Choose bandwidth, latency and path diversity according to application traffic and replication requirements—not simply the number of physical links.

Layer 02

Routing and reachability

BGP can advertise site prefixes and apply routing policy. Multipath forwarding may use multiple eligible routes when supported and correctly configured. Route selection and forwarding behaviour must be validated for the actual platform and topology.

Layer 03

Service traffic steering

DNS-based global load balancing, anycast, application load balancers or other steering mechanisms can direct clients towards available services. Health checks must reflect genuine service readiness, not just network reachability.

BGP: control the path, then validate the outcome

A common design advertises service or site prefixes from more than one location. The routing policy determines which advertisements are accepted and how paths are preferred. Depending on the design, engineers may use attributes such as LOCAL_PREF within an autonomous system, AS_PATH for inter-AS policy, or multipath features to support selected equal-cost paths.

The important distinction is between route availability and service availability. A router may have a valid route to a site whose application is unhealthy. Conversely, an application may be healthy while a routing or traffic-steering policy prevents clients from reaching it. These systems need coordinated health signals and carefully tested failure behaviour.

Design warning — avoid accidental traffic tromboning. If a client reaches a service in one site but the application retrieves data or calls a dependency in the other site, traffic may cross the DCI unnecessarily. This can increase latency, consume inter-site bandwidth and complicate failure handling. Keep traffic local where practical, while preserving the required consistency and availability guarantees.

What should happen during a failure?

DCI link failure: traffic should use a surviving path where one exists, with convergence and capacity validated under load.
Border router failure: alternate routing paths should remain available without creating a forwarding loop or black hole.
Site failure: route withdrawal and service-health mechanisms should steer new traffic away from unavailable services.
Application failure: health checks should stop advertising or selecting an endpoint that cannot deliver the required service.
Network partition: routing and application policies must avoid unsafe assumptions about the remote site's state or data ownership.

Architecture checks before deployment

Identify shared physical and logical failure domains across DCI paths.
Document prefix advertisements, routing policy, route filtering and intended path preferences.
Check convergence behaviour and whether surviving links can carry the expected load.
Validate inbound and outbound traffic steering independently; they may follow different paths.
Test application health checks alongside routing changes and site-isolation scenarios.

Engineering takeaway

BGP and DCI establish and control network reachability; they do not guarantee that an application is usable. A resilient active-active design coordinates routing, traffic steering, application health, capacity and data consistency—and verifies the result through controlled failure tests.

Interactive Lab 02 · Multi-site routing

Trace the Route Between Data Centers

Explore how inter-site connectivity, BGP reachability and routing policy influence traffic. Simulate a DCI link failure or route withdrawal, then select a router or network to inspect its role.

Select a routing scenario

Illustrative preferred path Illustrative alternate path Failed or withdrawn path

STATE: NORMAL ROUTING

Reachability is available through the configured paths

The diagram illustrates an available inter-site path and route advertisement. Actual forwarding depends on the installed route, routing policy, next-hop reachability and data-plane health.

Select a network component

Select either border router, an application network or the external AS in the diagram to explore its role in route propagation and traffic forwarding.

DCI statusAvailable

Illustrative routeAdvertised

Engineering focusValidate policy

Item	Illustrative state	What to verify
Inter-site path	Available	Independent physical paths, capacity and forwarding health.
Route advertisement	Present	Accepted prefixes, next hop, policy and route installation.
Service reachability	Check required	End-to-end application health, not only control-plane state.

Section 03 · Overlays and mobility

Layer 2 Extension, Overlays and Workload Mobility

Compare stretched Layer 2 networks with routed Layer 3 designs, and examine how VXLAN/EVPN overlays can support multi-site connectivity. Consider workload mobility, broadcast scope, failure propagation and the risk of traffic tromboning before extending networks between sites.

VXLAN / EVPNLayer 2 vs. Layer 3Workload mobility

Section 03 · Layer 2 and Overlay Design

Layer 2 Extension, Overlays and Workload Mobility

Workload mobility across data centers can simplify some application migrations and recovery scenarios. The challenge is deciding how much network continuity is actually required—and whether extending a Layer 2 domain is the right way to achieve it.

In a multi-site environment, applications may need to move between sites for maintenance, capacity balancing, disaster recovery or infrastructure changes. Some workloads can move while retaining their IP addresses; others can reconnect through a new endpoint or service address. The application’s requirements should determine the network design, rather than assuming that every workload needs a stretched subnet.

A Layer 2 extension carries an Ethernet broadcast domain across multiple locations. This can preserve subnet membership and support certain migration workflows, but it also extends the domain’s failure and operational characteristics. A Layer 3 design keeps subnets local to each site and uses routing to connect them, often reducing the scope of Layer 2 failure propagation.

Approach A

Stretched Layer 2

The same VLAN or subnet is available across sites, subject to the chosen extension technology and design constraints.

Can support address-preserving workload mobility.
May carry broadcast and unknown-unicast traffic across the DCI.
Requires careful control of loops, failure domains and inter-site latency.
Can make partial failures harder to diagnose.

Approach B

Routed Layer 3

Each site uses its own routed subnets, and connectivity between sites is provided through routing and application-level service discovery or traffic steering.

Provides clearer Layer 3 failure boundaries.
Supports route summarisation and explicit traffic policy.
May require application, DNS or load-balancer changes during migration.
Often simplifies fault isolation and scaling.

VXLAN and EVPN: separating the overlay from the underlay

VXLAN (Virtual Extensible LAN) encapsulates Layer 2 frames in UDP packets so that virtual networks can be carried across an IP underlay. VXLAN Tunnel Endpoints (VTEPs) perform encapsulation and decapsulation. EVPN (Ethernet VPN), commonly used as a control plane for VXLAN fabrics, distributes MAC and IP reachability information and supports policy-driven tenant connectivity.

This separation allows an IP-routed fabric to carry multiple logical networks without requiring the physical underlay itself to be one large Layer 2 domain. EVPN can support distributed gateway designs and multi-homing, depending on the implementation. It does not, however, remove the need to design inter-site routing, failure handling and application consistency deliberately.

Overlay network across two sites Conceptual VXLAN/EVPN model

Data Center A

IP underlay · Local VTEP

Workload A1 Workload A2

Data Center B

IP underlay · Local VTEP

Workload B1 Workload B2

VXLAN overlay · Logical network connectivity EVPN can distribute endpoint reachability; the underlay provides IP transport between VTEPs.

The diagram is illustrative. Actual VTEP placement, EVPN route exchange, gateway placement and whether an overlay spans both sites depend on the design.

Overlay connectivity can provide flexibility, but stretching a tenant network across sites should be a deliberate requirement—not the default for every application.

Where do OTV, FabricPath and LISP fit?

Legacy DCI

OTV

Overlay Transport Virtualization was designed to extend selected Layer 2 connectivity between data centers while limiting some traditional Layer 2 extension risks. Treat it as a platform- and lifecycle-dependent technology, not a universal modern default.

Fabric technology

FabricPath

Cisco FabricPath introduced a Layer 2 fabric approach using a fabric control plane rather than relying on conventional spanning tree for all forwarding decisions. It is mainly relevant when understanding existing or legacy environments.

Identifier / locator

LISP

Locator/ID Separation Protocol separates endpoint identity from routing location. It can support mobility and mapping use cases in appropriate architectures, but it is not interchangeable with VXLAN or EVPN.

Design principle: do not extend Layer 2 simply because an application currently uses a fixed IP address. First check whether the application can tolerate a different subnet, use a stable service endpoint, or reconnect through DNS or a load balancer. A routed design may meet the requirement with a smaller failure domain.

Workload mobility and traffic tromboning

Preserving a workload’s IP address does not guarantee that its traffic path remains optimal. After a workload moves, clients or dependent services may still reach it through a remote site. This can create traffic tromboning: packets travel across the DCI to reach a service and may cross back again to reach another dependency.

The result can be increased latency, unnecessary bandwidth consumption and a more complicated failure path. Mobility planning should therefore include endpoint discovery, routing updates, load-balancer behaviour, dependency locality and the time needed for existing sessions to recover.

Questions to resolve before extending a network

Does the workload genuinely require the same subnet at both sites?
What are the latency, bandwidth and MTU requirements across the DCI?
How are MAC/IP reachability and gateway behaviour handled after a move or failure?
Can a Layer 2 loop, broadcast event or control-plane issue affect both sites?
Will traffic remain local to the active workload and its dependencies?
What happens to established sessions when the workload or site becomes unavailable?

Engineering takeaway

VXLAN and EVPN can provide scalable overlay connectivity, while routed designs can keep site boundaries clearer. Choose Layer 2 extension only when the application and operational requirements justify it, and test mobility, failure isolation and traffic locality before relying on it for resilience.

Interactive design lab · 03

Layer 2 Extension, Overlays and Workload Mobility

Compare three ways to connect data-centre sites. Change the design, simulate an inter-site failure, and examine what happens to addressing, mobility and the fault domain.

1. Choose a connectivity design

2. Test the inter-site connection

✓

Shared segment is available

Both sites use the same example subnet. This can support address continuity, but it also extends Layer 2 dependencies across the sites.

Addressing Same subnet at both sites

Mobility approach Layer 2 continuity

Failure consideration Broadcast and loop control

Engineering trade-off

A stretched VLAN can make subnet continuity easier, but the design must deliberately control loop prevention, broadcast scope, gateway behaviour and inter-site failure handling.

Define the Layer 2 extension boundary
Test behaviour during link and site failure
Validate gateway redundancy and convergence
Confirm application recovery requirements

Educational topology only — not a live network simulator. VXLAN is an encapsulation technology; EVPN can provide a BGP-based control plane. Actual mobility, redundancy and failure behaviour depend on the chosen platform, configuration and application design.

Section 04 · Application resilience

Application, Storage and State Consistency

Network redundancy cannot guarantee application availability on its own. Examine stateless and stateful services, session continuity, synchronous and asynchronous replication, quorum, split-brain risk, and the trade-offs between recovery time and potential data loss.

Application stateReplicationRPO / RTO

Section 04 · Application and Data Resilience

Application, Storage and State Consistency

A network can keep routes available while an application remains unable to serve requests. Active-active resilience therefore depends on how application instances, user sessions and persistent data behave across sites—not just on redundant links and routers.

In an active-active environment, application instances in different data centers may process requests concurrently. That arrangement works well only when the application can manage shared state, concurrent operations and failures consistently. A stateless service is often easier to distribute because each request can be handled by any healthy instance, provided its dependencies are also available.

Stateful applications introduce more complexity. They may depend on databases, distributed caches, message queues, session stores or file systems. Each dependency needs an explicit availability and consistency strategy. Replicating the application tier alone does not make the whole service resilient if its database or identity provider remains a single point of failure.

Application and data layers across two sites Conceptual architecture

Data Center A

Active application instance

Application service A Local connection pool Data replica / access path

Data Center B

Active application instance

Application service B Local connection pool Data replica / access path

Replication may be synchronous, asynchronous or managed by a distributed data platform. The correct model depends on the database technology, latency, failure assumptions and application consistency requirements.

The drawing represents logical relationships, not a recommendation to permit unrestricted concurrent writes to every replicated data store.

Synchronous versus asynchronous replication

Replication determines how updates made at one site become available at another. The key trade-off is between acknowledgement latency, consistency guarantees and the amount of data that may be lost during a failure. These characteristics vary by platform and configuration; the terms alone do not guarantee a particular recovery outcome.

Replication model A

Synchronous replication

A write is acknowledged only after the configured synchronous durability or commit conditions have been met.

Can reduce the recovery point exposure for acknowledged writes.
Inter-site latency can affect write response times.
A link or site failure may affect write availability, depending on quorum and product design.
Requires careful planning for network partitions and failover.

Replication model B

Asynchronous replication

A write can be acknowledged before the remote replica has received or committed the update.

Can reduce the impact of remote-site latency on write acknowledgement.
Replication lag may leave the remote site temporarily behind.
A sudden failure can result in loss of updates not yet replicated.
Recovery procedures must account for lag and data reconciliation.

Important: synchronous replication does not automatically mean zero data loss in every failure scenario, and asynchronous replication does not always imply unacceptable loss. The real outcome depends on commit semantics, quorum, durable storage, topology and the failure being considered. Define the required Recovery Point Objective (RPO) and Recovery Time Objective (RTO), then validate them against the actual platform.

Three application-level challenges

Challenge 01

Session continuity

If a client moves between sites, its session may depend on local memory, a shared session store, a signed token or another mechanism. Design for reconnects and retries instead of assuming a transport path change preserves every session.

Challenge 02

Concurrent updates

Two active instances may attempt to modify the same logical record. Transactions, conflict resolution, idempotency and application-specific consistency rules must prevent duplicated or conflicting operations.

Challenge 03

Dependency availability

An application instance may be healthy while its database, authentication service, queue or external API is not. Service health should reflect the dependencies required to complete real user operations.

Quorum, network partitions and split-brain

A network partition can leave two sites running while preventing them from communicating. If both sides independently accept conflicting writes or claim ownership of the same resource, the system may enter a split-brain condition. Restoring connectivity does not necessarily reconcile the resulting state safely.

Distributed systems often use quorum, leases, fencing tokens, leader election or other coordination mechanisms to decide which operations are permitted during a partition. A two-site deployment may need a third failure-independent voting participant or witness, depending on the platform. A witness can help establish a decision, but it does not replace data replication or application-level recovery design.

Network partition is not the same as site failure. If the DCI fails, both sites may still be running and serving local clients. The architecture must decide whether each site can safely continue, whether one side must stop writes, and how service ownership is re-established when communication returns.

Questions to answer before production

Can application instances safely process requests concurrently in both sites?
Where is session state stored, and what happens when a client is redirected?
What RPO and RTO are required for each critical data store?
How are quorum, ownership and split-brain conditions handled during a partition?
Can the surviving site sustain the required transaction rate and connection load?
How will replication lag, failed writes, stale reads and recovery conflicts be detected?
Have failover and failback been tested with real application transactions, not just ping tests?

Engineering takeaway

Active-active application resilience requires an explicit strategy for state, replication, coordination and recovery. Network redundancy helps services communicate, but only the combined behaviour of the application, data platform and network determines whether service can continue safely during a failure.

Interactive design lab · 04

Application, Storage and State Consistency

A network can remain connected while an application becomes unsafe or unavailable. Explore replication modes, simulate an inter-site partition, and examine why two active sites need a deliberate write-ownership strategy.

1. Choose a data replication model

2. Choose a scenario

✓

Replication is operating normally

In this conceptual synchronous model, a write waits for the configured remote acknowledgement before it is confirmed. Real guarantees depend on the data platform and commit policy.

Write behaviour Waits for required acknowledgement

Data risk Depends on commit and quorum rules

Recovery priority Validate quorum and ownership

Engineering checks

A healthy network path does not prove that a transaction is durable or that both replicas agree. Check the application's consistency contract and the data platform's failure behaviour.

Confirm what a successful commit guarantees
Define quorum and write ownership
Measure replication delay and transaction errors
Test failover, fencing and safe recovery

Educational model only — no real transactions or replication occur. The diagram illustrates design trade-offs, not a guarantee that a particular database or storage platform behaves this way. Synchronous replication does not automatically prevent data loss in every failure; asynchronous replication may have a non-zero recovery point.

Section 05 · Operations and recovery

Failure Scenarios, Observability and Recovery

Move beyond theoretical redundancy and test how the architecture behaves under real failure conditions. Trace link loss, BGP changes, application dependency failures, replication lag and site isolation using telemetry, incident procedures and measured recovery objectives.

Failure testingTelemetryFailover and failback

Section 05 · Observability and Recovery

Failure Scenarios, Observability and Recovery

Active-active architecture must be designed around failure, not just normal operation. Reliable recovery depends on detecting what has failed, identifying the affected services, understanding the data state and directing traffic towards capacity that can safely serve it.

Redundant links and duplicated infrastructure reduce some risks, but they do not eliminate failures. A site can remain reachable while its applications are unhealthy; a routing session can remain established while packets are being dropped; and a database replica can be online while lagging behind the active data set. Effective operations require end-to-end observability across network, application and data layers.

Detect

Identify the real failure

Use telemetry and health checks to distinguish a failed link, an unreachable site, a degraded application and an unhealthy data dependency.

Decide

Choose a safe response

Determine whether traffic should move, whether writes can continue, and whether the surviving site has enough capacity to take additional work.

Validate

Prove service recovery

Confirm that users can complete meaningful transactions, not merely that interfaces are up or routing adjacencies have recovered.

Five failure scenarios to test

Inter-data-center link failure

One DCI path fails while both sites remain operational. Check route convergence, alternate-path capacity, packet loss and whether replication or application dependencies are affected.

Validate: path failover · latency · loss · capacity

Border router or routing failure

A border device fails, a route is withdrawn or a policy change directs traffic incorrectly. Verify that alternate routes are usable and that recovery does not create a forwarding loop or black hole.

Validate: BGP state · route changes · forwarding path

Application or dependency failure

Network connectivity remains available, but an application instance, database or identity service cannot complete requests. Check whether health signals remove the affected endpoint from service selection.

Validate: synthetic transactions · dependency health

Replication lag or data inconsistency

The remote data copy falls behind or a replication process stops. Establish whether the secondary site is safe to promote, what data could be missing and how consistency will be verified before normal service resumes.

Validate: replication lag · RPO · recovery procedure

Complete site loss or network partition

A site becomes unavailable, or both sites remain running but lose communication. Confirm the failover decision, capacity at the surviving site, data ownership and safeguards against split-brain.

Validate: site isolation · quorum · service continuity

A practical incident investigation sequence

During an incident, investigate from the observed user impact towards the underlying dependency. Correlating timestamps across routing events, interface counters, application logs and replication metrics helps distinguish the initiating failure from symptoms that appear later.

From alert to verified recovery Operational workflow

1 Scope impact

Identify affected users, services, sites and transactions.

2 Correlate signals

Compare network, application and data telemetry.

3 Apply safe recovery

Follow tested routing, failover and data procedures.

4 Verify service

Test transactions, data integrity and performance.

This is a general investigation pattern. Follow your incident procedures and service-specific recovery runbooks when making production changes.

What should you monitor?

Layer	Useful signals	What they help reveal
Network	Interface errors, packet loss, latency, BGP changes, path utilisation	Transport degradation, route changes and capacity pressure
Application	Request success rate, latency, error rate, synthetic transactions	Whether users can actually complete operations
Data	Replication lag, commit failures, consistency checks, storage health	Whether the remote site has a safe and sufficiently current data set
Platform	CPU, memory, connection pools, queue depth, storage throughput	Whether the surviving site can absorb additional workload
Recovery	Failover duration, RTO/RPO achievement, recovery test results	Whether the design meets its agreed recovery objectives

Do not confuse an infrastructure health check with a service health check. An interface can be up, a BGP session can be established and a server can respond to ping while a real transaction fails. Use application-aware probes and representative synthetic transactions wherever practical.

Failover, failback and recovery objectives

Recovery Time Objective (RTO) defines the target time to restore a service after disruption. Recovery Point Objective (RPO) defines the acceptable data-loss window, expressed as time or an equivalent recovery-point requirement. Different applications and data stores may need different objectives; a single pair of numbers should not automatically be applied to every service.

Failover is only half of the recovery story. Failback—returning service to its intended operating arrangement—can introduce another period of risk. Before returning workloads, confirm data consistency, replication health, routing policy, capacity and the correct ownership of stateful services. Avoid automatic failback if it could trigger repeated site switching or disrupt active sessions.

Operational readiness checklist

Document the expected behaviour for link, device, application, data and full-site failures.
Ensure alerts identify the affected service and provide enough context for investigation.
Test routing convergence and verify actual packet forwarding after route changes.
Confirm that the surviving site has enough compute, storage, network and application capacity.
Define who can authorise failover, data promotion and failback during an incident.
Record measured recovery times and data outcomes during controlled exercises.
Review failures and near misses to improve runbooks, monitoring and architecture.

Engineering takeaway

Resilience must be measured through observed service outcomes. Combine network telemetry, application health, data replication metrics and tested recovery procedures to determine whether the active-active architecture can withstand the failures it was designed to handle.

Interactive design lab · 05

Failure Scenarios, Observability and Recovery

Simulate a router failure, an inter-site partition or a complete site outage. Observe the illustrative network state, monitoring signals and recovery checks that matter before declaring an active-active service healthy.

1. Select a failure scenario

✓

Both sites are operational

The conceptual topology shows active services at both sites. Monitoring should confirm end-to-end transaction success, route health, replication status and available capacity.

Network reachability Primary and inter-site paths available

Application service Both sites serving

Recovery decision Monitor and validate health

Observability panel

Routing adjacencyHealthy

End-to-end transactionsHealthy

Inter-site connectivityHealthy

Replication / coordinationHealthy

Recovery checklist

A green interface or established BGP session is only one signal. Correlate network telemetry with application transactions, data health and the capacity of the surviving site.

Confirm the failure and its scope
Validate the alternate forwarding path
Check application and data integrity
Verify recovery objectives are met

Educational simulation only. No live devices, BGP sessions or monitoring systems are contacted. The alternate path is illustrative; real failover depends on topology, routing policy, detection timers, capacity, application dependencies and the platform's recovery behaviour.

Section 06 · Hybrid architecture

Cloud Connectivity and Modern Design Choices

Explore how data centers connect to cloud networks using private connectivity, VPNs, dynamic routing, VPC peering and hub-based services such as Google Cloud Network Connectivity Center. Compare resilience, route control, security, latency and cost without assuming one cloud model fits every deployment.

Hybrid cloudVPC and BGPConnectivity choices

Active-Active Data Center Design · Section 06

Cloud Connectivity and Modern Design Choices

Active-active design does not stop at the data center perimeter. Applications may depend on private cloud services, public cloud workloads, shared identity systems, or data platforms in another environment. The challenge is to connect these resources without creating hidden dependencies, unnecessary latency, or new single points of failure.

1. Separate data center interconnect from hybrid cloud connectivity

A data center interconnect (DCI) connects separate data center sites. Hybrid cloud connectivity connects an enterprise network or data center to cloud networks and services. These designs can overlap, but they solve different problems and should be assessed independently.

For example, two active data centers may exchange application traffic over a dedicated DCI while each site also has its own connection to a cloud provider. Another design may route both sites through a shared cloud connectivity hub. The right choice depends on application placement, traffic patterns, security boundaries, and failure-domain requirements.

Illustrative hybrid connectivity model

Data Center A Active workloads Local routing and security

Private DCI or routed inter-site path

Data Center B Active workloads Independent failure domain

Enterprise / DCI edge Routing policy and controlled egress

Private connectivity or VPN

Cloud network VPCs, cloud workloads and services

This is a conceptual view, not a required physical topology. Connectivity can be direct, hub-and-spoke, or provider-mediated. Redundancy must be verified across the complete path, not just at the visible connection.

2. Choose the connectivity model for the workload

Cloud connectivity is not one universal design. Dedicated private links, provider-managed private connectivity, and IPsec VPNs offer different trade-offs in capacity, latency, encryption, deployment time, and cost.

Private connectivity

Often suited to predictable, sustained traffic and workloads with demanding bandwidth or latency requirements. Validate provider availability, port and circuit diversity, routing hand-offs, and the actual end-to-end service commitment.

Site-to-site VPN

IPsec VPN can provide encrypted connectivity over an IP underlay. High-availability VPN designs use redundant tunnels or endpoints, but the underlying internet paths and provider failure domains still matter.

Cloud routing

Cloud routers and BGP sessions can exchange routes dynamically where supported. Define which prefixes are advertised, accepted, preferred, or filtered, and test behaviour when a peer or path fails.

Hub-and-spoke connectivity

A cloud transit or connectivity hub can simplify connections between multiple networks. It also becomes an important policy and operational component that needs its own resilience and capacity planning.

3. Understand VPCs, peering and cloud connectivity hubs

In Google Cloud, a Virtual Private Cloud (VPC) provides a logical network boundary for cloud resources. VPC connectivity, Cloud VPN, Cloud Router, and Network Connectivity Center (NCC) address different parts of hybrid and multi-network design. Other providers have comparable concepts, but their capabilities and terminology differ.

Building block	Design role	Important consideration
VPC network	Provides a logical network environment for cloud resources.	Plan address ranges, segmentation, route policy, and access controls.
VPC peering	Connects eligible VPC networks through a peering relationship.	Do not assume peering provides transitive routing through another VPC.
Cloud VPN	Provides IPsec-based connectivity between networks.	Plan tunnel redundancy, routing, encryption requirements, and failover testing.
Cloud Router / BGP	Supports dynamic route exchange for compatible connectivity services.	Validate advertised prefixes, learned routes, route priorities, and withdrawal behaviour.
Network Connectivity Center	Provides a hub-based framework for supported network connectivity patterns.	Check supported resources, topology constraints, route propagation, and configuration requirements.

Do not confuse connectivity with end-to-end resilience

A cloud hub, a pair of VPN tunnels, or two physical circuits does not automatically make an application highly available. Shared carrier infrastructure, common provider endpoints, route-policy errors, cloud-region dependencies, DNS, identity, and application state can still create a common failure mode.

4. Design the routing and failure behaviour

Once connectivity is established, define what happens when a link, tunnel, router, data center, or cloud attachment becomes unavailable. Routing must support the intended failover behaviour without creating loops, accidental transit, or unintended access between security zones.

Route control

Advertise only required prefixes. Apply explicit import and export policy, avoid overlapping address spaces, and understand how route preference changes during failure.

Path diversity

Check whether redundant connections use independent devices, facilities, carriers, cloud endpoints, and regions where the availability objective requires it.

Security boundaries

Apply segmentation, firewall policy, identity controls, and inspection requirements to both primary and failover paths. A backup route should not bypass required controls.

Application dependencies

Validate DNS, authentication, secrets, databases, monitoring, and other shared services. Network reachability alone does not prove that the service can operate.

5. Compare the design options before committing

Select connectivity according to measured workload requirements and operational constraints—not simply because a particular product or topology is available.

Design question	What to evaluate
Traffic profile	Peak and sustained throughput, burst behaviour, packet size, and growth expectations.
Performance	Latency, jitter, packet loss, regional distance, and application sensitivity.
Availability	Independent paths, redundant endpoints, provider failure domains, and tested recovery behaviour.
Routing	Address planning, BGP policy, route propagation, failover timing, and prevention of unintended transit.
Security	Encryption, segmentation, inspection, identity, and compliance requirements.
Operations and cost	Provisioning lead time, monitoring, support ownership, data transfer charges, and ongoing operational effort.

6. Validate the architecture with failure tests

Test the complete application path, including the network and the dependencies it uses. Record what users experience, how quickly the system detects a failure, which routes change, and whether recovery occurs within the required objective.

Disconnect one connectivity path and verify traffic moves as designed.
Withdraw a route and confirm that only intended prefixes change reachability.
Test a cloud attachment or VPN endpoint failure where the environment permits.
Verify that security policy remains enforced on the alternate path.
Measure application recovery, not just routing convergence.
Document ownership, alerts, escalation, and restoration procedures.

Engineering takeaway

Active-active data center design and hybrid cloud connectivity should be designed together where workloads depend on both—but they are not the same problem. Choose private connectivity, VPN, peering, or a hub-based model according to workload needs, then validate route policy, failure-domain independence, security, and application recovery end to end.

Interactive design explorer · Section 6

Cloud Connectivity and Modern Design Choices

Choose a connectivity pattern to explore how an on-premises network can connect to cloud networks. Compare the conceptual architecture, routing considerations and security decisions before selecting a design for a real environment.

01 / Select a connectivity pattern

02 / Architecture view Encrypted tunnel

Conceptual diagram, not a provider-specific reference architecture. Redundant tunnels, gateways and route exchange must be designed and tested explicitly.

03 / Design assessment

Site-to-site VPN

Build an encrypted connection between the enterprise network and the cloud provider's VPN endpoint. The internet may carry the tunnel traffic, while routing controls which prefixes can traverse it.

Connection model Encrypted tunnel

Routing focus Prefixes and failover

Security focus IKE / IPsec policy

Resilience focus Multiple tunnels

Engineering checks

Confirm tunnel encryption, authentication and key-management requirements.
Validate route advertisements, allowed prefixes and return-path symmetry.
Test failure detection, alternate paths, throughput and maximum packet size.

Watch-outs

A tunnel being established does not prove application reachability.
One tunnel or gateway can remain a single point of failure.

High availability is an end-to-end property

Consider independent circuits, gateways, cloud regions or zones where supported, route convergence, capacity under failure, and the application's own dependency model. Two connections that share the same failure domain may not provide meaningful redundancy.

Routing and security are separate decisions

Route exchange determines reachability; firewall policy, segmentation, identity and encryption determine what is permitted and protected. Validate both directions and test the application path rather than relying on a successful routing adjacency alone.

Network Insight · Article Summary

Active-Active Data Center Design: Key Takeaways

A practical recap of the architecture, routing, application dependencies, failure planning and cloud connectivity decisions behind a resilient multi-site data center.

Active-active data center design aims to use multiple sites to serve workloads concurrently while maintaining service continuity when links, network devices, applications or entire sites fail. Achieving this requires more than duplicating infrastructure: routing, application state, data consistency, security and recovery procedures must work together.

Architecture and workload placement

Define which services operate at each site, how traffic is distributed, and what happens when one location becomes unavailable. Active-active is a service and workload design decision, not simply a topology with two data centers.

Multi-site routing and BGP

Use explicit routing policy, controlled prefix advertisement and deliberate path selection. Validate convergence, asymmetric paths, route withdrawal and the capacity of the surviving site under failure conditions.

Layer 2 extension and overlays

Choose between routed inter-site connectivity, selective Layer 2 extension and overlay-based approaches according to workload requirements. Avoid extending broadcast domains across sites without a clear operational and failure-containment justification.

Application and data consistency

Understand replication lag, transaction behaviour, quorum, write ownership and recovery objectives. Network reachability alone cannot guarantee consistent data or prevent conflicting writes during a partition.

Failure detection and recovery

Monitor routing and infrastructure alongside application transactions, replication health and user-visible service availability. Test realistic failures and document safe failover, failback and recovery procedures.

Cloud connectivity and modern choices

Select VPN, private connectivity, network peering or hub-based connectivity based on reachability, routing, security and resilience requirements. Confirm provider-specific route propagation and service limitations before implementation.

Six engineering principles to carry into your design

Design around service availability, not device availability alone.
Ensure the surviving site has sufficient capacity for its intended failure role.
Keep routing policy, application behaviour and data consistency aligned.
Use Layer 2 extension only where the workload genuinely requires it.
Measure end-to-end service health, not just link and routing status.
Prove failover and recovery through repeatable tests and documented outcomes.

Final perspective: resilience must be engineered and tested

The strongest active-active designs make failure behaviour explicit. They define how traffic moves, how stateful applications continue safely, how data is protected, and how operators recognise and resolve degraded service. They also account for partial failures, where a site remains reachable but can no longer safely perform its intended role.

No single technology guarantees resilience. BGP, overlays, replication, cloud connectivity and redundant infrastructure are building blocks whose effectiveness depends on the requirements, implementation and operational testing around them.

The central takeaway

Build for the failure you need to survive, define the service outcome you must preserve, and test that outcome before relying on the architecture in production.

```

☕

Support Network Insight

Help support independent engineering guides, interactive networking tools, and future learning labs.

Support Network Insight ```

Data Center Network Design

August 11, 2014

by Matt Conran Blog

Data Center Network Design

Data centers are crucial in today’s digital landscape, serving as the backbone of numerous businesses and organizations. A well-designed data center network ensures optimal performance, scalability, and reliability. This blog post will explore the critical aspects of data center network design and its significance in modern IT infrastructure.

Data center network design involves the architectural planning and implementation of networking infrastructure within a data center environment. It encompasses various components such as switches, routers, cables, and protocols. A well-designed network ensures seamless communication, high availability, and efficient data flow.

The traditional three-tier network architecture is being replaced by more streamlined and flexible designs. Two popular approaches gaining traction are the spine-leaf architecture and the fabric-based architecture. The spine-leaf design offers low latency, high bandwidth, and improved scalability, making it ideal for large-scale data centers. On the other hand, fabric-based architectures provide a unified and simplified network fabric, enabling efficient management and enhanced performance.

Network virtualization, powered by technologies like SDN, is transforming data center network design. By decoupling the network control plane from the underlying hardware, SDN enables centralized network management, automation, and programmability. This results in improved agility, better resource allocation, and faster deployment of applications and services.

With the rising number of cyber threats, ensuring robust security and resilience has become paramount. Data center network design should incorporate advanced security measures such as firewalls, intrusion detection systems, and encryption protocols. Additionally, implementing redundant links, load balancing, and disaster recovery mechanisms enhances network resilience and minimizes downtime.

Matt Conran

Highlights: Data Center Network Design

Data Center Network Design

To embark on a successful network design journey, it is essential first to understand the data center’s specific requirements. Factors such as scalability, bandwidth, latency, and reliability need to be carefully assessed. By comprehending the data center’s unique needs, network architects can lay a solid foundation for an optimized design.

Efficiency and resilience are at the core of any well-designed data center network. Building on the requirements identified in the previous section, architects must consider redundancy, load balancing, and fault tolerance principles. The design should minimize single points of failure while maximizing resource utilization and network performance.

Various network topologies and architectures can be employed in data center network design. Each option offers unique advantages and trade-offs, from traditional hierarchical designs to modern approaches like leaf-spine architectures. This section will explore different topologies, highlighting their strengths and considerations.

Virtualization and SDN have revolutionized data center network design, offering increased flexibility and agility. By abstracting network functions from physical infrastructure, virtualization allows for dynamic resource allocation and improved scalability. SDN further enhances network programmability, enabling centralized management and automation. This section will delve into the benefits and implementation considerations of these technologies.

Network, security, and computing

– A data center architecture consists of three main components: the data center network, the data center security, and the data center computing architecture. In addition to these three types of architecture, there are also data center physical architectures and data center information architectures. The following are three typical compositions.

– Network architecture for data centers: Data center networks (DCNs) are arrangements of network devices interconnecting data center resources. They are a crucial research area for Internet companies and large cloud computing firms. The design of a data center depends on its network architecture.

– It is common for routers and switches to be arranged in hierarchies of two or three levels. There are three-tier DCNs: fat tree DCNs, DCells, and others. There has always been a focus on scalability, robustness, and reliability regarding data center network architectures.

– Data center security refers to physical practices and virtual technologies for protecting data centers from threats, attacks, and unauthorized access. It can be divided into two components: physical security and software security. A firewall between a data center’s external and internal networks can protect it from attack.

Data Center Network Design Considerations

a. Understanding the Requirements

Before embarking on the design process, it’s crucial to understand the data center’s unique requirements. Factors such as power and cooling, network connectivity, scalability, and security are vital in determining the design approach. By thoroughly assessing these requirements, architects can create a blueprint that aligns with the organization’s current and future needs.

b. Optimizing Physical Layout

The physical layout of a data center significantly impacts its efficiency and performance. This section will delve into rack placement, aisle design, cable management, and airflow optimization. By adopting best practices in physical layout design, data center operators can minimize energy consumption, reduce maintenance costs, and enhance overall operational efficiency.

c. Redundancy and Resilience

Data centers demand high levels of redundancy and resilience to ensure uninterrupted operations. This section will explore the concept of redundancy in power and cooling systems, backup generators, redundant network connectivity, and failover mechanisms. Implementing robust redundancy measures helps mitigate the risk of downtime and ensures continuous availability of critical services.

4. Security and Compliance

Data centers store sensitive and valuable information, making security a top priority. This section will discuss the importance of physical security measures, access controls, surveillance systems, and fire suppression mechanisms. Additionally, we will explore compliance standards and regulations that govern data center operations, such as SOC 2, ISO 27001, and GDPR.

5. Embracing Green Initiatives

As environmental sustainability gains importance, data centers seek ways to minimize their carbon footprint. This section will focus on energy-efficient design practices, including using renewable energy sources, efficient cooling techniques, and server virtualization. Data centers can contribute to a more sustainable future by adopting green initiatives.

Data Center Network Security

### What is Cloud Armor?

Cloud Armor is a security service offered by Google Cloud that provides protection against distributed denial-of-service (DDoS) attacks and other web-based threats. It leverages Google’s global infrastructure to offer scalable and reliable protection, ensuring that your applications and services remain available and secure even in the face of large-scale attacks.

### Key Features of Cloud Armor

Cloud Armor comes packed with several features that make it an indispensable tool for modern enterprises. Some of its key features include:

– **DDoS Protection:** Automatically detects and mitigates DDoS attacks, ensuring minimal disruption to your services.

– **Web Application Firewall (WAF):** Provides customizable rules to block malicious traffic and protect against common web vulnerabilities.

– **Edge Security Policies:** Allows you to define security policies at the edge of your network, ensuring threats are mitigated before they reach your core infrastructure.

– **Adaptive Protection:** Uses machine learning to identify and respond to evolving threats in real-time.

### Understanding Edge Security Policies

One of the standout features of Cloud Armor is its ability to implement edge security policies. These policies enable organizations to enforce security measures at the periphery of their network, providing an additional layer of defense. By stopping threats at the edge, you can prevent them from penetrating deeper into your network, thereby reducing the risk of data breaches and other security incidents.

Edge security policies can be tailored to your specific needs, allowing you to block traffic based on various criteria such as IP address, geographic location, and request patterns. This granular control helps you enforce stringent security measures while maintaining the performance and availability of your services.

### Benefits of Using Cloud Armor

Deploying Cloud Armor offers several benefits that can significantly enhance your security posture. These include:

– **Scalability:** Designed to handle traffic spikes and large-scale attacks, ensuring your services remain available even under heavy load.

– **Customization:** Flexible rules and policies allow you to tailor security measures to your unique requirements.

– **Proactive Defense:** Real-time threat detection and mitigation keep your applications protected against the latest cyber threats.

– **Cost-Effective:** By leveraging Google’s global infrastructure, you can achieve enterprise-level security without the need for significant upfront investment.

### What is Google Network Connectivity Center?

Google Network Connectivity Center is a unified platform designed to manage and monitor network connections across a variety of environments. Whether you’re dealing with on-premises data centers, cloud environments, or hybrid setups, NCC provides a centralized control point. It simplifies the complexities involved in network management, allowing IT teams to focus on optimizing performance rather than troubleshooting issues.

### Key Features of Google NCC

#### Unified Management

NCC offers a single pane of glass for managing network connections, making it easier to oversee and control your entire network infrastructure. This unified management approach reduces the need for multiple tools and interfaces, streamlining operations and increasing efficiency.

#### Flexible Connectivity Options

Google NCC supports a range of connectivity options, including VPNs, interconnects, and peering. This flexibility ensures that you can choose the best connectivity method for your specific needs, whether it’s connecting remote offices or integrating with third-party cloud services.

#### Real-Time Monitoring and Analytics

One of the standout features of NCC is its real-time monitoring and analytics capabilities. With detailed insights into network performance and traffic patterns, you can quickly identify and resolve issues, optimize resource allocation, and ensure consistent network performance.

Understanding Network Tiers

Network tiers are a concept that categorizes network traffic based on its importance and priority. By classifying traffic into different tiers, businesses can allocate resources accordingly and optimize their network usage. In the case of Google Cloud, there are two main network tiers: Premium Tier and Standard Tier.

The Premium Tier is designed to deliver exceptional performance and reliability. It leverages Google’s global network infrastructure, ensuring low latency and high throughput for critical applications. By utilizing the Premium Tier, businesses can enhance user experience, reduce latency-related issues, and improve overall network performance.

While the Premium Tier offers top-tier performance, the Standard Tier provides a cost-effective solution for non-critical workloads. It offers reliable network connectivity at a lower price point, making it an excellent choice for applications that do not require ultra-low latency or high bandwidth. By strategically utilizing the Standard Tier, businesses can optimize their network spend without compromising on reliability.

Understanding VPC Networking

VPC, or Virtual Private Cloud, is a virtual network dedicated to a specific Google Cloud project. It allows users to define and manage their network resources, including subnets, IP addresses, and firewall rules. With VPC networking, businesses can create isolated environments and control the flow of traffic within their cloud infrastructure.

Google Cloud’s VPC networking offers a range of powerful features. Firstly, it provides global connectivity, allowing businesses to connect resources across regions seamlessly. Additionally, VPC peering enables secure communication between different VPC networks, facilitating collaboration and data sharing. Moreover, VPC networking offers granular control through firewall rules, ensuring robust security for applications and services.

What is Google Cloud CDN?

Google Cloud CDN, short for Content Delivery Network, is a globally distributed network of servers designed to deliver content to users at blazing-fast speed. Cloud CDN minimizes latency and ensures a seamless user experience by caching your content in strategic locations worldwide. Whether it’s static assets, dynamic content, or even streaming media, Cloud CDN optimizes the delivery process, reducing the load on your origin servers and improving overall performance.

Cloud CDN operates by leveraging Google’s extensive network infrastructure. When a user requests content from your website or application, Cloud CDN intelligently routes the request to the nearest edge location. If the content is already cached at that edge location, it is immediately delivered to the user, eliminating the need for a round trip to the origin server. This not only reduces latency but also saves bandwidth and server resources.

Understanding VPC Network Peering

VPC network peering connects VPC networks from different projects or within the same project within Google Cloud. It enables direct communication between these networks, eliminating the need for complex VPN setups or public IP addresses. This seamless connectivity can significantly enhance collaboration, data sharing, and network management.

Enhanced Security: VPC network peering ensures that communication between peered networks remains isolated from the public internet. This adds an extra layer of security by reducing the exposure to potential cyber threats.

Improved Performance: By leveraging VPC network peering, data can be transferred at incredibly high speeds between peered networks. This enables faster resource access, reduces latency, and enhances overall application performance.

Simplified Network Architecture: VPC network peering allows for a more streamlined and simplified network architecture. Instead of relying on complex gateways or routers, communication between VPCs can be established directly, making network management and troubleshooting more straightforward.

Data Center Network Types

a. The Three-Tier Data Center Network

The three-tier DCN architecture has been a traditional approach in data center networking. It consists of three layers: the access layer, the aggregation layer, and the core layer. Each layer serves a specific purpose, from connecting end devices to aggregating traffic and providing high-speed connectivity. This hierarchical design allows for scalability and redundancy, making it a popular choice for many data centers.

b. Unleashing the Power of Fat Tree Data Center Networks

The fat tree DCN, also known as the Clos network, has gained prominence recently due to its ability to handle large-scale data center deployments. Unlike the three-tier DCN, a fat tree network provides multiple paths between devices, enabling better load balancing and higher bandwidth capacity. Fat tree networks offer low-latency communication and enhanced fault tolerance by utilizing a non-blocking switching fabric, making them ideal for mission-critical applications.

c. Exploring the Revolutionary DCell Approach

The DCell architecture takes a novel approach to data center networking and offers a unique perspective on scalability and fault tolerance. DCell networks are based on a hierarchical structure of cells, where each cell consists of a group of servers connected together. This decentralized design eliminates the need for traditional core switches and enables direct server-to-server communication. With its self-organizing capabilities, DCell networks provide excellent scalability, fault tolerance, and efficient resource utilization.

Composition of Data Center Architecture

Routing and Switching:

Routing is the backbone of a data center network, guiding data packets through the labyrinthine pathways. It involves determining the optimal path for data to travel from source to destination, considering network congestion, latency, and cost factors. Advanced routing protocols like Border Gateway Protocol (BGP) enable dynamic route selection, ensuring efficient and fault-tolerant data delivery.

Switching complements routing by facilitating efficient data transmission within a local network. At the heart of a data center, switches act as intelligent traffic controllers, directing data packets to their intended destinations. With features like VLANs (Virtual Local Area Networks) and Quality of Service (QoS), switches prioritize and prioritize traffic, optimizing network performance and ensuring seamless communication.

Example: Spanning Tree Uplink Fast

Spanning Tree Protocol (STP) prevents loops in Ethernet networks by creating a loop-free logical topology and blocking redundant paths. While STP ensures network stability, it can also introduce delays in network convergence. Network downtime caused by STP convergence can be a primary concern for businesses. Even a few seconds of downtime can result in significant losses in critical environments. This is where Spanning Tree Uplink Fast comes into play. Uplink Fast is an enhancement to STP that provides faster convergence times, reducing network downtime and improving overall network efficiency.

How Uplink Fast Works

Uplink Fast allows a switch to detect a link failure on its designated root port and immediately activate an alternate port. This process eliminates the need for the traditional STP convergence process, resulting in faster network recovery times. Uplink Fast is instrumental when network redundancy is crucial, such as in data centers or enterprise networks.

Introducing Spanning Tree MST

Spanning Tree MST enhances the traditional STP, providing a more efficient and flexible solution. MST allows network administrators to divide the network into multiple regions, each with its own Spanning Tree instance. By doing so, MST optimizes network resources and enables load balancing across multiple paths, leading to increased performance and redundancy.

To implement Spanning Tree MST, network switches need to be properly configured. This involves defining regions, assigning VLANs to instances, and configuring parameters such as root bridges and priorities. MST configuration can be complex, but with careful planning and understanding, it offers significant benefits.

Spanning Tree MST offers several key advantages. First, it enables efficient utilization of network resources by load-balancing traffic across multiple paths. Second, it provides enhanced redundancy, ensuring that if one path fails, traffic can automatically reroute through an alternate path. Third, MST simplifies network management by allowing administrators to control traffic flow and prioritize specific VLANs within each instance.

Data Center Security Technologies

Understanding the MAC Move Policy

The MAC Move Policy is a crucial feature in Cisco NX-OS devices that governs the movement of MAC addresses within a network. By defining specific rules and criteria, administrators can control how MAC addresses are learned, aged, and moved across different interfaces and VLANs.

Configuring the MAC Move Policy

Proper configuration is essential to effectively utilizing the MAC Move Policy. This section will guide you through the step-by-step process of configuring the policy on Cisco NX-OS devices. From defining the MAC move parameters to implementing the policy on specific interfaces or VLANs, we will cover all the necessary commands and considerations to ensure a seamless configuration experience.

Understanding MAC ACLs

MAC ACLs, also known as Ethernet ACLs or Layer 2 ACLs, operate at the data link layer of the OSI model. Unlike traditional IP-based ACLs, which focus on network layer addresses, MAC ACLs allow administrators to filter traffic based on MAC addresses. This enables granular control over network access, providing an additional layer of defense against unauthorized devices.

By implementing MAC ACLs on the Nexus 9000 series, network administrators can exercise enhanced control over their network environment. MAC ACLs prevent MAC address spoofing, mitigating the risk of unauthorized devices gaining access. Furthermore, they enable the isolation of specific devices or groups of devices, ensuring that only designated entities can communicate within a given VLAN or network segment.

Understanding VLANs and ACLs

Before we embark on our journey to explore VLAN ACLs’ potential, let’s establish a solid foundation by understanding VLANs and ACLs individually. VLANs (Virtual Local Area Networks) allow us to logically segment networks, improving performance, scalability, and network management. On the other hand, ACLs (Access Control Lists) act as gatekeepers, controlling traffic flow and enforcing security policies.

VLAN ACLs serve as a crucial layer of defense in protecting our networks from unauthorized access, malicious activities, and potential breaches. By implementing VLAN ACLs, we can define granular rules that filter and restrict traffic between VLANs, ensuring that only desired communication occurs. This level of control empowers network administrators to mitigate risks, maintain data integrity, and enforce compliance.

Understanding Nexus Switch Profiles

Nexus switch profiles are a feature of Cisco’s Nexus series switches that allow administrators to define and manage a group of switches as a single entity. By creating a profile, administrators can easily configure and monitor all switches within the group, eliminating the need for repetitive manual configurations. This centralization of management simplifies network administration and saves valuable time and resources.

One of the primary advantages of using Nexus switch profiles is the ability to streamline network operations. With a profile in place, administrators can make changes or updates to configurations across multiple switches simultaneously. This significantly reduces the risk of configuration errors and ensures consistent settings throughout the network. Furthermore, the centralized management approach simplifies troubleshooting and enables faster resolution of network issues.

Data Center Technologies

Understanding Layer 3 Etherchannel

Layer 3 Etherchannel is a link aggregation technique that combines multiple physical links between switches into a single logical channel. By bundling these links together, traffic can be distributed across them, increasing overall bandwidth capacity and providing load-balancing capabilities. Unlike Layer 2 Etherchannel, Layer 3 Etherchannel operates at the network layer, allowing traffic to be routed.

To configure Layer 3 Etherchannel, several steps need to be followed. First, the physical interfaces on the switches need to be identified and grouped into the Etherchannel bundle. Then, a logical interface, the Port-Channel interface, is created and assigned an IP address. Subsequently, routing protocols or static routes can be configured on the Port-Channel interface to enable communication between different networks.

Layer 3 Etherchannel supports various load-balancing algorithms, determining how traffic is distributed across the bundled links. Standard algorithms include source IP, destination IP, and round-robin. Each algorithm has advantages and considerations depending on the network requirements and traffic patterns.

Cisco Nexus 9000 Port Channel

Implementing Port Channels on Cisco Nexus 9000 switches offers several advantages. Firstly, it provides increased link bandwidth, allowing for efficient data transfer and reducing bottlenecks. Secondly, Port Channels enhance network resilience by providing link redundancy. In a link failure, traffic seamlessly switches to the remaining active links. Lastly, Port Channels enable load balancing, distributing network traffic evenly across the aggregated links for optimal utilization.

Setting up a Port Channel on Cisco Nexus 9000 switches is straightforward. Administrators can configure Port Channels using the Link Aggregation Control Protocol (LACP) or the Port Aggregation Protocol (PAgP). Administrators can maximize the benefits of this feature by adequately configuring interfaces and assigning them to the Port Channel.

Understanding Unidirectional Link Detection (UDLD)

UDLD is a layer 2 protocol that helps identify and mitigate the presence of unidirectional links in a network. It works by exchanging periodic messages between neighboring switches to verify bidirectional connectivity. By detecting unidirectional links, UDLD helps prevent potential network issues such as black holes, spanning-tree loops, and data loss.

Cisco Nexus 9000 switches offer seamless integration and support for UDLD. To enable UDLD on a Nexus 9000 switch, administrators can utilize simple commands within the switch configuration. By configuring UDLD timers, administrators can customize the frequency of UDLD messages exchanged between switches. Additionally, UDLD can be configured to operate in either standard or aggressive mode, depending on the specific needs of the network environment.

Understanding VRRP

VRRP, an essential networking protocol, provides automatic failover and load-balancing capabilities. It allows multiple routers to work as a virtual group, presenting a single IP address. By intelligently distributing network traffic, VRRP ensures seamless connectivity even in the face of router failures.

The Nexus 9000 Series, Cisco’s flagship product line, offers a range of cutting-edge features, including VRRP. Designed to meet the demands of modern networks, these switches deliver exceptional performance, scalability, and flexibility. With the Nexus 9000 Series, network administrators can harness the power of VRRP to build a robust and highly available network infrastructure.

Example: Data Center WAN Protocol

BGP, also known as the routing protocol of the Internet, is responsible for exchanging routing and reachability information among autonomous systems (AS). It enables routers to make intelligent decisions about the most optimal paths for data transmission. Unlike interior gateway protocols, BGP focuses on routing between different networks rather than within a single network.

BGP operates on a trust-based model, where routers form peer relationships to exchange routing information. These peers establish connections and exchange routing updates, allowing them to build a complete picture of network reachability. BGP uses a sophisticated algorithm that considers multiple factors, such as path length, quality of service, and policy-based decisions, to determine the best route for traffic.

Understanding BGP AS Prepend

AS Prepend involves adding additional Autonomous System (AS) numbers to the AS path attribute of BGP advertisements. By manipulating the AS path, network operators can influence inbound traffic routing decisions by neighboring autonomous systems. This technique makes a specific path appear less desirable, diverting traffic to alternative paths.

AS Prepend holds excellent potential for optimizing network routing in various scenarios. It can achieve load balancing across multiple links, redirect traffic to less congested paths, or prefer specific transit providers. By carefully implementing AS Prepend, network administrators can improve network performance, reduce latency, and enhance overall service quality.

Recap: Border Gateway Protocol (BGP) is data centers’ most commonly used routing protocol. It has been used to connect Internet systems worldwide for decades and can also be used outside a data center. The BGP protocol is a standard-based open-source software package. It’s more common to find BGP peering between data centers over the WAN. However, we see BGP often used purely inside the data center.

Understanding Leaf and Spine Networks

Leaf and spine networks, also known as Clos networks, are a modern approach to data center architecture. The design revolves around a hierarchical structure consisting of two key components: leaf switches and spine switches. Leaf switches connect directly to endpoints, while spine switches interconnect the leaf switches, forming a non-blocking fabric. This architecture eliminates bottlenecks and enables seamless scalability.

BGP (Border Gateway Protocol) is a crucial routing protocol in leaf and spine networks. It ensures efficient data forwarding between leaf switches using a set of rules known as BGP route advertisements. By default, BGP requires every router to have a full mesh of connections with all other routers in the network, which can be resource-intensive. This is where BGP route reflection comes into play.

Understanding BGP Route Reflection

BGP route reflection, at its core, is a method that allows a BGP speaker to reflect routing information to its peers, alleviating the need for full-mesh connectivity. Designating specific BGP routers as route reflectors streamlines and manages the network structure.

The utilization of BGP route reflection offers several advantages. First, it reduces the number of required BGP peering sessions, resulting in a simplified and less resource-intensive network. Second, route reflection enhances scalability by eliminating the need for full-mesh connectivity, particularly in large-scale networks. Third, it improves convergence time and reduces BGP update processing overhead, enhancing overall network performance.

The third wave of application architectures

Google and Amazon, two of the world’s leading web-scale pioneers, developed a modern data center. The third wave of application architectures represents these organizations’ search and cloud applications. Towards the end of the 20th century, client-server architectures and monolithic single-machine applications dominated the landscape. This third wave of applications has three primary characteristics:

Unlike client-server architectures, modern data center applications involve a lot of communication between servers. In client-server architectures, clients communicate with monolithic servers, which either handle the request entirely themselves or communicate with fewer than a handful of other servers, such as database servers. Search (or Hadoop, its more popular variant) employs many mappers and reducers instead of search. In the cloud, virtual machines can reside on different nodes but must communicate seamlessly. In some cases, VMs are deployed on servers with the least load, scaled out, or balanced loads.

A microservices architecture also increases server-to-server communication. This architecture is based on separating a single function into smaller building blocks and interacting with them. Each block can be used in several applications and enhanced, modified, and fixed independently in such an architecture. Since diagrams usually show servers next to each other, East-West traffic is often called server communication. Traffic flows north-south between local networks and external networks.

Scale and resilience

The sheer size of modern data centers is characterized by rows and rows of dark, humming, blinking machines. As opposed to the few hundred or so servers of the past, a modern data center contains between a few hundred and a hundred thousand servers. To address the connectivity requirements at such scales, as well as the need for increased server-to-server connectivity, network design must be rethought. Unlike older architectures, modern data center applications assume failures as a given. Failures should be limited to the smallest possible footprint. Failures must have a limited “blast radius.” By minimizing the impact of network or server failures on the end-user experience, we aim to provide a stable and reliable experience.

Data Center Goal: Interconnect networks

The goal of data center design and interconnection network is to transport end-user traffic from A to B without any packet drops, yet the metrics we use to achieve this goal can be very different. The data center is evolving and progressing through various topology and technology changes, resulting in multiple network designs. The new data center control planes we see today, such as Fabric Path, LISP, THRILL, and VXLAN, are driven by a change in the end user’s requirements; the application has changed. These new technologies may address new challenges, yet the fundamental question of where to create the Layer 2/Layer three boundaries and the need for Layer 2 in the access layer remains the same. The question stays the same, yet the technologies available to address this challenge have evolved.

Example Protocol: Understanding VXLAN

VXLAN, an encapsulation protocol, enables the creation of virtualized Layer 2 networks over an existing Layer 3 infrastructure. By extending the Layer 2 domain, VXLAN allows the seamless transfer of network traffic between geographically dispersed data centers. It achieves this by encapsulating Ethernet frames within IP packets, providing flexibility and scalability to network virtualization.

Scalability and Flexibility: VXLAN addresses the limitations of traditional VLANs by allowing for a significantly more significant number of virtual networks—up to 16 million—compared to the 4,096 limit of VLANs. This scalability enables organizations to allocate virtual networks more efficiently while accommodating the growing demands of cloud-based applications and services.

Enhanced Network Segmentation and Isolation: VXLAN provides improved network segmentation by creating logical networks that are isolated from one another, even if they share the same physical infrastructure. This isolation enhances security and enables more granular control over network traffic, facilitating efficient multi-tenancy in cloud environments.

Modern Data Centers

There is a vast difference between modern data centers and what they used to be just a few years ago. Physical servers have evolved into virtual networks that support applications and workloads across pools of physical infrastructure and into a multi-cloud environment. There are multiple data centers, the edge, and public and private clouds where data exists and is connected. Both on-premises and cloud-based data centers must be able to communicate. Data centers are even part of the public cloud. Cloud-hosted applications use the cloud provider’s data center resources.

Unified Fabric

Through Cisco’s fabric-based data center infrastructure, tiered silos and inefficiencies of multiple network domains are eliminated, and a unified, flat fabric is provided instead, which allows local area networks (LANs), storage area networks (SANs), and network-attached storage (NASs) to be consolidated into one high-performance, fault-tolerant network. Creating large pools of virtualized network resources that can be easily moved and rapidly reconfigured with Cisco Unified Fabric provides massive scalability and resiliency to the data center.

This approach automatically deploys virtual machines and applications, thereby reducing complexity. Thanks to deep integration between server and network architecture, secure IT services can be delivered from any device within the data center, between data centers, or beyond. In addition to Cisco Nexus switches, Cisco Unified Fabric uses Cisco NX-OS as its operating system.

The use of Open Networking

We also have the Open Networking Foundation ( ONF ), which provides open networking. Open networking describes a network that uses open standards and commodity hardware. So, consider open networking in terms of hardware and software. Unlike a vendor approach like Cisco, this gives you much more choice with what hardware and software you use to make up and design your network.

Data Center Performance Parameters

TCP Performance Parameters

TCP (Transmission Control Protocol) is the backbone of modern Internet communication, ensuring reliable data transmission across networks. However, various parameters that determine TCP’s behavior can influence its performance.

Understanding TCP Window Size: One crucial parameter that affects TCP performance is the window size. The TCP window size refers to the amount of data sent before an acknowledgment is required. A larger window size allows more data to be transmitted without waiting for acknowledgments, thus optimizing throughput. However, substantial window sizes can result in congestion and increased retransmissions.

Congestion Control Mechanisms: Congestion control mechanisms are vital in maintaining network stability and preventing congestion collapse. TCP utilizes algorithms such as Slow Start, Congestion Avoidance, and Fast Recovery to regulate data flow based on network conditions. These mechanisms ensure fairness and efficiency, improving TCP performance and avoiding network congestion.

Timeouts and Retransmission: TCP implements a reliable data transfer mechanism using acknowledgments and timeouts. When a packet is not acknowledged within a specific timeframe, it is considered lost, and TCP initiates retransmission. The selection of appropriate timeout values is crucial to balance reliability and responsiveness. Setting shorter timeouts may lead to unnecessary retransmissions, whereas longer ones can increase latency.

Selective Acknowledgments and SACK Options: Selective acknowledgments (SACK) enhance TCP performance and recovery from packet loss. SACK lets the receiver inform the sender about specific out-of-order packets received successfully. This enables the sender to retransmit only the necessary packets, reducing unnecessary retransmissions and improving overall efficiency.

Maximum Segment Size (MSS): The Maximum Segment Size (MSS) is another crucial TCP performance parameter defining the maximum amount of data encapsulated within a single TCP segment. Optimizing the MSS can significantly impact performance, especially when network links have different MTU (Maximum Transmission Unit) sizes.

Understanding TCP MSS

TCP MSS refers to the maximum amount of data encapsulated within a single TCP segment. It represents the size of the payload, excluding headers and other overhead. The MSS value is negotiated during the TCP handshake process and remains constant throughout the connection.

The TCP MSS value has a direct impact on network performance and efficiency. Setting an appropriate MSS value ensures optimal network resource utilization and avoids unnecessary data packet fragmentation. Properly configuring TCP MSS becomes crucial when networks have different MTU (Maximum Transmission Unit) sizes.

Fragmentation occurs when the MSS value exceeds the MTU of a network path. This fragmentation can lead to performance degradation, increased latency, and potential packet loss. By carefully managing the TCP MSS value, network administrators can prevent or minimize fragmentation issues and enhance overall network performance.

Configuring TCP MSS requires a thorough understanding of the network infrastructure and the devices involved. It involves adjusting the MSS value at various points within the network, such as routers, firewalls, and load balancers. Aligning the TCP MSS value with the MTU of the underlying network ensures efficient data transmission and avoids unnecessary fragmentation.

Advanced Topics

VXLAN Flood and Learn Mechanism

The flood-and-learn mechanism in VXLAN plays a crucial role in facilitating communication between virtual machines within the overlay network. When a virtual machine sends a broadcast or unknown unicast frame, the frame is encapsulated in a VXLAN packet and flooded throughout the network. Each VXLAN tunnel endpoint (VTEP) learns the source MAC address and VTEP association, enabling subsequent unicast traffic to be directly delivered.

Multicast is a fundamental component of VXLAN flood and learn, offering several benefits. First, using multicast VXLAN reduces bandwidth consumption compared to traditional flooding techniques. Second, multicast enables efficient replicating broadcast, multicast, and unknown unicast traffic across the overlay network. Third, it enhances network scalability by eliminating the need to maintain a multicast group per tenant.

BGP Multipath

Understanding BGP Multipath

BGP multipath is a feature that enables the installation and usage of multiple paths for a single prefix in the routing table. Traditionally, BGP selects a single best path based on factors such as AS path length, origin type, and path attributes. However, with multipath enabled, BGP can utilize multiple paths simultaneously, distributing traffic across them for load balancing and redundancy purposes.

The utilization of BGP multipath brings several advantages to network operators. First, it enhances network resilience by providing redundant paths. In the event of a link failure or congestion, traffic can be automatically rerouted through available alternate paths, ensuring continuous connectivity. Additionally, BGP multipath facilitates load balancing, enabling more efficient utilization of network resources and better traffic distribution across multiple links.

Understanding BGP Next Hop Tracking

BGP next-hop tracking monitors the reachability of the next-hop IP address associated with a particular route. It allows routers to dynamically adjust their routing tables based on changes in the network topology. Routers can make informed decisions about forwarding traffic by continuously tracking the next hop, ensuring optimal path selection.

Enhanced Network Resiliency: BGP next-hop tracking enables routers to detect and respond to network changes quickly. If a next hop becomes unreachable, routers can automatically reroute traffic to an alternative path, minimizing downtime and improving network resiliency.

Load Balancing and Traffic Engineering: Network administrators gain granular control over traffic distribution with BGP next-hop tracking. By monitoring the reachability of multiple next hops, routers can intelligently distribute traffic across different paths, optimizing resource utilization and improving overall network performance.

Improved Network Convergence: Rapid convergence is crucial in dynamic networks. BGP next hop tracking facilitates faster convergence by promptly updating routing tables when next hops become unreachable. This ensures routing decisions are based on current information, reducing packet loss and minimizing network disruptions.

Related: Before you proceed, you may find the following useful:

Data Center Network Design

The Rise of Overlay Networking

What has the industry introduced to overcome these limitations and address the new challenges? – Network virtualization and overlay networking. In its simplest form, an overlay is a dynamic tunnel between two endpoints that enables Layer 2 frames to be transported between them. In addition, these overlay-based technologies provide a level of indirection that allows switching table sizes to not increase in the order of the number of supported end hosts.

Today’s overlays are Cisco FabricPath, THRILL, LISP, VXLAN, NVGRE, OTV, PBB, and Shorted Path Bridging. They are essentially virtual networks that sit on top of a physical network, and often, the physical network is unaware of the virtual layer above it.

Traditional Data Center Network Design

How do routers create a broadcast domain boundary? Firstly, using the traditional core, distribution, and access model, the access layer is layer 2, and servers served to each other in the access layer are in the same IP subnet and VLAN. The same access VLAN will span the access layer switches for east-to-west traffic, and any outbound traffic is via a First Hop Redundancy Protocol ( FHRP ) like Hot Standby Router Protocol ( HSRP ).

Servers in different VLANs are isolated from each other and cannot communicate directly; inter-VLAN communications require a Layer 3 device. Virtualization’s humble beginnings started with VLANs, which were used to segment traffic at Layer 2. It was expected to find single VLANs spanning an entire data center fabric.

VLAN and Virtualization

The virtualization side of VLANs comes from two servers physically connected to different switches. Assuming the VLAN spans both switches, the same VLAN can communicate with each server. Each VLAN can be defined as a broadcast domain in a single Ethernet switch or shared among connected switches.

Whenever a switch interface belonging to a VLAN receives a broadcast frame (the destination MAC is ffff.ffff.ffff), the device must forward it to all other ports defined in the same VLAN.

This approach is straightforward in design and is almost like a plug-and-play network. The first question is, why not connect everything in the data center into one large Layer 2 broadcast domain? Layer 2 is a plug-and-play network, so why not? STP also blocks links to prevent loops.

The issues of Layer 2

The reason is that there are many scaling issues in large layer 2 networks. Layer 2 networks don’t have controlled / efficient network discovery protocols. Address Resolution Protocol ( ARP ) is used to locate end hosts and uses Broadcasts and Unicast replies. A single host might not generate much traffic, but imagine what would happen if 10,000 hosts were connected to the same broadcast domain. VLANs span an entire data center fabric, which can bring a lot of instability due to loops and broadcast storms.

No hierarchy in MAC addresses

MAC addressing also lacks hierarchy. Unlike Layer 3 networks, which allow summarization and hierarchy addressing, MAC addresses are flat. Adding several thousand hosts to a single broadcast domain will create large forwarding information tables.

Because end hosts are potentially not static, they are likely to be attached and removed from the network at regular intervals, creating a high rate of change in the control plane. Of course, you can have a large Layer 2 data center with multiple tenants if they don’t need to communicate with each other.

The shared services requirements, such as WAAS or load balancing, can be solved by spinning up the service VM in the tenant’s Layer 2 broadcast domain. This design will hit scaling and management issues. There is a consensus to move from a Layer 2 design to a more robust and scalable Layer 3 design.

But why is Layer 2 still needed in data center topologies? One solution is Layer 2 VPN with EVPN. But first, let us look at Cisco DFA.

The Requirement for Layer 2 in Data Center Network Design Servers that perform the same function might need to communicate with each other due to a clustering protocol or simply as part of the application’s inner functions. If the communication is clustering protocol heartbeats or some server-to-server application packets that are not routable, then you need this communication layer to be on the same VLAN, i.e., Layer 2 domain, as these types of packets are not routable and don’t understand the IP layer. Stateful devices such as firewalls and load balancers need Layer 2 adjacency as they constantly exchange connection and session state information. Dual-homed servers: Single server with two server NICs and one NIC to each switch will require a layer 2 adjacency if the adapter has a standby interface that uses the same MAC and IP addresses after a failure. In this situation, the active and standby interfaces must be on the same VLAN and use the same default gateway. Suppose your virtualization solutions cannot handle Layer 3 VM mobility. In that case, you may need to stretch VLANs between PODS / Virtual Resource Pools or even data centers so you can move VMs around the data center at Layer 2 ( without changing their IP address ).

Data Center Design and Cisco DFA

Cisco took a giant step and recently introduced a data center fabric with Dynamic Fabric Automaton ( DFA ), similar to Juniper QFabric. This fabric offers Layer 2 switching and Layer 3 routing at the access layer / ToR. Firstly, it has a Fabric Path ( IS-IS for Layer 2 connectivity ) in the core, which gives optimal Layer 2 forwarding between all the edges.

Then they configure the same Layer 3 address on all the edges, which gives you optimal Layer 3 forwarding across the whole Fabric.

On the edge, you can have Layer 3 Leaf switches, such as the Nexus 6000 series, or integrate with Layer 2-only devices, like the Nexus 5500 series or the Nexus 1000v. You can connect external routers, USC, or FEX to the Fabric. In addition to running IS-IS as the data center control plane, DFA uses MP-iBGP, with some Spine nodes being the Route Reflector to exchange IP forwarding information.

Cisco FabricPath

DFA also employs a Cisco FabricPath technique called “Conversational Learning.” The first packet triggers a full RIB lookup, and the subsequent packets are switched in the hardware-implemented switching cache.

This technology provides Layer 2 mobility throughout the data center while providing optimal traffic flow using Layer 3 routing. Cisco commented, “DFA provides a scale-out architecture without congestion points in the network while providing optimized forwarding for all applications.”

Terminating Layer 3 at the access / ToR has clear advantages and disadvantages. Other benefits include reducing the size of the broadcast domain, which comes at the cost of reducing the mobility domain across which VMs can be moved.

Terminating Layer 3 at the accesses can also result in sub-optimal routing because there will be hair pinning or traffic tromboning of across-subnet traffic, taking multiple and unnecessary hops across the data center fabric.

The role of the Cisco Fabricpath

Cisco FabricPath is a Layer 2 technology that provides Layer 3 benefits, such as multipathing the classical Layer 2 networks using IS-IS at Layer 2. This eliminates the need for spanning tree protocol, avoiding the pitfalls of having large Layer 2 networks. As a result, Fabric Path enables a massive Layer 2 network that supports multipath ( ECMP ). THRILL is an IEEE standard that, like Fabric Path, is a Layer 2 technology that provides the same Layer 3 benefits as Cisco FabricPath to the Layer 2 networks using IS-IS.

LISP is popular in Active data centers for DCI route optimization/mobility. It separates the host’s location from the identifier ( EID ), allowing VMs to move across subnet boundaries while keeping the endpoint identification. LISP is often referred to as an Internet locator.

That can enable some triangular routing designs. Popular encapsulation formats include VXLAN ( proposed by Cisco and VMware ) and STT (created by Nicira but will be deprecated over time as VXLAN comes to dominate ).

The role of OTV

OTV is a data center interconnect ( DCI ) technology enabling Layer 2 extension across data center sites. While Fabric Path can be a DCI technology with dark fiber over short distances, OTV has been explicitly designed for DCI. In contrast, the Fabric Path data center control plane is primarily used for intra-DC communications.

Failure boundary and site independence are preserved in OTV networks because OTV uses a data center control plane protocol to sync MAC addresses between sites and prevent unknown unicast floods. In addition, recent IOS versions can allow unknown unicast floods for certain VLANs, which are unavailable if you use Fabric Path as the DCI technology.

The Role of Software-defined Networking (SDN)

Another potential trade-off between data center control plane scaling, Layer 2 VM mobility, and optimal ingress/egress traffic flow would be software-defined networking ( SDN ). At a basic level, SDN can create direct paths through the network fabric to isolate private networks effectively.

An SDN network allows you to choose the correct forwarding information on a per-flow basis. This per-flow optimization eliminates VLAN separation in the data center fabric. Instead of using VLANs to enforce traffic separation, the SDN controller has a set of policies allowing traffic to be forwarded from a particular source to a destination.

The ACI Cisco borrows concepts of SDN to the data center. It operates over a leaf and spine design and traditional routing protocols such as BGP and IS-IS. However, it brings a new way to manage the data center with new constructs such as Endpoint Groups (EPGs). In addition, no more VLANs are needed in the data center as everything is routed over a Layer 3 core, with VXLAN as the overlay protocol.

Closing Points: Data Center Design

Data centers are the backbone of modern technology infrastructure, providing the foundation for storing, processing, and transmitting vast amounts of data. A critical aspect of data center design is the network architecture, which ensures efficient and reliable data transmission within and outside the facility. 1. Scalability and Flexibility

One of the primary goals of data center network design is to accommodate the ever-increasing demand for data processing and storage. Scalability ensures the network can grow seamlessly as the data center expands. This involves designing a network that supports many devices, servers, and users without compromising performance or reliability. Additionally, flexibility is essential to adapt to changing business requirements and technological advancements.

Redundancy and High Availability

Data centers must ensure uninterrupted access to data and services, making redundancy and high availability critical for network design. Redundancy involves duplicating essential components, such as switches, routers, and links, to eliminate single points of failure. This ensures that if one component fails, there are alternative paths for data transmission, minimizing downtime and maintaining uninterrupted operations. High availability further enhances reliability by providing automatic failover mechanisms and real-time monitoring to promptly detect and address network issues.

Traffic Optimization and Load Balancing

Efficient data flow within a data center is vital to prevent network congestion and bottlenecks. Traffic optimization techniques, such as Quality of Service (QoS) and traffic prioritization, can be implemented to ensure that critical applications and services receive the necessary bandwidth and resources. Load balancing is crucial in evenly distributing network traffic across multiple servers or paths, preventing overutilizing specific resources, and optimizing performance.

Security and Data Protection

Data centers house sensitive information and mission-critical applications, making security a top priority. The network design should incorporate robust security measures, including firewalls, intrusion detection systems, and encryption protocols, to safeguard data from unauthorized access and cyber threats. Data protection mechanisms, such as backups, replication, and disaster recovery plans, should also be integrated into the network design to ensure data integrity and availability.

Monitoring and Management

Proactive monitoring and effective management are essential for maintaining optimal network performance and addressing potential issues promptly. The network design should include comprehensive monitoring tools and centralized management systems that provide real-time visibility into network traffic, performance metrics, and security events. This enables administrators to promptly identify and resolve network bottlenecks, security breaches, and performance degradation.

Data center network design is critical in ensuring efficient, reliable, and secure data transmission within and outside the facility. Scalability, redundancy, traffic optimization, security, and monitoring are essential considerations for designing a robust, high-performance network. By implementing best practices and staying abreast of emerging technologies, data centers can build networks that meet the growing demands of the digital age while maintaining the highest levels of performance, availability, and security.

Example Product: Data Center Monitoring

#### Understanding Cisco ThousandEyes

Cisco ThousandEyes is a comprehensive network intelligence platform that offers deep insights into the performance and health of your data center. By leveraging cloud-based agents and on-premises appliances, ThousandEyes provides end-to-end visibility across your entire network, from your data center to the cloud and beyond. This holistic approach allows IT teams to quickly identify and resolve issues, ensuring that your data center operates at peak efficiency.

#### Key Features of Cisco ThousandEyes

One of the standout features of Cisco ThousandEyes is its ability to deliver real-time insights into network performance. With its advanced monitoring capabilities, ThousandEyes can detect anomalies, pinpoint bottlenecks, and provide actionable data to help you optimize your data center operations. Here are some of the key features that make ThousandEyes a valuable asset:

– **End-to-End Visibility:** Monitor the entire network path, from the user to the application, ensuring no blind spots.

– **Cloud and On-Premises Integration:** Seamlessly integrate with both cloud-based and on-premises infrastructure for comprehensive coverage.

– **Real-Time Alerts:** Receive instant notifications of any performance issues, allowing for swift resolution.

– **Detailed Reporting:** Generate in-depth reports that provide insights into network performance trends and potential areas for improvement.

#### Benefits of Using Cisco ThousandEyes for Data Center Performance

Implementing Cisco ThousandEyes in your data center can deliver a range of benefits that contribute to enhanced performance and reliability. Some of the key advantages include:

– **Proactive Issue Resolution:** By identifying potential problems before they escalate, ThousandEyes helps prevent downtime and ensures continuous service delivery.

– **Improved User Experience:** With optimized network performance, users enjoy faster, more reliable access to applications and services.

– **Cost Efficiency:** By reducing downtime and improving operational efficiency, ThousandEyes can help lower overall IT costs.

– **Scalability:** As your business grows, ThousandEyes can scale with you, providing consistent performance monitoring across expanding networks.

#### Real-World Applications

Many organizations have successfully leveraged Cisco ThousandEyes to boost their data center performance. For example, a global financial services company used ThousandEyes to monitor their network and quickly identify a latency issue affecting their trading platform. By resolving the issue promptly, they were able to maintain their competitive edge and deliver a seamless experience to their clients. Similarly, an e-commerce giant utilized ThousandEyes to ensure their website remained responsive during peak shopping seasons, resulting in increased customer satisfaction and sales.

Summary: Data Center Network Design

In today’s digital age, data centers are the backbone of countless industries, powering the storage, processing, and transmitting massive amounts of information. However, the efficiency and scalability of data center network design have become paramount concerns. In this blog post, we explored the challenges traditional data center network architectures face and delved into innovative solutions that are revolutionizing the field.

The Limitations of Traditional Designs

Traditional data center network designs, such as three-tier architectures, have long been the industry standard. However, these designs come with inherent limitations that hinder performance and flexibility. The oversubscription of network links, the complexity of managing multiple layers, and the lack of agility in scaling are just a few of the challenges that plague traditional designs.

Enter the Spine-and-Leaf Architecture

The spine-and-leaf architecture has emerged as a game-changer in data center network design. This approach replaces the hierarchical three-tier model with a more scalable and efficient structure. The spine-and-leaf design comprises spine switches, acting as the core, and leaf switches, connecting directly to the servers. This non-blocking, high-bandwidth architecture eliminates oversubscription and provides improved performance and scalability.

Embracing Software-Defined Networking (SDN)

Software-defined networking (SDN) is another revolutionary concept transforming data center network design. SDN abstracts the network control plane from the underlying infrastructure, allowing centralized network management and programmability. With SDN, data center administrators can dynamically allocate resources, optimize traffic flows, and respond rapidly to changing demands.

The Rise of Network Function Virtualization (NFV)

Network Function Virtualization (NFV) complements SDN by virtualizing network services traditionally implemented using dedicated hardware appliances. By decoupling network functions, such as firewalls, load balancers, and intrusion detection systems, from specialized hardware, NFV enables greater flexibility, scalability, and cost savings in data center network design.

Conclusion:

The landscape of data center network design is undergoing a significant transformation. Traditional architectures are being replaced by more scalable and efficient models like the spine-and-leaf architecture. Moreover, concepts like SDN and NFV empower administrators with unprecedented control and flexibility. As technology evolves, data center professionals must embrace these innovations and stay at the forefront of this paradigm shift.

What is OpenFlow?

Highlights: What is OpenFlow?

How does OpenFlow work?

The Role of OpenFlow Controllers

Introducing SDN

Data and control plane

Knowledge Check: Data & Control Plane

Building blocks: SDN Environment

OpenFlow Table & Routing Tables

What is OpenFlow?

What is OpenFlow?

What is SDN?

How OpenFlow Works:

What is OpenFlow SDN?

OpenFlow reference switch

**A key point: The lack of a session layer in the TCP/IP stack**

Control and data plane

**Control plane**

**Data plane**

The advent of VXLAN

Proactive versus reactive flow setup

Hop-by-hop versus path-based forwarding

Software-defined networking or traditional routing protocols?

Software-Defined Networking: Use Cases

SDN deployment models

Native OpenFlow

Native OpenFlow with Extensions

Hybrid ( Ships in the night )

Closing Points on OpenFlow

Summary: What is OpenFlow?

Advanced in IP Routing

OSPF Enhancements

BGP Enhancements

Recap BGP route advertisement:

Source packet routing

Prefix Hijacking in BGP

Cloud Enhancements – The Intercloud

Cisco’s One Platform Kit (onePK)

Cisco Switch Virtualization Nexus 1000v

Highlights: Cisco Switch Virtualization Nexus 1000v

Hypervisor and vSphere Introduction

A brief overview of the Nexus 1000V system

**Virtual routing and forwarding**

Network virtualization

Virtual Routing and Forwarding

Cisco Switch Virtualization: The additional overhead

Cisco switch virtualization: Many to one

Network Virtualization

NIC-per-VM design

Virtual-NIC-per-VM in Cisco UCS (Adapter FEX)

Software Virtual Switch

Network virtualization and Nexus 1000v ( Nexus 1000 )

**Nexus 1000 components: VEM and VSM**

Nexus 1000 and VSM best practices

Nexus 1000v InterCloud – Cisco switch virtualization

Nexus 1000v Hybrid Cloud Components

Prerequisites

VXLAN – Virtual Extensible LAN

VXLAN Mode Packet Functions

vPath – Service chaining

License Options

Nexus 1000 features and benefits

Advantages and disadvantages of the Nexus 1000

**Closing Points on Cisco Nexus 1000v**

Summary: Cisco Switch Virtualization Nexus 1000v

Spine–Leaf Architecture: Designing Scalable Data Center Fabrics

Explore the Spine–Leaf Architecture Guide

Traditional Three-Tier vs. Spine–Leaf Architecture

The traditional three-tier model

Three-tier hierarchy

Two-tier spine–leaf

Why the design approach changed

Key architectural differences

What engineers must still consider

Compare the Network Architectures

Leaf switches

Spine–Leaf Fabric Architecture: Routing, ECMP and Traffic Flow

Understanding the role of each layer

Leaf layer

Spine layer

A key point: The lack of a session layer in the TCP/IP stack

Control plane

Data plane

Virtual routing and forwarding

Nexus 1000 components: VEM and VSM

Closing Points on Cisco Nexus 1000v