OpenContrail

August 24, 2015

by Matt Conran Blog

OpenContrail

In today's fast-paced world, where cloud computing and virtualization have become the norm, the need for efficient and flexible networking solutions has never been greater. OpenContrail, an open-source software-defined networking (SDN) solution, has emerged as a powerful tool. This blog post explores the capabilities, benefits, and significance of OpenContrail in revolutionizing network management and delivering enhanced connectivity in the cloud era.

OpenContrail, initially developed by Juniper Networks, is an open-source SDN platform offering comprehensive network capabilities for cloud environments. It provides a scalable and flexible network infrastructure that enables automation, network virtualization, and secure multi-tenancy across distributed cloud deployments.

OpenContrail, an open-source network virtualization platform, is designed to simplify the management and orchestration of virtual networks. Built on well-established technologies such as OpenStack and SDN, it provides a comprehensive set of tools and APIs to create and manage virtualized network services. With OpenContrail, organizations can achieve greater scalability, security, and performance while reducing operational complexities.

Virtual Network Overlays: OpenContrail leverages virtual network overlays to create isolated and secure network segments, allowing for seamless multi-tenancy and network segmentation.

Network Policy and Security: It offers fine-grained network policies to control traffic flow, implement access control, and enforce security measures at the virtual network level.

Analytics and Monitoring: OpenContrail provides advanced analytics and monitoring capabilities, allowing administrators to gain insights into network performance, troubleshoot issues, and optimize resource allocation.

Cloud Service Providers: OpenContrail empowers cloud service providers to deliver scalable and secure network services to their customers. It enables seamless provisioning of virtual networks, ensuring high-performance connectivity and efficient resource utilization.

Enterprise Networks: Enterprises can leverage OpenContrail to build agile and flexible network infrastructures. It simplifies network management, enables seamless integration with existing infrastructure, and provides enhanced security measures.

Internet of Things (IoT): With the proliferation of IoT devices, OpenContrail offers a robust solution for managing and securing large-scale IoT deployments. It enables efficient communication between devices, ensures data privacy, and provides centralized control over IoT network resources.

OpenContrail proves to be a groundbreaking solution in the realm of network virtualization. Its feature-rich architecture, open-source nature, and diverse real-world applications make it an invaluable tool for organizations seeking to optimize network performance, enhance security, and embrace the future of virtualized networks.

Matt Conran

Highlights: OpenContrail

Understanding OpenContrail

OpenContrail is an open-source software-defined networking (SDN) solution that enables the creation and management of virtual networks. It provides a scalable and flexible networking platform that simplifies network provisioning, enhances security, and optimizes network performance. By leveraging OpenContrail, organizations can effectively address the challenges posed by traditional networking approaches.

**Key Features and Benefits**

OpenContrail offers a wide range of powerful features that set it apart from traditional networking solutions. One of its key features is network virtualization, which allows the creation of isolated virtual networks within a physical network infrastructure.

This enables organizations to achieve greater agility and scalability, as well as efficient resource utilization. Additionally, OpenContrail provides advanced security measures, including micro-segmentation, that help protect sensitive data and prevent unauthorized access.

**Use Cases and Industry Applications**

OpenContrail is versatile and can be applied across various industries and use cases. In the telecommunications sector, it supports network slicing and virtual network functions (VNFs), crucial for deploying 5G networks. Enterprises use OpenContrail to create agile and scalable cloud environments, facilitating faster application deployment and improving overall operational efficiency.

Additionally, OpenContrail’s robust security features make it a preferred choice for sectors that require stringent data protection measures, such as finance and healthcare. By providing micro-segmentation and advanced threat detection, OpenContrail helps organizations safeguard their sensitive information.

Open-source network virtualization platform

OpenContrail is an open-source network virtualization platform that enables the creation of virtual networks overlaying physical infrastructure. It provides a scalable and flexible solution for managing network resources, improving security, and enhancing overall network performance. By decoupling the network control plane from the data plane, OpenContrail brings a new level of agility and efficiency to network operations.

1. Virtual Network Creation: OpenContrail allows the creation of virtual networks, each with its own isolated environment, policies, and routing tables. This enables organizations to achieve multi-tenancy and securely isolate their applications and workloads.

2. Network Automation and Orchestration: With OpenContrail, network provisioning and management become automated and orchestrated. This reduces manual configuration efforts and brings more consistency and reliability to network operations.

3. Enhanced Security: OpenContrail provides advanced security features such as micro-segmentation, distributed firewalling, and traffic isolation. These capabilities ensure that applications and data remain protected and isolated, even in complex and dynamic network environments.

Understanding OpenContrail components

Controller Node: At the heart of OpenContrail lies the Controller Node, which acts as the brain of the network. It is responsible for managing and orchestrating all the network services, including configuration, control, and analytics. Through its intuitive and user-friendly interface, network administrators can easily define and enforce policies, monitor network performance, and troubleshoot issues.

vRouter: The vRouter, short for virtual router, is a critical component of OpenContrail that ensures efficient packet forwarding within the network. By combining the power of virtualization and routing, the vRouter enables seamless communication between virtual machines and physical hosts. It provides advanced networking capabilities, such as firewalling, NAT, and VPN, while ensuring high performance and scalability.

Analytics Node: To gain valuable insights into network behavior and performance, OpenContrail incorporates an Analytics Node. This component collects and analyzes network data, generating comprehensive reports and metrics. Network operators can leverage this information to optimize network utilization, identify bottlenecks, and proactively address potential issues. The Analytics Node plays a crucial role in ensuring the reliability and efficiency of the entire network infrastructure.

Web User Interface: OpenContrail offers a user-friendly Web User Interface (UI) that simplifies network management and configuration. With its intuitive design and powerful functionalities, network administrators can easily define network topologies, set up policies, and monitor network performance in real time. The Web UI provides a centralized platform for managing the entire network infrastructure, making deploying, scaling, and maintaining OpenContrail deployments easier.

The traditional network vs. SDN network

In a traditional network, each switch/router must be programmed individually because applications are loaded. These applications could include a load balancer, intrusion detection, monitoring, or Voice over IP (VoIP). Based on local logic, each switch/router decides where to route packets as traffic flows through the network. Changing applications or flows in this network requires systematically programming each switch/router.

A traditional network includes both a control plane and a forwarding plane. There are also applications loaded on each device, which must be configured separately.

In an SDN network, a switch/router is not connected to any applications or intelligence. By centralized control of all devices, the network becomes programmable. A controller interfaces with applications, which are then executed across a network. Traffic flows are now supervised by a centralized controller that distributes and manages a flow table for each switch/router. Several factors can be used to define very flexible flow tables.

The flow table also collects statistics, which are fed up to the controller. This improves both visibility and control of the network because issues are immediately reported to the controller, which, in turn, can make immediate adjustments across the entire network.

The role of The VM

Virtual machines have been around for a long time, but we are beginning to spread our computing workloads in several ways. When you throw in docker containers and bare metal servers, networking becomes more interesting. Network challenges arise when all these components require communication within the same subnet, access to Internet gateways, and Layer 3 MPLS/VPNs.

As a result, data center networks are moving towards IP underlay fabrics and Layer 2 overlays. Layer 3 data plane forwarding utilizes efficient Equal-cost multi-path routing (ECMP), but we lack Layer 2 multipathing by default. Now, similar to an SD WAN overlay approach, we can connect dispersed layer 2 segments and leverage all the good features of the IP underlay. To provide Layer 2 overlays and network virtualization, Juniper has introduced an SDN platform called Junipers OpenContrail in direct competition with

For additional pre-information, you may find the following post of use.

Highlights: OpenContrail

Key Features and Benefits:

Network Virtualization:

OpenContrail leverages network virtualization techniques to provide isolated virtual networks within a shared physical infrastructure. It offers a logical abstraction layer, enabling the creation of virtual networks that operate independently, complete with their own routing, security, and quality of service policies. This approach allows for the efficient utilization of resources, simplified network management, and improved scalability.

Secure Multi-Tenancy:

OpenContrail’s security features ensure tenants’ data and applications remain isolated and protected from unauthorized access. It employs micro-segmentation to enforce strict access control policies at the virtual machine level, reducing the risk of lateral movement within the network. Additionally, OpenContrail integrates with existing security solutions, enabling the implementation of comprehensive security measures.

Intelligent Automation:

OpenContrail automates various network provisioning, configuration, and management tasks, reducing manual intervention and minimizing human errors. Its programmable API and centralized control plane simplify the deployment of complex network topologies, accelerate service delivery, and enhance overall operational efficiency.

Scalability and Flexibility:

OpenContrail’s architecture is designed to scale seamlessly, supporting distributed cloud deployments across multiple locations. It offers a highly flexible solution that can adapt to changing network requirements, allowing administrators to dynamically allocate resources, establish new connectivity, and respond to evolving business needs.

OpenContrail in Practice:

OpenContrail has gained significant traction among cloud providers, service providers, and enterprises seeking to build robust, scalable, and secure networks. Its open-source nature has facilitated its adoption, encouraging collaboration, innovation, and customization. OpenContrail’s community-driven development model ensures continuous improvement and the availability of new features and enhancements.

Highlighting Junipers OpenContrail

OpenContrail is an open-source network virtualization platform. The commercial controller and open-source product are identical; they share the same checksum on the binary image. Maintenance and support are the only difference. Juniper decided to open source to fit into the open ecosystem, which wouldn’t have worked in a closed environment.

OpenContrail offers features similar to VMware NSX, can apply service chaining and high-level security policies, and provides connections to Layer 3 VPNs for WAN integration. OpenContrail works with any hardware, but integration with Juniper’s product sets offers additional rich analytics for the underlay network.

Underlay and overlay network visibility are essential for troubleshooting. You need to look further than the first header of the packet; you need to look deeper into the tunnel to understand what is happening entirely.

Network virtualization – Isolated networks

With a cloud architecture, network virtualization gives the illusion that each tenant has a separate isolated network. Virtual networks are independent of physical network location or state, and nodes within the physical underlay can fail without disrupting the overlay tenant. A tenant may be a customer or department, depending if it’s a public or private cloud.

The virtual network sits on top of a physical network, the same way the compute virtual machines sit on top of a physical server. Virtual networks are not created with VLANs; Contrail uses a virtual overlay network system for multi-tenancy and cross-tenant communication. Many problems exist with large-scale VLAN deployments for multi-tenancy in today’s networks.

They introduce a lot of states in the physical network, and the Spanning Tree Protocol (STP) also introduces well-documented problems. There are technologies (THRILL, SPB) to overcome these challenges, but they add complexity to the design of the network.

Service Chaining

Customers require the ability to apply policy at virtual network boundaries. Policies may include ACL and stateless firewalls provided within the virtual switch. However, once you require complicated policy pieces between virtual networks, you need a more sophisticated version of policy control and orchestration called service chaining. Service chaining applies intelligent services between traffic from one tenant to another.

For example, if a customer requires content caching and stateful services, you must introduce additional service appliances and force next-hop traffic through these appliances. Once you deploy a virtual appliance, you need a scale-out architecture.

The ability to Scale-out

Scale-out is the ability to instantiate multiple physical and virtual machine instances and load balance traffic across them. Customers may also require the ability to connect with different tenants in dispersed geographic locations or to workloads in a remote private cloud or public cloud. Usually, people build a private cloud for the norm and then burst into a public cloud.

Juniper has implemented a virtual networking architecture that meets these requirements. It is based on well-known technology, MPLS/layer 3 VPN. MPLS/layer 3 VPN is the base for Juniper designs.

Virtual Network Implementation

A – MPLS Overlay

The SDN controller is responsible for the networking aspects of virtualization. When creating virtual networks, initiate the Northbound API and issue an instruction that attaches the VM to the VN. The network responsibilities are delegated from Cloudstack or OpenStack to Contrail. The Contrail SDN controller automatically creates the overlay tunnel between virtual machines. The overlay can be either an MPLS overlay style with MPLS-over-GRE, MPLS-over-UDP, or VXLAN.

L3VPN for routed traffic and EVPN for bridged traffic

Juniper’s OpenContrail is still a pure MPLS overlay of MPLS/VPN, using L3VPN for routed traffic and EVPN for bridged traffic. Traffic forwarding between end nodes has one MPLS label (VPN label), but they use various encapsulation methods to carry labeled traffic across the IP fabric. As mentioned above, this includes MPLS-over-GRE – a traditional encapsulation mechanism, MPLS-over-UDP – a variation of MPLS-over-GRE that replaces the GRE headers with UDP headers. MPLS-over-VXLAN uses VXLAN packet format but stores the MPLS label in the Virtual Network Identifier (VNI) field.

B – The forwarding plane

The forwarding plane takes the packet from the VM and gives it to the “Vrouter,” which does a lookup and determines if the destination is a remote network. If it is, it encapsulates the packet and sends it across the tunnel. The underlay that sites between the workloads forward is based on tunnel source and destination only.

No state belongs to end hosts ‘VMs, MAC addresses, or IPs. This type of architecture gives the Core a cleaner and more precise role. Generally, as a best practice, keeping “state” in the Core is a lousy design principle.

C – Northbound and southbound interfaces

To implement policy and service chaining, use the Northbound Interface and express your policy at a high level. For example, you may require HTTP or NAT and force traffic via load balancers or virtual firewalls. Contrail does this automatically and issues instructions to the Vrouter, forcing traffic to the correct virtual appliance. In addition, it can create all the suitable routes and tunnels, causing traffic through the proper sequence of virtual machines.

Contrail achieves this automatically with southbound protocols, such as XMPP (Extensible Messaging and Presence Protocol) or BGP. XMPP is a communications protocol based on XML (Extensible Markup Language).

WAN Integration

Junipers OpenContrail can connect virtual networks to external Layer 3 MPLS VPN for WAN integration. In addition, they gave the controller the ability to peer BGP to gateway routers. For the data plane, they support MPLS-over-GRE, and for the control plane, they speak MP-BGP.

Contrail communicates directly with PE routers, exchanging VPNv4 routes with MP-BGP and using MPLS-over-GRE encapsulation to pass IP traffic between hypervisor hosts and PE routers. Using standards-based protocols lets you choose any hardware appliance as the gateway node.

This data and control plane makes integration to an MPLS/VPN backbone a simple task. First, MP-BGP between the controllers and PE-routers should be established. Inter-AS Option B next hop self-approach should be used to demonstrate some demarcation points.

OpenContrail has emerged as a game-changer in software-defined networking, empowering organizations to build agile, secure, and scalable networks in the cloud era. With its advanced features, such as network virtualization, secure multi-tenancy, intelligent automation, and scalability, OpenContrail offers a comprehensive solution that addresses the complex networking challenges of modern cloud environments.

As the demand for efficient and flexible network management continues to rise, OpenContrail provides a compelling option for organizations looking to optimize their network infrastructure and unlock the full potential of the cloud.

Summary: OpenContrail

OpenContrail is a powerful open-source software-defined networking (SDN) solution revolutionizing network management and connectivity. In this blog post, we will explore its key features, benefits, and use cases and showcase how it empowers organizations to build robust and scalable networks.

Understanding OpenContrail

OpenContrail, developed by Juniper Networks, is an open-source SDN controller that provides network virtualization and automation capabilities. It is a single control point for managing and orchestrating network resources, enabling organizations to simplify network operations and enhance flexibility. By decoupling the network control plane from the underlying physical infrastructure, OpenContrail brings agility and scalability to modern networks.

Key Features of OpenContrail

OpenContrail offers a wide range of features, making it a preferred choice for network administrators. Some notable features include:

1. Virtual Network Overlay: OpenContrail creates virtual network overlays, allowing multiple virtual networks to coexist on a shared physical infrastructure. This isolation ensures enhanced security and enables efficient resource utilization.

2. Policy-Driven Automation: With policy-driven automation, network administrators can define and enforce network policies and access controls across the infrastructure. OpenContrail simplifies the management and enforcement of complex policies, reducing operational overhead.

3. Analytics and Monitoring: OpenContrail provides extensive analytics and monitoring capabilities, offering real-time insights into network traffic, performance, and security. These insights help administrators optimize network resources and troubleshoot issues effectively.

Use Cases of OpenContrail

OpenContrail finds applications in various use cases across industries. Some prominent use cases include:

1. Cloud Infrastructure: OpenContrail enables cloud service providers to build and manage scalable and secure cloud infrastructures. It facilitates seamless integration with popular cloud platforms and offers rich networking capabilities.

2. Data Centers: OpenContrail simplifies network management in data center environments. It provides dynamic workload placement, automated provisioning, and seamless connectivity between virtual machines and containers, ensuring efficient resource utilization.

3. Multi-Cloud Networking: OpenContrail supports multi-cloud networking, allowing organizations to connect and manage multiple cloud environments securely. It provides seamless connectivity, consistent policies, and centralized control across cloud providers.

Conclusion:

OpenContrail presents a game-changing solution for organizations seeking to enhance their networking capabilities. With its rich feature set, including virtual network overlays, policy-driven automation, and advanced analytics, OpenContrail empowers organizations to build scalable, secure, and agile networks. Whether it’s cloud infrastructure, data centers, or multi-cloud networking, OpenContrail is a reliable and versatile SDN solution.

Overlay Virtual Networking | Overlay Virtual Networks

February 11, 2015

by Matt Conran Blog

Overlay Virtual Networks

In today's interconnected world, networks enable seamless communication and data transfer. Overlay virtual networking has emerged as a revolutionary approach to network connectivity, offering enhanced flexibility, scalability, and security. This blog post aims to delve into the concept of overlay virtual networking, exploring its benefits, use cases, and potential implications for modern network architectures.

Overlay virtual networking is a network virtualization technique that decouples the logical network from the underlying physical infrastructure. It creates a virtual network on top of the existing physical infrastructure, enabling the coexistence of multiple logical networks on the same physical infrastructure. By abstracting the network functions and services from the physical infrastructure, overlay virtual networking provides a flexible and scalable solution for managing complex network environments.

Scalability and Flexibility: Overlay virtual networks provide the ability to scale network resources on-demand without disrupting the underlying physical infrastructure. This enables organizations to expand their network capabilities swiftly and efficiently, catering to changing business requirements.

Enhanced Security: Overlay virtual networks offer heightened security by isolating traffic and providing secure communication channels. By segmenting the network into multiple virtual domains, potential threats can be contained, preventing unauthorized access to sensitive data.

Cloud Computing: Overlay virtual networks are extensively used in cloud computing environments. They allow multiple tenants to have their own isolated virtual networks, ensuring data privacy and security. Additionally, overlay networks enable seamless migration of virtual machines between physical hosts, enhancing resource utilization.

Software-Defined Networking (SDN): Overlay virtual networks align perfectly with the principles of Software-Defined Networking. By abstracting the logical network from the physical infrastructure, SDN controllers can dynamically manage and provision network resources, optimizing performance and efficiency.

Overlay virtual networks have emerged as a powerful networking solution, providing scalability, flexibility, and enhanced security. Their applications span across various domains, including cloud computing and software-defined networking. As technology continues to evolve, overlay virtual networks are poised to play a vital role in shaping the future of networking.

Matt Conran

Highlights: Overlay Virtual Networks

Overlay Network Architecture:

Overlay virtual networks are built on the existing physical network infrastructure, creating a logical network layer that operates independently. This architecture allows organizations to leverage the benefits of virtualization without disrupting their underlying network infrastructure.

The virtual network overlay software is at the heart of an overlay virtual network. This software handles the encapsulation and decapsulation of network packets, enabling communication between virtual machines (VMs) or containers across different physical hosts or data centers. It ensures data flows seamlessly within the overlay network, regardless of the underlying network topology.

To fully comprehend overlay virtual network architecture, it is crucial to understand its key components. These include:

1. Virtual Network Overlay: The virtual network overlay is the logical representation of a virtual network that operates on top of the physical infrastructure. It encompasses virtual switches, routers, and other network elements facilitating network connectivity.

2. Tunneling Protocols: Tunneling protocols play a vital role in overlay virtual network architecture by encapsulating network packets within other packets. Commonly used tunneling protocols include VXLAN (Virtual Extensible LAN), GRE (Generic Routing Encapsulation), and Geneve.

3. Network Virtualization Software: Network virtualization software is a crucial component that enables virtual network creation, provisioning, and management. It provides a centralized control plane and offers network segmentation, traffic isolation, and policy enforcement features.

### The Mechanics Behind the Magic

Overlay virtual networks function by encapsulating packets of data with additional headers, enabling them to traverse across multiple network segments seamlessly. This encapsulation is crucial as it allows for the creation of virtualized network paths that are independent of the physical network. Technologies like Virtual Extensible LAN (VXLAN) and Generic Network Virtualization Encapsulation (GENEVE) are commonly employed to facilitate this process. These technologies not only enhance the network’s scalability but also improve its ability to adapt to complex network demands.

### Benefits Driving Adoption

The benefits of overlay virtual networks are manifold. First and foremost, they offer unparalleled scalability, allowing organizations to expand their network infrastructure without the need for significant physical overhauls. Additionally, they provide enhanced network segmentation and isolation, which are critical for maintaining data privacy and security in a multi-tenant environment. Furthermore, by decoupling the virtual network from the physical infrastructure, overlay networks enable more agile and responsive network management, which is essential for businesses operating in dynamic digital ecosystems.

Types of Overlay Networks

1. Virtual Private Networks (VPNs):

VPNs are one of the most common types of overlay networks. They enable secure communication over public networks by creating an encrypted tunnel between the sender and receiver. Individuals and organizations widely use VPNs to protect sensitive data and maintain privacy. Additionally, they allow users to bypass geographical restrictions and access region-restricted content.

2. Software-Defined Networks (SDNs):

In network architecture, SDNs utilize overlay networks to separate the control plane from the data plane. SDNs provide centralized management, flexibility, and scalability by decoupling network control and forwarding functions. Overlay networks in SDNs enable the creation of virtual networks on top of the physical infrastructure, allowing for more efficient resource allocation and dynamic network provisioning.

3. Peer-to-Peer (P2P) Networks:

P2P overlay networks are decentralized systems that facilitate direct communication and file sharing between nodes without relying on a central server. They leverage overlay networks to establish direct connections between peers and enable efficient data distribution. These networks are widely used for content sharing, real-time streaming, and decentralized applications.

4. Content Delivery Networks (CDNs):

CDNs employ overlay networks to optimize content delivery by strategically distributing content across multiple servers in different geographic regions. By bringing content closer to end-users, CDNs reduce latency and improve performance. Overlay networks in CDNs enable efficient content caching, load balancing, and fault tolerance, resulting in faster and more reliable content delivery.

5. Overlay Multicast Networks:

Overlay multicast networks are designed to distribute data to multiple recipients simultaneously efficiently. These networks use overlay protocols to construct multicast trees and deliver data over these trees. Overlay multicast networks benefit applications such as video streaming, online gaming, and live events broadcasting, where data must be transmitted to many recipients in real-time.

VXLAN as the Modern Overlay for Scalable Multi‑Site Fabrics

VXLAN extends Layer‑2 networks across Layer‑3 underlays using lightweight UDP encapsulation, enabling massive scalability far beyond traditional VLAN limits. Instead of relying on physical broadcast domains, VXLAN uses a 24‑bit VNI to create millions of logical segments that can span racks, pods, data centers, and even private WANs. In this lab, VXLAN becomes a structured overlay object inside your JavaScript simulator: VTEPs, VNIs, tunnels, multicast groups, and EVPN route‑types all represented as dynamic JSON entities. This transforms VXLAN from a conceptual overlay into a measurable, testable system that reacts to topology changes, routing convergence, and control‑plane events in real time.

Your VXLAN lab uses a multi‑engine JavaScript pipeline to evaluate overlay stability. The VTEP engine checks tunnel health, encapsulation overhead, and latency. The VNI engine evaluates segment isolation, MAC learning, and flood‑and‑learn pressure. The EVPN engine analyzes BGP EVPN route‑types (RT‑2, RT‑3, RT‑5), MAC mobility, and control‑plane convergence. The underlay engine measures ECMP path diversity, routing stability, and packet loss during reconvergence. Each engine produces severity, timing, and root‑cause signals that feed into your cinematic CLI, creating a step‑by‑step visualization of VXLAN behavior across both overlay and underlay layers. To make VXLAN intuitive and screenshot‑ready, the lab uses the same ASCII‑based visuals you’ve standardized across your platform. The topology page shows VTEPs, tunnels, and underlay leaf‑spine paths using ASCII diagrams. The engines page displays shaded tables for VTEP health, VNI isolation, EVPN propagation, and underlay ECMP stability. The failures page animates tunnel drops, MAC mobility events, EVPN withdraws, and underlay reconvergence using timestamped ASCII timelines. The traffic page shows VXLAN encapsulated flows shifting across ECMP paths, with dynamic load bars and per‑VNI distribution. Together, these visuals turn VXLAN into a living overlay fabric that engineers can explore and analyze.

This image below elegantly visualizes the VXLAN v2 lab architecture and packet flow. On the left, the ASCII panels show the leaf‑spine underlay topology with four VTEPs and their performance grid — latency, loss, and health scores — giving a quick operational snapshot of the overlay fabric. Below that, the failure simulation outlines three critical scenarios: tunnel latency spikes, MAC mobility storms, and VNI isolation leaks, each describing how congestion or misconfiguration can disrupt VXLAN stability. On the right, the flow diagram breaks down the VXLAN packet structure, tracing encapsulation from Host A through VTEP 1, across the IP underlay network, and decapsulation at VTEP 2 before reaching Host B. It highlights the layered headers — Ethernet, VXLAN (24‑bit VNI), UDP 4789, IP, and outer Ethernet — and ties them to the EVPN/BGP control plane. Together, these visuals form a cohesive narrative of how VXLAN operates, encapsulates traffic, and maintains segmentation across a modern leaf‑spine data‑center fabric.

Google Cloud CDN

**What is Google Cloud CDN?**

Google Cloud CDN is a globally distributed network of servers designed to cache and deliver content closer to the end-users, reducing latency and improving load times. By leveraging Google’s robust infrastructure, Cloud CDN allows businesses to serve their content quickly and reliably to users all over the world. This service is particularly beneficial for websites, APIs, and applications that experience high traffic or need to operate at peak performance levels.

**Key Features of Google Cloud CDN**

One of the standout features of Google Cloud CDN is its integration with Google Cloud’s services. This seamless connection means you can easily manage your CDN alongside other Google Cloud products, streamlining operations. Additionally, Google Cloud CDN offers features like SSL/TLS support, advanced caching capabilities, and custom domain support, ensuring secure and flexible content delivery.

Another crucial feature is the global reach of Google Cloud CDN. With points of presence (PoPs) strategically located around the world, content is delivered from the nearest server to the user, ensuring quick and reliable access. The network is designed to handle high volumes of traffic, making it ideal for businesses of any size.

**Benefits of Using Google Cloud CDN**

The benefits of adopting Google Cloud CDN are manifold. Firstly, it significantly reduces latency, ensuring that users have a smooth and fast experience. This is particularly vital for businesses that rely on web traffic to drive sales or engagement. Faster load times can lead to increased user satisfaction and higher conversion rates.

Secondly, Google Cloud CDN enhances security. By offloading traffic to edge servers, it acts as a protective layer against DDoS attacks, mitigating threats before they reach your core infrastructure. This added security is invaluable for businesses that handle sensitive data or rely on uptime for operational success.

Lastly, Google Cloud CDN offers cost efficiency. By using caching and strategically distributing content, it reduces the need for expensive bandwidth and server resources. This optimization can lead to significant cost savings, making it an attractive option for businesses looking to maximize their cloud investment.

**How to Get Started with Google Cloud CDN**

Getting started with Google Cloud CDN is straightforward. First, you’ll need to set up a Google Cloud account if you don’t already have one. Once your account is ready, you can enable Cloud CDN through the Google Cloud Console. From there, you can configure your caching rules, secure your content with SSL/TLS, and monitor performance with Google Cloud’s comprehensive analytics tools.

To ensure optimal performance, it’s advisable to familiarize yourself with best practices for content caching and delivery. Google Cloud provides extensive documentation and support to help you make the most out of their CDN service.

Use Cases of Overlay Virtual Networking:

1. Multi-Tenancy: Overlay virtual networking provides an ideal solution for organizations to segregate their network resources securely. By creating virtual overlays, multiple tenants can coexist on a single physical network infrastructure without interference. This enables service providers and enterprises to offer distinct network environments to customers or departments while ensuring isolation and security.

2. Data Center Interconnect: Overlay virtual networking enables efficient and scalable data center interconnect (DCI). With traditional networking, interconnecting multiple data centers across geographies can be complex and costly. However, overlay virtual networking simplifies this process by abstracting the underlying physical infrastructure and providing a unified logical network. It allows organizations to seamlessly extend their networks across multiple data centers, enhancing workload mobility and disaster recovery capabilities.

3. Cloud Computing: Cloud computing heavily relies on overlay virtual networking to deliver agility and scalability. Cloud providers can dynamically provision and manage network resources by leveraging overlay networks, ensuring optimal customer performance and flexibility. Overlay virtual networking enables the creation of virtual networks that are isolated from each other, allowing for secure and efficient multi-tenant cloud environments.

4. Microservices and Containerization: The rise of microservices architecture and containerization has presented new networking challenges. Overlay virtual networking provides a solution by enabling seamless communication between microservices and containers, regardless of their physical location. It ensures that applications and services can communicate with each other, even across different hosts or clusters, without complex network configurations.

5. Network Segmentation and Security: Overlay virtual networking enables granular network segmentation, allowing organizations to implement fine-grained security policies. By creating overlay networks, administrators can isolate different workloads, departments, or applications, ensuring each segment has dedicated network resources and security policies. This enhances security by limiting the lateral movement of threats and reducing the attack surface.

Tunneling Protocols

Tunneling protocols play a crucial role in overlay virtual networks by facilitating the encapsulation and transportation of network packets over the underlying physical network. Popular tunneling protocols such as VXLAN (Virtual Extensible LAN), MPLS (Multiprotocol Label Switching ), NVGRE (Network Virtualization using Generic Routing Encapsulation), and Geneve provide the necessary mechanisms for creating virtual tunnels and encapsulating traffic.

The Network Virtualization Edge (NVE) acts as the endpoint for the overlay virtual network. It connects the physical network infrastructure to the virtual network, ensuring seamless communication between the two. NVEs perform functions like encapsulation, decapsulation, and mapping virtual network identifiers (VNIs) to the appropriate virtual machines or containers.

Example: Point-to-Point GRE

GRE, or Generic Routing Encapsulation, is a tunneling protocol widely used in overlay networks. It encapsulates various network layer protocols within IP packets, enabling virtual point-to-point connections over an existing IP network. GRE provides a mechanism to extend private IP addressing schemes over public networks, facilitating secure and efficient communication between remote locations.

Example: GRE and IPSec

GRE and IPSEC often work together to create secure tunnels across public networks. GRE provides the means for encapsulating and carrying different protocols, while IPSEC ensures the confidentiality and integrity of the encapsulated packets. By combining the strengths of both protocols, organizations can establish secure connections that protect sensitive data and enable secure communication between remote networks.

The combination of GRE and IPSEC offers several benefits and finds applications in various scenarios. Some of the key advantages include enhanced security, scalability, and flexibility. Organizations can utilize this technology to establish secure site-to-site VPNs, remote access VPNs, and even to facilitate secure multicast communication. Whether connecting branch offices, enabling remote employee access, or safeguarding critical data transfers, GRE and IPSEC are indispensable tools.

Example: MPLS Overlay Tunneling

MPLS overlay tunneling is a technique that enables the creation of virtual private networks (VPNs) over existing network infrastructures. It involves encapsulating data packets within additional headers to establish tunnels between network nodes. MPLS, or Multiprotocol Label Switching, is a versatile technique that facilitates the forwarding of network packets. It operates at the OSI (Open Systems Interconnection) model’s layer 2.5, combining the benefits of both circuit-switching and packet-switching technologies. By assigning labels to data packets, MPLS enables efficient routing and forwarding, enhancing network performance.

Overlay Network Control Plane

The control plane in an overlay virtual network manages and maintains the overall network connectivity. It handles tasks such as route distribution, network mapping, and keeping the overlay network’s forwarding tables. Border Gateway Protocol (BGP) and Virtual Extensible LAN Segment Identifier (VXLAN VNI) provide the necessary control plane mechanisms. The network can adapt to changing conditions and optimize performance through centralized or distributed control plane architectures.

Components of the Overlay Network Control Plane

a) Controller: The controller serves as the core component of the control plane, acting as a centralized entity that orchestrates network operations. It receives information from network devices, processes it, and then disseminates instructions to ensure proper network functioning.

b) Routing Protocols: Overlay networks employ various routing protocols to determine the optimal paths for data transmission. Protocols like BGP, OSPF, and IS-IS are commonly used to establish and maintain routes within the overlay network.

c) Virtual Network Mapping: This component maps virtual network topologies onto the physical infrastructure. It ensures that virtual network elements are appropriately placed and interconnected, optimizing resource utilization while maintaining network performance.

Underlay and Clos Fabric

The underlay of most modern data centers is a 3-stage or 5-stage Clos fabric, with the physical infrastructure and point-to-point Layer 3 interfaces between the spines and leaves. Network virtualization can be created by elevating the endpoints and applications connected to the network into this overlay, thus logically carving out different services on top of it.

Traditional data center network architectures, such as the three-tier architecture, were widely used. These architectures featured core, distribution, and access layers, each serving a specific purpose. However, as data traffic increased and workloads became more demanding, these architectures started to show limitations in terms of scalability and performance.

Introducing Leaf and Spine Architecture

Leaf and spine architecture emerged as a solution to overcome the shortcomings of traditional network architectures. This modern approach reimagines network connectivity by establishing a fabric of interconnected switches. The leaf switches act as access switches, while the spine switches provide high-speed interconnectivity between the leaf switches. This design increases scalability, reduces latency, and improves bandwidth utilization.

VXLAN and Leaf and Spine

In RFC 7348, a Virtual Extensible LAN (VXLAN) is a data plane encapsulation type capable of supporting Layer 2 and Layer 3 payloads. In addition to logically separating broadcast or bridging domains in a network, virtual LANs (VLANs) are limited in their scalability to 4K VLANs. By contrast, VXLAN provides a 24-bit VXLAN Network Identifier (VNI) in the VXLAN header, allowing the network administrator more flexibility to partition the network logically.

VXLAN is, in essence, a stateless tunnel originating at one endpoint and terminating at another because of its encapsulating trait. The VXLAN Tunnel Endpoints (VTEPs) are the endpoints that encapsulate and decapsulate the VXLAN tunnel. The first thing you need to understand about VXLAN is that these tunnels can originate and terminate on network devices or servers with the help of a virtual switch such as Open vSwitch, with a VXLAN module that is usually accelerated by hardware so that the CPU doesn’t have to process these packets in software.

Example: VXLAN Flood and Learn

Understanding VXLAN Flood and Learn

VXLAN Flood and Learn is a mechanism used in VXLAN networks to facilitate the dynamic learning of MAC addresses in a scalable manner. Traditionally, MAC learning relied on control-plane protocols, which could become a bottleneck in larger deployments. With VXLAN Flood and Learn, the burden of MAC address learning is offloaded to the data plane, allowing for greater scalability and efficiency.

Multicast plays a pivotal role in VXLAN Flood and Learn. It serves to transmit broadcast, unknown unicast, and multicast (BUM) traffic within the VXLAN overlay. By utilizing multicast, BUM traffic can be efficiently delivered to interested recipients across the VXLAN fabric, eliminating the need for flooding at Layer 2.

Adopting VXLAN Flood and Learn with Multicast brings several advantages to network operators. Firstly, it reduces the reliance on control-plane protocols, simplifying the network architecture and improving scalability. Additionally, it minimizes unnecessary traffic across the VXLAN fabric, resulting in enhanced efficiency. However, it’s essential to consider the scalability of the multicast infrastructure and the impact of multicast traffic on the underlying network.

VXLAN vs. Spanning Tree Protocol

Now, let’s compare VXLAN and STP across various aspects:

Scalability: VXLAN provides unparalleled scalability by enabling the creation of up to 16 million logical networks, addressing the limitations of traditional VLANs. In contrast, STP suffers from scalability issues due to its limited VLAN range and the potential for network loops.

Efficiency: VXLAN optimizes network utilization by allowing traffic to be load-balanced across multiple paths, resulting in improved performance. STP, on the other hand, blocks redundant paths, leading to underutilization of available network resources.

Convergence Time: VXLAN exhibits faster convergence time compared to STP. With VXLAN, network reconfigurations can be achieved dynamically without service interruption, while STP requires considerable time for convergence, causing potential service disruptions.

Multicast Overlay

VXLAN encapsulates Ethernet frames within UDP packets, allowing virtual machines (VMs) to communicate across different physical networks or data centers seamlessly. When combined, multicast and VXLAN offer a robust solution for scaling network virtualization environments. Multicast efficiently distributes traffic across VXLAN tunnels, ensuring optimal delivery to multiple hosts. By leveraging multicast, VXLAN eliminates the need for unnecessary packet replication, reducing network congestion and enhancing overall performance.

Advantages of Overlay Virtual Networks

Enhanced Security and Isolation:

One key advantage of overlay virtual networks is their ability to provide enhanced security and isolation. Encapsulating traffic within virtual tunnels allows overlay networks to establish secure communication channels between network segments. This isolation prevents unauthorized access and minimizes the potential for network breaches.

While VLANs offer flexibility and ease of network management, one of their significant disadvantages lies in their limited scalability. As networks expand and the number of VLANs increases, managing and maintaining the VLAN configurations becomes increasingly complex. Network administrators must carefully plan and allocate resources to prevent scalability issues and potential performance bottlenecks.

Simplified Network Management

Overlay virtual networks simplify network management. By decoupling the virtual network from the physical infrastructure, network administrators can easily configure and manage network policies and routing without affecting the underlying physical network. This abstraction layer streamlines network management tasks, resulting in increased operational efficiency.

Scalability and Flexibility

Scalability is a critical requirement in modern networks, and overlay virtual networks excel in this aspect. By leveraging the virtualization capabilities, overlay networks can dynamically allocate network resources based on demand. This flexibility enables seamless scaling of network services, accommodating evolving business needs and ensuring optimal network performance.

Performance Optimization

Overlay virtual networks also offer performance optimization features. By implementing intelligent traffic engineering techniques, overlay networks can intelligently route traffic and optimize network paths. This ensures efficient utilization of network resources and minimizes latency, resulting in improved application performance.

Advanced Topics

1. GETVPN:

Group Encrypted Transport VPN (GET VPN) is a set of Cisco IOS features that secure IP multicast group and unicast traffic over a private WAN. GET VPN secures IP multicast or unicast traffic by combining the keying protocol Group Domain of Interpretation (GDOI) with IP security (IPsec) encryption. With GET VPN, multicast and unicast traffic are protected without the need for tunnels, as nontunneled (that is, “native”) IP packets can be encrypted.

Key Components of Getvpn

GDOI Key Server: The GDOI Key Server is the central authority for key management in Getvpn. It distributes encryption keys to all participating network devices, ensuring secure communication. By centrally managing the keys, the GDOI Key Server simplifies adding or removing devices from the network.

Group Member: The Group Member is any device that is part of the Getvpn network. It can be a router, switch, or firewall. Group Members securely receive encryption keys from the GDOI Key Server and encrypt/decrypt traffic using these keys. This component ensures that data transmitted within the Getvpn network remains confidential and protected.

Group Domain of Interpretation (GDOI): The GDOI protocol is the backbone of Getvpn. It enables secure exchange and management between the GDOI Key Server and Group Members. Using IPsec for encryption and the Internet Key Exchange (IKE) protocol for key establishment, GDOI ensures the integrity and confidentiality of data transmitted over the Getvpn network.

2. DMVPN

DMVPN is a scalable and flexible networking solution that allows the creation of secure virtual private networks over a public infrastructure. Unlike traditional VPNs, DMVPN dynamically builds tunnels between network endpoints, providing a more efficient and cost-effective approach to network connectivity.

The underlay network forms the foundation of DMVPN. It represents the physical infrastructure that carries IP traffic between different sites. This network can be based on various technologies, such as MPLS, the Internet, or even a mix of both. It provides the necessary connectivity and routing capabilities to establish communication paths between the DMVPN sites.

While the underlay takes care of the physical connectivity, the overlay network is where DMVPN truly shines. This layer is built on top of the underlay and is responsible for creating a secure and efficient virtual network. Through the magic of tunneling protocols like GRE (Generic Routing Encapsulation) or IPsec (Internet Protocol Security), DMVPN overlays virtual tunnels over the underlay network, enabling seamless communication between sites.

1. Multipoint GRE Tunnels: One key component of DMVPN is the multipoint GRE (mGRE) tunnels. These tunnels allow multiple sites to communicate with each other over a shared IP network. Using a single tunnel interface makes scaling the network easier and reduces administrative overhead.

2. Next-Hop Resolution Protocol (NHRP): NHRP is another essential component of DMVPN. It allows mapping the tunnel IP address to the remote site’s physical IP address. This dynamic mapping allows efficient routing and eliminates the need for static or complex routing protocols.

3. IPsec Encryption: To ensure secure communication over the public network, DMVPN utilizes IPsec encryption. IPsec encrypts the data packets traveling between sites, making it nearly impossible for unauthorized entities to intercept or tamper with the data. This encryption provides confidentiality and integrity to the network traffic.

The Single Hub Dual Cloud Architecture

The single-hub dual cloud architecture takes the benefits of DMVPN to the next level. With this configuration, a central hub site is a connection point for multiple cloud service providers (CSPs). This architecture enables businesses to leverage the strengths of different CSPs simultaneously, ensuring high availability and redundancy.

One key advantage of the single hub dual cloud architecture is improved reliability. By distributing traffic across multiple CSPs, businesses can mitigate the risk of service disruption and minimize downtime. Additionally, this architecture provides enhanced performance by leveraging the geographic proximity of different CSPs to various remote sites.

Implementing the single-hub dual cloud architecture requires careful planning and consideration. Factors such as CSP selection, network design, and security measures must all be considered. It is crucial to assess your organization’s specific requirements and work closely with network engineers and CSP providers to ensure a smooth and successful deployment.

DMVPN vs GETVPN

DMVPN and GETVPN are two VPN technologies commonly used in Enterprise WAN setups, especially when connecting many remote sites to one hub. Both GETVPN and DMVPN technologies allow hub-to-spoke and spoke-to-spoke communication. Whenever any of these VPN solutions are deployed, especially on Cisco Routers, a security license is an additional overhead (cost).

Tunnel-less VPN technology, GETVPN, provides end-to-end security for network traffic across fully mesh topologies. DMVPN enables full mesh connectivity with a simple hub-and-spoke configuration. In DMVPN, IPsec tunnels are formed over dynamically/statically addressed spokes.

3. MPLS VPN

At its core, MPLS VPN is a technique that utilizes MPLS labels to route data securely over a shared network infrastructure. It enables the creation of virtual private networks, allowing businesses to establish private and isolated communication channels between their various sites. By leveraging MPLS technology, MPLS VPN ensures optimal performance, service quality, and enhanced data transmission security.

MPLS VPN Components

1. Provider Edge (PE) Routers: PE routers are located at the edge of the service provider’s network. They act as the entry and exit points for the customer’s data traffic. PE routers are responsible for applying labels to incoming packets and forwarding them based on the predetermined VPN routes.

2. Customer Edge (CE) Routers: CE routers are located at the customer’s premises and connect the customer’s local network to the service provider’s MPLS VPN network. They establish a secure connection with the PE routers and exchange routing information to ensure proper data forwarding.

3. Provider (P) Routers: P routers are the backbone of the service provider’s network. They form the core network and forward labeled packets between the PE routers. P routers do not participate in VPN-specific functions and only focus on efficient packet forwarding.

**Label Distribution Protocol (LDP)**

LDP is a key component of MPLS VPNs. It distributes labels across the network, ensuring each router has the necessary information to label and forward packets correctly. LDP establishes label-switched paths (LSPs) between PE routers, providing the foundation for efficient data transmission.

**Virtual Routing and Forwarding (VRF)**

VRF is a technology that enables the creation of multiple virtual routing tables within a single physical router. Each VRF instance represents a separate VPN, allowing for isolation and secure communication between different customer networks. VRF ensures that data from one VPN does not mix with another, providing enhanced privacy and security.

Use Case: Understanding Performance-Based Routing

Performance-based routing is a dynamic approach to network routing that considers real-time metrics such as latency, packet loss, and available bandwidth to determine the most optimal path for data transmission. Unlike traditional static routing protocols that rely on predetermined routes, performance-based routing adapts to the ever-changing network conditions, ensuring faster and more reliable data delivery.

Enhanced Network Performance: By leveraging performance-based routing algorithms, businesses can significantly improve network performance. This approach’s dynamic nature allows for intelligent decision-making, routing data through the most efficient paths, and avoiding congested or unreliable connections. This results in reduced latency, improved throughput, and enhanced us.

Cost Savings: Performance-based routing not only improves network performance but also leads to cost savings. Businesses can minimize bandwidth consumption by optimizing data transmission paths, effectively reducing operational expenses. Additionally, organizations can make more efficient use of their network infrastructure by avoiding underperforming or expensive routes.

Related: Before you proceed, you may find the following useful:

Overlay Virtual Networks

Concept of network virtualization

It’s worth mentioning that network virtualization is nothing new. The most common forms of network virtualization are virtual LANs (VLANs), virtual private networks (VPNs), and Multiprotocol Label Switching (MPLS). VLAN has been the first to extract the location of Layer 2 connectivity across multiple Layer 2 switches. VPN enables overlay networks across untrusted networks such as the WAN, while MPLS segments traffic based on labels.

These technologies enable the administrators to physically separate endpoints into logical groups, making them behave like they are all on the same local (physical) segment. The ability to do this allows for much greater efficiency in traffic control, security, and network management.

- Enhanced Connectivity:

One of the primary advantages of network overlay is its ability to enhance connectivity. By creating a virtual network layer, overlay networks enable seamless communication between devices and applications, irrespective of their physical location.

This means organizations can effortlessly connect geographically dispersed branches, data centers, and cloud environments, fostering collaboration and resource sharing. Moreover, network overlays offer greater flexibility by allowing organizations to dynamically adjust and optimize their network configurations to meet evolving business needs.

- Improved Scalability:

Traditional network infrastructures often struggle to keep up with the increasing demands of modern applications and services. Network overlay addresses this challenge by providing a scalable solution. By decoupling the virtual network from the physical infrastructure, overlay networks allow for more efficient resource utilization and easier scaling.

Organizations can easily add or remove network elements without disrupting the entire network. As a result, network overlays enable organizations to scale their networks rapidly and cost-effectively, ensuring optimal performance even during peak usage periods.

Tailored load balancing

Some customers may not require cloud load balancing services provided by the cloud services if they have optimized web delivery by deploying something like Squid or NGINX. Squid is a caching proxy that improves web request response times by caching frequently requested web pages. NGINX ( open source reverse proxy ) is used to load balance Hypertext Transfer Protocol ( HTTP ) among multiple servers.

Example: Traffic flow and the need for a virtual overlay

Traffic would flow to Web servers and trigger application and database requests. Each tier requires different segments, and in large environments, the limitations of using VLANs to create these segments will bring both scalability and performance problems.

This is why we need virtual overlay solutions. These subnets require Layer 3 and sometimes Layer 2 ( MAC ). Layer 2 connectivity might be for high availability services that rely on gratuitous Address Resolution Protocol ( ARP ) between devices or some other non-routable packet that can not communicate over IP. If the packet is not Layer 3 routable, it needs to communicate via Layer 2 VLANs.

Diagram: Virtual overlay networking and complex application tiers.

Scalability and Security Concerns

The weakest link in a security paradigm is the lowest application in that segment. Make each application an independent tenant so all other applications are unaffected if a security breach or misuse occurs in one application stack.

Designers should always attempt to design application stacks to minimize beachheading, i.e., an attacker compromising one box and using it to jump to another quickly. Public and private clouds should support multi-tenancy with each application stack.

However, scalability issues arise when you deploy each application as an individual segment. For example, customer X’s cloud application requires four segments; 4000 VLANs soon become 1000 applications. Media Access Control ( MAC ) visibility has an entire reach throughout Layer 2 domains.

Some switches support a low count number of MAC addresses. When a switch reaches its MAC limit, it starts flooding packets, increasing network load and consuming available bandwidth that should be used for production services.

…current broadcast domains can support … around 1,000 end hosts in a single bridged LAN of 100 bridges” (RFC 5556 – TRILL)

NIC in promiscuous mode and failure domains

Server administrators configure server NICs in promiscuous mode to save configuration time. NICs in promiscuous mode look at all frames passing even when the frame is not destined for them. Network cards acting in promiscuous mode are essentially the same as having one VLAN spanning the entire domain. Sniffer products set promiscuous modes to capture all data on a link and usually only act in this mode for troubleshooting purposes.

A well-known issue with Layer 2 networks is that they present a single failure domain with extreme scalability and operational challenges. This is related to Layer 2 Spanning Tree Protocol ( STP ); THRILL is also susceptible to broadcast storms and network meltdowns.

The rise of overlay virtual networks

Previously discussed scalability and operational concerns force vendors to develop new data center technologies. One of the most prevalent new technologies is overlay virtual networks, tunneling over IP. An overlay is a tunnel between two endpoints, allowing frames to be transported. The beauty of overlay architectures is that they enable switch table sizes not to increase as the number of hosts attached increases.

Vendors’ Answer: Virtual Overlay Solutions

Virtual Overlay Solution: Keep complexity to the edges.

Ideally, we should run virtual networks over IP like SKYPE runs Voice over IP. The recommended design retains complexity at the network’s edge; the IP transport network provides IP transport. A transport network does not need to be a Layer 2 network and can have as many IP subnets and router hops.

All data ( storage, vMotion, user traffic ) traffic becomes an IP application. The concept resembles how Border Gateway Protocol ( BGP ) applies to TCP. End hosts carry out encapsulation and use the network for transport. Again, complexity is at the edge, similar to the Internet. Keeping complexity to the edge makes Layer 3 fabrics efficient and scalable.

VXLAN, STT, and ( NV ) GRE

Numerous encapsulation methods can tunnel over the IP core. This is known as virtual overlay networking and includes VXLAN, STT, and ( NV ) GRE. The main difference between these technologies is the encapsulation method and minor technological differences with TCP offload and load balancing.

virtual overlay solutions — Diagram: Virtual overlay solution.

The Recommended Design: Leaf and Spine.

Like the ACI network, virtual overlay networks work best with Leaf and Spine fabric architectures. Leaf and Spine designs guarantee any two endpoints get equal bandwidth. VMs on the same Top-of-Rack ( ToR ) switch will have access to more bandwidth than if the VM had to communicate across the Spine layer.

Overlay networks assume that the underlying network has a central endpoint. The transport network should avoid oversubscription as much as possible. If security concerns you, you can always place similar VM appliances on dedicated clusters, one type per physical server.

( NV ) GRE, VXLAN, and STT do not have an built-in security features meaning the transport network MUST be secure.

TCP offload, load balancing & scale-out NAT

TCP can push huge segments down the physical NIC and slice the packet into individual TCP segments, improving TCP performance. For example, you can push 10Gbps from a VM with TCP offload. The problem is that NICs only support VLANs and not VXLANs.

NICIRA added another header in front of TCP segments. TCP is embedded in another TCP. Now, you can use the existing NIC to slice the current TCP segment into smaller TCP segments. It is dramatically improving performance.

STT and VXAN

STT and VXAN can use 5-tuple load balancing as they use port numbers. Therefore, traffic sent between a pair of VMs can use more than one link in the network. Unfortunately, not many switches can load balance based on the GRE payload used by NVGRE.

Scale-out NAT is difficult to implement as an asymmetric path is not guaranteed. Furthermore, the shared state is tied to an outside IP address, which limits scale-out options. To scale out effectively, the state has to be spread across all members of the NAT cluster. The new approach uses floating public IP addresses and one-to-one mapping between floating IP and the private IP address inside—there is no state due to the one-to-one mapping.

Distributed layer 2 & layer 3 forwarding

They distributed Layer 2 forwarding ( data plane ): Most Overlays offer distributed Layer 2 forwarding. VM can be sent to VM in the same segment. The big question is how they distribute MAC to VTEP – some use multicast and traditional Ethernet flooding, while others use control planes. The big question is how scalable is the control plane.

Distributed Layer 3 forwarding ( data plane ): On the other hand, if you have multiple IP subnets between segments ( not layer 2 ), you need to forward between them. The inter-subnet must not be a choke point. If your data center has lots of intra-traffic ( East to West traffic), avoid centralized inter-subnet forwarding, which will quickly become a traffic choke point.

The router will process ARP if you are doing Layer 3 forwarding. But if you are doing a mix of Layer 2 and 3, make sure you can reduce the flooding by intercepting ARP requests and caching ARP replies, known as distributed ARP Caching.

Scale-out control plane

Initial overlays used multicast and Ethernet-like learning. Now, some vendors are using controller-based overlays. Keep in mind that the controller can now become a scalability bottleneck. However, many vendors, such as Cisco ACI, can scale the controllers and have a quorum.

Efficient controller scalability is seen when controllers do not participate in the data plane ( do not reply to ARP ). This type of controller scales better than controllers that intercept data plane packets and perform data plane activity. So, the data plane will not be affected if a controller is offline. In the early days of Sofware-Defined Networking, this was not the case. If the controller was down, the network was down.

Scale-out controllers

Attempt to design scale-out controllers by building a cluster of controllers and having some protocol running between them. You now have clear failure domains. For example, controller A looks after VM segment A and Controller B, and control looks after VM segment B. For cloud deployments in multiple locations, deploy multiple controller clusters in each location.

Availability zones

Design availability zones with hierarchical failure domains by splitting infrastructures into regions. Problems arising in one region do not affect all other regions. You have one or more availability zones within an area for physical and logical isolation.

Availability zones limit the impact of a failure in a failure domain. An example of a failure domain could be a VLAN experiencing a broadcast storm. Attempt to determine the span of VLANs across availability zones—define VLANs to one-ToR switch. Never stretch VLANs as you create a single failure domain by merging two zones.

Do not stretch a VLAN across multiple availability zones. This is why we have network overlays in the first place, so we don’t need to stretch VLAN across the data center. For example, VXLAN uses the VNI to differentiate between Layer 2 and Layer 3 traffic over a routed underlay. We can use VXLAN as the overlay network to span large Layer 2 domains over a routed core.

Network Overlay Controllers

As a final note on controllers, controller-based SDN networks participate in data planes and perform activities such as MAC learning and ARP replies. As mentioned, this is not common nowadays but was at the start of the SDN days. If the controller performs activities such as MAC learning and APR replies and the controller fails, then you have network failure.

The more involved the controller is in the forwarding decisions, the worse the outage can be. All overlay networking vendors nowadays have controllers that set up the control plane so the data plane can forward traffic without getting involved in data plane activity. This design also allows the controller to be scaled without affecting the data plane activity.

Closing Point: Overlay Virtual Networks

Overlay virtual networking is a method where a virtual network is built on top of an existing physical network. This abstraction allows for greater flexibility in managing and deploying network resources. By decoupling the physical infrastructure from the network’s logical topology, businesses can create complex network architectures without the need to reconfigure physical hardware. This is particularly advantageous in environments like data centers, where dynamic and scalable network configurations are essential.

The primary advantage of overlay virtual networking lies in its ability to simplify and streamline network management. With overlays, network administrators can easily create, modify, and manage network segments without altering the underlying physical infrastructure. Additionally, overlay networks provide enhanced security through network segmentation, allowing for isolated virtual networks within the same physical environment. This capability reduces the risk of unauthorized access and data breaches.

Overlay virtual networking is increasingly being adopted in various sectors, including cloud computing, data centers, and enterprise networks. For cloud service providers, overlays enable the rapid deployment of virtual networks for different customers, ensuring that resources are efficiently utilized while maintaining customer privacy. In data centers, overlay networks facilitate the creation of agile and scalable environments to accommodate fluctuating workloads and demands. Enterprises, on the other hand, leverage overlays to integrate multiple branch offices seamlessly into a cohesive network, enhancing communication and collaboration.

While overlay virtual networking offers numerous benefits, it’s not without its challenges. Network administrators must consider factors such as increased complexity in troubleshooting, potential performance issues due to additional encapsulation, and the need for advanced skills to manage and implement overlay technologies effectively. It’s crucial for organizations to weigh these considerations against the benefits to determine the best approach for their specific needs.

Summary: Overlay Virtual Networks

Overlay networking has revolutionized the way we design and manage modern networks. In this blog post, we will delve into the fascinating world of overlay networking, exploring its benefits, applications, and critical components.

Understanding Overlay Networking

Overlay networking is a technique for creating virtual networks on top of an existing physical network infrastructure. By decoupling the network services from the underlying hardware, overlay networks provide flexibility, scalability, and enhanced security.

Benefits of Overlay Networking

One of the primary advantages of overlay networking is its ability to abstract the underlying physical infrastructure, allowing for seamless integration of different network technologies and protocols. This flexibility empowers organizations to adapt to changing network requirements without significant disruptions. Additionally, overlay networks facilitate the implementation of advanced network services, such as virtual private networks (VPNs) and load balancing, while maintaining a simplified management approach.

Applications of Overlay Networking

Overlay networking finds applications in various domains, ranging from data centers to cloud computing. In data center environments, overlay networks enable efficient multi-tenancy, allowing different applications or departments to operate within isolated virtual networks. Moreover, overlay networking facilitates the creation of hybrid cloud architectures, enabling seamless connectivity between on-premises infrastructure and public cloud resources.

Key Components of Overlay Networking

Understanding overlay networking’s key components is crucial to comprehending it. These include overlay protocols, which establish and manage virtual network connections, and software-defined networking (SDN) controllers, which orchestrate the overlay network. Additionally, virtual tunnel endpoints (VTEPs) play a vital role in encapsulating and decapsulating network packets, ensuring efficient communication within the overlay network.

Overlay networking has genuinely transformed the landscape of modern network architectures. By providing flexibility, scalability, and enhanced security, overlay networks have become indispensable in various industries. Whether it is for data centers, cloud environments, or enterprise networks, overlay networking offers a powerful solution to meet the evolving demands of the digital era.

Conclusion:

In conclusion, overlay networking has emerged as a game-changer in the world of networking. Its ability to abstract and virtualize network services brings immense value to organizations, enabling them to adapt quickly, enhance security, and optimize resource utilization. As technology continues to advance, overlay networking will likely play an even more significant role in shaping the future of network architectures.

Green data center with eco friendly electricity usage tiny person concept. Database server technology for file storage hosting with ecological and carbon neutral power source vector illustration.

LISP Hybrid Cloud Use Case

November 20, 2014

by Matt Conran Blog

LISP Hybrid Cloud Use Case

In the world of networking, the ability to efficiently manage and scale networks is of paramount importance. This is where LISP networking comes into play. LISP, which stands for Locator/ID Separation Protocol, is a powerful networking technology that offers numerous benefits to network administrators and operators. In this blog post, we will explore the world of LISP networking and its key features and advantages.

LISP networking is a revolutionary approach to IP addressing and routing that separates the identity of a device (ID) from its location (locator). Traditional IP addressing relies on combining these two aspects, making it challenging to scale networks and manage mobility. LISP overcomes these limitations by decoupling the device's identity and location, enabling more flexible and scalable network architectures.

LISP, at its core, is a routing architecture that separates location and identity information for IP addresses. By doing so, it enables scalable and efficient routing across networks. LISP hybrid cloud leverages this architecture to seamlessly integrate multiple cloud environments, including public, private, and on-premises clouds.

Enhanced Scalability: LISP hybrid cloud allows organizations to scale their cloud infrastructure effortlessly. By abstracting location information from IP addresses, it enables efficient traffic routing across cloud environments, ensuring optimal utilization of resources.

Improved Security and Privacy: With LISP hybrid cloud, organizations can establish secure and private connections between different cloud environments. This ensures that sensitive data remains protected while being seamlessly accessed across clouds, bolstering data security and compliance.

Simplified Network Management: By centralizing network policies and control, LISP hybrid cloud simplifies network management for organizations. It provides a unified view of the entire cloud infrastructure, enabling efficient monitoring, troubleshooting, and policy enforcement.

Seamless Data Migration: LISP hybrid cloud enables seamless migration of data between different clouds, eliminating the complexities associated with traditional data migration methods. It allows organizations to transfer large volumes of data quickly and efficiently, minimizing downtime and disruption.

Hybrid Application Deployment: Organizations can leverage LISP hybrid cloud to deploy applications across multiple cloud environments. This enables a flexible and scalable infrastructure, where applications can utilize resources from different clouds based on specific requirements, optimizing performance and cost-efficiency.

The LISP hybrid cloud use case presents a compelling solution for organizations seeking to enhance their cloud infrastructure. With its scalability, security, and simplified network management benefits, LISP hybrid cloud opens up a world of possibilities for seamless integration and optimization of multiple cloud environments. Embracing LISP hybrid cloud can drive efficiency, flexibility, and agility, empowering organizations to stay ahead in today's dynamic digital landscape.

Matt Conran

Highlights: LISP Hybrid Cloud Use Case

Understanding LISP

LISP, short for Locator/ID Separation Protocol, is a routing architecture that separates the endpoint identifier (ID) from its location (locator). By doing so, LISP enables efficient routing, scalability, and mobility in IP networks. This protocol has been widely adopted in modern networking to address the challenges posed by the growth of the Internet and the limitations of traditional IP addressing.

Hybrid cloud architecture combines the best of both worlds by integrating public and private cloud environments. It allows organizations to leverage the scalability and cost-effectiveness of public clouds while maintaining control over sensitive data and critical applications in private clouds. This flexible approach provides businesses with the agility to scale their resources up or down based on demand, ensuring optimal performance and cost-efficiency.

The Synergy of LISP and Hybrid Cloud

When LISP and hybrid cloud architecture merge, the result is a powerful combination that offers numerous advantages. LISP’s ability to separate the ID from the locator enables seamless mobility and efficient routing across hybrid cloud environments. By leveraging LISP, organizations can achieve enhanced scalability, simplified network management, and improved performance across their distributed infrastructure.

Highlighting real-world examples of LISP hybrid cloud use cases can shed light on its practical applications. From multinational corporations with geographically dispersed offices to service providers managing cloud-based services, LISP hybrid cloud use cases have demonstrated significant improvements in network performance, reduced latency, simplified network management, and increased overall agility.

Data Analytics with LISP Hybrid Cloud

Data analytics becomes the strategic visibility layer that reveals how LISP behaves when stretched across hybrid cloud environments. In a hybrid cloud, workloads move between on‑prem data centers, public cloud regions, and edge locations — all while keeping a stable identity. LISP’s separation of EID and RLOC is the mechanism that makes this mobility possible, and analytics exposes how well that mechanism performs. Telemetry from Map‑Servers, Map‑Resolvers, and xTRs shows how quickly mappings propagate across clouds, how locator‑sets shift during mobility events, and how mapping‑cache entries age under bursty traffic. Analytics highlights hybrid‑specific issues such as cloud‑induced latency spikes affecting map‑reply timing, inconsistent EID registration between domains, or RLOC reachability changes caused by dynamic cloud routing. By correlating mapping behavior with workload movement, engineers gain precise insight into how identity resolution behaves across hybrid boundaries.

The data‑plane becomes equally transparent when analytics is applied to LISP’s cross‑cloud forwarding paths. Encapsulation tunnels between on‑prem xTRs and cloud gateways generate measurable indicators: per‑RLOC latency, tunnel jitter, packet‑loss trends, cloud‑region path asymmetry, and mobility‑driven tunnel shifts. Analytics can detect when cloud routing changes silently alter RLOC selection, when a region becomes overloaded, or when a mobility event triggers triangular routing between clouds. When analytics feeds hybrid controllers and mapping systems in real time, the architecture becomes adaptive: locator‑sets can be rebalanced across cloud regions, unstable EIDs dampened, mapping lifetimes tuned for inter‑cloud latency, and flows redirected toward healthier RLOCs before users notice degradation. Data analytics transforms LISP hybrid cloud into a self‑optimizing identity fabric that maintains predictable performance even during mobility, failover, or multi‑region bursts.

Use Case: Hybrid Cloud

The hybrid cloud connects the public cloud provider to the private enterprise cloud. It consists of two or more distinct infrastructures in dispersed locations that remain unique. These unique entities are bound together logically via a network to enable data and application portability. LISP networking performs hybrid cloud and can overcome the negative drawback of stretched VLAN. How do you support intra-subnet traffic patterns among two dispersed cloud locations? Without a stretched VLAN spanning locations, instability may arise from broadcast storms and Layer 2 loops.

**End to End-to-end connectivity**

Enterprises want the ability to seamlessly insert their application right into the heart of the cloud provider without changing any parameters. Customers want to do this without changing the VM’s IP addresses and MAC addresses. This requires the VLAN to be stretched end-to-end. Unfortunately, IP routing cannot support VLAN extension, which puts pressure on the data center interconnect ( DCI ) link to enable extended VLANs. In reality, and from experience, this is not a good solution.

**LISP Architecture on Cisco Platforms**

There are various Cisco platforms that support LISP, but the platforms are mainly characterized by the operating system software they run. LISP is supported by Cisco’s IOS/IOS-XE, IOS-XR, and NX-OS operating systems. LISP offers several distinctive features and functions, including xTR/MS/MR, IGP Assist, and ESM/ASM Multi-hop. It is not true that all hardware supports all functions or features. Users need to verify that a platform supports key features before implementing it.

IOS-XR and NX-OS do not have distributed architectures, as does Cisco IOS/IOS-XE.RIB and Cisco Express Forwarding (CEF) provide the forwarding architecture for LISP on IOS/IOS-XE platforms using the LISP control process.

Before you proceed, you may find the following helpful:

LISP Hybrid Cloud Use Case

The LISP Network

The LISP network comprises a mapping system with a global database of RLOC-EID mapping entries. The mapping system is the control plane of the LISP network decoupled from the data plane. The mapping system is address-family agnostic; the EID can be an IPv4 address mapped to an RLOC IPv6 address and vice versa. Or the EID may be a Virtual Extensible LAN (VXLAN) Layer 2 virtual network identifier (L2VNI) mapped to a VXLAN tunnel endpoint (VTEP) address working as an RLOC IP address.

How Does LISP Networking Work? At its core, LISP networking introduces a new level of indirection between the device’s IP address and location. LISP relies on two key components: the xTR (eXternal Tunnel Router) and the mapping system. The xTR is responsible for encapsulating and forwarding traffic between different LISP sites, while the mapping system stores the mappings between the device’s identity and its current location.

**Benefits of LISP Networking**

Scalability: LISP provides a scalable solution for managing large networks by separating the device’s identity from its location. This allows for efficient routing and reduces the amount of routing table information that needs to be stored and exchanged.

Mobility: LISP networking offers seamless mobility support, enabling devices to change locations without disrupting ongoing communications. This is particularly beneficial in scenarios where mobile devices are constantly moving, such as IoT deployments or mobile networks.

Traffic Engineering: LISP allows network administrators to optimize traffic flow by manipulating the mappings between device IDs and locators. This provides greater control over network traffic and enables efficient load balancing and congestion management.

Security: LISP supports secure communications through the use of cryptographic techniques. It provides authentication and integrity verification mechanisms, ensuring the confidentiality and integrity of data transmitted over the network.

Use Cases for LISP Networking:

A – Data Centers: LISP can significantly simplify the management of large-scale data center networks by providing efficient traffic engineering and seamless mobility support for virtual machines.

B- Internet Service Providers (ISPs): LISP can help ISPs improve their network scalability and handle the increasing demand for IP addresses. It enables ISPs to optimize their routing tables and efficiently manage address space.

C- IoT Deployments: LISP’s mobility support and scalability make it an ideal choice for IoT deployments. It efficiently manages large devices and enables seamless connectivity as devices move across different networks.

LISP Networking and Stretched VLAN

Locator Identity Separation Protocol ( LISP ) can extend subnets without the VLAN. I am creating a LISP Hybrid Cloud. A subnet extension with LISP is far more appealing than a Layer 2 LAN extension. The LISP-enabled hybrid cloud solution allows Intra-subnet communication regardless of where the server is. This means you can have two servers in different locations, one in the public cloud and the other in the Enterprise domain; both servers can communicate as if they were on the same subnet.

LISP acts as an overlay technology

LISP operates like an overlay technology; it encapsulates the source packet with UDP and a header consisting of the source and destination RLOC ( RLOC are used to map EIDS). The result is that you can address the servers in the cloud according to your addressing scheme. There is no need to match your addressing scheme to the cloud addressing scheme.

LISP on the Cloud Service Router ( CRS ) 1000V ( virtual router ) solution provides a Layer-3-based approach to a hybrid cloud. It allows you to stretch subnets from the enterprise to the public cloud without needing a Layer 2 LAN extension.

LISP networking deployment key points:

LISP can be deployed with the CRS 1000V in the cloud and either a CRS 1000V or ASR 1000 in the enterprise domain.
The enterprise CRS must have at least two interfaces. One interface is the L3 routed interface to the core. The second interface is a Layer 2 interface to support VLAN connectivity for the servers that require mobility.
The enterprise CRS does not need to be the default gateway, and its interaction with the local infrastructure ( via the Layer 2 interface ) is based on Proxy-ARP. As a result, ARP packets must be allowed on the underlying networks.
The Cloud CRS is also deployed with at least two interfaces. One interface is facing the Internet or MPLS network. The second interface faces the local infrastructure, either by VLANs or Virtual Extensible LAN ( VXLAN ).
The CRS offers machine-level high availability and supports all the VMware high-availability features such as dynamic resource scheduling ( DRS ), vMotion, NIC load balancing, and teaming.

Hybrid Cloud — Hybrid cloud and CRS1000V

LISP is a network-based solution and is independent of the hypervisor. You can have different hypervisors in the Enterprise and the public cloud. No changes to virtual servers or hosts. It’s completely transparent.
The PxTR ( also used to forward to non-LISP sites ) is deployed in the enterprise cloud, and the xTR is deployed in the public cloud.
The CRS1000V deployed in the public cloud is secured by an IPSEC tunnel. Therefore, the LISP tunnel should be encrypted using IPSEC tunnel mode. Tunnel mode is preferred to support NAT.
Each CRS must have one unique outside IP address. This is used to form the IPSEC tunnel between the two endpoints.
Dynamic or static Routing must be enabled over the IPSEC tunnel. This is to announce the RLOC IP address used by the LISP mapping system.
The map-resolver ( MR ) and map server ( MS ) can be enabled on the xTR in the Enterprise or the xTR in the cloud.
Traffic symmetry is still required when you have stateful devices in the path.

LISP stretched subnets

The two modes of LISP operation are the LISP “Across” subnet and the LISP “Extended” subnet mode. Neither of these modes is used with the LISP-enabled CRS hybrid cloud deployment scenario. The mode of operation utilized is called the LISP stretched subnet model ( SSM ). The same subnet is used on both sides of the network, and mobility is performed between these two segments on the same subnet. You may think that this is the same as LISP “Extended” subnet mode, but in this case, we are not using a LAN extension between sites. Instead, the extended mode requires a LAN extension such as OTV.

Closing Points on LISP Hybrid Cloud

LISP, or Locator/ID Separation Protocol, is a novel architecture that aims to improve the scalability and manageability of IP networks. By decoupling the location and identity of network devices, LISP enables more efficient routing and simplifies network management. This separation allows for seamless integration between different cloud environments, making it an ideal choice for hybrid cloud implementations. In a LISP-enabled network, endpoints have two addresses: an Endpoint Identifier (EID) that identifies the device and a Routing Locator (RLOC) that identifies its location in the network.

1. **Enhanced Scalability**: One of the key advantages of using LISP in a hybrid cloud setup is its ability to scale effortlessly. By separating identity and location, LISP reduces the complexity of routing tables, allowing networks to grow without the traditional limitations.

2. **Improved Security**: Security is a paramount concern for any cloud implementation. LISP provides improved security features by enabling network segmentation and facilitating the creation of secure communication tunnels between different cloud environments.

3. **Seamless Mobility**: LISP’s architecture supports seamless mobility, which is crucial for hybrid cloud environments where workloads may move between on-premises and cloud-based resources. This mobility ensures minimal disruption and consistent performance.

To successfully implement LISP in a hybrid cloud environment, organizations should follow a structured approach:

1. **Assessment and Planning**: Begin by assessing your current network infrastructure and identifying areas where LISP can bring the most value. Develop a detailed implementation plan that outlines the steps, resources, and timelines required.

2. **Deployment**: Deploy LISP-enabled routers and configure them to support both EIDs and RLOCs. Ensure that your network devices are compatible with the LISP protocol and that they are properly configured to handle LISP traffic.

3. **Testing and Optimization**: Conduct thorough testing to ensure that LISP is functioning as expected. Monitor network performance and make necessary adjustments to optimize routing and security.

Summary: LISP Hybrid Cloud Use Case

In the rapidly evolving world of cloud computing, businesses constantly seek innovative solutions to optimize their operations. One such groundbreaking approach is the utilization of LISP (Locator/ID Separation Protocol) in hybrid cloud environments. In this blog post, we explored the fascinating use case of LISP Hybrid Cloud and delved into its benefits, implementation, and potential for revolutionizing the industry.

Understanding LISP Hybrid Cloud

LISP Hybrid Cloud combines the best of two worlds: the scalability and flexibility of public cloud services with the security and control of private cloud infrastructure. By separating the location and identity of network devices, LISP allows for seamless communication between public and private clouds. This breakthrough technology enables businesses to leverage the advantages of both environments and optimize their cloud strategies.

Benefits of LISP Hybrid Cloud

Enhanced Scalability: LISP Hybrid Cloud offers unparalleled scalability by allowing businesses to scale their operations across public and private clouds seamlessly. This ensures that organizations can meet evolving demands without compromising performance or security.

Improved Flexibility: With LISP Hybrid Cloud, businesses can choose the most suitable cloud resources. They can leverage the vast capabilities of public clouds for non-sensitive workloads while keeping critical data and applications secure within their private cloud infrastructure.

Enhanced Security: LISP Hybrid Cloud provides enhanced security by leveraging the inherent advantages of private clouds. Critical data and applications can remain within the organization’s secure network, minimizing the risk of unauthorized access or data breaches.

Implementation of LISP Hybrid Cloud

Implementing LISP Hybrid Cloud involves several key steps. First, organizations must evaluate their cloud requirements and determine the optimal balance between public and private cloud resources. Next, they must deploy the necessary LISP infrastructure, including LISP routers and mapping servers. Finally, businesses must establish secure communication channels between their public and private cloud environments, ensuring seamless data transfer and interconnectivity.

Conclusion:

In conclusion, LISP Hybrid Cloud represents a revolutionary approach to cloud computing. By harnessing the power of LISP, businesses can unlock the potential of hybrid cloud environments, enabling enhanced scalability, improved flexibility, and heightened security. As the cloud landscape continues to evolve, LISP Hybrid Cloud is poised to play a pivotal role in shaping the future of cloud computing.

What is OpenFlow

September 25, 2014

by Matt Conran Blog

What is OpenFlow?

In today's rapidly evolving digital landscape, network management and data flow control have become critical for businesses of all sizes. OpenFlow is one technology that has gained significant attention and is transforming how networks are managed. In this blog post, we will delve into the concept of OpenFlow, its advantages, and its implications for network control.

OpenFlow is an open-standard communications protocol that separates the control and data planes in a network architecture. It allows network administrators to have direct control over the behavior of network devices, such as switches and routers, by utilizing a centralized controller.

Traditional network architectures follow a closed model, where network devices make independent decisions on forwarding packets. On the other hand, OpenFlow introduces a centralized control plane that provides a global view of the network and allows administrators to define network policies and rules from a centralized location.

OpenFlow operates by establishing a secure channel between the centralized controller and the network switches. The controller is responsible for managing the flow tables within the switches, defining how traffic should be forwarded based on predefined rules and policies. This separation of control and data planes allows for dynamic network management and facilitates the implementation of innovative network protocols.

One of the key advantages of OpenFlow is its ability to simplify network management. By centralizing control, administrators can easily configure and manage the entire network from a single point of control. This reduces complexity and enhances the scalability of network infrastructure. Additionally, OpenFlow enables network programmability, allowing for the development of custom networking applications and services tailored to specific requirements.

OpenFlow plays a crucial role in network virtualization, as it allows for the creation and management of virtual networks on top of physical infrastructure. By abstracting the underlying network, OpenFlow empowers organizations to optimize resource utilization, improve security, and enhance network performance. It opens doors to dynamic provisioning, isolation, and efficient utilization of network resources.

Matt Conran

Highlights: What is OpenFlow?

How does OpenFlow work?

1: OpenFlow allows network controllers to determine the path of network packets in a network of switches. There is a difference between switches and controllers. With separate control and forwarding, traffic management can be more sophisticated than access control lists (ACLs) and routing protocols.

2: An OpenFlow protocol allows switches from different vendors, often with proprietary interfaces and scripting languages, to be managed remotely. Software-defined networking (SDN) is considered to be enabled by OpenFlow by its inventors.

3: With OpenFlow, Layer 3 switches can add, modify, and remove packet-matching rules and actions remotely. By doing so, routing decisions can be made periodically or ad hoc by the controller and translated into rules and actions with a configurable lifespan, which are then deployed to the switch’s flow table, where packets are forwarded at wire speed for the duration of the rule.

The Role of OpenFlow Controllers

If the switch cannot match packets, they can be sent to the controller. The controller can modify existing flow table rules or deploy new rules to prevent a structural traffic flow. It may even forward the traffic itself if the switch is instructed to forward packets rather than just their headers.

OpenFlow uses Transport Layer Security (TLS) over Transmission Control Protocol (TCP). Switches wishing to connect should listen on TCP port 6653. In earlier versions of OpenFlow, port 6633 was unofficially used. The protocol is mainly used between switches and controllers.

### Origins of OpenFlow

The inception of OpenFlow can be traced back to the early 2000s when researchers at Stanford University sought to create a more versatile and programmable network architecture. Traditional networking relied heavily on static and proprietary hardware configurations, which limited innovation and adaptability. OpenFlow emerged as a solution to these challenges, offering a standardized protocol that decouples the network control plane from the data plane. This separation allows for centralized control and dynamic adjustment of network traffic, fostering innovation and agility in network design.

### How OpenFlow Works

At its core, the OpenFlow protocol facilitates communication between network devices and a centralized SDN controller. It does this by using a series of flow tables within network switches, which are programmed by the controller. These flow tables dictate how packets should be handled, whether they are forwarded to a destination, dropped, or modified in some way. By leveraging OpenFlow, network administrators can deploy updates, optimize performance, and troubleshoot issues with unprecedented speed and precision, all from a single point of control.

Introducing SDN

Recent changes and requirements have driven networks and network services to become more flexible, virtualization-aware, and API-driven. One major trend affecting the future of networking is software-defined networking ( SDN ). The software-defined architecture aims to extract the entire network into a single switch.

Software-defined networking (SDN) is an evolving technology defined by the Open Networking Foundation ( ONF ). It involves the physical separation of the network control plane from the forwarding plane, where a control plane controls several devices. This differs significantly from traditional IP forwarding that you may have used in the past.

The Core Concepts of SDN

At its heart, Software-Defined Networking decouples the network control plane from the data plane, allowing network administrators to manage network services through abstraction of lower-level functionality. This separation enables centralized network control, which simplifies the management of complex networks. The control plane makes decisions about where traffic is sent, while the data plane forwards traffic to the selected destination. This approach allows for a more flexible, adaptable network infrastructure.

The activities around OpenFlow

Even though OpenFlow has received a lot of industry attention, programmable networks and decoupled control planes (control logic) from data planes have been around for many years. To enhance ATM, Internet, and mobile networks’ openness, extensibility, and programmability, the Open Signaling (OPENING) working group held workshops in 1995. A working group within the Internet Engineering Task Force (IETF) developed GSMP to control label switches based on these ideas. June 2002 marked the official end of this group, and GSMPv3 was published.

Data and control plane

Therefore, SDN separates the data and control plane. The main driving body behind software-defined networking (SDN) is the Open Networking Foundation ( ONF ). Introduced in 2008, the ONF is a non-profit organization that wants to provide an alternative to proprietary solutions that limit flexibility and create vendor lock-in.

The insertion of the ONF allowed its members to run proof of concepts on heterogeneous networking devices without requiring vendors to expose their software’s internal code. This creates a path for an open-source approach to networking and policy-based controllers.

Knowledge Check: Data & Control Plane

### Data Plane: The Highway for Your Data

The data plane, often referred to as the forwarding plane, is responsible for the actual movement of packets of data from source to destination. Imagine it as a network’s highway, where data travels at high speed. This component operates at the speed of light, handling massive amounts of data with minimal delay. It’s designed to process packets quickly, ensuring that the information arrives where it needs to be without interruption.

### Control Plane: The Brain Behind the Operation

While the data plane acts as the highway, the control plane is the brain that orchestrates the flow of traffic. It makes decisions about routing, managing the network topology, and controlling the data plane’s operations. The control plane uses protocols to determine the best paths for data and to update routing tables as needed. It’s responsible for ensuring that the network operates efficiently, adapting to changes, and maintaining optimal performance.

### Interplay Between Data and Control Planes

The synergy between the data and control planes is what enables modern networks to function effectively. The control plane provides the intelligence and decision-making necessary to guide the data plane. This interaction ensures that data packets take the best possible paths, reducing latency and maximizing throughput. As networks evolve, the lines between these planes may blur, but their distinct roles remain pivotal.

### Real-World Applications and Innovations

The concepts of data and control planes are not just theoretical—they have practical applications in technologies such as Software-Defined Networking (SDN) and Network Function Virtualization (NFV). These innovations allow for greater flexibility and scalability in managing network resources, offering businesses the agility needed to adapt to changing demands.

Building blocks: SDN Environment

As a fundamental building block of an SDN deployment, the controller, the SDN switch (for example, an OpenFlow switch), and the interfaces are present in the controller to communicate with forwarding devices, generally the southbound interface (OpenFlow) and the northbound interface (the network application interface).

In an SDN, switches function as basic forwarding hardware, accessible via an open interface, with the control logic and algorithms offloaded to controllers. Hybrid (OpenFlow-enabled) and pure (OpenFlow-only) OpenFlow switches are available.

OpenFlow switches rely entirely on a controller for forwarding decisions, without legacy features or onboard control. Hybrid switches support OpenFlow as well, in addition to traditional operation and protocols. Today, hybrid switches are the most common type of commercial switch. A flow table performs packet lookup and forwarding in an OpenFlow switch.

### The Role of the OpenFlow Controller

The OpenFlow controller is the brain of the SDN, orchestrating the flow of data across the network. It communicates with the network devices using the OpenFlow protocol to dictate how packets should be handled. The controller’s primary function is to make decisions on the path data packets should take, ensuring optimal network performance and resource utilization. This centralization of control allows for dynamic network configuration, paving the way for innovative applications and services.

### OpenFlow Switches: The Workhorses of the Network

While the controller is the brain, OpenFlow switches are the workhorses, executing the instructions they receive. These switches operate at the data plane, where they forward packets based on the rules set by the controller. Each switch maintains a flow table that matches incoming packets to particular actions, such as forwarding or dropping the packet. This separation of control and data planes is what sets SDN apart from traditional networking, offering unparalleled flexibility and control.

### Flow Tables: The Heart of OpenFlow Switches

Flow tables are the core component of OpenFlow switches, dictating how packets are handled. Each entry in a flow table consists of match fields, counters, and a set of instructions. Match fields identify the packets that should be affected by the rule, counters track the number of packets and bytes that match the entry, and instructions define the actions to be taken. This modular approach allows for precise traffic management and is essential for implementing advanced network policies.

OpenFlow Table & Routing Tables

### What is an OpenFlow Flow Table?

OpenFlow is a protocol that allows network controllers to interact with the forwarding plane of network devices like switches and routers. At the heart of OpenFlow is the flow table. This table contains a set of flow entries, each specifying actions to take on packets that match a particular pattern. Unlike traditional routing tables, which rely on predefined paths and protocols, OpenFlow flow tables provide flexibility and programmability, allowing dynamic changes in how packets are handled based on real-time network conditions.

### Understanding Routing Tables

Routing tables are a staple of traditional networking, used by routers to determine the best path for forwarding packets to their final destinations. These tables consist of a list of network destinations, with associated metrics that help in selecting the most efficient route. Routing protocols such as OSPF, BGP, and RIP are employed to maintain and update these tables, ensuring that data flows smoothly across the interconnected web of networks. While reliable, routing tables are less flexible compared to OpenFlow flow tables, as changes in network traffic patterns require updates to routing protocols and configurations.

Example of a Routing Table

### Key Differences Between OpenFlow Flow Table and Routing Table

The primary distinction between OpenFlow flow tables and routing tables lies in their approach to network management. OpenFlow flow tables are dynamic and programmable, allowing for real-time adjustments and fine-grained control over traffic flows. This makes them ideal for environments where network agility and customization are paramount. Conversely, routing tables offer a more static and predictable method of packet forwarding, which can be beneficial in stable networks where consistency and reliability are prioritized.

### Use Cases: When to Use Each

OpenFlow flow tables are particularly advantageous in software-defined networking (SDN) environments, where network administrators need to quickly adapt to changing conditions and optimize traffic flows. They are well-suited for data centers, virtualized networks, and scenarios requiring high levels of automation and scalability. On the other hand, traditional routing tables are best used in established networks with predictable traffic patterns, such as those found in enterprise or service provider settings, where reliability and stability are key.

You may find the following useful for pre-information:

What is OpenFlow?

OpenFlow was the first protocol of the Software-Defined Networking (SDN) trend and is the only protocol that allows the decoupling of a network device’s control plane from the data plane. In most straightforward terms, the control plane can be thought of as the brains of a network device. On the other hand, the data plane can be considered hardware or application-specific integrated circuits (ASICs) that perform packet forwarding.

Numerous devices also support running OpenFlow in a hybrid mode, meaning OpenFlow can be deployed on a given port, virtual local area network (VLAN), or even within a regular packet-forwarding pipeline such that if there is not a match in the OpenFlow table, then the existing forwarding tables (MAC, Routing, etc.) are used, making it more analogous to Policy Based Routing (PBR).

What is SDN?

Despite various modifications to the underlying architecture and devices (such as switches, routers, and firewalls), traditional network technologies have existed since the inception of networking. Using a similar approach, frames and packets have been forwarded and routed in a limited manner, resulting in low efficiency and high maintenance costs. Consequently, the architecture and operation of networks need to evolve, resulting in SDN.

By enabling network programmability, SDN promises to simplify network control and management and allow innovation in computer networking. Network engineers configure policies to respond to various network events and application scenarios. They can achieve the desired results by manually converting high-level policies into low-level configuration commands.

Often, minimal tools are available to accomplish these very complex tasks. Controlling network performance and tuning network management are challenging and error-prone tasks.

A modern network architecture consists of a control plane, a data plane, and a management plane; the control and data planes are merged into a machine called Inside the Box. To overcome these limitations, programmable networks have emerged.

How OpenFlow Works:

At the core of OpenFlow is the concept of a flow table, which resides in each OpenFlow-enabled switch. The flow table contains match-action rules defining how incoming packets should be processed and forwarded. The centralized controller determines these rules and communicates using the OpenFlow protocol with the switches.

When a packet arrives at an OpenFlow-enabled switch, it is first matched against the rules in the flow table. If a match is found, the corresponding action is executed, including forwarding the packet, dropping it, or sending it to the controller for further processing. This decoupling of the control and data planes allows for flexible and programmable network management.

What is OpenFlow SDN?

The main goal of SDN is to separate the control and data planes and transfer network intelligence and state to the control plane. These concepts have been exploited by technologies like Routing Control Platform (RCP), Secure Architecture for Network Enterprise (SANE), and, more recently, Ethane.

In addition, there is often a connection between SDN and OpenFlow. The Open Networking Foundation (ONF) is responsible for advancing SDN and standardizing OpenFlow, whose latest version is 1.5.0.

An SDN deployment starts with these building blocks.

For communication with forwarding devices, the controller has the SDN switch (for example, an OpenFlow switch), the SDN controller, and the interfaces. An SDN deployment is based on two basic building blocks: a southbound interface (OpenFlow) and a northbound interface (the network application interface).

As the control logic and algorithms are offloaded to a controller, switches in SDNs may be represented as basic forwarding hardware. Switches that support OpenFlow come in two varieties: pure (OpenFlow-only) and hybrid (OpenFlow-enabled).

Pure OpenFlow switches do not have legacy features or onboard control for forwarding decisions. A hybrid switch can operate with both traditional protocols and OpenFlow. Hybrid switches make up the majority of commercial switches available today. In an OpenFlow switch, a flow table performs packet lookup and forwarding.

OpenFlow reference switch

The OpenFlow protocol and interface allow OpenFlow switches to be accessed as essential forwarding elements. A flow-based SDN architecture like OpenFlow simplifies switching hardware. Still, it may require additional forwarding tables, buffer space, and statistical counters that are difficult to implement in traditional switches with integrated circuits tailored to specific applications.

There are two types of switches in an OpenFlow network: hybrids (which enable OpenFlow) and pores (which only support OpenFlow). OpenFlow is supported by hybrid switches and traditional protocols (L2/L3). OpenFlow switches rely entirely on a controller for forwarding decisions and do not have legacy features or onboard control.

Hybrid switches are the majority of the switches currently available on the market. This link must remain active and secure because OpenFlow switches are controlled over an open interface (through a TCP-based TLS session). OpenFlow is a messaging protocol that defines communication between OpenFlow switches and controllers, which can be viewed as an implementation of SDN-based controller-switch interactions.

Openflow switch — Diagram: OpenFlow switch. The source is cable solution.

Identify the Benefits of OpenFlow

Application-driven routing. Users can control the network paths.	The networks paths.A way to enhance link utilization.
An open solution for VM mobility. No VLAN reliability.	A means to traffic engineer without MPLS.
A solution to build very large Layer 2 networks.	A way to scale Firewalls and Load Balancers.
A way to configure an entire network as a whole as opposed to individual entities.	A way to build your own encryption solution. Off-the-box encryption.
A way to distribute policies from a central controller.	Customized flow forwarding. Based on a variety of bit patterns.
A solution to get a global view of the network and its state. End-to-end visibility.	A solution to use commodity switches in the network. Massive cost savings.

The following table lists the Software Networking ( SDN ) benefits and the problems encountered with existing control plane architecture:

Identify the benefits of OpenFlow and SDN	Problems with the existing approach
Faster software deployment.	Large scale provisioning and orchestration.
Programmable network elements.	Limited traffic engineering ( MPLS TE is cumbersome )
Faster provisioning.	Synchronized distribution policies.
Centralized intelligence with centralized controllers.	Routing of large elephant flows.
Decisions are based on end-to-end visibility.	Qos and load based forwarding models.
Granular control of flows.	Ability to scale with VLANs.
Decreases the dependence on network appliances like load balancers.

A key point: The lack of a session layer in the TCP/IP stack

Regardless of the hype and benefits of SDN, neither OpenFlow nor other SDN technologies address the real problems of the lack of a session layer in the TCP/IP protocol stack. The problem is that the client’s application ( Layer 7 ) connects to the server’s IP address ( Layer 3 ), and if you want to have persistent sessions, the server’s IP address must remain reachable.

This session’s persistence and the ability to connect to multiple Layer 3 addresses to reach the same device is the job of the OSI session layer. The session layer provides the services for opening, closing, and managing a session between end-user applications. In addition, it allows information from different sources to be correctly combined and synchronized.

The problem is the TCP/IP reference module does not consider a session layer, and there is none in the TCP/IP protocol stack. SDN does not solve this; it gives you different tools to implement today’s kludges.

Control and data plane

When we identify the benefits of OpenFlow, let us first examine traditional networking operations. Traditional networking devices have a control and forwarding plane, depicted in the diagram below. The control plane is responsible for setting up the necessary protocols and controls so the data plane can forward packets, resulting in end-to-end connectivity. These roles are shared on a single device, and the fast packet forwarding ( data path ) and the high-level routing decisions ( control path ) occur on the same device.

What is OpenFlow | SDN separates the data and control plane?

Control plane

The control plane is part of the router architecture and is responsible for drawing the network map in routing. When we mention control planes, you usually think about routing protocols, such as OSPF or BGP. But in reality, the control plane protocols perform numerous other functions, including:

Connectivity management ( BFD, CFM )	Interface state management ( PPP, LACP )
Service provisioning ( RSVP for InServ or MPLS TE)	Topology and reachability information exchange ( IP routing protocols, IS-IS in TRILL/SPB )
Adjacent device discovery via HELLO mechanism	ICMP

Control plane protocols run over data plane interfaces to ensure “shared fate” – if the packet forwarding fails, the control plane protocol fails as well.

Most control plane protocols ( BGP, OSPF, BFD ) are not data-driven. A BGP or BFD packet is never sent as a direct response to a data packet. There is a question mark over the validity of ICMP as a control plane protocol. The debate is whether it should be classed in the control or data plane category.

Some ICMP packets are sent as replies to other ICMP packets, and others are triggered by data plane packets, i.e., data-driven. My view is that ICMP is a control plane protocol that is triggered by data plane activity. After all, the “C” is ICMP does stand for “Control.”

Data plane

The data path is part of the routing architecture that decides what to do when a packet is received on its inbound interface. It is primarily focused on forwarding packets but also includes the following functions:

ACL logging	Netflow accounting
NAT session creation	NAT table maintenance

The data forwarding is usually performed in dedicated hardware, while the additional functions ( ACL logging, Netflow accounting ) typically happen on the device CPU, commonly known as “punting.” The data plane for an OpenFlow-enabled network can take a few forms.

However, the most common, even in the commercial offering, is the Open vSwitch, often called the OVS. The Open vSwitch is an open-source implementation of a distributed virtual multilayer switch. It enables a switching stack for virtualization environments while supporting multiple protocols and standards.

Software-defined networking changes the control and data plane architecture.

The concept of SDN separates these two planes, i.e., the control and forwarding planes are decoupled. This allows the networking devices in the forwarding path to focus solely on packet forwarding. An out-of-band network uses a separate controller ( orchestration system ) to set up the policies and controls. Hence, the forwarding plane has the correct information to forward packets efficiently.

In addition, it allows the network control plane to be moved to a centralized controller on a server instead of residing on the same box carrying out the forwarding. Moving the intelligence ( control plane ) of the data plane network devices to a controller enables companies to use low-cost, commodity hardware in the forwarding path. A significant benefit is that SDN separates the data and control plane, enabling new use cases.

A centralized computation and management plane makes more sense than a centralized control plane.

The controller maintains a view of the entire network and communicates with Openflow ( or, in some cases, BGP with BGP SDN ) with the different types of OpenFlow-enabled network boxes. The data path portion remains on the switch, such as the OVS bridge, while the high-level decisions are moved to a separate controller. The data path presents a clean flow table abstraction, and each flow table entry contains a set of packet fields to match, resulting in specific actions ( drop, redirect, send-out-port ).

When an OpenFlow switch receives a packet it has never seen before and doesn’t have a matching flow entry, it sends the packet to the controller for processing. The controller then decides what to do with the packet.

Applications could then be developed on top of this controller, performing security scrubbing, load balancing, traffic engineering, or customized packet forwarding. The centralized view of the network simplifies problems that were harder to overcome with traditional control plane protocols.

A single controller could potentially manage all OpenFlow-enabled switches. Instead of individually configuring each switch, the controller can push down policies to multiple switches simultaneously—a compelling example of many-to-one virtualization.

Now that SDN separates the data and control plane, the operator uses the centralized controller to choose the correct forwarding information per-flow basis. This allows better load balancing and traffic separation on the data plane. In addition, there is no need to enforce traffic separation based on VLANs, as the controller would have a set of policies and rules that would only allow traffic from one “VLAN” to be forwarded to other devices within that same “VLAN.”

The advent of VXLAN

With the advent of VXLAN, which allows up to 16 million logical entities, the benefits of SDN should not be purely associated with overcoming VLAN scaling issues. VXLAN already does an excellent job with this. It does make sense to deploy a centralized control plane in smaller independent islands; in my view, it should be at the edge of the network for security and policy enforcement roles. Using Openflow on one or more remote devices is easy to implement and scale.

It also decreases the impact of controller failure. If a controller fails and its sole job is implementing packet filters when a new user connects to the network, the only affecting element is that the new user cannot connect. If the controller is responsible for core changes, you may have interesting results with a failure. New users not being able to connect is bad, but losing your entire fabric is not as bad.

Spanning tree VXLAN — Diagram: Loop prevention. Source is Cisco

A traditional networking device runs all the control and data plane functions. The control plane, usually implemented in the central CPU or the supervisor module, downloads the forwarding instructions into the data plane structures. Every vendor needs communications protocols to bind the two planes together to download forward instructions.

Therefore, all distributed architects need a protocol between control and data plane elements. The protocol binding this communication path for traditional vendor devices is not open-source, and every vendor uses its proprietary protocol (Cisco uses IPC—InterProcess Communication ).

Openflow tries to define a standard protocol between the control plane and the associated data plane. When you think of Openflow, you should relate it to the communication protocol between the traditional supervisors and the line cards. OpenFlow is just a low-level tool.

OpenFlow is a control plane ( controller ) to data plane ( OpenFlow enabled device ) protocol that allows the control plane to modify forwarding entries in the data plane. It enables SDN to separate the data and control planes.

Proactive versus reactive flow setup

OpenFlow operations have two types of flow setups: Proactive and Reactive.

With Proactive, the controller can populate the flow tables ahead of time, similar to a typical routing. However, the packet-in event never occurs by pre-defining your flows and actions ahead of time in the switch’s flow tables. The result is all packets are forwarded at line rate. With Reactive, the network devices react to traffic, consult the OpenFlow controller, and create a rule in the flow table based on the instruction. The problem with this approach is that there can be many CPU hits.

The following table outlines the critical points for each type of flow setup:

Proactive flow setup	Reactive flow setup
Works well when the controller is emulating BGP or OSPF.	Used when no one can predict when and where a new MAC address will appear.
The controller must first discover the entire topology.	Punts unknown packets to the controller. Many CPU hits.
Discover endpoints ( MAC addresses, IP addresses, and IP subnets )	Compute forwarding paths on demand. Not off the box computation.
Compute off the box optimal forwarding.	Install flow entries based on actual traffic.
Download flow entries to the data plane switches.	Has many scalability concerns such as packet punting rate.
No data plane controller involvement with the exceptions of ARP and MAC learning. Line-rate performance.	Not a recommended setup.

Hop-by-hop versus path-based forwarding

The following table illustrates the key points for the two types of forwarding methods used by OpenFlow: hop-by-hop forwarding and path-based forwarding:

Hop-by-hop Forwarding	Path-based Forwarding
Similar to traditional IP Forwarding.	Similar to MPLS.
Installs identical flows on each switch on the data path.	Map flows to paths on ingress switches and assigns user traffic to paths at the edge node
Scalability concerns relating to flow updates after a change in topology.	Compute paths across the network and installs end-to-end path-forwarding entries.
Significant overhead in large-scale networks.	Works better than hop-by-hop forwarding in large-scale networks.
FIB update challenges. Convergence time.	Core switches don’t have to support the same granular functionality as edge switches.

Obviously, with any controller, the controller is a lucrative target for attack. Anyone who knows you are using a controller-based network will try to attack the controller and its control plane. The attacker may attempt to intercept the controller-to-switch communication and replace it with its commands, essentially attacking the control plane with whatever means they like.

An attacker may also try to insert a malformed packet or some other type of unknown packet into the controller ( fuzzing attack ), exploiting bugs in the controller and causing the controller to crash.

Fuzzing attacks can be carried out with application scanning software such as Burp Suite. It attempts to manipulate data in a particular way, breaking the application.

The best way to tighten security is to encrypt switch-to-controller communications with SSL and self-signed certificates to authenticate the switch and controller. It would also be best to minimize interaction with the data plane, except for ARP and MAC learning.

To prevent denial-of-service attacks on the controller, you can use Control Plane Policing ( CoPP ) on Ingress to avoid overloading the switch and the controller. Currently, NEC is the only vendor implementing CoPP.

The Hybrid deployment model is helpful from a security perspective. For example, you can group specific ports or VLANs to OpenFlow and other ports or VLANs to traditional forwarding, then use traditional forwarding to communicate with the OpenFlow controller.

Software-defined networking or traditional routing protocols?

The move to a Software-Defined Networking architecture has clear advantages. It’s agile and can react quickly to business needs, such as new product development. For businesses to succeed, they must have software that continues evolving.

Otherwise, your customers and staff may lose interest in your product and service. The following table displays the advantages and disadvantages of the existing routing protocol control architecture.

+Reliable and well known.	-Non-standard Forwarding models. Destination-only and not load-aware metrics**
+Proven with 20 plus years field experience.	-Loosely coupled.
+Deterministic and predictable.	-Lacks end-to-end transactional consistency and visibility.
+Self-Healing. Traffic can reroute around a failed node or link.	-Limited Topology discovery and extraction. Basic neighbor and topology tables.
+Autonomous.	-Lacks the ability to change existing control plane protocol behavior.
+Scalable.	-Lacks the ability to introduce new control plane protocols.
+Plenty of learning and reading materials.

** Basic EIGRP IETF originally proposed an Energy-Aware Control Plane, but the IETF later removed this.

Software-Defined Networking: Use Cases

Edge Security policy enforcement at the network edge.	Authenticate users or VMs and deploy per-user ACL before connecting a user to the network.
Custom routing and online TE.	The ability to route on a variety of business metrics aka routing for dollars. Allowing you to override the default routing behavior.
Custom traffic processing.	For analytics and encryption.
Programmable SPAN ports	Use Openflow entries to mirror selected traffic to the SPAN port.
DoS traffic blackholing & distributed DoS prevention.	Block DoS traffic as close to the source as possible with more selective traffic targeting than the original RTBH approach**. The traffic blocking is implemented in OpenFlow switches. Higher performance with significantly lower costs.
Traffic redirection and service insertion.	Redirect a subset of traffic to network appliances and install redirection flow entries wherever needed.
Network Monitoring.	The controller is the authoritative source of information on network topology and Forwarding paths.
Scale-Out Load Balancing.	Punt new flows to the Openflow controller and install per-session entries throughout the network.
IPS Scale-Out.	OpenFlow is used to distribute the load to multiple IDS appliances.

**Remote-Triggered Black Hole: RTBH refers to installing a host route to a bogus IP address ( RTBH address ) pointing to NULL interfaces on all routers. BGP is used to advertise the host routes to other BGP peers of the attacked hosts, with the next hop pointing to the RTBH address, and it is mainly automated in ISP environments.

SDN deployment models

Guidelines:

Start with small deployments away from the mission-critical production path, i.e., the Core. Ideally, start with device or service provisioning systems.
Start at the Edge and slowly integrate with the Core. Minimize the risk and blast radius. Start with packet filters at the Edge and tasks that can be easily automated ( VLANs ).
Integrate new technology with the existing network.
Gradually increase scale and gain trust. Experience is key.
Have the controller in a protected out-of-band network with SSL connectivity to the switches.

There are 4 different models for OpenFlow deployment, and the following sections list the key points of each model.

Native OpenFlow

They are commonly used for Greenfield deployments.
The controller performs all the intelligent functions.
The forwarding plane switches have little intelligence and solely perform packet forwarding.
The white box switches need IP connectivity to the controller for the OpenFlow control sessions. If you are forced to use an in-band network for this communication path using an isolated VLAN with STP, you should use an out-of-band network.
Fast convergence techniques such as BFD may be challenging to use with a central controller.
Many people believe that this approach does not work for a regular company. Companies implementing native OpenFlow, such as Google, have the time and resources to reinvent the wheel when implementing a new control-plane protocol ( OpenFlow ).

Native OpenFlow with Extensions

Some control plane functions are handled from the centralized controller to the forwarding plane switches. For example, the OpenFlow-enabled switches could load balancing across multiple links without the controller’s previous decision. You could also run STP, LACP, or ARP locally on the switch without interaction with the controller. This approach is helpful if you lose connectivity to the controller. If the low-level switches perform certain controller functions, packet forwarding will continue in the event of failure.
The local switches should support the specific OpenFlow extensions that let them perform functions on the controller’s behalf.

Hybrid ( Ships in the night )

This approach is used where OpenFlow runs in parallel with the production network.
The same network box is controlled by existing on-box and off-box control planes ( OpenFlow).
Suitable for pilot deployment models as switches still run traditional control plane protocols.
The Openflow controller manages only specific VLANs or ports on the network.
The big challenge is determining and investigating the conflict-free sharing of forwarding plane resources across multiple control planes.

Integrated OpenFlow

OpenFlow classifiers and forwarding entries are integrated with the existing control plane. For example, Juniper’s OpenFlow model follows this mode of operation where OpenFlow static routes can be redistributed into the other routing protocols.
No need for a new control plane.
No need to replace all forwarding hardware
It is the most practical approach as long as the vendor supports it.

Closing Points on OpenFlow

OpenFlow is a communication protocol that provides access to the forwarding plane of a network switch or router over the network. It was initially developed at Stanford University and has since been embraced by the Open Networking Foundation (ONF) as a core component of SDN. OpenFlow allows network administrators to program the control plane, enabling them to direct how packets are forwarded through the network. This decoupling of the control and data planes is what empowers SDN to offer more dynamic and flexible network management.

The architecture of OpenFlow is quite straightforward yet powerful. It consists of three main components: the controller, the switch, and the protocol itself. The controller is the brains of the operation, managing network traffic by sending instructions to OpenFlow-enabled switches. These switches, in turn, execute the instructions received, altering the flow of network data accordingly. This setup allows for centralized network control, offering unprecedented levels of automation and agility.

**Advantages of OpenFlow**

OpenFlow brings several critical advantages to network management and control:

1. Flexibility and Programmability: With OpenFlow, network administrators can dynamically reconfigure the behavior of network devices, allowing for greater adaptability to changing network requirements.

2. Centralized Control: By centralizing control in a single controller, network administrators gain a holistic view of the network, simplifying management and troubleshooting processes.

3. Innovation and Experimentation: OpenFlow enables researchers and developers to experiment with new network protocols and applications, fostering innovation in the networking industry.

4. Scalability: OpenFlow’s centralized control architecture provides the scalability needed to manage large-scale networks efficiently.

**Implications for Network Control**

OpenFlow has significant implications for network control, paving the way for new possibilities in network management:

1. Software-Defined Networking (SDN): OpenFlow is a critical component of the broader concept of SDN, which aims to decouple network control from the underlying hardware, providing a more flexible and programmable infrastructure.

2. Network Virtualization: OpenFlow facilitates network virtualization, allowing multiple virtual networks to coexist on a single physical infrastructure.

3. Traffic Engineering: By controlling the flow of packets at a granular level, OpenFlow enables advanced traffic engineering techniques, optimizing network performance and resource utilization.

OpenFlow represents a paradigm shift in network control, offering a more flexible, scalable, and programmable approach to managing networks. By separating the control and data planes, OpenFlow empowers network administrators to have fine-grained control over network behavior, improving efficiency, innovation, and adaptability. As the networking industry continues to evolve, OpenFlow and its related technologies will undoubtedly play a crucial role in shaping the future of network management.

Summary: What is OpenFlow?

In the rapidly evolving world of networking, OpenFlow has emerged as a game-changer. This revolutionary technology has transformed the way networks are managed, offering unprecedented flexibility, control, and efficiency. In this blog post, we will delve into the depths of OpenFlow, exploring its definition, key features, and benefits.

What is OpenFlow?

OpenFlow can be best described as an open standard communications protocol that enables the separation of the control plane and the data plane in network devices. It allows centralized control over a network’s forwarding elements, making it possible to program and manage network traffic dynamically. By decoupling the intelligence of the network from the underlying hardware, OpenFlow provides a flexible and programmable infrastructure for network administrators.

Key Features of OpenFlow

a) Centralized Control: One of the core features of OpenFlow is its ability to centralize network control, allowing administrators to define and implement policies from a single point of control. This centralized control improves network visibility and simplifies management tasks.

b) Programmability: OpenFlow’s programmability empowers network administrators to define how network traffic should be handled based on their specific requirements. Through the use of flow tables and match-action rules, administrators can dynamically control the behavior of network switches and routers.

c) Software-Defined Networking (SDN) Integration: OpenFlow plays a crucial role in the broader concept of Software-Defined Networking. It provides a standardized interface for SDN controllers to communicate with network devices, enabling dynamic and automated network provisioning.

Benefits of OpenFlow

a) Enhanced Network Flexibility: With OpenFlow, network administrators can easily adapt and customize their networks to suit evolving business needs. The ability to modify network behavior on the fly allows for efficient resource allocation and improved network performance.

b) Simplified Network Management: By centralizing network control, OpenFlow simplifies the management of complex network architectures. Policies and configurations can be applied uniformly across the network, reducing administrative overhead and minimizing the chances of configuration errors.

c) Innovation and Experimentation: OpenFlow fosters innovation by providing a platform for the development and deployment of new network protocols and applications. Researchers and developers can experiment with novel networking concepts, paving the way for future advancements in the field.

Conclusion

OpenFlow has ushered in a new era of network management, offering unparalleled flexibility and control. Its ability to separate the control plane from the data plane, coupled with centralized control and programmability, has opened up endless possibilities in network architecture design. As organizations strive for more agile and efficient networks, embracing OpenFlow and its associated technologies will undoubtedly be a wise choice.

What is VXLAN

August 28, 2014

by Matt Conran Blog

Network Insight · Data Centre Networking

What Is VXLAN? Understanding Modern Network Overlays

Modern data centres require networks that can scale beyond traditional VLAN boundaries, support virtualised workloads and deliver predictable connectivity across routed infrastructure. Virtual Extensible LAN (VXLAN) addresses these requirements by extending Layer 2 Ethernet segments across a Layer 3 IP network.

VXLAN separates the logical network from the physical transport infrastructure. Instead of relying on a large switched Layer 2 domain, VXLAN encapsulates Ethernet frames inside UDP/IP packets and transports them between VXLAN Tunnel Endpoints (VTEPs). This enables virtual networks to span multiple physical switches while the underlying fabric continues to use IP routing.

Each VXLAN segment is identified by a 24-bit VXLAN Network Identifier (VNI) . Compared with traditional 12-bit VLAN identifiers, VXLAN provides a substantially larger segmentation space, making it particularly useful for multi-tenant cloud platforms, software-defined data centres and large-scale virtualised environments.

24-bit VXLAN Network Identifier used to distinguish logical overlay segments

~16.7M Potential VNI values compared with approximately 4,094 usable VLAN IDs

UDP 4789 Standard VXLAN destination port for Ethernet encapsulation over IP

Architecture Preview · Layer 2 over Layer 3

Leaf A

VTEP 10.0.0.11

VLAN 10 → VNI 10010

VXLAN Overlay

UDP/IP Encapsulation

Routed IP Underlay

Leaf B

VTEP 10.0.0.12

VLAN 10 → VNI 10010

Two VTEPs extend the same logical Layer 2 segment across a routed IP fabric. Intermediate routers forward the outer IP packet without inspecting the inner Ethernet frame.

Why VXLAN Matters in Modern Data Centres

Traditional Ethernet networks face practical limitations when stretched across large data centre environments. VLAN identifiers provide a limited segmentation space, large broadcast domains increase operational complexity, and Spanning Tree Protocol can prevent redundant Layer 2 links from forwarding simultaneously.

VXLAN enables a different architectural model. A routed leaf-spine underlay can use Equal-Cost Multi-Path (ECMP) forwarding across multiple active paths, while the VXLAN overlay provides logical Layer 2 connectivity independently of the physical topology. When combined with BGP Ethernet VPN (EVPN), the fabric can distribute MAC and IP reachability information through a control plane, reducing reliance on data-plane flooding.

What You Will Learn

✓ Understand why VXLAN was introduced and how it addresses traditional VLAN scaling limitations.

✓ Explore the relationship between the IP underlay, VXLAN overlay, VTEPs and VNIs.

✓ Follow Ethernet frames through VXLAN encapsulation, routed forwarding and decapsulation.

✓ Compare flood-and-learn forwarding with BGP EVPN MAC and IP reachability distribution.

✓ Investigate tenant segmentation, inter-VNI routing and distributed gateway design.

✓ Diagnose common VXLAN failures using VTEP reachability, VNI mappings, EVPN state and telemetry.

From Architecture to Engineering Practice

This guide combines technical explanations with interactive network engineering exercises. You will explore VXLAN topology behaviour, examine encapsulated packets, compare forwarding mechanisms and investigate realistic overlay and underlay failures. The objective is not simply to understand what VXLAN is, but to understand how it behaves in a production data centre fabric.

Section 01 · VXLAN Fundamentals

Why VXLAN? Understanding the Limitations of Traditional Layer 2 Networks

Modern data centre networks must support thousands of virtual machines, multiple tenants, dynamic workload placement and high-bandwidth east-west traffic. Traditional Ethernet switching was not originally designed for this degree of scale, segmentation and mobility.

Conventional Layer 2 architectures rely on VLANs, MAC address learning and loop-prevention mechanisms such as Spanning Tree Protocol (STP). These technologies remain useful, but extending large Layer 2 domains across a data centre can introduce operational complexity, inefficient use of redundant links and limitations in the number of logical segments.

VXLAN was introduced to separate logical Layer 2 connectivity from the physical network topology. By encapsulating Ethernet frames inside UDP/IP packets, VXLAN allows a routed IP fabric to transport multiple isolated virtual networks without requiring the entire physical infrastructure to participate in the same Layer 2 broadcast domain.

VLAN Scaling Spanning Tree MAC Learning Server Virtualization Leaf-Spine Fabrics Network Overlays

1.1 — The Traditional Layer 2 Data Centre

In a traditional switched network, Ethernet frames are forwarded using destination MAC addresses. Switches learn source MAC addresses from incoming frames and maintain forwarding tables that associate MAC addresses with physical interfaces.

VLANs provide logical separation within this infrastructure. Each VLAN represents an independent Layer 2 broadcast domain, allowing different applications, departments or tenants to share physical switching hardware while maintaining separate Ethernet segments.

The challenge emerges when these networks grow. More servers create more MAC addresses, more applications require additional VLANs, and redundant physical links require careful loop prevention. Workload mobility can also require Layer 2 connectivity across multiple racks or physical locations.

Limited Segmentation Space

Traditional IEEE 802.1Q VLAN tags use a 12-bit VLAN identifier, providing 4,096 possible values, of which approximately 4,094 are available for normal VLAN use.

Spanning Tree Constraints

STP prevents Ethernet forwarding loops by selecting a loop-free active topology. Some redundant links may be blocked, preventing simultaneous forwarding.

MAC Address Table Growth

Virtualized servers can introduce many MAC addresses per physical switch port. Large Layer 2 domains increase forwarding table requirements and operational scale.

Workload Mobility

Moving workloads between physical hosts may require preserving network connectivity, addressing and policy across different switching locations.

Engineering Principle

The problem is not that Ethernet switching or VLANs are inherently unsuitable. The challenge is extending large numbers of logical Layer 2 segments across a growing physical network while preserving scalability, resilience and operational simplicity.

1.2 — VLAN Identifier Exhaustion

IEEE 802.1Q uses a 12-bit VLAN Identifier (VID). This provides 2¹², or 4,096, possible identifier values. VLAN IDs 0 and 4095 are reserved, leaving 4,094 usable VLAN identifiers in the normal VLAN namespace.

For a small enterprise, this may be sufficient. However, large cloud providers and multi-tenant data centres may require thousands of independent network segments for customers, applications, security zones and workload environments.

Logical Network Identifier Capacity

Traditional VLAN — 12 bits 4,096 values

VXLAN VNI — 24 bits 16,777,216 values

Visual comparison uses a compressed scale so both identifier spaces remain visible. VXLAN provides 4,096 times the theoretical identifier space of a 12-bit VLAN field. Actual supported VNI scale depends on platform hardware and software limits.

VXLAN introduces a 24-bit VXLAN Network Identifier (VNI). This significantly expands the available logical segment namespace. A VNI identifies an overlay segment, allowing different physical VLANs to map into the same logical VXLAN segment when required.

For example, VLAN 10 on one leaf switch and VLAN 20 on another leaf switch can both map to VNI 10010. The local VLAN numbers do not need to match, provided the VTEPs are configured to associate those VLANs with the same intended overlay bridge domain.

Engineering Example · Multi-Tenant Cloud

Consider a cloud provider operating infrastructure for 500 customers. Each customer requires eight isolated application networks for development, production, management, databases and supporting services.

500 Customers

8 Segments per Customer

4,000 Required Segments

This deployment already consumes almost the entire traditional VLAN identifier space if every segment requires a unique VLAN ID in one shared administrative namespace.

Adding more customers, environments or application tiers creates further demand. VXLAN allows the provider to allocate distinct VNIs without relying on a single global 12-bit VLAN namespace.

Important Design Distinction

VLAN IDs can be reused in separate Layer 2 domains, so the 4,094-VLAN limit is not a universal limit on the number of tenants in an entire network. VXLAN is valuable because it provides a much larger overlay namespace and decouples logical segmentation from local physical VLAN assignments.

1.3 — Spanning Tree and Blocked Links

Ethernet networks must prevent forwarding loops. Unlike IP packets, ordinary Ethernet frames do not contain a hop-limit field that automatically expires as frames traverse switches.

A Layer 2 forwarding loop can cause broadcast storms, repeated frame replication, unstable MAC learning and severe network disruption. Spanning Tree Protocol prevents these conditions by selecting a loop-free forwarding topology.

Although this provides stability, a conventional spanning-tree topology may block redundant links. Consequently, installed physical bandwidth is not always available for active forwarding. Convergence following topology changes can also affect application connectivity.

Traditional Layer 2 Design

Redundant Ethernet paths require loop prevention and may include blocked links.

Distribution A

Distribution B

Access A

Access B

Active forwarding path
Illustrative blocked redundant path

Routed Leaf-Spine Fabric

IP routing enables multiple equal-cost paths to forward traffic concurrently.

Spine 1

Spine 2

Leaf A

Leaf B

Routed links can remain active
ECMP distributes eligible flows across available next hops.

Modern leaf-spine architectures commonly replace the large switched core with a routed IP underlay. Each leaf connects to multiple spine switches, and routing protocols maintain reachability across the fabric.

Equal-Cost Multi-Path (ECMP) forwarding allows traffic to use multiple equal-cost routes. Rather than relying on a spanning tree to disable redundant Layer 2 paths, the fabric uses Layer 3 routing and its associated convergence mechanisms.

VXLAN complements this architecture by transporting logical Layer 2 segments over the routed underlay. The physical fabric can therefore remain Layer 3 while connected workloads continue to experience the Layer 2 adjacency required by their applications.

Architectural Advantage

VXLAN does not itself perform ECMP routing. ECMP is provided by the IP underlay. VXLAN enables Layer 2 overlay connectivity to use that routed infrastructure. The VXLAN UDP source port can provide entropy for flow-based ECMP hashing.

1.4 — Server Virtualization and MAC Scale

Server virtualization changes the relationship between physical switch ports and network endpoints. A single physical server can host many virtual machines, each with its own virtual network interface and MAC address.

In a conventional Ethernet design, switches learn the MAC addresses of communicating endpoints and store them in forwarding tables. As virtual machine populations increase, the number of learned addresses can grow substantially.

Large Layer 2 domains also require mechanisms for locating unknown destinations. When a destination MAC address is not known, unknown-unicast frames may be flooded within the relevant broadcast domain.

Scaling Factor	Traditional Layer 2 Impact	VXLAN Fabric Approach
More virtual machines	Increasing MAC address learning and forwarding-table requirements	Logical segments distributed across VTEPs; EVPN can advertise remote MAC reachability
More application networks	Greater demand for VLAN identifiers and trunk configuration	24-bit VNI namespace supports many overlay segments
Workload relocation	MAC location changes and possible VLAN extension requirements	Remote VTEP mappings can be updated through the overlay control plane
Unknown destinations	Flooding within the Layer 2 broadcast domain	BUM replication is still required in some cases; EVPN can reduce unnecessary flooding
Redundant fabric links	STP may block selected Layer 2 paths	Routed underlay supports ECMP across active IP paths

VXLAN does not automatically eliminate MAC table requirements. VTEPs still need forwarding information for local and remote endpoints. However, the overlay architecture allows logical segments to be distributed across a routed fabric.

When combined with BGP EVPN, MAC and IP reachability information can be exchanged through a control plane. This provides more structured endpoint discovery, supports MAC mobility and can reduce reliance on traditional flood-and-learn forwarding.

1.5 — Why Workload Mobility Matters

Virtualization allows workloads to move between physical hosts for maintenance, resource optimisation, resilience and operational flexibility.

In some deployment models, a workload must preserve its IP address and Layer 2 connectivity when it moves. Traditional approaches may require VLANs to be extended across multiple racks or switching domains.

VXLAN enables the same logical overlay segment to be present on different physical leaf switches. A workload attached to VNI 10010 on Leaf A can communicate with another endpoint attached to VNI 10010 on Leaf B, even though the underlying network forwards traffic using IP routing.

In EVPN-based fabrics, control-plane updates can advertise changes in endpoint location. Remote VTEPs can then update their forwarding state to direct traffic toward the appropriate destination VTEP.

Design Consideration

VXLAN provides the network mechanism for extending logical segments, but successful workload mobility also depends on application requirements, gateway design, security policy, address management and orchestration. Extending Layer 2 connectivity across distant sites is not always the preferred design.

1.6 — The Transition to Routed Overlay Fabrics

The evolution from traditional Layer 2 networks to VXLAN-based fabrics represents an important architectural shift. Instead of requiring the physical network to provide every logical Ethernet segment, the network separates two responsibilities:

The Underlay Network

Provides IP connectivity between VTEPs. Routing protocols maintain reachability, ECMP enables path diversity, and the physical topology can scale independently of tenant segmentation.

The VXLAN Overlay

Provides logical Ethernet segments identified by VNIs. VTEPs encapsulate and decapsulate traffic, while mechanisms such as BGP EVPN can distribute endpoint reachability.

This separation enables data centre architects to design a resilient routed fabric while delivering logical connectivity to workloads independently of their physical attachment points.

It also introduces new engineering responsibilities. The underlay must provide reliable VTEP reachability, the overlay must maintain correct endpoint mappings, and the fabric must account for encapsulation overhead, MTU, segmentation and control-plane convergence.

Design Dimension	Traditional Approach	VXLAN-Based Fabric
Network segmentation	12-bit VLAN identifiers	24-bit VXLAN VNIs
Core transport	Often extended Layer 2 switching	Routed IP underlay
Path redundancy	STP or other L2 redundancy designs	IP routing and ECMP
Endpoint learning	Data-plane MAC learning	Flood-and-learn or EVPN control-plane distribution
Workload mobility	May require extended VLAN domains	Logical overlay segment extension
Operational visibility	Switch ports, VLANs and MAC tables	Underlay routes, VTEPs, VNIs, EVPN state and packet encapsulation

Key Engineering Insight

VXLAN does not replace Ethernet, VLANs or IP routing. It combines them into an overlay architecture in which Ethernet segments are mapped to VNIs and transported across an IP network. Local VLANs still commonly connect endpoints to VTEPs, while the routed underlay transports encapsulated traffic.

1.7 — Design Trade-Offs and Operational Reality

VXLAN solves important scaling and architectural challenges, but it is not a universal replacement for traditional Layer 2 networking.

Overlay fabrics introduce additional headers, forwarding state and control-plane dependencies. Troubleshooting must distinguish between failures in the physical IP underlay and failures in the logical VXLAN overlay.

Engineers must also consider hardware forwarding capacity, VNI scale, EVPN route scale, broadcast replication, security policy, operational tooling and the ability to observe encapsulated traffic across the network.

VXLAN Advantage	Engineering Responsibility
Large logical segment space	VNI allocation, lifecycle management and tenant isolation
Routed underlay transport	IP reachability, routing convergence and ECMP path health
Flexible workload placement	MAC mobility, gateway consistency and policy enforcement
Overlay encapsulation	MTU planning, packet visibility and forwarding performance
EVPN control-plane learning	BGP session health, route import/export and endpoint state consistency

VXLAN also provides segmentation rather than encryption. Security still requires appropriate access controls, isolation policies and protection of the transport infrastructure.

Section 1 — Engineering Takeaways

✓ Traditional VLANs use a 12-bit identifier, while VXLAN introduces a 24-bit VNI for a much larger logical segment space.

✓ Large Layer 2 domains can create challenges involving STP, MAC learning, flooding and workload mobility.

✓ Routed leaf-spine fabrics use IP routing and ECMP to provide active path diversity without relying on STP in the physical fabric.

✓ VXLAN transports logical Ethernet segments across the IP underlay, separating tenant connectivity from physical topology.

✓ VXLAN does not eliminate the need for MAC learning, MTU planning, security or careful control-plane design.

Continue Learning · Section 02

VXLAN Architecture: Underlay, Overlay and VTEPs

Network Insight · Engineering Guide

Network Insight · Interactive Engineering Lab 01

Why VXLAN? Explore the Limits of Layer 2

Compare a traditional VLAN-based network with a routed VXLAN overlay. Select an engineering challenge, change the architecture and run a packet journey to see what changes.

Step 1 · Select Architecture

Engineering Focus

Step 2 · Select an Engineering Scenario

Live Network Architecture Traditional Layer 2

Active physical path STP-blocked alternative VXLAN logical tunnel

Ready to investigate

Engineering Analysis

VLAN Scaling in a Traditional Network

Traditional VLAN identifiers use a 12-bit field, providing 4094 normally usable VLAN IDs. Extending large numbers of VLANs across a physical switching fabric increases operational complexity and Layer 2 scope.

Architecture Indicators

Segment Identifier 12-bit

Segment ID Space 4,094

Core Forwarding Layer 2

Path Model STP

Engineering Event Timeline

Select an architecture and scenario. Run the packet journey to inspect forwarding events.

Key Engineering Principle

VXLAN separates logical Layer 2 segments from the physical transport network, enabling Ethernet overlays to operate across a routed IP fabric.

Section 02 · VXLAN Architecture

VXLAN Architecture: Understanding the Underlay, Overlay and VTEPs

VXLAN architecture separates the physical network responsible for IP packet transport from the logical network that provides Ethernet connectivity to applications. This separation is the foundation of modern routed data centre fabrics.

In a VXLAN environment, the underlay network provides routed IP connectivity between tunnel endpoints, while the overlay network represents the virtual Layer 2 segments transported across that infrastructure. VXLAN Tunnel Endpoints (VTEPs) connect these two architectural layers.

Understanding the relationship between the underlay, overlay, VTEPs and VXLAN Network Identifiers (VNIs) is essential for designing, operating and troubleshooting VXLAN fabrics. A failure in any of these components can interrupt workload connectivity even when other parts of the network appear healthy.

IP Underlay VXLAN Overlay VTEP VNI Leaf-Spine ECMP BGP EVPN

2.1 — The Three Architectural Layers

A VXLAN fabric can be understood through three primary architectural components: the physical IP underlay, the logical VXLAN overlay and the VTEPs that perform encapsulation and decapsulation.

TRANSPORT

IP Underlay Network

The routed physical network connecting leaf and spine switches. It provides reachability between VTEP IP addresses and forwards outer IP packets using normal Layer 3 routing.

VIRTUAL NETWORK

VXLAN Overlay Network

Logical Ethernet segments identified by VNIs. The overlay allows workloads connected to different physical leaf switches to participate in the same virtual Layer 2 network.

ENCAPSULATION

VXLAN Tunnel Endpoint

A VTEP encapsulates local Ethernet frames into VXLAN UDP/IP packets and decapsulates incoming VXLAN traffic before forwarding the inner frames toward connected endpoints.

SEGMENT IDENTIFIER

VXLAN Network Identifier

A 24-bit VNI identifies the logical overlay segment associated with encapsulated traffic. VNIs allow many isolated virtual networks to share the same routed fabric.

Architectural Principle

The underlay forwards packets using outer IP addresses. The overlay identifies the logical Ethernet segment using the VNI. The VTEP connects both functions by adding or removing the VXLAN encapsulation.

2.2 — VXLAN Leaf-Spine Fabric Architecture

Modern VXLAN deployments commonly use a leaf-spine architecture. Each leaf switch connects to multiple spine switches, creating a routed topology with predictable path diversity and multiple equal-cost forwarding options.

Leaf switches frequently act as VTEPs, connecting local servers or virtual machines to the overlay. Spine switches provide IP transport between leaf VTEPs and do not normally need to inspect the VXLAN headers or tenant Ethernet frames.

Reference Architecture · Two-Spine / Three-Leaf Fabric

Routed IP underlay Logical VXLAN overlay

The dashed VXLAN path represents a logical overlay relationship between VTEPs. Actual encapsulated packets traverse the routed spine-leaf underlay. Leaf C may participate in other VNIs or the same VNI if configured.

The leaf-spine topology is particularly suitable for VXLAN because it provides consistent routed connectivity between VTEPs and can distribute traffic across multiple available paths.

The physical network does not need to extend tenant VLANs through the spines. Instead, each leaf encapsulates local traffic using VXLAN and sends the resulting outer IP packet toward the remote VTEP.

2.3 — The IP Underlay Network

The underlay is responsible for transporting encapsulated VXLAN packets between VTEPs. It must provide reliable IP connectivity between the VTEP source and destination addresses.

In many deployments, each VTEP uses a loopback interface as its VXLAN source address. The underlay routing system advertises reachability to these loopback addresses so that remote VTEPs can forward encapsulated traffic correctly.

Common underlay routing approaches include OSPF, IS-IS and eBGP. The appropriate protocol depends on the fabric architecture, operational requirements and platform capabilities.

Underlay Component	Engineering Role	Operational Check
Physical links	Provide connectivity between leaf and spine switches	Interface status, errors, speed and utilization
Routing protocol	Advertises reachability to VTEP addresses	OSPF, IS-IS or BGP neighbor and route state
VTEP loopback	Stable source or destination address for VXLAN packets	Route lookup and loopback reachability
ECMP	Distributes eligible traffic across equal-cost paths	Installed next hops and flow distribution
MTU	Accommodates additional VXLAN encapsulation overhead	End-to-end packet size and fragmentation behavior

Underlay Failure Scenario

If Leaf A cannot route traffic to Leaf B's VTEP address, the VXLAN overlay cannot deliver encapsulated packets between them. The VNI configuration may be correct, but the data plane will still fail because the underlay lacks usable IP reachability.

2.4 — The VXLAN Overlay Network

The overlay represents the logical networks carried across the IP underlay. Each Layer 2 VXLAN segment is identified by a VNI, and participating VTEPs associate local attachment circuits or VLANs with that identifier.

Consider two servers connected to different leaf switches. Both servers belong to the same logical network, represented by VNI 10010. The servers can exchange Ethernet frames even though the physical switches between them operate as Layer 3 routers.

VXLAN encapsulation allows the original Ethernet frame to remain intact as it crosses the underlay. The receiving VTEP removes the outer encapsulation and forwards the inner frame within the appropriate local bridge domain.

LEAF A

Local VLAN 10

Host A connects to VLAN 10. Leaf A maps this VLAN to VNI 10010 and encapsulates traffic destined for remote endpoints in that overlay segment.

LEAF B

Local VLAN 20

Host B connects to VLAN 20. Leaf B maps VLAN 20 to the same VNI 10010, placing both endpoints in the intended logical Layer 2 segment.

VLAN-to-VNI Mapping

Local VLAN identifiers do not have to match across VTEPs. The VNI identifies the shared overlay bridge domain. However, the mappings must be consistent with the intended tenant segmentation and control-plane configuration.

2.5 — The Role of the VTEP

A VXLAN Tunnel Endpoint is the device or software component responsible for converting Ethernet traffic into VXLAN packets and back again.

VTEPs may exist in physical switches, virtual switches, hypervisors or other supported network devices. In a common hardware leaf-spine design, the leaf switches act as VTEPs.

VTEP Function	Description
Encapsulation	Adds VXLAN, UDP, IP and outer Ethernet headers to an original Ethernet frame
Decapsulation	Removes the outer headers and restores the original Ethernet frame for local forwarding
VNI mapping	Associates local bridge domains or VLANs with VXLAN segment identifiers
Remote endpoint forwarding	Determines the destination VTEP for known remote MAC addresses
BUM replication	Handles broadcast, unknown unicast and multicast traffic using the configured replication mechanism
EVPN integration	Uses advertised MAC/IP reachability information when BGP EVPN is deployed

A VTEP needs to determine both the logical segment associated with a frame and the remote VTEP that should receive it. The method used to discover remote endpoints depends on whether the fabric uses flood-and-learn or a control plane such as BGP EVPN.

2.6 — Example VTEP Configuration

The following simplified Cisco NX-OS-style example illustrates the relationship between a local VLAN, VNI and NVE interface. The example is conceptual and may require additional platform- specific commands, feature enablement and control-plane configuration.

Leaf A · Illustrative VXLAN Configuration NX-OS Style

feature nv overlay

vlan 10
  vn-segment 10010

interface loopback1
  ip address 10.0.0.11/32

interface nve1
  no shutdown
  host-reachability protocol bgp
  source-interface loopback1
  member vni 10010
    ingress-replication protocol bgp

Illustrative syntax only. A functional EVPN fabric also requires underlay reachability, BGP EVPN configuration, appropriate route targets and other platform-dependent settings.

In this example, VLAN 10 maps to VNI 10010. Loopback1 provides the local VTEP source address, and interface NVE1 represents the VXLAN tunnel interface.

The BGP EVPN control plane can advertise reachability information and support ingress replication membership. However, the exact configuration model varies between hardware platforms, operating systems and software versions.

2.7 — Packet Forwarding Across the Fabric

When a local endpoint sends an Ethernet frame to a remote endpoint in the same VNI, the ingress VTEP encapsulates the frame and forwards it across the IP underlay.

Host A

Sends an ordinary Ethernet frame

Leaf A

Maps VLAN to VNI and encapsulates

Underlay

Routes outer IP packet toward VTEP

Leaf B

Decapsulates and selects bridge domain

Host B

Receives original Ethernet frame

The original source and destination MAC addresses remain part of the inner Ethernet frame. The outer IP header identifies the source and destination VTEPs.

Intermediate spine routers forward the packet based on the outer destination IP address. They do not need to learn tenant MAC addresses or participate in the tenant Layer 2 bridge domain.

Packet Forwarding Insight

The VXLAN tunnel is a logical relationship between VTEPs. It is not necessarily a dedicated physical path. Encapsulated packets may traverse different equal-cost underlay routes depending on the ECMP hashing algorithm and network state.

2.8 — ECMP and Underlay Path Diversity

One of the major advantages of a routed leaf-spine underlay is the ability to use multiple equal-cost paths simultaneously.

VXLAN uses UDP encapsulation, allowing the underlay to distinguish flows through outer header fields. Many implementations derive the UDP source port from a hash of inner packet information to provide entropy for ECMP.

This enables different encapsulated flows to be distributed across available underlay next hops. However, ECMP is commonly flow-based, so a single flow may continue to use one selected path rather than being distributed packet by packet.

NORMAL OPERATION

Multiple Active Paths

Leaf A can reach Leaf B through Spine 1 or Spine 2. The underlay may install both equal-cost next hops and distribute eligible VXLAN flows across them.

LINK FAILURE

Underlay Reconvergence

If one path fails, the routing system removes the unavailable next hop. Traffic can continue over remaining paths, subject to detection, convergence and forwarding updates.

2.9 — Where BGP EVPN Fits

VXLAN defines a data-plane encapsulation mechanism. It does not, by itself, define a complete mechanism for distributing remote endpoint reachability throughout the fabric.

Early VXLAN deployments commonly relied on data-plane learning and multicast-based replication. Modern data centre fabrics frequently combine VXLAN with BGP Ethernet VPN (EVPN) to provide a scalable control-plane mechanism.

EVPN can advertise MAC/IP reachability, identify participating VTEPs and support endpoint mobility. It also enables features such as distributed gateway architectures and inter-subnet routing.

Component	Responsibility
VXLAN	Encapsulates Ethernet traffic inside UDP/IP packets
IP Underlay	Provides routed reachability between VTEP addresses
BGP EVPN	Distributes endpoint reachability and overlay control-plane information
VTEP	Performs encapsulation, decapsulation and overlay forwarding
VNI	Identifies the logical overlay segment or routing context

VXLAN and EVPN Are Complementary

VXLAN is the data-plane encapsulation technology. BGP EVPN is a control-plane technology used to distribute overlay reachability information. They are commonly deployed together, but they perform different architectural roles.

2.10 — VXLAN Architecture Design Requirements

A successful VXLAN fabric depends on more than enabling an NVE interface and assigning a VNI. The underlay and overlay must be designed as related but independently observable systems.

Requirement	Why It Matters
VTEP IP reachability	Remote tunnel endpoints must be reachable through the routed underlay
Consistent VNI mappings	Logical segments must map to the intended bridge domains
Appropriate MTU	Encapsulation adds header overhead beyond the original frame size
Control-plane stability	EVPN routes and endpoint mappings must remain accurate during changes
Resilient underlay	Routing and ECMP must recover from physical path failures
Tenant isolation	VNIs, VRFs and security policy must prevent unintended connectivity
Operational visibility	Engineers need to inspect routing, NVE peers, VNIs, EVPN routes and forwarding state

Troubleshooting should begin by determining whether the fault exists in the underlay, overlay or endpoint attachment configuration.

For example, a missing VTEP route indicates an underlay reachability problem, while a VNI mismatch or missing EVPN route may indicate an overlay configuration or control-plane problem.

Operational Design Warning

A healthy BGP EVPN session does not guarantee successful VXLAN forwarding. The fabric must also have correct VNI membership, valid remote endpoint mappings, usable VTEP routes, sufficient MTU and appropriate forwarding policies.

Section 2 — Engineering Takeaways

✓ The IP underlay provides routed transport between VXLAN tunnel endpoints.

✓ The VXLAN overlay provides logical Ethernet segments identified by 24-bit VNIs.

✓ VTEPs encapsulate and decapsulate Ethernet frames as they enter and leave the routed fabric.

✓ ECMP enables eligible encapsulated flows to use multiple active underlay paths.

✓ BGP EVPN complements VXLAN by distributing endpoint and overlay reachability information.

✓ VXLAN troubleshooting requires visibility into both the routed underlay and the logical overlay.

Continue Learning · Section 03

VXLAN Encapsulation and Packet Forwarding

Network Insight · Interactive Engineering Lab 02

VXLAN Architecture Explorer: Underlay, Overlay and VTEPs

Investigate a two-leaf, two-spine VXLAN fabric. Select individual devices to inspect their roles, switch between physical and logical network views, and trace the journey of an Ethernet frame across the IP underlay.

Step 1 · Select Architecture View

Interactive Leaf-Spine Architecture IP Underlay

Routed IP links Logical VXLAN tunnel Select any device to inspect it

Selected Network Component

Leaf A · VXLAN Tunnel Endpoint

Leaf A is a VTEP. It receives Ethernet frames from Host A, maps the local VLAN to a VNI and encapsulates frames for remote VTEPs.

Component Details

Role

VTEP

Address

10.0.0.11

Layer

Overlay Edge

Function

Encapsulation

Select a device or run a trace

Packet Forwarding Timeline

Select a topology component to inspect it. Run the trace to see encapsulation and forwarding.

Key Architecture Principle

The underlay transports IP packets between VTEPs. The VXLAN overlay provides logical Ethernet connectivity between workloads. The spines forward on outer IP headers and do not normally need to inspect the inner Ethernet frame.

Conceptual VXLAN architecture lab. The example illustrates same-subnet forwarding over one selected spine path. It does not emulate BGP EVPN, routing convergence or ECMP hashing.

Section 03 · VXLAN Data Plane

VXLAN Encapsulation and Packet Forwarding

VXLAN enables an Ethernet frame to cross a routed IP network without requiring intermediate routers to participate in the original Layer 2 broadcast domain. It achieves this by encapsulating the Ethernet frame inside a VXLAN header, UDP datagram and outer IP packet.

The original Ethernet frame remains the inner frame, while the newly added transport headers identify the source and destination VXLAN Tunnel Endpoints (VTEPs). This separation allows the physical network to forward traffic using ordinary IP routing while the overlay preserves logical Ethernet connectivity between workloads.

Understanding the exact encapsulation format is essential for troubleshooting VXLAN fabrics. Engineers must distinguish the inner and outer addressing information, verify the correct VXLAN Network Identifier (VNI), account for encapsulation overhead and confirm that the receiving VTEP can decapsulate and forward the original frame.

Ethernet Frame VXLAN Header UDP 4789 VNI VTEP MTU ECMP Hashing

3.1 — What Happens During VXLAN Encapsulation?

When Host A sends an Ethernet frame toward Host B in the same logical VXLAN segment, the ingress VTEP first determines the bridge domain and VNI associated with the incoming frame.

If the destination MAC address is reachable through a remote VTEP, the ingress device encapsulates the original Ethernet frame with a VXLAN header and additional UDP/IP transport headers.

The resulting packet is forwarded through the routed underlay toward the destination VTEP. Intermediate routers inspect the outer IP destination address and do not need to understand the tenant's inner Ethernet addressing.

INNER NETWORK

Original Ethernet Frame

Contains the original source and destination MAC addresses, optional VLAN information and the payload generated by the communicating hosts.

OUTER NETWORK

VXLAN Transport Headers

Include the VXLAN VNI, UDP header, outer source and destination IP addresses and the Ethernet header used on the current physical link.

Fundamental Data-Plane Principle

The inner Ethernet frame identifies the communicating endpoints. The outer IP header identifies the VTEPs transporting that frame. The VNI identifies the logical overlay segment in which the frame belongs.

3.2 — The Complete VXLAN Packet Structure

A conventional VXLAN packet consists of an original Ethernet frame encapsulated inside an 8-byte VXLAN header, an 8-byte UDP header and an outer IP packet. An outer Ethernet header is also used when the encapsulated packet is transmitted across an Ethernet underlay link.

Packet Anatomy · Ethernet over VXLAN/UDP/IPv4

Outer Ethernet

Physical next-hop MAC addresses · 14 bytes without an 802.1Q tag

Outer IPv4 Header

Source VTEP: 10.0.0.11 · Destination VTEP: 10.0.0.12 · 20 bytes without IP options

UDP Header

Variable source port · Destination port 4789 · 8 bytes

VXLAN Header

VNI 10010 8-byte header identifying the logical overlay segment

Inner Ethernet

Original source and destination host MAC addresses · 14 bytes without an inner VLAN tag

Inner Payload

Original IPv4/IPv6 packet, ARP message or other Ethernet payload

Packet layers are shown from the outer Ethernet header toward the encapsulated payload. The Ethernet FCS, optional VLAN tags and additional IP options are not included in this simplified diagram.

The outer Ethernet header changes as the encapsulated packet moves between physical Layer 3 hops. By contrast, the outer source and destination VTEP IP addresses normally remain constant across the underlay path.

The inner Ethernet frame is preserved through the VXLAN tunnel, allowing the destination VTEP to deliver it into the corresponding local bridge domain after decapsulation.

3.3 — Understanding the 8-Byte VXLAN Header

The standard VXLAN header defined by RFC 7348 is eight bytes long. Its most important field is the 24-bit VXLAN Network Identifier, which identifies the logical network segment associated with the encapsulated frame.

The header also contains flags and reserved fields. The I flag indicates that the VNI field is valid. In the original VXLAN format, the remaining reserved bits are transmitted as zero.

VXLAN Header · 8 Bytes / 64 Bits

Byte 0 Flags

Byte 1 RSVD

Byte 2 RSVD

Byte 3 RSVD

Byte 4 VNI

Byte 5 VNI

Byte 6 VNI

Byte 7 RSVD

The VNI occupies bytes 4–6. The I flag is bit 3 of the first byte, conventionally represented by the flag value 0x08. This diagram represents the original VXLAN header, not VXLAN-GPE.

Field	Size	Purpose
Flags	8 bits	Includes the I flag indicating that the VNI is valid
Reserved	24 bits	Reserved fields in the original VXLAN header
VNI	24 bits	Identifies the logical VXLAN segment
Reserved	8 bits	Reserved final byte

Why the VNI Matters

A VTEP uses the VNI to select the intended overlay bridge domain. If the receiving VTEP does not have the appropriate VNI configured, it cannot correctly deliver the frame into that logical segment.

3.4 — Inner and Outer Addressing

VXLAN packets contain two addressing contexts. The inner headers describe the communicating workloads, while the outer headers describe how the encapsulated packet travels through the physical IP fabric.

Address Field	Example	Function
Inner source MAC	00:11:22:33:44:01	Original Host A MAC
Inner destination MAC	00:11:22:33:44:02	Original Host B MAC
Inner source IP	192.168.10.11	Host A IP address
Inner destination IP	192.168.10.12	Host B IP address
Outer source IP	10.0.0.11	Leaf A VTEP address
Outer destination IP	10.0.0.12	Leaf B VTEP address
VXLAN VNI	10010	Overlay bridge domain

Spine routers use the outer destination IP address to forward the encapsulated packet. They do not need routes for the inner host IP addresses when simply providing underlay transport.

The receiving VTEP uses the VXLAN header and local forwarding information to identify the correct bridge domain and deliver the inner frame.

3.5 — End-to-End VXLAN Packet Journey

Consider Host A connected to Leaf A and Host B connected to Leaf B. Both hosts belong to VNI 10010, and the underlay provides IP connectivity between the two VTEPs.

Host A

Sends an Ethernet frame toward Host B

Leaf A

Selects VNI 10010 and encapsulates

Spine

Routes the outer IP packet

Leaf B

Removes VXLAN encapsulation

Host B

Receives the original frame

Step 1 — Original Ethernet Frame

Host A creates an Ethernet frame containing Host B's destination MAC address. The frame enters Leaf A through a local access interface associated with the intended bridge domain.

Step 2 — VNI and Remote VTEP Selection

Leaf A identifies the local bridge domain and associated VNI. Its forwarding information indicates that Host B's MAC address is reachable through Leaf B's VTEP.

This remote mapping may have been learned through flood-and-learn forwarding or advertised by BGP EVPN.

Step 3 — Encapsulation and IP Forwarding

Leaf A adds the VXLAN, UDP and outer IP headers. The outer destination becomes Leaf B's VTEP address, 10.0.0.12.

The underlay routing table determines the next hop. If multiple equal-cost paths exist, ECMP hashing may select one of them.

Step 4 — Decapsulation at Leaf B

Leaf B receives the VXLAN packet, validates the relevant encapsulation information and identifies VNI 10010.

It removes the outer headers and processes the original Ethernet frame in the associated local bridge domain.

Step 5 — Delivery to Host B

Leaf B forwards the recovered Ethernet frame toward Host B through the appropriate local attachment interface.

Forwarding Assumption

This example assumes that the destination MAC-to-VTEP mapping is already known. If the destination is unknown, the fabric may need to perform BUM replication or another endpoint discovery operation. These mechanisms are covered in Section 4.

3.6 — VXLAN UDP Ports and ECMP Entropy

Standard VXLAN uses UDP destination port 4789. The UDP source port is commonly derived from a hash of the inner packet fields.

This variable source port provides additional entropy for underlay ECMP hashing. Without suitable entropy, many VXLAN flows between the same pair of VTEPs could hash onto the same physical path.

UDP DESTINATION

Port 4789

The standard VXLAN destination port. Receiving VTEPs use the configured port to identify VXLAN traffic for processing.

UDP SOURCE

Flow-Based Entropy

A source port chosen using a hash of inner traffic fields can help distribute different flows across ECMP paths.

ECMP is typically flow-based rather than packet-based. A single flow may remain on one underlay path, while different flows can be distributed across other available paths.

Legacy VXLAN Port Configurations

Some older VXLAN implementations and lab environments used non-standard or configurable UDP destination ports. Your original VXLAN lab used port 1024 explicitly. Both VTEPs must use compatible port settings. For new standards-based deployments, UDP 4789 is the normal choice.

3.7 — VXLAN Encapsulation Overhead and MTU

Encapsulation increases packet size because additional headers are inserted around the original Ethernet frame.

For a typical VXLAN packet transported over IPv4, the additional headers consist of:

Engineering Calculation · VXLAN over IPv4

14 Inner Ethernet Header

8 VXLAN Header

8 UDP Header

20 Outer IPv4 Header

14 + 8 + 8 + 20 = 50 Bytes of Additional Encapsulated IP Payload

The 50-byte calculation assumes an untagged inner Ethernet frame, a minimum 20-byte outer IPv4 header and no optional extensions. It excludes the outer Ethernet header and FCS, which are link-layer overhead.

If the original host sends a 1,500-byte IP packet, the VXLAN encapsulation produces a 1,550-byte outer IPv4 packet when the inner Ethernet header is included.

Therefore, an underlay IP MTU of at least 1,550 bytes is required for this specific untagged IPv4 example without outer IP fragmentation.

Many data centre operators configure larger MTUs, often using jumbo-frame support, to provide additional headroom for encapsulation, VLAN tags and other transport requirements.

MTU Design Warning

VXLAN does not automatically require a particular jumbo MTU such as 9,000 bytes. The required underlay MTU depends on the inner packet size and encapsulation format. Insufficient MTU can cause fragmentation, packet drops or application-level failures.

3.8 — VXLAN Packet Capture Analysis

Packet captures are one of the most useful tools for understanding VXLAN forwarding. A capture taken on an underlay link reveals the outer Ethernet, IP, UDP and VXLAN headers.

In Wireshark, an engineer can inspect the outer source and destination VTEP addresses, UDP destination port, VNI and encapsulated Ethernet frame.

Illustrative Packet Capture · VXLAN over IPv4

Frame: VXLAN encapsulated ICMP

Outer Ethernet:
  Source MAC:      Underlay next-hop MAC
  Destination MAC: Underlay next-hop MAC

Outer IPv4:
  Source IP:       10.0.0.11
  Destination IP:  10.0.0.12
  Protocol:        UDP (17)

UDP:
  Source Port:     52341
  Destination:     4789

VXLAN:
  Flags:           0x08 (I flag set)
  VNI:             10010

Inner Ethernet:
  Source MAC:      00:11:22:33:44:01
  Destination MAC: 00:11:22:33:44:02

Inner IPv4:
  Source IP:       192.168.10.11
  Destination IP:  192.168.10.12
  Protocol:        ICMP

ICMP:
  Type:            Echo Request

Illustrative decoded packet fields, not an actual captured packet. MAC addresses and UDP source ports vary according to the physical link and implementation.

A useful first check is to confirm that the outer destination IP address matches the expected remote VTEP.

The next check is the VNI. A correct VTEP destination with an incorrect VNI can still result in failed forwarding.

Finally, the inner Ethernet and IP headers should match the expected workload traffic. This helps determine whether the issue exists before encapsulation, inside the overlay or after decapsulation.

3.9 — Common VXLAN Packet Forwarding Failures

Failure	Likely Symptom	Engineering Check
Incorrect VNI	Remote VTEP cannot deliver traffic into intended segment	Verify local VLAN-to-VNI mappings and NVE membership
Missing VTEP route	Encapsulated packets cannot reach remote VTEP	Check underlay routing and VTEP reachability
UDP port mismatch	Receiver may not recognise VXLAN packets	Verify configured VXLAN destination port
Insufficient MTU	Large packets fail while smaller packets work	Test end-to-end encapsulated packet sizes
Missing remote MAC mapping	Traffic is flooded or cannot be forwarded	Inspect MAC tables and EVPN reachability
Underlay path failure	Packet loss or temporary interruption	Verify routing convergence and ECMP next hops
Security policy drop	Encapsulated traffic is filtered in transit	Inspect ACLs, firewalls and UDP 4789 reachability

Troubleshooting Method

Follow the packet in order: original host frame, local bridge-domain mapping, remote MAC-to-VTEP resolution, VXLAN encapsulation, underlay IP forwarding, destination VTEP decapsulation and final host delivery. This method helps isolate the failing stage rather than treating the overlay as one opaque tunnel.

3.10 — Operational Verification Commands

On a Cisco NX-OS-based VXLAN fabric, several operational commands can help validate packet forwarding. Exact output and availability vary by platform and release.

VXLAN Verification · NX-OS-Style CLI

show nve interface
show nve peers
show nve vni

show mac address-table
show l2route evpn mac all

show ip route 10.0.0.12
show ip interface brief

show bgp l2vpn evpn summary
show bgp l2vpn evpn

show interface counters errors

Use the commands supported by the specific platform and NX-OS release. Some EVPN commands are applicable only when BGP EVPN is configured.

A successful troubleshooting investigation should confirm that the local VTEP is operational, the expected VNI is active, the remote VTEP is reachable and the required forwarding information exists.

If the fabric uses BGP EVPN, the engineer should also inspect EVPN routes and their imported reachability information. However, a healthy BGP session alone does not prove successful VXLAN data-plane forwarding.

Section 3 — Engineering Takeaways

✓ VXLAN encapsulates the original Ethernet frame inside an 8-byte VXLAN header, UDP and outer IP packet.

✓ The inner headers identify workloads, while the outer IP addresses identify the source and destination VTEPs.

✓ The 24-bit VNI identifies the logical overlay segment and is essential for correct decapsulation.

✓ UDP destination port 4789 is the standard VXLAN port, while source-port entropy can support underlay ECMP.

✓ VXLAN over IPv4 typically adds 50 bytes to the underlay IP packet size for an untagged inner Ethernet frame.

✓ Packet captures and VTEP/VNI verification help distinguish encapsulation faults from underlay routing and endpoint forwarding problems.

Continue Learning · Section 04

VXLAN Flood-and-Learn and the BGP EVPN Control Plane

Network Insight · Interactive Engineering Lab 03

VXLAN Encapsulation and Packet Header Explorer

Build and inspect a VXLAN packet. Explore how the original Ethernet frame is wrapped in VXLAN, UDP, IP and Ethernet headers, then trace the packet across the underlay and examine its MTU requirements.

VXLAN Network Identifier

Inner IP Packet Size

Underlay IP MTU

VXLAN Packet Journey Ready to Encapsulate

Select a Packet Layer to Inspect Its Fields

Packet Header Inspector

Inner Ethernet Header

The original Ethernet header identifies the source and destination MAC addresses. VXLAN transports this frame across the routed IP network without requiring intermediate spines to inspect its inner MAC addresses.

Header Fields

Field 1 Source MAC

Field 2 Destination MAC

Header Size 14 bytes

Purpose Tenant Ethernet

Original Inner IP Packet 1500 B Excludes inner Ethernet header

Outer IP Packet Size 1550 B Compared with underlay IP MTU

Encapsulation Overhead +50 B 14 inner Ethernet + 20 IPv4 + 8 UDP + 8 VXLAN

Underlay MTU is sufficient for this packet.

Select a header or run the animation

Packet Encapsulation Event Timeline

[READY] Host A has an Ethernet frame. [READY] Select a header to inspect its fields. [READY] Run the animation to trace encapsulation.

Key Engineering Principle

VXLAN preserves the original Ethernet frame inside a new transport packet. The underlay forwards using the outer IP destination, while the remote VTEP removes encapsulation and delivers the original Ethernet frame. Always account for encapsulation overhead when planning the underlay MTU.

Section 04 · VXLAN Control Plane

VXLAN Flood-and-Learn and the BGP EVPN Control Plane

VXLAN defines how Ethernet frames are encapsulated and transported across an IP network. However, encapsulation alone does not explain how a VTEP discovers remote MAC addresses, identifies participating tunnel endpoints or determines where unknown traffic should be sent.

These functions are handled through endpoint learning and control-plane mechanisms. Traditional VXLAN deployments commonly relied on flood-and-learn forwarding, while modern data centre fabrics frequently use BGP Ethernet VPN (EVPN) to distribute endpoint reachability and overlay membership information.

Understanding both approaches is important because they solve similar forwarding problems through fundamentally different mechanisms. Flood-and-learn depends heavily on data-plane observation, whereas EVPN enables endpoint information to be advertised through a routing control plane.

Flood-and-Learn BUM Traffic Multicast Ingress Replication BGP EVPN Route Type 2 Route Type 3 ARP Suppression

4.1 — Why VXLAN Needs Endpoint Discovery

Consider two servers connected to different leaf switches in the same VXLAN segment. Host A is connected to Leaf A, while Host B is connected to Leaf B.

Before Leaf A can send a known-unicast VXLAN packet directly to Leaf B, it must determine which remote VTEP is responsible for Host B's destination MAC address.

This requires a mapping between the remote MAC address and the destination VTEP IP address.

ENDPOINT LOCATION

MAC-to-VTEP Mapping

Associates a remote endpoint MAC address with the VTEP IP address through which the endpoint is reachable.

OVERLAY MEMBERSHIP

VNI Participation

Identifies the VTEPs participating in a particular VXLAN segment, allowing appropriate forwarding and replication.

The Endpoint Discovery Problem

VXLAN encapsulation provides a way to transport a frame, but the ingress VTEP must still determine the correct remote VTEP destination. Flood-and-learn and BGP EVPN provide different methods for discovering and maintaining that forwarding information.

4.2 — Traditional VXLAN Flood-and-Learn

Traditional VXLAN flood-and-learn operates in a way that resembles conventional Ethernet switching. A VTEP learns MAC addresses from observed traffic and uses flooding when the destination is unknown.

When an ingress VTEP receives a frame destined for an unknown MAC address, it replicates the frame to other participating VTEPs using the configured BUM delivery mechanism.

When the destination endpoint responds, the receiving VTEPs can learn the association between the source MAC address and the originating VTEP address.

Unknown Destination MAC

Host A sends a frame toward Host B. Leaf A does not yet know which remote VTEP owns Host B's MAC address.

Flooding Across the Overlay

Leaf A sends copies to the appropriate remote VTEPs using multicast or ingress replication.

Destination Receives the Frame

Leaf B forwards the received frame into the local bridge domain containing Host B.

Source MAC Learning

VTEPs learn remote source MAC-to-VTEP associations from observed encapsulated traffic.

Subsequent Known Unicast

Once the required destination mapping is learned, traffic can be sent directly to the appropriate VTEP.

Flood-and-learn is conceptually straightforward, but it introduces operational considerations around flooding scope, replication, MAC aging and endpoint mobility.

4.3 — Understanding BUM Traffic

BUM stands for Broadcast, Unknown Unicast and Multicast. These traffic categories require special handling because the ingress VTEP may not have a single known destination VTEP.

Traffic Type	Example	Typical VXLAN Handling
Broadcast	ARP request	Replicated to participating VTEPs unless locally suppressed or otherwise handled
Unknown Unicast	Frame with an unknown destination MAC	Flooded within the relevant overlay segment when unknown-unicast flooding is enabled
Multicast	Application multicast Ethernet traffic	Delivered according to the configured overlay multicast and replication behavior

BUM replication must be constrained to the appropriate VXLAN segment. A VTEP should not replicate tenant traffic indiscriminately across unrelated VNIs.

Flooding and Scale

Excessive BUM traffic can consume bandwidth and processing resources. Large VXLAN environments therefore benefit from careful replication design, endpoint learning controls and features such as ARP suppression.

4.4 — Multicast Versus Ingress Replication

VXLAN can distribute BUM traffic through underlay multicast or through ingress replication. Both approaches deliver copies to participating VTEPs, but they differ in where packet replication occurs.

UNDERLAY MULTICAST

Multicast-Based Replication

VTEPs send VXLAN BUM traffic toward a multicast group. Multicast-capable underlay devices replicate traffic along the distribution tree.

HEAD-END REPLICATION

Ingress Replication

The ingress VTEP creates individual VXLAN copies for remote VTEPs in the replication list. Underlay multicast is not required.

Characteristic	Multicast	Ingress Replication
Underlay multicast	Required	Not required
Replication location	Multicast distribution tree in the network	Ingress VTEP
Source VTEP load	Generally fewer source-generated copies	Increases with the number of remote replication destinations
Operational dependency	Multicast routing and group membership	Accurate remote VTEP replication list
Common deployment	Traditional multicast-based VXLAN fabrics	Frequently used with BGP EVPN

Ingress replication is popular in EVPN fabrics because the control plane can distribute the information needed to construct replication lists without requiring multicast routing throughout the underlay.

4.5 — Introducing the BGP EVPN Control Plane

BGP EVPN uses Multiprotocol BGP to distribute overlay reachability information between participating network devices.

Rather than relying exclusively on observed VXLAN traffic to learn remote endpoint locations, EVPN allows VTEPs to advertise reachability information through BGP routes.

This separates the control plane, responsible for distributing endpoint information, from the data plane, responsible for carrying encapsulated Ethernet traffic.

Reference Architecture · EVPN Control Plane and VXLAN Data Plane

IP underlay EVPN control-plane sessions

Conceptual EVPN route-reflector topology. The dashed green paths represent logical BGP EVPN sessions, not dedicated physical links. The route reflector distributes EVPN reachability information, while VXLAN data traffic travels directly between VTEPs through the routed underlay.

A BGP EVPN route reflector can distribute overlay reachability information between leaf VTEPs without becoming part of the VXLAN data path.

This architecture supports scalable endpoint learning and allows network engineers to inspect the control-plane state independently from the forwarding path.

4.6 — EVPN Route Type 2: MAC/IP Advertisement

EVPN Route Type 2 is the MAC/IP Advertisement Route. It allows a VTEP to advertise endpoint MAC reachability and, when applicable, associated IP address information.

For example, Leaf B can advertise that Host B's MAC address is reachable through Leaf B's VTEP.

Leaf A can import the appropriate EVPN route and install forwarding information for Host B without needing to discover the remote MAC location solely through flooding.

Conceptual EVPN Type 2 Advertisement

EVPN Route Type: 2
Route: MAC/IP Advertisement

MAC Address:
  00:11:22:33:44:02

IP Address:
  192.168.10.12

Advertising VTEP:
  10.0.0.12

Overlay Segment:
  VNI 10010

BGP Next Hop:
  10.0.0.12

Result:
  Remote MAC reachability learned
  through the EVPN control plane

Simplified route illustration. Actual EVPN NLRI includes additional fields such as the route distinguisher, Ethernet segment identifier and Ethernet tag ID. Route targets and encapsulation-related attributes influence import and forwarding.

Control-Plane Learning

Type 2 routes reduce dependence on data-plane flooding for discovering known remote endpoints. The receiving VTEP can install the advertised MAC-to-VTEP mapping after validating and importing the route.

4.7 — EVPN Route Type 3: Inclusive Multicast Ethernet Tag

EVPN Route Type 3 is the Inclusive Multicast Ethernet Tag (IMET) Route. It advertises information used to establish participation in a broadcast domain and construct BUM delivery state.

In an ingress-replication design, IMET routes can help VTEPs discover the remote peers that should receive replicated BUM traffic for the relevant overlay segment.

EVPN ROUTE TYPE 2

Endpoint Reachability

Advertises MAC addresses and optional IP bindings, enabling remote endpoint forwarding information to be installed.

EVPN ROUTE TYPE 3

Broadcast Domain Membership

Supports discovery of participating VTEPs and establishment of BUM delivery state, including ingress replication lists.

Route Type	Primary Role	Operational Importance
Type 2	MAC/IP Advertisement	Known-unicast endpoint reachability
Type 3	Inclusive Multicast Ethernet Tag	BUM membership and replication information
Type 1	Ethernet Auto-Discovery	Multihoming procedures and mass withdrawal
Type 4	Ethernet Segment	Ethernet segment discovery and designated forwarder election
Type 5	IP Prefix Advertisement	IP prefix reachability in EVPN-based routing architectures

Type 2 and Type 3 are particularly important when understanding basic VXLAN EVPN Layer 2 forwarding.

Other EVPN route types support more advanced designs, including multihoming, distributed routing and IP prefix advertisement.

4.8 — EVPN Endpoint Learning Sequence

The following example shows how Host B's reachability can become available to Leaf A through BGP EVPN.

Host B Becomes Active

Host B sends traffic through Leaf B. Leaf B learns the local endpoint MAC address and, where applicable, its IP binding.

Leaf B Advertises Type 2

Leaf B originates an EVPN MAC/IP Advertisement Route containing the endpoint reachability information.

Route Reflector Distributes EVPN

The EVPN route reflector distributes the route to appropriate BGP EVPN peers according to routing policy.

Leaf A Imports the Route

Leaf A validates the route and imports the endpoint mapping into the appropriate overlay forwarding context.

Known-Unicast VXLAN Forwarding

Leaf A can encapsulate traffic for Host B directly toward Leaf B's VTEP, assuming underlay reachability and valid forwarding state.

Control Plane Versus Data Plane

EVPN distributes information about where an endpoint is located. VXLAN carries the actual Ethernet frame. The BGP route reflector normally participates in control-plane distribution but does not need to forward the encapsulated tenant data traffic.

4.9 — ARP Suppression in VXLAN EVPN

In a conventional Layer 2 network, ARP requests are broadcast to discover the MAC address associated with an IPv4 address.

In a VXLAN fabric, uncontrolled ARP broadcasting can create additional BUM traffic across the overlay.

With EVPN-based ARP suppression, a VTEP can use known IP-to-MAC bindings to answer certain ARP requests locally, reducing the need to flood those requests to remote VTEPs.

WITHOUT SUPPRESSION

ARP Flooding

An ARP request may be replicated across the VXLAN segment to discover the destination MAC address.

WITH SUPPRESSION

Local ARP Response

A VTEP with a valid learned IP-to-MAC binding may answer the ARP request locally without flooding it.

Why ARP Suppression Helps

ARP suppression reduces unnecessary broadcast traffic when valid endpoint bindings are available. Unknown or unresolved requests may still require flooding or other resolution mechanisms, depending on platform behavior and configuration.

4.10 — MAC Mobility and Endpoint Movement

Virtual machines and workloads may move between hosts or leaf switches while retaining their original MAC and IP addresses.

This creates a forwarding challenge: remote VTEPs must update their endpoint mappings when a MAC address becomes reachable through a different VTEP.

BGP EVPN supports MAC mobility procedures, including a MAC Mobility extended community and sequence numbers used to help distinguish newer endpoint location advertisements.

Original Endpoint Location

Host B is reachable through Leaf B. Remote VTEPs have imported the corresponding EVPN Type 2 route.

Workload Moves to Leaf C

Host B moves to an attachment point behind Leaf C while retaining its MAC address.

Updated EVPN Advertisement

Leaf C advertises updated MAC reachability. MAC mobility information helps remote devices select the newer endpoint location.

Forwarding State Converges

Remote VTEPs update the MAC-to-VTEP mapping and forward subsequent traffic toward Leaf C.

MAC mobility is especially important in virtualized environments where workload placement can change without requiring the endpoint to obtain a new IP address.

MAC Flapping Versus Legitimate Mobility

Repeated movement of the same MAC address between VTEPs can indicate legitimate workload migration, but it may also indicate a loop, duplicate MAC address or configuration error. Engineers should inspect mobility events and sequence numbers before assuming the movement is expected.

4.11 — Flood-and-Learn Versus BGP EVPN

Capability	Flood-and-Learn	BGP EVPN
Remote MAC learning	Observed data-plane traffic	Control-plane Type 2 advertisements
Unknown destinations	Typically flooded	May be flooded or handled by policy and platform features
VTEP discovery	Multicast membership or configured peer lists	EVPN membership information and replication state
ARP suppression	Not inherent to basic flood-and-learn	Supported when suitable IP/MAC bindings are available
MAC mobility	Data-plane relearning	EVPN mobility procedures and sequence numbers
Operational visibility	Primarily forwarding and MAC tables	BGP EVPN routes, forwarding state and VTEP tables
Typical architecture	Simpler or legacy VXLAN environments	Modern scalable VXLAN EVPN fabrics

BGP EVPN provides a more structured control plane for endpoint distribution, but it also introduces additional operational dependencies.

Engineers must verify BGP neighbor state, EVPN address-family activation, route import policies, route targets, VNI mappings and underlay reachability.

4.12 — EVPN Control-Plane Verification

A healthy VXLAN EVPN fabric requires consistency between BGP EVPN routes and the actual VXLAN forwarding state installed on the VTEPs.

The following Cisco NX-OS-style commands provide useful starting points for verifying EVPN reachability and replication.

EVPN Operational Verification · NX-OS Style

show bgp l2vpn evpn summary

show bgp l2vpn evpn

show bgp l2vpn evpn route-type 2

show bgp l2vpn evpn route-type 3

show nve peers

show nve vni

show l2route evpn mac all

show mac address-table

show ip route 10.0.0.12

Illustrative NX-OS commands. Exact syntax, supported route-type filters and output vary by platform and software release.

Check	Expected Result
BGP EVPN neighbors	Expected sessions are Established
Type 2 routes	Remote endpoint reachability is advertised and imported
Type 3 routes	Expected overlay membership and BUM replication state exist
VNI state	Required VNIs are configured and operational
Remote VTEP route	Underlay provides usable reachability to remote VTEP IP
MAC forwarding table	Remote MAC addresses resolve to the correct VTEP

EVPN Troubleshooting Warning

An Established BGP EVPN session does not prove that remote MAC routes have been imported correctly. Route-target mismatches, incorrect VNI mappings, missing endpoint advertisements or unresolved VTEP next hops can prevent forwarding even when the BGP session itself is healthy.

4.13 — Engineering Design Considerations

Selecting the correct VXLAN control-plane design requires consideration of scalability, replication overhead, operational visibility and endpoint mobility.

Design Area	Engineering Consideration
EVPN route reflectors	Provide scalable BGP route distribution with appropriate redundancy and policy
Route targets	Control which EVPN routes are imported into the intended overlay contexts
BUM replication	Balance source VTEP replication load against underlay multicast complexity
ARP suppression	Reduce unnecessary ARP flooding while maintaining accurate endpoint bindings
MAC mobility	Ensure endpoint moves converge correctly without persistent stale forwarding state
Underlay reachability	Maintain resilient routing between all required VTEPs
Monitoring	Correlate EVPN control-plane state with installed forwarding entries and packet delivery

Recommended Troubleshooting Order

Verify underlay VTEP reachability first, followed by BGP EVPN neighbor state, imported Type 2 and Type 3 routes, VNI membership, remote MAC forwarding entries and actual VXLAN packet delivery. This sequence helps separate transport failures from control-plane and forwarding-state problems.

Section 4 — Engineering Takeaways

✓ Traditional VXLAN flood-and-learn uses observed traffic and flooding to discover remote MAC locations.

✓ BUM traffic includes broadcast, unknown unicast and multicast frames that may require replication.

✓ Underlay multicast and ingress replication provide different mechanisms for distributing BUM traffic.

✓ EVPN Type 2 routes advertise endpoint MAC/IP reachability, while Type 3 routes support broadcast domain membership and BUM delivery state.

✓ ARP suppression can reduce broadcast traffic when valid endpoint bindings are available.

✓ EVPN MAC mobility procedures help update forwarding information when workloads move between VTEPs.

✓ Successful VXLAN forwarding depends on both valid EVPN control-plane information and operational underlay and data-plane state.

Continue Learning · Section 05

VXLAN EVPN Routing, Anycast Gateways and Inter-VNI Connectivity

Network Insight · Interactive Engineering Lab 04

Flood-and-Learn vs BGP EVPN: Explore the VXLAN Control Plane

Investigate how remote endpoint reachability is discovered. Compare data-plane flooding with BGP EVPN route advertisements, inspect the resulting forwarding state and simulate endpoint mobility between VTEPs.

Step 1 · Select Control-Plane Model

Step 2 · Generate a Network Event

VXLAN Fabric · VNI 10010 Data-Plane Learning

BUM Replication Copies (Current Event) 0

EVPN Route Advertisements (Current Event) 0

Remote MAC Entry on Leaf A Unknown

Leaf A · Remote MAC Reachability

Destination	Remote VTEP	Source
No remote entry learned

EVPN Control-Plane State

Flood-and-Learn Mode

Leaf A initially has no remote MAC mapping. Unknown unicast traffic is replicated to other VTEPs participating in the VNI.

EVPN Route	State
Type 2 · MAC/IP	Not used
Type 3 · IMET	Not used

Control-Plane and Data-Plane Event Timeline

[READY] Select a control-plane model. [READY] Generate unknown unicast, ARP or mobility events.

Engineering Takeaway

Flood-and-learn discovers remote MAC reachability through data-plane traffic. BGP EVPN can advertise MAC/IP reachability in the control plane, reducing the need for unknown-unicast flooding.

Educational event-driven model. Assumes three participating VTEPs, ingress replication and pre-established EVPN sessions in EVPN mode. The widget simplifies MAC learning, route withdrawal, mobility sequencing and ARP suppression; it is not a full protocol emulator.

Section 05 · Distributed Layer 3 Overlay

VXLAN EVPN Routing, Anycast Gateways and Inter-VNI Connectivity

VXLAN is often introduced as a technology that extends Layer 2 networks across an IP fabric. However, modern VXLAN EVPN architectures also support distributed Layer 3 routing between different subnets and virtual networks.

This capability allows workloads connected to different leaf switches and different VLANs to communicate through distributed routing gateways rather than sending all inter-subnet traffic to a centralized router.

The main building blocks include distributed anycast gateways, integrated routing and bridging (IRB), Layer 2 VNIs, Layer 3 VNIs, VRFs and EVPN route advertisements. Together, these components enable a scalable routed overlay while the physical underlay continues to provide ordinary IP connectivity between VTEPs.

Anycast Gateway Inter-VNI Routing Symmetric IRB Asymmetric IRB L2 VNI L3 VNI VRF EVPN Type 5

5.1 — Why Layer 3 Routing Matters in VXLAN Fabrics

A VXLAN bridge domain provides Layer 2 connectivity for endpoints belonging to the same logical network segment.

However, real data centre applications often consist of multiple subnets. Web servers, application servers and databases may be assigned to different VLANs and IP networks.

Communication between those networks requires Layer 3 routing. A modern VXLAN EVPN fabric can perform this routing directly on the leaf switches where workloads connect.

SAME SUBNET

Layer 2 VXLAN Forwarding

Traffic between endpoints in the same bridge domain can be transported across the overlay using the associated L2 VNI.

DIFFERENT SUBNETS

Layer 3 EVPN Routing

Traffic between IP subnets is routed through an IRB gateway and can cross the overlay using an L3 VNI.

Architectural Principle

The underlay provides IP transport between VTEPs. The overlay defines tenant Layer 2 and Layer 3 forwarding contexts. Distributed routing allows the leaf fabric to route tenant traffic without requiring a centralized physical router for every inter-subnet flow.

5.2 — Distributed Anycast Gateways

A distributed anycast gateway allows multiple leaf switches to present the same default gateway IP address and virtual gateway MAC address to endpoints in a given subnet.

For example, hosts in subnet 192.168.10.0/24 might use 192.168.10.1 as their default gateway, regardless of which leaf switch they are connected to.

Each participating leaf can perform local routing for traffic that must leave the source subnet. This reduces the need to send packets across the fabric merely to reach a centralized first-hop gateway.

Gateway Attribute	Example	Purpose
Subnet	192.168.10.0/24	Endpoint IP network
Gateway IP	192.168.10.1	Shared default gateway address
Virtual gateway MAC	0001.0001.0001	Common gateway MAC used by participating leaf switches
Leaf A	VTEP 10.0.0.11	Local first-hop routing for its hosts
Leaf B	VTEP 10.0.0.12	Local first-hop routing for its hosts

The gateway IP and virtual MAC must be consistently configured for the same subnet across participating leaves.

Distributed anycast gateways also support workload mobility: a virtual machine moving to another leaf can continue using the same first-hop gateway identity.

Why Anycast Gateways Improve Traffic Flow

Hosts can reach their default gateway locally on the attached leaf. This can reduce unnecessary hairpinning and allows the fabric to route traffic closer to the source endpoint.

5.3 — Layer 2 VNIs and Layer 3 VNIs

VXLAN EVPN fabrics commonly distinguish between Layer 2 VNIs and Layer 3 VNIs.

An L2 VNI identifies a Layer 2 bridge domain, often associated with a VLAN. An L3 VNI identifies a routed overlay context associated with a VRF.

Component	Example	Function
VLAN 10	192.168.10.0/24	Web subnet
L2 VNI 10010	VLAN 10	Web Layer 2 bridge domain
VLAN 20	192.168.20.0/24	Application subnet
L2 VNI 10020	VLAN 20	Application Layer 2 bridge domain
VRF TENANT-A	Tenant routing table	Isolated Layer 3 forwarding context
L3 VNI 50000	VRF TENANT-A	Routed VXLAN overlay context

In a symmetric IRB design, the L3 VNI is used to transport routed traffic between VTEPs in the relevant tenant VRF.

The L2 VNI continues to provide Layer 2 connectivity for endpoints within the corresponding bridge domain.

VNI Mapping Is Platform-Specific

L2 and L3 VNI numbers are locally configured design values. The examples here illustrate a common VLAN-to-L2-VNI and VRF-to-L3-VNI mapping; the numbers themselves are not reserved or mandatory.

5.4 — VXLAN EVPN Routed Fabric Architecture

The following example contains two leaf VTEPs, two routed spine switches and endpoints in different IP subnets.

Host A belongs to VLAN 10 / VNI 10010, while Host B belongs to VLAN 20 / VNI 10020. Both subnets are associated with VRF TENANT-A and L3 VNI 50000.

Reference Topology · Distributed Routing over VXLAN EVPN

Physical IP underlay Logical routed VXLAN overlay Host attachment

Simplified symmetric IRB example. Each leaf performs local routing for its attached subnet, while the L3 VNI carries routed traffic between VTEPs. The dashed overlay path is logical and is transported through the routed spine fabric. In a distributed gateway deployment, gateway interfaces for a subnet are configured consistently on all leaves participating in that subnet.

5.5 — Integrated Routing and Bridging (IRB)

Integrated Routing and Bridging combines Layer 2 bridging and Layer 3 routing within a single forwarding architecture.

In a VXLAN EVPN fabric, IRB allows a VTEP to bridge frames within an L2 VNI and route packets between different IP subnets associated with the same tenant VRF.

Two major IRB forwarding models are commonly discussed: asymmetric IRB and symmetric IRB.

ASYMMETRIC IRB

Route at Ingress, Bridge at Egress

The ingress VTEP routes into the destination bridge domain and encapsulates traffic using the destination L2 VNI. The egress VTEP then bridges the frame locally.

SYMMETRIC IRB

Route at Both VTEPs

The ingress VTEP routes traffic into the tenant L3 VNI. The egress VTEP performs the corresponding routing lookup and delivers traffic into the destination bridge domain.

5.6 — Asymmetric IRB Explained

In asymmetric IRB, the ingress VTEP performs the Layer 3 routing decision and selects the destination subnet's bridge domain.

The packet is then encapsulated using the destination L2 VNI and transported to the remote VTEP. The egress VTEP bridges the resulting frame to the destination.

Because the ingress leaf may need to route into multiple destination bridge domains, asymmetric designs can require broad distribution of VLAN and L2 VNI configuration across the participating fabric.

Asymmetric IRB Scaling Consideration

In common asymmetric IRB implementations, ingress leaves need the destination subnet's L2 forwarding context. As the number of subnets increases, this can increase the number of VLANs and L2 VNIs that must be provisioned across the fabric.

5.7 — Symmetric IRB Explained

Symmetric IRB performs Layer 3 forwarding at both the ingress and egress VTEPs.

The ingress VTEP routes the packet from the source bridge domain into the tenant VRF and encapsulates it using the L3 VNI.

The egress VTEP decapsulates the routed overlay packet, performs the appropriate destination forwarding operation and delivers it through the local destination bridge domain.

This design allows each leaf to host only the local Layer 2 segments it requires, while participating in the shared tenant Layer 3 routing context.

Characteristic	Asymmetric IRB	Symmetric IRB
Ingress VTEP	Routes into destination L2 segment	Routes into tenant L3 VNI
Egress VTEP	Bridges to destination	Routes/forwards into destination bridge domain
Inter-VTEP VNI	Destination L2 VNI	Tenant L3 VNI
Remote L2 VNI presence	Often needed at ingress for destination subnets	Not generally required for every remote subnet
Scalability	Can increase L2 state across the fabric	Better separation of local L2 state and tenant routing
Common use	Smaller or specific EVPN IRB designs	Large distributed routed EVPN fabrics

Design Preference

Symmetric IRB is widely used in modern VXLAN EVPN data centre fabrics because it avoids the need for every ingress leaf to instantiate every destination Layer 2 segment. The exact forwarding behavior depends on the platform and EVPN implementation.

5.8 — Symmetric IRB Packet Journey

Consider Host A (192.168.10.11) communicating with Host B (192.168.20.12). Both networks belong to VRF TENANT-A.

The example assumes that the necessary endpoint routes, VNI mappings and underlay reachability are already installed.

Host A Sends to Its Default Gateway

Host A recognizes that 192.168.20.12 belongs to another subnet. It sends the packet to gateway 192.168.10.1 on Leaf A.

Leaf A Performs a VRF Routing Lookup

Leaf A routes the packet in VRF TENANT-A. The installed EVPN forwarding information identifies Leaf B as the remote VTEP for the destination.

Routed VXLAN Encapsulation

Leaf A encapsulates the routed packet using L3 VNI 50000 with outer source VTEP 10.0.0.11 and destination VTEP 10.0.0.12.

Underlay IP Forwarding

The spine fabric forwards the encapsulated packet toward Leaf B using the outer destination IP. ECMP may select an available path.

Leaf B Decapsulates and Routes

Leaf B identifies the tenant L3 VNI, processes the packet in the appropriate VRF and resolves the destination through its local VLAN 20 bridge domain.

Host B Receives the Packet

Leaf B delivers the packet to Host B on 192.168.20.12. The original source and destination IPs remain unchanged in this example.

Inner Header Handling

Symmetric IRB does not simply preserve the original host Ethernet frame unchanged. The ingress routing operation creates the appropriate routed overlay forwarding context. The exact inner Ethernet header, router MAC information and egress processing depend on the EVPN forwarding model and platform.

5.9 — VRFs and Tenant Isolation

A Virtual Routing and Forwarding instance, or VRF, maintains a separate Layer 3 routing table.

In a VXLAN EVPN fabric, VRFs allow multiple tenants or application environments to use isolated routing contexts over the same physical underlay.

A VRF may contain multiple local subnets and L2 VNIs. A corresponding L3 VNI can provide routed connectivity between VTEPs participating in that VRF.

TENANT-A

VRF TENANT-A

Web and application subnets use their own routing table, L2 VNIs and routed L3 VNI.

TENANT-B

VRF TENANT-B

A separate tenant can use independent routing and overlay identifiers while sharing the same physical IP fabric.

VRF Isolation Is Not a Complete Security Policy

VRFs provide separate routing contexts, but intentional route leaking, shared services or external connectivity can introduce paths between them. Firewall policy, segmentation controls and route import rules must still be designed appropriately.

5.10 — EVPN Route Type 5: IP Prefix Advertisement

EVPN Route Type 5 is the IP Prefix Advertisement Route. It provides a way to advertise IP prefix reachability through the EVPN control plane.

While Type 2 routes primarily describe MAC reachability and optional host IP bindings, Type 5 routes advertise IP prefixes independently of individual endpoint MAC advertisements.

This is useful for advertising routed networks, external prefixes and aggregate reachability in EVPN-based Layer 3 overlays.

EVPN Route	Primary Information	Typical Role
Type 2	MAC address and optional IP address	Endpoint reachability, host routes and MAC/IP bindings
Type 3	Inclusive multicast Ethernet tag information	Overlay membership and BUM delivery
Type 5	IP prefix	Routed prefix advertisement in EVPN

Conceptual EVPN Route Type 5 Advertisement

EVPN Route Type: 5
Route: IP Prefix Advertisement

Tenant VRF:
  TENANT-A

Advertised Prefix:
  172.16.50.0/24

Advertising VTEP:
  10.0.0.12

Tenant Routed Overlay:
  L3 VNI 50000

Route Import:
  Controlled by EVPN route targets

Result:
  Remote VTEPs can install
  tenant IP prefix reachability

Conceptual example rather than raw BGP NLRI. Actual Type 5 advertisements include route distinguisher, Ethernet tag, IP prefix and gateway-related fields, with associated BGP attributes and extended communities. Forwarding resolution is implementation- and deployment-dependent.

Type 2 Versus Type 5

Type 2 can advertise host reachability with MAC/IP bindings, while Type 5 supports IP prefix advertisements. Both can participate in EVPN Layer 3 designs; Type 5 is not mandatory for every symmetric IRB deployment.

5.11 — Route Targets and Route Import

EVPN uses BGP route distinguishers and route targets for different purposes.

A Route Distinguisher (RD) helps make otherwise overlapping VPN routes unique in the BGP control plane.

A Route Target (RT) is an extended community used to control which VPN routes are imported into the appropriate local contexts.

Attribute	Purpose	Example
RD	Makes EVPN route identifiers unique	65001:50000
RT Import	Selects EVPN routes imported into a local context	65001:50000
RT Export	Marks advertised EVPN routes for intended import	65001:50000
L3 VNI	Identifies tenant routed VXLAN traffic	50000

Incorrect route targets can cause a VTEP to receive EVPN routes without importing them into the expected VRF or bridge domain.

This is an important troubleshooting distinction: a route visible in the BGP EVPN table is not necessarily installed in the intended tenant forwarding table.

5.12 — VXLAN EVPN Routing Verification

When troubleshooting inter-VNI routing, engineers should verify the tenant routing context, anycast gateway configuration, EVPN reachability and installed forwarding state.

Cisco NX-OS-Style Verification Commands

show vrf

show ip route vrf TENANT-A

show ip interface brief vrf TENANT-A

show nve interface

show nve vni

show nve peers

show bgp l2vpn evpn summary

show bgp l2vpn evpn route-type 2

show bgp l2vpn evpn route-type 5

show l2route evpn mac all

show mac address-table

Illustrative verification commands. Exact syntax, VRF filtering, route-type support and output depend on the NX-OS platform and software release.

Verification	Expected State	Possible Failure
VRF routing table	Destination route installed	Missing or incorrectly imported EVPN route
Anycast gateway	Correct gateway IP and MAC	Incorrect SVI or virtual gateway configuration
L2 VNI	Local bridge domain mapped correctly	VLAN-to-VNI mismatch
L3 VNI	Tenant routed overlay operational	Missing VRF-to-VNI mapping
EVPN routes	Correct remote host or prefix reachability	RT mismatch or missing advertisement
VTEP reachability	Outer destination reachable through underlay	Underlay routing failure
Packet forwarding	Encapsulation, decapsulation and local delivery succeed	Incorrect forwarding entry or policy drop

5.13 — Common Inter-VNI Routing Failures

A VXLAN fabric may provide successful same-subnet communication while failing to route between different subnets.

This often indicates that the L2 overlay is operational but the distributed Layer 3 forwarding path is incomplete or misconfigured.

GATEWAY ISSUE

Incorrect Anycast Gateway

Inconsistent gateway IP or MAC configuration can prevent reliable first-hop routing.

VRF ISSUE

Wrong Routing Context

The source or destination subnet may be associated with the wrong tenant VRF.

EVPN ISSUE

Missing Route Import

EVPN routes may be received but rejected or not imported because of route-target policy.

DATA-PLANE ISSUE

Incorrect L3 VNI

A routed overlay mismatch can prevent the receiving VTEP from processing the packet in the correct VRF.

Recommended Troubleshooting Sequence

Confirm local gateway reachability, verify the source VRF routing table, inspect the imported EVPN route, validate the remote VTEP next hop, confirm L3 VNI encapsulation and finally inspect egress routing and local host delivery.

5.14 — Design Considerations for Distributed Routing

Distributed EVPN routing introduces additional flexibility, but consistent design remains essential.

Design Area	Recommendation
Gateway consistency	Use consistent anycast gateway IP and virtual MAC settings for each subnet across participating leaves
VNI planning	Maintain a documented mapping between VLANs, L2 VNIs, VRFs and L3 VNIs
IRB model	Select symmetric or asymmetric IRB based on platform support, scale and forwarding requirements
EVPN policy	Validate route targets and import/export policy for every tenant context
Underlay routing	Provide resilient VTEP reachability with suitable ECMP and MTU design
Segmentation	Combine VRF isolation with appropriate security and route leaking controls
Operational visibility	Correlate BGP EVPN routes with installed tenant routes and actual packet forwarding

Section 5 — Engineering Takeaways

✓ VXLAN EVPN supports distributed Layer 3 routing between different IP subnets, not only Layer 2 extension.

✓ Distributed anycast gateways allow endpoints to use a consistent default gateway identity across participating leaf switches.

✓ L2 VNIs identify bridge domains, while L3 VNIs support routed tenant overlays.

✓ Symmetric IRB routes traffic at both ingress and egress VTEPs and is widely used in scalable EVPN fabrics.

✓ VRFs provide separate tenant routing tables, while route targets control EVPN route import.

✓ EVPN Type 5 routes support IP prefix advertisements, complementing Type 2 endpoint reachability.

✓ Successful inter-VNI forwarding requires correct gateway, VRF, VNI, EVPN and underlay state.

Continue Learning · Section 06

VXLAN Deployment, Verification and Troubleshooting

Network Insight · Interactive Engineering Lab 05

VXLAN EVPN Routing: Anycast Gateways and Inter-VNI Connectivity

Follow a packet between tenant workloads. Compare Layer 2 bridging with Layer 3 routing, inspect symmetric and asymmetric IRB, and investigate what happens when the tenant VRF or EVPN reachability is misconfigured.

Step 1 · Choose Traffic Scenario

Step 2 · Select IRB Model

Distributed Anycast Gateway Fabric Symmetric IRB · Ready

Ready to trace

Transport VNI 50000

Forwarding Operation Route → Bridge

Forwarding Result Ready

Tenant Forwarding Table

Destination	Next Hop / VTEP	Context

Control-Plane and Gateway State

Component	Status
Tenant VRF	TENANT-A · Active
EVPN Reachability	Installed
Anycast Gateway	Active on both leaves
Routing Model	Symmetric IRB

Packet Forwarding and EVPN Investigation Timeline

[READY] TENANT-A VRF initialized. [READY] L2VNIs 10010 and 10020 configured. [READY] L3VNI 50000 available. [READY] Select a scenario and trace the packet.

Engineering Explanation

With symmetric IRB, the ingress leaf routes traffic into the tenant L3VNI. The remote leaf receives the routed VXLAN packet, performs destination-side forwarding and delivers it through the appropriate L2VNI or local VLAN. Distributed anycast gateways provide consistent default-gateway addresses across the fabric.

Simplified conceptual EVPN forwarding model. Assumes pre-established underlay and EVPN reachability, local gateway configuration and a single tenant VRF. Actual symmetric IRB forwarding also uses router MAC information and platform-specific EVPN route programming. This lab does not emulate a real BGP or ASIC pipeline.

Section 06 · Deployment and Operations

VXLAN Deployment, Verification and Troubleshooting

Designing a VXLAN EVPN fabric requires more than configuring a VXLAN tunnel. A successful deployment depends on the interaction between the physical IP underlay, VTEP interfaces, overlay segment mappings, BGP EVPN control-plane advertisements and the forwarding tables installed on each leaf switch.

When a VXLAN network fails, the visible symptom may appear to be a simple connectivity problem. However, the root cause could exist in the underlay routing table, EVPN route distribution, VLAN-to-VNI mapping, tenant VRF or encapsulated packet path.

A structured operational approach separates these layers and verifies each dependency in the correct order.

Underlay Routing VTEP Reachability NVE Interface BGP EVPN VNI Mapping Route Targets Packet Capture Incident Diagnosis

6.1 — The VXLAN Deployment Model

A typical VXLAN EVPN deployment is built in logical stages. The underlay must first provide reliable IP reachability between the VTEP addresses.

Once the underlay is operational, the overlay can establish VNI mappings and BGP EVPN route distribution.

Tenant forwarding is then enabled through the appropriate Layer 2 and Layer 3 interfaces.

Build the Routed Underlay

Configure physical interfaces, routed leaf-spine links, loopbacks and an appropriate underlay routing protocol.

Establish VTEP Reachability

Confirm that each leaf can reach the remote VTEP source IP addresses through the underlay.

Configure VLAN and VNI Mappings

Associate local bridge domains with their intended L2 VNIs and configure tenant VRFs and L3 VNIs where needed.

Configure the NVE Interface

Define the VXLAN tunnel endpoint source and associate the required VNIs with the network virtualization interface.

Establish BGP EVPN

Configure BGP EVPN neighbors, address families, route distinguishers and route-target policies.

Validate Endpoint Learning

Confirm local and remote MAC learning, EVPN route import, VTEP peer discovery and forwarding state.

Test Tenant Connectivity

Verify same-subnet VXLAN bridging, inter-subnet routing, failure recovery and expected traffic isolation.

Recommended Deployment Order

Validate the underlay before introducing overlay complexity. A correctly configured BGP EVPN control plane cannot compensate for missing IP reachability between VTEPs.

6.2 — VXLAN EVPN Reference Topology

The following reference design uses two spine switches and two leaf VTEPs.

The leaf switches advertise their loopback VTEP addresses through the underlay. BGP EVPN distributes overlay reachability information.

Reference Deployment · Two-Leaf VXLAN EVPN Fabric

IP underlay Logical VXLAN overlay Endpoint access

Simplified deployment topology. VXLAN traffic uses UDP port 4789 and travels through the routed spine fabric between VTEPs. BGP EVPN control-plane sessions are logical routing adjacencies and may use dedicated route reflectors or another supported peering design.

6.3 — Underlay Routing Requirements

The VXLAN underlay is the routed IP network that transports encapsulated packets between VTEPs.

The underlay can use routing protocols such as OSPF, IS-IS or BGP, depending on the design and platform.

Its primary purpose is to provide resilient IP reachability to the loopback addresses used as VTEP sources.

Underlay Requirement	Engineering Check
VTEP IP reachability	Remote VTEP loopbacks are present in the routing table
Routing convergence	Failed links trigger an appropriate alternate path
ECMP	Multiple equal-cost paths are installed where intended
MTU	Underlay supports the additional VXLAN encapsulation overhead
UDP transport	VXLAN UDP traffic is permitted through relevant network policies
Source address	NVE interface uses the intended VTEP loopback address

Underlay Verification · NX-OS-Style Commands

show ip interface brief

show ip route 10.0.0.12

show ip ospf neighbors

show ip route

ping 10.0.0.12 source 10.0.0.11

traceroute 10.0.0.12 source 10.0.0.11

Example assumes OSPF as the underlay protocol. For an IS-IS or BGP underlay, use the corresponding routing verification commands. Source-aware ping and traceroute syntax varies by release.

MTU and VXLAN Encapsulation

VXLAN adds outer Ethernet, IP, UDP and VXLAN headers around the original Ethernet frame. For a common IPv4 transport, this adds approximately 50 bytes of overhead, excluding additional tags or encapsulations. The underlay MTU must accommodate the resulting packet size.

6.4 — Configuring the VTEP and NVE Interface

A VXLAN Tunnel Endpoint, or VTEP, performs VXLAN encapsulation and decapsulation.

On Cisco NX-OS platforms, VXLAN forwarding is commonly configured through a Network Virtualization Edge interface such as NVE1.

The NVE interface references a source loopback and the VNIs that participate in the overlay.

Illustrative Cisco NX-OS VXLAN EVPN Configuration

feature nv overlay
feature vn-segment-vlan-based
feature bgp

nv overlay evpn

vlan 10
  vn-segment 10010

interface loopback1
  ip address 10.0.0.11/32

interface nve1
  no shutdown
  host-reachability protocol bgp
  source-interface loopback1
  member vni 10010
    ingress-replication protocol bgp

evpn
  vni 10010 l2
    rd auto
    route-target import auto
    route-target export auto

Illustrative configuration fragment for a compatible NX-OS VXLAN EVPN deployment. It is not a complete production configuration. The physical underlay, loopback advertisement, BGP EVPN neighbors, access interfaces and other platform-specific requirements must also be configured. Exact syntax and automatic route-target behavior depend on platform and release.

In this example, VLAN 10 is associated with VNI 10010. Loopback1 provides the local VTEP source address.

The NVE interface enables VXLAN encapsulation and uses BGP EVPN for remote endpoint learning and ingress replication state.

6.5 — Verifying BGP EVPN Control-Plane State

The next step is to verify that BGP EVPN sessions are established and that the expected EVPN routes are being exchanged.

Engineers should inspect both the BGP neighbor state and the actual route information installed for the overlay.

BGP EVPN Verification

show bgp l2vpn evpn summary

show bgp l2vpn evpn

show bgp l2vpn evpn route-type 2

show bgp l2vpn evpn route-type 3

show bgp l2vpn evpn route-type 5

show nve peers

show nve vni

Commands are representative of Cisco NX-OS. Support for specific route-type filters and output formats varies.

Verification	Expected Result	Failure Indicator
BGP EVPN neighbor	Established	Idle, Active or repeated resets
Type 2 route	Remote MAC/IP reachability present	Missing endpoint advertisement
Type 3 route	Correct overlay membership state	Missing BUM replication peer
Type 5 route	Expected routed prefixes present when used	Missing prefix or import mismatch
NVE peers	Expected remote VTEPs operational	Missing or down remote peer
VNI status	Required VNIs operational	Inactive VNI or incorrect mapping

BGP Established Does Not Mean VXLAN Works

BGP EVPN can be Established while endpoint routes are missing, route targets are incorrect, VNIs are inactive or remote VTEPs are unreachable. Verify the installed forwarding state rather than relying on neighbor status alone.

6.6 — MAC Learning and Forwarding Validation

Once the control plane is operational, inspect the forwarding tables to confirm that endpoint reachability has been programmed correctly.

Local MAC addresses should resolve toward access interfaces. Remote MAC addresses should resolve toward the appropriate remote VTEP.

Endpoint Forwarding Verification

show mac address-table

show l2route evpn mac all

show nve peers

show nve vni

show ip arp

show ip route vrf TENANT-A

Inspect local versus remote MAC entries, VNI association, ARP state and tenant routes. VRF-specific commands apply when Layer 3 EVPN routing is configured.

LOCAL ENDPOINT

Access-Side Learning

Confirm that the endpoint MAC is associated with the correct VLAN and local interface.

REMOTE ENDPOINT

EVPN Remote MAC Learning

Confirm that the remote MAC is associated with the expected VNI and remote VTEP destination.

6.7 — Packet Capture and VXLAN Encapsulation

Packet captures are particularly useful when the control plane appears healthy but data traffic still fails.

A VXLAN packet typically contains an outer Ethernet header, outer IP header, UDP header, VXLAN header and original inner Ethernet frame.

The standard VXLAN UDP destination port is 4789. The VXLAN header carries the VNI identifying the relevant overlay segment.

Packet Field	Example	What to Verify
Outer source IP	10.0.0.11	Correct ingress VTEP
Outer destination IP	10.0.0.12	Correct remote VTEP
UDP destination	4789	VXLAN transport
VNI	10010	Correct overlay segment
Inner source IP	192.168.10.11	Original endpoint source
Inner destination IP	192.168.10.12	Original endpoint destination

Example Linux Packet Capture

sudo tcpdump -ni eth0 udp dst port 4789

sudo tcpdump -ni eth0 \
  'host 10.0.0.12 and udp port 4789' \
  -vv

Run on an interface where encapsulated VXLAN traffic is visible. Hardware-offloaded traffic may not appear in ordinary host or switch CPU captures. SPAN, ERSPAN or platform-specific capture facilities may be required.

Capture Location Matters

Capturing traffic at the server interface normally shows the original tenant packet. Capturing on an underlay link can reveal the VXLAN-encapsulated packet. These views answer different questions during troubleshooting.

6.8 — Common VXLAN Failure Scenarios

VXLAN incidents often fall into recognizable categories. The table below maps typical symptoms to likely areas of investigation.

Symptom	Possible Cause	Investigation
Remote VTEP unreachable	Underlay routing or physical link failure	Check routes, interfaces and source-aware ping
BGP EVPN neighbor down	BGP configuration, transport or policy issue	Inspect neighbor state, peer addresses and EVPN activation
Missing remote MAC	No Type 2 route or route import failure	Check EVPN routes, route targets and MAC learning
Broadcast traffic fails	Missing Type 3 membership or replication state	Inspect IMET routes, VNI state and BUM replication
Same-subnet works, inter-subnet fails	Gateway, VRF, L3 VNI or routing issue	Check SVI, tenant routes and symmetric IRB
Small packets work, large packets fail	MTU mismatch or fragmentation issue	Test path MTU and inspect encapsulated sizes
Endpoint intermittently unreachable	MAC mobility, stale state or inconsistent forwarding	Check mobility events and forwarding updates
Only one VNI fails	Incorrect VLAN/VNI mapping or route target	Compare affected and healthy VNI configuration

6.9 — Seven-Stage VXLAN Troubleshooting Workflow

The most effective troubleshooting method is to locate the first point at which expected network state differs from actual state.

Begin with the physical transport and progress toward tenant forwarding.

Validate Physical and Routed Underlay

Confirm interface status, routing adjacencies, VTEP loopback reachability and expected ECMP paths.

Validate NVE and VNI State

Check the NVE source interface, operational status, VNI membership and VLAN mappings.

Validate BGP EVPN Neighbors

Confirm EVPN address-family sessions and route exchange with the expected peers or route reflectors.

Inspect EVPN Route Import

Verify relevant Type 2, Type 3 and, where used, Type 5 routes. Check route targets and import policy.

Inspect Forwarding Tables

Compare EVPN routes with remote MAC, ARP, VNI and tenant VRF forwarding state.

Trace the Encapsulated Packet

Verify outer VTEP addresses, UDP port, VNI and successful transport through the underlay.

Verify Egress Delivery and Policy

Confirm decapsulation, local VLAN forwarding, gateway processing, ACLs and destination endpoint reachability.

Engineering Troubleshooting Principle

Avoid changing several configuration elements simultaneously. Identify the failing layer, form a testable hypothesis and compare expected versus observed state before applying a fix.

6.10 — Example Incident: Remote Host Unreachable

Consider a VXLAN fabric where Host A on Leaf A cannot reach Host B on Leaf B.

Both hosts belong to VNI 10010. BGP EVPN sessions are Established, but traffic still fails.

Investigation Step	Observed Result	Interpretation
Underlay route to 10.0.0.12	Present	Remote VTEP route exists
BGP EVPN neighbor	Established	Control-plane transport is operational
EVPN Type 2 route	Remote Host B MAC advertised	Endpoint information is being distributed
Remote MAC forwarding	Entry absent from VNI 10010	Route not programmed into expected forwarding context
Route-target inspection	Import RT mismatch	Likely reason route was not imported

In this example, the underlay and BGP neighbor state are healthy. The failure occurs when the received EVPN route is not imported into the intended local forwarding context.

Correcting the route-target policy and validating the resulting remote MAC entry would be the appropriate next investigation step.

Avoid the Wrong Fix

Restarting BGP or modifying underlay routing would not directly address the route import mismatch identified in this example. The evidence points toward overlay policy and forwarding installation.

6.11 — VXLAN Performance and Scale

Beyond basic connectivity, production VXLAN fabrics must be designed for scale, predictable convergence and operational visibility.

Design Area	Operational Consideration
MAC scale	Monitor local and remote MAC table capacity
EVPN route scale	Monitor BGP EVPN routes, memory and route-reflector capacity
BUM traffic	Control unnecessary replication and broadcast amplification
ECMP	Validate load distribution across available spine paths
MTU	Ensure encapsulated packets fit within the supported underlay MTU
MAC mobility	Track workload movement and unexpected MAC flapping
Convergence	Test link, leaf and control-plane failure recovery
Security	Apply segmentation, route policy and appropriate access controls

6.12 — Pre-Production Validation Checklist

Before a VXLAN EVPN fabric enters production, validate the following operational requirements.

✓ All expected underlay routing adjacencies are operational.

✓ Every required VTEP loopback is reachable.

✓ Underlay MTU supports VXLAN encapsulation.

✓ VLAN-to-VNI mappings are consistent.

✓ NVE interfaces and required VNIs are operational.

✓ BGP EVPN neighbors are Established.

✓ Expected EVPN routes are received and imported.

✓ Remote MAC forwarding entries resolve correctly.

✓ Same-subnet connectivity works across VTEPs.

✓ Inter-subnet routing works where configured.

✓ Tenant isolation and security policy are verified.

✓ Link and node failure recovery has been tested.

Operational Baseline

Capture a known-good baseline of underlay routes, BGP EVPN neighbors, VNI state, MAC forwarding entries and tenant routing tables. Comparing healthy and failed states can substantially reduce incident investigation time.

Section 6 — Engineering Takeaways

✓ VXLAN EVPN deployment begins with a reliable routed IP underlay.

✓ VTEP loopback reachability, NVE state and VNI mappings must be correct before overlay forwarding can succeed.

✓ BGP EVPN sessions distribute endpoint and overlay reachability, but Established neighbors alone do not prove forwarding works.

✓ Remote MAC and tenant routing tables must be checked alongside EVPN control-plane advertisements.

✓ VXLAN packet captures reveal outer VTEP addresses, UDP transport and the overlay VNI.

✓ Route-target mismatches, incorrect VNI mappings, MTU problems and stale MAC state are common investigation areas.

✓ A structured troubleshooting workflow isolates underlay, control-plane and data-plane failures before configuration changes are made.

Next Learning Activity

Interactive VXLAN Engineering Labs and Troubleshooting Exercises

Network Insight · Interactive Engineering Lab 06

VXLAN EVPN CLI Verification and Troubleshooting Workbench

Investigate a broken VXLAN fabric using operational commands. Inspect the underlay, BGP EVPN sessions, NVE state, VNI mappings and remote MAC entries. Identify the fault, repair it and verify end-to-end connectivity.

Step 1 · Select Network Incident

Investigation Objective

Restore Host A → Host B

Underlay Down BGP EVPN Established NVE Up

VXLAN Fabric · VNI 10010 · AS 65001

Underlay Reachability Failed

EVPN Route State Installed

End-to-End Traffic Not Tested

LEAF-A · Simulated NX-OS CLI Operational Verification Console

Network Insight VXLAN EVPN Investigation Lab Leaf-A# Type help to list available commands.

Quick Verification Commands

Verification Checklist

Check	Operational State
Underlay VTEP route	Missing
BGP EVPN neighbor	Established
NVE interface	Up
VNI consistency	Matched
Remote MAC route	Installed

Incident Investigation Notes

Underlay Reachability Failure

The routed underlay has lost reachability to the remote VTEP. Inspect the route table and test connectivity to 10.0.0.12.

Start with show ip route 10.0.0.12.

Engineering Investigation Principle

Verify VXLAN systematically: underlay IP reachability first, then BGP EVPN control-plane state, NVE peer status, VNI configuration, remote MAC/IP reachability and finally end-to-end traffic. A successful BGP session alone does not guarantee working VXLAN forwarding.

Section 07 · Production Fabric Architecture

VXLAN Design Best Practices, Security and Production Readiness

VXLAN EVPN provides a scalable framework for extending Layer 2 networks and building distributed Layer 3 overlays. However, deploying the technology successfully requires careful planning around segmentation, resilience, control-plane security, operational scale and automation.

The goal of production design is not simply to establish VXLAN connectivity. It is to build a fabric that remains predictable during failures, supports controlled tenant communication and provides sufficient visibility to diagnose problems quickly.

Fabric Architecture Segmentation EVPN Security High Availability Scale Automation Observability Production Readiness

7.1 — Principles of a Production VXLAN Fabric

A production VXLAN EVPN fabric should separate physical transport from tenant forwarding and use consistent design standards across all participating leaf switches.

This separation allows the underlay to focus on resilient IP transport while the overlay manages tenant segments, endpoint reachability and distributed routing.

PRINCIPLE 01

Keep the Underlay Simple

Use a predictable routed leaf-spine design with clear loopback addressing, ECMP and resilient VTEP reachability.

PRINCIPLE 02

Standardize Overlay Mappings

Document VLAN, L2 VNI, VRF, L3 VNI and route-target assignments.

PRINCIPLE 03

Limit Failure Domains

Avoid unnecessary Layer 2 extension and design explicit boundaries for tenant and application networks.

PRINCIPLE 04

Engineer for Visibility

Monitor underlay routing, EVPN advertisements, VNI state, endpoint mobility and forwarding behavior.

Design Objective

A well-designed fabric should make failures easier to isolate, not merely provide more network scale. Clear separation between underlay, EVPN control plane and tenant forwarding is central to operational simplicity.

7.2 — Reference Production Fabric Architecture

The following conceptual topology illustrates two tenants sharing a routed leaf-spine underlay while using separate overlay routing contexts.

Reference Architecture · Shared Underlay, Isolated Overlays

Conceptual multi-tenant design. The same physical leaf-spine underlay carries multiple VXLAN overlays. Tenant isolation is provided through separate forwarding contexts and controlled EVPN route import. The drawing is logical rather than a complete physical cabling diagram.

7.3 — Segmentation and Tenant Isolation

VXLAN enables multiple logical networks to share the same physical infrastructure. VNIs identify Layer 2 segments, while VRFs provide separate Layer 3 routing contexts.

This allows applications or tenants to maintain independent forwarding domains without requiring dedicated physical network hardware for every environment.

Segmentation Layer	Mechanism	Purpose
Layer 2	L2 VNI	Separate Ethernet bridge domains
Layer 3	VRF and L3 VNI	Separate tenant routing contexts
EVPN control plane	Route targets	Control which routes are imported
Security policy	ACLs / Firewalls	Restrict permitted application flows
Inter-tenant services	Controlled route leaking and service insertion	Permit explicitly approved communication

VXLAN Does Not Provide Encryption by Default

VXLAN encapsulates Ethernet traffic but does not inherently encrypt the payload or authenticate VTEPs. Where confidentiality or transport integrity is required, additional protections such as IPsec, MACsec or an appropriately secured transport environment must be considered.

7.4 — EVPN Control-Plane Security

BGP EVPN distributes information that influences endpoint reachability and tenant forwarding.

Protecting the EVPN control plane is therefore an important part of fabric security.

BGP PEERING

Restrict Neighbor Access

Limit EVPN BGP peering to approved network devices and routing endpoints.

ROUTE POLICY

Control Route Import

Validate route targets, tenant boundaries and route-leaking requirements.

MANAGEMENT

Protect Management Access

Use authenticated, authorized and monitored access for configuration and operational tools.

MONITORING

Detect Unexpected Changes

Track unusual EVPN advertisements, MAC mobility and unexpected routing changes.

Security Boundary

Route targets are route distribution controls, not a substitute for firewall enforcement. A complete security design must also account for endpoint access, policy enforcement and authorized connectivity between routing domains.

7.5 — High Availability and Failure Domains

Leaf-spine architectures are commonly designed with redundant physical paths.

Equal-Cost Multipath routing can distribute underlay traffic across available next hops, while the routing protocol reacts to link or node failures.

VXLAN EVPN overlays depend on this underlay resilience to maintain connectivity between VTEPs.

Failure Scenario	Expected Design Behavior
Single spine link failure	Underlay converges onto remaining valid paths
Spine switch failure	Reachability continues through surviving spine paths where capacity permits
Leaf failure	Dual-attached endpoints may retain service through an alternate leaf, depending on multihoming design
EVPN peer failure	Redundant control-plane peers support route distribution continuity
Endpoint migration	EVPN MAC mobility procedures update endpoint reachability

Resilience Must Be Tested

Redundant topology does not guarantee zero packet loss or instantaneous convergence. Validate actual failure recovery, forwarding behavior, convergence time and available bandwidth during degraded operation.

7.6 — EVPN Scale and Capacity Planning

VXLAN supports a 24-bit VNI field, but the theoretical identifier space should not be confused with the practical scale supported by a specific network platform.

Real fabric scale is influenced by hardware forwarding capacity, EVPN route count, MAC and ARP table resources, tenant VRFs and operational control-plane limits.

Scale Dimension	What to Measure
VNIs	Supported L2 and L3 VNI capacity
MAC entries	Local and remote MAC forwarding state
EVPN routes	Route count, memory and convergence load
ARP / ND entries	Endpoint neighbor resolution state
VRFs	Supported tenant routing instances
ECMP paths	Hardware path capacity and hash distribution
BUM replication	Replication fan-out and bandwidth impact

7.7 — Automation and Configuration Consistency

Manual VXLAN EVPN deployments can become difficult to maintain when many leaf switches, tenants and VNIs are involved.

Automation helps maintain consistent configuration standards and reduces the risk of mapping or route-target errors.

Define the Intended Network State

Maintain a source of truth for devices, loopbacks, VLANs, VNIs, VRFs and route targets.

Generate Consistent Configurations

Use templates or automation tooling to apply approved fabric design standards.

Validate Before Deployment

Check duplicate identifiers, missing mappings, route-target inconsistencies and unsupported platform features.

Compare Intended and Actual State

Verify operational NVE, BGP EVPN, routing and forwarding state after each deployment.

Automation Principle

Configuration deployment is only one part of automation. Production workflows should also validate the resulting network state and detect drift from the intended fabric design.

7.8 — VXLAN Observability and Analytics

Modern data centre operations require visibility across the underlay, EVPN control plane and tenant data plane.

Useful telemetry includes interface utilization, ECMP distribution, route changes, endpoint mobility, packet drops and tunnel health.

Telemetry Area	Operational Value
Underlay interfaces	Detect congestion, errors and link failures
Routing adjacencies	Identify unstable underlay or EVPN sessions
EVPN route changes	Correlate route withdrawals with service disruption
MAC mobility	Detect workload movement or unexpected MAC flapping
VNI statistics	Monitor overlay forwarding and traffic distribution
Queue depth / drops	Identify congestion and microburst effects
Packet traces	Validate actual encapsulation and delivery paths

Analytics Do Not Automatically Optimize VXLAN

Telemetry provides information about network behavior. Automated optimization requires additional control systems, supported platform capabilities and explicit operational policy. VXLAN encapsulation itself does not automatically rebalance or optimize traffic.

7.9 — Recommended Engineering Validation Commands

The following Cisco NX-OS-style commands provide a useful starting point for production-state validation.

VXLAN EVPN Production Verification

show ip route

show ip interface brief

show nve interface

show nve peers

show nve vni

show bgp l2vpn evpn summary

show bgp l2vpn evpn route-type 2

show bgp l2vpn evpn route-type 3

show bgp l2vpn evpn route-type 5

show l2route evpn mac all

show mac address-table

show vrf

show ip route vrf TENANT-A

show interface counters errors

Illustrative verification commands. Exact syntax, feature support and output depend on the hardware platform and software release.

7.10 — Production Architecture Decision Matrix

Not every VXLAN EVPN deployment needs the same design choices. The correct architecture depends on workload mobility, tenant isolation, operational scale and resilience requirements.

Decision	Recommended Evaluation
Underlay protocol	Choose based on operational familiarity, scale and convergence requirements
EVPN peering model	Evaluate route reflectors, redundancy and control-plane scale
IRB model	Compare symmetric and asymmetric forwarding requirements
Tenant segmentation	Define VRF boundaries, route import and firewall enforcement
BUM handling	Select supported ingress replication or multicast transport models
High availability	Evaluate dual attachment, multihoming and failover behavior
Automation	Establish a source of truth and validation workflow
Observability	Define operational metrics, baselines and alert thresholds

7.11 — Production Readiness Checklist

✓ Underlay routing is resilient and VTEP reachability verified.

✓ VLAN, VNI and VRF mappings are documented and consistent.

✓ EVPN peering and route-target policies are validated.

✓ Tenant isolation and permitted route leaking are tested.

✓ BGP and management access are appropriately protected.

✓ Hardware MAC, route, VNI and VRF scale is understood.

✓ Underlay MTU accommodates VXLAN overhead.

✓ Failure recovery and degraded-capacity operation are tested.

✓ Monitoring covers underlay, EVPN and data-plane state.

✓ Automation includes pre-deployment and post-deployment validation.

Section 7 — Engineering Takeaways

✓ Production VXLAN fabrics require consistent design across underlay, overlay and tenant routing.

✓ VNIs, VRFs and route targets support logical segmentation, but security policy must be enforced separately.

✓ VXLAN does not inherently provide encryption or endpoint authentication.

✓ Fabric scale is constrained by actual platform resources, not simply the size of the VNI field.

✓ Redundancy, ECMP and EVPN convergence should be validated under realistic failure conditions.

✓ Automation and telemetry improve consistency and operational visibility when combined with explicit validation and control policies.

Continue Learning

VXLAN Interactive Architecture and Engineering Labs

Network Insight · Interactive Engineering Lab 07

VXLAN EVPN Fabric Design and Production Readiness

Act as the network architect. Configure resilience, security, routing and observability controls, test failure scenarios and evaluate whether your VXLAN EVPN fabric is ready for production.

Dual-Spine EVPN VXLAN Fabric Design Assessment

Design Readiness Score 0%

Engineering Controls Enabled 0 / 8

Production Assessment Review

1. Configure the VXLAN Fabric

Enable or disable individual design controls. Each decision affects the engineering readiness assessment and selected resilience tests.

01 · ECMP Underlay Resilience

Maintain multiple equal-cost routed paths between VTEPs through independent spines.

02 · BGP EVPN Control Plane

Distribute MAC/IP reachability and VNI membership through BGP EVPN.

03 · Underlay MTU Headroom

Allow sufficient MTU for VXLAN encapsulation without unwanted fragmentation.

04 · Tenant VRF Isolation

Separate tenant routing domains and control inter-VRF connectivity.

05 · Distributed Anycast Gateway

Provide consistent default gateways on participating leaf switches.

06 · Tenant Security Policy

Apply explicit filtering and restrict unauthorized tenant-to-tenant traffic.

07 · Fabric Telemetry

Monitor BGP, NVE, interfaces, MAC mobility and forwarding health.

08 · ARP Suppression

Use verified EVPN MAC/IP bindings to reduce unnecessary ARP flooding.

2. Run Production Failure Tests

Select an engineering scenario to assess how your chosen design would behave.

Fabric Design Assessment and Event Report

[READY] Fabric architecture loaded. [READY] Configure design controls. [READY] Run a failure scenario or generate a report.

Design Assessment Ready

Enable the appropriate engineering controls and run resilience tests to evaluate your VXLAN EVPN fabric.

Section 08 · Advanced VXLAN EVPN Architecture

VXLAN EVPN Multi-Site, Data Centre Interconnect and Migration

VXLAN EVPN is not limited to a single leaf-spine fabric. With an appropriate multi-site architecture, organizations can connect data centres, extend selected network segments, distribute routing information and support workload mobility across geographically separate environments.

However, interconnecting VXLAN fabrics introduces design challenges that do not exist within a single data centre. These include inter-site latency, failure isolation, route distribution, gateway placement, broadcast containment and disaster recovery.

The objective is to extend only the network services that require cross-site connectivity, while preserving predictable routing and independent site operation.

VXLAN EVPN Multi-Site Data Centre Interconnect Border Gateways Workload Mobility Distributed Anycast Gateway Layer 2 Extension Migration Disaster Recovery

8.1 — Why Extend VXLAN Between Data Centres?

Enterprises often operate multiple data centres for capacity, availability, geographic coverage or disaster recovery.

VXLAN EVPN can form part of a DCI design that supports selected Layer 2 extensions and Layer 3 connectivity between sites.

USE CASE 01

Workload Mobility

Support workloads that require selected network identities or subnets to remain reachable across sites during planned migration.

USE CASE 02

Disaster Recovery

Provide network connectivity for applications and services activated in an alternate data centre.

USE CASE 03

Application Distribution

Enable controlled communication between application components located in different data centres.

USE CASE 04

Infrastructure Modernization

Connect legacy networks with modern EVPN fabrics during a phased migration.

Design Recommendation

Prefer routed inter-site connectivity when application requirements allow it. Extend Layer 2 only where there is a clear operational or application dependency. Unnecessary Layer 2 extension increases the scope of broadcast and failure domains.

8.2 — VXLAN EVPN Multi-Site Architecture

A multi-site architecture connects separate data centre fabrics through an inter-site transport network.

Depending on the vendor architecture, dedicated or designated border gateways can exchange overlay reachability and forward VXLAN traffic between sites.

Reference Architecture · Two VXLAN EVPN Data Centres

Site underlay Inter-site overlay Tenant attachment

Simplified logical architecture. Border gateways may be implemented as dedicated devices or designated leaf roles. Actual multi-site designs require additional redundancy, routing policy and vendor-specific inter-site mechanisms.

8.3 — The Role of Border Gateways

Border gateways provide a controlled point of connectivity between a local VXLAN fabric and the DCI transport network.

In supported multi-site architectures, they can exchange EVPN reachability information and forward encapsulated traffic between participating sites.

Border Gateway Function	Engineering Purpose
Inter-site forwarding	Carry selected overlay traffic between fabrics
EVPN route exchange	Advertise or import permitted endpoint and tenant routes
Policy enforcement	Restrict inter-site route distribution and connectivity
BUM containment	Control replication behavior between participating sites
Failure isolation	Limit the impact of remote site or transport disruptions
Redundancy	Support resilient inter-site paths when multiple gateways are deployed

Multi-Site Is Architecture-Specific

The term VXLAN EVPN Multi-Site is also used for particular vendor implementations. Border-gateway roles, control-plane behavior, anycast addressing, BUM replication and failure handling vary by platform and design. Verify the supported architecture rather than assuming that two independent EVPN fabrics can simply be connected without additional configuration.

8.4 — Layer 2 Extension Versus Layer 3 Interconnect

One of the most important DCI decisions is whether a subnet must extend across sites.

Layer 2 extension allows endpoints in separate data centres to participate in the same bridge domain.

Layer 3 interconnect instead routes between distinct subnets and routing domains.

Design Characteristic	Layer 2 Extension	Layer 3 Interconnect
Subnet model	Shared subnet	Separate subnets
Broadcast domain	Extends between sites	Remains site-local
Workload mobility	Can preserve an IP subnet during migration	Usually requires address or application-level mobility planning
Failure domain	Potentially wider	More clearly bounded
Operational complexity	Greater dependency on MAC and BUM behavior	Primarily routed reachability and policy
Typical preference	Specific mobility requirements	General-purpose inter-site connectivity

Recommended Architecture

Use Layer 3 boundaries by default and introduce stretched Layer 2 segments only when justified. This reduces unnecessary broadcast extension and simplifies site-level fault isolation.

8.5 — Distributed Anycast Gateway Across Sites

A distributed anycast gateway allows multiple participating leaf switches to provide a common default gateway identity for a tenant subnet.

In a supported multi-site design, this can allow workloads to use local first-hop routing even when the subnet is present at multiple sites.

Gateway consistency is critical. IP addressing, gateway MAC behavior, EVPN advertisements and tenant routing policies must follow the selected architecture.

LOCAL ROUTING

First-Hop Gateway

The local leaf can route traffic using the distributed gateway rather than sending every packet to a remote site.

CONSISTENCY

Shared Gateway Identity

Gateway addresses and MAC behavior must be consistent with the supported anycast gateway implementation.

ROUTING POLICY

Symmetric IRB

Symmetric integrated routing and bridging can support scalable routed tenant connectivity through L3 VNIs.

FAILURE RECOVERY

Site Independence

Design gateway and route behavior so local services can continue where feasible during DCI failure.

8.6 — Workload Mobility and MAC Mobility

EVPN includes mechanisms for advertising MAC reachability and signaling endpoint mobility.

When a workload moves from one VTEP to another, the control plane must update reachability so remote devices forward traffic toward the new location.

Workload Located in Site A

The local VTEP learns the endpoint and advertises the appropriate EVPN MAC/IP reachability.

Workload Migrates to Site B

The endpoint becomes attached to a VTEP in the destination site.

EVPN Reachability Changes

The new location advertises updated reachability. MAC mobility procedures help remote devices distinguish the newer location.

Remote Forwarding Updates

Participating VTEPs update forwarding state to direct traffic toward the destination site.

Validate Application Continuity

Test routing, ARP/ND state, firewall sessions, load balancers and application dependencies.

Mobility Does Not Guarantee Session Continuity

EVPN can update network reachability after endpoint movement, but application continuity also depends on transport sessions, firewall state, load balancing, storage and application architecture. VXLAN alone does not guarantee seamless migration.

8.7 — Inter-Site Failure and Disaster Recovery

DCI failure must be considered separately from failures inside a single data centre fabric.

Each site should have a defined behavior when the inter-site network becomes unavailable.

Failure	Potential Impact	Design Consideration
DCI link failure	Remote overlays become unreachable	Local routing independence and alternate paths
Border gateway failure	Inter-site forwarding is disrupted	Redundant gateways and convergence
Complete site outage	Workloads and network services may be unavailable	Recovery orchestration, route changes and application failover
Control-plane instability	EVPN route churn or inconsistent forwarding	Route filtering, monitoring and failure isolation
Extended Layer 2 fault	Broadcast or endpoint-learning disruption	Minimize stretched segments and control BUM traffic

Split-Brain Risk

When two sites lose communication but continue operating independently, application and network failover mechanisms must avoid conflicting active ownership. VXLAN EVPN connectivity is not a substitute for application-level quorum, storage consistency or disaster recovery coordination.

8.8 — Migrating from Traditional VLAN Networks

Moving from a traditional Layer 2 network to VXLAN EVPN is usually best performed incrementally.

A phased migration allows engineers to validate the new fabric while preserving existing application connectivity.

Assess the Existing Network

Document VLANs, subnets, default gateways, trunks, spanning tree, routing and application dependencies.

Build and Validate the Underlay

Establish routed leaf-spine connectivity, loopback reachability, ECMP and MTU consistency.

Deploy the EVPN Overlay

Configure VTEPs, L2 VNIs, tenant VRFs, L3 VNIs and BGP EVPN route exchange.

Establish a Controlled Migration Boundary

Use supported Layer 2 or Layer 3 handoff mechanisms between legacy and EVPN environments. Avoid uncontrolled loops and duplicate gateways.

Migrate a Pilot Workload

Test a limited application or subnet, validate endpoint learning and confirm expected routing and security policy.

Expand in Controlled Phases

Migrate additional segments using documented change windows, validation tests and rollback plans.

Remove Legacy Dependencies

Retire unnecessary VLAN extensions, legacy trunks and obsolete gateway paths after successful validation.

8.9 — Example Migration Mapping

A VLAN-to-VNI mapping plan provides a consistent reference when moving network segments into the EVPN fabric.

Existing VLAN	New L2 VNI	Tenant VRF	Migration Approach
10	10010	TENANT-A	Pilot application migration
20	10020	TENANT-A	Phased server migration
30	20030	TENANT-B	Separate tenant cutover
40	20040	TENANT-B	Gateway migration after validation

Gateway Migration Requires Special Care

Moving the default gateway from a traditional switch to a distributed anycast gateway must be planned carefully. Duplicate active gateway ownership, inconsistent ARP behavior or incorrect routing boundaries can disrupt application traffic during cutover.

8.10 — Multi-Site Operational Verification

Engineers should verify both the local EVPN fabric and the inter-site transport.

The following commands illustrate common checks on Cisco NX-OS VXLAN EVPN deployments.

VXLAN EVPN / DCI Verification

show ip route

show nve interface

show nve peers

show nve vni

show bgp l2vpn evpn summary

show bgp l2vpn evpn route-type 2

show bgp l2vpn evpn route-type 3

show bgp l2vpn evpn route-type 5

show l2route evpn mac all

show mac address-table

show vrf

show ip route vrf TENANT-A

Generic NX-OS-style commands. Vendor-specific multi-site features may require additional border-gateway, site identifier, route-policy and inter-site peer verification commands. Availability varies by software release.

Validation Layer	What to Confirm
Local underlay	VTEP reachability and stable routing adjacencies
DCI transport	Border gateway connectivity, MTU and permitted traffic
EVPN control plane	Expected inter-site route exchange
Route policy	Correct import, export and tenant boundaries
Data plane	Encapsulation, remote VTEP forwarding and endpoint delivery
Failure recovery	Behavior during DCI or border gateway outage

8.11 — Multi-Site Design Checklist

✓ Each site has a validated local underlay and overlay.

✓ DCI routing, MTU and transport capacity are verified.

✓ Border gateway redundancy and failover are tested.

✓ Inter-site EVPN route distribution follows approved policy.

✓ Layer 2 extension is limited to justified application requirements.

✓ Anycast gateway behavior is consistent across participating sites.

✓ MAC mobility and endpoint movement are tested.

✓ Site isolation and DCI failure behavior are documented.

✓ Application and firewall dependencies are included in migration testing.

✓ Migration rollback and disaster recovery procedures are validated.

Section 8 — Engineering Takeaways

✓ VXLAN EVPN can support selected inter-site Layer 2 and Layer 3 connectivity when deployed using a supported DCI architecture.

✓ Border gateways can provide controlled inter-site forwarding and route exchange.

✓ Routed interconnects should generally be preferred over unnecessary Layer 2 extension.

✓ Distributed anycast gateways and EVPN MAC mobility can support workload movement, but application continuity requires additional planning.

✓ Multi-site designs must account for DCI failures, site isolation, route policy and disaster recovery coordination.

✓ Legacy VLAN migration should follow a phased approach with explicit gateway ownership, validation and rollback procedures.

Next Learning Activity

VXLAN EVPN Multi-Site Architecture and Migration Investigation Lab

Network Insight · Interactive Engineering Lab 08

VXLAN EVPN Multi-Site, Data Centre Interconnect and Migration Lab

Connect two data centres, compare inter-site architectures, investigate DCI failures and plan a controlled migration from legacy Layer 2 networking to VXLAN EVPN.

Step 1 · Select the Inter-Site Architecture

Two-Site VXLAN EVPN Fabric EVPN Multi-Site

DCI Architecture EVPN Multi-Site

Inter-Site Paths 2 Available

Traffic Result Ready

Step 2 · Inject a Failure Failure Scenario

Both DCI paths are operational.

Architecture Analysis

EVPN Multi-Site

Border gateways connect independent EVPN fabrics while maintaining inter-site reachability for selected tenant networks.

Inter-Site Packet and Failure Investigation Timeline

[READY] DC1 and DC2 initialized. [READY] EVPN Multi-Site selected. [READY] Both DCI paths available. [READY] Trace an inter-site packet.

Step 3 · Migration Planning

Legacy Layer 2 to VXLAN EVPN Migration

Work through five controlled migration stages. Review the required checks before approving each stage.

Migration Stage 01 / 05

Discovery and Dependency Mapping

Identify VLANs, gateways, IP subnets, application dependencies and existing failure domains before migration.

Complete the stage checklist before approval.

Engineering Design Principle

Extending Layer 2 between data centres increases the scope of broadcast and failure domains. Routed DCI is generally preferable when applications do not require subnet continuity. EVPN Multi-Site can support selected extensions, but border-gateway redundancy, loop prevention, failure isolation and operational validation remain essential.

Section 09 · Final Engineering Summary

VXLAN: Engineering Summary, Key Takeaways and Next Steps

VXLAN has become an important building block of modern data centre network architecture. By transporting Ethernet segments across an IP underlay, VXLAN enables scalable overlay networks without requiring the physical fabric to operate as one large Layer 2 domain.

When combined with BGP EVPN, VXLAN supports control-plane distribution of endpoint reachability, multi-tenant segmentation, distributed routing and advanced leaf-spine network designs.

This final section brings together the central engineering concepts explored throughout the article.

Underlay Overlay VTEPs Encapsulation BGP EVPN Tenant Segmentation Operations Multi-Site

9.1 — The Complete VXLAN Architecture

A VXLAN EVPN fabric can be understood through four closely related architectural components.

FOUNDATION 01

IP Underlay

The routed physical network provides IP reachability between VTEPs. Routing protocols and ECMP support resilient transport.

FOUNDATION 02

VXLAN Overlay

VXLAN encapsulates Ethernet frames in UDP packets. The VNI identifies the associated overlay segment.

FOUNDATION 03

BGP EVPN Control Plane

EVPN distributes endpoint and routing information, reducing reliance on traditional data-plane learning across the overlay.

FOUNDATION 04

Distributed Tenant Routing

VRFs, L2 VNIs, L3 VNIs and distributed gateways enable tenant-aware forwarding and inter-subnet routing.

Architecture Recap · From Workload to Workload

Conceptual same-subnet VXLAN forwarding. Leaf A encapsulates the original Ethernet frame, the IP underlay transports it to Leaf B, and Leaf B decapsulates it before delivering the frame to Host B.

9.2 — Traditional VLAN Networks Versus VXLAN EVPN

Traditional VLAN networks and VXLAN EVPN fabrics solve different scaling and operational challenges.

Characteristic	Traditional VLAN Network	VXLAN EVPN Fabric
Segmentation identifier	12-bit VLAN ID	24-bit VNI
Transport model	Ethernet switching	Overlay over routed IP
Loop avoidance	Often relies on spanning tree	Routed underlay avoids Layer 2 loops in the core
Multipath forwarding	Depends on Layer 2 design	Underlay ECMP can use multiple routed paths
Endpoint reachability	Primarily data-plane MAC learning	EVPN control-plane advertisements plus local learning
Tenant routing	Traditional SVI/router model	VRFs and distributed IRB
Multi-site connectivity	Requires suitable DCI technology	Supported EVPN DCI architectures
Operational complexity	Simpler for small deployments	More components and control-plane state to manage

Important Distinction

VXLAN is the data-plane encapsulation technology. BGP EVPN is a separate control-plane technology commonly used to distribute endpoint and routing information. They work together, but they are not the same protocol.

9.3 — The End-to-End Packet Journey

Understanding a complete forwarding sequence is one of the most useful skills for troubleshooting VXLAN networks.

Ingress

Host A sends an Ethernet frame to Leaf A.

Lookup

Leaf A resolves the destination MAC and remote VTEP mapping.

Encapsulation

Leaf A adds VXLAN, UDP and outer IP headers.

Transport

The underlay routes traffic toward the remote VTEP.

Delivery

Leaf B removes encapsulation and forwards the original frame.

Engineering Mental Model

When a VXLAN packet fails, investigate the path in layers: local VLAN and VNI mapping, endpoint learning, EVPN reachability, remote VTEP connectivity, underlay routing, encapsulation and destination forwarding.

9.4 — The Most Important VXLAN Design Decisions

Successful deployments depend on a series of architecture choices rather than VXLAN encapsulation alone.

DESIGN 01

Underlay Routing

Select a suitable routing protocol, define loopback addressing and validate ECMP and convergence.

DESIGN 02

VNI and VRF Allocation

Maintain consistent mappings and clear tenant routing boundaries.

DESIGN 03

EVPN Route Policy

Define route import and export behavior, route reflectors and supported EVPN route types.

DESIGN 04

Distributed Gateway

Choose a supported IRB architecture and validate first-hop gateway behavior.

DESIGN 05

Security and Segmentation

Use VRFs, route policy and security enforcement to control tenant connectivity.

DESIGN 06

Operational Visibility

Monitor EVPN sessions, VTEPs, MAC mobility, underlay health and packet drops.

9.5 — Production Engineering Checklist

Before deploying a VXLAN EVPN fabric into production, validate the following core requirements.

✓ Routed underlay provides reliable VTEP reachability.

✓ Underlay MTU supports VXLAN encapsulation overhead.

✓ VLAN-to-VNI mappings are correct and consistent.

✓ EVPN BGP sessions are established and stable.

✓ Expected MAC/IP advertisements are present.

✓ VRF and route-target policies are validated.

✓ Distributed gateway behavior is tested.

✓ Tenant isolation and security policies work.

✓ ECMP forwarding and failover are verified.

✓ MAC mobility and endpoint movement are tested where required.

✓ Hardware forwarding and route scale are understood.

✓ Monitoring, rollback and recovery procedures are documented.

Final Security Reminder

VXLAN does not inherently encrypt traffic or authenticate tunnel endpoints. Route targets are not firewall rules, and logical segmentation alone does not guarantee a complete security boundary. Apply appropriate transport protection and traffic enforcement according to your security requirements.

9.6 — Final Engineering Takeaways

What You Should Remember About VXLAN

✓ VXLAN extends Layer 2 segments across a routed IP infrastructure.

✓ The 24-bit VNI supports a much larger segmentation identifier space than traditional VLAN IDs.

✓ VTEPs perform encapsulation and decapsulation at the overlay boundary.

✓ BGP EVPN provides a scalable control plane for distributing MAC/IP and routing information.

✓ VRFs, L2 VNIs, L3 VNIs and distributed gateways support multi-tenant routing and segmentation.

✓ Production VXLAN requires sound underlay design, MTU planning, security controls, monitoring and failure testing.

✓ Multi-site EVPN can support selected DCI and workload mobility requirements, but should not replace careful application and disaster recovery design.

9.7 — Continue Your VXLAN Learning Journey

Understanding VXLAN architecture is the first step. The next stage is applying these concepts through topology investigation, packet tracing and realistic network engineering scenarios.

Network Insight · Continue Learning

Move from VXLAN Theory to Network Engineering Practice

Explore the wider data centre networking architecture, including leaf-spine design, BGP, routing, traffic investigation and modern overlay technologies.

Explore Data Centre Workspace → Review VXLAN Fundamentals ↺

End of VXLAN Editorial Learning Path. You have completed the nine-section progression from traditional Ethernet limitations to production VXLAN EVPN architecture, operations and multi-site design.

Network Insight · Interactive Engineering Lab 09

VXLAN EVPN Engineering Capstone: Diagnose, Repair and Validate

Complete six engineering investigations covering underlay architecture, VXLAN encapsulation, EVPN control-plane routes, distributed routing, troubleshooting and production design. Use the simulated CLI, select your engineering decisions and complete the final assessment.

Engineering Score 0 / 600

Stages Completed 0 / 6

Current Stage 01

Assessment Status In Progress

Engineering Investigation Fabric Investigation Active

Stage 01 / 06 · Underlay Architecture

Verify the VXLAN Underlay

The EVPN overlay depends on routed IP reachability between VTEPs. Investigate the fabric before making your engineering decision.

Which underlay condition is required before VXLAN encapsulated traffic can be forwarded?

Investigation Ready

Review the evidence and select the most appropriate engineering decision.

Simulated Operational CLI

Use NX-OS-style verification commands to inspect the current scenario. Type help for available commands.

LEAF-A · Engineering Console

Network Insight VXLAN EVPN Capstone Leaf-A#

Engineering Investigation Timeline

[READY] Capstone initialized. [READY] Stage 01 loaded. [READY] Review evidence and submit your decision.

Final Engineering Assessment

VXLAN EVPN Capstone Complete

0 / 600

☕

Support Network Insight

Help support the development of interactive networking labs, BGP simulations, and educational content.

Support the Project

Network Insight · Continue Your Learning Journey

From VXLAN Theory to Data Centre Engineering

You've explored VXLAN encapsulation, BGP EVPN, distributed routing and fabric design. Now take the next step into an interactive data centre networking environment.

Data Centre
Workspace Explore · Trace · Diagnose

Leaf-Spine
Fabrics

VXLAN
Overlays

BGP
Routing

Traffic
Investigation

Fabric
Resilience

Explore network architecture, follow packet paths, investigate failures and practise engineering decisions through guided interactive activities.

Explore the Data Centre Network Workspace

Network Insight · Interactive Learning Workspace

From theory to interactive practice

Don't Just Read the Network. Explore It.

You've explored data centre architecture, routing and resilient design. Now take the next step. Use the interactive simulator to investigate how network components and paths work together. Move beyond diagrams and start exploring network behaviour.

Network topology Traffic paths Routing decisions Failure scenarios

LAUNCH THE BGP CORE SIMULATOR Opens in a new browser tab.

Network in motion Illustrative

STEP 01 / OBSERVE Inspect the network Explore the topology, devices and available network state.

STEP 02 / TRACE Follow the behaviour Investigate routes, forwarding decisions and the paths traffic can take.

STEP 03 / INVESTIGATE Think like an engineer Use what you observe to reason about changes, redundancy and failures.

The topology above is illustrative. Open the simulator to explore the functionality it provides.

Data Center Topologies

August 14, 2014

by Matt Conran Blog

Network Architecture / Data Center Design

Data Center Topology: From Three-Tier Networks to Leaf-Spine Fabrics

Data center topology defines how switches, servers, storage systems and external networks connect—and ultimately how application traffic moves through the infrastructure. The architecture you choose influences latency, available bandwidth, resilience, operational complexity and the ability to scale as workloads grow.

Traditional hierarchical networks remain relevant in many environments, but modern cloud platforms and distributed applications increasingly rely on scalable Layer 3 fabrics, leaf-spine architectures and equal-cost multipath routing. Understanding these designs means looking beyond the network diagram to examine routing behaviour, traffic patterns, failure domains and the control plane that keeps the fabric connected.

Design principle: build around application requirements

A good topology is not simply the one with the most links or the most switches. It is the one that delivers the required capacity, predictable forwarding, appropriate redundancy and manageable failure behaviour at an acceptable cost.

01 / STRUCTURE

Topology and Fabric Design

Compare three-tier, centralized, top-of-rack, end-of-row and leaf-spine architectures.

02 / ROUTING

BGP and ECMP

Explore routing adjacencies, route selection, multipath forwarding and convergence during failures.

03 / OVERLAYS

VXLAN and EVPN

Understand how Layer 3 underlays support virtual networks and distributed endpoint reachability.

04 / OPERATIONS

Capacity and Resilience

Evaluate oversubscription, traffic distribution, observability and failure-domain behaviour.

In this guide: Work from fundamental topology choices to modern data center fabrics, BGP-based routing, traffic engineering considerations and practical troubleshooting.

NETWORK INSIGHT · FIELD GUIDE

Explore Data Center Topology

Navigate the architecture, routing, traffic flows and resilience principles behind modern data center networks.

06 sections

01 / ARCHITECTURE FUNDAMENTALS

Understanding Data Center Architectures

Data center topology defines how servers, storage systems, switches and external networks connect. The architecture determines how traffic moves through the infrastructure and influences available bandwidth, latency, redundancy, scalability and operational complexity.

This section introduces traditional three-tier networks, centralized switching, top-of-rack (ToR), end-of-row (EoR) and leaf-spine architectures. Compare their connectivity models and trade-offs to understand which designs best match different workload and infrastructure requirements.

Engineering focus Choose topology according to traffic patterns, growth expectations, capacity, failure domains and the cost of operating the network.

Section 01 / Architecture Fundamentals

The Main Data Center Architectures

Data center networks can be built using hierarchical switching layers, centralized switching platforms, distributed access switches or fabric-based designs. Each approach creates a different balance between cabling, port utilization, bandwidth, failure isolation and operational complexity.

The right starting point is the workload. A smaller environment with limited server-to-server traffic may not need the same architecture as a cloud platform supporting thousands of virtual machines, containers or data-intensive applications. Topology selection should reflect traffic patterns, growth expectations, availability targets and the cost of operating the network over time.

Common architectural approaches

Hierarchical

Traditional Three-Tier Architecture

A traditional enterprise design separates access, aggregation or distribution, and core functions. Access switches connect servers, the aggregation layer concentrates traffic and may enforce policy, and the core provides high-speed connectivity between network blocks.

This model remains useful where existing infrastructure, established operational practices or clear policy boundaries make a hierarchical design appropriate. However, multiple switching tiers can increase path length and create bottlenecks when east-west traffic grows.

Centralized

Centralized Switching

In a centralized design, servers or server racks connect back to a central switching location. Concentrating switching resources can simplify management and make port allocation efficient in smaller environments.

As the facility grows, cable lengths, pathway congestion, centralized capacity limits and the consequences of equipment failure must be evaluated carefully.

Distributed Access

Top-of-Rack and End-of-Row

Top-of-rack (ToR) places access switches in individual server racks. End-of-row (EoR) places switching equipment in a dedicated rack serving a row of servers. Both approaches can provide redundant connectivity when designed with suitable independent paths and devices.

ToR generally shortens server cabling and can isolate access failures to a rack. EoR can consolidate switching resources, but may require more structured cabling. Neither approach automatically determines whether the wider network uses a three-tier or leaf-spine architecture.

Fabric-Based

Leaf-Spine Architecture

A leaf-spine fabric connects each leaf switch to multiple spine switches. Servers and other endpoints connect to the leaf layer, while the spine layer provides paths between different leaves. A Layer 3 underlay commonly uses equal-cost multipath routing to distribute eligible traffic across available paths.

This architecture provides predictable path structure and supports incremental expansion. Its capacity and resilience depend on link speeds, the number of spine switches, oversubscription, routing design and failure behaviour.

Important distinction: ToR and EoR describe where access switching is placed. Three-tier and leaf-spine describe broader network architectures. A data center can use ToR switches as its leaf layer in a leaf-spine fabric.

Compare the design trade-offs

Design consideration	Three-tier / centralized	Leaf-spine fabric
Traffic paths	May traverse several hierarchical layers.	Typically uses a consistent leaf-to-spine-to-leaf path.
Scaling	Expansion depends on available tier capacity and design.	Can expand by adding leafs and, when needed, spine capacity.
Multipath routing	Depends on the routing and switching design.	Layer 3 ECMP is commonly used between leafs and spines.
Operational considerations	May suit established environments and existing investments.	Requires deliberate underlay, routing, capacity and failure planning.

Before selecting a topology

Traffic patterns

Determine whether workloads primarily communicate with external networks or generate substantial east-west traffic between servers.

Capacity and growth

Estimate endpoint growth, uplink bandwidth, oversubscription limits and the cost of adding future capacity.

Failure domains

Identify the impact of losing a switch, link, rack or entire network block, including the capacity that remains.

Operations and cost

Balance equipment and cabling costs against monitoring, troubleshooting, automation and lifecycle requirements.

NETWORK INSIGHT · INTERACTIVE ARCHITECTURE LAB

Explore Data Center Architectures

Compare four common ways to connect compute, access and switching. Select an architecture, then click a device in the topology to inspect its role in the network.

Three-tier architecture Illustrative topology

Selectable device Network connection Highlighted path

Selected device

Core switch

The core provides high-speed connectivity between distribution blocks and upstream networks in a traditional three-tier design.

Architecture insight

Separate layers make roles easy to understand, but traffic may traverse multiple tiers and scaling can increase operational complexity.

Layered Design structure

Multi-hop Typical path shape

Layer by layer Common scaling approach

Design strengths

Clear separation of network roles
Familiar operational model
Can suit hierarchical traffic patterns

Design considerations

Potentially longer paths between servers
Capacity planning across tiers
Additional layers to operate and troubleshoot

These diagrams illustrate common design patterns, not vendor-specific configurations. Actual forwarding paths, redundancy and oversubscription depend on the implementation. ToR means Top of Rack; EoR means End of Row; Clos describes a multistage network fabric.

02 / FABRIC ARCHITECTURE

Leaf-Spine and Clos Fabrics

Leaf-spine architecture connects each leaf switch to multiple spine switches, creating a consistent path structure between different server racks. It is widely used in modern data centers where distributed applications, virtualization and cloud workloads generate substantial east-west traffic.

Explore how leaf and spine layers work together, how two-tier fabrics relate to Clos network designs, and how adding switches can expand connectivity. The section also examines oversubscription, available path diversity and the limits of relying on topology alone to guarantee performance.

Engineering focus Understand fabric stages, leaf-to-spine connectivity, expansion options and the capacity that remains when a device or link fails.

Section 02 / Fabric Architecture

Leaf-Spine and Clos Fabrics

Leaf-spine architecture is a common foundation for modern data center networks. Instead of sending traffic through a succession of hierarchical switching layers, it connects access-layer leaf switches to a set of spine switches. This creates a fabric with consistent paths between leaf switches and provides a repeatable way to expand connectivity as workloads grow.

The architecture is particularly useful for environments with substantial east-west traffic, including virtualized platforms, container infrastructure, distributed storage and cloud-native applications. Its performance depends on the actual link speeds, number of paths, traffic distribution and capacity provisioned throughout the fabric.

The physical fabric

Spine layer

Spine 1

Spine 2

Spine 3

Every leaf connects to every spine

Leaf layer

Leaf 1

Leaf 2

Leaf 3

Leaf 4

Server and workload connectivity

Servers

Storage

Compute

Services

Conceptual topology, not a physical cabling diagram. Each leaf has links to each spine; server-facing connections are shown separately. The simplified graphic illustrates connectivity rather than every individual link.

1. The role of leaf switches

Leaf switches provide connectivity for servers, storage, firewalls, load balancers and other data center endpoints. They may also perform Layer 3 routing, enforce network policy and act as VXLAN tunnel endpoints when an overlay is deployed.

In a Layer 3 fabric, endpoints in different subnets can communicate through routed paths. When VXLAN EVPN is used, the leaf layer can additionally provide overlay connectivity and distribute endpoint reachability through the EVPN control plane.

2. The role of spine switches

Spine switches interconnect the leaf layer and provide transit paths between leaf switches. In a conventional two-tier leaf-spine fabric, spine switches do not connect directly to servers, and leaf switches do not connect directly to other leaf switches.

Adding spine capacity can increase the number of available paths and aggregate fabric bandwidth, provided the leaf uplinks, interfaces, routing configuration and downstream capacity are designed to use it. A spine failure can reduce available capacity even when connectivity survives.

Why path predictability matters: Traffic between endpoints attached to different leaf switches normally traverses one spine switch. This creates a consistent switching-hop structure, although actual latency still depends on link speeds, device processing, congestion and the forwarding path.

How the architecture scales

01 / ADD LEAF CAPACITY

Expand endpoint connectivity

Add leaf switches with uplinks to the fabric's spine layer. Confirm that the existing spine and uplink capacity can support the additional traffic.

02 / ADD SPINE CAPACITY

Increase fabric bandwidth

Additional spine switches can create more equal-cost paths. Validate routing limits, optics, cabling, interface capacity and platform support before expanding.

03 / MANAGE OVERSUBSCRIPTION

Plan for real traffic demand

Compare aggregate server-facing bandwidth with available uplink bandwidth. Size the fabric according to workload behaviour and acceptable contention.

04 / CONTROL FAILURE IMPACT

Test the remaining capacity

Evaluate whether surviving spines and links can carry traffic after a failure, not simply whether alternate paths exist.

Two-tier and multi-tier Clos designs

Leaf-spine is a practical implementation of a two-tier Clos-style fabric. Larger environments may introduce a super-spine or additional fabric stages to interconnect multiple pods, rooms or data halls. These designs extend the architecture while requiring careful consideration of path diversity, capacity and failure boundaries.

Design	Typical purpose	Key consideration
Two-tier leaf-spine	Connect endpoints across a single fabric.	Spine capacity, leaf uplinks and ECMP behaviour.
Multi-pod fabric	Connect modular groups of racks or infrastructure.	Inter-pod connectivity and failure isolation.
Super-spine / multi-stage Clos	Scale connectivity across larger fabrics or multiple data halls.	Additional stages, path design and end-to-end capacity.

NETWORK INSIGHT · FABRIC EXPLORER

Explore Leaf-Spine and Clos Fabrics

Investigate how leaf and spine switches connect, trace an illustrative server-to-server path, simulate a spine-link failure, and see why uplink capacity and oversubscription matter.

Explore:

Two-tier leaf-spine fabric All illustrated links up

Network device Available link Illustrative selected path Failed link

Topology focus

Leaf switches

Leaf switches connect servers and provide uplinks into the spine layer. In a conventional two-tier fabric, each leaf connects to every spine.

Fabric principle

A full-mesh leaf-to-spine design provides multiple potential paths between different leaves. Routing and ECMP determine which next hops are installed and used.

Trace a flow

Source leaf

Destination leaf

2 Illustrated spine paths between selected leaves

8 Leaf-to-spine links currently available

Healthy Illustrative fabric state

Explore leaf uplink oversubscription 3:1 example

1:1 · equal nominal capacity 5:1 · more aggregate server capacity

A 3:1 illustrative oversubscription ratio means aggregate nominal server-facing bandwidth is three times the uplink bandwidth. Actual throughput depends on traffic patterns, link rates and congestion.

Why use a Clos-style fabric?

A multistage design can provide scalable connectivity and multiple paths without requiring every leaf to connect directly to every other leaf.

What should engineers verify?

Check spine and leaf capacity, uplink speed, ECMP behavior, failure convergence and the bandwidth remaining after a link or device failure.

This is an educational model, not a live network or packet-level simulator. The four-leaf, two-spine diagram is illustrative. ECMP distributes eligible flows across available next hops; it does not guarantee that each physical link carries exactly the same load.

03 / TRAFFIC ENGINEERING

Data Center Traffic Patterns

A data center network must accommodate the traffic generated by applications, storage systems, virtual machines and distributed services. Understanding where traffic originates and where it travels is essential for designing a fabric with sufficient capacity and predictable performance.

This section distinguishes east-west traffic between internal workloads from north-south traffic entering or leaving the data center. It explores how flow size, concurrent demand, ECMP hashing, uplink capacity and congestion influence the actual behaviour of a leaf-spine network.

Engineering focus Size the fabric around real workload demand, identify potential bottlenecks and validate performance during normal operation and failure conditions.

Section 03 / Traffic Engineering

Data Center Traffic Patterns

A data center topology must be designed around the traffic it carries, not simply the number of switches and links it contains. Application placement, storage access, replication, service dependencies and external connectivity all influence where traffic flows and where congestion can occur.

Leaf-spine fabrics are particularly suited to distributed east-west communication, but north-south traffic, large data transfers and uneven workload demand can still create bottlenecks. Understanding these patterns helps engineers size uplinks, evaluate redundancy and choose appropriate routing and forwarding designs.

Two fundamental traffic directions

01 / EAST-WEST

Workload-to-workload traffic

Traffic between servers, virtual machines, containers, storage systems and distributed application components. In modern environments, this can represent a substantial share of total network demand.

02 / NORTH-SOUTH

Traffic entering or leaving the fabric

Traffic between internal workloads and external networks, users, WANs, internet services or other data centers. Its path may include border leaf switches, firewalls, routers and load balancers.

Example / east-west traffic path

Communication between two server racks

Server A
Rack 1

Leaf 1

Spine

Leaf 2

Server B
Rack 2

Simplified logical path. In a conventional two-tier leaf-spine fabric, inter-leaf traffic normally traverses a spine. The forwarding plane selects an available path according to the installed routes and forwarding configuration.

Why east-west traffic affects fabric design

Distributed applications frequently exchange data across multiple servers. Examples include database replication, distributed storage, service-to-service requests, cluster coordination and virtual machine communication. If these flows cross leaf switches, the spine layer and leaf uplinks must provide enough capacity for their combined demand.

Traffic demand is rarely perfectly uniform. Some flows are short and latency-sensitive, while others transfer large volumes of data. Multiple simultaneous flows may also compete for the same links. Consequently, aggregate bandwidth alone does not reveal every congestion risk.

Important distinction: A fabric can have several equal-cost paths and still experience congestion. ECMP provides a mechanism for distributing eligible flows across paths; it does not guarantee that every flow uses a different link or that traffic is perfectly balanced.

North-south traffic and border connectivity

External traffic commonly enters or leaves through designated border leaf switches or dedicated edge devices. These connect the fabric to WAN routers, internet edge infrastructure, firewalls, load balancers and other networks.

The design must consider both bandwidth and resilience. Concentrating external connectivity on a small number of devices may simplify policy enforcement, but it can also create capacity constraints or larger failure domains. Distributing border functions can improve scale, provided routing policy and security controls remain consistent.

Traffic patterns and engineering decisions

Traffic pattern	Potential pressure point	Design response
Server-to-server	Leaf uplinks and spine capacity.	Assess aggregate demand, path diversity and ECMP.
Storage replication	Sustained high-throughput flows.	Measure peak demand and validate link utilization.
Internet or WAN access	Border links, security appliances and external routing.	Plan edge capacity, policy and redundant paths.
Large data transfers	Flow imbalance and transient congestion.	Evaluate hashing behaviour, flow sizes and telemetry.
Failure or maintenance	Reduced capacity on surviving paths.	Validate traffic behaviour after losing a link or device.

Measure the traffic, not just the topology

Traffic engineering should use interface counters, utilization trends, flow telemetry, packet drops, latency measurements and application performance data. These observations help distinguish a routing issue from a capacity problem or an application-generated traffic burst.

Engineers should also examine behaviour during failure conditions. A fabric that operates comfortably during normal traffic may become congested when a spine or uplink is unavailable and flows are redistributed across fewer remaining paths.

NETWORK INSIGHT · TRAFFIC EXPLORER

Explore Data Center Traffic Patterns

Follow traffic between servers, across the fabric and toward external networks. Adjust the illustrative load to understand why workload patterns, path capacity and congestion matter when designing a data center.

East-west: server to server Illustrative traffic flow

Network link East-west flow North-south flow Potential congestion

Traffic focus

East-west traffic

Traffic between workloads inside the data center. In a leaf-spine fabric, communication between servers on different leaves normally traverses a spine.

Engineering insight

Distributed applications, storage replication and service-to-service calls can generate substantial east-west traffic. Design for workload demand rather than assuming all traffic exits the data center.

Example flow

East-West Selected traffic direction

Leaf → Spine → Leaf Illustrative forwarding path

Moderate Illustrative link demand

Simulate traffic demand 45% illustrative load

Low demand High demand

At this illustrative load, the selected links have moderate demand. Real congestion depends on offered traffic, available link capacity, buffering, queueing and the distribution of flows.

Why traffic direction matters

East-west traffic can stay inside the fabric and use leaf uplinks and spine capacity. North-south traffic crosses the boundary toward external networks or services.

What engineers should measure

Compare interface utilization, drops, queue depth, flow records and application latency. A high average utilization can hide short bursts that cause packet loss or delay.

This widget is a conceptual teaching model, not a live traffic generator or telemetry system. The load slider represents illustrative demand, not measured utilization. Actual paths and capacity depend on topology, routing, link rates, hashing and workload behavior.

04 / ROUTING CONTROL PLANE

BGP in Data Center Fabrics

Border Gateway Protocol (BGP) provides a scalable way to distribute network reachability information across modern data center fabrics. In a routed leaf-spine design, BGP can establish the routes that allow traffic to move between endpoints while supporting controlled route propagation and multipath forwarding.

Explore eBGP and iBGP design approaches, route advertisements, best-path selection, ECMP and convergence. The section also introduces the relationship between a routed underlay and a VXLAN EVPN overlay, showing why control-plane state must be checked alongside the actual forwarding table.

Engineering focus Trace route propagation, validate next-hop reachability, understand multipath installation and troubleshoot routing changes without confusing BGP state with end-to-end connectivity.

Section 04 / Routing Control Plane

BGP in Data Center Fabrics

Border Gateway Protocol (BGP) is widely used to exchange reachability information in modern data center networks. In a routed leaf-spine fabric, it can provide scalable route distribution, controlled reachability, multipath forwarding and predictable responses to topology changes.

BGP does not replace the physical fabric. It supplies control-plane information that routers use to decide which network destinations are reachable and which paths can be installed for forwarding. The resulting behaviour depends on the routing design, policy, platform capabilities and operational configuration.

Keep the layers distinct: the physical topology defines available links; the routing control plane distributes reachability; the forwarding plane moves packets using installed forwarding entries. A healthy BGP session alone does not prove that the intended traffic path is working.

01 / Choosing a BGP fabric design

DESIGN A

eBGP leaf-spine

Leaf and spine switches use different autonomous system numbers (ASNs), and BGP sessions between them operate as external BGP. This is a common design for routed Clos fabrics because it can keep routing relationships explicit and support repeatable deployment patterns.

DESIGN B

iBGP-based fabric

Routers exchange BGP routes within the same autonomous system. Larger designs may use route reflectors to distribute routes without requiring a full mesh of iBGP sessions between every participating router.

DESIGN C

Hybrid routing

A fabric may combine an IGP or static routing for underlay reachability with BGP for other routing functions. The exact division depends on the design; there is no requirement that every data center use one universal routing model.

DESIGN D

Border and external routing

Border leaf switches or edge routers exchange routes with external networks. Import and export policies determine which prefixes can enter or leave the fabric and how they are represented to other routing domains.

Example: eBGP in a leaf-spine fabric

In this simplified example, each leaf has an eBGP session to each spine. Leaf-to-spine routing sessions exchange the reachability needed to forward traffic across the fabric. The example illustrates the relationship rather than prescribing a specific ASN plan or configuration.

Illustrative two-tier routed fabric

SPINE LAYER

Spine 1 AS 65101

Spine 2 AS 65102

eBGP sessions between each leaf and both spines

LEAF LAYER

Leaf 1 AS 65001

Leaf 2 AS 65002

Conceptual relationship only; individual physical links and separate BGP sessions are not drawn. ASNs are illustrative. This design assumes the chosen BGP configuration permits the required route exchange between the relevant devices.

Design caveat: eBGP does not automatically guarantee a working fabric or multipath forwarding. ASN allocation, next-hop reachability, route policy, prefix limits, supported BGP features and ECMP configuration must all be validated.

02 / Underlay and overlay routing

The BGP underlay

The underlay is the routed IP infrastructure that connects the switches and provides IP reachability across the fabric. In a BGP-routed underlay, BGP can advertise the infrastructure prefixes needed to reach loopbacks and other routed destinations. These routes provide the foundation for control-plane sessions and overlay tunnel endpoints where applicable.

The VXLAN EVPN overlay

In a VXLAN EVPN design, VXLAN encapsulates tenant traffic across the IP fabric, while EVPN uses BGP to distribute overlay reachability information. This can include information about MAC addresses, IP addresses and tenant segments, depending on the EVPN route type and deployment.

Component	Primary responsibility	Example question
Underlay routing	IP reachability between routed fabric endpoints.	Can this VTEP reach the remote VTEP address?
BGP EVPN control plane	Advertises overlay reachability and related information.	Has the expected MAC/IP or prefix route been learned?
VXLAN data plane	Encapsulates tenant packets across the IP underlay.	Can the encapsulated packet traverse the fabric?
Forwarding and policy	Applies installed forwarding entries and applicable policies.	Does traffic reach the correct destination and tenant?

These functions are related but not interchangeable. A healthy underlay does not prove that EVPN routes are correct, and receiving an EVPN route does not by itself prove that end-to-end forwarding succeeds.

03 / How BGP selects routes

From route advertisement to forwarding

Receive: a BGP neighbor advertises a prefix and its associated path attributes. The receiving router evaluates the update against session state, import policy and protocol rules.

Validate: the route must be acceptable under policy and protocol requirements. Its next hop must also be resolvable for the route to be usable for forwarding.

Select: the router evaluates eligible paths using its BGP decision process and implementation-specific policy. Attributes such as local preference, AS path, origin and MED can influence the outcome; their exact effect depends on the decision context and configuration.

Install: the selected route or eligible set of multipaths may be installed into the routing table and forwarding hardware, subject to platform limits, configuration and interactions with other routing sources.

Forward: packets are sent using the installed forwarding entries. The actual path depends on the forwarding implementation, ECMP eligibility, hashing and current topology.

Best path is not the same as every installed path. BGP implementations can select a best path while also installing multiple eligible paths for multipath forwarding. Multipath rules and supported attributes vary by platform and configuration.

Why BGP policy matters

BGP exchanges reachability, but policy determines which routes are accepted, preferred, modified or advertised onward. In a data center, an incorrect policy can suppress a required prefix, permit an unintended route or cause traffic to prefer an unexpected exit.

Engineers should explicitly document intended import and export behaviour, allowed prefixes, ASN expectations, next-hop handling and any route attributes used to influence path selection. Reusable policy templates can reduce configuration drift across large fabrics.

04 / ECMP and BGP multipath

How equal-cost paths support fabric capacity

Equal-Cost Multipath (ECMP) allows a router to forward traffic across multiple eligible next hops for a destination. BGP can supply candidate paths, but the routing implementation and configuration determine whether multiple paths are considered equivalent and installed.

Once multiple next hops are installed, forwarding hardware commonly uses a hash of packet or flow fields to select a path. This helps distribute traffic while typically preserving packet ordering within a flow. It does not mean every individual packet is sent down a different path, nor does it guarantee perfectly even utilization.

Mechanism	Role	Common misconception
BGP best-path selection	Chooses a preferred route according to policy and decision rules.	It does not necessarily install every received path.
BGP multipath	Allows multiple eligible BGP paths to be installed.	It is not automatic on every platform or configuration.
ECMP forwarding	Distributes eligible flows across equal-cost next hops.	It does not guarantee perfect load balance.
LACP	Negotiates and manages a Layer 2 link aggregation group.	It is not the same mechanism as Layer 3 ECMP.

ECMP and LACP can both use hashing in forwarding or distribution decisions, but they operate at different levels. ECMP selects among Layer 3 next hops; LACP negotiates membership in a Layer 2 aggregated link. A design may use one or both, depending on its physical and logical architecture.

05 / Convergence and failure handling

What happens when a path fails?

When a link or device fails, the fabric must detect the event, update the relevant routing state and adjust forwarding. BGP may withdraw affected routes or advertise revised reachability. The forwarding plane then changes its installed entries as the control plane and hardware respond.

Convergence time depends on failure detection, session behaviour, routing policy, protocol timers, implementation, hardware programming and the scope of the change. BFD may help detect some failures faster when supported and correctly configured, but it is not a universal guarantee of a particular recovery time.

FAILURE 01

Single link loss

If alternate paths remain available and the routing design recognizes the failure, traffic may be redirected through surviving paths. Check both route changes and actual forwarding.

FAILURE 02

Spine or peer loss

Routes or next hops associated with the failed device may be removed. Surviving paths carry the remaining demand, potentially increasing utilization and exposing oversubscription.

FAILURE 03

Policy or route withdrawal

A prefix may disappear even though physical links and BGP sessions remain up. Investigate advertisements, filters, next-hop resolution and the routing table.

FAILURE 04

Session instability

Repeated BGP resets can cause route churn and traffic disruption. Check interface errors, reachability, timers, logs and control-plane resource conditions.

Resilience must be tested under load. Redundant links are useful only if surviving paths have sufficient capacity and forwarding converges as intended. Test representative link and device failures, and measure packet loss, recovery time and post-failure utilization.

06 / Practical BGP troubleshooting

Work from session state to packet forwarding

When a prefix is missing or traffic takes an unexpected path, avoid starting with a broad configuration change. Follow the route through the control plane and forwarding plane, checking each stage before moving to the next.

Verify the session. Confirm the intended BGP neighbor is Established, and inspect logs, reset reasons, interface status and reachability between peer addresses.

Check received routes. Confirm the expected prefix is being advertised by the peer and received locally. Use the platform's received-route or neighbor-route commands where supported.

Inspect policy. Review prefix lists, route maps, import/export policies, community handling, maximum-prefix controls and any policy that could reject or modify the route.

Inspect best path and next hop. Identify why a candidate route was selected or rejected. Confirm that the selected next hop is resolvable and reachable.

Verify multipath installation. If several paths are expected, check the routing table for multiple installed next hops and confirm that multipath settings and platform capabilities permit them.

Validate forwarding. Inspect the forwarding table, interface counters, flow telemetry and packet path. For VXLAN EVPN, also verify underlay reachability, relevant EVPN routes, VTEP reachability and tenant forwarding state.

Example diagnostic commands

These are illustrative Cisco IOS/IOS XE-style commands. Exact syntax and available detail vary by operating system and platform; use the equivalent commands for your switches.

show ip bgp summary
show ip bgp
show ip route
show ip cef
show running-config | section router bgp

Start with the neighbor summary, then inspect the target prefix and its candidate paths. Check the routing table to determine which route was selected, and inspect the forwarding table to verify the next hop used for packet forwarding. Review the BGP configuration and policies when the observed result differs from the intended design.

The relevant BGP neighbor is Established.
The expected prefix is received and passes import policy.
The selected route has a valid, reachable next hop.
Expected ECMP next hops are installed when multipath is intended.
Export policy allows the required route to reach downstream peers.
Forwarding state and traffic telemetry agree with the expected path.
Failure tests confirm acceptable convergence and surviving capacity.

Continue with the BGP Learning Centre

Explore route propagation, best-path selection and fault investigation through the Network Insight BGP Learning Centre. It complements this architecture section with a practical, interactive way to examine BGP behaviour.

Open the BGP Learning Centre: Explore. Trace. Diagnose. →

NETWORK INSIGHT · BGP FABRIC EXPLORER

Explore BGP in a Leaf-Spine Fabric

Select a router, inspect the illustrative BGP sessions, trace how a prefix is learned, and simulate a path failure. See how routing information and forwarding paths work together in a data center network.

Illustrative BGP fabric Sessions established

iBGP eBGP Failed path

Selected element

Spine 1

A spine switch provides fabric transit. Depending on the design, it may establish eBGP sessions with leaf switches and exchange underlay reachability.

Engineering insight

BGP distributes reachability information. The forwarding plane uses the installed routes to move packets; a BGP session being established does not alone prove that an application path is working.

Interactive controls

6Illustrative BGP sessions

2Illustrative available paths

EstablishedSelected scenario state

Follow a route advertisement 10.10.10.0/24

1Prefix learnedA peer advertises reachability to the prefix.

2BGP evaluatesAttributes and policy influence route selection.

3Route installedAn eligible route is installed for forwarding.

4Packet forwardedThe forwarding table directs traffic to a next hop.

Underlay reachability

The underlay provides IP reachability between fabric devices and their loopbacks or next hops. eBGP is often used between leaf and spine switches in routed fabrics, though designs vary.

Route selection and forwarding

BGP applies policy and path-selection rules to choose eligible routes. ECMP may install multiple equal-cost next hops when supported by the routing design and platform.

Educational model only. This topology and its route progression are illustrative, not a live BGP implementation. Real session counts, ASNs, route attributes, ECMP eligibility, convergence and forwarding behavior depend on the configured network and platform.

05 / Observe

Network Visibility, Traffic Intelligence and BGP

You cannot reliably protect a network if you cannot establish what is happening within it. Flow records, packet captures, encrypted-traffic metadata and security logs reveal different aspects of network behaviour and provide evidence for troubleshooting and threat detection.

This section connects traffic analysis with BGP route visibility, helping engineers investigate unexpected communication patterns, routing changes and reachability problems using evidence from multiple layers.

05 / Observe

Network Visibility: From Traffic Flows to BGP Intelligence

Security controls determine what should be allowed. Observability helps engineers establish what is actually happening. By correlating flow records, packet-level evidence, encrypted traffic metadata and routing information, network teams can identify suspicious behaviour, investigate connectivity failures and understand how an incident moves across the network.

A firewall log may show that a connection was permitted, but it rarely explains the entire conversation. A packet capture may reveal a protocol exchange, yet only represent a short interval or one observation point. Flow telemetry can expose communication patterns across a longer period, while routing telemetry can explain which paths were available and why traffic may have taken an unexpected route.

Engineering principle: No single telemetry source provides the complete picture. Strong network investigations correlate evidence from multiple layers rather than treating an alert, a flow record or a routing change as conclusive proof on its own.

5.1 — Choose the right observation method

Flow telemetry

NetFlow and IPFIX

Summarise conversations using fields such as source and destination addresses, ports, protocol, packet counts, byte counts and timestamps. IPFIX provides an extensible framework for exporting structured flow information.

Sampling

sFlow

Uses statistical sampling of packets and interface counters to provide scalable visibility into traffic patterns, particularly in large switched and routed networks. Sampling can miss individual events.

Packet evidence

SPAN and packet capture

SPAN mirrors selected switch traffic to an observation port. Packet captures can reveal protocol headers, retransmissions, handshakes and other details that summaries cannot show. Mirror oversubscription can cause packet loss.

Encrypted traffic

Metadata and behaviour

Connection timing, volume, direction, destination reputation, DNS context where available, TLS handshake metadata and endpoint telemetry can help detect anomalies without assuming that encrypted payloads are readable.

Method	Best used for	Important limitation
NetFlow / IPFIX	Traffic accounting, unusual communication patterns and historical investigation	Typically lacks full packet payload and detailed packet-by-packet sequence
sFlow	High-volume traffic trends and scalable network sampling	Sampling may omit short-lived or low-volume activity
SPAN / packet capture	Protocol troubleshooting and detailed session analysis	Limited observation point, capture duration and possible mirror-port loss
Endpoint and security logs	Process attribution, identity, host events and security outcomes	Coverage and reliability depend on instrumentation and log integrity
BGP and routing telemetry	Route changes, path selection, reachability and control-plane anomalies	A valid route does not prove that the application or forwarding path works

5.2 — Build an evidence chain

Consider a server that suddenly transfers much more data than usual. A single bandwidth graph might identify the spike, but it cannot establish whether the cause is a legitimate backup, a compromised service or an unexpected change in traffic routing. A structured investigation combines several sources.

DetectBaseline deviation

→

CorrelateFlows and identities

→

InspectPackets and logs

→

ValidateRoutes and forwarding

First, establish when the anomaly began. Next, identify the communicating endpoints, destination ports, volume and duration. Compare the behaviour against the normal baseline. Then inspect available DNS, endpoint, firewall and packet evidence. Finally, check whether a route change, next-hop failure or policy update coincided with the event.

Interactive investigation explorer

Select an observation to see what it can establish and what should be checked next.

INVESTIGATION VIEW Flow records: establish who communicated

Compare source and destination addresses, ports, protocol, bytes, packets and timestamps. Identify unusual destinations, repeated connections or a sudden change in traffic volume. Then correlate the flow with endpoint and security logs.

5.3 — Encrypted traffic: visibility without payload inspection

Encryption protects confidentiality, but it does not make every aspect of a connection invisible. Depending on the protocol, configuration and observation point, engineers may still have access to connection duration, byte counts, packet sizes, timing, DNS information, destination addresses and some handshake metadata.

Behavioural baselines

Detect changes in connection frequency, data volume, session duration and communication direction compared with a service's established pattern.

Identity correlation

Combine network observations with endpoint, workload and authentication records to associate a connection with a device, user or application where the evidence supports it.

Protocol-aware analysis

Use available TLS, DNS and transport metadata carefully. Encrypted Client Hello and encrypted DNS can limit visibility into some fields, while QUIC changes what traditional passive inspection can observe.

Important distinction: A large encrypted transfer to an unfamiliar destination is an investigation lead, not proof of data exfiltration. Validate the destination, expected application behaviour, endpoint context and any applicable organisational policy before drawing conclusions.

5.4 — BGP visibility: understand the route behind the traffic

Border Gateway Protocol (BGP) exchanges reachability information between autonomous systems and distributes routes within an autonomous system through iBGP. Its control plane influences the paths that IP packets may follow. For network security and troubleshooting, this means traffic anomalies should sometimes be investigated alongside BGP updates, routing-table changes, next-hop resolution and forwarding state.

Routing intelligence

Three autonomous systems, one destination prefix

Illustrative topology for correlating route propagation with observed traffic.

AS65002 External BGP peer

AS65001 Internal BGP and routing policy

AS65003 External BGP peer

Example destination: 10.10.10.0/24

# Illustrative route observation Prefix: 10.10.10.0/24 Protocol: BGP AS_PATH: 65002 64496 NEXT_HOP: 192.0.2.2 Observation: Route present in BGP table Validation: Confirm best path, next-hop reachability and FIB

The values above are illustrative. They do not describe a live network or establish that this route is selected, reachable or safe.

What should engineers observe?

UPDATE activity: identify announcements and withdrawals, including the prefix affected and the time of the change.
AS_PATH changes: investigate unexpected path changes, unusual origins and route attributes that may affect selection.
Best-path selection: compare competing routes, local preference, AS_PATH length and other attributes in the context of the vendor's decision process and policy.
Next-hop resolution: verify that the selected next hop is reachable through the relevant routing information.
Forwarding state: confirm that the selected route is programmed into the forwarding information base (FIB) and that traffic can actually traverse the path.
Session health: correlate BGP neighbour state transitions, resets, timers and logs with the time the traffic anomaly occurred.

Security note: BGP route leaks and route hijacks can affect reachability or redirect traffic. Monitoring is only one defence. Apply appropriate prefix filtering, route-policy controls, peer validation and, where supported and operationally suitable, RPKI-based Route Origin Validation. ROV validates route-origin authorisation; it does not by itself validate the entire AS_PATH or prevent every routing attack.

5.5 — BGP and traffic telemetry must be correlated

A route appearing in the BGP table is not the same as a successful application connection. BGP provides control-plane reachability information, while the forwarding plane handles packets. An investigation must distinguish a route that has been received from one that has been selected, installed and successfully used.

Observation	Possible interpretation	Next validation step
Traffic volume drops suddenly	Application failure, link problem, filtering or a route change	Correlate interface counters, flow records, route events and endpoint health
AS_PATH changes unexpectedly	Legitimate policy change, upstream change or suspicious announcement	Check the source peer, prefix origin, route policy and trusted routing evidence
BGP route exists, but traffic fails	Unresolved next hop, FIB inconsistency, ACL, MTU issue or downstream failure	Check route selection, next-hop recursion, FIB, interface state and packet path
Repeated BGP session resets	Transport instability, peer configuration, resource pressure or timer expiry	Inspect neighbour logs, TCP connectivity, interface errors and control-plane load
Unexpected outbound traffic	New workload behaviour, misconfiguration or potentially malicious activity	Correlate destination, workload identity, DNS, security events and route history

5.6 — Operational requirements for reliable visibility

Observability systems are only as useful as the quality and consistency of their data. Time synchronisation is essential when correlating events across routers, firewalls, switches, hosts and collectors. Retention policies should preserve enough history to establish a baseline and investigate delayed reports. Telemetry collectors should be monitored for gaps, overload and export failures.

Synchronise timestamps across network devices, endpoints and telemetry platforms.
Document flow sampling rates, SPAN limitations and which interfaces are actually observed.
Monitor telemetry loss, collector health, storage capacity and data-retention windows.
Restrict access to sensitive flow records, packet captures and routing logs.
Protect management and telemetry channels against unauthorised access or tampering.
Establish normal traffic and routing baselines before treating deviations as incidents.

Section takeaway: Effective visibility connects traffic behaviour, security events and routing decisions. Flow records show communication patterns; packet captures provide deeper protocol evidence; encrypted-traffic metadata supports behavioural analysis; and BGP telemetry helps explain changes in network reachability. Correlation turns those separate signals into a defensible engineering investigation.

NETWORK INSIGHT · VISIBILITY WORKBENCH

Investigate the Data Center with Network Telemetry

Explore how flow records, interface counters, packet captures and BGP telemetry reveal different parts of a network problem. Select a diagnostic view to correlate traffic behavior with routing and forwarding state.

Traffic and telemetry sources Illustrative telemetry

Network connection Selected diagnostic path Warning signal Potential fault

Selected source

Flow telemetry

Flow records summarize conversations between endpoints. They help identify major talkers, traffic direction and volume, but do not normally contain every packet's full contents.

Engineering insight

No single telemetry source explains every fault. Correlating traffic records, interface counters, packet evidence and routing state can narrow the cause much faster.

Diagnostic actions

Flow recordsSelected evidence source

Traffic patternWhat it can reveal

PartialEvidence coverage

Illustrative diagnostic evidence

Signal	Example observation	Interpretation	Status
Flow records	Large app-to-database flow	Identify traffic source and volume	Useful
Interface counters	Output drops increasing	Investigate congestion or queue pressure	Check
Packet capture	Retransmissions observed	Correlate loss, delay and endpoint behavior	Check
BGP telemetry	Next-hop route changed	Verify route selection and forwarding state	Useful

What the data tells you

Flow telemetry helps locate large or unexpected conversations. It is valuable for traffic engineering and capacity planning, but its detail depends on the exporter, sampling and collection interval.

Correlate before concluding

Compare timestamps and affected paths across telemetry sources. A counter increase, a packet symptom and a routing change are stronger evidence when they align in time and relate to the same flow.

Educational demonstration only. The evidence values are illustrative examples, not live measurements. Available telemetry, sampling, counter definitions, packet visibility and BGP monitoring capabilities vary by platform and deployment.

06 / Respond

Incident Response, Threat Detection and Network Forensics

When suspicious activity is detected, the priority shifts from observation to controlled investigation and response. Engineers must validate the evidence, determine the scope of the incident, preserve relevant records and contain the threat without creating unnecessary operational disruption.

Explore the incident response lifecycle, practical forensic techniques, containment decisions and BGP incident investigation, from the first alert through recovery and lessons learned.

06 / Respond

Incident Response, Threat Detection and Network Forensics

Detecting suspicious activity is only the beginning. Effective incident response requires engineers to validate the alert, establish the scope of the incident, preserve evidence, contain the threat and restore services safely. Network telemetry, endpoint records, security controls and routing information provide complementary evidence that helps teams understand what happened and determine the appropriate response.

A security alert does not automatically establish that a system has been compromised. A sudden increase in outbound traffic, an unfamiliar destination, repeated authentication failures or an unexpected BGP route change may indicate a security incident, a configuration error or legitimate operational activity. The objective is to investigate systematically rather than act on an assumption.

Core principle: Incident response must balance containment, evidence preservation, service availability and business risk. Every action should have a clear purpose, an authorised scope and a way to verify its outcome.

6.1 — The incident response lifecycle

A practical response process moves from preparation through detection, analysis, containment, eradication and recovery. Lessons learned then feed back into monitoring, configuration standards and security controls. In a serious incident, these activities may overlap, and responders may need to revisit earlier stages as new evidence emerges.

PrepareTools, roles and playbooks

→

DetectAlert and validate

→

ContainLimit further impact

→

EradicateRemove root cause

→

RecoverRestore and monitor

Phase 1

Preparation

Define incident ownership, escalation paths, approved containment actions, logging requirements, backup procedures and communication channels before an incident occurs.

Phase 2

Detection and analysis

Validate the alert, establish a timeline, identify affected assets and correlate evidence from network, endpoint, identity and security systems.

Phase 3

Containment

Reduce the opportunity for further harm using proportionate, authorised controls. Consider dependencies and the operational consequences of isolating a device or blocking a communication path.

Phase 4

Eradication and recovery

Remove the confirmed cause, remediate affected systems, restore trusted configurations and verify that services are operating correctly before returning to normal operations.

6.2 — Detect and validate suspicious network activity

Detection systems compare observed activity against rules, threat intelligence, behavioural baselines and known indicators. The resulting alerts must be evaluated in context. An IP address associated with a threat feed, for example, is a useful investigation lead but may be outdated, shared by multiple services or unrelated to the activity under examination.

Signal	Potential concern	Validation approach
Unusual outbound data volume	Possible data transfer or unexpected application behaviour	Check flow records, destination, workload identity, scheduled jobs and endpoint events.
Repeated authentication failures	Possible credential guessing or account misuse	Correlate identity logs, source addresses, successful logins and account activity.
Repeated DNS queries	Possible misconfiguration, automated activity or suspicious domain generation	Inspect query patterns, resolver logs, domain context and the originating endpoint.
Unexpected east-west traffic	Possible lateral movement or a newly introduced dependency	Compare with workload communication baselines, segmentation policy and process telemetry.
Unexpected BGP announcement	Possible route leak, misconfiguration or unauthorised route propagation	Verify the peer, prefix, origin, policy, route history and expected maintenance activity.
Repeated connection resets	Possible service instability, filtering or hostile connection activity	Correlate packet captures, firewall decisions, server logs and interface counters.

6.3 — Interactive incident investigation

Select a scenario to explore the evidence an engineer should collect before choosing a response. These are illustrative investigation paths, not automated diagnoses.

Investigation workbench

Select a scenario to reveal its evidence requirements and recommended next steps.

INVESTIGATION SCENARIO Unusual outbound data transfer

Establish which host initiated the transfer, the destination, data volume, timing and expected application behaviour. Correlate flow records with endpoint activity, DNS context and available security logs before deciding whether containment is necessary.

6.4 — Network forensics and evidence preservation

Network forensics reconstructs relevant events from available evidence. Depending on the environment, that evidence may include packet captures, NetFlow or IPFIX records, DNS logs, firewall events, authentication records, endpoint telemetry, configuration changes and BGP update history.

Packet captures

Preserve the relevant capture interval and observation point. Record any capture filters, truncation settings, packet loss indicators and limitations affecting interpretation.

Centralised logs

Preserve original records where possible, document collection methods and correlate events using consistent timestamps and asset identifiers.

Configuration history

Compare approved and current firewall rules, ACLs, routing policies, interface configuration and relevant changes made before or during the incident.

Evidence handling: Follow organisational policy and applicable legal requirements. Restrict access to sensitive records, preserve provenance, document collection and handling, and use cryptographic hashes where appropriate to help demonstrate that collected files have not changed.

# Example: calculate a SHA-256 hash of a collected evidence file # Run on an authorised system with access to the evidence file. shasum -a 256 incident-capture.pcap # Record the hash, filename, collection time, collector and storage location. # A hash supports integrity verification; it does not prove the original # capture was complete or that its contents are accurate.

6.5 — Containment: reduce risk without creating another outage

Containment aims to limit the impact of an incident while preserving enough operational capability to investigate and recover. The right action depends on confidence in the evidence, the criticality of the affected service, the likely threat and the available alternatives.

Containment option	Potential benefit	Risk to consider
Isolate an endpoint	Limits some network communications from a suspected host	May interrupt critical services or remove access needed for investigation.
Apply a targeted firewall rule	Restricts a verified malicious flow or exposure	An overly broad rule may block legitimate dependencies.
Disable compromised credentials	Reduces the opportunity for further unauthorised access	Can interrupt automated workloads or legitimate administrative activity.
Change routing policy	May mitigate an established routing threat or isolate an affected path	Incorrect changes can cause route withdrawal, traffic blackholing or wider reachability loss.
Increase targeted monitoring	Improves evidence collection when immediate isolation is not justified	Monitoring alone may be insufficient when active harm is continuing.

Operational safeguard: High-impact containment actions should follow the incident response plan and appropriate authorisation. In a critical network, coordinate changes with service owners, assess failover behaviour and maintain a documented rollback plan wherever feasible.

6.6 — BGP incidents: distinguish control-plane anomalies from forwarding failures

BGP-related incidents require particular care because an incorrect response can affect reachability well beyond a single router. Unexpected route advertisements, route leaks, session instability and policy errors may alter the paths available to traffic. However, a route change alone does not establish malicious intent.

# Illustrative BGP incident triage checklist 1. Identify the affected prefix and first observed change. 2. Determine which peer advertised or withdrew the route. 3. Compare AS_PATH, origin and relevant route attributes. 4. Verify import/export policy and expected maintenance changes. 5. Check the selected route and next-hop resolution. 6. Confirm the FIB and test the affected forwarding path. 7. Apply an authorised mitigation and verify the result.

In an interdomain routing incident, useful safeguards may include carefully maintained prefix filters, explicit import and export policies, maximum-prefix controls, appropriate peer authentication and RPKI Route Origin Validation where supported. ROV helps validate whether an origin AS is authorised to originate a prefix; it does not validate every aspect of the route or prevent every possible BGP attack.

Within an autonomous system, engineers should also check iBGP propagation, route-reflector behaviour, next-hop reachability, IGP dependencies and forwarding state. A route may be visible in one routing table but absent from another, or selected by BGP without being usable for forwarding.

6.7 — Recovery, verification and lessons learned

Recovery is more than removing a block or restarting a service. Before returning affected systems to normal operation, teams should verify that the original cause has been addressed, required security controls are working and relevant traffic is following expected paths.

Confirm the affected systems, accounts, prefixes or services have been identified.
Remove the confirmed cause and remediate the relevant configuration or vulnerability.
Restore systems from trusted sources where required and verify their security posture.
Validate routing, next-hop reachability, forwarding state and application connectivity.
Maintain heightened monitoring for recurrence and unexpected changes.
Document the timeline, decisions, evidence, impact, recovery steps and outstanding risks.
Update detection rules, runbooks, access controls and training based on lessons learned.

6.8 — Measure incident response effectiveness

Response metrics help organisations identify gaps and improve operational readiness. They should be interpreted alongside incident severity, service impact and investigation quality rather than treated as isolated performance targets.

Detection

Mean time to detect

Measures the elapsed time between a defined incident starting point and detection. The starting point must be consistently defined for comparisons to be meaningful.

Response

Mean time to contain

Measures how long it takes to implement an effective containment measure after the relevant response clock starts.

Quality

Recurrence and false positives

Review recurring incident causes, unnecessary escalations, detection coverage and whether mitigations actually reduce risk without unacceptable disruption.

Section takeaway: Effective incident response joins detection, evidence preservation, containment, remediation and recovery into a controlled engineering process. Network telemetry and BGP visibility provide important context, but reliable decisions depend on corroborating evidence, understanding operational dependencies and verifying that corrective actions work.

NETWORK INSIGHT · RESILIENCE LAB

What Happens When a Data Center Fabric Fails?

Simulate illustrative link and spine failures in a leaf-spine fabric. Follow the surviving paths, inspect the remaining connectivity and identify what engineers should verify before declaring the fabric resilient.

Leaf-spine fabric connectivity Normal operation

Available connection Traffic path Failed component or link Illustrative alternate path

Scenario analysis

Normal operation

Both spine switches are available. Each leaf has two uplinks, allowing traffic to use multiple fabric paths when routing and forwarding are configured accordingly.

Engineering lesson

Redundancy is only useful when alternate paths are operational, routing converges correctly and the remaining links can carry the offered load.

Test controls

8 / 8Available leaf-to-spine links

2 / 2Available spine switches

2Illustrative paths between selected leaves

Failure investigation checklist

Check	Question	Why it matters	Status
Physical state	Are the links and devices operational?	Confirms the failure boundary	Baseline
Routing	Have paths been withdrawn or recalculated?	Control-plane state influences path selection	Baseline
Forwarding	Are alternate next hops installed?	Confirms traffic has a usable path	Baseline
Capacity	Can surviving links carry the load?	Prevents congestion after failover	Baseline

Resilience is more than redundancy

Multiple links and switches create alternate paths, but they do not guarantee uninterrupted service. Detection time, routing convergence, forwarding updates, hashing and remaining capacity all affect the outcome.

Validate the recovery

After a failure, verify reachability and application behavior, inspect routing and forwarding state, compare interface counters and confirm that the surviving fabric is not overloaded.

Educational model only. This is a simplified two-spine, four-leaf fabric. The diagram illustrates connectivity and possible alternate paths; it does not calculate live routing convergence, ECMP hashing, throughput, packet loss or application availability. Actual behavior depends on topology, routing policy, hardware and configuration.

Network Insight · Architecture Briefing

Designing Data Center Networks for Scale, Visibility and Resilience

Data center topology is more than a diagram of switches and links. It determines how workloads communicate, how capacity scales, how routing adapts to change and how engineers isolate faults when the network behaves unexpectedly.

Six ideas to take away

From architectural foundations to operational troubleshooting, each design decision affects the next.

Choose the right architecture

Three-tier designs separate access, aggregation and core functions. Leaf-spine fabrics provide a more consistent path structure for many modern, east-west-heavy workloads.

Understand the fabric

Leaf switches connect endpoints; spine switches interconnect the leaves. Clos-style designs can scale horizontally, subject to port count, link speed and oversubscription choices.

Design for real traffic

North-south and east-west flows create different demands. Capacity planning must consider traffic concentration, ECMP distribution, burstiness and workload communication patterns.

Make routing part of the design

BGP can provide scalable underlay routing in a leaf-spine fabric. Route policy, next-hop reachability, ECMP and convergence behavior all influence the forwarding outcome.

Make the network observable

Flow telemetry, interface counters, packet captures and routing telemetry reveal different parts of the story. Correlating them helps connect symptoms to causes.

Prove resilience under failure

Redundant links are only the beginning. Test failover, verify routing and forwarding state, check surviving capacity and confirm that affected applications recover.

A simplified fabric mental model

SPINE LAYER Path diversity

Leaf 1Leaf 2Leaf 3Leaf 4

ComputeStorageApplications

Think in layers, paths and failure domains.

A useful fabric design connects physical topology, routing behavior and application requirements. Each layer answers a different engineering question: what is connected, where can traffic travel and what happens when a component is lost?

✓Topology: Are there enough paths between the required endpoints?

✓Routing: Are the intended next hops available and installed?

✓Capacity: Can the remaining fabric carry the expected load?

✓Operations: Can telemetry identify the fault and verify recovery?

The engineering takeaway

A resilient data center is not simply one with redundant links. It is one whose behavior can be understood, measured and tested.

Build around the workload, understand the trade-offs in the chosen topology, validate routing and forwarding, and exercise failure scenarios before they occur in production. The goal is not just connectivity—it is predictable performance and recoverable service.

```

☕

Support Network Insight

Help support independent engineering guides, interactive networking tools, and future learning labs.

Support Network Insight ```

Data Center Network Design

August 11, 2014

by Matt Conran Blog

Data Center Network Design

Data centers are crucial in today’s digital landscape, serving as the backbone of numerous businesses and organizations. A well-designed data center network ensures optimal performance, scalability, and reliability. This blog post will explore the critical aspects of data center network design and its significance in modern IT infrastructure.

Data center network design involves the architectural planning and implementation of networking infrastructure within a data center environment. It encompasses various components such as switches, routers, cables, and protocols. A well-designed network ensures seamless communication, high availability, and efficient data flow.

The traditional three-tier network architecture is being replaced by more streamlined and flexible designs. Two popular approaches gaining traction are the spine-leaf architecture and the fabric-based architecture. The spine-leaf design offers low latency, high bandwidth, and improved scalability, making it ideal for large-scale data centers. On the other hand, fabric-based architectures provide a unified and simplified network fabric, enabling efficient management and enhanced performance.

Network virtualization, powered by technologies like SDN, is transforming data center network design. By decoupling the network control plane from the underlying hardware, SDN enables centralized network management, automation, and programmability. This results in improved agility, better resource allocation, and faster deployment of applications and services.

With the rising number of cyber threats, ensuring robust security and resilience has become paramount. Data center network design should incorporate advanced security measures such as firewalls, intrusion detection systems, and encryption protocols. Additionally, implementing redundant links, load balancing, and disaster recovery mechanisms enhances network resilience and minimizes downtime.

Matt Conran

Highlights: Data Center Network Design

Data Center Network Design

To embark on a successful network design journey, it is essential first to understand the data center’s specific requirements. Factors such as scalability, bandwidth, latency, and reliability need to be carefully assessed. By comprehending the data center’s unique needs, network architects can lay a solid foundation for an optimized design.

Efficiency and resilience are at the core of any well-designed data center network. Building on the requirements identified in the previous section, architects must consider redundancy, load balancing, and fault tolerance principles. The design should minimize single points of failure while maximizing resource utilization and network performance.

Various network topologies and architectures can be employed in data center network design. Each option offers unique advantages and trade-offs, from traditional hierarchical designs to modern approaches like leaf-spine architectures. This section will explore different topologies, highlighting their strengths and considerations.

Virtualization and SDN have revolutionized data center network design, offering increased flexibility and agility. By abstracting network functions from physical infrastructure, virtualization allows for dynamic resource allocation and improved scalability. SDN further enhances network programmability, enabling centralized management and automation. This section will delve into the benefits and implementation considerations of these technologies.

Network, security, and computing

– A data center architecture consists of three main components: the data center network, the data center security, and the data center computing architecture. In addition to these three types of architecture, there are also data center physical architectures and data center information architectures. The following are three typical compositions.

– Network architecture for data centers: Data center networks (DCNs) are arrangements of network devices interconnecting data center resources. They are a crucial research area for Internet companies and large cloud computing firms. The design of a data center depends on its network architecture.

– It is common for routers and switches to be arranged in hierarchies of two or three levels. There are three-tier DCNs: fat tree DCNs, DCells, and others. There has always been a focus on scalability, robustness, and reliability regarding data center network architectures.

– Data center security refers to physical practices and virtual technologies for protecting data centers from threats, attacks, and unauthorized access. It can be divided into two components: physical security and software security. A firewall between a data center’s external and internal networks can protect it from attack.

Data Center Network Design Considerations

a. Understanding the Requirements

Before embarking on the design process, it’s crucial to understand the data center’s unique requirements. Factors such as power and cooling, network connectivity, scalability, and security are vital in determining the design approach. By thoroughly assessing these requirements, architects can create a blueprint that aligns with the organization’s current and future needs.

b. Optimizing Physical Layout

The physical layout of a data center significantly impacts its efficiency and performance. This section will delve into rack placement, aisle design, cable management, and airflow optimization. By adopting best practices in physical layout design, data center operators can minimize energy consumption, reduce maintenance costs, and enhance overall operational efficiency.

c. Redundancy and Resilience

Data centers demand high levels of redundancy and resilience to ensure uninterrupted operations. This section will explore the concept of redundancy in power and cooling systems, backup generators, redundant network connectivity, and failover mechanisms. Implementing robust redundancy measures helps mitigate the risk of downtime and ensures continuous availability of critical services.

4. Security and Compliance

Data centers store sensitive and valuable information, making security a top priority. This section will discuss the importance of physical security measures, access controls, surveillance systems, and fire suppression mechanisms. Additionally, we will explore compliance standards and regulations that govern data center operations, such as SOC 2, ISO 27001, and GDPR.

5. Embracing Green Initiatives

As environmental sustainability gains importance, data centers seek ways to minimize their carbon footprint. This section will focus on energy-efficient design practices, including using renewable energy sources, efficient cooling techniques, and server virtualization. Data centers can contribute to a more sustainable future by adopting green initiatives.

Data Center Network Security

### What is Cloud Armor?

Cloud Armor is a security service offered by Google Cloud that provides protection against distributed denial-of-service (DDoS) attacks and other web-based threats. It leverages Google’s global infrastructure to offer scalable and reliable protection, ensuring that your applications and services remain available and secure even in the face of large-scale attacks.

### Key Features of Cloud Armor

Cloud Armor comes packed with several features that make it an indispensable tool for modern enterprises. Some of its key features include:

– **DDoS Protection:** Automatically detects and mitigates DDoS attacks, ensuring minimal disruption to your services.

– **Web Application Firewall (WAF):** Provides customizable rules to block malicious traffic and protect against common web vulnerabilities.

– **Edge Security Policies:** Allows you to define security policies at the edge of your network, ensuring threats are mitigated before they reach your core infrastructure.

– **Adaptive Protection:** Uses machine learning to identify and respond to evolving threats in real-time.

### Understanding Edge Security Policies

One of the standout features of Cloud Armor is its ability to implement edge security policies. These policies enable organizations to enforce security measures at the periphery of their network, providing an additional layer of defense. By stopping threats at the edge, you can prevent them from penetrating deeper into your network, thereby reducing the risk of data breaches and other security incidents.

Edge security policies can be tailored to your specific needs, allowing you to block traffic based on various criteria such as IP address, geographic location, and request patterns. This granular control helps you enforce stringent security measures while maintaining the performance and availability of your services.

### Benefits of Using Cloud Armor

Deploying Cloud Armor offers several benefits that can significantly enhance your security posture. These include:

– **Scalability:** Designed to handle traffic spikes and large-scale attacks, ensuring your services remain available even under heavy load.

– **Customization:** Flexible rules and policies allow you to tailor security measures to your unique requirements.

– **Proactive Defense:** Real-time threat detection and mitigation keep your applications protected against the latest cyber threats.

– **Cost-Effective:** By leveraging Google’s global infrastructure, you can achieve enterprise-level security without the need for significant upfront investment.

### What is Google Network Connectivity Center?

Google Network Connectivity Center is a unified platform designed to manage and monitor network connections across a variety of environments. Whether you’re dealing with on-premises data centers, cloud environments, or hybrid setups, NCC provides a centralized control point. It simplifies the complexities involved in network management, allowing IT teams to focus on optimizing performance rather than troubleshooting issues.

### Key Features of Google NCC

#### Unified Management

NCC offers a single pane of glass for managing network connections, making it easier to oversee and control your entire network infrastructure. This unified management approach reduces the need for multiple tools and interfaces, streamlining operations and increasing efficiency.

#### Flexible Connectivity Options

Google NCC supports a range of connectivity options, including VPNs, interconnects, and peering. This flexibility ensures that you can choose the best connectivity method for your specific needs, whether it’s connecting remote offices or integrating with third-party cloud services.

#### Real-Time Monitoring and Analytics

One of the standout features of NCC is its real-time monitoring and analytics capabilities. With detailed insights into network performance and traffic patterns, you can quickly identify and resolve issues, optimize resource allocation, and ensure consistent network performance.

Understanding Network Tiers

Network tiers are a concept that categorizes network traffic based on its importance and priority. By classifying traffic into different tiers, businesses can allocate resources accordingly and optimize their network usage. In the case of Google Cloud, there are two main network tiers: Premium Tier and Standard Tier.

The Premium Tier is designed to deliver exceptional performance and reliability. It leverages Google’s global network infrastructure, ensuring low latency and high throughput for critical applications. By utilizing the Premium Tier, businesses can enhance user experience, reduce latency-related issues, and improve overall network performance.

While the Premium Tier offers top-tier performance, the Standard Tier provides a cost-effective solution for non-critical workloads. It offers reliable network connectivity at a lower price point, making it an excellent choice for applications that do not require ultra-low latency or high bandwidth. By strategically utilizing the Standard Tier, businesses can optimize their network spend without compromising on reliability.

Understanding VPC Networking

VPC, or Virtual Private Cloud, is a virtual network dedicated to a specific Google Cloud project. It allows users to define and manage their network resources, including subnets, IP addresses, and firewall rules. With VPC networking, businesses can create isolated environments and control the flow of traffic within their cloud infrastructure.

Google Cloud’s VPC networking offers a range of powerful features. Firstly, it provides global connectivity, allowing businesses to connect resources across regions seamlessly. Additionally, VPC peering enables secure communication between different VPC networks, facilitating collaboration and data sharing. Moreover, VPC networking offers granular control through firewall rules, ensuring robust security for applications and services.

What is Google Cloud CDN?

Google Cloud CDN, short for Content Delivery Network, is a globally distributed network of servers designed to deliver content to users at blazing-fast speed. Cloud CDN minimizes latency and ensures a seamless user experience by caching your content in strategic locations worldwide. Whether it’s static assets, dynamic content, or even streaming media, Cloud CDN optimizes the delivery process, reducing the load on your origin servers and improving overall performance.

Cloud CDN operates by leveraging Google’s extensive network infrastructure. When a user requests content from your website or application, Cloud CDN intelligently routes the request to the nearest edge location. If the content is already cached at that edge location, it is immediately delivered to the user, eliminating the need for a round trip to the origin server. This not only reduces latency but also saves bandwidth and server resources.

Understanding VPC Network Peering

VPC network peering connects VPC networks from different projects or within the same project within Google Cloud. It enables direct communication between these networks, eliminating the need for complex VPN setups or public IP addresses. This seamless connectivity can significantly enhance collaboration, data sharing, and network management.

Enhanced Security: VPC network peering ensures that communication between peered networks remains isolated from the public internet. This adds an extra layer of security by reducing the exposure to potential cyber threats.

Improved Performance: By leveraging VPC network peering, data can be transferred at incredibly high speeds between peered networks. This enables faster resource access, reduces latency, and enhances overall application performance.

Simplified Network Architecture: VPC network peering allows for a more streamlined and simplified network architecture. Instead of relying on complex gateways or routers, communication between VPCs can be established directly, making network management and troubleshooting more straightforward.

Data Center Network Types

a. The Three-Tier Data Center Network

The three-tier DCN architecture has been a traditional approach in data center networking. It consists of three layers: the access layer, the aggregation layer, and the core layer. Each layer serves a specific purpose, from connecting end devices to aggregating traffic and providing high-speed connectivity. This hierarchical design allows for scalability and redundancy, making it a popular choice for many data centers.

b. Unleashing the Power of Fat Tree Data Center Networks

The fat tree DCN, also known as the Clos network, has gained prominence recently due to its ability to handle large-scale data center deployments. Unlike the three-tier DCN, a fat tree network provides multiple paths between devices, enabling better load balancing and higher bandwidth capacity. Fat tree networks offer low-latency communication and enhanced fault tolerance by utilizing a non-blocking switching fabric, making them ideal for mission-critical applications.

c. Exploring the Revolutionary DCell Approach

The DCell architecture takes a novel approach to data center networking and offers a unique perspective on scalability and fault tolerance. DCell networks are based on a hierarchical structure of cells, where each cell consists of a group of servers connected together. This decentralized design eliminates the need for traditional core switches and enables direct server-to-server communication. With its self-organizing capabilities, DCell networks provide excellent scalability, fault tolerance, and efficient resource utilization.

Composition of Data Center Architecture

Routing and Switching:

Routing is the backbone of a data center network, guiding data packets through the labyrinthine pathways. It involves determining the optimal path for data to travel from source to destination, considering network congestion, latency, and cost factors. Advanced routing protocols like Border Gateway Protocol (BGP) enable dynamic route selection, ensuring efficient and fault-tolerant data delivery.

Switching complements routing by facilitating efficient data transmission within a local network. At the heart of a data center, switches act as intelligent traffic controllers, directing data packets to their intended destinations. With features like VLANs (Virtual Local Area Networks) and Quality of Service (QoS), switches prioritize and prioritize traffic, optimizing network performance and ensuring seamless communication.

Example: Spanning Tree Uplink Fast

Spanning Tree Protocol (STP) prevents loops in Ethernet networks by creating a loop-free logical topology and blocking redundant paths. While STP ensures network stability, it can also introduce delays in network convergence. Network downtime caused by STP convergence can be a primary concern for businesses. Even a few seconds of downtime can result in significant losses in critical environments. This is where Spanning Tree Uplink Fast comes into play. Uplink Fast is an enhancement to STP that provides faster convergence times, reducing network downtime and improving overall network efficiency.

How Uplink Fast Works

Uplink Fast allows a switch to detect a link failure on its designated root port and immediately activate an alternate port. This process eliminates the need for the traditional STP convergence process, resulting in faster network recovery times. Uplink Fast is instrumental when network redundancy is crucial, such as in data centers or enterprise networks.

Introducing Spanning Tree MST

Spanning Tree MST enhances the traditional STP, providing a more efficient and flexible solution. MST allows network administrators to divide the network into multiple regions, each with its own Spanning Tree instance. By doing so, MST optimizes network resources and enables load balancing across multiple paths, leading to increased performance and redundancy.

To implement Spanning Tree MST, network switches need to be properly configured. This involves defining regions, assigning VLANs to instances, and configuring parameters such as root bridges and priorities. MST configuration can be complex, but with careful planning and understanding, it offers significant benefits.

Spanning Tree MST offers several key advantages. First, it enables efficient utilization of network resources by load-balancing traffic across multiple paths. Second, it provides enhanced redundancy, ensuring that if one path fails, traffic can automatically reroute through an alternate path. Third, MST simplifies network management by allowing administrators to control traffic flow and prioritize specific VLANs within each instance.

Data Center Security Technologies

Understanding the MAC Move Policy

The MAC Move Policy is a crucial feature in Cisco NX-OS devices that governs the movement of MAC addresses within a network. By defining specific rules and criteria, administrators can control how MAC addresses are learned, aged, and moved across different interfaces and VLANs.

Configuring the MAC Move Policy

Proper configuration is essential to effectively utilizing the MAC Move Policy. This section will guide you through the step-by-step process of configuring the policy on Cisco NX-OS devices. From defining the MAC move parameters to implementing the policy on specific interfaces or VLANs, we will cover all the necessary commands and considerations to ensure a seamless configuration experience.

Understanding MAC ACLs

MAC ACLs, also known as Ethernet ACLs or Layer 2 ACLs, operate at the data link layer of the OSI model. Unlike traditional IP-based ACLs, which focus on network layer addresses, MAC ACLs allow administrators to filter traffic based on MAC addresses. This enables granular control over network access, providing an additional layer of defense against unauthorized devices.

By implementing MAC ACLs on the Nexus 9000 series, network administrators can exercise enhanced control over their network environment. MAC ACLs prevent MAC address spoofing, mitigating the risk of unauthorized devices gaining access. Furthermore, they enable the isolation of specific devices or groups of devices, ensuring that only designated entities can communicate within a given VLAN or network segment.

Understanding VLANs and ACLs

Before we embark on our journey to explore VLAN ACLs’ potential, let’s establish a solid foundation by understanding VLANs and ACLs individually. VLANs (Virtual Local Area Networks) allow us to logically segment networks, improving performance, scalability, and network management. On the other hand, ACLs (Access Control Lists) act as gatekeepers, controlling traffic flow and enforcing security policies.

VLAN ACLs serve as a crucial layer of defense in protecting our networks from unauthorized access, malicious activities, and potential breaches. By implementing VLAN ACLs, we can define granular rules that filter and restrict traffic between VLANs, ensuring that only desired communication occurs. This level of control empowers network administrators to mitigate risks, maintain data integrity, and enforce compliance.

Understanding Nexus Switch Profiles

Nexus switch profiles are a feature of Cisco’s Nexus series switches that allow administrators to define and manage a group of switches as a single entity. By creating a profile, administrators can easily configure and monitor all switches within the group, eliminating the need for repetitive manual configurations. This centralization of management simplifies network administration and saves valuable time and resources.

One of the primary advantages of using Nexus switch profiles is the ability to streamline network operations. With a profile in place, administrators can make changes or updates to configurations across multiple switches simultaneously. This significantly reduces the risk of configuration errors and ensures consistent settings throughout the network. Furthermore, the centralized management approach simplifies troubleshooting and enables faster resolution of network issues.

Data Center Technologies

Understanding Layer 3 Etherchannel

Layer 3 Etherchannel is a link aggregation technique that combines multiple physical links between switches into a single logical channel. By bundling these links together, traffic can be distributed across them, increasing overall bandwidth capacity and providing load-balancing capabilities. Unlike Layer 2 Etherchannel, Layer 3 Etherchannel operates at the network layer, allowing traffic to be routed.

To configure Layer 3 Etherchannel, several steps need to be followed. First, the physical interfaces on the switches need to be identified and grouped into the Etherchannel bundle. Then, a logical interface, the Port-Channel interface, is created and assigned an IP address. Subsequently, routing protocols or static routes can be configured on the Port-Channel interface to enable communication between different networks.

Layer 3 Etherchannel supports various load-balancing algorithms, determining how traffic is distributed across the bundled links. Standard algorithms include source IP, destination IP, and round-robin. Each algorithm has advantages and considerations depending on the network requirements and traffic patterns.

Cisco Nexus 9000 Port Channel

Implementing Port Channels on Cisco Nexus 9000 switches offers several advantages. Firstly, it provides increased link bandwidth, allowing for efficient data transfer and reducing bottlenecks. Secondly, Port Channels enhance network resilience by providing link redundancy. In a link failure, traffic seamlessly switches to the remaining active links. Lastly, Port Channels enable load balancing, distributing network traffic evenly across the aggregated links for optimal utilization.

Setting up a Port Channel on Cisco Nexus 9000 switches is straightforward. Administrators can configure Port Channels using the Link Aggregation Control Protocol (LACP) or the Port Aggregation Protocol (PAgP). Administrators can maximize the benefits of this feature by adequately configuring interfaces and assigning them to the Port Channel.

Understanding Unidirectional Link Detection (UDLD)

UDLD is a layer 2 protocol that helps identify and mitigate the presence of unidirectional links in a network. It works by exchanging periodic messages between neighboring switches to verify bidirectional connectivity. By detecting unidirectional links, UDLD helps prevent potential network issues such as black holes, spanning-tree loops, and data loss.

Cisco Nexus 9000 switches offer seamless integration and support for UDLD. To enable UDLD on a Nexus 9000 switch, administrators can utilize simple commands within the switch configuration. By configuring UDLD timers, administrators can customize the frequency of UDLD messages exchanged between switches. Additionally, UDLD can be configured to operate in either standard or aggressive mode, depending on the specific needs of the network environment.

Understanding VRRP

VRRP, an essential networking protocol, provides automatic failover and load-balancing capabilities. It allows multiple routers to work as a virtual group, presenting a single IP address. By intelligently distributing network traffic, VRRP ensures seamless connectivity even in the face of router failures.

The Nexus 9000 Series, Cisco’s flagship product line, offers a range of cutting-edge features, including VRRP. Designed to meet the demands of modern networks, these switches deliver exceptional performance, scalability, and flexibility. With the Nexus 9000 Series, network administrators can harness the power of VRRP to build a robust and highly available network infrastructure.

Example: Data Center WAN Protocol

BGP, also known as the routing protocol of the Internet, is responsible for exchanging routing and reachability information among autonomous systems (AS). It enables routers to make intelligent decisions about the most optimal paths for data transmission. Unlike interior gateway protocols, BGP focuses on routing between different networks rather than within a single network.

BGP operates on a trust-based model, where routers form peer relationships to exchange routing information. These peers establish connections and exchange routing updates, allowing them to build a complete picture of network reachability. BGP uses a sophisticated algorithm that considers multiple factors, such as path length, quality of service, and policy-based decisions, to determine the best route for traffic.

Understanding BGP AS Prepend

AS Prepend involves adding additional Autonomous System (AS) numbers to the AS path attribute of BGP advertisements. By manipulating the AS path, network operators can influence inbound traffic routing decisions by neighboring autonomous systems. This technique makes a specific path appear less desirable, diverting traffic to alternative paths.

AS Prepend holds excellent potential for optimizing network routing in various scenarios. It can achieve load balancing across multiple links, redirect traffic to less congested paths, or prefer specific transit providers. By carefully implementing AS Prepend, network administrators can improve network performance, reduce latency, and enhance overall service quality.

Recap: Border Gateway Protocol (BGP) is data centers’ most commonly used routing protocol. It has been used to connect Internet systems worldwide for decades and can also be used outside a data center. The BGP protocol is a standard-based open-source software package. It’s more common to find BGP peering between data centers over the WAN. However, we see BGP often used purely inside the data center.

Understanding Leaf and Spine Networks

Leaf and spine networks, also known as Clos networks, are a modern approach to data center architecture. The design revolves around a hierarchical structure consisting of two key components: leaf switches and spine switches. Leaf switches connect directly to endpoints, while spine switches interconnect the leaf switches, forming a non-blocking fabric. This architecture eliminates bottlenecks and enables seamless scalability.

BGP (Border Gateway Protocol) is a crucial routing protocol in leaf and spine networks. It ensures efficient data forwarding between leaf switches using a set of rules known as BGP route advertisements. By default, BGP requires every router to have a full mesh of connections with all other routers in the network, which can be resource-intensive. This is where BGP route reflection comes into play.

Understanding BGP Route Reflection

BGP route reflection, at its core, is a method that allows a BGP speaker to reflect routing information to its peers, alleviating the need for full-mesh connectivity. Designating specific BGP routers as route reflectors streamlines and manages the network structure.

The utilization of BGP route reflection offers several advantages. First, it reduces the number of required BGP peering sessions, resulting in a simplified and less resource-intensive network. Second, route reflection enhances scalability by eliminating the need for full-mesh connectivity, particularly in large-scale networks. Third, it improves convergence time and reduces BGP update processing overhead, enhancing overall network performance.

The third wave of application architectures

Google and Amazon, two of the world’s leading web-scale pioneers, developed a modern data center. The third wave of application architectures represents these organizations’ search and cloud applications. Towards the end of the 20th century, client-server architectures and monolithic single-machine applications dominated the landscape. This third wave of applications has three primary characteristics:

Unlike client-server architectures, modern data center applications involve a lot of communication between servers. In client-server architectures, clients communicate with monolithic servers, which either handle the request entirely themselves or communicate with fewer than a handful of other servers, such as database servers. Search (or Hadoop, its more popular variant) employs many mappers and reducers instead of search. In the cloud, virtual machines can reside on different nodes but must communicate seamlessly. In some cases, VMs are deployed on servers with the least load, scaled out, or balanced loads.

A microservices architecture also increases server-to-server communication. This architecture is based on separating a single function into smaller building blocks and interacting with them. Each block can be used in several applications and enhanced, modified, and fixed independently in such an architecture. Since diagrams usually show servers next to each other, East-West traffic is often called server communication. Traffic flows north-south between local networks and external networks.

Scale and resilience

The sheer size of modern data centers is characterized by rows and rows of dark, humming, blinking machines. As opposed to the few hundred or so servers of the past, a modern data center contains between a few hundred and a hundred thousand servers. To address the connectivity requirements at such scales, as well as the need for increased server-to-server connectivity, network design must be rethought. Unlike older architectures, modern data center applications assume failures as a given. Failures should be limited to the smallest possible footprint. Failures must have a limited “blast radius.” By minimizing the impact of network or server failures on the end-user experience, we aim to provide a stable and reliable experience.

Data Center Goal: Interconnect networks

The goal of data center design and interconnection network is to transport end-user traffic from A to B without any packet drops, yet the metrics we use to achieve this goal can be very different. The data center is evolving and progressing through various topology and technology changes, resulting in multiple network designs. The new data center control planes we see today, such as Fabric Path, LISP, THRILL, and VXLAN, are driven by a change in the end user’s requirements; the application has changed. These new technologies may address new challenges, yet the fundamental question of where to create the Layer 2/Layer three boundaries and the need for Layer 2 in the access layer remains the same. The question stays the same, yet the technologies available to address this challenge have evolved.

Example Protocol: Understanding VXLAN

VXLAN, an encapsulation protocol, enables the creation of virtualized Layer 2 networks over an existing Layer 3 infrastructure. By extending the Layer 2 domain, VXLAN allows the seamless transfer of network traffic between geographically dispersed data centers. It achieves this by encapsulating Ethernet frames within IP packets, providing flexibility and scalability to network virtualization.

Scalability and Flexibility: VXLAN addresses the limitations of traditional VLANs by allowing for a significantly more significant number of virtual networks—up to 16 million—compared to the 4,096 limit of VLANs. This scalability enables organizations to allocate virtual networks more efficiently while accommodating the growing demands of cloud-based applications and services.

Enhanced Network Segmentation and Isolation: VXLAN provides improved network segmentation by creating logical networks that are isolated from one another, even if they share the same physical infrastructure. This isolation enhances security and enables more granular control over network traffic, facilitating efficient multi-tenancy in cloud environments.

Modern Data Centers

There is a vast difference between modern data centers and what they used to be just a few years ago. Physical servers have evolved into virtual networks that support applications and workloads across pools of physical infrastructure and into a multi-cloud environment. There are multiple data centers, the edge, and public and private clouds where data exists and is connected. Both on-premises and cloud-based data centers must be able to communicate. Data centers are even part of the public cloud. Cloud-hosted applications use the cloud provider’s data center resources.

Unified Fabric

Through Cisco’s fabric-based data center infrastructure, tiered silos and inefficiencies of multiple network domains are eliminated, and a unified, flat fabric is provided instead, which allows local area networks (LANs), storage area networks (SANs), and network-attached storage (NASs) to be consolidated into one high-performance, fault-tolerant network. Creating large pools of virtualized network resources that can be easily moved and rapidly reconfigured with Cisco Unified Fabric provides massive scalability and resiliency to the data center.

This approach automatically deploys virtual machines and applications, thereby reducing complexity. Thanks to deep integration between server and network architecture, secure IT services can be delivered from any device within the data center, between data centers, or beyond. In addition to Cisco Nexus switches, Cisco Unified Fabric uses Cisco NX-OS as its operating system.

The use of Open Networking

We also have the Open Networking Foundation ( ONF ), which provides open networking. Open networking describes a network that uses open standards and commodity hardware. So, consider open networking in terms of hardware and software. Unlike a vendor approach like Cisco, this gives you much more choice with what hardware and software you use to make up and design your network.

Data Center Performance Parameters

TCP Performance Parameters

TCP (Transmission Control Protocol) is the backbone of modern Internet communication, ensuring reliable data transmission across networks. However, various parameters that determine TCP’s behavior can influence its performance.

Understanding TCP Window Size: One crucial parameter that affects TCP performance is the window size. The TCP window size refers to the amount of data sent before an acknowledgment is required. A larger window size allows more data to be transmitted without waiting for acknowledgments, thus optimizing throughput. However, substantial window sizes can result in congestion and increased retransmissions.

Congestion Control Mechanisms: Congestion control mechanisms are vital in maintaining network stability and preventing congestion collapse. TCP utilizes algorithms such as Slow Start, Congestion Avoidance, and Fast Recovery to regulate data flow based on network conditions. These mechanisms ensure fairness and efficiency, improving TCP performance and avoiding network congestion.

Timeouts and Retransmission: TCP implements a reliable data transfer mechanism using acknowledgments and timeouts. When a packet is not acknowledged within a specific timeframe, it is considered lost, and TCP initiates retransmission. The selection of appropriate timeout values is crucial to balance reliability and responsiveness. Setting shorter timeouts may lead to unnecessary retransmissions, whereas longer ones can increase latency.

Selective Acknowledgments and SACK Options: Selective acknowledgments (SACK) enhance TCP performance and recovery from packet loss. SACK lets the receiver inform the sender about specific out-of-order packets received successfully. This enables the sender to retransmit only the necessary packets, reducing unnecessary retransmissions and improving overall efficiency.

Maximum Segment Size (MSS): The Maximum Segment Size (MSS) is another crucial TCP performance parameter defining the maximum amount of data encapsulated within a single TCP segment. Optimizing the MSS can significantly impact performance, especially when network links have different MTU (Maximum Transmission Unit) sizes.

Understanding TCP MSS

TCP MSS refers to the maximum amount of data encapsulated within a single TCP segment. It represents the size of the payload, excluding headers and other overhead. The MSS value is negotiated during the TCP handshake process and remains constant throughout the connection.

The TCP MSS value has a direct impact on network performance and efficiency. Setting an appropriate MSS value ensures optimal network resource utilization and avoids unnecessary data packet fragmentation. Properly configuring TCP MSS becomes crucial when networks have different MTU (Maximum Transmission Unit) sizes.

Fragmentation occurs when the MSS value exceeds the MTU of a network path. This fragmentation can lead to performance degradation, increased latency, and potential packet loss. By carefully managing the TCP MSS value, network administrators can prevent or minimize fragmentation issues and enhance overall network performance.

Configuring TCP MSS requires a thorough understanding of the network infrastructure and the devices involved. It involves adjusting the MSS value at various points within the network, such as routers, firewalls, and load balancers. Aligning the TCP MSS value with the MTU of the underlying network ensures efficient data transmission and avoids unnecessary fragmentation.

Advanced Topics

VXLAN Flood and Learn Mechanism

The flood-and-learn mechanism in VXLAN plays a crucial role in facilitating communication between virtual machines within the overlay network. When a virtual machine sends a broadcast or unknown unicast frame, the frame is encapsulated in a VXLAN packet and flooded throughout the network. Each VXLAN tunnel endpoint (VTEP) learns the source MAC address and VTEP association, enabling subsequent unicast traffic to be directly delivered.

Multicast is a fundamental component of VXLAN flood and learn, offering several benefits. First, using multicast VXLAN reduces bandwidth consumption compared to traditional flooding techniques. Second, multicast enables efficient replicating broadcast, multicast, and unknown unicast traffic across the overlay network. Third, it enhances network scalability by eliminating the need to maintain a multicast group per tenant.

BGP Multipath

Understanding BGP Multipath

BGP multipath is a feature that enables the installation and usage of multiple paths for a single prefix in the routing table. Traditionally, BGP selects a single best path based on factors such as AS path length, origin type, and path attributes. However, with multipath enabled, BGP can utilize multiple paths simultaneously, distributing traffic across them for load balancing and redundancy purposes.

The utilization of BGP multipath brings several advantages to network operators. First, it enhances network resilience by providing redundant paths. In the event of a link failure or congestion, traffic can be automatically rerouted through available alternate paths, ensuring continuous connectivity. Additionally, BGP multipath facilitates load balancing, enabling more efficient utilization of network resources and better traffic distribution across multiple links.

Understanding BGP Next Hop Tracking

BGP next-hop tracking monitors the reachability of the next-hop IP address associated with a particular route. It allows routers to dynamically adjust their routing tables based on changes in the network topology. Routers can make informed decisions about forwarding traffic by continuously tracking the next hop, ensuring optimal path selection.

Enhanced Network Resiliency: BGP next-hop tracking enables routers to detect and respond to network changes quickly. If a next hop becomes unreachable, routers can automatically reroute traffic to an alternative path, minimizing downtime and improving network resiliency.

Load Balancing and Traffic Engineering: Network administrators gain granular control over traffic distribution with BGP next-hop tracking. By monitoring the reachability of multiple next hops, routers can intelligently distribute traffic across different paths, optimizing resource utilization and improving overall network performance.

Improved Network Convergence: Rapid convergence is crucial in dynamic networks. BGP next hop tracking facilitates faster convergence by promptly updating routing tables when next hops become unreachable. This ensures routing decisions are based on current information, reducing packet loss and minimizing network disruptions.

Related: Before you proceed, you may find the following useful:

Data Center Network Design

The Rise of Overlay Networking

What has the industry introduced to overcome these limitations and address the new challenges? – Network virtualization and overlay networking. In its simplest form, an overlay is a dynamic tunnel between two endpoints that enables Layer 2 frames to be transported between them. In addition, these overlay-based technologies provide a level of indirection that allows switching table sizes to not increase in the order of the number of supported end hosts.

Today’s overlays are Cisco FabricPath, THRILL, LISP, VXLAN, NVGRE, OTV, PBB, and Shorted Path Bridging. They are essentially virtual networks that sit on top of a physical network, and often, the physical network is unaware of the virtual layer above it.

Traditional Data Center Network Design

How do routers create a broadcast domain boundary? Firstly, using the traditional core, distribution, and access model, the access layer is layer 2, and servers served to each other in the access layer are in the same IP subnet and VLAN. The same access VLAN will span the access layer switches for east-to-west traffic, and any outbound traffic is via a First Hop Redundancy Protocol ( FHRP ) like Hot Standby Router Protocol ( HSRP ).

Servers in different VLANs are isolated from each other and cannot communicate directly; inter-VLAN communications require a Layer 3 device. Virtualization’s humble beginnings started with VLANs, which were used to segment traffic at Layer 2. It was expected to find single VLANs spanning an entire data center fabric.

VLAN and Virtualization

The virtualization side of VLANs comes from two servers physically connected to different switches. Assuming the VLAN spans both switches, the same VLAN can communicate with each server. Each VLAN can be defined as a broadcast domain in a single Ethernet switch or shared among connected switches.

Whenever a switch interface belonging to a VLAN receives a broadcast frame (the destination MAC is ffff.ffff.ffff), the device must forward it to all other ports defined in the same VLAN.

This approach is straightforward in design and is almost like a plug-and-play network. The first question is, why not connect everything in the data center into one large Layer 2 broadcast domain? Layer 2 is a plug-and-play network, so why not? STP also blocks links to prevent loops.

The issues of Layer 2

The reason is that there are many scaling issues in large layer 2 networks. Layer 2 networks don’t have controlled / efficient network discovery protocols. Address Resolution Protocol ( ARP ) is used to locate end hosts and uses Broadcasts and Unicast replies. A single host might not generate much traffic, but imagine what would happen if 10,000 hosts were connected to the same broadcast domain. VLANs span an entire data center fabric, which can bring a lot of instability due to loops and broadcast storms.

No hierarchy in MAC addresses

MAC addressing also lacks hierarchy. Unlike Layer 3 networks, which allow summarization and hierarchy addressing, MAC addresses are flat. Adding several thousand hosts to a single broadcast domain will create large forwarding information tables.

Because end hosts are potentially not static, they are likely to be attached and removed from the network at regular intervals, creating a high rate of change in the control plane. Of course, you can have a large Layer 2 data center with multiple tenants if they don’t need to communicate with each other.

The shared services requirements, such as WAAS or load balancing, can be solved by spinning up the service VM in the tenant’s Layer 2 broadcast domain. This design will hit scaling and management issues. There is a consensus to move from a Layer 2 design to a more robust and scalable Layer 3 design.

But why is Layer 2 still needed in data center topologies? One solution is Layer 2 VPN with EVPN. But first, let us look at Cisco DFA.

The Requirement for Layer 2 in Data Center Network Design Servers that perform the same function might need to communicate with each other due to a clustering protocol or simply as part of the application’s inner functions. If the communication is clustering protocol heartbeats or some server-to-server application packets that are not routable, then you need this communication layer to be on the same VLAN, i.e., Layer 2 domain, as these types of packets are not routable and don’t understand the IP layer. Stateful devices such as firewalls and load balancers need Layer 2 adjacency as they constantly exchange connection and session state information. Dual-homed servers: Single server with two server NICs and one NIC to each switch will require a layer 2 adjacency if the adapter has a standby interface that uses the same MAC and IP addresses after a failure. In this situation, the active and standby interfaces must be on the same VLAN and use the same default gateway. Suppose your virtualization solutions cannot handle Layer 3 VM mobility. In that case, you may need to stretch VLANs between PODS / Virtual Resource Pools or even data centers so you can move VMs around the data center at Layer 2 ( without changing their IP address ).

Data Center Design and Cisco DFA

Cisco took a giant step and recently introduced a data center fabric with Dynamic Fabric Automaton ( DFA ), similar to Juniper QFabric. This fabric offers Layer 2 switching and Layer 3 routing at the access layer / ToR. Firstly, it has a Fabric Path ( IS-IS for Layer 2 connectivity ) in the core, which gives optimal Layer 2 forwarding between all the edges.

Then they configure the same Layer 3 address on all the edges, which gives you optimal Layer 3 forwarding across the whole Fabric.

On the edge, you can have Layer 3 Leaf switches, such as the Nexus 6000 series, or integrate with Layer 2-only devices, like the Nexus 5500 series or the Nexus 1000v. You can connect external routers, USC, or FEX to the Fabric. In addition to running IS-IS as the data center control plane, DFA uses MP-iBGP, with some Spine nodes being the Route Reflector to exchange IP forwarding information.

Cisco FabricPath

DFA also employs a Cisco FabricPath technique called “Conversational Learning.” The first packet triggers a full RIB lookup, and the subsequent packets are switched in the hardware-implemented switching cache.

This technology provides Layer 2 mobility throughout the data center while providing optimal traffic flow using Layer 3 routing. Cisco commented, “DFA provides a scale-out architecture without congestion points in the network while providing optimized forwarding for all applications.”

Terminating Layer 3 at the access / ToR has clear advantages and disadvantages. Other benefits include reducing the size of the broadcast domain, which comes at the cost of reducing the mobility domain across which VMs can be moved.

Terminating Layer 3 at the accesses can also result in sub-optimal routing because there will be hair pinning or traffic tromboning of across-subnet traffic, taking multiple and unnecessary hops across the data center fabric.

The role of the Cisco Fabricpath

Cisco FabricPath is a Layer 2 technology that provides Layer 3 benefits, such as multipathing the classical Layer 2 networks using IS-IS at Layer 2. This eliminates the need for spanning tree protocol, avoiding the pitfalls of having large Layer 2 networks. As a result, Fabric Path enables a massive Layer 2 network that supports multipath ( ECMP ). THRILL is an IEEE standard that, like Fabric Path, is a Layer 2 technology that provides the same Layer 3 benefits as Cisco FabricPath to the Layer 2 networks using IS-IS.

LISP is popular in Active data centers for DCI route optimization/mobility. It separates the host’s location from the identifier ( EID ), allowing VMs to move across subnet boundaries while keeping the endpoint identification. LISP is often referred to as an Internet locator.

That can enable some triangular routing designs. Popular encapsulation formats include VXLAN ( proposed by Cisco and VMware ) and STT (created by Nicira but will be deprecated over time as VXLAN comes to dominate ).

The role of OTV

OTV is a data center interconnect ( DCI ) technology enabling Layer 2 extension across data center sites. While Fabric Path can be a DCI technology with dark fiber over short distances, OTV has been explicitly designed for DCI. In contrast, the Fabric Path data center control plane is primarily used for intra-DC communications.

Failure boundary and site independence are preserved in OTV networks because OTV uses a data center control plane protocol to sync MAC addresses between sites and prevent unknown unicast floods. In addition, recent IOS versions can allow unknown unicast floods for certain VLANs, which are unavailable if you use Fabric Path as the DCI technology.

The Role of Software-defined Networking (SDN)

Another potential trade-off between data center control plane scaling, Layer 2 VM mobility, and optimal ingress/egress traffic flow would be software-defined networking ( SDN ). At a basic level, SDN can create direct paths through the network fabric to isolate private networks effectively.

An SDN network allows you to choose the correct forwarding information on a per-flow basis. This per-flow optimization eliminates VLAN separation in the data center fabric. Instead of using VLANs to enforce traffic separation, the SDN controller has a set of policies allowing traffic to be forwarded from a particular source to a destination.

The ACI Cisco borrows concepts of SDN to the data center. It operates over a leaf and spine design and traditional routing protocols such as BGP and IS-IS. However, it brings a new way to manage the data center with new constructs such as Endpoint Groups (EPGs). In addition, no more VLANs are needed in the data center as everything is routed over a Layer 3 core, with VXLAN as the overlay protocol.

Closing Points: Data Center Design

Data centers are the backbone of modern technology infrastructure, providing the foundation for storing, processing, and transmitting vast amounts of data. A critical aspect of data center design is the network architecture, which ensures efficient and reliable data transmission within and outside the facility. 1. Scalability and Flexibility

One of the primary goals of data center network design is to accommodate the ever-increasing demand for data processing and storage. Scalability ensures the network can grow seamlessly as the data center expands. This involves designing a network that supports many devices, servers, and users without compromising performance or reliability. Additionally, flexibility is essential to adapt to changing business requirements and technological advancements.

Redundancy and High Availability

Data centers must ensure uninterrupted access to data and services, making redundancy and high availability critical for network design. Redundancy involves duplicating essential components, such as switches, routers, and links, to eliminate single points of failure. This ensures that if one component fails, there are alternative paths for data transmission, minimizing downtime and maintaining uninterrupted operations. High availability further enhances reliability by providing automatic failover mechanisms and real-time monitoring to promptly detect and address network issues.

Traffic Optimization and Load Balancing

Efficient data flow within a data center is vital to prevent network congestion and bottlenecks. Traffic optimization techniques, such as Quality of Service (QoS) and traffic prioritization, can be implemented to ensure that critical applications and services receive the necessary bandwidth and resources. Load balancing is crucial in evenly distributing network traffic across multiple servers or paths, preventing overutilizing specific resources, and optimizing performance.

Security and Data Protection

Data centers house sensitive information and mission-critical applications, making security a top priority. The network design should incorporate robust security measures, including firewalls, intrusion detection systems, and encryption protocols, to safeguard data from unauthorized access and cyber threats. Data protection mechanisms, such as backups, replication, and disaster recovery plans, should also be integrated into the network design to ensure data integrity and availability.

Monitoring and Management

Proactive monitoring and effective management are essential for maintaining optimal network performance and addressing potential issues promptly. The network design should include comprehensive monitoring tools and centralized management systems that provide real-time visibility into network traffic, performance metrics, and security events. This enables administrators to promptly identify and resolve network bottlenecks, security breaches, and performance degradation.

Data center network design is critical in ensuring efficient, reliable, and secure data transmission within and outside the facility. Scalability, redundancy, traffic optimization, security, and monitoring are essential considerations for designing a robust, high-performance network. By implementing best practices and staying abreast of emerging technologies, data centers can build networks that meet the growing demands of the digital age while maintaining the highest levels of performance, availability, and security.

Example Product: Data Center Monitoring

#### Understanding Cisco ThousandEyes

Cisco ThousandEyes is a comprehensive network intelligence platform that offers deep insights into the performance and health of your data center. By leveraging cloud-based agents and on-premises appliances, ThousandEyes provides end-to-end visibility across your entire network, from your data center to the cloud and beyond. This holistic approach allows IT teams to quickly identify and resolve issues, ensuring that your data center operates at peak efficiency.

#### Key Features of Cisco ThousandEyes

One of the standout features of Cisco ThousandEyes is its ability to deliver real-time insights into network performance. With its advanced monitoring capabilities, ThousandEyes can detect anomalies, pinpoint bottlenecks, and provide actionable data to help you optimize your data center operations. Here are some of the key features that make ThousandEyes a valuable asset:

– **End-to-End Visibility:** Monitor the entire network path, from the user to the application, ensuring no blind spots.

– **Cloud and On-Premises Integration:** Seamlessly integrate with both cloud-based and on-premises infrastructure for comprehensive coverage.

– **Real-Time Alerts:** Receive instant notifications of any performance issues, allowing for swift resolution.

– **Detailed Reporting:** Generate in-depth reports that provide insights into network performance trends and potential areas for improvement.

#### Benefits of Using Cisco ThousandEyes for Data Center Performance

Implementing Cisco ThousandEyes in your data center can deliver a range of benefits that contribute to enhanced performance and reliability. Some of the key advantages include:

– **Proactive Issue Resolution:** By identifying potential problems before they escalate, ThousandEyes helps prevent downtime and ensures continuous service delivery.

– **Improved User Experience:** With optimized network performance, users enjoy faster, more reliable access to applications and services.

– **Cost Efficiency:** By reducing downtime and improving operational efficiency, ThousandEyes can help lower overall IT costs.

– **Scalability:** As your business grows, ThousandEyes can scale with you, providing consistent performance monitoring across expanding networks.

#### Real-World Applications

Many organizations have successfully leveraged Cisco ThousandEyes to boost their data center performance. For example, a global financial services company used ThousandEyes to monitor their network and quickly identify a latency issue affecting their trading platform. By resolving the issue promptly, they were able to maintain their competitive edge and deliver a seamless experience to their clients. Similarly, an e-commerce giant utilized ThousandEyes to ensure their website remained responsive during peak shopping seasons, resulting in increased customer satisfaction and sales.

Summary: Data Center Network Design

In today’s digital age, data centers are the backbone of countless industries, powering the storage, processing, and transmitting massive amounts of information. However, the efficiency and scalability of data center network design have become paramount concerns. In this blog post, we explored the challenges traditional data center network architectures face and delved into innovative solutions that are revolutionizing the field.

The Limitations of Traditional Designs

Traditional data center network designs, such as three-tier architectures, have long been the industry standard. However, these designs come with inherent limitations that hinder performance and flexibility. The oversubscription of network links, the complexity of managing multiple layers, and the lack of agility in scaling are just a few of the challenges that plague traditional designs.

Enter the Spine-and-Leaf Architecture

The spine-and-leaf architecture has emerged as a game-changer in data center network design. This approach replaces the hierarchical three-tier model with a more scalable and efficient structure. The spine-and-leaf design comprises spine switches, acting as the core, and leaf switches, connecting directly to the servers. This non-blocking, high-bandwidth architecture eliminates oversubscription and provides improved performance and scalability.

Embracing Software-Defined Networking (SDN)

Software-defined networking (SDN) is another revolutionary concept transforming data center network design. SDN abstracts the network control plane from the underlying infrastructure, allowing centralized network management and programmability. With SDN, data center administrators can dynamically allocate resources, optimize traffic flows, and respond rapidly to changing demands.

The Rise of Network Function Virtualization (NFV)

Network Function Virtualization (NFV) complements SDN by virtualizing network services traditionally implemented using dedicated hardware appliances. By decoupling network functions, such as firewalls, load balancers, and intrusion detection systems, from specialized hardware, NFV enables greater flexibility, scalability, and cost savings in data center network design.

Conclusion:

The landscape of data center network design is undergoing a significant transformation. Traditional architectures are being replaced by more scalable and efficient models like the spine-and-leaf architecture. Moreover, concepts like SDN and NFV empower administrators with unprecedented control and flexibility. As technology evolves, data center professionals must embrace these innovations and stay at the forefront of this paradigm shift.

OpenContrail

Highlights: OpenContrail

Understanding OpenContrail

Open-source network virtualization platform

Understanding OpenContrail components

The traditional network vs. SDN network

Highlights: OpenContrail

Key Features and Benefits:

Highlighting Junipers OpenContrail

Network virtualization – Isolated networks

Service Chaining

The ability to Scale-out

Virtual Network Implementation

A – MPLS Overlay

B – The forwarding plane

C – Northbound and southbound interfaces

WAN Integration

Summary: OpenContrail

Overlay Virtual Networks

Highlights: Overlay Virtual Networks

Overlay Network Architecture:

**Types of Overlay Networks**

VXLAN as the Modern Overlay for Scalable Multi‑Site Fabrics

Google Cloud CDN

Use Cases of Overlay Virtual Networking:

Tunneling Protocols

Example: Point-to-Point GRE

Example: GRE and IPSec

Example: MPLS Overlay Tunneling

**Overlay Network Control Plane**

Underlay and Clos Fabric

VXLAN and Leaf and Spine

Example: VXLAN Flood and Learn

**VXLAN vs. Spanning Tree Protocol**

Multicast Overlay

Advantages of Overlay Virtual Networks

Advanced Topics

1. GETVPN:

2. DMVPN

The Single Hub Dual Cloud Architecture

**DMVPN vs GETVPN**

3. MPLS VPN

Use Case: Understanding Performance-Based Routing

Overlay Virtual Networks

Concept of network virtualization

Tailored load balancing

Scalability and Security Concerns

NIC in promiscuous mode and failure domains

The rise of overlay virtual networks

Virtual Overlay Solution: Keep complexity to the edges.

VXLAN, STT, and ( NV ) GRE

The Recommended Design: Leaf and Spine.

TCP offload, load balancing & scale-out NAT

STT and VXAN

Distributed layer 2 & layer 3 forwarding

**Scale-out control plane**

Scale-out controllers

Availability zones

Network Overlay Controllers

Closing Point: Overlay Virtual Networks

Summary: Overlay Virtual Networks

LISP Hybrid Cloud Use Case

Highlights: LISP Hybrid Cloud Use Case

Understanding LISP

Data Analytics with LISP Hybrid Cloud

Use Case: Hybrid Cloud

LISP Hybrid Cloud Use Case

The LISP Network

LISP Networking and Stretched VLAN

LISP acts as an overlay technology

LISP networking deployment key points:

LISP stretched subnets

Closing Points on LISP Hybrid Cloud

Summary: LISP Hybrid Cloud Use Case

What is OpenFlow?

Highlights: What is OpenFlow?

How does OpenFlow work?

The Role of OpenFlow Controllers

Introducing SDN

Data and control plane

Types of Overlay Networks

Overlay Network Control Plane

VXLAN vs. Spanning Tree Protocol

DMVPN vs GETVPN

Scale-out control plane

A key point: The lack of a session layer in the TCP/IP stack

Control plane

Data plane