What is VXLAN

What is VXLAN

Network Insight · Data Centre Networking

What Is VXLAN? Understanding Modern Network Overlays

Modern data centres require networks that can scale beyond traditional VLAN boundaries, support virtualised workloads and deliver predictable connectivity across routed infrastructure. Virtual Extensible LAN (VXLAN) addresses these requirements by extending Layer 2 Ethernet segments across a Layer 3 IP network.

VXLAN separates the logical network from the physical transport infrastructure. Instead of relying on a large switched Layer 2 domain, VXLAN encapsulates Ethernet frames inside UDP/IP packets and transports them between VXLAN Tunnel Endpoints (VTEPs). This enables virtual networks to span multiple physical switches while the underlying fabric continues to use IP routing.

Each VXLAN segment is identified by a 24-bit VXLAN Network Identifier (VNI) . Compared with traditional 12-bit VLAN identifiers, VXLAN provides a substantially larger segmentation space, making it particularly useful for multi-tenant cloud platforms, software-defined data centres and large-scale virtualised environments.

24-bit VXLAN Network Identifier used to distinguish logical overlay segments
~16.7M Potential VNI values compared with approximately 4,094 usable VLAN IDs
UDP 4789 Standard VXLAN destination port for Ethernet encapsulation over IP
Architecture Preview · Layer 2 over Layer 3
Leaf A
VTEP 10.0.0.11
VLAN 10 → VNI 10010
VXLAN Overlay
UDP/IP Encapsulation
Routed IP Underlay
Leaf B
VTEP 10.0.0.12
VLAN 10 → VNI 10010
Two VTEPs extend the same logical Layer 2 segment across a routed IP fabric. Intermediate routers forward the outer IP packet without inspecting the inner Ethernet frame.

Why VXLAN Matters in Modern Data Centres

Traditional Ethernet networks face practical limitations when stretched across large data centre environments. VLAN identifiers provide a limited segmentation space, large broadcast domains increase operational complexity, and Spanning Tree Protocol can prevent redundant Layer 2 links from forwarding simultaneously.

VXLAN enables a different architectural model. A routed leaf-spine underlay can use Equal-Cost Multi-Path (ECMP) forwarding across multiple active paths, while the VXLAN overlay provides logical Layer 2 connectivity independently of the physical topology. When combined with BGP Ethernet VPN (EVPN), the fabric can distribute MAC and IP reachability information through a control plane, reducing reliance on data-plane flooding.

What You Will Learn

✓ Understand why VXLAN was introduced and how it addresses traditional VLAN scaling limitations.
✓ Explore the relationship between the IP underlay, VXLAN overlay, VTEPs and VNIs.
✓ Follow Ethernet frames through VXLAN encapsulation, routed forwarding and decapsulation.
✓ Compare flood-and-learn forwarding with BGP EVPN MAC and IP reachability distribution.
✓ Investigate tenant segmentation, inter-VNI routing and distributed gateway design.
✓ Diagnose common VXLAN failures using VTEP reachability, VNI mappings, EVPN state and telemetry.
From Architecture to Engineering Practice

This guide combines technical explanations with interactive network engineering exercises. You will explore VXLAN topology behaviour, examine encapsulated packets, compare forwarding mechanisms and investigate realistic overlay and underlay failures. The objective is not simply to understand what VXLAN is, but to understand how it behaves in a production data centre fabric.

Section 01 · VXLAN Fundamentals

Why VXLAN? Understanding the Limitations of Traditional Layer 2 Networks

Modern data centre networks must support thousands of virtual machines, multiple tenants, dynamic workload placement and high-bandwidth east-west traffic. Traditional Ethernet switching was not originally designed for this degree of scale, segmentation and mobility.

Conventional Layer 2 architectures rely on VLANs, MAC address learning and loop-prevention mechanisms such as Spanning Tree Protocol (STP). These technologies remain useful, but extending large Layer 2 domains across a data centre can introduce operational complexity, inefficient use of redundant links and limitations in the number of logical segments.

VXLAN was introduced to separate logical Layer 2 connectivity from the physical network topology. By encapsulating Ethernet frames inside UDP/IP packets, VXLAN allows a routed IP fabric to transport multiple isolated virtual networks without requiring the entire physical infrastructure to participate in the same Layer 2 broadcast domain.

VLAN Scaling Spanning Tree MAC Learning Server Virtualization Leaf-Spine Fabrics Network Overlays

1.1 — The Traditional Layer 2 Data Centre

In a traditional switched network, Ethernet frames are forwarded using destination MAC addresses. Switches learn source MAC addresses from incoming frames and maintain forwarding tables that associate MAC addresses with physical interfaces.

VLANs provide logical separation within this infrastructure. Each VLAN represents an independent Layer 2 broadcast domain, allowing different applications, departments or tenants to share physical switching hardware while maintaining separate Ethernet segments.

The challenge emerges when these networks grow. More servers create more MAC addresses, more applications require additional VLANs, and redundant physical links require careful loop prevention. Workload mobility can also require Layer 2 connectivity across multiple racks or physical locations.

01
Limited Segmentation Space

Traditional IEEE 802.1Q VLAN tags use a 12-bit VLAN identifier, providing 4,096 possible values, of which approximately 4,094 are available for normal VLAN use.

02
Spanning Tree Constraints

STP prevents Ethernet forwarding loops by selecting a loop-free active topology. Some redundant links may be blocked, preventing simultaneous forwarding.

03
MAC Address Table Growth

Virtualized servers can introduce many MAC addresses per physical switch port. Large Layer 2 domains increase forwarding table requirements and operational scale.

04
Workload Mobility

Moving workloads between physical hosts may require preserving network connectivity, addressing and policy across different switching locations.

Engineering Principle

The problem is not that Ethernet switching or VLANs are inherently unsuitable. The challenge is extending large numbers of logical Layer 2 segments across a growing physical network while preserving scalability, resilience and operational simplicity.


1.2 — VLAN Identifier Exhaustion

IEEE 802.1Q uses a 12-bit VLAN Identifier (VID). This provides 212, or 4,096, possible identifier values. VLAN IDs 0 and 4095 are reserved, leaving 4,094 usable VLAN identifiers in the normal VLAN namespace.

For a small enterprise, this may be sufficient. However, large cloud providers and multi-tenant data centres may require thousands of independent network segments for customers, applications, security zones and workload environments.

Logical Network Identifier Capacity
Traditional VLAN — 12 bits 4,096 values
VXLAN VNI — 24 bits 16,777,216 values
Visual comparison uses a compressed scale so both identifier spaces remain visible. VXLAN provides 4,096 times the theoretical identifier space of a 12-bit VLAN field. Actual supported VNI scale depends on platform hardware and software limits.

VXLAN introduces a 24-bit VXLAN Network Identifier (VNI). This significantly expands the available logical segment namespace. A VNI identifies an overlay segment, allowing different physical VLANs to map into the same logical VXLAN segment when required.

For example, VLAN 10 on one leaf switch and VLAN 20 on another leaf switch can both map to VNI 10010. The local VLAN numbers do not need to match, provided the VTEPs are configured to associate those VLANs with the same intended overlay bridge domain.

Engineering Example · Multi-Tenant Cloud

Consider a cloud provider operating infrastructure for 500 customers. Each customer requires eight isolated application networks for development, production, management, databases and supporting services.

500 Customers
8 Segments per Customer
4,000 Required Segments

This deployment already consumes almost the entire traditional VLAN identifier space if every segment requires a unique VLAN ID in one shared administrative namespace.

Adding more customers, environments or application tiers creates further demand. VXLAN allows the provider to allocate distinct VNIs without relying on a single global 12-bit VLAN namespace.

Important Design Distinction

VLAN IDs can be reused in separate Layer 2 domains, so the 4,094-VLAN limit is not a universal limit on the number of tenants in an entire network. VXLAN is valuable because it provides a much larger overlay namespace and decouples logical segmentation from local physical VLAN assignments.


1.3 — Spanning Tree and Blocked Links

Ethernet networks must prevent forwarding loops. Unlike IP packets, ordinary Ethernet frames do not contain a hop-limit field that automatically expires as frames traverse switches.

A Layer 2 forwarding loop can cause broadcast storms, repeated frame replication, unstable MAC learning and severe network disruption. Spanning Tree Protocol prevents these conditions by selecting a loop-free forwarding topology.

Although this provides stability, a conventional spanning-tree topology may block redundant links. Consequently, installed physical bandwidth is not always available for active forwarding. Convergence following topology changes can also affect application connectivity.

Traditional Layer 2 Design
Redundant Ethernet paths require loop prevention and may include blocked links.
Distribution A
Distribution B
Access A
Access B
Active forwarding path
Illustrative blocked redundant path
Routed Leaf-Spine Fabric
IP routing enables multiple equal-cost paths to forward traffic concurrently.
Spine 1
Spine 2
Leaf A
Leaf B
Routed links can remain active
ECMP distributes eligible flows across available next hops.

Modern leaf-spine architectures commonly replace the large switched core with a routed IP underlay. Each leaf connects to multiple spine switches, and routing protocols maintain reachability across the fabric.

Equal-Cost Multi-Path (ECMP) forwarding allows traffic to use multiple equal-cost routes. Rather than relying on a spanning tree to disable redundant Layer 2 paths, the fabric uses Layer 3 routing and its associated convergence mechanisms.

VXLAN complements this architecture by transporting logical Layer 2 segments over the routed underlay. The physical fabric can therefore remain Layer 3 while connected workloads continue to experience the Layer 2 adjacency required by their applications.

Architectural Advantage

VXLAN does not itself perform ECMP routing. ECMP is provided by the IP underlay. VXLAN enables Layer 2 overlay connectivity to use that routed infrastructure. The VXLAN UDP source port can provide entropy for flow-based ECMP hashing.


1.4 — Server Virtualization and MAC Scale

Server virtualization changes the relationship between physical switch ports and network endpoints. A single physical server can host many virtual machines, each with its own virtual network interface and MAC address.

In a conventional Ethernet design, switches learn the MAC addresses of communicating endpoints and store them in forwarding tables. As virtual machine populations increase, the number of learned addresses can grow substantially.

Large Layer 2 domains also require mechanisms for locating unknown destinations. When a destination MAC address is not known, unknown-unicast frames may be flooded within the relevant broadcast domain.

Scaling Factor Traditional Layer 2 Impact VXLAN Fabric Approach
More virtual machines Increasing MAC address learning and forwarding-table requirements Logical segments distributed across VTEPs; EVPN can advertise remote MAC reachability
More application networks Greater demand for VLAN identifiers and trunk configuration 24-bit VNI namespace supports many overlay segments
Workload relocation MAC location changes and possible VLAN extension requirements Remote VTEP mappings can be updated through the overlay control plane
Unknown destinations Flooding within the Layer 2 broadcast domain BUM replication is still required in some cases; EVPN can reduce unnecessary flooding
Redundant fabric links STP may block selected Layer 2 paths Routed underlay supports ECMP across active IP paths

VXLAN does not automatically eliminate MAC table requirements. VTEPs still need forwarding information for local and remote endpoints. However, the overlay architecture allows logical segments to be distributed across a routed fabric.

When combined with BGP EVPN, MAC and IP reachability information can be exchanged through a control plane. This provides more structured endpoint discovery, supports MAC mobility and can reduce reliance on traditional flood-and-learn forwarding.


1.5 — Why Workload Mobility Matters

Virtualization allows workloads to move between physical hosts for maintenance, resource optimisation, resilience and operational flexibility.

In some deployment models, a workload must preserve its IP address and Layer 2 connectivity when it moves. Traditional approaches may require VLANs to be extended across multiple racks or switching domains.

VXLAN enables the same logical overlay segment to be present on different physical leaf switches. A workload attached to VNI 10010 on Leaf A can communicate with another endpoint attached to VNI 10010 on Leaf B, even though the underlying network forwards traffic using IP routing.

In EVPN-based fabrics, control-plane updates can advertise changes in endpoint location. Remote VTEPs can then update their forwarding state to direct traffic toward the appropriate destination VTEP.

Design Consideration

VXLAN provides the network mechanism for extending logical segments, but successful workload mobility also depends on application requirements, gateway design, security policy, address management and orchestration. Extending Layer 2 connectivity across distant sites is not always the preferred design.


1.6 — The Transition to Routed Overlay Fabrics

The evolution from traditional Layer 2 networks to VXLAN-based fabrics represents an important architectural shift. Instead of requiring the physical network to provide every logical Ethernet segment, the network separates two responsibilities:

L3
The Underlay Network

Provides IP connectivity between VTEPs. Routing protocols maintain reachability, ECMP enables path diversity, and the physical topology can scale independently of tenant segmentation.

L2
The VXLAN Overlay

Provides logical Ethernet segments identified by VNIs. VTEPs encapsulate and decapsulate traffic, while mechanisms such as BGP EVPN can distribute endpoint reachability.

This separation enables data centre architects to design a resilient routed fabric while delivering logical connectivity to workloads independently of their physical attachment points.

It also introduces new engineering responsibilities. The underlay must provide reliable VTEP reachability, the overlay must maintain correct endpoint mappings, and the fabric must account for encapsulation overhead, MTU, segmentation and control-plane convergence.

Design Dimension Traditional Approach VXLAN-Based Fabric
Network segmentation 12-bit VLAN identifiers 24-bit VXLAN VNIs
Core transport Often extended Layer 2 switching Routed IP underlay
Path redundancy STP or other L2 redundancy designs IP routing and ECMP
Endpoint learning Data-plane MAC learning Flood-and-learn or EVPN control-plane distribution
Workload mobility May require extended VLAN domains Logical overlay segment extension
Operational visibility Switch ports, VLANs and MAC tables Underlay routes, VTEPs, VNIs, EVPN state and packet encapsulation
Key Engineering Insight

VXLAN does not replace Ethernet, VLANs or IP routing. It combines them into an overlay architecture in which Ethernet segments are mapped to VNIs and transported across an IP network. Local VLANs still commonly connect endpoints to VTEPs, while the routed underlay transports encapsulated traffic.


1.7 — Design Trade-Offs and Operational Reality

VXLAN solves important scaling and architectural challenges, but it is not a universal replacement for traditional Layer 2 networking.

Overlay fabrics introduce additional headers, forwarding state and control-plane dependencies. Troubleshooting must distinguish between failures in the physical IP underlay and failures in the logical VXLAN overlay.

Engineers must also consider hardware forwarding capacity, VNI scale, EVPN route scale, broadcast replication, security policy, operational tooling and the ability to observe encapsulated traffic across the network.

VXLAN Advantage Engineering Responsibility
Large logical segment space VNI allocation, lifecycle management and tenant isolation
Routed underlay transport IP reachability, routing convergence and ECMP path health
Flexible workload placement MAC mobility, gateway consistency and policy enforcement
Overlay encapsulation MTU planning, packet visibility and forwarding performance
EVPN control-plane learning BGP session health, route import/export and endpoint state consistency

VXLAN also provides segmentation rather than encryption. Security still requires appropriate access controls, isolation policies and protection of the transport infrastructure.

Section 1 — Engineering Takeaways

✓ Traditional VLANs use a 12-bit identifier, while VXLAN introduces a 24-bit VNI for a much larger logical segment space.
✓ Large Layer 2 domains can create challenges involving STP, MAC learning, flooding and workload mobility.
✓ Routed leaf-spine fabrics use IP routing and ECMP to provide active path diversity without relying on STP in the physical fabric.
✓ VXLAN transports logical Ethernet segments across the IP underlay, separating tenant connectivity from physical topology.
✓ VXLAN does not eliminate the need for MAC learning, MTU planning, security or careful control-plane design.
Continue Learning · Section 02
VXLAN Architecture: Underlay, Overlay and VTEPs
Network Insight · Engineering Guide
Network Insight · Interactive Engineering Lab 01

Why VXLAN? Explore the Limits of Layer 2

Compare a traditional VLAN-based network with a routed VXLAN overlay. Select an engineering challenge, change the architecture and run a packet journey to see what changes.

Step 1 · Select Architecture
Engineering Focus
Step 2 · Select an Engineering Scenario
Live Network Architecture Traditional Layer 2
TRADITIONAL LAYER 2 CORE Core Switch A Layer 2 switching Core Switch B STP alternate path Access Switch A VLAN 10 Access Switch B VLAN 10 Host A 192.168.10.11 Host B 192.168.10.12 VXLAN VNI 10010 Select a scenario and run the packet journey
Active physical path STP-blocked alternative VXLAN logical tunnel
Ready to investigate
Engineering Analysis
VLAN Scaling in a Traditional Network
Traditional VLAN identifiers use a 12-bit field, providing 4094 normally usable VLAN IDs. Extending large numbers of VLANs across a physical switching fabric increases operational complexity and Layer 2 scope.
Architecture Indicators
Segment Identifier 12-bit
Segment ID Space 4,094
Core Forwarding Layer 2
Path Model STP
Engineering Event Timeline
Select an architecture and scenario. Run the packet journey to inspect forwarding events.
Key Engineering Principle

VXLAN separates logical Layer 2 segments from the physical transport network, enabling Ethernet overlays to operate across a routed IP fabric.

Section 02 · VXLAN Architecture

VXLAN Architecture: Understanding the Underlay, Overlay and VTEPs

VXLAN architecture separates the physical network responsible for IP packet transport from the logical network that provides Ethernet connectivity to applications. This separation is the foundation of modern routed data centre fabrics.

In a VXLAN environment, the underlay network provides routed IP connectivity between tunnel endpoints, while the overlay network represents the virtual Layer 2 segments transported across that infrastructure. VXLAN Tunnel Endpoints (VTEPs) connect these two architectural layers.

Understanding the relationship between the underlay, overlay, VTEPs and VXLAN Network Identifiers (VNIs) is essential for designing, operating and troubleshooting VXLAN fabrics. A failure in any of these components can interrupt workload connectivity even when other parts of the network appear healthy.

IP Underlay VXLAN Overlay VTEP VNI Leaf-Spine ECMP BGP EVPN

2.1 — The Three Architectural Layers

A VXLAN fabric can be understood through three primary architectural components: the physical IP underlay, the logical VXLAN overlay and the VTEPs that perform encapsulation and decapsulation.

TRANSPORT
IP Underlay Network

The routed physical network connecting leaf and spine switches. It provides reachability between VTEP IP addresses and forwards outer IP packets using normal Layer 3 routing.

VIRTUAL NETWORK
VXLAN Overlay Network

Logical Ethernet segments identified by VNIs. The overlay allows workloads connected to different physical leaf switches to participate in the same virtual Layer 2 network.

ENCAPSULATION
VXLAN Tunnel Endpoint

A VTEP encapsulates local Ethernet frames into VXLAN UDP/IP packets and decapsulates incoming VXLAN traffic before forwarding the inner frames toward connected endpoints.

SEGMENT IDENTIFIER
VXLAN Network Identifier

A 24-bit VNI identifies the logical overlay segment associated with encapsulated traffic. VNIs allow many isolated virtual networks to share the same routed fabric.

Architectural Principle

The underlay forwards packets using outer IP addresses. The overlay identifies the logical Ethernet segment using the VNI. The VTEP connects both functions by adding or removing the VXLAN encapsulation.


2.2 — VXLAN Leaf-Spine Fabric Architecture

Modern VXLAN deployments commonly use a leaf-spine architecture. Each leaf switch connects to multiple spine switches, creating a routed topology with predictable path diversity and multiple equal-cost forwarding options.

Leaf switches frequently act as VTEPs, connecting local servers or virtual machines to the overlay. Spine switches provide IP transport between leaf VTEPs and do not normally need to inspect the VXLAN headers or tenant Ethernet frames.

Reference Architecture · Two-Spine / Three-Leaf Fabric
Spine 1 IP Underlay Spine 2 IP Underlay Leaf A VTEP 10.0.0.11 Leaf B VTEP 10.0.0.12 Leaf C VTEP 10.0.0.13 VNI 10010 Host A Host B Host C
Routed IP underlay Logical VXLAN overlay
The dashed VXLAN path represents a logical overlay relationship between VTEPs. Actual encapsulated packets traverse the routed spine-leaf underlay. Leaf C may participate in other VNIs or the same VNI if configured.

The leaf-spine topology is particularly suitable for VXLAN because it provides consistent routed connectivity between VTEPs and can distribute traffic across multiple available paths.

The physical network does not need to extend tenant VLANs through the spines. Instead, each leaf encapsulates local traffic using VXLAN and sends the resulting outer IP packet toward the remote VTEP.


2.3 — The IP Underlay Network

The underlay is responsible for transporting encapsulated VXLAN packets between VTEPs. It must provide reliable IP connectivity between the VTEP source and destination addresses.

In many deployments, each VTEP uses a loopback interface as its VXLAN source address. The underlay routing system advertises reachability to these loopback addresses so that remote VTEPs can forward encapsulated traffic correctly.

Common underlay routing approaches include OSPF, IS-IS and eBGP. The appropriate protocol depends on the fabric architecture, operational requirements and platform capabilities.

Underlay Component Engineering Role Operational Check
Physical links Provide connectivity between leaf and spine switches Interface status, errors, speed and utilization
Routing protocol Advertises reachability to VTEP addresses OSPF, IS-IS or BGP neighbor and route state
VTEP loopback Stable source or destination address for VXLAN packets Route lookup and loopback reachability
ECMP Distributes eligible traffic across equal-cost paths Installed next hops and flow distribution
MTU Accommodates additional VXLAN encapsulation overhead End-to-end packet size and fragmentation behavior
Underlay Failure Scenario

If Leaf A cannot route traffic to Leaf B's VTEP address, the VXLAN overlay cannot deliver encapsulated packets between them. The VNI configuration may be correct, but the data plane will still fail because the underlay lacks usable IP reachability.


2.4 — The VXLAN Overlay Network

The overlay represents the logical networks carried across the IP underlay. Each Layer 2 VXLAN segment is identified by a VNI, and participating VTEPs associate local attachment circuits or VLANs with that identifier.

Consider two servers connected to different leaf switches. Both servers belong to the same logical network, represented by VNI 10010. The servers can exchange Ethernet frames even though the physical switches between them operate as Layer 3 routers.

VXLAN encapsulation allows the original Ethernet frame to remain intact as it crosses the underlay. The receiving VTEP removes the outer encapsulation and forwards the inner frame within the appropriate local bridge domain.

LEAF A
Local VLAN 10

Host A connects to VLAN 10. Leaf A maps this VLAN to VNI 10010 and encapsulates traffic destined for remote endpoints in that overlay segment.

LEAF B
Local VLAN 20

Host B connects to VLAN 20. Leaf B maps VLAN 20 to the same VNI 10010, placing both endpoints in the intended logical Layer 2 segment.

VLAN-to-VNI Mapping

Local VLAN identifiers do not have to match across VTEPs. The VNI identifies the shared overlay bridge domain. However, the mappings must be consistent with the intended tenant segmentation and control-plane configuration.


2.5 — The Role of the VTEP

A VXLAN Tunnel Endpoint is the device or software component responsible for converting Ethernet traffic into VXLAN packets and back again.

VTEPs may exist in physical switches, virtual switches, hypervisors or other supported network devices. In a common hardware leaf-spine design, the leaf switches act as VTEPs.

VTEP Function Description
Encapsulation Adds VXLAN, UDP, IP and outer Ethernet headers to an original Ethernet frame
Decapsulation Removes the outer headers and restores the original Ethernet frame for local forwarding
VNI mapping Associates local bridge domains or VLANs with VXLAN segment identifiers
Remote endpoint forwarding Determines the destination VTEP for known remote MAC addresses
BUM replication Handles broadcast, unknown unicast and multicast traffic using the configured replication mechanism
EVPN integration Uses advertised MAC/IP reachability information when BGP EVPN is deployed

A VTEP needs to determine both the logical segment associated with a frame and the remote VTEP that should receive it. The method used to discover remote endpoints depends on whether the fabric uses flood-and-learn or a control plane such as BGP EVPN.


2.6 — Example VTEP Configuration

The following simplified Cisco NX-OS-style example illustrates the relationship between a local VLAN, VNI and NVE interface. The example is conceptual and may require additional platform- specific commands, feature enablement and control-plane configuration.

Leaf A · Illustrative VXLAN Configuration NX-OS Style
feature nv overlay

vlan 10
  vn-segment 10010

interface loopback1
  ip address 10.0.0.11/32

interface nve1
  no shutdown
  host-reachability protocol bgp
  source-interface loopback1
  member vni 10010
    ingress-replication protocol bgp
Illustrative syntax only. A functional EVPN fabric also requires underlay reachability, BGP EVPN configuration, appropriate route targets and other platform-dependent settings.

In this example, VLAN 10 maps to VNI 10010. Loopback1 provides the local VTEP source address, and interface NVE1 represents the VXLAN tunnel interface.

The BGP EVPN control plane can advertise reachability information and support ingress replication membership. However, the exact configuration model varies between hardware platforms, operating systems and software versions.


2.7 — Packet Forwarding Across the Fabric

When a local endpoint sends an Ethernet frame to a remote endpoint in the same VNI, the ingress VTEP encapsulates the frame and forwards it across the IP underlay.

01
Host A
Sends an ordinary Ethernet frame
02
Leaf A
Maps VLAN to VNI and encapsulates
03
Underlay
Routes outer IP packet toward VTEP
04
Leaf B
Decapsulates and selects bridge domain
05
Host B
Receives original Ethernet frame

The original source and destination MAC addresses remain part of the inner Ethernet frame. The outer IP header identifies the source and destination VTEPs.

Intermediate spine routers forward the packet based on the outer destination IP address. They do not need to learn tenant MAC addresses or participate in the tenant Layer 2 bridge domain.

Packet Forwarding Insight

The VXLAN tunnel is a logical relationship between VTEPs. It is not necessarily a dedicated physical path. Encapsulated packets may traverse different equal-cost underlay routes depending on the ECMP hashing algorithm and network state.


2.8 — ECMP and Underlay Path Diversity

One of the major advantages of a routed leaf-spine underlay is the ability to use multiple equal-cost paths simultaneously.

VXLAN uses UDP encapsulation, allowing the underlay to distinguish flows through outer header fields. Many implementations derive the UDP source port from a hash of inner packet information to provide entropy for ECMP.

This enables different encapsulated flows to be distributed across available underlay next hops. However, ECMP is commonly flow-based, so a single flow may continue to use one selected path rather than being distributed packet by packet.

NORMAL OPERATION
Multiple Active Paths

Leaf A can reach Leaf B through Spine 1 or Spine 2. The underlay may install both equal-cost next hops and distribute eligible VXLAN flows across them.

LINK FAILURE
Underlay Reconvergence

If one path fails, the routing system removes the unavailable next hop. Traffic can continue over remaining paths, subject to detection, convergence and forwarding updates.


2.9 — Where BGP EVPN Fits

VXLAN defines a data-plane encapsulation mechanism. It does not, by itself, define a complete mechanism for distributing remote endpoint reachability throughout the fabric.

Early VXLAN deployments commonly relied on data-plane learning and multicast-based replication. Modern data centre fabrics frequently combine VXLAN with BGP Ethernet VPN (EVPN) to provide a scalable control-plane mechanism.

EVPN can advertise MAC/IP reachability, identify participating VTEPs and support endpoint mobility. It also enables features such as distributed gateway architectures and inter-subnet routing.

Component Responsibility
VXLAN Encapsulates Ethernet traffic inside UDP/IP packets
IP Underlay Provides routed reachability between VTEP addresses
BGP EVPN Distributes endpoint reachability and overlay control-plane information
VTEP Performs encapsulation, decapsulation and overlay forwarding
VNI Identifies the logical overlay segment or routing context
VXLAN and EVPN Are Complementary

VXLAN is the data-plane encapsulation technology. BGP EVPN is a control-plane technology used to distribute overlay reachability information. They are commonly deployed together, but they perform different architectural roles.


2.10 — VXLAN Architecture Design Requirements

A successful VXLAN fabric depends on more than enabling an NVE interface and assigning a VNI. The underlay and overlay must be designed as related but independently observable systems.

Requirement Why It Matters
VTEP IP reachability Remote tunnel endpoints must be reachable through the routed underlay
Consistent VNI mappings Logical segments must map to the intended bridge domains
Appropriate MTU Encapsulation adds header overhead beyond the original frame size
Control-plane stability EVPN routes and endpoint mappings must remain accurate during changes
Resilient underlay Routing and ECMP must recover from physical path failures
Tenant isolation VNIs, VRFs and security policy must prevent unintended connectivity
Operational visibility Engineers need to inspect routing, NVE peers, VNIs, EVPN routes and forwarding state

Troubleshooting should begin by determining whether the fault exists in the underlay, overlay or endpoint attachment configuration.

For example, a missing VTEP route indicates an underlay reachability problem, while a VNI mismatch or missing EVPN route may indicate an overlay configuration or control-plane problem.

Operational Design Warning

A healthy BGP EVPN session does not guarantee successful VXLAN forwarding. The fabric must also have correct VNI membership, valid remote endpoint mappings, usable VTEP routes, sufficient MTU and appropriate forwarding policies.

Section 2 — Engineering Takeaways

✓ The IP underlay provides routed transport between VXLAN tunnel endpoints.
✓ The VXLAN overlay provides logical Ethernet segments identified by 24-bit VNIs.
✓ VTEPs encapsulate and decapsulate Ethernet frames as they enter and leave the routed fabric.
✓ ECMP enables eligible encapsulated flows to use multiple active underlay paths.
✓ BGP EVPN complements VXLAN by distributing endpoint and overlay reachability information.
✓ VXLAN troubleshooting requires visibility into both the routed underlay and the logical overlay.
Continue Learning · Section 03
VXLAN Encapsulation and Packet Forwarding
Network Insight · Interactive Engineering Lab 02

VXLAN Architecture Explorer: Underlay, Overlay and VTEPs

Investigate a two-leaf, two-spine VXLAN fabric. Select individual devices to inspect their roles, switch between physical and logical network views, and trace the journey of an Ethernet frame across the IP underlay.

Step 1 · Select Architecture View
Interactive Leaf-Spine Architecture IP Underlay
ROUTED LEAF-SPINE FABRIC VXLAN · VNI 10010 Spine 1 IP Router Spine 2 IP Router Leaf A VTEP 10.0.0.11 Leaf B VTEP 10.0.0.12 Host A 192.168.10.11 Host B 192.168.10.12
Routed IP links Logical VXLAN tunnel Select any device to inspect it
Selected Network Component

Leaf A · VXLAN Tunnel Endpoint

Leaf A is a VTEP. It receives Ethernet frames from Host A, maps the local VLAN to a VNI and encapsulates frames for remote VTEPs.

Component Details
Role
VTEP
Address
10.0.0.11
Layer
Overlay Edge
Function
Encapsulation
Select a device or run a trace
Packet Forwarding Timeline
Select a topology component to inspect it. Run the trace to see encapsulation and forwarding.
Key Architecture Principle

The underlay transports IP packets between VTEPs. The VXLAN overlay provides logical Ethernet connectivity between workloads. The spines forward on outer IP headers and do not normally need to inspect the inner Ethernet frame.

Conceptual VXLAN architecture lab. The example illustrates same-subnet forwarding over one selected spine path. It does not emulate BGP EVPN, routing convergence or ECMP hashing.
Section 03 · VXLAN Data Plane

VXLAN Encapsulation and Packet Forwarding

VXLAN enables an Ethernet frame to cross a routed IP network without requiring intermediate routers to participate in the original Layer 2 broadcast domain. It achieves this by encapsulating the Ethernet frame inside a VXLAN header, UDP datagram and outer IP packet.

The original Ethernet frame remains the inner frame, while the newly added transport headers identify the source and destination VXLAN Tunnel Endpoints (VTEPs). This separation allows the physical network to forward traffic using ordinary IP routing while the overlay preserves logical Ethernet connectivity between workloads.

Understanding the exact encapsulation format is essential for troubleshooting VXLAN fabrics. Engineers must distinguish the inner and outer addressing information, verify the correct VXLAN Network Identifier (VNI), account for encapsulation overhead and confirm that the receiving VTEP can decapsulate and forward the original frame.

Ethernet Frame VXLAN Header UDP 4789 VNI VTEP MTU ECMP Hashing

3.1 — What Happens During VXLAN Encapsulation?

When Host A sends an Ethernet frame toward Host B in the same logical VXLAN segment, the ingress VTEP first determines the bridge domain and VNI associated with the incoming frame.

If the destination MAC address is reachable through a remote VTEP, the ingress device encapsulates the original Ethernet frame with a VXLAN header and additional UDP/IP transport headers.

The resulting packet is forwarded through the routed underlay toward the destination VTEP. Intermediate routers inspect the outer IP destination address and do not need to understand the tenant's inner Ethernet addressing.

INNER NETWORK
Original Ethernet Frame

Contains the original source and destination MAC addresses, optional VLAN information and the payload generated by the communicating hosts.

OUTER NETWORK
VXLAN Transport Headers

Include the VXLAN VNI, UDP header, outer source and destination IP addresses and the Ethernet header used on the current physical link.

Fundamental Data-Plane Principle

The inner Ethernet frame identifies the communicating endpoints. The outer IP header identifies the VTEPs transporting that frame. The VNI identifies the logical overlay segment in which the frame belongs.


3.2 — The Complete VXLAN Packet Structure

A conventional VXLAN packet consists of an original Ethernet frame encapsulated inside an 8-byte VXLAN header, an 8-byte UDP header and an outer IP packet. An outer Ethernet header is also used when the encapsulated packet is transmitted across an Ethernet underlay link.

Packet Anatomy · Ethernet over VXLAN/UDP/IPv4
Outer Ethernet
Physical next-hop MAC addresses · 14 bytes without an 802.1Q tag
Outer IPv4 Header
Source VTEP: 10.0.0.11 · Destination VTEP: 10.0.0.12 · 20 bytes without IP options
UDP Header
Variable source port · Destination port 4789 · 8 bytes
VXLAN Header
VNI 10010   8-byte header identifying the logical overlay segment
Inner Ethernet
Original source and destination host MAC addresses · 14 bytes without an inner VLAN tag
Inner Payload
Original IPv4/IPv6 packet, ARP message or other Ethernet payload
Packet layers are shown from the outer Ethernet header toward the encapsulated payload. The Ethernet FCS, optional VLAN tags and additional IP options are not included in this simplified diagram.

The outer Ethernet header changes as the encapsulated packet moves between physical Layer 3 hops. By contrast, the outer source and destination VTEP IP addresses normally remain constant across the underlay path.

The inner Ethernet frame is preserved through the VXLAN tunnel, allowing the destination VTEP to deliver it into the corresponding local bridge domain after decapsulation.


3.3 — Understanding the 8-Byte VXLAN Header

The standard VXLAN header defined by RFC 7348 is eight bytes long. Its most important field is the 24-bit VXLAN Network Identifier, which identifies the logical network segment associated with the encapsulated frame.

The header also contains flags and reserved fields. The I flag indicates that the VNI field is valid. In the original VXLAN format, the remaining reserved bits are transmitted as zero.

VXLAN Header · 8 Bytes / 64 Bits
Byte 0 Flags
Byte 1 RSVD
Byte 2 RSVD
Byte 3 RSVD
Byte 4 VNI
Byte 5 VNI
Byte 6 VNI
Byte 7 RSVD
The VNI occupies bytes 4–6. The I flag is bit 3 of the first byte, conventionally represented by the flag value 0x08. This diagram represents the original VXLAN header, not VXLAN-GPE.
Field Size Purpose
Flags 8 bits Includes the I flag indicating that the VNI is valid
Reserved 24 bits Reserved fields in the original VXLAN header
VNI 24 bits Identifies the logical VXLAN segment
Reserved 8 bits Reserved final byte
Why the VNI Matters

A VTEP uses the VNI to select the intended overlay bridge domain. If the receiving VTEP does not have the appropriate VNI configured, it cannot correctly deliver the frame into that logical segment.


3.4 — Inner and Outer Addressing

VXLAN packets contain two addressing contexts. The inner headers describe the communicating workloads, while the outer headers describe how the encapsulated packet travels through the physical IP fabric.

Address Field Example Function
Inner source MAC 00:11:22:33:44:01 Original Host A MAC
Inner destination MAC 00:11:22:33:44:02 Original Host B MAC
Inner source IP 192.168.10.11 Host A IP address
Inner destination IP 192.168.10.12 Host B IP address
Outer source IP 10.0.0.11 Leaf A VTEP address
Outer destination IP 10.0.0.12 Leaf B VTEP address
VXLAN VNI 10010 Overlay bridge domain

Spine routers use the outer destination IP address to forward the encapsulated packet. They do not need routes for the inner host IP addresses when simply providing underlay transport.

The receiving VTEP uses the VXLAN header and local forwarding information to identify the correct bridge domain and deliver the inner frame.


3.5 — End-to-End VXLAN Packet Journey

Consider Host A connected to Leaf A and Host B connected to Leaf B. Both hosts belong to VNI 10010, and the underlay provides IP connectivity between the two VTEPs.

01
Host A
Sends an Ethernet frame toward Host B
02
Leaf A
Selects VNI 10010 and encapsulates
03
Spine
Routes the outer IP packet
04
Leaf B
Removes VXLAN encapsulation
05
Host B
Receives the original frame
Step 1 — Original Ethernet Frame

Host A creates an Ethernet frame containing Host B's destination MAC address. The frame enters Leaf A through a local access interface associated with the intended bridge domain.

Step 2 — VNI and Remote VTEP Selection

Leaf A identifies the local bridge domain and associated VNI. Its forwarding information indicates that Host B's MAC address is reachable through Leaf B's VTEP.

This remote mapping may have been learned through flood-and-learn forwarding or advertised by BGP EVPN.

Step 3 — Encapsulation and IP Forwarding

Leaf A adds the VXLAN, UDP and outer IP headers. The outer destination becomes Leaf B's VTEP address, 10.0.0.12.

The underlay routing table determines the next hop. If multiple equal-cost paths exist, ECMP hashing may select one of them.

Step 4 — Decapsulation at Leaf B

Leaf B receives the VXLAN packet, validates the relevant encapsulation information and identifies VNI 10010.

It removes the outer headers and processes the original Ethernet frame in the associated local bridge domain.

Step 5 — Delivery to Host B

Leaf B forwards the recovered Ethernet frame toward Host B through the appropriate local attachment interface.

Forwarding Assumption

This example assumes that the destination MAC-to-VTEP mapping is already known. If the destination is unknown, the fabric may need to perform BUM replication or another endpoint discovery operation. These mechanisms are covered in Section 4.


3.6 — VXLAN UDP Ports and ECMP Entropy

Standard VXLAN uses UDP destination port 4789. The UDP source port is commonly derived from a hash of the inner packet fields.

This variable source port provides additional entropy for underlay ECMP hashing. Without suitable entropy, many VXLAN flows between the same pair of VTEPs could hash onto the same physical path.

UDP DESTINATION
Port 4789

The standard VXLAN destination port. Receiving VTEPs use the configured port to identify VXLAN traffic for processing.

UDP SOURCE
Flow-Based Entropy

A source port chosen using a hash of inner traffic fields can help distribute different flows across ECMP paths.

ECMP is typically flow-based rather than packet-based. A single flow may remain on one underlay path, while different flows can be distributed across other available paths.

Legacy VXLAN Port Configurations

Some older VXLAN implementations and lab environments used non-standard or configurable UDP destination ports. Your original VXLAN lab used port 1024 explicitly. Both VTEPs must use compatible port settings. For new standards-based deployments, UDP 4789 is the normal choice.


3.7 — VXLAN Encapsulation Overhead and MTU

Encapsulation increases packet size because additional headers are inserted around the original Ethernet frame.

For a typical VXLAN packet transported over IPv4, the additional headers consist of:

Engineering Calculation · VXLAN over IPv4
14 Inner Ethernet Header
8 VXLAN Header
8 UDP Header
20 Outer IPv4 Header
14 + 8 + 8 + 20 = 50 Bytes of Additional Encapsulated IP Payload

The 50-byte calculation assumes an untagged inner Ethernet frame, a minimum 20-byte outer IPv4 header and no optional extensions. It excludes the outer Ethernet header and FCS, which are link-layer overhead.

If the original host sends a 1,500-byte IP packet, the VXLAN encapsulation produces a 1,550-byte outer IPv4 packet when the inner Ethernet header is included.

Therefore, an underlay IP MTU of at least 1,550 bytes is required for this specific untagged IPv4 example without outer IP fragmentation.

Many data centre operators configure larger MTUs, often using jumbo-frame support, to provide additional headroom for encapsulation, VLAN tags and other transport requirements.

MTU Design Warning

VXLAN does not automatically require a particular jumbo MTU such as 9,000 bytes. The required underlay MTU depends on the inner packet size and encapsulation format. Insufficient MTU can cause fragmentation, packet drops or application-level failures.


3.8 — VXLAN Packet Capture Analysis

Packet captures are one of the most useful tools for understanding VXLAN forwarding. A capture taken on an underlay link reveals the outer Ethernet, IP, UDP and VXLAN headers.

In Wireshark, an engineer can inspect the outer source and destination VTEP addresses, UDP destination port, VNI and encapsulated Ethernet frame.

Illustrative Packet Capture · VXLAN over IPv4
Frame: VXLAN encapsulated ICMP

Outer Ethernet:
  Source MAC:      Underlay next-hop MAC
  Destination MAC: Underlay next-hop MAC

Outer IPv4:
  Source IP:       10.0.0.11
  Destination IP:  10.0.0.12
  Protocol:        UDP (17)

UDP:
  Source Port:     52341
  Destination:     4789

VXLAN:
  Flags:           0x08 (I flag set)
  VNI:             10010

Inner Ethernet:
  Source MAC:      00:11:22:33:44:01
  Destination MAC: 00:11:22:33:44:02

Inner IPv4:
  Source IP:       192.168.10.11
  Destination IP:  192.168.10.12
  Protocol:        ICMP

ICMP:
  Type:            Echo Request
Illustrative decoded packet fields, not an actual captured packet. MAC addresses and UDP source ports vary according to the physical link and implementation.

A useful first check is to confirm that the outer destination IP address matches the expected remote VTEP.

The next check is the VNI. A correct VTEP destination with an incorrect VNI can still result in failed forwarding.

Finally, the inner Ethernet and IP headers should match the expected workload traffic. This helps determine whether the issue exists before encapsulation, inside the overlay or after decapsulation.


3.9 — Common VXLAN Packet Forwarding Failures

Failure Likely Symptom Engineering Check
Incorrect VNI Remote VTEP cannot deliver traffic into intended segment Verify local VLAN-to-VNI mappings and NVE membership
Missing VTEP route Encapsulated packets cannot reach remote VTEP Check underlay routing and VTEP reachability
UDP port mismatch Receiver may not recognise VXLAN packets Verify configured VXLAN destination port
Insufficient MTU Large packets fail while smaller packets work Test end-to-end encapsulated packet sizes
Missing remote MAC mapping Traffic is flooded or cannot be forwarded Inspect MAC tables and EVPN reachability
Underlay path failure Packet loss or temporary interruption Verify routing convergence and ECMP next hops
Security policy drop Encapsulated traffic is filtered in transit Inspect ACLs, firewalls and UDP 4789 reachability
Troubleshooting Method

Follow the packet in order: original host frame, local bridge-domain mapping, remote MAC-to-VTEP resolution, VXLAN encapsulation, underlay IP forwarding, destination VTEP decapsulation and final host delivery. This method helps isolate the failing stage rather than treating the overlay as one opaque tunnel.


3.10 — Operational Verification Commands

On a Cisco NX-OS-based VXLAN fabric, several operational commands can help validate packet forwarding. Exact output and availability vary by platform and release.

VXLAN Verification · NX-OS-Style CLI
show nve interface
show nve peers
show nve vni

show mac address-table
show l2route evpn mac all

show ip route 10.0.0.12
show ip interface brief

show bgp l2vpn evpn summary
show bgp l2vpn evpn

show interface counters errors
Use the commands supported by the specific platform and NX-OS release. Some EVPN commands are applicable only when BGP EVPN is configured.

A successful troubleshooting investigation should confirm that the local VTEP is operational, the expected VNI is active, the remote VTEP is reachable and the required forwarding information exists.

If the fabric uses BGP EVPN, the engineer should also inspect EVPN routes and their imported reachability information. However, a healthy BGP session alone does not prove successful VXLAN data-plane forwarding.

Section 3 — Engineering Takeaways

✓ VXLAN encapsulates the original Ethernet frame inside an 8-byte VXLAN header, UDP and outer IP packet.
✓ The inner headers identify workloads, while the outer IP addresses identify the source and destination VTEPs.
✓ The 24-bit VNI identifies the logical overlay segment and is essential for correct decapsulation.
✓ UDP destination port 4789 is the standard VXLAN port, while source-port entropy can support underlay ECMP.
✓ VXLAN over IPv4 typically adds 50 bytes to the underlay IP packet size for an untagged inner Ethernet frame.
✓ Packet captures and VTEP/VNI verification help distinguish encapsulation faults from underlay routing and endpoint forwarding problems.
Continue Learning · Section 04
VXLAN Flood-and-Learn and the BGP EVPN Control Plane
Network Insight · Interactive Engineering Lab 03

VXLAN Encapsulation and Packet Header Explorer

Build and inspect a VXLAN packet. Explore how the original Ethernet frame is wrapped in VXLAN, UDP, IP and Ethernet headers, then trace the packet across the underlay and examine its MTU requirements.

VXLAN Packet Journey Ready to Encapsulate
Host A Leaf A Spine Leaf B Host B Source Encapsulate IP Route Decapsulate Destination VXLAN Transport · UDP Destination Port 4789 Original Ethernet frame awaits encapsulation
Select a Packet Layer to Inspect Its Fields
Packet Header Inspector

Inner Ethernet Header

The original Ethernet header identifies the source and destination MAC addresses. VXLAN transports this frame across the routed IP network without requiring intermediate spines to inspect its inner MAC addresses.

Header Fields
Field 1 Source MAC
Field 2 Destination MAC
Header Size 14 bytes
Purpose Tenant Ethernet
Original Inner IP Packet 1500 B Excludes inner Ethernet header
Outer IP Packet Size 1550 B Compared with underlay IP MTU
Encapsulation Overhead +50 B 14 inner Ethernet + 20 IPv4 + 8 UDP + 8 VXLAN
Underlay MTU is sufficient for this packet.
Select a header or run the animation
Packet Encapsulation Event Timeline
[READY] Host A has an Ethernet frame. [READY] Select a header to inspect its fields. [READY] Run the animation to trace encapsulation.
Key Engineering Principle

VXLAN preserves the original Ethernet frame inside a new transport packet. The underlay forwards using the outer IP destination, while the remote VTEP removes encapsulation and delivers the original Ethernet frame. Always account for encapsulation overhead when planning the underlay MTU.

Section 04 · VXLAN Control Plane

VXLAN Flood-and-Learn and the BGP EVPN Control Plane

VXLAN defines how Ethernet frames are encapsulated and transported across an IP network. However, encapsulation alone does not explain how a VTEP discovers remote MAC addresses, identifies participating tunnel endpoints or determines where unknown traffic should be sent.

These functions are handled through endpoint learning and control-plane mechanisms. Traditional VXLAN deployments commonly relied on flood-and-learn forwarding, while modern data centre fabrics frequently use BGP Ethernet VPN (EVPN) to distribute endpoint reachability and overlay membership information.

Understanding both approaches is important because they solve similar forwarding problems through fundamentally different mechanisms. Flood-and-learn depends heavily on data-plane observation, whereas EVPN enables endpoint information to be advertised through a routing control plane.

Flood-and-Learn BUM Traffic Multicast Ingress Replication BGP EVPN Route Type 2 Route Type 3 ARP Suppression

4.1 — Why VXLAN Needs Endpoint Discovery

Consider two servers connected to different leaf switches in the same VXLAN segment. Host A is connected to Leaf A, while Host B is connected to Leaf B.

Before Leaf A can send a known-unicast VXLAN packet directly to Leaf B, it must determine which remote VTEP is responsible for Host B's destination MAC address.

This requires a mapping between the remote MAC address and the destination VTEP IP address.

ENDPOINT LOCATION
MAC-to-VTEP Mapping

Associates a remote endpoint MAC address with the VTEP IP address through which the endpoint is reachable.

OVERLAY MEMBERSHIP
VNI Participation

Identifies the VTEPs participating in a particular VXLAN segment, allowing appropriate forwarding and replication.

The Endpoint Discovery Problem

VXLAN encapsulation provides a way to transport a frame, but the ingress VTEP must still determine the correct remote VTEP destination. Flood-and-learn and BGP EVPN provide different methods for discovering and maintaining that forwarding information.


4.2 — Traditional VXLAN Flood-and-Learn

Traditional VXLAN flood-and-learn operates in a way that resembles conventional Ethernet switching. A VTEP learns MAC addresses from observed traffic and uses flooding when the destination is unknown.

When an ingress VTEP receives a frame destined for an unknown MAC address, it replicates the frame to other participating VTEPs using the configured BUM delivery mechanism.

When the destination endpoint responds, the receiving VTEPs can learn the association between the source MAC address and the originating VTEP address.

01
Unknown Destination MAC
Host A sends a frame toward Host B. Leaf A does not yet know which remote VTEP owns Host B's MAC address.
02
Flooding Across the Overlay
Leaf A sends copies to the appropriate remote VTEPs using multicast or ingress replication.
03
Destination Receives the Frame
Leaf B forwards the received frame into the local bridge domain containing Host B.
04
Source MAC Learning
VTEPs learn remote source MAC-to-VTEP associations from observed encapsulated traffic.
05
Subsequent Known Unicast
Once the required destination mapping is learned, traffic can be sent directly to the appropriate VTEP.

Flood-and-learn is conceptually straightforward, but it introduces operational considerations around flooding scope, replication, MAC aging and endpoint mobility.


4.3 — Understanding BUM Traffic

BUM stands for Broadcast, Unknown Unicast and Multicast. These traffic categories require special handling because the ingress VTEP may not have a single known destination VTEP.

Traffic Type Example Typical VXLAN Handling
Broadcast ARP request Replicated to participating VTEPs unless locally suppressed or otherwise handled
Unknown Unicast Frame with an unknown destination MAC Flooded within the relevant overlay segment when unknown-unicast flooding is enabled
Multicast Application multicast Ethernet traffic Delivered according to the configured overlay multicast and replication behavior

BUM replication must be constrained to the appropriate VXLAN segment. A VTEP should not replicate tenant traffic indiscriminately across unrelated VNIs.

Flooding and Scale

Excessive BUM traffic can consume bandwidth and processing resources. Large VXLAN environments therefore benefit from careful replication design, endpoint learning controls and features such as ARP suppression.


4.4 — Multicast Versus Ingress Replication

VXLAN can distribute BUM traffic through underlay multicast or through ingress replication. Both approaches deliver copies to participating VTEPs, but they differ in where packet replication occurs.

UNDERLAY MULTICAST
Multicast-Based Replication

VTEPs send VXLAN BUM traffic toward a multicast group. Multicast-capable underlay devices replicate traffic along the distribution tree.

HEAD-END REPLICATION
Ingress Replication

The ingress VTEP creates individual VXLAN copies for remote VTEPs in the replication list. Underlay multicast is not required.

Characteristic Multicast Ingress Replication
Underlay multicast Required Not required
Replication location Multicast distribution tree in the network Ingress VTEP
Source VTEP load Generally fewer source-generated copies Increases with the number of remote replication destinations
Operational dependency Multicast routing and group membership Accurate remote VTEP replication list
Common deployment Traditional multicast-based VXLAN fabrics Frequently used with BGP EVPN

Ingress replication is popular in EVPN fabrics because the control plane can distribute the information needed to construct replication lists without requiring multicast routing throughout the underlay.


4.5 — Introducing the BGP EVPN Control Plane

BGP EVPN uses Multiprotocol BGP to distribute overlay reachability information between participating network devices.

Rather than relying exclusively on observed VXLAN traffic to learn remote endpoint locations, EVPN allows VTEPs to advertise reachability information through BGP routes.

This separates the control plane, responsible for distributing endpoint information, from the data plane, responsible for carrying encapsulated Ethernet traffic.

Reference Architecture · EVPN Control Plane and VXLAN Data Plane
BGP EVPN RR Overlay Route Distribution Spine 1 Spine 2 Leaf A VTEP .11 Leaf B VTEP .12 Leaf C VTEP .13 Host A Host B Host C
IP underlay EVPN control-plane sessions
Conceptual EVPN route-reflector topology. The dashed green paths represent logical BGP EVPN sessions, not dedicated physical links. The route reflector distributes EVPN reachability information, while VXLAN data traffic travels directly between VTEPs through the routed underlay.

A BGP EVPN route reflector can distribute overlay reachability information between leaf VTEPs without becoming part of the VXLAN data path.

This architecture supports scalable endpoint learning and allows network engineers to inspect the control-plane state independently from the forwarding path.


4.6 — EVPN Route Type 2: MAC/IP Advertisement

EVPN Route Type 2 is the MAC/IP Advertisement Route. It allows a VTEP to advertise endpoint MAC reachability and, when applicable, associated IP address information.

For example, Leaf B can advertise that Host B's MAC address is reachable through Leaf B's VTEP.

Leaf A can import the appropriate EVPN route and install forwarding information for Host B without needing to discover the remote MAC location solely through flooding.

Conceptual EVPN Type 2 Advertisement
EVPN Route Type: 2
Route: MAC/IP Advertisement

MAC Address:
  00:11:22:33:44:02

IP Address:
  192.168.10.12

Advertising VTEP:
  10.0.0.12

Overlay Segment:
  VNI 10010

BGP Next Hop:
  10.0.0.12

Result:
  Remote MAC reachability learned
  through the EVPN control plane
Simplified route illustration. Actual EVPN NLRI includes additional fields such as the route distinguisher, Ethernet segment identifier and Ethernet tag ID. Route targets and encapsulation-related attributes influence import and forwarding.
Control-Plane Learning

Type 2 routes reduce dependence on data-plane flooding for discovering known remote endpoints. The receiving VTEP can install the advertised MAC-to-VTEP mapping after validating and importing the route.


4.7 — EVPN Route Type 3: Inclusive Multicast Ethernet Tag

EVPN Route Type 3 is the Inclusive Multicast Ethernet Tag (IMET) Route. It advertises information used to establish participation in a broadcast domain and construct BUM delivery state.

In an ingress-replication design, IMET routes can help VTEPs discover the remote peers that should receive replicated BUM traffic for the relevant overlay segment.

EVPN ROUTE TYPE 2
Endpoint Reachability

Advertises MAC addresses and optional IP bindings, enabling remote endpoint forwarding information to be installed.

EVPN ROUTE TYPE 3
Broadcast Domain Membership

Supports discovery of participating VTEPs and establishment of BUM delivery state, including ingress replication lists.

Route Type Primary Role Operational Importance
Type 2 MAC/IP Advertisement Known-unicast endpoint reachability
Type 3 Inclusive Multicast Ethernet Tag BUM membership and replication information
Type 1 Ethernet Auto-Discovery Multihoming procedures and mass withdrawal
Type 4 Ethernet Segment Ethernet segment discovery and designated forwarder election
Type 5 IP Prefix Advertisement IP prefix reachability in EVPN-based routing architectures

Type 2 and Type 3 are particularly important when understanding basic VXLAN EVPN Layer 2 forwarding.

Other EVPN route types support more advanced designs, including multihoming, distributed routing and IP prefix advertisement.


4.8 — EVPN Endpoint Learning Sequence

The following example shows how Host B's reachability can become available to Leaf A through BGP EVPN.

01
Host B Becomes Active
Host B sends traffic through Leaf B. Leaf B learns the local endpoint MAC address and, where applicable, its IP binding.
02
Leaf B Advertises Type 2
Leaf B originates an EVPN MAC/IP Advertisement Route containing the endpoint reachability information.
03
Route Reflector Distributes EVPN
The EVPN route reflector distributes the route to appropriate BGP EVPN peers according to routing policy.
04
Leaf A Imports the Route
Leaf A validates the route and imports the endpoint mapping into the appropriate overlay forwarding context.
05
Known-Unicast VXLAN Forwarding
Leaf A can encapsulate traffic for Host B directly toward Leaf B's VTEP, assuming underlay reachability and valid forwarding state.
Control Plane Versus Data Plane

EVPN distributes information about where an endpoint is located. VXLAN carries the actual Ethernet frame. The BGP route reflector normally participates in control-plane distribution but does not need to forward the encapsulated tenant data traffic.


4.9 — ARP Suppression in VXLAN EVPN

In a conventional Layer 2 network, ARP requests are broadcast to discover the MAC address associated with an IPv4 address.

In a VXLAN fabric, uncontrolled ARP broadcasting can create additional BUM traffic across the overlay.

With EVPN-based ARP suppression, a VTEP can use known IP-to-MAC bindings to answer certain ARP requests locally, reducing the need to flood those requests to remote VTEPs.

WITHOUT SUPPRESSION
ARP Flooding

An ARP request may be replicated across the VXLAN segment to discover the destination MAC address.

WITH SUPPRESSION
Local ARP Response

A VTEP with a valid learned IP-to-MAC binding may answer the ARP request locally without flooding it.

Why ARP Suppression Helps

ARP suppression reduces unnecessary broadcast traffic when valid endpoint bindings are available. Unknown or unresolved requests may still require flooding or other resolution mechanisms, depending on platform behavior and configuration.


4.10 — MAC Mobility and Endpoint Movement

Virtual machines and workloads may move between hosts or leaf switches while retaining their original MAC and IP addresses.

This creates a forwarding challenge: remote VTEPs must update their endpoint mappings when a MAC address becomes reachable through a different VTEP.

BGP EVPN supports MAC mobility procedures, including a MAC Mobility extended community and sequence numbers used to help distinguish newer endpoint location advertisements.

01
Original Endpoint Location
Host B is reachable through Leaf B. Remote VTEPs have imported the corresponding EVPN Type 2 route.
02
Workload Moves to Leaf C
Host B moves to an attachment point behind Leaf C while retaining its MAC address.
03
Updated EVPN Advertisement
Leaf C advertises updated MAC reachability. MAC mobility information helps remote devices select the newer endpoint location.
04
Forwarding State Converges
Remote VTEPs update the MAC-to-VTEP mapping and forward subsequent traffic toward Leaf C.

MAC mobility is especially important in virtualized environments where workload placement can change without requiring the endpoint to obtain a new IP address.

MAC Flapping Versus Legitimate Mobility

Repeated movement of the same MAC address between VTEPs can indicate legitimate workload migration, but it may also indicate a loop, duplicate MAC address or configuration error. Engineers should inspect mobility events and sequence numbers before assuming the movement is expected.


4.11 — Flood-and-Learn Versus BGP EVPN

Capability Flood-and-Learn BGP EVPN
Remote MAC learning Observed data-plane traffic Control-plane Type 2 advertisements
Unknown destinations Typically flooded May be flooded or handled by policy and platform features
VTEP discovery Multicast membership or configured peer lists EVPN membership information and replication state
ARP suppression Not inherent to basic flood-and-learn Supported when suitable IP/MAC bindings are available
MAC mobility Data-plane relearning EVPN mobility procedures and sequence numbers
Operational visibility Primarily forwarding and MAC tables BGP EVPN routes, forwarding state and VTEP tables
Typical architecture Simpler or legacy VXLAN environments Modern scalable VXLAN EVPN fabrics

BGP EVPN provides a more structured control plane for endpoint distribution, but it also introduces additional operational dependencies.

Engineers must verify BGP neighbor state, EVPN address-family activation, route import policies, route targets, VNI mappings and underlay reachability.


4.12 — EVPN Control-Plane Verification

A healthy VXLAN EVPN fabric requires consistency between BGP EVPN routes and the actual VXLAN forwarding state installed on the VTEPs.

The following Cisco NX-OS-style commands provide useful starting points for verifying EVPN reachability and replication.

EVPN Operational Verification · NX-OS Style
show bgp l2vpn evpn summary

show bgp l2vpn evpn

show bgp l2vpn evpn route-type 2

show bgp l2vpn evpn route-type 3

show nve peers

show nve vni

show l2route evpn mac all

show mac address-table

show ip route 10.0.0.12
Illustrative NX-OS commands. Exact syntax, supported route-type filters and output vary by platform and software release.
Check Expected Result
BGP EVPN neighbors Expected sessions are Established
Type 2 routes Remote endpoint reachability is advertised and imported
Type 3 routes Expected overlay membership and BUM replication state exist
VNI state Required VNIs are configured and operational
Remote VTEP route Underlay provides usable reachability to remote VTEP IP
MAC forwarding table Remote MAC addresses resolve to the correct VTEP
EVPN Troubleshooting Warning

An Established BGP EVPN session does not prove that remote MAC routes have been imported correctly. Route-target mismatches, incorrect VNI mappings, missing endpoint advertisements or unresolved VTEP next hops can prevent forwarding even when the BGP session itself is healthy.


4.13 — Engineering Design Considerations

Selecting the correct VXLAN control-plane design requires consideration of scalability, replication overhead, operational visibility and endpoint mobility.

Design Area Engineering Consideration
EVPN route reflectors Provide scalable BGP route distribution with appropriate redundancy and policy
Route targets Control which EVPN routes are imported into the intended overlay contexts
BUM replication Balance source VTEP replication load against underlay multicast complexity
ARP suppression Reduce unnecessary ARP flooding while maintaining accurate endpoint bindings
MAC mobility Ensure endpoint moves converge correctly without persistent stale forwarding state
Underlay reachability Maintain resilient routing between all required VTEPs
Monitoring Correlate EVPN control-plane state with installed forwarding entries and packet delivery
Recommended Troubleshooting Order

Verify underlay VTEP reachability first, followed by BGP EVPN neighbor state, imported Type 2 and Type 3 routes, VNI membership, remote MAC forwarding entries and actual VXLAN packet delivery. This sequence helps separate transport failures from control-plane and forwarding-state problems.

Section 4 — Engineering Takeaways

✓ Traditional VXLAN flood-and-learn uses observed traffic and flooding to discover remote MAC locations.
✓ BUM traffic includes broadcast, unknown unicast and multicast frames that may require replication.
✓ Underlay multicast and ingress replication provide different mechanisms for distributing BUM traffic.
✓ EVPN Type 2 routes advertise endpoint MAC/IP reachability, while Type 3 routes support broadcast domain membership and BUM delivery state.
✓ ARP suppression can reduce broadcast traffic when valid endpoint bindings are available.
✓ EVPN MAC mobility procedures help update forwarding information when workloads move between VTEPs.
✓ Successful VXLAN forwarding depends on both valid EVPN control-plane information and operational underlay and data-plane state.
Continue Learning · Section 05
VXLAN EVPN Routing, Anycast Gateways and Inter-VNI Connectivity
Network Insight · Interactive Engineering Lab 04

Flood-and-Learn vs BGP EVPN: Explore the VXLAN Control Plane

Investigate how remote endpoint reachability is discovered. Compare data-plane flooding with BGP EVPN route advertisements, inspect the resulting forwarding state and simulate endpoint mobility between VTEPs.

Step 1 · Select Control-Plane Model
Step 2 · Generate a Network Event
VXLAN Fabric · VNI 10010 Data-Plane Learning
ROUTED IP UNDERLAY Spine IP Forwarding Leaf A VTEP 10.0.0.11 Leaf B VTEP 10.0.0.12 Leaf C VTEP 10.0.0.13 VXLAN OVERLAY · VNI 10010 Host A 192.168.10.11 Host B 192.168.10.12
BUM Replication Copies (Current Event) 0
EVPN Route Advertisements (Current Event) 0
Remote MAC Entry on Leaf A Unknown
Leaf A · Remote MAC Reachability
Destination Remote VTEP Source
No remote entry learned
EVPN Control-Plane State

Flood-and-Learn Mode

Leaf A initially has no remote MAC mapping. Unknown unicast traffic is replicated to other VTEPs participating in the VNI.

EVPN Route State
Type 2 · MAC/IP Not used
Type 3 · IMET Not used
Control-Plane and Data-Plane Event Timeline
[READY] Select a control-plane model. [READY] Generate unknown unicast, ARP or mobility events.
Engineering Takeaway

Flood-and-learn discovers remote MAC reachability through data-plane traffic. BGP EVPN can advertise MAC/IP reachability in the control plane, reducing the need for unknown-unicast flooding.

Educational event-driven model. Assumes three participating VTEPs, ingress replication and pre-established EVPN sessions in EVPN mode. The widget simplifies MAC learning, route withdrawal, mobility sequencing and ARP suppression; it is not a full protocol emulator.
Section 05 · Distributed Layer 3 Overlay

VXLAN EVPN Routing, Anycast Gateways and Inter-VNI Connectivity

VXLAN is often introduced as a technology that extends Layer 2 networks across an IP fabric. However, modern VXLAN EVPN architectures also support distributed Layer 3 routing between different subnets and virtual networks.

This capability allows workloads connected to different leaf switches and different VLANs to communicate through distributed routing gateways rather than sending all inter-subnet traffic to a centralized router.

The main building blocks include distributed anycast gateways, integrated routing and bridging (IRB), Layer 2 VNIs, Layer 3 VNIs, VRFs and EVPN route advertisements. Together, these components enable a scalable routed overlay while the physical underlay continues to provide ordinary IP connectivity between VTEPs.

Anycast Gateway Inter-VNI Routing Symmetric IRB Asymmetric IRB L2 VNI L3 VNI VRF EVPN Type 5

5.1 — Why Layer 3 Routing Matters in VXLAN Fabrics

A VXLAN bridge domain provides Layer 2 connectivity for endpoints belonging to the same logical network segment.

However, real data centre applications often consist of multiple subnets. Web servers, application servers and databases may be assigned to different VLANs and IP networks.

Communication between those networks requires Layer 3 routing. A modern VXLAN EVPN fabric can perform this routing directly on the leaf switches where workloads connect.

SAME SUBNET
Layer 2 VXLAN Forwarding

Traffic between endpoints in the same bridge domain can be transported across the overlay using the associated L2 VNI.

DIFFERENT SUBNETS
Layer 3 EVPN Routing

Traffic between IP subnets is routed through an IRB gateway and can cross the overlay using an L3 VNI.

Architectural Principle

The underlay provides IP transport between VTEPs. The overlay defines tenant Layer 2 and Layer 3 forwarding contexts. Distributed routing allows the leaf fabric to route tenant traffic without requiring a centralized physical router for every inter-subnet flow.


5.2 — Distributed Anycast Gateways

A distributed anycast gateway allows multiple leaf switches to present the same default gateway IP address and virtual gateway MAC address to endpoints in a given subnet.

For example, hosts in subnet 192.168.10.0/24 might use 192.168.10.1 as their default gateway, regardless of which leaf switch they are connected to.

Each participating leaf can perform local routing for traffic that must leave the source subnet. This reduces the need to send packets across the fabric merely to reach a centralized first-hop gateway.

Gateway Attribute Example Purpose
Subnet 192.168.10.0/24 Endpoint IP network
Gateway IP 192.168.10.1 Shared default gateway address
Virtual gateway MAC 0001.0001.0001 Common gateway MAC used by participating leaf switches
Leaf A VTEP 10.0.0.11 Local first-hop routing for its hosts
Leaf B VTEP 10.0.0.12 Local first-hop routing for its hosts

The gateway IP and virtual MAC must be consistently configured for the same subnet across participating leaves.

Distributed anycast gateways also support workload mobility: a virtual machine moving to another leaf can continue using the same first-hop gateway identity.

Why Anycast Gateways Improve Traffic Flow

Hosts can reach their default gateway locally on the attached leaf. This can reduce unnecessary hairpinning and allows the fabric to route traffic closer to the source endpoint.


5.3 — Layer 2 VNIs and Layer 3 VNIs

VXLAN EVPN fabrics commonly distinguish between Layer 2 VNIs and Layer 3 VNIs.

An L2 VNI identifies a Layer 2 bridge domain, often associated with a VLAN. An L3 VNI identifies a routed overlay context associated with a VRF.

Component Example Function
VLAN 10 192.168.10.0/24 Web subnet
L2 VNI 10010 VLAN 10 Web Layer 2 bridge domain
VLAN 20 192.168.20.0/24 Application subnet
L2 VNI 10020 VLAN 20 Application Layer 2 bridge domain
VRF TENANT-A Tenant routing table Isolated Layer 3 forwarding context
L3 VNI 50000 VRF TENANT-A Routed VXLAN overlay context

In a symmetric IRB design, the L3 VNI is used to transport routed traffic between VTEPs in the relevant tenant VRF.

The L2 VNI continues to provide Layer 2 connectivity for endpoints within the corresponding bridge domain.

VNI Mapping Is Platform-Specific

L2 and L3 VNI numbers are locally configured design values. The examples here illustrate a common VLAN-to-L2-VNI and VRF-to-L3-VNI mapping; the numbers themselves are not reserved or mandatory.


5.4 — VXLAN EVPN Routed Fabric Architecture

The following example contains two leaf VTEPs, two routed spine switches and endpoints in different IP subnets.

Host A belongs to VLAN 10 / VNI 10010, while Host B belongs to VLAN 20 / VNI 10020. Both subnets are associated with VRF TENANT-A and L3 VNI 50000.

Reference Topology · Distributed Routing over VXLAN EVPN
VRF TENANT-A Routed Overlay · L3 VNI 50000 Spine 1 IP Underlay Spine 2 IP Underlay L3 VNI 50000 Leaf A VTEP 10.0.0.11 Anycast GW 192.168.10.1 VLAN 10 · L2 VNI 10010 Leaf B VTEP 10.0.0.12 Anycast GW 192.168.20.1 VLAN 20 · L2 VNI 10020 Host A · Web 192.168.10.11/24 Gateway 192.168.10.1 Host B · Application 192.168.20.12/24 Gateway 192.168.20.1
Physical IP underlay Logical routed VXLAN overlay Host attachment
Simplified symmetric IRB example. Each leaf performs local routing for its attached subnet, while the L3 VNI carries routed traffic between VTEPs. The dashed overlay path is logical and is transported through the routed spine fabric. In a distributed gateway deployment, gateway interfaces for a subnet are configured consistently on all leaves participating in that subnet.

5.5 — Integrated Routing and Bridging (IRB)

Integrated Routing and Bridging combines Layer 2 bridging and Layer 3 routing within a single forwarding architecture.

In a VXLAN EVPN fabric, IRB allows a VTEP to bridge frames within an L2 VNI and route packets between different IP subnets associated with the same tenant VRF.

Two major IRB forwarding models are commonly discussed: asymmetric IRB and symmetric IRB.

ASYMMETRIC IRB
Route at Ingress, Bridge at Egress

The ingress VTEP routes into the destination bridge domain and encapsulates traffic using the destination L2 VNI. The egress VTEP then bridges the frame locally.

SYMMETRIC IRB
Route at Both VTEPs

The ingress VTEP routes traffic into the tenant L3 VNI. The egress VTEP performs the corresponding routing lookup and delivers traffic into the destination bridge domain.


5.6 — Asymmetric IRB Explained

In asymmetric IRB, the ingress VTEP performs the Layer 3 routing decision and selects the destination subnet's bridge domain.

The packet is then encapsulated using the destination L2 VNI and transported to the remote VTEP. The egress VTEP bridges the resulting frame to the destination.

Because the ingress leaf may need to route into multiple destination bridge domains, asymmetric designs can require broad distribution of VLAN and L2 VNI configuration across the participating fabric.

Asymmetric IRB Scaling Consideration

In common asymmetric IRB implementations, ingress leaves need the destination subnet's L2 forwarding context. As the number of subnets increases, this can increase the number of VLANs and L2 VNIs that must be provisioned across the fabric.


5.7 — Symmetric IRB Explained

Symmetric IRB performs Layer 3 forwarding at both the ingress and egress VTEPs.

The ingress VTEP routes the packet from the source bridge domain into the tenant VRF and encapsulates it using the L3 VNI.

The egress VTEP decapsulates the routed overlay packet, performs the appropriate destination forwarding operation and delivers it through the local destination bridge domain.

This design allows each leaf to host only the local Layer 2 segments it requires, while participating in the shared tenant Layer 3 routing context.

Characteristic Asymmetric IRB Symmetric IRB
Ingress VTEP Routes into destination L2 segment Routes into tenant L3 VNI
Egress VTEP Bridges to destination Routes/forwards into destination bridge domain
Inter-VTEP VNI Destination L2 VNI Tenant L3 VNI
Remote L2 VNI presence Often needed at ingress for destination subnets Not generally required for every remote subnet
Scalability Can increase L2 state across the fabric Better separation of local L2 state and tenant routing
Common use Smaller or specific EVPN IRB designs Large distributed routed EVPN fabrics
Design Preference

Symmetric IRB is widely used in modern VXLAN EVPN data centre fabrics because it avoids the need for every ingress leaf to instantiate every destination Layer 2 segment. The exact forwarding behavior depends on the platform and EVPN implementation.


5.8 — Symmetric IRB Packet Journey

Consider Host A (192.168.10.11) communicating with Host B (192.168.20.12). Both networks belong to VRF TENANT-A.

The example assumes that the necessary endpoint routes, VNI mappings and underlay reachability are already installed.

01
Host A Sends to Its Default Gateway
Host A recognizes that 192.168.20.12 belongs to another subnet. It sends the packet to gateway 192.168.10.1 on Leaf A.
02
Leaf A Performs a VRF Routing Lookup
Leaf A routes the packet in VRF TENANT-A. The installed EVPN forwarding information identifies Leaf B as the remote VTEP for the destination.
03
Routed VXLAN Encapsulation
Leaf A encapsulates the routed packet using L3 VNI 50000 with outer source VTEP 10.0.0.11 and destination VTEP 10.0.0.12.
04
Underlay IP Forwarding
The spine fabric forwards the encapsulated packet toward Leaf B using the outer destination IP. ECMP may select an available path.
05
Leaf B Decapsulates and Routes
Leaf B identifies the tenant L3 VNI, processes the packet in the appropriate VRF and resolves the destination through its local VLAN 20 bridge domain.
06
Host B Receives the Packet
Leaf B delivers the packet to Host B on 192.168.20.12. The original source and destination IPs remain unchanged in this example.
Inner Header Handling

Symmetric IRB does not simply preserve the original host Ethernet frame unchanged. The ingress routing operation creates the appropriate routed overlay forwarding context. The exact inner Ethernet header, router MAC information and egress processing depend on the EVPN forwarding model and platform.


5.9 — VRFs and Tenant Isolation

A Virtual Routing and Forwarding instance, or VRF, maintains a separate Layer 3 routing table.

In a VXLAN EVPN fabric, VRFs allow multiple tenants or application environments to use isolated routing contexts over the same physical underlay.

A VRF may contain multiple local subnets and L2 VNIs. A corresponding L3 VNI can provide routed connectivity between VTEPs participating in that VRF.

TENANT-A
VRF TENANT-A

Web and application subnets use their own routing table, L2 VNIs and routed L3 VNI.

TENANT-B
VRF TENANT-B

A separate tenant can use independent routing and overlay identifiers while sharing the same physical IP fabric.

VRF Isolation Is Not a Complete Security Policy

VRFs provide separate routing contexts, but intentional route leaking, shared services or external connectivity can introduce paths between them. Firewall policy, segmentation controls and route import rules must still be designed appropriately.


5.10 — EVPN Route Type 5: IP Prefix Advertisement

EVPN Route Type 5 is the IP Prefix Advertisement Route. It provides a way to advertise IP prefix reachability through the EVPN control plane.

While Type 2 routes primarily describe MAC reachability and optional host IP bindings, Type 5 routes advertise IP prefixes independently of individual endpoint MAC advertisements.

This is useful for advertising routed networks, external prefixes and aggregate reachability in EVPN-based Layer 3 overlays.

EVPN Route Primary Information Typical Role
Type 2 MAC address and optional IP address Endpoint reachability, host routes and MAC/IP bindings
Type 3 Inclusive multicast Ethernet tag information Overlay membership and BUM delivery
Type 5 IP prefix Routed prefix advertisement in EVPN
Conceptual EVPN Route Type 5 Advertisement
EVPN Route Type: 5
Route: IP Prefix Advertisement

Tenant VRF:
  TENANT-A

Advertised Prefix:
  172.16.50.0/24

Advertising VTEP:
  10.0.0.12

Tenant Routed Overlay:
  L3 VNI 50000

Route Import:
  Controlled by EVPN route targets

Result:
  Remote VTEPs can install
  tenant IP prefix reachability
Conceptual example rather than raw BGP NLRI. Actual Type 5 advertisements include route distinguisher, Ethernet tag, IP prefix and gateway-related fields, with associated BGP attributes and extended communities. Forwarding resolution is implementation- and deployment-dependent.
Type 2 Versus Type 5

Type 2 can advertise host reachability with MAC/IP bindings, while Type 5 supports IP prefix advertisements. Both can participate in EVPN Layer 3 designs; Type 5 is not mandatory for every symmetric IRB deployment.


5.11 — Route Targets and Route Import

EVPN uses BGP route distinguishers and route targets for different purposes.

A Route Distinguisher (RD) helps make otherwise overlapping VPN routes unique in the BGP control plane.

A Route Target (RT) is an extended community used to control which VPN routes are imported into the appropriate local contexts.

Attribute Purpose Example
RD Makes EVPN route identifiers unique 65001:50000
RT Import Selects EVPN routes imported into a local context 65001:50000
RT Export Marks advertised EVPN routes for intended import 65001:50000
L3 VNI Identifies tenant routed VXLAN traffic 50000

Incorrect route targets can cause a VTEP to receive EVPN routes without importing them into the expected VRF or bridge domain.

This is an important troubleshooting distinction: a route visible in the BGP EVPN table is not necessarily installed in the intended tenant forwarding table.


5.12 — VXLAN EVPN Routing Verification

When troubleshooting inter-VNI routing, engineers should verify the tenant routing context, anycast gateway configuration, EVPN reachability and installed forwarding state.

Cisco NX-OS-Style Verification Commands
show vrf

show ip route vrf TENANT-A

show ip interface brief vrf TENANT-A

show nve interface

show nve vni

show nve peers

show bgp l2vpn evpn summary

show bgp l2vpn evpn route-type 2

show bgp l2vpn evpn route-type 5

show l2route evpn mac all

show mac address-table
Illustrative verification commands. Exact syntax, VRF filtering, route-type support and output depend on the NX-OS platform and software release.
Verification Expected State Possible Failure
VRF routing table Destination route installed Missing or incorrectly imported EVPN route
Anycast gateway Correct gateway IP and MAC Incorrect SVI or virtual gateway configuration
L2 VNI Local bridge domain mapped correctly VLAN-to-VNI mismatch
L3 VNI Tenant routed overlay operational Missing VRF-to-VNI mapping
EVPN routes Correct remote host or prefix reachability RT mismatch or missing advertisement
VTEP reachability Outer destination reachable through underlay Underlay routing failure
Packet forwarding Encapsulation, decapsulation and local delivery succeed Incorrect forwarding entry or policy drop

5.13 — Common Inter-VNI Routing Failures

A VXLAN fabric may provide successful same-subnet communication while failing to route between different subnets.

This often indicates that the L2 overlay is operational but the distributed Layer 3 forwarding path is incomplete or misconfigured.

GATEWAY ISSUE
Incorrect Anycast Gateway

Inconsistent gateway IP or MAC configuration can prevent reliable first-hop routing.

VRF ISSUE
Wrong Routing Context

The source or destination subnet may be associated with the wrong tenant VRF.

EVPN ISSUE
Missing Route Import

EVPN routes may be received but rejected or not imported because of route-target policy.

DATA-PLANE ISSUE
Incorrect L3 VNI

A routed overlay mismatch can prevent the receiving VTEP from processing the packet in the correct VRF.

Recommended Troubleshooting Sequence

Confirm local gateway reachability, verify the source VRF routing table, inspect the imported EVPN route, validate the remote VTEP next hop, confirm L3 VNI encapsulation and finally inspect egress routing and local host delivery.


5.14 — Design Considerations for Distributed Routing

Distributed EVPN routing introduces additional flexibility, but consistent design remains essential.

Design Area Recommendation
Gateway consistency Use consistent anycast gateway IP and virtual MAC settings for each subnet across participating leaves
VNI planning Maintain a documented mapping between VLANs, L2 VNIs, VRFs and L3 VNIs
IRB model Select symmetric or asymmetric IRB based on platform support, scale and forwarding requirements
EVPN policy Validate route targets and import/export policy for every tenant context
Underlay routing Provide resilient VTEP reachability with suitable ECMP and MTU design
Segmentation Combine VRF isolation with appropriate security and route leaking controls
Operational visibility Correlate BGP EVPN routes with installed tenant routes and actual packet forwarding

Section 5 — Engineering Takeaways

✓ VXLAN EVPN supports distributed Layer 3 routing between different IP subnets, not only Layer 2 extension.
✓ Distributed anycast gateways allow endpoints to use a consistent default gateway identity across participating leaf switches.
✓ L2 VNIs identify bridge domains, while L3 VNIs support routed tenant overlays.
✓ Symmetric IRB routes traffic at both ingress and egress VTEPs and is widely used in scalable EVPN fabrics.
✓ VRFs provide separate tenant routing tables, while route targets control EVPN route import.
✓ EVPN Type 5 routes support IP prefix advertisements, complementing Type 2 endpoint reachability.
✓ Successful inter-VNI forwarding requires correct gateway, VRF, VNI, EVPN and underlay state.
Continue Learning · Section 06
VXLAN Deployment, Verification and Troubleshooting
Network Insight · Interactive Engineering Lab 05

VXLAN EVPN Routing: Anycast Gateways and Inter-VNI Connectivity

Follow a packet between tenant workloads. Compare Layer 2 bridging with Layer 3 routing, inspect symmetric and asymmetric IRB, and investigate what happens when the tenant VRF or EVPN reachability is misconfigured.

Step 1 · Choose Traffic Scenario
Step 2 · Select IRB Model
Distributed Anycast Gateway Fabric Symmetric IRB · Ready
TENANT-A · EVPN VXLAN FABRIC Spine Routed IP Underlay L3VNI 50000 · TENANT-A Leaf A VTEP 10.0.0.11 Leaf B VTEP 10.0.0.12 Anycast Gateway Anycast Gateway Host A 192.168.10.11 · VNI 10010 Host B 192.168.20.12 · VNI 10020
Ready to trace
Transport VNI 50000
Forwarding Operation Route → Bridge
Forwarding Result Ready
Tenant Forwarding Table
Destination Next Hop / VTEP Context
Control-Plane and Gateway State
Component Status
Tenant VRF TENANT-A · Active
EVPN Reachability Installed
Anycast Gateway Active on both leaves
Routing Model Symmetric IRB
Packet Forwarding and EVPN Investigation Timeline
[READY] TENANT-A VRF initialized. [READY] L2VNIs 10010 and 10020 configured. [READY] L3VNI 50000 available. [READY] Select a scenario and trace the packet.
Engineering Explanation

With symmetric IRB, the ingress leaf routes traffic into the tenant L3VNI. The remote leaf receives the routed VXLAN packet, performs destination-side forwarding and delivers it through the appropriate L2VNI or local VLAN. Distributed anycast gateways provide consistent default-gateway addresses across the fabric.

Simplified conceptual EVPN forwarding model. Assumes pre-established underlay and EVPN reachability, local gateway configuration and a single tenant VRF. Actual symmetric IRB forwarding also uses router MAC information and platform-specific EVPN route programming. This lab does not emulate a real BGP or ASIC pipeline.
Section 06 · Deployment and Operations

VXLAN Deployment, Verification and Troubleshooting

Designing a VXLAN EVPN fabric requires more than configuring a VXLAN tunnel. A successful deployment depends on the interaction between the physical IP underlay, VTEP interfaces, overlay segment mappings, BGP EVPN control-plane advertisements and the forwarding tables installed on each leaf switch.

When a VXLAN network fails, the visible symptom may appear to be a simple connectivity problem. However, the root cause could exist in the underlay routing table, EVPN route distribution, VLAN-to-VNI mapping, tenant VRF or encapsulated packet path.

A structured operational approach separates these layers and verifies each dependency in the correct order.

Underlay Routing VTEP Reachability NVE Interface BGP EVPN VNI Mapping Route Targets Packet Capture Incident Diagnosis

6.1 — The VXLAN Deployment Model

A typical VXLAN EVPN deployment is built in logical stages. The underlay must first provide reliable IP reachability between the VTEP addresses.

Once the underlay is operational, the overlay can establish VNI mappings and BGP EVPN route distribution.

Tenant forwarding is then enabled through the appropriate Layer 2 and Layer 3 interfaces.

01
Build the Routed Underlay
Configure physical interfaces, routed leaf-spine links, loopbacks and an appropriate underlay routing protocol.
02
Establish VTEP Reachability
Confirm that each leaf can reach the remote VTEP source IP addresses through the underlay.
03
Configure VLAN and VNI Mappings
Associate local bridge domains with their intended L2 VNIs and configure tenant VRFs and L3 VNIs where needed.
04
Configure the NVE Interface
Define the VXLAN tunnel endpoint source and associate the required VNIs with the network virtualization interface.
05
Establish BGP EVPN
Configure BGP EVPN neighbors, address families, route distinguishers and route-target policies.
06
Validate Endpoint Learning
Confirm local and remote MAC learning, EVPN route import, VTEP peer discovery and forwarding state.
07
Test Tenant Connectivity
Verify same-subnet VXLAN bridging, inter-subnet routing, failure recovery and expected traffic isolation.
Recommended Deployment Order

Validate the underlay before introducing overlay complexity. A correctly configured BGP EVPN control plane cannot compensate for missing IP reachability between VTEPs.


6.2 — VXLAN EVPN Reference Topology

The following reference design uses two spine switches and two leaf VTEPs.

The leaf switches advertise their loopback VTEP addresses through the underlay. BGP EVPN distributes overlay reachability information.

Reference Deployment · Two-Leaf VXLAN EVPN Fabric
Spine 1 IP Routing Spine 2 IP Routing Routed IP Underlay OSPF / IS-IS / BGP VXLAN VNI 10010 Leaf A VTEP 10.0.0.11 NVE1 · VLAN 10 L2 VNI 10010 Leaf B VTEP 10.0.0.12 NVE1 · VLAN 10 L2 VNI 10010 Host A 192.168.10.11 Host B 192.168.10.12
IP underlay Logical VXLAN overlay Endpoint access
Simplified deployment topology. VXLAN traffic uses UDP port 4789 and travels through the routed spine fabric between VTEPs. BGP EVPN control-plane sessions are logical routing adjacencies and may use dedicated route reflectors or another supported peering design.

6.3 — Underlay Routing Requirements

The VXLAN underlay is the routed IP network that transports encapsulated packets between VTEPs.

The underlay can use routing protocols such as OSPF, IS-IS or BGP, depending on the design and platform.

Its primary purpose is to provide resilient IP reachability to the loopback addresses used as VTEP sources.

Underlay Requirement Engineering Check
VTEP IP reachability Remote VTEP loopbacks are present in the routing table
Routing convergence Failed links trigger an appropriate alternate path
ECMP Multiple equal-cost paths are installed where intended
MTU Underlay supports the additional VXLAN encapsulation overhead
UDP transport VXLAN UDP traffic is permitted through relevant network policies
Source address NVE interface uses the intended VTEP loopback address
Underlay Verification · NX-OS-Style Commands
show ip interface brief

show ip route 10.0.0.12

show ip ospf neighbors

show ip route

ping 10.0.0.12 source 10.0.0.11

traceroute 10.0.0.12 source 10.0.0.11
Example assumes OSPF as the underlay protocol. For an IS-IS or BGP underlay, use the corresponding routing verification commands. Source-aware ping and traceroute syntax varies by release.
MTU and VXLAN Encapsulation

VXLAN adds outer Ethernet, IP, UDP and VXLAN headers around the original Ethernet frame. For a common IPv4 transport, this adds approximately 50 bytes of overhead, excluding additional tags or encapsulations. The underlay MTU must accommodate the resulting packet size.


6.4 — Configuring the VTEP and NVE Interface

A VXLAN Tunnel Endpoint, or VTEP, performs VXLAN encapsulation and decapsulation.

On Cisco NX-OS platforms, VXLAN forwarding is commonly configured through a Network Virtualization Edge interface such as NVE1.

The NVE interface references a source loopback and the VNIs that participate in the overlay.

Illustrative Cisco NX-OS VXLAN EVPN Configuration
feature nv overlay
feature vn-segment-vlan-based
feature bgp

nv overlay evpn

vlan 10
  vn-segment 10010

interface loopback1
  ip address 10.0.0.11/32

interface nve1
  no shutdown
  host-reachability protocol bgp
  source-interface loopback1
  member vni 10010
    ingress-replication protocol bgp

evpn
  vni 10010 l2
    rd auto
    route-target import auto
    route-target export auto
Illustrative configuration fragment for a compatible NX-OS VXLAN EVPN deployment. It is not a complete production configuration. The physical underlay, loopback advertisement, BGP EVPN neighbors, access interfaces and other platform-specific requirements must also be configured. Exact syntax and automatic route-target behavior depend on platform and release.

In this example, VLAN 10 is associated with VNI 10010. Loopback1 provides the local VTEP source address.

The NVE interface enables VXLAN encapsulation and uses BGP EVPN for remote endpoint learning and ingress replication state.


6.5 — Verifying BGP EVPN Control-Plane State

The next step is to verify that BGP EVPN sessions are established and that the expected EVPN routes are being exchanged.

Engineers should inspect both the BGP neighbor state and the actual route information installed for the overlay.

BGP EVPN Verification
show bgp l2vpn evpn summary

show bgp l2vpn evpn

show bgp l2vpn evpn route-type 2

show bgp l2vpn evpn route-type 3

show bgp l2vpn evpn route-type 5

show nve peers

show nve vni
Commands are representative of Cisco NX-OS. Support for specific route-type filters and output formats varies.
Verification Expected Result Failure Indicator
BGP EVPN neighbor Established Idle, Active or repeated resets
Type 2 route Remote MAC/IP reachability present Missing endpoint advertisement
Type 3 route Correct overlay membership state Missing BUM replication peer
Type 5 route Expected routed prefixes present when used Missing prefix or import mismatch
NVE peers Expected remote VTEPs operational Missing or down remote peer
VNI status Required VNIs operational Inactive VNI or incorrect mapping
BGP Established Does Not Mean VXLAN Works

BGP EVPN can be Established while endpoint routes are missing, route targets are incorrect, VNIs are inactive or remote VTEPs are unreachable. Verify the installed forwarding state rather than relying on neighbor status alone.


6.6 — MAC Learning and Forwarding Validation

Once the control plane is operational, inspect the forwarding tables to confirm that endpoint reachability has been programmed correctly.

Local MAC addresses should resolve toward access interfaces. Remote MAC addresses should resolve toward the appropriate remote VTEP.

Endpoint Forwarding Verification
show mac address-table

show l2route evpn mac all

show nve peers

show nve vni

show ip arp

show ip route vrf TENANT-A
Inspect local versus remote MAC entries, VNI association, ARP state and tenant routes. VRF-specific commands apply when Layer 3 EVPN routing is configured.
LOCAL ENDPOINT
Access-Side Learning

Confirm that the endpoint MAC is associated with the correct VLAN and local interface.

REMOTE ENDPOINT
EVPN Remote MAC Learning

Confirm that the remote MAC is associated with the expected VNI and remote VTEP destination.


6.7 — Packet Capture and VXLAN Encapsulation

Packet captures are particularly useful when the control plane appears healthy but data traffic still fails.

A VXLAN packet typically contains an outer Ethernet header, outer IP header, UDP header, VXLAN header and original inner Ethernet frame.

The standard VXLAN UDP destination port is 4789. The VXLAN header carries the VNI identifying the relevant overlay segment.

Packet Field Example What to Verify
Outer source IP 10.0.0.11 Correct ingress VTEP
Outer destination IP 10.0.0.12 Correct remote VTEP
UDP destination 4789 VXLAN transport
VNI 10010 Correct overlay segment
Inner source IP 192.168.10.11 Original endpoint source
Inner destination IP 192.168.10.12 Original endpoint destination
Example Linux Packet Capture
sudo tcpdump -ni eth0 udp dst port 4789

sudo tcpdump -ni eth0 \
  'host 10.0.0.12 and udp port 4789' \
  -vv
Run on an interface where encapsulated VXLAN traffic is visible. Hardware-offloaded traffic may not appear in ordinary host or switch CPU captures. SPAN, ERSPAN or platform-specific capture facilities may be required.
Capture Location Matters

Capturing traffic at the server interface normally shows the original tenant packet. Capturing on an underlay link can reveal the VXLAN-encapsulated packet. These views answer different questions during troubleshooting.


6.8 — Common VXLAN Failure Scenarios

VXLAN incidents often fall into recognizable categories. The table below maps typical symptoms to likely areas of investigation.

Symptom Possible Cause Investigation
Remote VTEP unreachable Underlay routing or physical link failure Check routes, interfaces and source-aware ping
BGP EVPN neighbor down BGP configuration, transport or policy issue Inspect neighbor state, peer addresses and EVPN activation
Missing remote MAC No Type 2 route or route import failure Check EVPN routes, route targets and MAC learning
Broadcast traffic fails Missing Type 3 membership or replication state Inspect IMET routes, VNI state and BUM replication
Same-subnet works, inter-subnet fails Gateway, VRF, L3 VNI or routing issue Check SVI, tenant routes and symmetric IRB
Small packets work, large packets fail MTU mismatch or fragmentation issue Test path MTU and inspect encapsulated sizes
Endpoint intermittently unreachable MAC mobility, stale state or inconsistent forwarding Check mobility events and forwarding updates
Only one VNI fails Incorrect VLAN/VNI mapping or route target Compare affected and healthy VNI configuration

6.9 — Seven-Stage VXLAN Troubleshooting Workflow

The most effective troubleshooting method is to locate the first point at which expected network state differs from actual state.

Begin with the physical transport and progress toward tenant forwarding.

01
Validate Physical and Routed Underlay
Confirm interface status, routing adjacencies, VTEP loopback reachability and expected ECMP paths.
02
Validate NVE and VNI State
Check the NVE source interface, operational status, VNI membership and VLAN mappings.
03
Validate BGP EVPN Neighbors
Confirm EVPN address-family sessions and route exchange with the expected peers or route reflectors.
04
Inspect EVPN Route Import
Verify relevant Type 2, Type 3 and, where used, Type 5 routes. Check route targets and import policy.
05
Inspect Forwarding Tables
Compare EVPN routes with remote MAC, ARP, VNI and tenant VRF forwarding state.
06
Trace the Encapsulated Packet
Verify outer VTEP addresses, UDP port, VNI and successful transport through the underlay.
07
Verify Egress Delivery and Policy
Confirm decapsulation, local VLAN forwarding, gateway processing, ACLs and destination endpoint reachability.
Engineering Troubleshooting Principle

Avoid changing several configuration elements simultaneously. Identify the failing layer, form a testable hypothesis and compare expected versus observed state before applying a fix.


6.10 — Example Incident: Remote Host Unreachable

Consider a VXLAN fabric where Host A on Leaf A cannot reach Host B on Leaf B.

Both hosts belong to VNI 10010. BGP EVPN sessions are Established, but traffic still fails.

Investigation Step Observed Result Interpretation
Underlay route to 10.0.0.12 Present Remote VTEP route exists
BGP EVPN neighbor Established Control-plane transport is operational
EVPN Type 2 route Remote Host B MAC advertised Endpoint information is being distributed
Remote MAC forwarding Entry absent from VNI 10010 Route not programmed into expected forwarding context
Route-target inspection Import RT mismatch Likely reason route was not imported

In this example, the underlay and BGP neighbor state are healthy. The failure occurs when the received EVPN route is not imported into the intended local forwarding context.

Correcting the route-target policy and validating the resulting remote MAC entry would be the appropriate next investigation step.

Avoid the Wrong Fix

Restarting BGP or modifying underlay routing would not directly address the route import mismatch identified in this example. The evidence points toward overlay policy and forwarding installation.


6.11 — VXLAN Performance and Scale

Beyond basic connectivity, production VXLAN fabrics must be designed for scale, predictable convergence and operational visibility.

Design Area Operational Consideration
MAC scale Monitor local and remote MAC table capacity
EVPN route scale Monitor BGP EVPN routes, memory and route-reflector capacity
BUM traffic Control unnecessary replication and broadcast amplification
ECMP Validate load distribution across available spine paths
MTU Ensure encapsulated packets fit within the supported underlay MTU
MAC mobility Track workload movement and unexpected MAC flapping
Convergence Test link, leaf and control-plane failure recovery
Security Apply segmentation, route policy and appropriate access controls

6.12 — Pre-Production Validation Checklist

Before a VXLAN EVPN fabric enters production, validate the following operational requirements.

✓ All expected underlay routing adjacencies are operational.
✓ Every required VTEP loopback is reachable.
✓ Underlay MTU supports VXLAN encapsulation.
✓ VLAN-to-VNI mappings are consistent.
✓ NVE interfaces and required VNIs are operational.
✓ BGP EVPN neighbors are Established.
✓ Expected EVPN routes are received and imported.
✓ Remote MAC forwarding entries resolve correctly.
✓ Same-subnet connectivity works across VTEPs.
✓ Inter-subnet routing works where configured.
✓ Tenant isolation and security policy are verified.
✓ Link and node failure recovery has been tested.
Operational Baseline

Capture a known-good baseline of underlay routes, BGP EVPN neighbors, VNI state, MAC forwarding entries and tenant routing tables. Comparing healthy and failed states can substantially reduce incident investigation time.

Section 6 — Engineering Takeaways

✓ VXLAN EVPN deployment begins with a reliable routed IP underlay.
✓ VTEP loopback reachability, NVE state and VNI mappings must be correct before overlay forwarding can succeed.
✓ BGP EVPN sessions distribute endpoint and overlay reachability, but Established neighbors alone do not prove forwarding works.
✓ Remote MAC and tenant routing tables must be checked alongside EVPN control-plane advertisements.
✓ VXLAN packet captures reveal outer VTEP addresses, UDP transport and the overlay VNI.
✓ Route-target mismatches, incorrect VNI mappings, MTU problems and stale MAC state are common investigation areas.
✓ A structured troubleshooting workflow isolates underlay, control-plane and data-plane failures before configuration changes are made.
Next Learning Activity
Interactive VXLAN Engineering Labs and Troubleshooting Exercises
Network Insight · Interactive Engineering Lab 06

VXLAN EVPN CLI Verification and Troubleshooting Workbench

Investigate a broken VXLAN fabric using operational commands. Inspect the underlay, BGP EVPN sessions, NVE state, VNI mappings and remote MAC entries. Identify the fault, repair it and verify end-to-end connectivity.

Investigation Objective
Restore Host A → Host B
Underlay Down BGP EVPN Established NVE Up
VXLAN Fabric · VNI 10010 · AS 65001
Spine IP Underlay Leaf A 10.0.0.11 Leaf B 10.0.0.12 Host A 192.168.10.11 Host B 192.168.10.12 EVPN VXLAN OVERLAY
Underlay Reachability Failed
EVPN Route State Installed
End-to-End Traffic Not Tested
LEAF-A · Simulated NX-OS CLI Operational Verification Console
Network Insight VXLAN EVPN Investigation Lab Leaf-A# Type help to list available commands.
Leaf-A#
Quick Verification Commands
Verification Checklist
Check Operational State
Underlay VTEP route Missing
BGP EVPN neighbor Established
NVE interface Up
VNI consistency Matched
Remote MAC route Installed
Incident Investigation Notes

Underlay Reachability Failure

The routed underlay has lost reachability to the remote VTEP. Inspect the route table and test connectivity to 10.0.0.12.

Start with show ip route 10.0.0.12.

Engineering Investigation Principle

Verify VXLAN systematically: underlay IP reachability first, then BGP EVPN control-plane state, NVE peer status, VNI configuration, remote MAC/IP reachability and finally end-to-end traffic. A successful BGP session alone does not guarantee working VXLAN forwarding.

Section 07 · Production Fabric Architecture

VXLAN Design Best Practices, Security and Production Readiness

VXLAN EVPN provides a scalable framework for extending Layer 2 networks and building distributed Layer 3 overlays. However, deploying the technology successfully requires careful planning around segmentation, resilience, control-plane security, operational scale and automation.

The goal of production design is not simply to establish VXLAN connectivity. It is to build a fabric that remains predictable during failures, supports controlled tenant communication and provides sufficient visibility to diagnose problems quickly.

Fabric Architecture Segmentation EVPN Security High Availability Scale Automation Observability Production Readiness

7.1 — Principles of a Production VXLAN Fabric

A production VXLAN EVPN fabric should separate physical transport from tenant forwarding and use consistent design standards across all participating leaf switches.

This separation allows the underlay to focus on resilient IP transport while the overlay manages tenant segments, endpoint reachability and distributed routing.

PRINCIPLE 01
Keep the Underlay Simple

Use a predictable routed leaf-spine design with clear loopback addressing, ECMP and resilient VTEP reachability.

PRINCIPLE 02
Standardize Overlay Mappings

Document VLAN, L2 VNI, VRF, L3 VNI and route-target assignments.

PRINCIPLE 03
Limit Failure Domains

Avoid unnecessary Layer 2 extension and design explicit boundaries for tenant and application networks.

PRINCIPLE 04
Engineer for Visibility

Monitor underlay routing, EVPN advertisements, VNI state, endpoint mobility and forwarding behavior.

Design Objective

A well-designed fabric should make failures easier to isolate, not merely provide more network scale. Clear separation between underlay, EVPN control plane and tenant forwarding is central to operational simplicity.


7.2 — Reference Production Fabric Architecture

The following conceptual topology illustrates two tenants sharing a routed leaf-spine underlay while using separate overlay routing contexts.

Reference Architecture · Shared Underlay, Isolated Overlays
Shared Routed Underlay Spine 1 IP / ECMP Spine 2 IP / ECMP Leaf A VTEP 10.0.0.11 EVPN / NVE1 Leaf B VTEP 10.0.0.12 EVPN / NVE1 VXLAN EVPN Tenant A VRF TENANT-A L2 VNI 10010 L3 VNI 50000 Tenant A VRF TENANT-A L2 VNI 10010 L3 VNI 50000 Tenant B VRF TENANT-B L2 VNI 20010 L3 VNI 60000 Tenant B TENANT-B VNI 20010 L3 60000
Conceptual multi-tenant design. The same physical leaf-spine underlay carries multiple VXLAN overlays. Tenant isolation is provided through separate forwarding contexts and controlled EVPN route import. The drawing is logical rather than a complete physical cabling diagram.

7.3 — Segmentation and Tenant Isolation

VXLAN enables multiple logical networks to share the same physical infrastructure. VNIs identify Layer 2 segments, while VRFs provide separate Layer 3 routing contexts.

This allows applications or tenants to maintain independent forwarding domains without requiring dedicated physical network hardware for every environment.

Segmentation Layer Mechanism Purpose
Layer 2 L2 VNI Separate Ethernet bridge domains
Layer 3 VRF and L3 VNI Separate tenant routing contexts
EVPN control plane Route targets Control which routes are imported
Security policy ACLs / Firewalls Restrict permitted application flows
Inter-tenant services Controlled route leaking and service insertion Permit explicitly approved communication
VXLAN Does Not Provide Encryption by Default

VXLAN encapsulates Ethernet traffic but does not inherently encrypt the payload or authenticate VTEPs. Where confidentiality or transport integrity is required, additional protections such as IPsec, MACsec or an appropriately secured transport environment must be considered.


7.4 — EVPN Control-Plane Security

BGP EVPN distributes information that influences endpoint reachability and tenant forwarding.

Protecting the EVPN control plane is therefore an important part of fabric security.

BGP PEERING
Restrict Neighbor Access

Limit EVPN BGP peering to approved network devices and routing endpoints.

ROUTE POLICY
Control Route Import

Validate route targets, tenant boundaries and route-leaking requirements.

MANAGEMENT
Protect Management Access

Use authenticated, authorized and monitored access for configuration and operational tools.

MONITORING
Detect Unexpected Changes

Track unusual EVPN advertisements, MAC mobility and unexpected routing changes.

Security Boundary

Route targets are route distribution controls, not a substitute for firewall enforcement. A complete security design must also account for endpoint access, policy enforcement and authorized connectivity between routing domains.


7.5 — High Availability and Failure Domains

Leaf-spine architectures are commonly designed with redundant physical paths.

Equal-Cost Multipath routing can distribute underlay traffic across available next hops, while the routing protocol reacts to link or node failures.

VXLAN EVPN overlays depend on this underlay resilience to maintain connectivity between VTEPs.

Failure Scenario Expected Design Behavior
Single spine link failure Underlay converges onto remaining valid paths
Spine switch failure Reachability continues through surviving spine paths where capacity permits
Leaf failure Dual-attached endpoints may retain service through an alternate leaf, depending on multihoming design
EVPN peer failure Redundant control-plane peers support route distribution continuity
Endpoint migration EVPN MAC mobility procedures update endpoint reachability
Resilience Must Be Tested

Redundant topology does not guarantee zero packet loss or instantaneous convergence. Validate actual failure recovery, forwarding behavior, convergence time and available bandwidth during degraded operation.


7.6 — EVPN Scale and Capacity Planning

VXLAN supports a 24-bit VNI field, but the theoretical identifier space should not be confused with the practical scale supported by a specific network platform.

Real fabric scale is influenced by hardware forwarding capacity, EVPN route count, MAC and ARP table resources, tenant VRFs and operational control-plane limits.

Scale Dimension What to Measure
VNIs Supported L2 and L3 VNI capacity
MAC entries Local and remote MAC forwarding state
EVPN routes Route count, memory and convergence load
ARP / ND entries Endpoint neighbor resolution state
VRFs Supported tenant routing instances
ECMP paths Hardware path capacity and hash distribution
BUM replication Replication fan-out and bandwidth impact

7.7 — Automation and Configuration Consistency

Manual VXLAN EVPN deployments can become difficult to maintain when many leaf switches, tenants and VNIs are involved.

Automation helps maintain consistent configuration standards and reduces the risk of mapping or route-target errors.

01
Define the Intended Network State
Maintain a source of truth for devices, loopbacks, VLANs, VNIs, VRFs and route targets.
02
Generate Consistent Configurations
Use templates or automation tooling to apply approved fabric design standards.
03
Validate Before Deployment
Check duplicate identifiers, missing mappings, route-target inconsistencies and unsupported platform features.
04
Compare Intended and Actual State
Verify operational NVE, BGP EVPN, routing and forwarding state after each deployment.
Automation Principle

Configuration deployment is only one part of automation. Production workflows should also validate the resulting network state and detect drift from the intended fabric design.


7.8 — VXLAN Observability and Analytics

Modern data centre operations require visibility across the underlay, EVPN control plane and tenant data plane.

Useful telemetry includes interface utilization, ECMP distribution, route changes, endpoint mobility, packet drops and tunnel health.

Telemetry Area Operational Value
Underlay interfaces Detect congestion, errors and link failures
Routing adjacencies Identify unstable underlay or EVPN sessions
EVPN route changes Correlate route withdrawals with service disruption
MAC mobility Detect workload movement or unexpected MAC flapping
VNI statistics Monitor overlay forwarding and traffic distribution
Queue depth / drops Identify congestion and microburst effects
Packet traces Validate actual encapsulation and delivery paths
Analytics Do Not Automatically Optimize VXLAN

Telemetry provides information about network behavior. Automated optimization requires additional control systems, supported platform capabilities and explicit operational policy. VXLAN encapsulation itself does not automatically rebalance or optimize traffic.


7.9 — Recommended Engineering Validation Commands

The following Cisco NX-OS-style commands provide a useful starting point for production-state validation.

VXLAN EVPN Production Verification
show ip route

show ip interface brief

show nve interface

show nve peers

show nve vni

show bgp l2vpn evpn summary

show bgp l2vpn evpn route-type 2

show bgp l2vpn evpn route-type 3

show bgp l2vpn evpn route-type 5

show l2route evpn mac all

show mac address-table

show vrf

show ip route vrf TENANT-A

show interface counters errors
Illustrative verification commands. Exact syntax, feature support and output depend on the hardware platform and software release.

7.10 — Production Architecture Decision Matrix

Not every VXLAN EVPN deployment needs the same design choices. The correct architecture depends on workload mobility, tenant isolation, operational scale and resilience requirements.

Decision Recommended Evaluation
Underlay protocol Choose based on operational familiarity, scale and convergence requirements
EVPN peering model Evaluate route reflectors, redundancy and control-plane scale
IRB model Compare symmetric and asymmetric forwarding requirements
Tenant segmentation Define VRF boundaries, route import and firewall enforcement
BUM handling Select supported ingress replication or multicast transport models
High availability Evaluate dual attachment, multihoming and failover behavior
Automation Establish a source of truth and validation workflow
Observability Define operational metrics, baselines and alert thresholds

7.11 — Production Readiness Checklist

✓ Underlay routing is resilient and VTEP reachability verified.
✓ VLAN, VNI and VRF mappings are documented and consistent.
✓ EVPN peering and route-target policies are validated.
✓ Tenant isolation and permitted route leaking are tested.
✓ BGP and management access are appropriately protected.
✓ Hardware MAC, route, VNI and VRF scale is understood.
✓ Underlay MTU accommodates VXLAN overhead.
✓ Failure recovery and degraded-capacity operation are tested.
✓ Monitoring covers underlay, EVPN and data-plane state.
✓ Automation includes pre-deployment and post-deployment validation.

Section 7 — Engineering Takeaways

✓ Production VXLAN fabrics require consistent design across underlay, overlay and tenant routing.
✓ VNIs, VRFs and route targets support logical segmentation, but security policy must be enforced separately.
✓ VXLAN does not inherently provide encryption or endpoint authentication.
✓ Fabric scale is constrained by actual platform resources, not simply the size of the VNI field.
✓ Redundancy, ECMP and EVPN convergence should be validated under realistic failure conditions.
✓ Automation and telemetry improve consistency and operational visibility when combined with explicit validation and control policies.
Continue Learning
VXLAN Interactive Architecture and Engineering Labs
Network Insight · Interactive Engineering Lab 07

VXLAN EVPN Fabric Design and Production Readiness

Act as the network architect. Configure resilience, security, routing and observability controls, test failure scenarios and evaluate whether your VXLAN EVPN fabric is ready for production.

Dual-Spine EVPN VXLAN Fabric Design Assessment
VXLAN EVPN FABRIC · AS 65001 Spine 1 ECMP Path A Spine 2 ECMP Path B Leaf A VTEP 10.0.0.11 Leaf B VTEP 10.0.0.12 Leaf C VTEP 10.0.0.13 Tenant A Tenant B Shared Services
Design Readiness Score 0%
Engineering Controls Enabled 0 / 8
Production Assessment Review

1. Configure the VXLAN Fabric

Enable or disable individual design controls. Each decision affects the engineering readiness assessment and selected resilience tests.

01 · ECMP Underlay Resilience

Maintain multiple equal-cost routed paths between VTEPs through independent spines.

02 · BGP EVPN Control Plane

Distribute MAC/IP reachability and VNI membership through BGP EVPN.

03 · Underlay MTU Headroom

Allow sufficient MTU for VXLAN encapsulation without unwanted fragmentation.

04 · Tenant VRF Isolation

Separate tenant routing domains and control inter-VRF connectivity.

05 · Distributed Anycast Gateway

Provide consistent default gateways on participating leaf switches.

06 · Tenant Security Policy

Apply explicit filtering and restrict unauthorized tenant-to-tenant traffic.

07 · Fabric Telemetry

Monitor BGP, NVE, interfaces, MAC mobility and forwarding health.

08 · ARP Suppression

Use verified EVPN MAC/IP bindings to reduce unnecessary ARP flooding.

2. Run Production Failure Tests

Select an engineering scenario to assess how your chosen design would behave.

Fabric Design Assessment and Event Report
[READY] Fabric architecture loaded. [READY] Configure design controls. [READY] Run a failure scenario or generate a report.
Design Assessment Ready

Enable the appropriate engineering controls and run resilience tests to evaluate your VXLAN EVPN fabric.

Section 08 · Advanced VXLAN EVPN Architecture

VXLAN EVPN Multi-Site, Data Centre Interconnect and Migration

VXLAN EVPN is not limited to a single leaf-spine fabric. With an appropriate multi-site architecture, organizations can connect data centres, extend selected network segments, distribute routing information and support workload mobility across geographically separate environments.

However, interconnecting VXLAN fabrics introduces design challenges that do not exist within a single data centre. These include inter-site latency, failure isolation, route distribution, gateway placement, broadcast containment and disaster recovery.

The objective is to extend only the network services that require cross-site connectivity, while preserving predictable routing and independent site operation.

VXLAN EVPN Multi-Site Data Centre Interconnect Border Gateways Workload Mobility Distributed Anycast Gateway Layer 2 Extension Migration Disaster Recovery

8.1 — Why Extend VXLAN Between Data Centres?

Enterprises often operate multiple data centres for capacity, availability, geographic coverage or disaster recovery.

VXLAN EVPN can form part of a DCI design that supports selected Layer 2 extensions and Layer 3 connectivity between sites.

USE CASE 01
Workload Mobility

Support workloads that require selected network identities or subnets to remain reachable across sites during planned migration.

USE CASE 02
Disaster Recovery

Provide network connectivity for applications and services activated in an alternate data centre.

USE CASE 03
Application Distribution

Enable controlled communication between application components located in different data centres.

USE CASE 04
Infrastructure Modernization

Connect legacy networks with modern EVPN fabrics during a phased migration.

Design Recommendation

Prefer routed inter-site connectivity when application requirements allow it. Extend Layer 2 only where there is a clear operational or application dependency. Unnecessary Layer 2 extension increases the scope of broadcast and failure domains.


8.2 — VXLAN EVPN Multi-Site Architecture

A multi-site architecture connects separate data centre fabrics through an inter-site transport network.

Depending on the vendor architecture, dedicated or designated border gateways can exchange overlay reachability and forward VXLAN traffic between sites.

Reference Architecture · Two VXLAN EVPN Data Centres
Data Centre A Data Centre B Spine A1 Spine A2 Spine B1 Spine B2 Leaf A1 Leaf A2 Leaf B1 Leaf B2 VTEP VTEP VTEP VTEP Border Gateway A Inter-site VXLAN Border Gateway B Inter-site VXLAN DCI IP WAN Tenant A Tenant B Tenant A Tenant B
Site underlay Inter-site overlay Tenant attachment
Simplified logical architecture. Border gateways may be implemented as dedicated devices or designated leaf roles. Actual multi-site designs require additional redundancy, routing policy and vendor-specific inter-site mechanisms.

8.3 — The Role of Border Gateways

Border gateways provide a controlled point of connectivity between a local VXLAN fabric and the DCI transport network.

In supported multi-site architectures, they can exchange EVPN reachability information and forward encapsulated traffic between participating sites.

Border Gateway Function Engineering Purpose
Inter-site forwarding Carry selected overlay traffic between fabrics
EVPN route exchange Advertise or import permitted endpoint and tenant routes
Policy enforcement Restrict inter-site route distribution and connectivity
BUM containment Control replication behavior between participating sites
Failure isolation Limit the impact of remote site or transport disruptions
Redundancy Support resilient inter-site paths when multiple gateways are deployed
Multi-Site Is Architecture-Specific

The term VXLAN EVPN Multi-Site is also used for particular vendor implementations. Border-gateway roles, control-plane behavior, anycast addressing, BUM replication and failure handling vary by platform and design. Verify the supported architecture rather than assuming that two independent EVPN fabrics can simply be connected without additional configuration.


8.4 — Layer 2 Extension Versus Layer 3 Interconnect

One of the most important DCI decisions is whether a subnet must extend across sites.

Layer 2 extension allows endpoints in separate data centres to participate in the same bridge domain.

Layer 3 interconnect instead routes between distinct subnets and routing domains.

Design Characteristic Layer 2 Extension Layer 3 Interconnect
Subnet model Shared subnet Separate subnets
Broadcast domain Extends between sites Remains site-local
Workload mobility Can preserve an IP subnet during migration Usually requires address or application-level mobility planning
Failure domain Potentially wider More clearly bounded
Operational complexity Greater dependency on MAC and BUM behavior Primarily routed reachability and policy
Typical preference Specific mobility requirements General-purpose inter-site connectivity
Recommended Architecture

Use Layer 3 boundaries by default and introduce stretched Layer 2 segments only when justified. This reduces unnecessary broadcast extension and simplifies site-level fault isolation.


8.5 — Distributed Anycast Gateway Across Sites

A distributed anycast gateway allows multiple participating leaf switches to provide a common default gateway identity for a tenant subnet.

In a supported multi-site design, this can allow workloads to use local first-hop routing even when the subnet is present at multiple sites.

Gateway consistency is critical. IP addressing, gateway MAC behavior, EVPN advertisements and tenant routing policies must follow the selected architecture.

LOCAL ROUTING
First-Hop Gateway

The local leaf can route traffic using the distributed gateway rather than sending every packet to a remote site.

CONSISTENCY
Shared Gateway Identity

Gateway addresses and MAC behavior must be consistent with the supported anycast gateway implementation.

ROUTING POLICY
Symmetric IRB

Symmetric integrated routing and bridging can support scalable routed tenant connectivity through L3 VNIs.

FAILURE RECOVERY
Site Independence

Design gateway and route behavior so local services can continue where feasible during DCI failure.


8.6 — Workload Mobility and MAC Mobility

EVPN includes mechanisms for advertising MAC reachability and signaling endpoint mobility.

When a workload moves from one VTEP to another, the control plane must update reachability so remote devices forward traffic toward the new location.

01
Workload Located in Site A
The local VTEP learns the endpoint and advertises the appropriate EVPN MAC/IP reachability.
02
Workload Migrates to Site B
The endpoint becomes attached to a VTEP in the destination site.
03
EVPN Reachability Changes
The new location advertises updated reachability. MAC mobility procedures help remote devices distinguish the newer location.
04
Remote Forwarding Updates
Participating VTEPs update forwarding state to direct traffic toward the destination site.
05
Validate Application Continuity
Test routing, ARP/ND state, firewall sessions, load balancers and application dependencies.
Mobility Does Not Guarantee Session Continuity

EVPN can update network reachability after endpoint movement, but application continuity also depends on transport sessions, firewall state, load balancing, storage and application architecture. VXLAN alone does not guarantee seamless migration.


8.7 — Inter-Site Failure and Disaster Recovery

DCI failure must be considered separately from failures inside a single data centre fabric.

Each site should have a defined behavior when the inter-site network becomes unavailable.

Failure Potential Impact Design Consideration
DCI link failure Remote overlays become unreachable Local routing independence and alternate paths
Border gateway failure Inter-site forwarding is disrupted Redundant gateways and convergence
Complete site outage Workloads and network services may be unavailable Recovery orchestration, route changes and application failover
Control-plane instability EVPN route churn or inconsistent forwarding Route filtering, monitoring and failure isolation
Extended Layer 2 fault Broadcast or endpoint-learning disruption Minimize stretched segments and control BUM traffic
Split-Brain Risk

When two sites lose communication but continue operating independently, application and network failover mechanisms must avoid conflicting active ownership. VXLAN EVPN connectivity is not a substitute for application-level quorum, storage consistency or disaster recovery coordination.


8.8 — Migrating from Traditional VLAN Networks

Moving from a traditional Layer 2 network to VXLAN EVPN is usually best performed incrementally.

A phased migration allows engineers to validate the new fabric while preserving existing application connectivity.

01
Assess the Existing Network
Document VLANs, subnets, default gateways, trunks, spanning tree, routing and application dependencies.
02
Build and Validate the Underlay
Establish routed leaf-spine connectivity, loopback reachability, ECMP and MTU consistency.
03
Deploy the EVPN Overlay
Configure VTEPs, L2 VNIs, tenant VRFs, L3 VNIs and BGP EVPN route exchange.
04
Establish a Controlled Migration Boundary
Use supported Layer 2 or Layer 3 handoff mechanisms between legacy and EVPN environments. Avoid uncontrolled loops and duplicate gateways.
05
Migrate a Pilot Workload
Test a limited application or subnet, validate endpoint learning and confirm expected routing and security policy.
06
Expand in Controlled Phases
Migrate additional segments using documented change windows, validation tests and rollback plans.
07
Remove Legacy Dependencies
Retire unnecessary VLAN extensions, legacy trunks and obsolete gateway paths after successful validation.

8.9 — Example Migration Mapping

A VLAN-to-VNI mapping plan provides a consistent reference when moving network segments into the EVPN fabric.

Existing VLAN New L2 VNI Tenant VRF Migration Approach
10 10010 TENANT-A Pilot application migration
20 10020 TENANT-A Phased server migration
30 20030 TENANT-B Separate tenant cutover
40 20040 TENANT-B Gateway migration after validation
Gateway Migration Requires Special Care

Moving the default gateway from a traditional switch to a distributed anycast gateway must be planned carefully. Duplicate active gateway ownership, inconsistent ARP behavior or incorrect routing boundaries can disrupt application traffic during cutover.


8.10 — Multi-Site Operational Verification

Engineers should verify both the local EVPN fabric and the inter-site transport.

The following commands illustrate common checks on Cisco NX-OS VXLAN EVPN deployments.

VXLAN EVPN / DCI Verification
show ip route

show nve interface

show nve peers

show nve vni

show bgp l2vpn evpn summary

show bgp l2vpn evpn route-type 2

show bgp l2vpn evpn route-type 3

show bgp l2vpn evpn route-type 5

show l2route evpn mac all

show mac address-table

show vrf

show ip route vrf TENANT-A
Generic NX-OS-style commands. Vendor-specific multi-site features may require additional border-gateway, site identifier, route-policy and inter-site peer verification commands. Availability varies by software release.
Validation Layer What to Confirm
Local underlay VTEP reachability and stable routing adjacencies
DCI transport Border gateway connectivity, MTU and permitted traffic
EVPN control plane Expected inter-site route exchange
Route policy Correct import, export and tenant boundaries
Data plane Encapsulation, remote VTEP forwarding and endpoint delivery
Failure recovery Behavior during DCI or border gateway outage

8.11 — Multi-Site Design Checklist

✓ Each site has a validated local underlay and overlay.
✓ DCI routing, MTU and transport capacity are verified.
✓ Border gateway redundancy and failover are tested.
✓ Inter-site EVPN route distribution follows approved policy.
✓ Layer 2 extension is limited to justified application requirements.
✓ Anycast gateway behavior is consistent across participating sites.
✓ MAC mobility and endpoint movement are tested.
✓ Site isolation and DCI failure behavior are documented.
✓ Application and firewall dependencies are included in migration testing.
✓ Migration rollback and disaster recovery procedures are validated.

Section 8 — Engineering Takeaways

✓ VXLAN EVPN can support selected inter-site Layer 2 and Layer 3 connectivity when deployed using a supported DCI architecture.
✓ Border gateways can provide controlled inter-site forwarding and route exchange.
✓ Routed interconnects should generally be preferred over unnecessary Layer 2 extension.
✓ Distributed anycast gateways and EVPN MAC mobility can support workload movement, but application continuity requires additional planning.
✓ Multi-site designs must account for DCI failures, site isolation, route policy and disaster recovery coordination.
✓ Legacy VLAN migration should follow a phased approach with explicit gateway ownership, validation and rollback procedures.
Next Learning Activity
VXLAN EVPN Multi-Site Architecture and Migration Investigation Lab
Step 1 · Select the Inter-Site Architecture
Two-Site VXLAN EVPN Fabric EVPN Multi-Site
DATA CENTRE 1 DATA CENTRE 2 Leaf A VTEP 10.1.0.11 Leaf B VTEP 10.2.0.11 BGW 1A Site Border BGW 1B Redundant Border BGW 2A Site Border BGW 2B Redundant Border Host A 192.168.10.11 Host B 192.168.10.12 EVPN MULTI-SITE
DCI Architecture EVPN Multi-Site
Inter-Site Paths 2 Available
Traffic Result Ready
Step 2 · Inject a Failure

Both DCI paths are operational.

Architecture Analysis

EVPN Multi-Site

Border gateways connect independent EVPN fabrics while maintaining inter-site reachability for selected tenant networks.

Inter-Site Packet and Failure Investigation Timeline
[READY] DC1 and DC2 initialized. [READY] EVPN Multi-Site selected. [READY] Both DCI paths available. [READY] Trace an inter-site packet.
Step 3 · Migration Planning

Legacy Layer 2 to VXLAN EVPN Migration

Work through five controlled migration stages. Review the required checks before approving each stage.

Migration Stage 01 / 05

Discovery and Dependency Mapping

Identify VLANs, gateways, IP subnets, application dependencies and existing failure domains before migration.

Complete the stage checklist before approval.

Engineering Design Principle

Extending Layer 2 between data centres increases the scope of broadcast and failure domains. Routed DCI is generally preferable when applications do not require subnet continuity. EVPN Multi-Site can support selected extensions, but border-gateway redundancy, loop prevention, failure isolation and operational validation remain essential.

Section 09 · Final Engineering Summary

VXLAN: Engineering Summary, Key Takeaways and Next Steps

VXLAN has become an important building block of modern data centre network architecture. By transporting Ethernet segments across an IP underlay, VXLAN enables scalable overlay networks without requiring the physical fabric to operate as one large Layer 2 domain.

When combined with BGP EVPN, VXLAN supports control-plane distribution of endpoint reachability, multi-tenant segmentation, distributed routing and advanced leaf-spine network designs.

This final section brings together the central engineering concepts explored throughout the article.

Underlay Overlay VTEPs Encapsulation BGP EVPN Tenant Segmentation Operations Multi-Site

9.1 — The Complete VXLAN Architecture

A VXLAN EVPN fabric can be understood through four closely related architectural components.

FOUNDATION 01
IP Underlay

The routed physical network provides IP reachability between VTEPs. Routing protocols and ECMP support resilient transport.

FOUNDATION 02
VXLAN Overlay

VXLAN encapsulates Ethernet frames in UDP packets. The VNI identifies the associated overlay segment.

FOUNDATION 03
BGP EVPN Control Plane

EVPN distributes endpoint and routing information, reducing reliance on traditional data-plane learning across the overlay.

FOUNDATION 04
Distributed Tenant Routing

VRFs, L2 VNIs, L3 VNIs and distributed gateways enable tenant-aware forwarding and inter-subnet routing.

Architecture Recap · From Workload to Workload
BGP EVPN Control Plane MAC/IP and tenant route distribution Routed Spine Fabric IP underlay · ECMP Leaf A VTEP 10.0.0.11 VLAN 10 / VNI 10010 Leaf B VTEP 10.0.0.12 VLAN 10 / VNI 10010 VXLAN · UDP 4789 Host A 192.168.10.11 Host B 192.168.10.12
Conceptual same-subnet VXLAN forwarding. Leaf A encapsulates the original Ethernet frame, the IP underlay transports it to Leaf B, and Leaf B decapsulates it before delivering the frame to Host B.

9.2 — Traditional VLAN Networks Versus VXLAN EVPN

Traditional VLAN networks and VXLAN EVPN fabrics solve different scaling and operational challenges.

Characteristic Traditional VLAN Network VXLAN EVPN Fabric
Segmentation identifier 12-bit VLAN ID 24-bit VNI
Transport model Ethernet switching Overlay over routed IP
Loop avoidance Often relies on spanning tree Routed underlay avoids Layer 2 loops in the core
Multipath forwarding Depends on Layer 2 design Underlay ECMP can use multiple routed paths
Endpoint reachability Primarily data-plane MAC learning EVPN control-plane advertisements plus local learning
Tenant routing Traditional SVI/router model VRFs and distributed IRB
Multi-site connectivity Requires suitable DCI technology Supported EVPN DCI architectures
Operational complexity Simpler for small deployments More components and control-plane state to manage
Important Distinction

VXLAN is the data-plane encapsulation technology. BGP EVPN is a separate control-plane technology commonly used to distribute endpoint and routing information. They work together, but they are not the same protocol.


9.3 — The End-to-End Packet Journey

Understanding a complete forwarding sequence is one of the most useful skills for troubleshooting VXLAN networks.

01
Ingress
Host A sends an Ethernet frame to Leaf A.
02
Lookup
Leaf A resolves the destination MAC and remote VTEP mapping.
03
Encapsulation
Leaf A adds VXLAN, UDP and outer IP headers.
04
Transport
The underlay routes traffic toward the remote VTEP.
05
Delivery
Leaf B removes encapsulation and forwards the original frame.
Engineering Mental Model

When a VXLAN packet fails, investigate the path in layers: local VLAN and VNI mapping, endpoint learning, EVPN reachability, remote VTEP connectivity, underlay routing, encapsulation and destination forwarding.


9.4 — The Most Important VXLAN Design Decisions

Successful deployments depend on a series of architecture choices rather than VXLAN encapsulation alone.

DESIGN 01
Underlay Routing

Select a suitable routing protocol, define loopback addressing and validate ECMP and convergence.

DESIGN 02
VNI and VRF Allocation

Maintain consistent mappings and clear tenant routing boundaries.

DESIGN 03
EVPN Route Policy

Define route import and export behavior, route reflectors and supported EVPN route types.

DESIGN 04
Distributed Gateway

Choose a supported IRB architecture and validate first-hop gateway behavior.

DESIGN 05
Security and Segmentation

Use VRFs, route policy and security enforcement to control tenant connectivity.

DESIGN 06
Operational Visibility

Monitor EVPN sessions, VTEPs, MAC mobility, underlay health and packet drops.


9.5 — Production Engineering Checklist

Before deploying a VXLAN EVPN fabric into production, validate the following core requirements.

✓ Routed underlay provides reliable VTEP reachability.
✓ Underlay MTU supports VXLAN encapsulation overhead.
✓ VLAN-to-VNI mappings are correct and consistent.
✓ EVPN BGP sessions are established and stable.
✓ Expected MAC/IP advertisements are present.
✓ VRF and route-target policies are validated.
✓ Distributed gateway behavior is tested.
✓ Tenant isolation and security policies work.
✓ ECMP forwarding and failover are verified.
✓ MAC mobility and endpoint movement are tested where required.
✓ Hardware forwarding and route scale are understood.
✓ Monitoring, rollback and recovery procedures are documented.
Final Security Reminder

VXLAN does not inherently encrypt traffic or authenticate tunnel endpoints. Route targets are not firewall rules, and logical segmentation alone does not guarantee a complete security boundary. Apply appropriate transport protection and traffic enforcement according to your security requirements.


9.6 — Final Engineering Takeaways

What You Should Remember About VXLAN

✓ VXLAN extends Layer 2 segments across a routed IP infrastructure.
✓ The 24-bit VNI supports a much larger segmentation identifier space than traditional VLAN IDs.
✓ VTEPs perform encapsulation and decapsulation at the overlay boundary.
✓ BGP EVPN provides a scalable control plane for distributing MAC/IP and routing information.
✓ VRFs, L2 VNIs, L3 VNIs and distributed gateways support multi-tenant routing and segmentation.
✓ Production VXLAN requires sound underlay design, MTU planning, security controls, monitoring and failure testing.
✓ Multi-site EVPN can support selected DCI and workload mobility requirements, but should not replace careful application and disaster recovery design.

9.7 — Continue Your VXLAN Learning Journey

Understanding VXLAN architecture is the first step. The next stage is applying these concepts through topology investigation, packet tracing and realistic network engineering scenarios.

Network Insight · Continue Learning

Move from VXLAN Theory to Network Engineering Practice

Explore the wider data centre networking architecture, including leaf-spine design, BGP, routing, traffic investigation and modern overlay technologies.

End of VXLAN Editorial Learning Path. You have completed the nine-section progression from traditional Ethernet limitations to production VXLAN EVPN architecture, operations and multi-site design.
Network Insight · Interactive Engineering Lab 09

VXLAN EVPN Engineering Capstone: Diagnose, Repair and Validate

Complete six engineering investigations covering underlay architecture, VXLAN encapsulation, EVPN control-plane routes, distributed routing, troubleshooting and production design. Use the simulated CLI, select your engineering decisions and complete the final assessment.

Engineering Score 0 / 600
Stages Completed 0 / 6
Current Stage 01
Assessment Status In Progress
Engineering Investigation Fabric Investigation Active
Spine 1 IP Underlay Spine 2 ECMP Leaf A VTEP 10.0.0.11 Leaf B VTEP 10.0.0.12 Host A 192.168.10.11 Host B 192.168.10.12
Stage 01 / 06 · Underlay Architecture

Verify the VXLAN Underlay

The EVPN overlay depends on routed IP reachability between VTEPs. Investigate the fabric before making your engineering decision.

Which underlay condition is required before VXLAN encapsulated traffic can be forwarded?
Simulated Operational CLI

Use NX-OS-style verification commands to inspect the current scenario. Type help for available commands.

LEAF-A · Engineering Console
Network Insight VXLAN EVPN Capstone Leaf-A#
Engineering Investigation Timeline
[READY] Capstone initialized. [READY] Stage 01 loaded. [READY] Review evidence and submit your decision.
Final Engineering Assessment

VXLAN EVPN Capstone Complete

0 / 600

☕
Support Network Insight
Help support the development of interactive networking labs, BGP simulations, and educational content.
Support the Project
Network Insight · Continue Your Learning Journey

From VXLAN Theory to Data Centre Engineering

You've explored VXLAN encapsulation, BGP EVPN, distributed routing and fabric design. Now take the next step into an interactive data centre networking environment.

Explore network architecture, follow packet paths, investigate failures and practise engineering decisions through guided interactive activities.

Explore the Data Centre Network Workspace
Network Insight · Interactive Learning Workspace
From theory to interactive practice

Don't Just Read the Network. Explore It.

You've explored data centre architecture, routing and resilient design. Now take the next step. Use the interactive simulator to investigate how network components and paths work together. Move beyond diagrams and start exploring network behaviour.

Network topology Traffic paths Routing decisions Failure scenarios
LAUNCH THE BGP CORE SIMULATOR Opens in a new browser tab.
Network in motion Illustrative
Illustrative leaf-spine topology Two spine switches connect to two leaf switches. Endpoint devices connect to the leaf switches. Animated dashed lines represent example traffic paths. SPINE 1 SPINE 2 LEAF 1 LEAF 2 HOST A HOST B HOST C HOST D
STEP 01 / OBSERVE Inspect the network Explore the topology, devices and available network state.
STEP 02 / TRACE Follow the behaviour Investigate routes, forwarding decisions and the paths traffic can take.
STEP 03 / INVESTIGATE Think like an engineer Use what you observe to reason about changes, redundancy and failures.
The topology above is illustrative. Open the simulator to explore the functionality it provides.
Matt Conran

Comments are closed.