Hello, I recently completed a 3 part package for Uniken. Part 1 can be found here – Matt Conran & Uniken. Stayed tuned for Part 2 and Part 3!
Paessler – NetFlow for Cybersecurity
Paessler purchased a three package article on NetFlow for Cybersecurity. Generally, the three posts relate to how NetFlow can be used to battle the ongoing threats of cybercriminals. Kindly, click the links. Part 1 – Matt Conran Paessler, Part 2 – Matt Conran Outbound DDoS, Part 3 – Matt Conran Cyberhunter.
Aviatrix Hybrid Cloud – Active Directory
Aviatrix, a hybrid cloud networking specialist used a 3 part blog package to formulate a solution brief – ActiveDirectoryTechBriefWhitePaperR1
Nominum Security Report
I had the pleasure to contribute to Nominum’s Security Report. Kindly click on the link to register and download – Matt Conran with Nominum
“Nominum Data Science just released a new Data Science and Security report that investigates the largest threats affecting organizations and individuals, including ransomware, DDoS, mobile device malware, IoT-based attacks and more. Below is an excerpt:
October 21, 2016, was a day many security professionals will remember. Internet users around the world couldn’t access their favorite sites like Twitter, Paypal, The New York Times, Box, Netflix, and Spotify, to name a few. The culprit: a massive Distributed Denial of Service (DDoS) attack against a managed Domain Name System (DNS) provider not well-known outside technology circles. We were quickly reminded how critical the DNS is to the internet as well as its vulnerability. Many theorize that this attack was merely a Proof of Concept, with far bigger attacks to come”
NS1 – Adding Intelligence to the Internet
I recently completed a two-part guest post for DNS-based company NS1. It discusses Internet challenges and introduces the NS1 traffic management solution – Pulsar. Part 1, kindly click – Matt Conran with NS1, and Part 2, kindly click – Matt Conran with NS1 Traffic Management.
“Application and service delivery over the public Internet is subject to various network performance challenges. This is because the Internet comprises different fabrics, connection points, and management entities, all of which are dynamic, creating unpredictable traffic paths and unreliable conditions. While there is an inherent lack of visibility into end-to-end performance metrics, for the most part, the Internet works, and packets eventually reach their final destination. In this post, we’ll discuss key challenges affecting application performance and examine the birth of new technologies,multi-CDNN designs, and how they affect DNS. Finally, we’ll look at Pulsar and our real-time telemetry engine developed specifically for overcoming many performance challenges by adding intelligence at the DNS lookup stage.”
SD WAN Tutorial: Nuage Networks
Nuage Networks
The following post details Nuage Netowrk and its response to SD-WAN. Part 2 can be found here with Nuage Network and SD-WAN. It’s a 24/7 connected world, and traffic diversity puts the Wide Area Network (WAN) edge to the test. Today’s applications should not be hindered by underlying network issues or a poorly designed WAN. Instead, the business requires designers to find a better way to manage the WAN by adding intelligence via an SD WAN Overlay with improved flow management, visibility, and control.
The WAN Monitoring role has changed from providing basic inter-site connectivity to adapting technology to meet business applications’ demands. It must proactively manage flows over all available paths, regardless of transport type. Business requirements should drive today’s networks, and the business should dictate the directions of flows, not the limitations of a routing protocol. The remainder of the post relates to Nuage Network and services as a good foundation for an SD WAN tutorial.
For additional information, you may find the following posts helpful:
Nuage SD WAN. |
|
The building blocks of the WAN have remained stagnant while the application environment has dynamically shifted; sure, speeds and feeds have increased, but the same architectural choices that were best practice 10 or 15 years ago are still being applied, hindering rapid growth in business evolution. So how will the traditional WAN edge keep up with new application requirements?
Nuage SD WAN
Nuage Networks SD-WAN solution challenges this space and overcomes existing WAN limitations by bringing intelligence to routing at an application level. Now, policy decisions are made by a central platform that has full WAN and data center visibility. A transport-agnostic WAN optimizes the network and the decisions you make about it. In the eyes of Nuage, “every packet counts,” and mission-critical applications are always available on protected premium paths.
Routing Protocols at the WAN Edge
Routing protocols assist in the forwarding decisions for traffic based on destinations, with decisions made hop-by-hop. This limits the number of paths the application traffic can take. Paths are further limited to routing loop restrictions – routing protocols will not take a path that could potentially result in a forwarding loop. Couple this with the traditional forwarding paradigms of primitive WAN designs, resulting in a network that cannot match today’s application requirements. We need to find more granular ways to forward traffic.
There has always been a problem with complex routing for the WAN. BGP supports the best path, and ECMP provides some options for path selection. Solutions like Dynamic Multipoint VPN (DMVPN) operate with multiple control planes that are hard to design and operate. It’s painful to configure QOS policies per-link basis and design WAN solutions to incorporate multiple failure scenarios. The WAN is the most complex module of any network yet so important as it acts as the gateway to other networks such as the branch LAN and data center.
Best path & failover only.
At the network edge, where there are two possible exit paths, choosing a path based on a unique business characteristic is often desirable. For example, use a historical jitter link for web traffic or premium links for mission-critical applications. The granularity for exit path selection should be flexible and selected based on business and application requirements. Criteria for exit points should be application-independent, allowing end-to-end network segmentation.
External policy-based protocol
BGP is an external policy-based protocol commonly used to control path selection. BGP peers with other BGP routers to exchange Network Layer Reachability Information (NLRI). Its flexible policy-orientated approach and outbound traffic engineering offer tailored control for that network slice. As a result, it offers more control than an Interior Gateway Protocol (IGP) and reduces network complexity in large networks. These factors have made BGP the de facto WAN edge routing protocol.
However, the path attributes that influence BGP does not consider any specifically tailored characteristics, such as unique metrics, transit performance, or transit brownouts. When BGP receives multiple paths to the same destination, it runs the best path algorithm to decide the best path to install in the IP routing table; generally, this path selection is based on AS-Path. Unfortunately, AS-Path is not an efficient measure of end-to-end transit. It misses the shape of the network, which can result in long path selection or paths experiencing packet loss.
The traditional WAN
Traditional WAN routes down one path and, by default, have no awareness of what’s happening at the application level (packet loss, jitter, retransmissions). There have been many attempts to enhance the WANs behavior. For example, SLA steering based on enhanced object tracking would poll a metric such as Round Trip Time (RTT).
These methods are popular and widely implemented, but failover events occur on a configurable metric. All these extra configuration parameters make the WAN more complex. Simply acting as band-aids for a network that is under increasing pressure.
“Nuage Networks sponsor this post. All thoughts and opinions expressed are the authors.”
DMVPN Phases | DMVPN Phase 1 2 3
Highlighting DMVPN Phase 1 2 3
Dynamic Multipoint Virtual Private Network ( DMVPN ) is a dynamic virtual private network ( VPN ) form that allows a mesh of VPNs without needing to pre-configure all tunnel endpoints, i.e., spokes. Tunnels on spokes establish on-demand based on traffic patterns without repeated configuration on hubs or spokes. The design is based on DMVPN Phase 1 2 3.
- Point-to-multipoint Layer 3 overlay VPN
In its simplest form, DMVPN is a point-to-multipoint Layer 3 overlay VPN enabling logical hub and spoke topology supporting direct spoke-to-spoke communications depending on DMVPN design ( DMVPN Phases: Phase 1, Phase 2, and Phase 3 ) selection. The DMVPN Phase selection significantly affects routing protocol configuration and how it works over the logical topology. However, parallels between frame-relay routing and DMVPN routing protocols are evident from a routing point of view.
- Dynamic routing capabilities
DMVPN is one of the most scalable technologies when building large IPsec-based VPN networks with dynamic routing functionality. It seems simple, but you could encounter interesting design challenges when your deployment has more than a few spoke routers. This post will help you understand the DMVPN phases and their configurations.
DMVPN Phases. |
|
DMVPN allows the creation of full mesh GRE or IPsec tunnels with a simple template of configuration. From a provisioning point of view, DMPVN is simple.
Before you proceed, you may find the following useful:
- A key point: Video on the DMVPN Phases
The following video discusses the DMVPN phases. The demonstration is performed with Cisco modeling labs, and I go through a few different types of topologies. At the same time, I was comparing the configurations for DMVPN Phase 1 and DMVPN Phase 3. There is also some on-the-fly troubleshooting that you will find helpful and deepen your understanding of DMVPN.
Back to basics with DMVPN.
- Highlighting DMVPN
DMVPN is a Cisco solution providing scalable VPN architecture. In its simplest form, DMVPN is a point-to-multipoint Layer 3 overlay VPN enabling logical hub and spoke topology supporting direct spoke-to-spoke communications depending on DMVPN design ( DMVPN Phases: Phase 1, Phase 2, and Phase 3 ) selection. The DMVPN Phase selection significantly affects routing protocol configuration and how it works over the logical topology. However, parallels between frame-relay routing and DMVPN routing protocols are evident from a routing point of view.
- Introduction to DMVPN technologies
DMVPN uses industry-standardized technologies ( NHRP, GRE, and IPsec ) to build the overlay network. DMVPN uses Generic Routing Encapsulation (GRE) for tunneling, Next Hop Resolution Protocol (NHRP) for on-demand forwarding and mapping information, and IPsec to provide a secure overlay network to address the deficiencies of site-to-site VPN tunnels while providing full-mesh connectivity.
In particular, DMVPN uses Multipoint GRE (mGRE) encapsulation and supports dynamic routing protocols, eliminating many other support issues associated with other VPN technologies. The DMVPN network is classified as an overlay network because the GRE tunnels are built on top of existing transports, also known as an underlay network.
DMVPN Is a Combination of 4 Technologies:
mGRE: In concept, GRE tunnels behave like point-to-point serial links. mGRE behaves like LAN, so many neighbors are reachable over the same interface. The “M” in mGRE stands for multipoint.
Dynamic Next Hop Resolution Protocol ( NHRP ) with Next Hop Server ( NHS ): LAN environments utilize Address Resolution Protocol ( ARP ) to determine the MAC address of your neighbor ( inverse ARP for frame relay ). mGRE, the role of ARP is replaced by NHRP. NHRP binds the logical IP address on the tunnel with the physical IP address used on the outgoing link ( tunnel source ).
The resolution process determines if you want to form a tunnel destination to X and what address tunnel X resolve towards. DMVPN binds IP-to-IP instead of ARP, which binds destination IP to destination MAC address.
- A key point: Lab guide on Next Hop Resolution Protocol (NHRP)
So we know that NHRP is a dynamic routing protocol that focuses on resolving the next hop address for packet forwarding in a network. Unlike traditional static routing protocols, DNHRP adapts to changing network conditions and dynamically determines the optimal path for data transmission. It works with a client-server model. The DMVPN hub is the NHS, and the Spokes are the NHC.
In the following lab topology, we have R11 as the hub with two spokes, R31 and R41. The spokes need to explicitly configure the next hop server (NHS) information with the command: IP nhrp NHS 192.168.100.11 nbma 172.16.11.1. Notice we have the “multicast” keyword at the end of the configuration line. This is used to allow multicast traffic.
As the routing protocol over the DMVPN tunnel, I am running EIGRP, which requires multicast Hellos to form EIGRP neighbor relationships. To form neighbor relationships with BGP, you use TCP, so you would not need the “multicast” keyword.
IPsec tunnel protection and IPsec fault tolerance: DMVPN is a routing technique not directly related to encryption. IPsec is optional and used primarily over public networks. Potential designs exist for DMVPN in public networks with GETVPN, which allows the grouping of tunnels to a single Security Association ( SA ).
Routing: Designers are implementing DMVPN without IPsec for MPLS-based networks to improve convergence as DMVPN acts independently of service provider routing policy. The sites only need IP connectivity to each other to form a DMVPN network. It would be best to ping the tunnel endpoints and route IP between the sites. End customers decide on the routing policy, not the service provider, offering more flexibility than sites connected by MPLS. MPLS-connected sites, the service provider determines routing protocol policies.
Map IP to IP: If you want to reach my private address, you need to GRE encapsulate it and send it to my public address. Spoke registration process.
DMVPN Phases Explained
DMVPN Phases: DMVPN phase 1 2 3
The DMVPN phase selected influence spoke-to-spoke traffic patterns supported routing designs and scalability.
- DMVPN Phase 1: All traffic flows through the hub. The hub is used in the network’s control and data plane paths.
- DMVPN Phase 2: Allows spoke-to-spoke tunnels. Spoke-to-spoke communication does not need the hub in the actual data plane. Spoke-to-spoke tunnels are on-demand based on spoke traffic triggering the tunnel. Routing protocol design limitations exist. The hub is used for the control plane but, unlike phase 1, not necessarily in the data plane.
- DMVPN Phase 3: Improves scalability of Phase 2. We can use any Routing Protocol with any setup. “NHRP redirects” and “shortcuts” take care of traffic flows.
- A key point: Video on DMVPN
In the following video, we will start with the core block of DMVPN, GRE. Generic Routing Encapsulation (GRE) is a tunneling protocol developed by Cisco Systems that can encapsulate a wide variety of network layer protocols inside virtual point-to-point links or point-to-multipoint links over an Internet Protocol network.
We will then move to add the DMVPN configuration parameters. Depending on the DMVPN phase you want to implement, DMVPN can be enabled with just a few commands. Obviously, it would help if you had the underlay in place.
As you know, DMVPN operates as the overlay that lays up an existing underlay network. This demonstration will go through DMVPN Phase 1, which was the starting point of DMVPN, and we will touch on DMVPN Phase 3. We will look at the various DMVPN and NHRP configuration parameters along with the show commands.
The DMVPN Phases
DMVPN Phase 1
- Phase 1 consists of mGRE on the hub and point-to-point GRE tunnels on the spoke.
Hub can reach any spoke over the tunnel interface, but spokes can only go through the hub. No direct Spoke-to-Spoke. Spoke only needs to reach the hub, so a host route to the hub is required. Perfect for default route design from the hub. Design against any routing protocol, as long as you set the next hop to the hub device.
Multicast ( routing protocol control plane ) exchanged between the hub and spoke and not spoke-to-spoke.
On spoke, enter adjust MMS to help with environments where Path MTU is broken. It must be 40 bytes lower than the MTU – IP MTU 1400 & IP TCP adjust-mss 1360. In addition, it inserts the max segment size option in TCP SYN packets, so even if Path MTU does not work, at least TCP sessions are unaffected.
- A key point: Tunnel keys
Tunnel keys are optional for hubs with a single tunnel interface. They can be used for parallel tunnels, usually in conjunction with VRF-light designs. Two tunnels between the hub and spoke, the hub cannot determine which tunnel it belongs to based on destination or source IP address. Tunnel keys identify tunnels and help map incoming GRE packets to multiple tunnel interfaces.
Tunnel Keys on 6500 and 7600: Hardware cannot use tunnel keys. It cannot look that deep in the packet. The CPU switches all incoming traffic, so performance goes down by 100. You should use a different source for each parallel tunnel to overcome this. If you have a static configuration and the network is stable, you could use a “hold-time” and “registration timeout” based on hours, not the 60-second default.
In carrier Ethernet and Cable networks, the spoke IP is assigned by DHCP and can change regularly. Also, in xDSL environments, PPPoE sessions can be cleared, and spokes get a new IP address. Therefore, non-Unique NHRP Registration works efficiently here.
Routing Protocol
Routing for Phase 1 is simple. Summarization and default routing at the hub are allowed. The hub constantly changes next-hop on spokes; the hub is always the next hop. Spoke needs to first communicate with the hub; sending them all the routing information makes no sense. So instead, send them a default route.
Careful with recursive routing – sometimes, the Spoke can advertise its physical address over the tunnel. Hence, the hub attempts to send a DMVPN packet to the spoke via the tunnel, resulting in tunnel flaps.
DMVPN phase 1 OSPF routing
Recommended design should use different routing protocols over DMVPN, but you can extend the OSPF domain by adding the DMVPN network into a separate OSPF Area. Possible to have one big area but with a large number of spokes; try to minimize the topology information spokes have to process.
Redundant set-up with spoke running two tunnels to redundant Hubs, i.e., Tunnel 1 to Hub 1 and Tunnel 2 to Hub 2—designed to have the tunnel interfaces in the same non-backbone area. Having them in separate areas will cause spoke to become Area Border Router ( ABR ). Every OSPF ABR must have a link to Area 0. Resulting in complex OSPF Virtual-Link configuration and additional unnecessary Shortest Path First ( SPF ) runs.
Make sure the SPF algorithm does not consume too much spoke resource. If Spoke is a high-end router with a good CPU, you do not care about SPF running on Spoke. Usually, they are low-end routers, and maintaining efficient resource levels is critical. Potentially design the DMVPN area as a stub or totally stub area. This design prevents changes (for example, prefix additions ) on the non-DVMPN part from causing full or partial SPFs.
Low-end spoke routers can handle 50 routers in single OSPF area.
Configure OSPF point-to-multipoint. Mandatory on the hub and recommended on the spoke. Spoke has GRE tunnels, by default, use OSPF point-to-point network type. Timers need to match for OSPF adjacency to come up.
OSPF is hierarchical by design and not scalable. OSPF over DMVPN is fine if you have fewer spoke sites, i.e., below 100.
DMVPN phase 1 EIGRP routing
On the hub, disable split horizon and perform summarization. Then, deploy EIGRP leak maps for redundant remote sites. Two routers connecting the DMVPN and leak maps specify which information ( routes ) can leak to each redundant spoke.
Deploy spokes as Stub routers. Without stub routing, whenever a change occurs ( prefix lost ), the hub will query all spokes for path information.
Essential to specify interface Bandwidth.
- A key point: Lab guide with DMVPN phase 1 EIGRP.
In the following lab guide, I show how to turn on and off split horizon at the hub sites, R11. So when you turn on split-horizon, the spokes will only see the routes behind R11; in this case, it’s actually only one route. They will not see routes from the other spokes. In addition, I have performed summarization on the hub site. Notice how the spoke only see the summary route.
Turning the split horizon on with summarization, too, will not affect spoke reachability as the hub summarizes the routes. So, if you are performing summarization at the hub site, you can also have split horizon turned on at the hub site, R11,
DMVPN phase 1 BGP routing
Recommended using EBGP. Hub must have next-hop-self on all BGP neighbors. To save resources and configuration steps, possible to use policy templates. Avoid routing updates to spokes by filtering BGP updates or advertising the default route to spoke devices.
In recent IOS, we have dynamic BGP neighbors. Configure the range on the hub with command BGP listens to range 192.168.0.0/24 peer-group spokes. Inbound BGP sessions are accepted if the source IP address is in the specified range of 192.168.0.0/24.
DMVPN Phase 2
Phase 2 allowed mGRE on the hub and spoke, permitting spoke-to-spoke on-demand tunnels. Phase 2 consists of no changes on the hub router; change tunnel mode on spokes to GRE multipoint – tunnel mode gre multipoint. Tunnel keys are mandatory when multiple tunnels share the same source interface.
Multicast traffic still flows between the hub and spoke only, but data traffic can now flow from spoke to spoke.
DMVPN Packet Flows and Routing
DMVPN phase 2 packet flow
| -For initial packet flow, even though the routing table displays the spoke as the Next Hop, all packets are sent to the hub router. Shortcut not established. |
| -The spokes send NHRP requests to the Hub and ask the hub about the IP address of the other spokes. |
| -Reply is received and stored on the NHRP dynamic cache on the spoke router. |
| -Now, spokes attempt to set up IPSEC and IKE sessions with other spokes directly. |
| -Once IKE and IPSEC become operational, the NHRP entry is also operational, and the CEF table is modified so spokes can send traffic directly to spokes. |
The process is unidirectional. Reverse traffic from other spoke triggers the exact mechanism. Spokes don’t establish two unidirectional IPsec sessions; Only one.
There are more routing protocol restrictions with Phase 2 than DMVPN Phases 1 and 3. For example, summarization and default routing is NOT allowed at the hub, and the hub always preserves the next hop on spokes. Spokes need specific routes to each other networks.
DMVPN phase 2 OSPF routing
Recommended using OSPF network type Broadcast. Ensure the hub is DR. You will have a disaster if a spoke becomes a Designated Router ( DR ). For that reason, set the spoke OSPF priority to “ZERO.”
OSPF multicast packets are delivered to the hub only. Due to configured static or dynamic NHRP multicast maps, OSPF neighbor relationships only formed between the hub and spoke.
The spoke router needs all routes from all other spokes, so default routing is impossible for the hub.
DMVPN phase 2 EIGRP routing
No changes to the spoke. Add no IP next-hop-self on a hub only—Disable EIRP split-horizon on hub routers to propagate updates between spokes.
Do not use summarization; if you configure summarization on spokes, routes will not arrive in other spokes. Resulting in spoke-to-spoke traffic going to the hub.
DMVPN phase 2 BGP pouting
Remove the next-hop-self on hub routers.
Split default routing
Split default routing may be used if you have the requirement for default routing to the hub: maybe for central firewall design, and you want all traffic to go there before proceeding to the Internet. However, the problem with Phase 2 allows spoke-to-spoke traffic, so even though we would default route pointing to the hub, we need the default route point to the Internet.
Require two routing perspectives; one for GRE and IPsec packets and another for data traversing the enterprise WAN. Possible to configure Policy Based Routing ( PBR ) but only as a temporary measure. PBR can run into bugs and is difficult to troubleshoot. Split routing with VRF is much cleaner. Routing tables for different VRFs may contain default routes. Routing in one VRF will not affect routing in another VRF.
Multi-homed remote site
To make it complicated, the spoke needs two 0.0.0/0. One for each DMVPN Hub network. Now, we have two default routes in the same INTERNET VRF. We need a mechanism to tell us which one to use and for which DMVPN cloud.
Even if the tunnel source is for mGRE-B ISP-B, the routing table could send the traffic to ISP-A. ISP-A may perform uRFC to prevent address spoofing. It results in packet drops.
The problem is that the outgoing link ( ISP-A ) selection depends on Cisco Express Forwarding ( CEF ) hashing, which you cannot influence. So, we have a problem: the outgoing packet has to use the correct outgoing link based on the source and not the destination IP address. The solution is Tunnel Route-via – Policy routing for GRE. To get this to work with IPsec, install two VRFs for each ISP.
DMVPN Phase 3
Phase 3 consists of mGRE on the hub and mGRE tunnels on the spoke. Allows spoke-to-spoke on-demand tunnels. The difference is that when the hub receives an NHRP request, it can redirect the remote spoke to tell them to update their routing table.
- A key point: Lab on DMVPN Phase 3 configuration
The following lab configuration shows an example of DMVPN Phase 3. The command: Tunnel mode gre multipoint GRE is on both the hub and the spokes. This contrasts with DMVPN Phase 1, where we must explicitly configure the tunnel destination on the spokes. Notice the command: Show IP nhrp. We have two spokes. dynamically learned via the NHRP resolution process with the flag “registered nhop.” However, this is only part of the picture for DMVPN Phase 3. We need configurations to enable dynamic spoke-to-spoke tunnels, and this is discussed next.
Phase 3 redirect features
The Phase 3 DMVPN configuration for the hub router adds the interface parameter command ip nhrp redirect on the hub router. This command checks the flow of packets on the tunnel interface and sends a redirect message to the source spoke router when it detects packets hair pinning out of the DMVPN cloud.
Hairpinning means traffic is received and sent to an interface in the same cloud (identified by the NHRP network ID). For instance, hair pinning occurs when packets come in and go out of the same tunnel interface. The Phase 3 DMVPN configuration for spoke routers uses the mGRE tunnel interface and the command ip nhrp shortcut on the tunnel interface.
Note: Placing ip nhrp shortcut and ip nhrp redirect on the same DMVPN tunnel interface has no adverse effects.
Phase 3 allows spoke-to-spoke communication even with default routing. So even though the routing table points to the hub, the traffic flows between spokes. No limits on routing; we still get spoke-to-spoke traffic flow even when you use default routes.
“Traffic-driven-redirect”; hub notices the spoke is sending data to it, and it sends a redirect back to the spoke, saying use this other spoke. Redirect informs the sender of a better path. The spoke will install this shortcut and initiate IPsec with another spoke. Use ip nhrp redirect on hub routers & ip nhrp shortcuts on spoke routers.
No restrictions on routing protocol or which routes are received by spokes. Summarization and default routing is allowed. The next hop is always the hub.
- A key point: Lab guide on DMVPN Phase 3
I have the command in the following lab guide: IP nhrp shortcut on the spoke, R31. I also have the “redirect” command on the hub, R11. So, we don’t see the actual command on the hub, but we do see that R11 is sending a “Traffic Indication” message to the spokes. This was sent when spoke-to-spoke traffic is initiated, informing the spokes that a better and more optimal path exists without going to the hub.
 | Main Checklist Points To Consider
|
What is OpenFlow
What is OpenFlow?
In today's rapidly evolving digital landscape, network management and data flow control have become critical for businesses of all sizes. OpenFlow is one technology that has gained significant attention and is transforming how networks are managed. In this blog post, we will delve into the concept of OpenFlow, its advantages, and its implications for network control.
OpenFlow is an open-standard communications protocol that separates the control and data planes in a network architecture. It allows network administrators to have direct control over the behavior of network devices, such as switches and routers, by utilizing a centralized controller.
Traditional network architectures follow a closed model, where network devices make independent decisions on forwarding packets. On the other hand, OpenFlow introduces a centralized control plane that provides a global view of the network and allows administrators to define network policies and rules from a centralized location.
OpenFlow operates by establishing a secure channel between the centralized controller and the network switches. The controller is responsible for managing the flow tables within the switches, defining how traffic should be forwarded based on predefined rules and policies. This separation of control and data planes allows for dynamic network management and facilitates the implementation of innovative network protocols.
One of the key advantages of OpenFlow is its ability to simplify network management. By centralizing control, administrators can easily configure and manage the entire network from a single point of control. This reduces complexity and enhances the scalability of network infrastructure. Additionally, OpenFlow enables network programmability, allowing for the development of custom networking applications and services tailored to specific requirements.
OpenFlow plays a crucial role in network virtualization, as it allows for the creation and management of virtual networks on top of physical infrastructure. By abstracting the underlying network, OpenFlow empowers organizations to optimize resource utilization, improve security, and enhance network performance. It opens doors to dynamic provisioning, isolation, and efficient utilization of network resources.
Highlights: What is OpenFlow?
How does OpenFlow work?
1: OpenFlow allows network controllers to determine the path of network packets in a network of switches. There is a difference between switches and controllers. With separate control and forwarding, traffic management can be more sophisticated than access control lists (ACLs) and routing protocols.
2: An OpenFlow protocol allows switches from different vendors, often with proprietary interfaces and scripting languages, to be managed remotely. Software-defined networking (SDN) is considered to be enabled by OpenFlow by its inventors.
3: With OpenFlow, Layer 3 switches can add, modify, and remove packet-matching rules and actions remotely. By doing so, routing decisions can be made periodically or ad hoc by the controller and translated into rules and actions with a configurable lifespan, which are then deployed to the switch’s flow table, where packets are forwarded at wire speed for the duration of the rule.
The Role of OpenFlow Controllers
If the switch cannot match packets, they can be sent to the controller. The controller can modify existing flow table rules or deploy new rules to prevent a structural traffic flow. It may even forward the traffic itself if the switch is instructed to forward packets rather than just their headers.
OpenFlow uses Transport Layer Security (TLS) over Transmission Control Protocol (TCP). Switches wishing to connect should listen on TCP port 6653. In earlier versions of OpenFlow, port 6633 was unofficially used. The protocol is mainly used between switches and controllers.
### Origins of OpenFlow
The inception of OpenFlow can be traced back to the early 2000s when researchers at Stanford University sought to create a more versatile and programmable network architecture. Traditional networking relied heavily on static and proprietary hardware configurations, which limited innovation and adaptability. OpenFlow emerged as a solution to these challenges, offering a standardized protocol that decouples the network control plane from the data plane. This separation allows for centralized control and dynamic adjustment of network traffic, fostering innovation and agility in network design.
### How OpenFlow Works
At its core, the OpenFlow protocol facilitates communication between network devices and a centralized SDN controller. It does this by using a series of flow tables within network switches, which are programmed by the controller. These flow tables dictate how packets should be handled, whether they are forwarded to a destination, dropped, or modified in some way. By leveraging OpenFlow, network administrators can deploy updates, optimize performance, and troubleshoot issues with unprecedented speed and precision, all from a single point of control.
Introducing SDN
Recent changes and requirements have driven networks and network services to become more flexible, virtualization-aware, and API-driven. One major trend affecting the future of networking is software-defined networking ( SDN ). The software-defined architecture aims to extract the entire network into a single switch.
Software-defined networking (SDN) is an evolving technology defined by the Open Networking Foundation ( ONF ). It involves the physical separation of the network control plane from the forwarding plane, where a control plane controls several devices. This differs significantly from traditional IP forwarding that you may have used in the past.
The Core Concepts of SDN
At its heart, Software-Defined Networking decouples the network control plane from the data plane, allowing network administrators to manage network services through abstraction of lower-level functionality. This separation enables centralized network control, which simplifies the management of complex networks. The control plane makes decisions about where traffic is sent, while the data plane forwards traffic to the selected destination. This approach allows for a more flexible, adaptable network infrastructure.
The activities around OpenFlow
Even though OpenFlow has received a lot of industry attention, programmable networks and decoupled control planes (control logic) from data planes have been around for many years. To enhance ATM, Internet, and mobile networks’ openness, extensibility, and programmability, the Open Signaling (OPENING) working group held workshops in 1995. A working group within the Internet Engineering Task Force (IETF) developed GSMP to control label switches based on these ideas. June 2002 marked the official end of this group, and GSMPv3 was published.
Data and control plane
Therefore, SDN separates the data and control plane. The main driving body behind software-defined networking (SDN) is the Open Networking Foundation ( ONF ). Introduced in 2008, the ONF is a non-profit organization that wants to provide an alternative to proprietary solutions that limit flexibility and create vendor lock-in.
The insertion of the ONF allowed its members to run proof of concepts on heterogeneous networking devices without requiring vendors to expose their software’s internal code. This creates a path for an open-source approach to networking and policy-based controllers.
Knowledge Check: Data & Control Plane
### Data Plane: The Highway for Your Data
The data plane, often referred to as the forwarding plane, is responsible for the actual movement of packets of data from source to destination. Imagine it as a network’s highway, where data travels at high speed. This component operates at the speed of light, handling massive amounts of data with minimal delay. It’s designed to process packets quickly, ensuring that the information arrives where it needs to be without interruption.
### Control Plane: The Brain Behind the Operation
While the data plane acts as the highway, the control plane is the brain that orchestrates the flow of traffic. It makes decisions about routing, managing the network topology, and controlling the data plane’s operations. The control plane uses protocols to determine the best paths for data and to update routing tables as needed. It’s responsible for ensuring that the network operates efficiently, adapting to changes, and maintaining optimal performance.
### Interplay Between Data and Control Planes
The synergy between the data and control planes is what enables modern networks to function effectively. The control plane provides the intelligence and decision-making necessary to guide the data plane. This interaction ensures that data packets take the best possible paths, reducing latency and maximizing throughput. As networks evolve, the lines between these planes may blur, but their distinct roles remain pivotal.
### Real-World Applications and Innovations
The concepts of data and control planes are not just theoretical—they have practical applications in technologies such as Software-Defined Networking (SDN) and Network Function Virtualization (NFV). These innovations allow for greater flexibility and scalability in managing network resources, offering businesses the agility needed to adapt to changing demands.
Building blocks: SDN Environment
As a fundamental building block of an SDN deployment, the controller, the SDN switch (for example, an OpenFlow switch), and the interfaces are present in the controller to communicate with forwarding devices, generally the southbound interface (OpenFlow) and the northbound interface (the network application interface).
In an SDN, switches function as basic forwarding hardware, accessible via an open interface, with the control logic and algorithms offloaded to controllers. Hybrid (OpenFlow-enabled) and pure (OpenFlow-only) OpenFlow switches are available.
OpenFlow switches rely entirely on a controller for forwarding decisions, without legacy features or onboard control. Hybrid switches support OpenFlow as well, in addition to traditional operation and protocols. Today, hybrid switches are the most common type of commercial switch. A flow table performs packet lookup and forwarding in an OpenFlow switch.
### The Role of the OpenFlow Controller
The OpenFlow controller is the brain of the SDN, orchestrating the flow of data across the network. It communicates with the network devices using the OpenFlow protocol to dictate how packets should be handled. The controller’s primary function is to make decisions on the path data packets should take, ensuring optimal network performance and resource utilization. This centralization of control allows for dynamic network configuration, paving the way for innovative applications and services.
### OpenFlow Switches: The Workhorses of the Network
While the controller is the brain, OpenFlow switches are the workhorses, executing the instructions they receive. These switches operate at the data plane, where they forward packets based on the rules set by the controller. Each switch maintains a flow table that matches incoming packets to particular actions, such as forwarding or dropping the packet. This separation of control and data planes is what sets SDN apart from traditional networking, offering unparalleled flexibility and control.
### Flow Tables: The Heart of OpenFlow Switches
Flow tables are the core component of OpenFlow switches, dictating how packets are handled. Each entry in a flow table consists of match fields, counters, and a set of instructions. Match fields identify the packets that should be affected by the rule, counters track the number of packets and bytes that match the entry, and instructions define the actions to be taken. This modular approach allows for precise traffic management and is essential for implementing advanced network policies.
OpenFlow Table & Routing Tables
### What is an OpenFlow Flow Table?
OpenFlow is a protocol that allows network controllers to interact with the forwarding plane of network devices like switches and routers. At the heart of OpenFlow is the flow table. This table contains a set of flow entries, each specifying actions to take on packets that match a particular pattern. Unlike traditional routing tables, which rely on predefined paths and protocols, OpenFlow flow tables provide flexibility and programmability, allowing dynamic changes in how packets are handled based on real-time network conditions.
### Understanding Routing Tables
Routing tables are a staple of traditional networking, used by routers to determine the best path for forwarding packets to their final destinations. These tables consist of a list of network destinations, with associated metrics that help in selecting the most efficient route. Routing protocols such as OSPF, BGP, and RIP are employed to maintain and update these tables, ensuring that data flows smoothly across the interconnected web of networks. While reliable, routing tables are less flexible compared to OpenFlow flow tables, as changes in network traffic patterns require updates to routing protocols and configurations.
Example of a Routing Table
### Key Differences Between OpenFlow Flow Table and Routing Table
The primary distinction between OpenFlow flow tables and routing tables lies in their approach to network management. OpenFlow flow tables are dynamic and programmable, allowing for real-time adjustments and fine-grained control over traffic flows. This makes them ideal for environments where network agility and customization are paramount. Conversely, routing tables offer a more static and predictable method of packet forwarding, which can be beneficial in stable networks where consistency and reliability are prioritized.
### Use Cases: When to Use Each
OpenFlow flow tables are particularly advantageous in software-defined networking (SDN) environments, where network administrators need to quickly adapt to changing conditions and optimize traffic flows. They are well-suited for data centers, virtualized networks, and scenarios requiring high levels of automation and scalability. On the other hand, traditional routing tables are best used in established networks with predictable traffic patterns, such as those found in enterprise or service provider settings, where reliability and stability are key.
You may find the following useful for pre-information:
What is OpenFlow?
What is OpenFlow?
OpenFlow was the first protocol of the Software-Defined Networking (SDN) trend and is the only protocol that allows the decoupling of a network device’s control plane from the data plane. In most straightforward terms, the control plane can be thought of as the brains of a network device. On the other hand, the data plane can be considered hardware or application-specific integrated circuits (ASICs) that perform packet forwarding.
Numerous devices also support running OpenFlow in a hybrid mode, meaning OpenFlow can be deployed on a given port, virtual local area network (VLAN), or even within a regular packet-forwarding pipeline such that if there is not a match in the OpenFlow table, then the existing forwarding tables (MAC, Routing, etc.) are used, making it more analogous to Policy Based Routing (PBR).
What is SDN?
Despite various modifications to the underlying architecture and devices (such as switches, routers, and firewalls), traditional network technologies have existed since the inception of networking. Using a similar approach, frames and packets have been forwarded and routed in a limited manner, resulting in low efficiency and high maintenance costs. Consequently, the architecture and operation of networks need to evolve, resulting in SDN.
By enabling network programmability, SDN promises to simplify network control and management and allow innovation in computer networking. Network engineers configure policies to respond to various network events and application scenarios. They can achieve the desired results by manually converting high-level policies into low-level configuration commands.
Often, minimal tools are available to accomplish these very complex tasks. Controlling network performance and tuning network management are challenging and error-prone tasks.
A modern network architecture consists of a control plane, a data plane, and a management plane; the control and data planes are merged into a machine called Inside the Box. To overcome these limitations, programmable networks have emerged.
How OpenFlow Works:
At the core of OpenFlow is the concept of a flow table, which resides in each OpenFlow-enabled switch. The flow table contains match-action rules defining how incoming packets should be processed and forwarded. The centralized controller determines these rules and communicates using the OpenFlow protocol with the switches.
When a packet arrives at an OpenFlow-enabled switch, it is first matched against the rules in the flow table. If a match is found, the corresponding action is executed, including forwarding the packet, dropping it, or sending it to the controller for further processing. This decoupling of the control and data planes allows for flexible and programmable network management.
What is OpenFlow SDN?
The main goal of SDN is to separate the control and data planes and transfer network intelligence and state to the control plane. These concepts have been exploited by technologies like Routing Control Platform (RCP), Secure Architecture for Network Enterprise (SANE), and, more recently, Ethane.
In addition, there is often a connection between SDN and OpenFlow. The Open Networking Foundation (ONF) is responsible for advancing SDN and standardizing OpenFlow, whose latest version is 1.5.0.
An SDN deployment starts with these building blocks.
For communication with forwarding devices, the controller has the SDN switch (for example, an OpenFlow switch), the SDN controller, and the interfaces. An SDN deployment is based on two basic building blocks: a southbound interface (OpenFlow) and a northbound interface (the network application interface).
As the control logic and algorithms are offloaded to a controller, switches in SDNs may be represented as basic forwarding hardware. Switches that support OpenFlow come in two varieties: pure (OpenFlow-only) and hybrid (OpenFlow-enabled).
Pure OpenFlow switches do not have legacy features or onboard control for forwarding decisions. A hybrid switch can operate with both traditional protocols and OpenFlow. Hybrid switches make up the majority of commercial switches available today. In an OpenFlow switch, a flow table performs packet lookup and forwarding.
OpenFlow reference switch
The OpenFlow protocol and interface allow OpenFlow switches to be accessed as essential forwarding elements. A flow-based SDN architecture like OpenFlow simplifies switching hardware. Still, it may require additional forwarding tables, buffer space, and statistical counters that are difficult to implement in traditional switches with integrated circuits tailored to specific applications.
There are two types of switches in an OpenFlow network: hybrids (which enable OpenFlow) and pores (which only support OpenFlow). OpenFlow is supported by hybrid switches and traditional protocols (L2/L3). OpenFlow switches rely entirely on a controller for forwarding decisions and do not have legacy features or onboard control.
Hybrid switches are the majority of the switches currently available on the market. This link must remain active and secure because OpenFlow switches are controlled over an open interface (through a TCP-based TLS session). OpenFlow is a messaging protocol that defines communication between OpenFlow switches and controllers, which can be viewed as an implementation of SDN-based controller-switch interactions.
Identify the Benefits of OpenFlow
Application-driven routing. Users can control the network paths. | The networks paths.A way to enhance link utilization. |
An open solution for VM mobility. No VLAN reliability. | A means to traffic engineer without MPLS. |
A solution to build very large Layer 2 networks. | A way to scale Firewalls and Load Balancers. |
A way to configure an entire network as a whole as opposed to individual entities. | A way to build your own encryption solution. Off-the-box encryption. |
A way to distribute policies from a central controller. | Customized flow forwarding. Based on a variety of bit patterns. |
A solution to get a global view of the network and its state. End-to-end visibility. | A solution to use commodity switches in the network. Massive cost savings. |
The following table lists the Software Networking ( SDN ) benefits and the problems encountered with existing control plane architecture:
Identify the benefits of OpenFlow and SDN | Problems with the existing approach |
Faster software deployment. | Large scale provisioning and orchestration. |
Programmable network elements. | Limited traffic engineering ( MPLS TE is cumbersome ) |
Faster provisioning. | Synchronized distribution policies. |
Centralized intelligence with centralized controllers. | Routing of large elephant flows. |
Decisions are based on end-to-end visibility. | Qos and load based forwarding models. |
Granular control of flows. | Ability to scale with VLANs. |
Decreases the dependence on network appliances like load balancers. |
**A key point: The lack of a session layer in the TCP/IP stack**
Regardless of the hype and benefits of SDN, neither OpenFlow nor other SDN technologies address the real problems of the lack of a session layer in the TCP/IP protocol stack. The problem is that the client’s application ( Layer 7 ) connects to the server’s IP address ( Layer 3 ), and if you want to have persistent sessions, the server’s IP address must remain reachable.
This session’s persistence and the ability to connect to multiple Layer 3 addresses to reach the same device is the job of the OSI session layer. The session layer provides the services for opening, closing, and managing a session between end-user applications. In addition, it allows information from different sources to be correctly combined and synchronized.
The problem is the TCP/IP reference module does not consider a session layer, and there is none in the TCP/IP protocol stack. SDN does not solve this; it gives you different tools to implement today’s kludges.
Control and data plane
When we identify the benefits of OpenFlow, let us first examine traditional networking operations. Traditional networking devices have a control and forwarding plane, depicted in the diagram below. The control plane is responsible for setting up the necessary protocols and controls so the data plane can forward packets, resulting in end-to-end connectivity. These roles are shared on a single device, and the fast packet forwarding ( data path ) and the high-level routing decisions ( control path ) occur on the same device.
What is OpenFlow | SDN separates the data and control plane?
**Control plane**
The control plane is part of the router architecture and is responsible for drawing the network map in routing. When we mention control planes, you usually think about routing protocols, such as OSPF or BGP. But in reality, the control plane protocols perform numerous other functions, including:
Connectivity management ( BFD, CFM ) | Interface state management ( PPP, LACP ) |
Service provisioning ( RSVP for InServ or MPLS TE) | Topology and reachability information exchange ( IP routing protocols, IS-IS in TRILL/SPB ) |
Adjacent device discovery via HELLO mechanism | ICMP |
Control plane protocols run over data plane interfaces to ensure “shared fate” – if the packet forwarding fails, the control plane protocol fails as well.
Most control plane protocols ( BGP, OSPF, BFD ) are not data-driven. A BGP or BFD packet is never sent as a direct response to a data packet. There is a question mark over the validity of ICMP as a control plane protocol. The debate is whether it should be classed in the control or data plane category.
Some ICMP packets are sent as replies to other ICMP packets, and others are triggered by data plane packets, i.e., data-driven. My view is that ICMP is a control plane protocol that is triggered by data plane activity. After all, the “C” is ICMP does stand for “Control.”
**Data plane**
The data path is part of the routing architecture that decides what to do when a packet is received on its inbound interface. It is primarily focused on forwarding packets but also includes the following functions:
ACL logging | Netflow accounting |
NAT session creation | NAT table maintenance |
The data forwarding is usually performed in dedicated hardware, while the additional functions ( ACL logging, Netflow accounting ) typically happen on the device CPU, commonly known as “punting.” The data plane for an OpenFlow-enabled network can take a few forms.
However, the most common, even in the commercial offering, is the Open vSwitch, often called the OVS. The Open vSwitch is an open-source implementation of a distributed virtual multilayer switch. It enables a switching stack for virtualization environments while supporting multiple protocols and standards.
Software-defined networking changes the control and data plane architecture.
The concept of SDN separates these two planes, i.e., the control and forwarding planes are decoupled. This allows the networking devices in the forwarding path to focus solely on packet forwarding. An out-of-band network uses a separate controller ( orchestration system ) to set up the policies and controls. Hence, the forwarding plane has the correct information to forward packets efficiently.
In addition, it allows the network control plane to be moved to a centralized controller on a server instead of residing on the same box carrying out the forwarding. Moving the intelligence ( control plane ) of the data plane network devices to a controller enables companies to use low-cost, commodity hardware in the forwarding path. A significant benefit is that SDN separates the data and control plane, enabling new use cases.
A centralized computation and management plane makes more sense than a centralized control plane.
The controller maintains a view of the entire network and communicates with Openflow ( or, in some cases, BGP with BGP SDN ) with the different types of OpenFlow-enabled network boxes. The data path portion remains on the switch, such as the OVS bridge, while the high-level decisions are moved to a separate controller. The data path presents a clean flow table abstraction, and each flow table entry contains a set of packet fields to match, resulting in specific actions ( drop, redirect, send-out-port ).
When an OpenFlow switch receives a packet it has never seen before and doesn’t have a matching flow entry, it sends the packet to the controller for processing. The controller then decides what to do with the packet.
Applications could then be developed on top of this controller, performing security scrubbing, load balancing, traffic engineering, or customized packet forwarding. The centralized view of the network simplifies problems that were harder to overcome with traditional control plane protocols.
A single controller could potentially manage all OpenFlow-enabled switches. Instead of individually configuring each switch, the controller can push down policies to multiple switches simultaneously—a compelling example of many-to-one virtualization.
Now that SDN separates the data and control plane, the operator uses the centralized controller to choose the correct forwarding information per-flow basis. This allows better load balancing and traffic separation on the data plane. In addition, there is no need to enforce traffic separation based on VLANs, as the controller would have a set of policies and rules that would only allow traffic from one “VLAN” to be forwarded to other devices within that same “VLAN.”
The advent of VXLAN
With the advent of VXLAN, which allows up to 16 million logical entities, the benefits of SDN should not be purely associated with overcoming VLAN scaling issues. VXLAN already does an excellent job with this. It does make sense to deploy a centralized control plane in smaller independent islands; in my view, it should be at the edge of the network for security and policy enforcement roles. Using Openflow on one or more remote devices is easy to implement and scale.
It also decreases the impact of controller failure. If a controller fails and its sole job is implementing packet filters when a new user connects to the network, the only affecting element is that the new user cannot connect. If the controller is responsible for core changes, you may have interesting results with a failure. New users not being able to connect is bad, but losing your entire fabric is not as bad.
A traditional networking device runs all the control and data plane functions. The control plane, usually implemented in the central CPU or the supervisor module, downloads the forwarding instructions into the data plane structures. Every vendor needs communications protocols to bind the two planes together to download forward instructions.
Therefore, all distributed architects need a protocol between control and data plane elements. The protocol binding this communication path for traditional vendor devices is not open-source, and every vendor uses its proprietary protocol (Cisco uses IPC—InterProcess Communication ).
Openflow tries to define a standard protocol between the control plane and the associated data plane. When you think of Openflow, you should relate it to the communication protocol between the traditional supervisors and the line cards. OpenFlow is just a low-level tool.
OpenFlow is a control plane ( controller ) to data plane ( OpenFlow enabled device ) protocol that allows the control plane to modify forwarding entries in the data plane. It enables SDN to separate the data and control planes.
Proactive versus reactive flow setup
OpenFlow operations have two types of flow setups: Proactive and Reactive.
With Proactive, the controller can populate the flow tables ahead of time, similar to a typical routing. However, the packet-in event never occurs by pre-defining your flows and actions ahead of time in the switch’s flow tables. The result is all packets are forwarded at line rate. With Reactive, the network devices react to traffic, consult the OpenFlow controller, and create a rule in the flow table based on the instruction. The problem with this approach is that there can be many CPU hits.
The following table outlines the critical points for each type of flow setup:
Proactive flow setup | Reactive flow setup |
Works well when the controller is emulating BGP or OSPF. | Used when no one can predict when and where a new MAC address will appear. |
The controller must first discover the entire topology. | Punts unknown packets to the controller. Many CPU hits. |
Discover endpoints ( MAC addresses, IP addresses, and IP subnets ) | Compute forwarding paths on demand. Not off the box computation. |
Compute off the box optimal forwarding. | Install flow entries based on actual traffic. |
Download flow entries to the data plane switches. | Has many scalability concerns such as packet punting rate. |
No data plane controller involvement with the exceptions of ARP and MAC learning. Line-rate performance. | Not a recommended setup. |
Hop-by-hop versus path-based forwarding
The following table illustrates the key points for the two types of forwarding methods used by OpenFlow: hop-by-hop forwarding and path-based forwarding:
Hop-by-hop Forwarding | Path-based Forwarding |
Similar to traditional IP Forwarding. | Similar to MPLS. |
Installs identical flows on each switch on the data path. | Map flows to paths on ingress switches and assigns user traffic to paths at the edge node |
Scalability concerns relating to flow updates after a change in topology. | Compute paths across the network and installs end-to-end path-forwarding entries. |
Significant overhead in large-scale networks. | Works better than hop-by-hop forwarding in large-scale networks. |
FIB update challenges. Convergence time. | Core switches don’t have to support the same granular functionality as edge switches. |
Obviously, with any controller, the controller is a lucrative target for attack. Anyone who knows you are using a controller-based network will try to attack the controller and its control plane. The attacker may attempt to intercept the controller-to-switch communication and replace it with its commands, essentially attacking the control plane with whatever means they like.
An attacker may also try to insert a malformed packet or some other type of unknown packet into the controller ( fuzzing attack ), exploiting bugs in the controller and causing the controller to crash.
Fuzzing attacks can be carried out with application scanning software such as Burp Suite. It attempts to manipulate data in a particular way, breaking the application.
The best way to tighten security is to encrypt switch-to-controller communications with SSL and self-signed certificates to authenticate the switch and controller. It would also be best to minimize interaction with the data plane, except for ARP and MAC learning.
To prevent denial-of-service attacks on the controller, you can use Control Plane Policing ( CoPP ) on Ingress to avoid overloading the switch and the controller. Currently, NEC is the only vendor implementing CoPP.
The Hybrid deployment model is helpful from a security perspective. For example, you can group specific ports or VLANs to OpenFlow and other ports or VLANs to traditional forwarding, then use traditional forwarding to communicate with the OpenFlow controller.
Software-defined networking or traditional routing protocols?
The move to a Software-Defined Networking architecture has clear advantages. It’s agile and can react quickly to business needs, such as new product development. For businesses to succeed, they must have software that continues evolving.
Otherwise, your customers and staff may lose interest in your product and service. The following table displays the advantages and disadvantages of the existing routing protocol control architecture.
+Reliable and well known. | -Non-standard Forwarding models. Destination-only and not load-aware metrics** |
+Proven with 20 plus years field experience. | -Loosely coupled. |
+Deterministic and predictable. | -Lacks end-to-end transactional consistency and visibility. |
+Self-Healing. Traffic can reroute around a failed node or link. | -Limited Topology discovery and extraction. Basic neighbor and topology tables. |
+Autonomous. | -Lacks the ability to change existing control plane protocol behavior. |
+Scalable. | -Lacks the ability to introduce new control plane protocols. |
+Plenty of learning and reading materials. |
** Basic EIGRP IETF originally proposed an Energy-Aware Control Plane, but the IETF later removed this.
Software-Defined Networking: Use Cases
Edge Security policy enforcement at the network edge. | Authenticate users or VMs and deploy per-user ACL before connecting a user to the network. |
Custom routing and online TE. | The ability to route on a variety of business metrics aka routing for dollars. Allowing you to override the default routing behavior. |
Custom traffic processing. | For analytics and encryption. |
Programmable SPAN ports | Use Openflow entries to mirror selected traffic to the SPAN port. |
DoS traffic blackholing & distributed DoS prevention. | Block DoS traffic as close to the source as possible with more selective traffic targeting than the original RTBH approach**. The traffic blocking is implemented in OpenFlow switches. Higher performance with significantly lower costs. |
Traffic redirection and service insertion. | Redirect a subset of traffic to network appliances and install redirection flow entries wherever needed. |
Network Monitoring. | The controller is the authoritative source of information on network topology and Forwarding paths. |
Scale-Out Load Balancing. | Punt new flows to the Openflow controller and install per-session entries throughout the network. |
IPS Scale-Out. | OpenFlow is used to distribute the load to multiple IDS appliances. |
**Remote-Triggered Black Hole: RTBH refers to installing a host route to a bogus IP address ( RTBH address ) pointing to NULL interfaces on all routers. BGP is used to advertise the host routes to other BGP peers of the attacked hosts, with the next hop pointing to the RTBH address, and it is mainly automated in ISP environments.
SDN deployment models
Guidelines:
- Start with small deployments away from the mission-critical production path, i.e., the Core. Ideally, start with device or service provisioning systems.
- Start at the Edge and slowly integrate with the Core. Minimize the risk and blast radius. Start with packet filters at the Edge and tasks that can be easily automated ( VLANs ).
- Integrate new technology with the existing network.
- Gradually increase scale and gain trust. Experience is key.
- Have the controller in a protected out-of-band network with SSL connectivity to the switches.
There are 4 different models for OpenFlow deployment, and the following sections list the key points of each model.
Native OpenFlow
- They are commonly used for Greenfield deployments.
- The controller performs all the intelligent functions.
- The forwarding plane switches have little intelligence and solely perform packet forwarding.
- The white box switches need IP connectivity to the controller for the OpenFlow control sessions. If you are forced to use an in-band network for this communication path using an isolated VLAN with STP, you should use an out-of-band network.
- Fast convergence techniques such as BFD may be challenging to use with a central controller.
- Many people believe that this approach does not work for a regular company. Companies implementing native OpenFlow, such as Google, have the time and resources to reinvent the wheel when implementing a new control-plane protocol ( OpenFlow ).
Native OpenFlow with Extensions
- Some control plane functions are handled from the centralized controller to the forwarding plane switches. For example, the OpenFlow-enabled switches could load balancing across multiple links without the controller’s previous decision. You could also run STP, LACP, or ARP locally on the switch without interaction with the controller. This approach is helpful if you lose connectivity to the controller. If the low-level switches perform certain controller functions, packet forwarding will continue in the event of failure.
- The local switches should support the specific OpenFlow extensions that let them perform functions on the controller’s behalf.
Hybrid ( Ships in the night )
- This approach is used where OpenFlow runs in parallel with the production network.
- The same network box is controlled by existing on-box and off-box control planes ( OpenFlow).
- Suitable for pilot deployment models as switches still run traditional control plane protocols.
- The Openflow controller manages only specific VLANs or ports on the network.
- The big challenge is determining and investigating the conflict-free sharing of forwarding plane resources across multiple control planes.
Integrated OpenFlow
- OpenFlow classifiers and forwarding entries are integrated with the existing control plane. For example, Juniper’s OpenFlow model follows this mode of operation where OpenFlow static routes can be redistributed into the other routing protocols.
- No need for a new control plane.
- No need to replace all forwarding hardware
- It is the most practical approach as long as the vendor supports it.
Closing Points on OpenFlow
OpenFlow is a communication protocol that provides access to the forwarding plane of a network switch or router over the network. It was initially developed at Stanford University and has since been embraced by the Open Networking Foundation (ONF) as a core component of SDN. OpenFlow allows network administrators to program the control plane, enabling them to direct how packets are forwarded through the network. This decoupling of the control and data planes is what empowers SDN to offer more dynamic and flexible network management.
The architecture of OpenFlow is quite straightforward yet powerful. It consists of three main components: the controller, the switch, and the protocol itself. The controller is the brains of the operation, managing network traffic by sending instructions to OpenFlow-enabled switches. These switches, in turn, execute the instructions received, altering the flow of network data accordingly. This setup allows for centralized network control, offering unprecedented levels of automation and agility.
**Advantages of OpenFlow**
OpenFlow brings several critical advantages to network management and control:
1. Flexibility and Programmability: With OpenFlow, network administrators can dynamically reconfigure the behavior of network devices, allowing for greater adaptability to changing network requirements.
2. Centralized Control: By centralizing control in a single controller, network administrators gain a holistic view of the network, simplifying management and troubleshooting processes.
3. Innovation and Experimentation: OpenFlow enables researchers and developers to experiment with new network protocols and applications, fostering innovation in the networking industry.
4. Scalability: OpenFlow’s centralized control architecture provides the scalability needed to manage large-scale networks efficiently.
**Implications for Network Control**
OpenFlow has significant implications for network control, paving the way for new possibilities in network management:
1. Software-Defined Networking (SDN): OpenFlow is a critical component of the broader concept of SDN, which aims to decouple network control from the underlying hardware, providing a more flexible and programmable infrastructure.
2. Network Virtualization: OpenFlow facilitates network virtualization, allowing multiple virtual networks to coexist on a single physical infrastructure.
3. Traffic Engineering: By controlling the flow of packets at a granular level, OpenFlow enables advanced traffic engineering techniques, optimizing network performance and resource utilization.
OpenFlow represents a paradigm shift in network control, offering a more flexible, scalable, and programmable approach to managing networks. By separating the control and data planes, OpenFlow empowers network administrators to have fine-grained control over network behavior, improving efficiency, innovation, and adaptability. As the networking industry continues to evolve, OpenFlow and its related technologies will undoubtedly play a crucial role in shaping the future of network management.
Summary: What is OpenFlow?
In the rapidly evolving world of networking, OpenFlow has emerged as a game-changer. This revolutionary technology has transformed the way networks are managed, offering unprecedented flexibility, control, and efficiency. In this blog post, we will delve into the depths of OpenFlow, exploring its definition, key features, and benefits.
What is OpenFlow?
OpenFlow can be best described as an open standard communications protocol that enables the separation of the control plane and the data plane in network devices. It allows centralized control over a network’s forwarding elements, making it possible to program and manage network traffic dynamically. By decoupling the intelligence of the network from the underlying hardware, OpenFlow provides a flexible and programmable infrastructure for network administrators.
Key Features of OpenFlow
a) Centralized Control: One of the core features of OpenFlow is its ability to centralize network control, allowing administrators to define and implement policies from a single point of control. This centralized control improves network visibility and simplifies management tasks.
b) Programmability: OpenFlow’s programmability empowers network administrators to define how network traffic should be handled based on their specific requirements. Through the use of flow tables and match-action rules, administrators can dynamically control the behavior of network switches and routers.
c) Software-Defined Networking (SDN) Integration: OpenFlow plays a crucial role in the broader concept of Software-Defined Networking. It provides a standardized interface for SDN controllers to communicate with network devices, enabling dynamic and automated network provisioning.
Benefits of OpenFlow
a) Enhanced Network Flexibility: With OpenFlow, network administrators can easily adapt and customize their networks to suit evolving business needs. The ability to modify network behavior on the fly allows for efficient resource allocation and improved network performance.
b) Simplified Network Management: By centralizing network control, OpenFlow simplifies the management of complex network architectures. Policies and configurations can be applied uniformly across the network, reducing administrative overhead and minimizing the chances of configuration errors.
c) Innovation and Experimentation: OpenFlow fosters innovation by providing a platform for the development and deployment of new network protocols and applications. Researchers and developers can experiment with novel networking concepts, paving the way for future advancements in the field.
Conclusion
OpenFlow has ushered in a new era of network management, offering unparalleled flexibility and control. Its ability to separate the control plane from the data plane, coupled with centralized control and programmability, has opened up endless possibilities in network architecture design. As organizations strive for more agile and efficient networks, embracing OpenFlow and its associated technologies will undoubtedly be a wise choice.
What is VXLAN
Network Insight · Data Centre Networking
What Is VXLAN? Understanding Modern Network Overlays
Modern data centres require networks that can scale beyond traditional VLAN boundaries, support virtualised workloads and deliver predictable connectivity across routed infrastructure. Virtual Extensible LAN (VXLAN) addresses these requirements by extending Layer 2 Ethernet segments across a Layer 3 IP network.
VXLAN separates the logical network from the physical transport infrastructure. Instead of relying on a large switched Layer 2 domain, VXLAN encapsulates Ethernet frames inside UDP/IP packets and transports them between VXLAN Tunnel Endpoints (VTEPs). This enables virtual networks to span multiple physical switches while the underlying fabric continues to use IP routing.
Each VXLAN segment is identified by a 24-bit VXLAN Network Identifier (VNI) . Compared with traditional 12-bit VLAN identifiers, VXLAN provides a substantially larger segmentation space, making it particularly useful for multi-tenant cloud platforms, software-defined data centres and large-scale virtualised environments.
24-bit
VXLAN Network Identifier used to
distinguish logical overlay segments
~16.7M
Potential VNI values compared with
approximately 4,094 usable VLAN IDs
UDP 4789
Standard VXLAN destination port
for Ethernet encapsulation over IP
Architecture Preview · Layer 2 over Layer 3
Leaf A
VTEP 10.0.0.11
VLAN 10 → VNI 10010
VXLAN Overlay
UDP/IP Encapsulation
Routed IP Underlay
Leaf B
VTEP 10.0.0.12
VLAN 10 → VNI 10010
Two VTEPs extend the same logical Layer 2
segment across a routed IP fabric.
Intermediate routers forward the outer IP
packet without inspecting the inner Ethernet frame.
Why VXLAN Matters in Modern Data Centres
Traditional Ethernet networks face practical limitations when stretched across large data centre environments. VLAN identifiers provide a limited segmentation space, large broadcast domains increase operational complexity, and Spanning Tree Protocol can prevent redundant Layer 2 links from forwarding simultaneously.
VXLAN enables a different architectural model. A routed leaf-spine underlay can use Equal-Cost Multi-Path (ECMP) forwarding across multiple active paths, while the VXLAN overlay provides logical Layer 2 connectivity independently of the physical topology. When combined with BGP Ethernet VPN (EVPN), the fabric can distribute MAC and IP reachability information through a control plane, reducing reliance on data-plane flooding.
What You Will Learn
✓
Understand why VXLAN was introduced
and how it addresses traditional
VLAN scaling limitations.
✓
Explore the relationship between
the IP underlay, VXLAN overlay,
VTEPs and VNIs.
✓
Follow Ethernet frames through
VXLAN encapsulation, routed
forwarding and decapsulation.
✓
Compare flood-and-learn forwarding
with BGP EVPN MAC and IP
reachability distribution.
✓
Investigate tenant segmentation,
inter-VNI routing and distributed
gateway design.
✓
Diagnose common VXLAN failures
using VTEP reachability, VNI
mappings, EVPN state and telemetry.
From Architecture to Engineering Practice
This guide combines technical explanations with interactive network engineering exercises. You will explore VXLAN topology behaviour, examine encapsulated packets, compare forwarding mechanisms and investigate realistic overlay and underlay failures. The objective is not simply to understand what VXLAN is, but to understand how it behaves in a production data centre fabric.
Section 01 · VXLAN Fundamentals
Why VXLAN? Understanding the Limitations of Traditional Layer 2 Networks
Modern data centre networks must support thousands of virtual machines, multiple tenants, dynamic workload placement and high-bandwidth east-west traffic. Traditional Ethernet switching was not originally designed for this degree of scale, segmentation and mobility.
Conventional Layer 2 architectures rely on VLANs, MAC address learning and loop-prevention mechanisms such as Spanning Tree Protocol (STP). These technologies remain useful, but extending large Layer 2 domains across a data centre can introduce operational complexity, inefficient use of redundant links and limitations in the number of logical segments.
VXLAN was introduced to separate logical Layer 2 connectivity from the physical network topology. By encapsulating Ethernet frames inside UDP/IP packets, VXLAN allows a routed IP fabric to transport multiple isolated virtual networks without requiring the entire physical infrastructure to participate in the same Layer 2 broadcast domain.
VLAN Scaling
Spanning Tree
MAC Learning
Server Virtualization
Leaf-Spine Fabrics
Network Overlays
1.1 — The Traditional Layer 2 Data Centre
In a traditional switched network, Ethernet frames are forwarded using destination MAC addresses. Switches learn source MAC addresses from incoming frames and maintain forwarding tables that associate MAC addresses with physical interfaces.
VLANs provide logical separation within this infrastructure. Each VLAN represents an independent Layer 2 broadcast domain, allowing different applications, departments or tenants to share physical switching hardware while maintaining separate Ethernet segments.
The challenge emerges when these networks grow. More servers create more MAC addresses, more applications require additional VLANs, and redundant physical links require careful loop prevention. Workload mobility can also require Layer 2 connectivity across multiple racks or physical locations.
01
Limited Segmentation Space
Traditional IEEE 802.1Q VLAN tags use a 12-bit VLAN identifier, providing 4,096 possible values, of which approximately 4,094 are available for normal VLAN use.
02
Spanning Tree Constraints
STP prevents Ethernet forwarding loops by selecting a loop-free active topology. Some redundant links may be blocked, preventing simultaneous forwarding.
03
MAC Address Table Growth
Virtualized servers can introduce many MAC addresses per physical switch port. Large Layer 2 domains increase forwarding table requirements and operational scale.
04
Workload Mobility
Moving workloads between physical hosts may require preserving network connectivity, addressing and policy across different switching locations.
Engineering Principle
The problem is not that Ethernet switching or VLANs are inherently unsuitable. The challenge is extending large numbers of logical Layer 2 segments across a growing physical network while preserving scalability, resilience and operational simplicity.
1.2 — VLAN Identifier Exhaustion
IEEE 802.1Q uses a 12-bit VLAN Identifier (VID). This provides 212, or 4,096, possible identifier values. VLAN IDs 0 and 4095 are reserved, leaving 4,094 usable VLAN identifiers in the normal VLAN namespace.
For a small enterprise, this may be sufficient. However, large cloud providers and multi-tenant data centres may require thousands of independent network segments for customers, applications, security zones and workload environments.
Logical Network Identifier Capacity
Traditional VLAN — 12 bits
4,096 values
VXLAN VNI — 24 bits
16,777,216 values
Visual comparison uses a compressed scale
so both identifier spaces remain visible.
VXLAN provides 4,096 times the theoretical
identifier space of a 12-bit VLAN field.
Actual supported VNI scale depends on
platform hardware and software limits.
VXLAN introduces a 24-bit VXLAN Network Identifier (VNI). This significantly expands the available logical segment namespace. A VNI identifies an overlay segment, allowing different physical VLANs to map into the same logical VXLAN segment when required.
For example, VLAN 10 on one leaf switch and VLAN 20 on another leaf switch can both map to VNI 10010. The local VLAN numbers do not need to match, provided the VTEPs are configured to associate those VLANs with the same intended overlay bridge domain.
Engineering Example · Multi-Tenant Cloud
Consider a cloud provider operating infrastructure for 500 customers. Each customer requires eight isolated application networks for development, production, management, databases and supporting services.
500
Customers
8
Segments per Customer
4,000
Required Segments
This deployment already consumes almost the entire traditional VLAN identifier space if every segment requires a unique VLAN ID in one shared administrative namespace.
Adding more customers, environments or application tiers creates further demand. VXLAN allows the provider to allocate distinct VNIs without relying on a single global 12-bit VLAN namespace.
Important Design Distinction
VLAN IDs can be reused in separate Layer 2 domains, so the 4,094-VLAN limit is not a universal limit on the number of tenants in an entire network. VXLAN is valuable because it provides a much larger overlay namespace and decouples logical segmentation from local physical VLAN assignments.
1.3 — Spanning Tree and Blocked Links
Ethernet networks must prevent forwarding loops. Unlike IP packets, ordinary Ethernet frames do not contain a hop-limit field that automatically expires as frames traverse switches.
A Layer 2 forwarding loop can cause broadcast storms, repeated frame replication, unstable MAC learning and severe network disruption. Spanning Tree Protocol prevents these conditions by selecting a loop-free forwarding topology.
Although this provides stability, a conventional spanning-tree topology may block redundant links. Consequently, installed physical bandwidth is not always available for active forwarding. Convergence following topology changes can also affect application connectivity.
Traditional Layer 2 Design
Redundant Ethernet paths require loop
prevention and may include blocked links.
Distribution A
Distribution B
Access A
Access B
Active forwarding path
Illustrative blocked redundant path
Illustrative blocked redundant path
Routed Leaf-Spine Fabric
IP routing enables multiple equal-cost
paths to forward traffic concurrently.
Spine 1
Spine 2
Leaf A
Leaf B
Routed links can remain active
ECMP distributes eligible flows across available next hops.
ECMP distributes eligible flows across available next hops.
Modern leaf-spine architectures commonly replace the large switched core with a routed IP underlay. Each leaf connects to multiple spine switches, and routing protocols maintain reachability across the fabric.
Equal-Cost Multi-Path (ECMP) forwarding allows traffic to use multiple equal-cost routes. Rather than relying on a spanning tree to disable redundant Layer 2 paths, the fabric uses Layer 3 routing and its associated convergence mechanisms.
VXLAN complements this architecture by transporting logical Layer 2 segments over the routed underlay. The physical fabric can therefore remain Layer 3 while connected workloads continue to experience the Layer 2 adjacency required by their applications.
Architectural Advantage
VXLAN does not itself perform ECMP routing. ECMP is provided by the IP underlay. VXLAN enables Layer 2 overlay connectivity to use that routed infrastructure. The VXLAN UDP source port can provide entropy for flow-based ECMP hashing.
1.4 — Server Virtualization and MAC Scale
Server virtualization changes the relationship between physical switch ports and network endpoints. A single physical server can host many virtual machines, each with its own virtual network interface and MAC address.
In a conventional Ethernet design, switches learn the MAC addresses of communicating endpoints and store them in forwarding tables. As virtual machine populations increase, the number of learned addresses can grow substantially.
Large Layer 2 domains also require mechanisms for locating unknown destinations. When a destination MAC address is not known, unknown-unicast frames may be flooded within the relevant broadcast domain.
| Scaling Factor | Traditional Layer 2 Impact | VXLAN Fabric Approach |
|---|---|---|
| More virtual machines | Increasing MAC address learning and forwarding-table requirements | Logical segments distributed across VTEPs; EVPN can advertise remote MAC reachability |
| More application networks | Greater demand for VLAN identifiers and trunk configuration | 24-bit VNI namespace supports many overlay segments |
| Workload relocation | MAC location changes and possible VLAN extension requirements | Remote VTEP mappings can be updated through the overlay control plane |
| Unknown destinations | Flooding within the Layer 2 broadcast domain | BUM replication is still required in some cases; EVPN can reduce unnecessary flooding |
| Redundant fabric links | STP may block selected Layer 2 paths | Routed underlay supports ECMP across active IP paths |
VXLAN does not automatically eliminate MAC table requirements. VTEPs still need forwarding information for local and remote endpoints. However, the overlay architecture allows logical segments to be distributed across a routed fabric.
When combined with BGP EVPN, MAC and IP reachability information can be exchanged through a control plane. This provides more structured endpoint discovery, supports MAC mobility and can reduce reliance on traditional flood-and-learn forwarding.
1.5 — Why Workload Mobility Matters
Virtualization allows workloads to move between physical hosts for maintenance, resource optimisation, resilience and operational flexibility.
In some deployment models, a workload must preserve its IP address and Layer 2 connectivity when it moves. Traditional approaches may require VLANs to be extended across multiple racks or switching domains.
VXLAN enables the same logical overlay segment to be present on different physical leaf switches. A workload attached to VNI 10010 on Leaf A can communicate with another endpoint attached to VNI 10010 on Leaf B, even though the underlying network forwards traffic using IP routing.
In EVPN-based fabrics, control-plane updates can advertise changes in endpoint location. Remote VTEPs can then update their forwarding state to direct traffic toward the appropriate destination VTEP.
Design Consideration
VXLAN provides the network mechanism for extending logical segments, but successful workload mobility also depends on application requirements, gateway design, security policy, address management and orchestration. Extending Layer 2 connectivity across distant sites is not always the preferred design.
1.6 — The Transition to Routed Overlay Fabrics
The evolution from traditional Layer 2 networks to VXLAN-based fabrics represents an important architectural shift. Instead of requiring the physical network to provide every logical Ethernet segment, the network separates two responsibilities:
L3
The Underlay Network
Provides IP connectivity between VTEPs. Routing protocols maintain reachability, ECMP enables path diversity, and the physical topology can scale independently of tenant segmentation.
L2
The VXLAN Overlay
Provides logical Ethernet segments identified by VNIs. VTEPs encapsulate and decapsulate traffic, while mechanisms such as BGP EVPN can distribute endpoint reachability.
This separation enables data centre architects to design a resilient routed fabric while delivering logical connectivity to workloads independently of their physical attachment points.
It also introduces new engineering responsibilities. The underlay must provide reliable VTEP reachability, the overlay must maintain correct endpoint mappings, and the fabric must account for encapsulation overhead, MTU, segmentation and control-plane convergence.
| Design Dimension | Traditional Approach | VXLAN-Based Fabric |
|---|---|---|
| Network segmentation | 12-bit VLAN identifiers | 24-bit VXLAN VNIs |
| Core transport | Often extended Layer 2 switching | Routed IP underlay |
| Path redundancy | STP or other L2 redundancy designs | IP routing and ECMP |
| Endpoint learning | Data-plane MAC learning | Flood-and-learn or EVPN control-plane distribution |
| Workload mobility | May require extended VLAN domains | Logical overlay segment extension |
| Operational visibility | Switch ports, VLANs and MAC tables | Underlay routes, VTEPs, VNIs, EVPN state and packet encapsulation |
Key Engineering Insight
VXLAN does not replace Ethernet, VLANs or IP routing. It combines them into an overlay architecture in which Ethernet segments are mapped to VNIs and transported across an IP network. Local VLANs still commonly connect endpoints to VTEPs, while the routed underlay transports encapsulated traffic.
1.7 — Design Trade-Offs and Operational Reality
VXLAN solves important scaling and architectural challenges, but it is not a universal replacement for traditional Layer 2 networking.
Overlay fabrics introduce additional headers, forwarding state and control-plane dependencies. Troubleshooting must distinguish between failures in the physical IP underlay and failures in the logical VXLAN overlay.
Engineers must also consider hardware forwarding capacity, VNI scale, EVPN route scale, broadcast replication, security policy, operational tooling and the ability to observe encapsulated traffic across the network.
| VXLAN Advantage | Engineering Responsibility |
|---|---|
| Large logical segment space | VNI allocation, lifecycle management and tenant isolation |
| Routed underlay transport | IP reachability, routing convergence and ECMP path health |
| Flexible workload placement | MAC mobility, gateway consistency and policy enforcement |
| Overlay encapsulation | MTU planning, packet visibility and forwarding performance |
| EVPN control-plane learning | BGP session health, route import/export and endpoint state consistency |
VXLAN also provides segmentation rather than encryption. Security still requires appropriate access controls, isolation policies and protection of the transport infrastructure.
Section 1 — Engineering Takeaways
✓
Traditional VLANs use a 12-bit identifier,
while VXLAN introduces a 24-bit VNI
for a much larger logical segment space.
✓
Large Layer 2 domains can create
challenges involving STP, MAC
learning, flooding and workload mobility.
✓
Routed leaf-spine fabrics use IP
routing and ECMP to provide active
path diversity without relying on
STP in the physical fabric.
✓
VXLAN transports logical Ethernet
segments across the IP underlay,
separating tenant connectivity
from physical topology.
✓
VXLAN does not eliminate the need
for MAC learning, MTU planning,
security or careful control-plane
design.
Network Insight · Interactive Engineering Lab 01
Why VXLAN? Explore the Limits of Layer 2
Compare a traditional VLAN-based network with a routed VXLAN overlay. Select an engineering challenge, change the architecture and run a packet journey to see what changes.
Step 1 · Select Architecture
Engineering Focus
Live Network Architecture
Traditional Layer 2
Active physical path
STP-blocked alternative
VXLAN logical tunnel
Ready to investigate
Engineering Analysis
VLAN Scaling in a Traditional Network
Traditional VLAN identifiers use
a 12-bit field, providing
4094 normally usable VLAN IDs.
Extending large numbers of VLANs
across a physical switching
fabric increases operational
complexity and Layer 2 scope.
Architecture Indicators
Segment Identifier
12-bit
Segment ID Space
4,094
Core Forwarding
Layer 2
Path Model
STP
Engineering Event Timeline
Select an architecture and scenario.
Run the packet journey to inspect forwarding events.
Key Engineering Principle
VXLAN separates logical Layer 2 segments from the physical transport network, enabling Ethernet overlays to operate across a routed IP fabric.
Section 02 · VXLAN Architecture
VXLAN Architecture: Understanding the Underlay, Overlay and VTEPs
VXLAN architecture separates the physical network responsible for IP packet transport from the logical network that provides Ethernet connectivity to applications. This separation is the foundation of modern routed data centre fabrics.
In a VXLAN environment, the underlay network provides routed IP connectivity between tunnel endpoints, while the overlay network represents the virtual Layer 2 segments transported across that infrastructure. VXLAN Tunnel Endpoints (VTEPs) connect these two architectural layers.
Understanding the relationship between the underlay, overlay, VTEPs and VXLAN Network Identifiers (VNIs) is essential for designing, operating and troubleshooting VXLAN fabrics. A failure in any of these components can interrupt workload connectivity even when other parts of the network appear healthy.
IP Underlay
VXLAN Overlay
VTEP
VNI
Leaf-Spine
ECMP
BGP EVPN
2.1 — The Three Architectural Layers
A VXLAN fabric can be understood through three primary architectural components: the physical IP underlay, the logical VXLAN overlay and the VTEPs that perform encapsulation and decapsulation.
TRANSPORT
IP Underlay Network
The routed physical network connecting leaf and spine switches. It provides reachability between VTEP IP addresses and forwards outer IP packets using normal Layer 3 routing.
VIRTUAL NETWORK
VXLAN Overlay Network
Logical Ethernet segments identified by VNIs. The overlay allows workloads connected to different physical leaf switches to participate in the same virtual Layer 2 network.
ENCAPSULATION
VXLAN Tunnel Endpoint
A VTEP encapsulates local Ethernet frames into VXLAN UDP/IP packets and decapsulates incoming VXLAN traffic before forwarding the inner frames toward connected endpoints.
SEGMENT IDENTIFIER
VXLAN Network Identifier
A 24-bit VNI identifies the logical overlay segment associated with encapsulated traffic. VNIs allow many isolated virtual networks to share the same routed fabric.
Architectural Principle
The underlay forwards packets using outer IP addresses. The overlay identifies the logical Ethernet segment using the VNI. The VTEP connects both functions by adding or removing the VXLAN encapsulation.
2.2 — VXLAN Leaf-Spine Fabric Architecture
Modern VXLAN deployments commonly use a leaf-spine architecture. Each leaf switch connects to multiple spine switches, creating a routed topology with predictable path diversity and multiple equal-cost forwarding options.
Leaf switches frequently act as VTEPs, connecting local servers or virtual machines to the overlay. Spine switches provide IP transport between leaf VTEPs and do not normally need to inspect the VXLAN headers or tenant Ethernet frames.
Reference Architecture · Two-Spine / Three-Leaf Fabric
Routed IP underlay
Logical VXLAN overlay
The dashed VXLAN path represents a logical
overlay relationship between VTEPs.
Actual encapsulated packets traverse the
routed spine-leaf underlay.
Leaf C may participate in other VNIs
or the same VNI if configured.
The leaf-spine topology is particularly suitable for VXLAN because it provides consistent routed connectivity between VTEPs and can distribute traffic across multiple available paths.
The physical network does not need to extend tenant VLANs through the spines. Instead, each leaf encapsulates local traffic using VXLAN and sends the resulting outer IP packet toward the remote VTEP.
2.3 — The IP Underlay Network
The underlay is responsible for transporting encapsulated VXLAN packets between VTEPs. It must provide reliable IP connectivity between the VTEP source and destination addresses.
In many deployments, each VTEP uses a loopback interface as its VXLAN source address. The underlay routing system advertises reachability to these loopback addresses so that remote VTEPs can forward encapsulated traffic correctly.
Common underlay routing approaches include OSPF, IS-IS and eBGP. The appropriate protocol depends on the fabric architecture, operational requirements and platform capabilities.
| Underlay Component | Engineering Role | Operational Check |
|---|---|---|
| Physical links | Provide connectivity between leaf and spine switches | Interface status, errors, speed and utilization |
| Routing protocol | Advertises reachability to VTEP addresses | OSPF, IS-IS or BGP neighbor and route state |
| VTEP loopback | Stable source or destination address for VXLAN packets | Route lookup and loopback reachability |
| ECMP | Distributes eligible traffic across equal-cost paths | Installed next hops and flow distribution |
| MTU | Accommodates additional VXLAN encapsulation overhead | End-to-end packet size and fragmentation behavior |
Underlay Failure Scenario
If Leaf A cannot route traffic to Leaf B's VTEP address, the VXLAN overlay cannot deliver encapsulated packets between them. The VNI configuration may be correct, but the data plane will still fail because the underlay lacks usable IP reachability.
2.4 — The VXLAN Overlay Network
The overlay represents the logical networks carried across the IP underlay. Each Layer 2 VXLAN segment is identified by a VNI, and participating VTEPs associate local attachment circuits or VLANs with that identifier.
Consider two servers connected to different leaf switches. Both servers belong to the same logical network, represented by VNI 10010. The servers can exchange Ethernet frames even though the physical switches between them operate as Layer 3 routers.
VXLAN encapsulation allows the original Ethernet frame to remain intact as it crosses the underlay. The receiving VTEP removes the outer encapsulation and forwards the inner frame within the appropriate local bridge domain.
LEAF A
Local VLAN 10
Host A connects to VLAN 10. Leaf A maps this VLAN to VNI 10010 and encapsulates traffic destined for remote endpoints in that overlay segment.
LEAF B
Local VLAN 20
Host B connects to VLAN 20. Leaf B maps VLAN 20 to the same VNI 10010, placing both endpoints in the intended logical Layer 2 segment.
VLAN-to-VNI Mapping
Local VLAN identifiers do not have to match across VTEPs. The VNI identifies the shared overlay bridge domain. However, the mappings must be consistent with the intended tenant segmentation and control-plane configuration.
2.5 — The Role of the VTEP
A VXLAN Tunnel Endpoint is the device or software component responsible for converting Ethernet traffic into VXLAN packets and back again.
VTEPs may exist in physical switches, virtual switches, hypervisors or other supported network devices. In a common hardware leaf-spine design, the leaf switches act as VTEPs.
| VTEP Function | Description |
|---|---|
| Encapsulation | Adds VXLAN, UDP, IP and outer Ethernet headers to an original Ethernet frame |
| Decapsulation | Removes the outer headers and restores the original Ethernet frame for local forwarding |
| VNI mapping | Associates local bridge domains or VLANs with VXLAN segment identifiers |
| Remote endpoint forwarding | Determines the destination VTEP for known remote MAC addresses |
| BUM replication | Handles broadcast, unknown unicast and multicast traffic using the configured replication mechanism |
| EVPN integration | Uses advertised MAC/IP reachability information when BGP EVPN is deployed |
A VTEP needs to determine both the logical segment associated with a frame and the remote VTEP that should receive it. The method used to discover remote endpoints depends on whether the fabric uses flood-and-learn or a control plane such as BGP EVPN.
2.6 — Example VTEP Configuration
The following simplified Cisco NX-OS-style example illustrates the relationship between a local VLAN, VNI and NVE interface. The example is conceptual and may require additional platform- specific commands, feature enablement and control-plane configuration.
Leaf A · Illustrative VXLAN Configuration
NX-OS Style
feature nv overlay
vlan 10
vn-segment 10010
interface loopback1
ip address 10.0.0.11/32
interface nve1
no shutdown
host-reachability protocol bgp
source-interface loopback1
member vni 10010
ingress-replication protocol bgp
Illustrative syntax only. A functional
EVPN fabric also requires underlay
reachability, BGP EVPN configuration,
appropriate route targets and other
platform-dependent settings.
In this example, VLAN 10 maps to VNI 10010. Loopback1 provides the local VTEP source address, and interface NVE1 represents the VXLAN tunnel interface.
The BGP EVPN control plane can advertise reachability information and support ingress replication membership. However, the exact configuration model varies between hardware platforms, operating systems and software versions.
2.7 — Packet Forwarding Across the Fabric
When a local endpoint sends an Ethernet frame to a remote endpoint in the same VNI, the ingress VTEP encapsulates the frame and forwards it across the IP underlay.
01
Host A
Sends an ordinary
Ethernet frame
02
Leaf A
Maps VLAN to VNI
and encapsulates
03
Underlay
Routes outer IP
packet toward VTEP
04
Leaf B
Decapsulates and
selects bridge domain
05
Host B
Receives original
Ethernet frame
The original source and destination MAC addresses remain part of the inner Ethernet frame. The outer IP header identifies the source and destination VTEPs.
Intermediate spine routers forward the packet based on the outer destination IP address. They do not need to learn tenant MAC addresses or participate in the tenant Layer 2 bridge domain.
Packet Forwarding Insight
The VXLAN tunnel is a logical relationship between VTEPs. It is not necessarily a dedicated physical path. Encapsulated packets may traverse different equal-cost underlay routes depending on the ECMP hashing algorithm and network state.
2.8 — ECMP and Underlay Path Diversity
One of the major advantages of a routed leaf-spine underlay is the ability to use multiple equal-cost paths simultaneously.
VXLAN uses UDP encapsulation, allowing the underlay to distinguish flows through outer header fields. Many implementations derive the UDP source port from a hash of inner packet information to provide entropy for ECMP.
This enables different encapsulated flows to be distributed across available underlay next hops. However, ECMP is commonly flow-based, so a single flow may continue to use one selected path rather than being distributed packet by packet.
NORMAL OPERATION
Multiple Active Paths
Leaf A can reach Leaf B through Spine 1 or Spine 2. The underlay may install both equal-cost next hops and distribute eligible VXLAN flows across them.
LINK FAILURE
Underlay Reconvergence
If one path fails, the routing system removes the unavailable next hop. Traffic can continue over remaining paths, subject to detection, convergence and forwarding updates.
2.9 — Where BGP EVPN Fits
VXLAN defines a data-plane encapsulation mechanism. It does not, by itself, define a complete mechanism for distributing remote endpoint reachability throughout the fabric.
Early VXLAN deployments commonly relied on data-plane learning and multicast-based replication. Modern data centre fabrics frequently combine VXLAN with BGP Ethernet VPN (EVPN) to provide a scalable control-plane mechanism.
EVPN can advertise MAC/IP reachability, identify participating VTEPs and support endpoint mobility. It also enables features such as distributed gateway architectures and inter-subnet routing.
| Component | Responsibility |
|---|---|
| VXLAN | Encapsulates Ethernet traffic inside UDP/IP packets |
| IP Underlay | Provides routed reachability between VTEP addresses |
| BGP EVPN | Distributes endpoint reachability and overlay control-plane information |
| VTEP | Performs encapsulation, decapsulation and overlay forwarding |
| VNI | Identifies the logical overlay segment or routing context |
VXLAN and EVPN Are Complementary
VXLAN is the data-plane encapsulation technology. BGP EVPN is a control-plane technology used to distribute overlay reachability information. They are commonly deployed together, but they perform different architectural roles.
2.10 — VXLAN Architecture Design Requirements
A successful VXLAN fabric depends on more than enabling an NVE interface and assigning a VNI. The underlay and overlay must be designed as related but independently observable systems.
| Requirement | Why It Matters |
|---|---|
| VTEP IP reachability | Remote tunnel endpoints must be reachable through the routed underlay |
| Consistent VNI mappings | Logical segments must map to the intended bridge domains |
| Appropriate MTU | Encapsulation adds header overhead beyond the original frame size |
| Control-plane stability | EVPN routes and endpoint mappings must remain accurate during changes |
| Resilient underlay | Routing and ECMP must recover from physical path failures |
| Tenant isolation | VNIs, VRFs and security policy must prevent unintended connectivity |
| Operational visibility | Engineers need to inspect routing, NVE peers, VNIs, EVPN routes and forwarding state |
Troubleshooting should begin by determining whether the fault exists in the underlay, overlay or endpoint attachment configuration.
For example, a missing VTEP route indicates an underlay reachability problem, while a VNI mismatch or missing EVPN route may indicate an overlay configuration or control-plane problem.
Operational Design Warning
A healthy BGP EVPN session does not guarantee successful VXLAN forwarding. The fabric must also have correct VNI membership, valid remote endpoint mappings, usable VTEP routes, sufficient MTU and appropriate forwarding policies.
Section 2 — Engineering Takeaways
✓
The IP underlay provides
routed transport between
VXLAN tunnel endpoints.
✓
The VXLAN overlay provides
logical Ethernet segments
identified by 24-bit VNIs.
✓
VTEPs encapsulate and
decapsulate Ethernet frames
as they enter and leave
the routed fabric.
✓
ECMP enables eligible
encapsulated flows to
use multiple active
underlay paths.
✓
BGP EVPN complements
VXLAN by distributing
endpoint and overlay
reachability information.
✓
VXLAN troubleshooting
requires visibility into
both the routed underlay
and the logical overlay.
Network Insight · Interactive Engineering Lab 02
VXLAN Architecture Explorer: Underlay, Overlay and VTEPs
Investigate a two-leaf, two-spine VXLAN fabric. Select individual devices to inspect their roles, switch between physical and logical network views, and trace the journey of an Ethernet frame across the IP underlay.
Step 1 · Select Architecture View
Interactive Leaf-Spine Architecture
IP Underlay
Routed IP links
Logical VXLAN tunnel
Select any device to inspect it
Selected Network Component
Leaf A · VXLAN Tunnel Endpoint
Leaf A is a VTEP. It receives Ethernet frames from Host A, maps the local VLAN to a VNI and encapsulates frames for remote VTEPs.
Component Details
Role
VTEP
Address
10.0.0.11
Layer
Overlay Edge
Function
Encapsulation
Select a device or run a trace
Packet Forwarding Timeline
Select a topology component to inspect it.
Run the trace to see encapsulation and forwarding.
Key Architecture Principle
The underlay transports IP packets between VTEPs. The VXLAN overlay provides logical Ethernet connectivity between workloads. The spines forward on outer IP headers and do not normally need to inspect the inner Ethernet frame.
Conceptual VXLAN architecture lab.
The example illustrates same-subnet
forwarding over one selected spine path.
It does not emulate BGP EVPN,
routing convergence or ECMP hashing.
Section 03 · VXLAN Data Plane
VXLAN Encapsulation and Packet Forwarding
VXLAN enables an Ethernet frame to cross a routed IP network without requiring intermediate routers to participate in the original Layer 2 broadcast domain. It achieves this by encapsulating the Ethernet frame inside a VXLAN header, UDP datagram and outer IP packet.
The original Ethernet frame remains the inner frame, while the newly added transport headers identify the source and destination VXLAN Tunnel Endpoints (VTEPs). This separation allows the physical network to forward traffic using ordinary IP routing while the overlay preserves logical Ethernet connectivity between workloads.
Understanding the exact encapsulation format is essential for troubleshooting VXLAN fabrics. Engineers must distinguish the inner and outer addressing information, verify the correct VXLAN Network Identifier (VNI), account for encapsulation overhead and confirm that the receiving VTEP can decapsulate and forward the original frame.
Ethernet Frame
VXLAN Header
UDP 4789
VNI
VTEP
MTU
ECMP Hashing
3.1 — What Happens During VXLAN Encapsulation?
When Host A sends an Ethernet frame toward Host B in the same logical VXLAN segment, the ingress VTEP first determines the bridge domain and VNI associated with the incoming frame.
If the destination MAC address is reachable through a remote VTEP, the ingress device encapsulates the original Ethernet frame with a VXLAN header and additional UDP/IP transport headers.
The resulting packet is forwarded through the routed underlay toward the destination VTEP. Intermediate routers inspect the outer IP destination address and do not need to understand the tenant's inner Ethernet addressing.
INNER NETWORK
Original Ethernet Frame
Contains the original source and destination MAC addresses, optional VLAN information and the payload generated by the communicating hosts.
OUTER NETWORK
VXLAN Transport Headers
Include the VXLAN VNI, UDP header, outer source and destination IP addresses and the Ethernet header used on the current physical link.
Fundamental Data-Plane Principle
The inner Ethernet frame identifies the communicating endpoints. The outer IP header identifies the VTEPs transporting that frame. The VNI identifies the logical overlay segment in which the frame belongs.
3.2 — The Complete VXLAN Packet Structure
A conventional VXLAN packet consists of an original Ethernet frame encapsulated inside an 8-byte VXLAN header, an 8-byte UDP header and an outer IP packet. An outer Ethernet header is also used when the encapsulated packet is transmitted across an Ethernet underlay link.
Packet Anatomy · Ethernet over VXLAN/UDP/IPv4
Outer Ethernet
Physical next-hop MAC addresses ·
14 bytes without an 802.1Q tag
Outer IPv4 Header
Source VTEP: 10.0.0.11 ·
Destination VTEP: 10.0.0.12 ·
20 bytes without IP options
UDP Header
Variable source port ·
Destination port 4789 · 8 bytes
VXLAN Header
VNI 10010
8-byte header identifying
the logical overlay segment
Inner Ethernet
Original source and destination
host MAC addresses · 14 bytes
without an inner VLAN tag
Inner Payload
Original IPv4/IPv6 packet,
ARP message or other
Ethernet payload
Packet layers are shown from the outer
Ethernet header toward the encapsulated
payload. The Ethernet FCS, optional VLAN
tags and additional IP options are not
included in this simplified diagram.
The outer Ethernet header changes as the encapsulated packet moves between physical Layer 3 hops. By contrast, the outer source and destination VTEP IP addresses normally remain constant across the underlay path.
The inner Ethernet frame is preserved through the VXLAN tunnel, allowing the destination VTEP to deliver it into the corresponding local bridge domain after decapsulation.
3.3 — Understanding the 8-Byte VXLAN Header
The standard VXLAN header defined by RFC 7348 is eight bytes long. Its most important field is the 24-bit VXLAN Network Identifier, which identifies the logical network segment associated with the encapsulated frame.
The header also contains flags and reserved fields. The I flag indicates that the VNI field is valid. In the original VXLAN format, the remaining reserved bits are transmitted as zero.
VXLAN Header · 8 Bytes / 64 Bits
Byte 0
Flags
Byte 1
RSVD
Byte 2
RSVD
Byte 3
RSVD
Byte 4
VNI
Byte 5
VNI
Byte 6
VNI
Byte 7
RSVD
The VNI occupies bytes 4–6.
The I flag is bit 3 of the first
byte, conventionally represented
by the flag value 0x08.
This diagram represents the
original VXLAN header, not VXLAN-GPE.
| Field | Size | Purpose |
|---|---|---|
| Flags | 8 bits | Includes the I flag indicating that the VNI is valid |
| Reserved | 24 bits | Reserved fields in the original VXLAN header |
| VNI | 24 bits | Identifies the logical VXLAN segment |
| Reserved | 8 bits | Reserved final byte |
Why the VNI Matters
A VTEP uses the VNI to select the intended overlay bridge domain. If the receiving VTEP does not have the appropriate VNI configured, it cannot correctly deliver the frame into that logical segment.
3.4 — Inner and Outer Addressing
VXLAN packets contain two addressing contexts. The inner headers describe the communicating workloads, while the outer headers describe how the encapsulated packet travels through the physical IP fabric.
| Address Field | Example | Function |
|---|---|---|
| Inner source MAC | 00:11:22:33:44:01 | Original Host A MAC |
| Inner destination MAC | 00:11:22:33:44:02 | Original Host B MAC |
| Inner source IP | 192.168.10.11 | Host A IP address |
| Inner destination IP | 192.168.10.12 | Host B IP address |
| Outer source IP | 10.0.0.11 | Leaf A VTEP address |
| Outer destination IP | 10.0.0.12 | Leaf B VTEP address |
| VXLAN VNI | 10010 | Overlay bridge domain |
Spine routers use the outer destination IP address to forward the encapsulated packet. They do not need routes for the inner host IP addresses when simply providing underlay transport.
The receiving VTEP uses the VXLAN header and local forwarding information to identify the correct bridge domain and deliver the inner frame.
3.5 — End-to-End VXLAN Packet Journey
Consider Host A connected to Leaf A and Host B connected to Leaf B. Both hosts belong to VNI 10010, and the underlay provides IP connectivity between the two VTEPs.
01
Host A
Sends an Ethernet
frame toward Host B
02
Leaf A
Selects VNI 10010
and encapsulates
03
Spine
Routes the outer
IP packet
04
Leaf B
Removes VXLAN
encapsulation
05
Host B
Receives the
original frame
Step 1 — Original Ethernet Frame
Host A creates an Ethernet frame containing Host B's destination MAC address. The frame enters Leaf A through a local access interface associated with the intended bridge domain.
Step 2 — VNI and Remote VTEP Selection
Leaf A identifies the local bridge domain and associated VNI. Its forwarding information indicates that Host B's MAC address is reachable through Leaf B's VTEP.
This remote mapping may have been learned through flood-and-learn forwarding or advertised by BGP EVPN.
Step 3 — Encapsulation and IP Forwarding
Leaf A adds the VXLAN, UDP and outer IP headers. The outer destination becomes Leaf B's VTEP address, 10.0.0.12.
The underlay routing table determines the next hop. If multiple equal-cost paths exist, ECMP hashing may select one of them.
Step 4 — Decapsulation at Leaf B
Leaf B receives the VXLAN packet, validates the relevant encapsulation information and identifies VNI 10010.
It removes the outer headers and processes the original Ethernet frame in the associated local bridge domain.
Step 5 — Delivery to Host B
Leaf B forwards the recovered Ethernet frame toward Host B through the appropriate local attachment interface.
Forwarding Assumption
This example assumes that the destination MAC-to-VTEP mapping is already known. If the destination is unknown, the fabric may need to perform BUM replication or another endpoint discovery operation. These mechanisms are covered in Section 4.
3.6 — VXLAN UDP Ports and ECMP Entropy
Standard VXLAN uses UDP destination port 4789. The UDP source port is commonly derived from a hash of the inner packet fields.
This variable source port provides additional entropy for underlay ECMP hashing. Without suitable entropy, many VXLAN flows between the same pair of VTEPs could hash onto the same physical path.
UDP DESTINATION
Port 4789
The standard VXLAN destination port. Receiving VTEPs use the configured port to identify VXLAN traffic for processing.
UDP SOURCE
Flow-Based Entropy
A source port chosen using a hash of inner traffic fields can help distribute different flows across ECMP paths.
ECMP is typically flow-based rather than packet-based. A single flow may remain on one underlay path, while different flows can be distributed across other available paths.
Legacy VXLAN Port Configurations
Some older VXLAN implementations and lab environments used non-standard or configurable UDP destination ports. Your original VXLAN lab used port 1024 explicitly. Both VTEPs must use compatible port settings. For new standards-based deployments, UDP 4789 is the normal choice.
3.7 — VXLAN Encapsulation Overhead and MTU
Encapsulation increases packet size because additional headers are inserted around the original Ethernet frame.
For a typical VXLAN packet transported over IPv4, the additional headers consist of:
Engineering Calculation · VXLAN over IPv4
14
Inner Ethernet Header
8
VXLAN Header
8
UDP Header
20
Outer IPv4 Header
14 + 8 + 8 + 20 = 50 Bytes
of Additional Encapsulated IP Payload
The 50-byte calculation assumes an untagged inner Ethernet frame, a minimum 20-byte outer IPv4 header and no optional extensions. It excludes the outer Ethernet header and FCS, which are link-layer overhead.
If the original host sends a 1,500-byte IP packet, the VXLAN encapsulation produces a 1,550-byte outer IPv4 packet when the inner Ethernet header is included.
Therefore, an underlay IP MTU of at least 1,550 bytes is required for this specific untagged IPv4 example without outer IP fragmentation.
Many data centre operators configure larger MTUs, often using jumbo-frame support, to provide additional headroom for encapsulation, VLAN tags and other transport requirements.
MTU Design Warning
VXLAN does not automatically require a particular jumbo MTU such as 9,000 bytes. The required underlay MTU depends on the inner packet size and encapsulation format. Insufficient MTU can cause fragmentation, packet drops or application-level failures.
3.8 — VXLAN Packet Capture Analysis
Packet captures are one of the most useful tools for understanding VXLAN forwarding. A capture taken on an underlay link reveals the outer Ethernet, IP, UDP and VXLAN headers.
In Wireshark, an engineer can inspect the outer source and destination VTEP addresses, UDP destination port, VNI and encapsulated Ethernet frame.
Illustrative Packet Capture · VXLAN over IPv4
Frame: VXLAN encapsulated ICMP Outer Ethernet: Source MAC: Underlay next-hop MAC Destination MAC: Underlay next-hop MAC Outer IPv4: Source IP: 10.0.0.11 Destination IP: 10.0.0.12 Protocol: UDP (17) UDP: Source Port: 52341 Destination: 4789 VXLAN: Flags: 0x08 (I flag set) VNI: 10010 Inner Ethernet: Source MAC: 00:11:22:33:44:01 Destination MAC: 00:11:22:33:44:02 Inner IPv4: Source IP: 192.168.10.11 Destination IP: 192.168.10.12 Protocol: ICMP ICMP: Type: Echo Request
Illustrative decoded packet fields,
not an actual captured packet.
MAC addresses and UDP source
ports vary according to the
physical link and implementation.
A useful first check is to confirm that the outer destination IP address matches the expected remote VTEP.
The next check is the VNI. A correct VTEP destination with an incorrect VNI can still result in failed forwarding.
Finally, the inner Ethernet and IP headers should match the expected workload traffic. This helps determine whether the issue exists before encapsulation, inside the overlay or after decapsulation.
3.9 — Common VXLAN Packet Forwarding Failures
| Failure | Likely Symptom | Engineering Check |
|---|---|---|
| Incorrect VNI | Remote VTEP cannot deliver traffic into intended segment | Verify local VLAN-to-VNI mappings and NVE membership |
| Missing VTEP route | Encapsulated packets cannot reach remote VTEP | Check underlay routing and VTEP reachability |
| UDP port mismatch | Receiver may not recognise VXLAN packets | Verify configured VXLAN destination port |
| Insufficient MTU | Large packets fail while smaller packets work | Test end-to-end encapsulated packet sizes |
| Missing remote MAC mapping | Traffic is flooded or cannot be forwarded | Inspect MAC tables and EVPN reachability |
| Underlay path failure | Packet loss or temporary interruption | Verify routing convergence and ECMP next hops |
| Security policy drop | Encapsulated traffic is filtered in transit | Inspect ACLs, firewalls and UDP 4789 reachability |
Troubleshooting Method
Follow the packet in order: original host frame, local bridge-domain mapping, remote MAC-to-VTEP resolution, VXLAN encapsulation, underlay IP forwarding, destination VTEP decapsulation and final host delivery. This method helps isolate the failing stage rather than treating the overlay as one opaque tunnel.
3.10 — Operational Verification Commands
On a Cisco NX-OS-based VXLAN fabric, several operational commands can help validate packet forwarding. Exact output and availability vary by platform and release.
VXLAN Verification · NX-OS-Style CLI
show nve interface show nve peers show nve vni show mac address-table show l2route evpn mac all show ip route 10.0.0.12 show ip interface brief show bgp l2vpn evpn summary show bgp l2vpn evpn show interface counters errors
Use the commands supported by
the specific platform and
NX-OS release. Some EVPN
commands are applicable
only when BGP EVPN is
configured.
A successful troubleshooting investigation should confirm that the local VTEP is operational, the expected VNI is active, the remote VTEP is reachable and the required forwarding information exists.
If the fabric uses BGP EVPN, the engineer should also inspect EVPN routes and their imported reachability information. However, a healthy BGP session alone does not prove successful VXLAN data-plane forwarding.
Section 3 — Engineering Takeaways
✓
VXLAN encapsulates the original
Ethernet frame inside an
8-byte VXLAN header,
UDP and outer IP packet.
✓
The inner headers identify
workloads, while the outer
IP addresses identify
the source and destination VTEPs.
✓
The 24-bit VNI identifies
the logical overlay
segment and is essential
for correct decapsulation.
✓
UDP destination port 4789
is the standard VXLAN port,
while source-port entropy
can support underlay ECMP.
✓
VXLAN over IPv4 typically
adds 50 bytes to the
underlay IP packet size
for an untagged inner
Ethernet frame.
✓
Packet captures and
VTEP/VNI verification
help distinguish
encapsulation faults
from underlay routing
and endpoint forwarding
problems.
Network Insight · Interactive Engineering Lab 03
VXLAN Encapsulation and Packet Header Explorer
Build and inspect a VXLAN packet. Explore how the original Ethernet frame is wrapped in VXLAN, UDP, IP and Ethernet headers, then trace the packet across the underlay and examine its MTU requirements.
VXLAN Packet Journey
Ready to Encapsulate
Select a Packet Layer to Inspect Its Fields
Packet Header Inspector
Inner Ethernet Header
The original Ethernet header identifies the source and destination MAC addresses. VXLAN transports this frame across the routed IP network without requiring intermediate spines to inspect its inner MAC addresses.
Header Fields
Field 1
Source MAC
Field 2
Destination MAC
Header Size
14 bytes
Purpose
Tenant Ethernet
Original Inner IP Packet
1500 B
Excludes inner Ethernet header
Outer IP Packet Size
1550 B
Compared with underlay IP MTU
Encapsulation Overhead
+50 B
14 inner Ethernet + 20 IPv4 +
8 UDP + 8 VXLAN
Underlay MTU is sufficient for this packet.
Select a header or run the animation
Packet Encapsulation Event Timeline
[READY] Host A has an Ethernet frame.
[READY] Select a header to inspect its fields.
[READY] Run the animation to trace encapsulation.
Key Engineering Principle
VXLAN preserves the original Ethernet frame inside a new transport packet. The underlay forwards using the outer IP destination, while the remote VTEP removes encapsulation and delivers the original Ethernet frame. Always account for encapsulation overhead when planning the underlay MTU.
Section 04 · VXLAN Control Plane
VXLAN Flood-and-Learn and the BGP EVPN Control Plane
VXLAN defines how Ethernet frames are encapsulated and transported across an IP network. However, encapsulation alone does not explain how a VTEP discovers remote MAC addresses, identifies participating tunnel endpoints or determines where unknown traffic should be sent.
These functions are handled through endpoint learning and control-plane mechanisms. Traditional VXLAN deployments commonly relied on flood-and-learn forwarding, while modern data centre fabrics frequently use BGP Ethernet VPN (EVPN) to distribute endpoint reachability and overlay membership information.
Understanding both approaches is important because they solve similar forwarding problems through fundamentally different mechanisms. Flood-and-learn depends heavily on data-plane observation, whereas EVPN enables endpoint information to be advertised through a routing control plane.
Flood-and-Learn
BUM Traffic
Multicast
Ingress Replication
BGP EVPN
Route Type 2
Route Type 3
ARP Suppression
4.1 — Why VXLAN Needs Endpoint Discovery
Consider two servers connected to different leaf switches in the same VXLAN segment. Host A is connected to Leaf A, while Host B is connected to Leaf B.
Before Leaf A can send a known-unicast VXLAN packet directly to Leaf B, it must determine which remote VTEP is responsible for Host B's destination MAC address.
This requires a mapping between the remote MAC address and the destination VTEP IP address.
ENDPOINT LOCATION
MAC-to-VTEP Mapping
Associates a remote endpoint MAC address with the VTEP IP address through which the endpoint is reachable.
OVERLAY MEMBERSHIP
VNI Participation
Identifies the VTEPs participating in a particular VXLAN segment, allowing appropriate forwarding and replication.
The Endpoint Discovery Problem
VXLAN encapsulation provides a way to transport a frame, but the ingress VTEP must still determine the correct remote VTEP destination. Flood-and-learn and BGP EVPN provide different methods for discovering and maintaining that forwarding information.
4.2 — Traditional VXLAN Flood-and-Learn
Traditional VXLAN flood-and-learn operates in a way that resembles conventional Ethernet switching. A VTEP learns MAC addresses from observed traffic and uses flooding when the destination is unknown.
When an ingress VTEP receives a frame destined for an unknown MAC address, it replicates the frame to other participating VTEPs using the configured BUM delivery mechanism.
When the destination endpoint responds, the receiving VTEPs can learn the association between the source MAC address and the originating VTEP address.
01
Unknown Destination MAC
Host A sends a frame
toward Host B. Leaf A
does not yet know which
remote VTEP owns Host B's
MAC address.
02
Flooding Across the Overlay
Leaf A sends copies to
the appropriate remote
VTEPs using multicast
or ingress replication.
03
Destination Receives the Frame
Leaf B forwards the
received frame into
the local bridge domain
containing Host B.
04
Source MAC Learning
VTEPs learn remote
source MAC-to-VTEP
associations from
observed encapsulated
traffic.
05
Subsequent Known Unicast
Once the required
destination mapping
is learned, traffic
can be sent directly
to the appropriate VTEP.
Flood-and-learn is conceptually straightforward, but it introduces operational considerations around flooding scope, replication, MAC aging and endpoint mobility.
4.3 — Understanding BUM Traffic
BUM stands for Broadcast, Unknown Unicast and Multicast. These traffic categories require special handling because the ingress VTEP may not have a single known destination VTEP.
| Traffic Type | Example | Typical VXLAN Handling |
|---|---|---|
| Broadcast | ARP request | Replicated to participating VTEPs unless locally suppressed or otherwise handled |
| Unknown Unicast | Frame with an unknown destination MAC | Flooded within the relevant overlay segment when unknown-unicast flooding is enabled |
| Multicast | Application multicast Ethernet traffic | Delivered according to the configured overlay multicast and replication behavior |
BUM replication must be constrained to the appropriate VXLAN segment. A VTEP should not replicate tenant traffic indiscriminately across unrelated VNIs.
Flooding and Scale
Excessive BUM traffic can consume bandwidth and processing resources. Large VXLAN environments therefore benefit from careful replication design, endpoint learning controls and features such as ARP suppression.
4.4 — Multicast Versus Ingress Replication
VXLAN can distribute BUM traffic through underlay multicast or through ingress replication. Both approaches deliver copies to participating VTEPs, but they differ in where packet replication occurs.
UNDERLAY MULTICAST
Multicast-Based Replication
VTEPs send VXLAN BUM traffic toward a multicast group. Multicast-capable underlay devices replicate traffic along the distribution tree.
HEAD-END REPLICATION
Ingress Replication
The ingress VTEP creates individual VXLAN copies for remote VTEPs in the replication list. Underlay multicast is not required.
| Characteristic | Multicast | Ingress Replication |
|---|---|---|
| Underlay multicast | Required | Not required |
| Replication location | Multicast distribution tree in the network | Ingress VTEP |
| Source VTEP load | Generally fewer source-generated copies | Increases with the number of remote replication destinations |
| Operational dependency | Multicast routing and group membership | Accurate remote VTEP replication list |
| Common deployment | Traditional multicast-based VXLAN fabrics | Frequently used with BGP EVPN |
Ingress replication is popular in EVPN fabrics because the control plane can distribute the information needed to construct replication lists without requiring multicast routing throughout the underlay.
4.5 — Introducing the BGP EVPN Control Plane
BGP EVPN uses Multiprotocol BGP to distribute overlay reachability information between participating network devices.
Rather than relying exclusively on observed VXLAN traffic to learn remote endpoint locations, EVPN allows VTEPs to advertise reachability information through BGP routes.
This separates the control plane, responsible for distributing endpoint information, from the data plane, responsible for carrying encapsulated Ethernet traffic.
Reference Architecture · EVPN Control Plane and VXLAN Data Plane
IP underlay
EVPN control-plane sessions
Conceptual EVPN route-reflector topology.
The dashed green paths represent logical
BGP EVPN sessions, not dedicated physical
links. The route reflector distributes
EVPN reachability information, while
VXLAN data traffic travels directly
between VTEPs through the routed underlay.
A BGP EVPN route reflector can distribute overlay reachability information between leaf VTEPs without becoming part of the VXLAN data path.
This architecture supports scalable endpoint learning and allows network engineers to inspect the control-plane state independently from the forwarding path.
4.6 — EVPN Route Type 2: MAC/IP Advertisement
EVPN Route Type 2 is the MAC/IP Advertisement Route. It allows a VTEP to advertise endpoint MAC reachability and, when applicable, associated IP address information.
For example, Leaf B can advertise that Host B's MAC address is reachable through Leaf B's VTEP.
Leaf A can import the appropriate EVPN route and install forwarding information for Host B without needing to discover the remote MAC location solely through flooding.
Conceptual EVPN Type 2 Advertisement
EVPN Route Type: 2 Route: MAC/IP Advertisement MAC Address: 00:11:22:33:44:02 IP Address: 192.168.10.12 Advertising VTEP: 10.0.0.12 Overlay Segment: VNI 10010 BGP Next Hop: 10.0.0.12 Result: Remote MAC reachability learned through the EVPN control plane
Simplified route illustration.
Actual EVPN NLRI includes additional
fields such as the route distinguisher,
Ethernet segment identifier and
Ethernet tag ID. Route targets and
encapsulation-related attributes
influence import and forwarding.
Control-Plane Learning
Type 2 routes reduce dependence on data-plane flooding for discovering known remote endpoints. The receiving VTEP can install the advertised MAC-to-VTEP mapping after validating and importing the route.
4.7 — EVPN Route Type 3: Inclusive Multicast Ethernet Tag
EVPN Route Type 3 is the Inclusive Multicast Ethernet Tag (IMET) Route. It advertises information used to establish participation in a broadcast domain and construct BUM delivery state.
In an ingress-replication design, IMET routes can help VTEPs discover the remote peers that should receive replicated BUM traffic for the relevant overlay segment.
EVPN ROUTE TYPE 2
Endpoint Reachability
Advertises MAC addresses and optional IP bindings, enabling remote endpoint forwarding information to be installed.
EVPN ROUTE TYPE 3
Broadcast Domain Membership
Supports discovery of participating VTEPs and establishment of BUM delivery state, including ingress replication lists.
| Route Type | Primary Role | Operational Importance |
|---|---|---|
| Type 2 | MAC/IP Advertisement | Known-unicast endpoint reachability |
| Type 3 | Inclusive Multicast Ethernet Tag | BUM membership and replication information |
| Type 1 | Ethernet Auto-Discovery | Multihoming procedures and mass withdrawal |
| Type 4 | Ethernet Segment | Ethernet segment discovery and designated forwarder election |
| Type 5 | IP Prefix Advertisement | IP prefix reachability in EVPN-based routing architectures |
Type 2 and Type 3 are particularly important when understanding basic VXLAN EVPN Layer 2 forwarding.
Other EVPN route types support more advanced designs, including multihoming, distributed routing and IP prefix advertisement.
4.8 — EVPN Endpoint Learning Sequence
The following example shows how Host B's reachability can become available to Leaf A through BGP EVPN.
01
Host B Becomes Active
Host B sends traffic
through Leaf B.
Leaf B learns the
local endpoint MAC
address and, where
applicable, its
IP binding.
02
Leaf B Advertises Type 2
Leaf B originates an
EVPN MAC/IP Advertisement
Route containing the
endpoint reachability
information.
03
Route Reflector Distributes EVPN
The EVPN route reflector
distributes the route
to appropriate BGP
EVPN peers according
to routing policy.
04
Leaf A Imports the Route
Leaf A validates the
route and imports
the endpoint mapping
into the appropriate
overlay forwarding
context.
05
Known-Unicast VXLAN Forwarding
Leaf A can encapsulate
traffic for Host B
directly toward
Leaf B's VTEP,
assuming underlay
reachability and
valid forwarding state.
Control Plane Versus Data Plane
EVPN distributes information about where an endpoint is located. VXLAN carries the actual Ethernet frame. The BGP route reflector normally participates in control-plane distribution but does not need to forward the encapsulated tenant data traffic.
4.9 — ARP Suppression in VXLAN EVPN
In a conventional Layer 2 network, ARP requests are broadcast to discover the MAC address associated with an IPv4 address.
In a VXLAN fabric, uncontrolled ARP broadcasting can create additional BUM traffic across the overlay.
With EVPN-based ARP suppression, a VTEP can use known IP-to-MAC bindings to answer certain ARP requests locally, reducing the need to flood those requests to remote VTEPs.
WITHOUT SUPPRESSION
ARP Flooding
An ARP request may be replicated across the VXLAN segment to discover the destination MAC address.
WITH SUPPRESSION
Local ARP Response
A VTEP with a valid learned IP-to-MAC binding may answer the ARP request locally without flooding it.
Why ARP Suppression Helps
ARP suppression reduces unnecessary broadcast traffic when valid endpoint bindings are available. Unknown or unresolved requests may still require flooding or other resolution mechanisms, depending on platform behavior and configuration.
4.10 — MAC Mobility and Endpoint Movement
Virtual machines and workloads may move between hosts or leaf switches while retaining their original MAC and IP addresses.
This creates a forwarding challenge: remote VTEPs must update their endpoint mappings when a MAC address becomes reachable through a different VTEP.
BGP EVPN supports MAC mobility procedures, including a MAC Mobility extended community and sequence numbers used to help distinguish newer endpoint location advertisements.
01
Original Endpoint Location
Host B is reachable
through Leaf B.
Remote VTEPs have
imported the
corresponding
EVPN Type 2 route.
02
Workload Moves to Leaf C
Host B moves to
an attachment point
behind Leaf C
while retaining
its MAC address.
03
Updated EVPN Advertisement
Leaf C advertises
updated MAC reachability.
MAC mobility information
helps remote devices
select the newer
endpoint location.
04
Forwarding State Converges
Remote VTEPs update
the MAC-to-VTEP
mapping and forward
subsequent traffic
toward Leaf C.
MAC mobility is especially important in virtualized environments where workload placement can change without requiring the endpoint to obtain a new IP address.
MAC Flapping Versus Legitimate Mobility
Repeated movement of the same MAC address between VTEPs can indicate legitimate workload migration, but it may also indicate a loop, duplicate MAC address or configuration error. Engineers should inspect mobility events and sequence numbers before assuming the movement is expected.
4.11 — Flood-and-Learn Versus BGP EVPN
| Capability | Flood-and-Learn | BGP EVPN |
|---|---|---|
| Remote MAC learning | Observed data-plane traffic | Control-plane Type 2 advertisements |
| Unknown destinations | Typically flooded | May be flooded or handled by policy and platform features |
| VTEP discovery | Multicast membership or configured peer lists | EVPN membership information and replication state |
| ARP suppression | Not inherent to basic flood-and-learn | Supported when suitable IP/MAC bindings are available |
| MAC mobility | Data-plane relearning | EVPN mobility procedures and sequence numbers |
| Operational visibility | Primarily forwarding and MAC tables | BGP EVPN routes, forwarding state and VTEP tables |
| Typical architecture | Simpler or legacy VXLAN environments | Modern scalable VXLAN EVPN fabrics |
BGP EVPN provides a more structured control plane for endpoint distribution, but it also introduces additional operational dependencies.
Engineers must verify BGP neighbor state, EVPN address-family activation, route import policies, route targets, VNI mappings and underlay reachability.
4.12 — EVPN Control-Plane Verification
A healthy VXLAN EVPN fabric requires consistency between BGP EVPN routes and the actual VXLAN forwarding state installed on the VTEPs.
The following Cisco NX-OS-style commands provide useful starting points for verifying EVPN reachability and replication.
EVPN Operational Verification · NX-OS Style
show bgp l2vpn evpn summary show bgp l2vpn evpn show bgp l2vpn evpn route-type 2 show bgp l2vpn evpn route-type 3 show nve peers show nve vni show l2route evpn mac all show mac address-table show ip route 10.0.0.12
Illustrative NX-OS commands.
Exact syntax, supported route-type
filters and output vary by
platform and software release.
| Check | Expected Result |
|---|---|
| BGP EVPN neighbors | Expected sessions are Established |
| Type 2 routes | Remote endpoint reachability is advertised and imported |
| Type 3 routes | Expected overlay membership and BUM replication state exist |
| VNI state | Required VNIs are configured and operational |
| Remote VTEP route | Underlay provides usable reachability to remote VTEP IP |
| MAC forwarding table | Remote MAC addresses resolve to the correct VTEP |
EVPN Troubleshooting Warning
An Established BGP EVPN session does not prove that remote MAC routes have been imported correctly. Route-target mismatches, incorrect VNI mappings, missing endpoint advertisements or unresolved VTEP next hops can prevent forwarding even when the BGP session itself is healthy.
4.13 — Engineering Design Considerations
Selecting the correct VXLAN control-plane design requires consideration of scalability, replication overhead, operational visibility and endpoint mobility.
| Design Area | Engineering Consideration |
|---|---|
| EVPN route reflectors | Provide scalable BGP route distribution with appropriate redundancy and policy |
| Route targets | Control which EVPN routes are imported into the intended overlay contexts |
| BUM replication | Balance source VTEP replication load against underlay multicast complexity |
| ARP suppression | Reduce unnecessary ARP flooding while maintaining accurate endpoint bindings |
| MAC mobility | Ensure endpoint moves converge correctly without persistent stale forwarding state |
| Underlay reachability | Maintain resilient routing between all required VTEPs |
| Monitoring | Correlate EVPN control-plane state with installed forwarding entries and packet delivery |
Recommended Troubleshooting Order
Verify underlay VTEP reachability first, followed by BGP EVPN neighbor state, imported Type 2 and Type 3 routes, VNI membership, remote MAC forwarding entries and actual VXLAN packet delivery. This sequence helps separate transport failures from control-plane and forwarding-state problems.
Section 4 — Engineering Takeaways
✓
Traditional VXLAN
flood-and-learn uses
observed traffic
and flooding to
discover remote
MAC locations.
✓
BUM traffic includes
broadcast, unknown
unicast and multicast
frames that may
require replication.
✓
Underlay multicast
and ingress replication
provide different
mechanisms for
distributing BUM traffic.
✓
EVPN Type 2 routes
advertise endpoint
MAC/IP reachability,
while Type 3 routes
support broadcast
domain membership
and BUM delivery state.
✓
ARP suppression
can reduce broadcast
traffic when valid
endpoint bindings
are available.
✓
EVPN MAC mobility
procedures help
update forwarding
information when
workloads move
between VTEPs.
✓
Successful VXLAN
forwarding depends
on both valid
EVPN control-plane
information and
operational underlay
and data-plane state.
Network Insight · Interactive Engineering Lab 04
Flood-and-Learn vs BGP EVPN: Explore the VXLAN Control Plane
Investigate how remote endpoint reachability is discovered. Compare data-plane flooding with BGP EVPN route advertisements, inspect the resulting forwarding state and simulate endpoint mobility between VTEPs.
Step 1 · Select Control-Plane Model
Step 2 · Generate a Network Event
VXLAN Fabric · VNI 10010
Data-Plane Learning
BUM Replication Copies (Current Event)
0
EVPN Route Advertisements (Current Event)
0
Remote MAC Entry on Leaf A
Unknown
Leaf A · Remote MAC Reachability
| Destination | Remote VTEP | Source |
|---|---|---|
| No remote entry learned | ||
EVPN Control-Plane State
Flood-and-Learn Mode
Leaf A initially has no remote MAC mapping. Unknown unicast traffic is replicated to other VTEPs participating in the VNI.
| EVPN Route | State |
|---|---|
| Type 2 · MAC/IP | Not used |
| Type 3 · IMET | Not used |
Control-Plane and Data-Plane Event Timeline
[READY] Select a control-plane model.
[READY] Generate unknown unicast, ARP or mobility events.
Engineering Takeaway
Flood-and-learn discovers remote MAC reachability through data-plane traffic. BGP EVPN can advertise MAC/IP reachability in the control plane, reducing the need for unknown-unicast flooding.
Educational event-driven model. Assumes three
participating VTEPs, ingress replication and
pre-established EVPN sessions in EVPN mode.
The widget simplifies MAC learning, route
withdrawal, mobility sequencing and ARP
suppression; it is not a full protocol emulator.
Section 05 · Distributed Layer 3 Overlay
VXLAN EVPN Routing, Anycast Gateways and Inter-VNI Connectivity
VXLAN is often introduced as a technology that extends Layer 2 networks across an IP fabric. However, modern VXLAN EVPN architectures also support distributed Layer 3 routing between different subnets and virtual networks.
This capability allows workloads connected to different leaf switches and different VLANs to communicate through distributed routing gateways rather than sending all inter-subnet traffic to a centralized router.
The main building blocks include distributed anycast gateways, integrated routing and bridging (IRB), Layer 2 VNIs, Layer 3 VNIs, VRFs and EVPN route advertisements. Together, these components enable a scalable routed overlay while the physical underlay continues to provide ordinary IP connectivity between VTEPs.
Anycast Gateway
Inter-VNI Routing
Symmetric IRB
Asymmetric IRB
L2 VNI
L3 VNI
VRF
EVPN Type 5
5.1 — Why Layer 3 Routing Matters in VXLAN Fabrics
A VXLAN bridge domain provides Layer 2 connectivity for endpoints belonging to the same logical network segment.
However, real data centre applications often consist of multiple subnets. Web servers, application servers and databases may be assigned to different VLANs and IP networks.
Communication between those networks requires Layer 3 routing. A modern VXLAN EVPN fabric can perform this routing directly on the leaf switches where workloads connect.
SAME SUBNET
Layer 2 VXLAN Forwarding
Traffic between endpoints in the same bridge domain can be transported across the overlay using the associated L2 VNI.
DIFFERENT SUBNETS
Layer 3 EVPN Routing
Traffic between IP subnets is routed through an IRB gateway and can cross the overlay using an L3 VNI.
Architectural Principle
The underlay provides IP transport between VTEPs. The overlay defines tenant Layer 2 and Layer 3 forwarding contexts. Distributed routing allows the leaf fabric to route tenant traffic without requiring a centralized physical router for every inter-subnet flow.
5.2 — Distributed Anycast Gateways
A distributed anycast gateway allows multiple leaf switches to present the same default gateway IP address and virtual gateway MAC address to endpoints in a given subnet.
For example, hosts in subnet 192.168.10.0/24 might use 192.168.10.1 as their default gateway, regardless of which leaf switch they are connected to.
Each participating leaf can perform local routing for traffic that must leave the source subnet. This reduces the need to send packets across the fabric merely to reach a centralized first-hop gateway.
| Gateway Attribute | Example | Purpose |
|---|---|---|
| Subnet | 192.168.10.0/24 | Endpoint IP network |
| Gateway IP | 192.168.10.1 | Shared default gateway address |
| Virtual gateway MAC | 0001.0001.0001 | Common gateway MAC used by participating leaf switches |
| Leaf A | VTEP 10.0.0.11 | Local first-hop routing for its hosts |
| Leaf B | VTEP 10.0.0.12 | Local first-hop routing for its hosts |
The gateway IP and virtual MAC must be consistently configured for the same subnet across participating leaves.
Distributed anycast gateways also support workload mobility: a virtual machine moving to another leaf can continue using the same first-hop gateway identity.
Why Anycast Gateways Improve Traffic Flow
Hosts can reach their default gateway locally on the attached leaf. This can reduce unnecessary hairpinning and allows the fabric to route traffic closer to the source endpoint.
5.3 — Layer 2 VNIs and Layer 3 VNIs
VXLAN EVPN fabrics commonly distinguish between Layer 2 VNIs and Layer 3 VNIs.
An L2 VNI identifies a Layer 2 bridge domain, often associated with a VLAN. An L3 VNI identifies a routed overlay context associated with a VRF.
| Component | Example | Function |
|---|---|---|
| VLAN 10 | 192.168.10.0/24 | Web subnet |
| L2 VNI 10010 | VLAN 10 | Web Layer 2 bridge domain |
| VLAN 20 | 192.168.20.0/24 | Application subnet |
| L2 VNI 10020 | VLAN 20 | Application Layer 2 bridge domain |
| VRF TENANT-A | Tenant routing table | Isolated Layer 3 forwarding context |
| L3 VNI 50000 | VRF TENANT-A | Routed VXLAN overlay context |
In a symmetric IRB design, the L3 VNI is used to transport routed traffic between VTEPs in the relevant tenant VRF.
The L2 VNI continues to provide Layer 2 connectivity for endpoints within the corresponding bridge domain.
VNI Mapping Is Platform-Specific
L2 and L3 VNI numbers are locally configured design values. The examples here illustrate a common VLAN-to-L2-VNI and VRF-to-L3-VNI mapping; the numbers themselves are not reserved or mandatory.
5.4 — VXLAN EVPN Routed Fabric Architecture
The following example contains two leaf VTEPs, two routed spine switches and endpoints in different IP subnets.
Host A belongs to VLAN 10 / VNI 10010, while Host B belongs to VLAN 20 / VNI 10020. Both subnets are associated with VRF TENANT-A and L3 VNI 50000.
Reference Topology · Distributed Routing over VXLAN EVPN
Physical IP underlay
Logical routed VXLAN overlay
Host attachment
Simplified symmetric IRB example.
Each leaf performs local routing
for its attached subnet, while
the L3 VNI carries routed traffic
between VTEPs. The dashed overlay
path is logical and is transported
through the routed spine fabric.
In a distributed gateway deployment,
gateway interfaces for a subnet
are configured consistently on
all leaves participating in that subnet.
5.5 — Integrated Routing and Bridging (IRB)
Integrated Routing and Bridging combines Layer 2 bridging and Layer 3 routing within a single forwarding architecture.
In a VXLAN EVPN fabric, IRB allows a VTEP to bridge frames within an L2 VNI and route packets between different IP subnets associated with the same tenant VRF.
Two major IRB forwarding models are commonly discussed: asymmetric IRB and symmetric IRB.
ASYMMETRIC IRB
Route at Ingress, Bridge at Egress
The ingress VTEP routes into the destination bridge domain and encapsulates traffic using the destination L2 VNI. The egress VTEP then bridges the frame locally.
SYMMETRIC IRB
Route at Both VTEPs
The ingress VTEP routes traffic into the tenant L3 VNI. The egress VTEP performs the corresponding routing lookup and delivers traffic into the destination bridge domain.
5.6 — Asymmetric IRB Explained
In asymmetric IRB, the ingress VTEP performs the Layer 3 routing decision and selects the destination subnet's bridge domain.
The packet is then encapsulated using the destination L2 VNI and transported to the remote VTEP. The egress VTEP bridges the resulting frame to the destination.
Because the ingress leaf may need to route into multiple destination bridge domains, asymmetric designs can require broad distribution of VLAN and L2 VNI configuration across the participating fabric.
Asymmetric IRB Scaling Consideration
In common asymmetric IRB implementations, ingress leaves need the destination subnet's L2 forwarding context. As the number of subnets increases, this can increase the number of VLANs and L2 VNIs that must be provisioned across the fabric.
5.7 — Symmetric IRB Explained
Symmetric IRB performs Layer 3 forwarding at both the ingress and egress VTEPs.
The ingress VTEP routes the packet from the source bridge domain into the tenant VRF and encapsulates it using the L3 VNI.
The egress VTEP decapsulates the routed overlay packet, performs the appropriate destination forwarding operation and delivers it through the local destination bridge domain.
This design allows each leaf to host only the local Layer 2 segments it requires, while participating in the shared tenant Layer 3 routing context.
| Characteristic | Asymmetric IRB | Symmetric IRB |
|---|---|---|
| Ingress VTEP | Routes into destination L2 segment | Routes into tenant L3 VNI |
| Egress VTEP | Bridges to destination | Routes/forwards into destination bridge domain |
| Inter-VTEP VNI | Destination L2 VNI | Tenant L3 VNI |
| Remote L2 VNI presence | Often needed at ingress for destination subnets | Not generally required for every remote subnet |
| Scalability | Can increase L2 state across the fabric | Better separation of local L2 state and tenant routing |
| Common use | Smaller or specific EVPN IRB designs | Large distributed routed EVPN fabrics |
Design Preference
Symmetric IRB is widely used in modern VXLAN EVPN data centre fabrics because it avoids the need for every ingress leaf to instantiate every destination Layer 2 segment. The exact forwarding behavior depends on the platform and EVPN implementation.
5.8 — Symmetric IRB Packet Journey
Consider Host A (192.168.10.11) communicating with Host B (192.168.20.12). Both networks belong to VRF TENANT-A.
The example assumes that the necessary endpoint routes, VNI mappings and underlay reachability are already installed.
01
Host A Sends to Its Default Gateway
Host A recognizes that
192.168.20.12 belongs
to another subnet.
It sends the packet
to gateway 192.168.10.1
on Leaf A.
02
Leaf A Performs a VRF Routing Lookup
Leaf A routes the
packet in VRF TENANT-A.
The installed EVPN
forwarding information
identifies Leaf B as
the remote VTEP for
the destination.
03
Routed VXLAN Encapsulation
Leaf A encapsulates
the routed packet
using L3 VNI 50000
with outer source
VTEP 10.0.0.11
and destination
VTEP 10.0.0.12.
04
Underlay IP Forwarding
The spine fabric
forwards the
encapsulated packet
toward Leaf B
using the outer
destination IP.
ECMP may select
an available path.
05
Leaf B Decapsulates and Routes
Leaf B identifies
the tenant L3 VNI,
processes the packet
in the appropriate
VRF and resolves
the destination
through its local
VLAN 20 bridge domain.
06
Host B Receives the Packet
Leaf B delivers
the packet to
Host B on
192.168.20.12.
The original
source and
destination IPs
remain unchanged
in this example.
Inner Header Handling
Symmetric IRB does not simply preserve the original host Ethernet frame unchanged. The ingress routing operation creates the appropriate routed overlay forwarding context. The exact inner Ethernet header, router MAC information and egress processing depend on the EVPN forwarding model and platform.
5.9 — VRFs and Tenant Isolation
A Virtual Routing and Forwarding instance, or VRF, maintains a separate Layer 3 routing table.
In a VXLAN EVPN fabric, VRFs allow multiple tenants or application environments to use isolated routing contexts over the same physical underlay.
A VRF may contain multiple local subnets and L2 VNIs. A corresponding L3 VNI can provide routed connectivity between VTEPs participating in that VRF.
TENANT-A
VRF TENANT-A
Web and application subnets use their own routing table, L2 VNIs and routed L3 VNI.
TENANT-B
VRF TENANT-B
A separate tenant can use independent routing and overlay identifiers while sharing the same physical IP fabric.
VRF Isolation Is Not a Complete Security Policy
VRFs provide separate routing contexts, but intentional route leaking, shared services or external connectivity can introduce paths between them. Firewall policy, segmentation controls and route import rules must still be designed appropriately.
5.10 — EVPN Route Type 5: IP Prefix Advertisement
EVPN Route Type 5 is the IP Prefix Advertisement Route. It provides a way to advertise IP prefix reachability through the EVPN control plane.
While Type 2 routes primarily describe MAC reachability and optional host IP bindings, Type 5 routes advertise IP prefixes independently of individual endpoint MAC advertisements.
This is useful for advertising routed networks, external prefixes and aggregate reachability in EVPN-based Layer 3 overlays.
| EVPN Route | Primary Information | Typical Role |
|---|---|---|
| Type 2 | MAC address and optional IP address | Endpoint reachability, host routes and MAC/IP bindings |
| Type 3 | Inclusive multicast Ethernet tag information | Overlay membership and BUM delivery |
| Type 5 | IP prefix | Routed prefix advertisement in EVPN |
Conceptual EVPN Route Type 5 Advertisement
EVPN Route Type: 5 Route: IP Prefix Advertisement Tenant VRF: TENANT-A Advertised Prefix: 172.16.50.0/24 Advertising VTEP: 10.0.0.12 Tenant Routed Overlay: L3 VNI 50000 Route Import: Controlled by EVPN route targets Result: Remote VTEPs can install tenant IP prefix reachability
Conceptual example rather than
raw BGP NLRI. Actual Type 5
advertisements include route
distinguisher, Ethernet tag,
IP prefix and gateway-related
fields, with associated BGP
attributes and extended
communities. Forwarding
resolution is implementation-
and deployment-dependent.
Type 2 Versus Type 5
Type 2 can advertise host reachability with MAC/IP bindings, while Type 5 supports IP prefix advertisements. Both can participate in EVPN Layer 3 designs; Type 5 is not mandatory for every symmetric IRB deployment.
5.11 — Route Targets and Route Import
EVPN uses BGP route distinguishers and route targets for different purposes.
A Route Distinguisher (RD) helps make otherwise overlapping VPN routes unique in the BGP control plane.
A Route Target (RT) is an extended community used to control which VPN routes are imported into the appropriate local contexts.
| Attribute | Purpose | Example |
|---|---|---|
| RD | Makes EVPN route identifiers unique | 65001:50000 |
| RT Import | Selects EVPN routes imported into a local context | 65001:50000 |
| RT Export | Marks advertised EVPN routes for intended import | 65001:50000 |
| L3 VNI | Identifies tenant routed VXLAN traffic | 50000 |
Incorrect route targets can cause a VTEP to receive EVPN routes without importing them into the expected VRF or bridge domain.
This is an important troubleshooting distinction: a route visible in the BGP EVPN table is not necessarily installed in the intended tenant forwarding table.
5.12 — VXLAN EVPN Routing Verification
When troubleshooting inter-VNI routing, engineers should verify the tenant routing context, anycast gateway configuration, EVPN reachability and installed forwarding state.
Cisco NX-OS-Style Verification Commands
show vrf show ip route vrf TENANT-A show ip interface brief vrf TENANT-A show nve interface show nve vni show nve peers show bgp l2vpn evpn summary show bgp l2vpn evpn route-type 2 show bgp l2vpn evpn route-type 5 show l2route evpn mac all show mac address-table
Illustrative verification commands.
Exact syntax, VRF filtering,
route-type support and output
depend on the NX-OS platform
and software release.
| Verification | Expected State | Possible Failure |
|---|---|---|
| VRF routing table | Destination route installed | Missing or incorrectly imported EVPN route |
| Anycast gateway | Correct gateway IP and MAC | Incorrect SVI or virtual gateway configuration |
| L2 VNI | Local bridge domain mapped correctly | VLAN-to-VNI mismatch |
| L3 VNI | Tenant routed overlay operational | Missing VRF-to-VNI mapping |
| EVPN routes | Correct remote host or prefix reachability | RT mismatch or missing advertisement |
| VTEP reachability | Outer destination reachable through underlay | Underlay routing failure |
| Packet forwarding | Encapsulation, decapsulation and local delivery succeed | Incorrect forwarding entry or policy drop |
5.13 — Common Inter-VNI Routing Failures
A VXLAN fabric may provide successful same-subnet communication while failing to route between different subnets.
This often indicates that the L2 overlay is operational but the distributed Layer 3 forwarding path is incomplete or misconfigured.
GATEWAY ISSUE
Incorrect Anycast Gateway
Inconsistent gateway IP or MAC configuration can prevent reliable first-hop routing.
VRF ISSUE
Wrong Routing Context
The source or destination subnet may be associated with the wrong tenant VRF.
EVPN ISSUE
Missing Route Import
EVPN routes may be received but rejected or not imported because of route-target policy.
DATA-PLANE ISSUE
Incorrect L3 VNI
A routed overlay mismatch can prevent the receiving VTEP from processing the packet in the correct VRF.
Recommended Troubleshooting Sequence
Confirm local gateway reachability, verify the source VRF routing table, inspect the imported EVPN route, validate the remote VTEP next hop, confirm L3 VNI encapsulation and finally inspect egress routing and local host delivery.
5.14 — Design Considerations for Distributed Routing
Distributed EVPN routing introduces additional flexibility, but consistent design remains essential.
| Design Area | Recommendation |
|---|---|
| Gateway consistency | Use consistent anycast gateway IP and virtual MAC settings for each subnet across participating leaves |
| VNI planning | Maintain a documented mapping between VLANs, L2 VNIs, VRFs and L3 VNIs |
| IRB model | Select symmetric or asymmetric IRB based on platform support, scale and forwarding requirements |
| EVPN policy | Validate route targets and import/export policy for every tenant context |
| Underlay routing | Provide resilient VTEP reachability with suitable ECMP and MTU design |
| Segmentation | Combine VRF isolation with appropriate security and route leaking controls |
| Operational visibility | Correlate BGP EVPN routes with installed tenant routes and actual packet forwarding |
Section 5 — Engineering Takeaways
✓
VXLAN EVPN supports
distributed Layer 3
routing between
different IP subnets,
not only Layer 2
extension.
✓
Distributed anycast
gateways allow
endpoints to use
a consistent default
gateway identity
across participating
leaf switches.
✓
L2 VNIs identify
bridge domains,
while L3 VNIs
support routed
tenant overlays.
✓
Symmetric IRB routes
traffic at both
ingress and egress
VTEPs and is widely
used in scalable
EVPN fabrics.
✓
VRFs provide separate
tenant routing tables,
while route targets
control EVPN
route import.
✓
EVPN Type 5 routes
support IP prefix
advertisements,
complementing
Type 2 endpoint
reachability.
✓
Successful inter-VNI
forwarding requires
correct gateway,
VRF, VNI, EVPN
and underlay state.
Network Insight · Interactive Engineering Lab 05
VXLAN EVPN Routing: Anycast Gateways and Inter-VNI Connectivity
Follow a packet between tenant workloads. Compare Layer 2 bridging with Layer 3 routing, inspect symmetric and asymmetric IRB, and investigate what happens when the tenant VRF or EVPN reachability is misconfigured.
Step 1 · Choose Traffic Scenario
Step 2 · Select IRB Model
Distributed Anycast Gateway Fabric
Symmetric IRB · Ready
Ready to trace
Transport VNI
50000
Forwarding Operation
Route → Bridge
Forwarding Result
Ready
Tenant Forwarding Table
| Destination | Next Hop / VTEP | Context |
|---|
Control-Plane and Gateway State
| Component | Status |
|---|---|
| Tenant VRF | TENANT-A · Active |
| EVPN Reachability | Installed |
| Anycast Gateway | Active on both leaves |
| Routing Model | Symmetric IRB |
Packet Forwarding and EVPN Investigation Timeline
[READY] TENANT-A VRF initialized.
[READY] L2VNIs 10010 and 10020 configured.
[READY] L3VNI 50000 available.
[READY] Select a scenario and trace the packet.
Engineering Explanation
With symmetric IRB, the ingress leaf routes traffic into the tenant L3VNI. The remote leaf receives the routed VXLAN packet, performs destination-side forwarding and delivers it through the appropriate L2VNI or local VLAN. Distributed anycast gateways provide consistent default-gateway addresses across the fabric.
Simplified conceptual EVPN forwarding model.
Assumes pre-established underlay and EVPN
reachability, local gateway configuration and
a single tenant VRF. Actual symmetric IRB
forwarding also uses router MAC information
and platform-specific EVPN route programming.
This lab does not emulate a real BGP or ASIC pipeline.
Section 06 · Deployment and Operations
VXLAN Deployment, Verification and Troubleshooting
Designing a VXLAN EVPN fabric requires more than configuring a VXLAN tunnel. A successful deployment depends on the interaction between the physical IP underlay, VTEP interfaces, overlay segment mappings, BGP EVPN control-plane advertisements and the forwarding tables installed on each leaf switch.
When a VXLAN network fails, the visible symptom may appear to be a simple connectivity problem. However, the root cause could exist in the underlay routing table, EVPN route distribution, VLAN-to-VNI mapping, tenant VRF or encapsulated packet path.
A structured operational approach separates these layers and verifies each dependency in the correct order.
Underlay Routing
VTEP Reachability
NVE Interface
BGP EVPN
VNI Mapping
Route Targets
Packet Capture
Incident Diagnosis
6.1 — The VXLAN Deployment Model
A typical VXLAN EVPN deployment is built in logical stages. The underlay must first provide reliable IP reachability between the VTEP addresses.
Once the underlay is operational, the overlay can establish VNI mappings and BGP EVPN route distribution.
Tenant forwarding is then enabled through the appropriate Layer 2 and Layer 3 interfaces.
01
Build the Routed Underlay
Configure physical
interfaces, routed
leaf-spine links,
loopbacks and
an appropriate
underlay routing
protocol.
02
Establish VTEP Reachability
Confirm that each
leaf can reach
the remote VTEP
source IP addresses
through the underlay.
03
Configure VLAN and VNI Mappings
Associate local
bridge domains
with their intended
L2 VNIs and configure
tenant VRFs and
L3 VNIs where needed.
04
Configure the NVE Interface
Define the VXLAN
tunnel endpoint
source and associate
the required VNIs
with the network
virtualization
interface.
05
Establish BGP EVPN
Configure BGP EVPN
neighbors, address
families, route
distinguishers
and route-target
policies.
06
Validate Endpoint Learning
Confirm local and
remote MAC learning,
EVPN route import,
VTEP peer discovery
and forwarding state.
07
Test Tenant Connectivity
Verify same-subnet
VXLAN bridging,
inter-subnet routing,
failure recovery
and expected
traffic isolation.
Recommended Deployment Order
Validate the underlay before introducing overlay complexity. A correctly configured BGP EVPN control plane cannot compensate for missing IP reachability between VTEPs.
6.2 — VXLAN EVPN Reference Topology
The following reference design uses two spine switches and two leaf VTEPs.
The leaf switches advertise their loopback VTEP addresses through the underlay. BGP EVPN distributes overlay reachability information.
Reference Deployment · Two-Leaf VXLAN EVPN Fabric
IP underlay
Logical VXLAN overlay
Endpoint access
Simplified deployment topology.
VXLAN traffic uses UDP port 4789
and travels through the routed
spine fabric between VTEPs.
BGP EVPN control-plane sessions
are logical routing adjacencies
and may use dedicated route
reflectors or another supported
peering design.
6.3 — Underlay Routing Requirements
The VXLAN underlay is the routed IP network that transports encapsulated packets between VTEPs.
The underlay can use routing protocols such as OSPF, IS-IS or BGP, depending on the design and platform.
Its primary purpose is to provide resilient IP reachability to the loopback addresses used as VTEP sources.
| Underlay Requirement | Engineering Check |
|---|---|
| VTEP IP reachability | Remote VTEP loopbacks are present in the routing table |
| Routing convergence | Failed links trigger an appropriate alternate path |
| ECMP | Multiple equal-cost paths are installed where intended |
| MTU | Underlay supports the additional VXLAN encapsulation overhead |
| UDP transport | VXLAN UDP traffic is permitted through relevant network policies |
| Source address | NVE interface uses the intended VTEP loopback address |
Underlay Verification · NX-OS-Style Commands
show ip interface brief show ip route 10.0.0.12 show ip ospf neighbors show ip route ping 10.0.0.12 source 10.0.0.11 traceroute 10.0.0.12 source 10.0.0.11
Example assumes OSPF as the
underlay protocol. For an
IS-IS or BGP underlay,
use the corresponding
routing verification
commands. Source-aware
ping and traceroute syntax
varies by release.
MTU and VXLAN Encapsulation
VXLAN adds outer Ethernet, IP, UDP and VXLAN headers around the original Ethernet frame. For a common IPv4 transport, this adds approximately 50 bytes of overhead, excluding additional tags or encapsulations. The underlay MTU must accommodate the resulting packet size.
6.4 — Configuring the VTEP and NVE Interface
A VXLAN Tunnel Endpoint, or VTEP, performs VXLAN encapsulation and decapsulation.
On Cisco NX-OS platforms, VXLAN forwarding is commonly configured through a Network Virtualization Edge interface such as NVE1.
The NVE interface references a source loopback and the VNIs that participate in the overlay.
Illustrative Cisco NX-OS VXLAN EVPN Configuration
feature nv overlay
feature vn-segment-vlan-based
feature bgp
nv overlay evpn
vlan 10
vn-segment 10010
interface loopback1
ip address 10.0.0.11/32
interface nve1
no shutdown
host-reachability protocol bgp
source-interface loopback1
member vni 10010
ingress-replication protocol bgp
evpn
vni 10010 l2
rd auto
route-target import auto
route-target export auto
Illustrative configuration fragment
for a compatible NX-OS VXLAN EVPN
deployment. It is not a complete
production configuration. The
physical underlay, loopback
advertisement, BGP EVPN neighbors,
access interfaces and other
platform-specific requirements
must also be configured.
Exact syntax and automatic
route-target behavior depend
on platform and release.
In this example, VLAN 10 is associated with VNI 10010. Loopback1 provides the local VTEP source address.
The NVE interface enables VXLAN encapsulation and uses BGP EVPN for remote endpoint learning and ingress replication state.
6.5 — Verifying BGP EVPN Control-Plane State
The next step is to verify that BGP EVPN sessions are established and that the expected EVPN routes are being exchanged.
Engineers should inspect both the BGP neighbor state and the actual route information installed for the overlay.
BGP EVPN Verification
show bgp l2vpn evpn summary show bgp l2vpn evpn show bgp l2vpn evpn route-type 2 show bgp l2vpn evpn route-type 3 show bgp l2vpn evpn route-type 5 show nve peers show nve vni
Commands are representative
of Cisco NX-OS. Support for
specific route-type filters
and output formats varies.
| Verification | Expected Result | Failure Indicator |
|---|---|---|
| BGP EVPN neighbor | Established | Idle, Active or repeated resets |
| Type 2 route | Remote MAC/IP reachability present | Missing endpoint advertisement |
| Type 3 route | Correct overlay membership state | Missing BUM replication peer |
| Type 5 route | Expected routed prefixes present when used | Missing prefix or import mismatch |
| NVE peers | Expected remote VTEPs operational | Missing or down remote peer |
| VNI status | Required VNIs operational | Inactive VNI or incorrect mapping |
BGP Established Does Not Mean VXLAN Works
BGP EVPN can be Established while endpoint routes are missing, route targets are incorrect, VNIs are inactive or remote VTEPs are unreachable. Verify the installed forwarding state rather than relying on neighbor status alone.
6.6 — MAC Learning and Forwarding Validation
Once the control plane is operational, inspect the forwarding tables to confirm that endpoint reachability has been programmed correctly.
Local MAC addresses should resolve toward access interfaces. Remote MAC addresses should resolve toward the appropriate remote VTEP.
Endpoint Forwarding Verification
show mac address-table show l2route evpn mac all show nve peers show nve vni show ip arp show ip route vrf TENANT-A
Inspect local versus remote
MAC entries, VNI association,
ARP state and tenant routes.
VRF-specific commands apply
when Layer 3 EVPN routing
is configured.
LOCAL ENDPOINT
Access-Side Learning
Confirm that the endpoint MAC is associated with the correct VLAN and local interface.
REMOTE ENDPOINT
EVPN Remote MAC Learning
Confirm that the remote MAC is associated with the expected VNI and remote VTEP destination.
6.7 — Packet Capture and VXLAN Encapsulation
Packet captures are particularly useful when the control plane appears healthy but data traffic still fails.
A VXLAN packet typically contains an outer Ethernet header, outer IP header, UDP header, VXLAN header and original inner Ethernet frame.
The standard VXLAN UDP destination port is 4789. The VXLAN header carries the VNI identifying the relevant overlay segment.
| Packet Field | Example | What to Verify |
|---|---|---|
| Outer source IP | 10.0.0.11 | Correct ingress VTEP |
| Outer destination IP | 10.0.0.12 | Correct remote VTEP |
| UDP destination | 4789 | VXLAN transport |
| VNI | 10010 | Correct overlay segment |
| Inner source IP | 192.168.10.11 | Original endpoint source |
| Inner destination IP | 192.168.10.12 | Original endpoint destination |
Example Linux Packet Capture
sudo tcpdump -ni eth0 udp dst port 4789 sudo tcpdump -ni eth0 \ 'host 10.0.0.12 and udp port 4789' \ -vv
Run on an interface where
encapsulated VXLAN traffic
is visible. Hardware-offloaded
traffic may not appear in
ordinary host or switch CPU
captures. SPAN, ERSPAN or
platform-specific capture
facilities may be required.
Capture Location Matters
Capturing traffic at the server interface normally shows the original tenant packet. Capturing on an underlay link can reveal the VXLAN-encapsulated packet. These views answer different questions during troubleshooting.
6.8 — Common VXLAN Failure Scenarios
VXLAN incidents often fall into recognizable categories. The table below maps typical symptoms to likely areas of investigation.
| Symptom | Possible Cause | Investigation |
|---|---|---|
| Remote VTEP unreachable | Underlay routing or physical link failure | Check routes, interfaces and source-aware ping |
| BGP EVPN neighbor down | BGP configuration, transport or policy issue | Inspect neighbor state, peer addresses and EVPN activation |
| Missing remote MAC | No Type 2 route or route import failure | Check EVPN routes, route targets and MAC learning |
| Broadcast traffic fails | Missing Type 3 membership or replication state | Inspect IMET routes, VNI state and BUM replication |
| Same-subnet works, inter-subnet fails | Gateway, VRF, L3 VNI or routing issue | Check SVI, tenant routes and symmetric IRB |
| Small packets work, large packets fail | MTU mismatch or fragmentation issue | Test path MTU and inspect encapsulated sizes |
| Endpoint intermittently unreachable | MAC mobility, stale state or inconsistent forwarding | Check mobility events and forwarding updates |
| Only one VNI fails | Incorrect VLAN/VNI mapping or route target | Compare affected and healthy VNI configuration |
6.9 — Seven-Stage VXLAN Troubleshooting Workflow
The most effective troubleshooting method is to locate the first point at which expected network state differs from actual state.
Begin with the physical transport and progress toward tenant forwarding.
01
Validate Physical and Routed Underlay
Confirm interface
status, routing
adjacencies,
VTEP loopback
reachability
and expected
ECMP paths.
02
Validate NVE and VNI State
Check the NVE
source interface,
operational status,
VNI membership
and VLAN mappings.
03
Validate BGP EVPN Neighbors
Confirm EVPN
address-family
sessions and
route exchange
with the expected
peers or route
reflectors.
04
Inspect EVPN Route Import
Verify relevant
Type 2, Type 3
and, where used,
Type 5 routes.
Check route targets
and import policy.
05
Inspect Forwarding Tables
Compare EVPN routes
with remote MAC,
ARP, VNI and
tenant VRF
forwarding state.
06
Trace the Encapsulated Packet
Verify outer VTEP
addresses, UDP port,
VNI and successful
transport through
the underlay.
07
Verify Egress Delivery and Policy
Confirm decapsulation,
local VLAN forwarding,
gateway processing,
ACLs and destination
endpoint reachability.
Engineering Troubleshooting Principle
Avoid changing several configuration elements simultaneously. Identify the failing layer, form a testable hypothesis and compare expected versus observed state before applying a fix.
6.10 — Example Incident: Remote Host Unreachable
Consider a VXLAN fabric where Host A on Leaf A cannot reach Host B on Leaf B.
Both hosts belong to VNI 10010. BGP EVPN sessions are Established, but traffic still fails.
| Investigation Step | Observed Result | Interpretation |
|---|---|---|
| Underlay route to 10.0.0.12 | Present | Remote VTEP route exists |
| BGP EVPN neighbor | Established | Control-plane transport is operational |
| EVPN Type 2 route | Remote Host B MAC advertised | Endpoint information is being distributed |
| Remote MAC forwarding | Entry absent from VNI 10010 | Route not programmed into expected forwarding context |
| Route-target inspection | Import RT mismatch | Likely reason route was not imported |
In this example, the underlay and BGP neighbor state are healthy. The failure occurs when the received EVPN route is not imported into the intended local forwarding context.
Correcting the route-target policy and validating the resulting remote MAC entry would be the appropriate next investigation step.
Avoid the Wrong Fix
Restarting BGP or modifying underlay routing would not directly address the route import mismatch identified in this example. The evidence points toward overlay policy and forwarding installation.
6.11 — VXLAN Performance and Scale
Beyond basic connectivity, production VXLAN fabrics must be designed for scale, predictable convergence and operational visibility.
| Design Area | Operational Consideration |
|---|---|
| MAC scale | Monitor local and remote MAC table capacity |
| EVPN route scale | Monitor BGP EVPN routes, memory and route-reflector capacity |
| BUM traffic | Control unnecessary replication and broadcast amplification |
| ECMP | Validate load distribution across available spine paths |
| MTU | Ensure encapsulated packets fit within the supported underlay MTU |
| MAC mobility | Track workload movement and unexpected MAC flapping |
| Convergence | Test link, leaf and control-plane failure recovery |
| Security | Apply segmentation, route policy and appropriate access controls |
6.12 — Pre-Production Validation Checklist
Before a VXLAN EVPN fabric enters production, validate the following operational requirements.
All expected underlay routing
adjacencies are operational.
Every required VTEP loopback
is reachable.
Underlay MTU supports
VXLAN encapsulation.
VLAN-to-VNI mappings
are consistent.
NVE interfaces and
required VNIs are operational.
BGP EVPN neighbors
are Established.
Expected EVPN routes
are received and imported.
Remote MAC forwarding
entries resolve correctly.
Same-subnet connectivity
works across VTEPs.
Inter-subnet routing works
where configured.
Tenant isolation and
security policy are verified.
Link and node failure
recovery has been tested.
Operational Baseline
Capture a known-good baseline of underlay routes, BGP EVPN neighbors, VNI state, MAC forwarding entries and tenant routing tables. Comparing healthy and failed states can substantially reduce incident investigation time.
Section 6 — Engineering Takeaways
✓
VXLAN EVPN deployment
begins with a
reliable routed
IP underlay.
✓
VTEP loopback
reachability,
NVE state and
VNI mappings
must be correct
before overlay
forwarding can
succeed.
✓
BGP EVPN sessions
distribute endpoint
and overlay
reachability,
but Established
neighbors alone
do not prove
forwarding works.
✓
Remote MAC and
tenant routing
tables must be
checked alongside
EVPN control-plane
advertisements.
✓
VXLAN packet
captures reveal
outer VTEP
addresses, UDP
transport and
the overlay VNI.
✓
Route-target
mismatches, incorrect
VNI mappings,
MTU problems
and stale MAC
state are common
investigation areas.
✓
A structured
troubleshooting
workflow isolates
underlay, control-plane
and data-plane
failures before
configuration changes
are made.
Network Insight · Interactive Engineering Lab 06
VXLAN EVPN CLI Verification and Troubleshooting Workbench
Investigate a broken VXLAN fabric using operational commands. Inspect the underlay, BGP EVPN sessions, NVE state, VNI mappings and remote MAC entries. Identify the fault, repair it and verify end-to-end connectivity.
Investigation Objective
Restore Host A → Host B
VXLAN Fabric · VNI 10010 · AS 65001
Underlay Reachability
Failed
EVPN Route State
Installed
End-to-End Traffic
Not Tested
LEAF-A · Simulated NX-OS CLI
Operational Verification Console
Network Insight VXLAN EVPN Investigation Lab
Leaf-A#
Type help to list available commands.
Quick Verification Commands
Verification Checklist
| Check | Operational State |
|---|---|
| Underlay VTEP route | Missing |
| BGP EVPN neighbor | Established |
| NVE interface | Up |
| VNI consistency | Matched |
| Remote MAC route | Installed |
Incident Investigation Notes
Underlay Reachability Failure
The routed underlay has lost reachability to the remote VTEP. Inspect the route table and test connectivity to 10.0.0.12.
Start with show ip route 10.0.0.12.
Engineering Investigation Principle
Verify VXLAN systematically: underlay IP reachability first, then BGP EVPN control-plane state, NVE peer status, VNI configuration, remote MAC/IP reachability and finally end-to-end traffic. A successful BGP session alone does not guarantee working VXLAN forwarding.
Section 07 · Production Fabric Architecture
VXLAN Design Best Practices, Security and Production Readiness
VXLAN EVPN provides a scalable framework for extending Layer 2 networks and building distributed Layer 3 overlays. However, deploying the technology successfully requires careful planning around segmentation, resilience, control-plane security, operational scale and automation.
The goal of production design is not simply to establish VXLAN connectivity. It is to build a fabric that remains predictable during failures, supports controlled tenant communication and provides sufficient visibility to diagnose problems quickly.
Fabric Architecture
Segmentation
EVPN Security
High Availability
Scale
Automation
Observability
Production Readiness
7.1 — Principles of a Production VXLAN Fabric
A production VXLAN EVPN fabric should separate physical transport from tenant forwarding and use consistent design standards across all participating leaf switches.
This separation allows the underlay to focus on resilient IP transport while the overlay manages tenant segments, endpoint reachability and distributed routing.
PRINCIPLE 01
Keep the Underlay Simple
Use a predictable routed leaf-spine design with clear loopback addressing, ECMP and resilient VTEP reachability.
PRINCIPLE 02
Standardize Overlay Mappings
Document VLAN, L2 VNI, VRF, L3 VNI and route-target assignments.
PRINCIPLE 03
Limit Failure Domains
Avoid unnecessary Layer 2 extension and design explicit boundaries for tenant and application networks.
PRINCIPLE 04
Engineer for Visibility
Monitor underlay routing, EVPN advertisements, VNI state, endpoint mobility and forwarding behavior.
Design Objective
A well-designed fabric should make failures easier to isolate, not merely provide more network scale. Clear separation between underlay, EVPN control plane and tenant forwarding is central to operational simplicity.
7.2 — Reference Production Fabric Architecture
The following conceptual topology illustrates two tenants sharing a routed leaf-spine underlay while using separate overlay routing contexts.
Reference Architecture · Shared Underlay, Isolated Overlays
Conceptual multi-tenant design.
The same physical leaf-spine
underlay carries multiple
VXLAN overlays. Tenant
isolation is provided
through separate forwarding
contexts and controlled
EVPN route import.
The drawing is logical
rather than a complete
physical cabling diagram.
7.3 — Segmentation and Tenant Isolation
VXLAN enables multiple logical networks to share the same physical infrastructure. VNIs identify Layer 2 segments, while VRFs provide separate Layer 3 routing contexts.
This allows applications or tenants to maintain independent forwarding domains without requiring dedicated physical network hardware for every environment.
| Segmentation Layer | Mechanism | Purpose |
|---|---|---|
| Layer 2 | L2 VNI | Separate Ethernet bridge domains |
| Layer 3 | VRF and L3 VNI | Separate tenant routing contexts |
| EVPN control plane | Route targets | Control which routes are imported |
| Security policy | ACLs / Firewalls | Restrict permitted application flows |
| Inter-tenant services | Controlled route leaking and service insertion | Permit explicitly approved communication |
VXLAN Does Not Provide Encryption by Default
VXLAN encapsulates Ethernet traffic but does not inherently encrypt the payload or authenticate VTEPs. Where confidentiality or transport integrity is required, additional protections such as IPsec, MACsec or an appropriately secured transport environment must be considered.
7.4 — EVPN Control-Plane Security
BGP EVPN distributes information that influences endpoint reachability and tenant forwarding.
Protecting the EVPN control plane is therefore an important part of fabric security.
BGP PEERING
Restrict Neighbor Access
Limit EVPN BGP peering to approved network devices and routing endpoints.
ROUTE POLICY
Control Route Import
Validate route targets, tenant boundaries and route-leaking requirements.
MANAGEMENT
Protect Management Access
Use authenticated, authorized and monitored access for configuration and operational tools.
MONITORING
Detect Unexpected Changes
Track unusual EVPN advertisements, MAC mobility and unexpected routing changes.
Security Boundary
Route targets are route distribution controls, not a substitute for firewall enforcement. A complete security design must also account for endpoint access, policy enforcement and authorized connectivity between routing domains.
7.5 — High Availability and Failure Domains
Leaf-spine architectures are commonly designed with redundant physical paths.
Equal-Cost Multipath routing can distribute underlay traffic across available next hops, while the routing protocol reacts to link or node failures.
VXLAN EVPN overlays depend on this underlay resilience to maintain connectivity between VTEPs.
| Failure Scenario | Expected Design Behavior |
|---|---|
| Single spine link failure | Underlay converges onto remaining valid paths |
| Spine switch failure | Reachability continues through surviving spine paths where capacity permits |
| Leaf failure | Dual-attached endpoints may retain service through an alternate leaf, depending on multihoming design |
| EVPN peer failure | Redundant control-plane peers support route distribution continuity |
| Endpoint migration | EVPN MAC mobility procedures update endpoint reachability |
Resilience Must Be Tested
Redundant topology does not guarantee zero packet loss or instantaneous convergence. Validate actual failure recovery, forwarding behavior, convergence time and available bandwidth during degraded operation.
7.6 — EVPN Scale and Capacity Planning
VXLAN supports a 24-bit VNI field, but the theoretical identifier space should not be confused with the practical scale supported by a specific network platform.
Real fabric scale is influenced by hardware forwarding capacity, EVPN route count, MAC and ARP table resources, tenant VRFs and operational control-plane limits.
| Scale Dimension | What to Measure |
|---|---|
| VNIs | Supported L2 and L3 VNI capacity |
| MAC entries | Local and remote MAC forwarding state |
| EVPN routes | Route count, memory and convergence load |
| ARP / ND entries | Endpoint neighbor resolution state |
| VRFs | Supported tenant routing instances |
| ECMP paths | Hardware path capacity and hash distribution |
| BUM replication | Replication fan-out and bandwidth impact |
7.7 — Automation and Configuration Consistency
Manual VXLAN EVPN deployments can become difficult to maintain when many leaf switches, tenants and VNIs are involved.
Automation helps maintain consistent configuration standards and reduces the risk of mapping or route-target errors.
01
Define the Intended Network State
Maintain a source
of truth for
devices, loopbacks,
VLANs, VNIs,
VRFs and route
targets.
02
Generate Consistent Configurations
Use templates
or automation
tooling to
apply approved
fabric design
standards.
03
Validate Before Deployment
Check duplicate
identifiers,
missing mappings,
route-target
inconsistencies
and unsupported
platform features.
04
Compare Intended and Actual State
Verify operational
NVE, BGP EVPN,
routing and
forwarding state
after each
deployment.
Automation Principle
Configuration deployment is only one part of automation. Production workflows should also validate the resulting network state and detect drift from the intended fabric design.
7.8 — VXLAN Observability and Analytics
Modern data centre operations require visibility across the underlay, EVPN control plane and tenant data plane.
Useful telemetry includes interface utilization, ECMP distribution, route changes, endpoint mobility, packet drops and tunnel health.
| Telemetry Area | Operational Value |
|---|---|
| Underlay interfaces | Detect congestion, errors and link failures |
| Routing adjacencies | Identify unstable underlay or EVPN sessions |
| EVPN route changes | Correlate route withdrawals with service disruption |
| MAC mobility | Detect workload movement or unexpected MAC flapping |
| VNI statistics | Monitor overlay forwarding and traffic distribution |
| Queue depth / drops | Identify congestion and microburst effects |
| Packet traces | Validate actual encapsulation and delivery paths |
Analytics Do Not Automatically Optimize VXLAN
Telemetry provides information about network behavior. Automated optimization requires additional control systems, supported platform capabilities and explicit operational policy. VXLAN encapsulation itself does not automatically rebalance or optimize traffic.
7.9 — Recommended Engineering Validation Commands
The following Cisco NX-OS-style commands provide a useful starting point for production-state validation.
VXLAN EVPN Production Verification
show ip route show ip interface brief show nve interface show nve peers show nve vni show bgp l2vpn evpn summary show bgp l2vpn evpn route-type 2 show bgp l2vpn evpn route-type 3 show bgp l2vpn evpn route-type 5 show l2route evpn mac all show mac address-table show vrf show ip route vrf TENANT-A show interface counters errors
Illustrative verification commands.
Exact syntax, feature support
and output depend on the
hardware platform and
software release.
7.10 — Production Architecture Decision Matrix
Not every VXLAN EVPN deployment needs the same design choices. The correct architecture depends on workload mobility, tenant isolation, operational scale and resilience requirements.
| Decision | Recommended Evaluation |
|---|---|
| Underlay protocol | Choose based on operational familiarity, scale and convergence requirements |
| EVPN peering model | Evaluate route reflectors, redundancy and control-plane scale |
| IRB model | Compare symmetric and asymmetric forwarding requirements |
| Tenant segmentation | Define VRF boundaries, route import and firewall enforcement |
| BUM handling | Select supported ingress replication or multicast transport models |
| High availability | Evaluate dual attachment, multihoming and failover behavior |
| Automation | Establish a source of truth and validation workflow |
| Observability | Define operational metrics, baselines and alert thresholds |
7.11 — Production Readiness Checklist
Underlay routing is resilient
and VTEP reachability verified.
VLAN, VNI and VRF mappings
are documented and consistent.
EVPN peering and route-target
policies are validated.
Tenant isolation and
permitted route leaking
are tested.
BGP and management access
are appropriately protected.
Hardware MAC, route,
VNI and VRF scale
is understood.
Underlay MTU accommodates
VXLAN overhead.
Failure recovery and
degraded-capacity operation
are tested.
Monitoring covers underlay,
EVPN and data-plane state.
Automation includes
pre-deployment and
post-deployment validation.
Section 7 — Engineering Takeaways
✓
Production VXLAN
fabrics require
consistent design
across underlay,
overlay and
tenant routing.
✓
VNIs, VRFs and
route targets
support logical
segmentation,
but security
policy must
be enforced
separately.
✓
VXLAN does not
inherently provide
encryption or
endpoint authentication.
✓
Fabric scale is
constrained by
actual platform
resources, not
simply the size
of the VNI field.
✓
Redundancy,
ECMP and EVPN
convergence
should be
validated under
realistic failure
conditions.
✓
Automation and
telemetry improve
consistency and
operational visibility
when combined with
explicit validation
and control policies.
Network Insight · Interactive Engineering Lab 07
VXLAN EVPN Fabric Design and Production Readiness
Act as the network architect. Configure resilience, security, routing and observability controls, test failure scenarios and evaluate whether your VXLAN EVPN fabric is ready for production.
Dual-Spine EVPN VXLAN Fabric
Design Assessment
Design Readiness Score
0%
Engineering Controls Enabled
0 / 8
Production Assessment
Review
1. Configure the VXLAN Fabric
Enable or disable individual design controls. Each decision affects the engineering readiness assessment and selected resilience tests.
01 · ECMP Underlay Resilience
Maintain multiple equal-cost routed paths between VTEPs through independent spines.
02 · BGP EVPN Control Plane
Distribute MAC/IP reachability and VNI membership through BGP EVPN.
03 · Underlay MTU Headroom
Allow sufficient MTU for VXLAN encapsulation without unwanted fragmentation.
04 · Tenant VRF Isolation
Separate tenant routing domains and control inter-VRF connectivity.
05 · Distributed Anycast Gateway
Provide consistent default gateways on participating leaf switches.
06 · Tenant Security Policy
Apply explicit filtering and restrict unauthorized tenant-to-tenant traffic.
07 · Fabric Telemetry
Monitor BGP, NVE, interfaces, MAC mobility and forwarding health.
08 · ARP Suppression
Use verified EVPN MAC/IP bindings to reduce unnecessary ARP flooding.
2. Run Production Failure Tests
Select an engineering scenario to assess how your chosen design would behave.
Fabric Design Assessment and Event Report
[READY] Fabric architecture loaded.
[READY] Configure design controls.
[READY] Run a failure scenario or generate a report.
Design Assessment Ready
Enable the appropriate engineering controls and run resilience tests to evaluate your VXLAN EVPN fabric.
Section 08 · Advanced VXLAN EVPN Architecture
VXLAN EVPN Multi-Site, Data Centre Interconnect and Migration
VXLAN EVPN is not limited to a single leaf-spine fabric. With an appropriate multi-site architecture, organizations can connect data centres, extend selected network segments, distribute routing information and support workload mobility across geographically separate environments.
However, interconnecting VXLAN fabrics introduces design challenges that do not exist within a single data centre. These include inter-site latency, failure isolation, route distribution, gateway placement, broadcast containment and disaster recovery.
The objective is to extend only the network services that require cross-site connectivity, while preserving predictable routing and independent site operation.
VXLAN EVPN Multi-Site
Data Centre Interconnect
Border Gateways
Workload Mobility
Distributed Anycast Gateway
Layer 2 Extension
Migration
Disaster Recovery
8.1 — Why Extend VXLAN Between Data Centres?
Enterprises often operate multiple data centres for capacity, availability, geographic coverage or disaster recovery.
VXLAN EVPN can form part of a DCI design that supports selected Layer 2 extensions and Layer 3 connectivity between sites.
USE CASE 01
Workload Mobility
Support workloads that require selected network identities or subnets to remain reachable across sites during planned migration.
USE CASE 02
Disaster Recovery
Provide network connectivity for applications and services activated in an alternate data centre.
USE CASE 03
Application Distribution
Enable controlled communication between application components located in different data centres.
USE CASE 04
Infrastructure Modernization
Connect legacy networks with modern EVPN fabrics during a phased migration.
Design Recommendation
Prefer routed inter-site connectivity when application requirements allow it. Extend Layer 2 only where there is a clear operational or application dependency. Unnecessary Layer 2 extension increases the scope of broadcast and failure domains.
8.2 — VXLAN EVPN Multi-Site Architecture
A multi-site architecture connects separate data centre fabrics through an inter-site transport network.
Depending on the vendor architecture, dedicated or designated border gateways can exchange overlay reachability and forward VXLAN traffic between sites.
Reference Architecture · Two VXLAN EVPN Data Centres
Site underlay
Inter-site overlay
Tenant attachment
Simplified logical architecture.
Border gateways may be
implemented as dedicated
devices or designated
leaf roles. Actual multi-site
designs require additional
redundancy, routing policy
and vendor-specific
inter-site mechanisms.
8.3 — The Role of Border Gateways
Border gateways provide a controlled point of connectivity between a local VXLAN fabric and the DCI transport network.
In supported multi-site architectures, they can exchange EVPN reachability information and forward encapsulated traffic between participating sites.
| Border Gateway Function | Engineering Purpose |
|---|---|
| Inter-site forwarding | Carry selected overlay traffic between fabrics |
| EVPN route exchange | Advertise or import permitted endpoint and tenant routes |
| Policy enforcement | Restrict inter-site route distribution and connectivity |
| BUM containment | Control replication behavior between participating sites |
| Failure isolation | Limit the impact of remote site or transport disruptions |
| Redundancy | Support resilient inter-site paths when multiple gateways are deployed |
Multi-Site Is Architecture-Specific
The term VXLAN EVPN Multi-Site is also used for particular vendor implementations. Border-gateway roles, control-plane behavior, anycast addressing, BUM replication and failure handling vary by platform and design. Verify the supported architecture rather than assuming that two independent EVPN fabrics can simply be connected without additional configuration.
8.4 — Layer 2 Extension Versus Layer 3 Interconnect
One of the most important DCI decisions is whether a subnet must extend across sites.
Layer 2 extension allows endpoints in separate data centres to participate in the same bridge domain.
Layer 3 interconnect instead routes between distinct subnets and routing domains.
| Design Characteristic | Layer 2 Extension | Layer 3 Interconnect |
|---|---|---|
| Subnet model | Shared subnet | Separate subnets |
| Broadcast domain | Extends between sites | Remains site-local |
| Workload mobility | Can preserve an IP subnet during migration | Usually requires address or application-level mobility planning |
| Failure domain | Potentially wider | More clearly bounded |
| Operational complexity | Greater dependency on MAC and BUM behavior | Primarily routed reachability and policy |
| Typical preference | Specific mobility requirements | General-purpose inter-site connectivity |
Recommended Architecture
Use Layer 3 boundaries by default and introduce stretched Layer 2 segments only when justified. This reduces unnecessary broadcast extension and simplifies site-level fault isolation.
8.5 — Distributed Anycast Gateway Across Sites
A distributed anycast gateway allows multiple participating leaf switches to provide a common default gateway identity for a tenant subnet.
In a supported multi-site design, this can allow workloads to use local first-hop routing even when the subnet is present at multiple sites.
Gateway consistency is critical. IP addressing, gateway MAC behavior, EVPN advertisements and tenant routing policies must follow the selected architecture.
LOCAL ROUTING
First-Hop Gateway
The local leaf can route traffic using the distributed gateway rather than sending every packet to a remote site.
CONSISTENCY
Shared Gateway Identity
Gateway addresses and MAC behavior must be consistent with the supported anycast gateway implementation.
ROUTING POLICY
Symmetric IRB
Symmetric integrated routing and bridging can support scalable routed tenant connectivity through L3 VNIs.
FAILURE RECOVERY
Site Independence
Design gateway and route behavior so local services can continue where feasible during DCI failure.
8.6 — Workload Mobility and MAC Mobility
EVPN includes mechanisms for advertising MAC reachability and signaling endpoint mobility.
When a workload moves from one VTEP to another, the control plane must update reachability so remote devices forward traffic toward the new location.
01
Workload Located in Site A
The local VTEP
learns the endpoint
and advertises
the appropriate
EVPN MAC/IP
reachability.
02
Workload Migrates to Site B
The endpoint
becomes attached
to a VTEP
in the
destination site.
03
EVPN Reachability Changes
The new location
advertises updated
reachability.
MAC mobility
procedures help
remote devices
distinguish the
newer location.
04
Remote Forwarding Updates
Participating
VTEPs update
forwarding state
to direct traffic
toward the
destination site.
05
Validate Application Continuity
Test routing,
ARP/ND state,
firewall sessions,
load balancers
and application
dependencies.
Mobility Does Not Guarantee Session Continuity
EVPN can update network reachability after endpoint movement, but application continuity also depends on transport sessions, firewall state, load balancing, storage and application architecture. VXLAN alone does not guarantee seamless migration.
8.7 — Inter-Site Failure and Disaster Recovery
DCI failure must be considered separately from failures inside a single data centre fabric.
Each site should have a defined behavior when the inter-site network becomes unavailable.
| Failure | Potential Impact | Design Consideration |
|---|---|---|
| DCI link failure | Remote overlays become unreachable | Local routing independence and alternate paths |
| Border gateway failure | Inter-site forwarding is disrupted | Redundant gateways and convergence |
| Complete site outage | Workloads and network services may be unavailable | Recovery orchestration, route changes and application failover |
| Control-plane instability | EVPN route churn or inconsistent forwarding | Route filtering, monitoring and failure isolation |
| Extended Layer 2 fault | Broadcast or endpoint-learning disruption | Minimize stretched segments and control BUM traffic |
Split-Brain Risk
When two sites lose communication but continue operating independently, application and network failover mechanisms must avoid conflicting active ownership. VXLAN EVPN connectivity is not a substitute for application-level quorum, storage consistency or disaster recovery coordination.
8.8 — Migrating from Traditional VLAN Networks
Moving from a traditional Layer 2 network to VXLAN EVPN is usually best performed incrementally.
A phased migration allows engineers to validate the new fabric while preserving existing application connectivity.
01
Assess the Existing Network
Document VLANs,
subnets, default
gateways, trunks,
spanning tree,
routing and
application
dependencies.
02
Build and Validate the Underlay
Establish routed
leaf-spine
connectivity,
loopback reachability,
ECMP and MTU
consistency.
03
Deploy the EVPN Overlay
Configure VTEPs,
L2 VNIs,
tenant VRFs,
L3 VNIs and
BGP EVPN
route exchange.
04
Establish a Controlled Migration Boundary
Use supported
Layer 2 or
Layer 3 handoff
mechanisms between
legacy and
EVPN environments.
Avoid uncontrolled
loops and
duplicate gateways.
05
Migrate a Pilot Workload
Test a limited
application or
subnet, validate
endpoint learning
and confirm
expected routing
and security policy.
06
Expand in Controlled Phases
Migrate additional
segments using
documented change
windows, validation
tests and
rollback plans.
07
Remove Legacy Dependencies
Retire unnecessary
VLAN extensions,
legacy trunks
and obsolete
gateway paths
after successful
validation.
8.9 — Example Migration Mapping
A VLAN-to-VNI mapping plan provides a consistent reference when moving network segments into the EVPN fabric.
| Existing VLAN | New L2 VNI | Tenant VRF | Migration Approach |
|---|---|---|---|
| 10 | 10010 | TENANT-A | Pilot application migration |
| 20 | 10020 | TENANT-A | Phased server migration |
| 30 | 20030 | TENANT-B | Separate tenant cutover |
| 40 | 20040 | TENANT-B | Gateway migration after validation |
Gateway Migration Requires Special Care
Moving the default gateway from a traditional switch to a distributed anycast gateway must be planned carefully. Duplicate active gateway ownership, inconsistent ARP behavior or incorrect routing boundaries can disrupt application traffic during cutover.
8.10 — Multi-Site Operational Verification
Engineers should verify both the local EVPN fabric and the inter-site transport.
The following commands illustrate common checks on Cisco NX-OS VXLAN EVPN deployments.
VXLAN EVPN / DCI Verification
show ip route show nve interface show nve peers show nve vni show bgp l2vpn evpn summary show bgp l2vpn evpn route-type 2 show bgp l2vpn evpn route-type 3 show bgp l2vpn evpn route-type 5 show l2route evpn mac all show mac address-table show vrf show ip route vrf TENANT-A
Generic NX-OS-style commands.
Vendor-specific multi-site
features may require
additional border-gateway,
site identifier, route-policy
and inter-site peer
verification commands.
Availability varies by
software release.
| Validation Layer | What to Confirm |
|---|---|
| Local underlay | VTEP reachability and stable routing adjacencies |
| DCI transport | Border gateway connectivity, MTU and permitted traffic |
| EVPN control plane | Expected inter-site route exchange |
| Route policy | Correct import, export and tenant boundaries |
| Data plane | Encapsulation, remote VTEP forwarding and endpoint delivery |
| Failure recovery | Behavior during DCI or border gateway outage |
8.11 — Multi-Site Design Checklist
Each site has a validated
local underlay and overlay.
DCI routing, MTU and
transport capacity
are verified.
Border gateway redundancy
and failover are tested.
Inter-site EVPN route
distribution follows
approved policy.
Layer 2 extension is
limited to justified
application requirements.
Anycast gateway behavior
is consistent across
participating sites.
MAC mobility and
endpoint movement
are tested.
Site isolation and
DCI failure behavior
are documented.
Application and
firewall dependencies
are included in
migration testing.
Migration rollback
and disaster recovery
procedures are validated.
Section 8 — Engineering Takeaways
✓
VXLAN EVPN can
support selected
inter-site Layer 2
and Layer 3
connectivity when
deployed using
a supported
DCI architecture.
✓
Border gateways
can provide
controlled inter-site
forwarding and
route exchange.
✓
Routed interconnects
should generally
be preferred over
unnecessary Layer 2
extension.
✓
Distributed anycast
gateways and EVPN
MAC mobility
can support
workload movement,
but application
continuity requires
additional planning.
✓
Multi-site designs
must account for
DCI failures,
site isolation,
route policy
and disaster
recovery coordination.
✓
Legacy VLAN
migration should
follow a phased
approach with
explicit gateway
ownership, validation
and rollback procedures.
Network Insight · Interactive Engineering Lab 08
VXLAN EVPN Multi-Site, Data Centre Interconnect and Migration Lab
Connect two data centres, compare inter-site architectures, investigate DCI failures and plan a controlled migration from legacy Layer 2 networking to VXLAN EVPN.
Step 1 · Select the Inter-Site Architecture
Two-Site VXLAN EVPN Fabric
EVPN Multi-Site
DCI Architecture
EVPN Multi-Site
Inter-Site Paths
2 Available
Traffic Result
Ready
Step 2 · Inject a Failure
Both DCI paths are operational.
Architecture Analysis
EVPN Multi-Site
Border gateways connect independent EVPN fabrics while maintaining inter-site reachability for selected tenant networks.
Inter-Site Packet and Failure Investigation Timeline
[READY] DC1 and DC2 initialized.
[READY] EVPN Multi-Site selected.
[READY] Both DCI paths available.
[READY] Trace an inter-site packet.
Step 3 · Migration Planning
Legacy Layer 2 to VXLAN EVPN Migration
Work through five controlled migration stages. Review the required checks before approving each stage.
Migration Stage 01 / 05
Discovery and Dependency Mapping
Identify VLANs, gateways, IP subnets, application dependencies and existing failure domains before migration.
Complete the stage checklist before approval.
Engineering Design Principle
Extending Layer 2 between data centres increases the scope of broadcast and failure domains. Routed DCI is generally preferable when applications do not require subnet continuity. EVPN Multi-Site can support selected extensions, but border-gateway redundancy, loop prevention, failure isolation and operational validation remain essential.
Section 09 · Final Engineering Summary
VXLAN: Engineering Summary, Key Takeaways and Next Steps
VXLAN has become an important building block of modern data centre network architecture. By transporting Ethernet segments across an IP underlay, VXLAN enables scalable overlay networks without requiring the physical fabric to operate as one large Layer 2 domain.
When combined with BGP EVPN, VXLAN supports control-plane distribution of endpoint reachability, multi-tenant segmentation, distributed routing and advanced leaf-spine network designs.
This final section brings together the central engineering concepts explored throughout the article.
Underlay
Overlay
VTEPs
Encapsulation
BGP EVPN
Tenant Segmentation
Operations
Multi-Site
9.1 — The Complete VXLAN Architecture
A VXLAN EVPN fabric can be understood through four closely related architectural components.
FOUNDATION 01
IP Underlay
The routed physical network provides IP reachability between VTEPs. Routing protocols and ECMP support resilient transport.
FOUNDATION 02
VXLAN Overlay
VXLAN encapsulates Ethernet frames in UDP packets. The VNI identifies the associated overlay segment.
FOUNDATION 03
BGP EVPN Control Plane
EVPN distributes endpoint and routing information, reducing reliance on traditional data-plane learning across the overlay.
FOUNDATION 04
Distributed Tenant Routing
VRFs, L2 VNIs, L3 VNIs and distributed gateways enable tenant-aware forwarding and inter-subnet routing.
Architecture Recap · From Workload to Workload
Conceptual same-subnet VXLAN
forwarding. Leaf A encapsulates
the original Ethernet frame,
the IP underlay transports
it to Leaf B, and Leaf B
decapsulates it before
delivering the frame
to Host B.
9.2 — Traditional VLAN Networks Versus VXLAN EVPN
Traditional VLAN networks and VXLAN EVPN fabrics solve different scaling and operational challenges.
| Characteristic | Traditional VLAN Network | VXLAN EVPN Fabric |
|---|---|---|
| Segmentation identifier | 12-bit VLAN ID | 24-bit VNI |
| Transport model | Ethernet switching | Overlay over routed IP |
| Loop avoidance | Often relies on spanning tree | Routed underlay avoids Layer 2 loops in the core |
| Multipath forwarding | Depends on Layer 2 design | Underlay ECMP can use multiple routed paths |
| Endpoint reachability | Primarily data-plane MAC learning | EVPN control-plane advertisements plus local learning |
| Tenant routing | Traditional SVI/router model | VRFs and distributed IRB |
| Multi-site connectivity | Requires suitable DCI technology | Supported EVPN DCI architectures |
| Operational complexity | Simpler for small deployments | More components and control-plane state to manage |
Important Distinction
VXLAN is the data-plane encapsulation technology. BGP EVPN is a separate control-plane technology commonly used to distribute endpoint and routing information. They work together, but they are not the same protocol.
9.3 — The End-to-End Packet Journey
Understanding a complete forwarding sequence is one of the most useful skills for troubleshooting VXLAN networks.
01
Ingress
Host A sends
an Ethernet
frame to Leaf A.
02
Lookup
Leaf A resolves
the destination
MAC and remote
VTEP mapping.
03
Encapsulation
Leaf A adds
VXLAN, UDP
and outer
IP headers.
04
Transport
The underlay
routes traffic
toward the
remote VTEP.
05
Delivery
Leaf B removes
encapsulation
and forwards
the original
frame.
Engineering Mental Model
When a VXLAN packet fails, investigate the path in layers: local VLAN and VNI mapping, endpoint learning, EVPN reachability, remote VTEP connectivity, underlay routing, encapsulation and destination forwarding.
9.4 — The Most Important VXLAN Design Decisions
Successful deployments depend on a series of architecture choices rather than VXLAN encapsulation alone.
DESIGN 01
Underlay Routing
Select a suitable routing protocol, define loopback addressing and validate ECMP and convergence.
DESIGN 02
VNI and VRF Allocation
Maintain consistent mappings and clear tenant routing boundaries.
DESIGN 03
EVPN Route Policy
Define route import and export behavior, route reflectors and supported EVPN route types.
DESIGN 04
Distributed Gateway
Choose a supported IRB architecture and validate first-hop gateway behavior.
DESIGN 05
Security and Segmentation
Use VRFs, route policy and security enforcement to control tenant connectivity.
DESIGN 06
Operational Visibility
Monitor EVPN sessions, VTEPs, MAC mobility, underlay health and packet drops.
9.5 — Production Engineering Checklist
Before deploying a VXLAN EVPN fabric into production, validate the following core requirements.
Routed underlay provides
reliable VTEP reachability.
Underlay MTU supports
VXLAN encapsulation overhead.
VLAN-to-VNI mappings
are correct and consistent.
EVPN BGP sessions
are established and stable.
Expected MAC/IP
advertisements are present.
VRF and route-target
policies are validated.
Distributed gateway
behavior is tested.
Tenant isolation and
security policies work.
ECMP forwarding and
failover are verified.
MAC mobility and
endpoint movement
are tested where required.
Hardware forwarding
and route scale
are understood.
Monitoring, rollback
and recovery procedures
are documented.
Final Security Reminder
VXLAN does not inherently encrypt traffic or authenticate tunnel endpoints. Route targets are not firewall rules, and logical segmentation alone does not guarantee a complete security boundary. Apply appropriate transport protection and traffic enforcement according to your security requirements.
9.6 — Final Engineering Takeaways
What You Should Remember About VXLAN
✓
VXLAN extends
Layer 2 segments
across a routed
IP infrastructure.
✓
The 24-bit VNI
supports a much
larger segmentation
identifier space
than traditional
VLAN IDs.
✓
VTEPs perform
encapsulation
and decapsulation
at the overlay
boundary.
✓
BGP EVPN provides
a scalable
control plane
for distributing
MAC/IP and
routing information.
✓
VRFs, L2 VNIs,
L3 VNIs and
distributed gateways
support multi-tenant
routing and
segmentation.
✓
Production VXLAN
requires sound
underlay design,
MTU planning,
security controls,
monitoring and
failure testing.
✓
Multi-site EVPN
can support
selected DCI
and workload
mobility requirements,
but should not
replace careful
application and
disaster recovery
design.
9.7 — Continue Your VXLAN Learning Journey
Understanding VXLAN architecture is the first step. The next stage is applying these concepts through topology investigation, packet tracing and realistic network engineering scenarios.
Network Insight · Continue Learning
Move from VXLAN Theory to Network Engineering Practice
Explore the wider data centre networking architecture, including leaf-spine design, BGP, routing, traffic investigation and modern overlay technologies.
End of VXLAN Editorial Learning Path.
You have completed the nine-section
progression from traditional
Ethernet limitations to
production VXLAN EVPN
architecture, operations
and multi-site design.
Network Insight · Interactive Engineering Lab 09
VXLAN EVPN Engineering Capstone: Diagnose, Repair and Validate
Complete six engineering investigations covering underlay architecture, VXLAN encapsulation, EVPN control-plane routes, distributed routing, troubleshooting and production design. Use the simulated CLI, select your engineering decisions and complete the final assessment.
Engineering Score
0 / 600
Stages Completed
0 / 6
Current Stage
01
Assessment Status
In Progress
Engineering Investigation Fabric
Investigation Active
Stage 01 / 06 · Underlay Architecture
Verify the VXLAN Underlay
The EVPN overlay depends on routed IP reachability between VTEPs. Investigate the fabric before making your engineering decision.
Which underlay condition is required
before VXLAN encapsulated traffic
can be forwarded?
Investigation Ready
Review the evidence and select the most appropriate engineering decision.
Simulated Operational CLI
Use NX-OS-style verification commands
to inspect the current scenario.
Type help for available commands.
LEAF-A · Engineering Console
Network Insight VXLAN EVPN Capstone
Leaf-A#
Engineering Investigation Timeline
[READY] Capstone initialized.
[READY] Stage 01 loaded.
[READY] Review evidence and submit your decision.
Final Engineering Assessment
VXLAN EVPN Capstone Complete
0 / 600
Support Network Insight
Help support the development of interactive networking labs,
BGP simulations, and educational content.
Network Insight · Continue Your Learning Journey
From VXLAN Theory to Data Centre Engineering
You've explored VXLAN encapsulation, BGP EVPN, distributed routing and fabric design. Now take the next step into an interactive data centre networking environment.
Data Centre
Workspace Explore · Trace · Diagnose
Workspace Explore · Trace · Diagnose
Leaf-Spine
Fabrics
Fabrics
VXLAN
Overlays
Overlays
BGP
Routing
Routing
Traffic
Investigation
Investigation
Fabric
Resilience
Resilience
Explore network architecture, follow packet paths, investigate failures and practise engineering decisions through guided interactive activities.
Explore the Data Centre Network Workspace
Network Insight · Interactive Learning Workspace
From theory to interactive practice
Don't Just Read the Network. Explore It.
You've explored data centre architecture, routing and resilient design. Now take the next step. Use the interactive simulator to investigate how network components and paths work together. Move beyond diagrams and start exploring network behaviour.
Network topology
Traffic paths
Routing decisions
Failure scenarios
LAUNCH THE BGP CORE SIMULATOR
Opens in a new browser tab.
Network in motion
Illustrative
STEP 01 / OBSERVE
Inspect the network
Explore the topology, devices and available network state.
STEP 02 / TRACE
Follow the behaviour
Investigate routes, forwarding decisions and the paths traffic can take.
STEP 03 / INVESTIGATE
Think like an engineer
Use what you observe to reason about changes, redundancy and failures.
The topology above is illustrative. Open the simulator to explore the functionality it provides.
Routing Convergence
Network Engineering Guide
Routing Convergence: From Failure Detection to Forwarding Recovery
Routing convergence determines how a network responds when its topology changes. When a link fails, a router becomes unreachable, or a route is withdrawn, routing protocols must react, select usable paths, and update forwarding state. Understanding each stage helps engineers reduce disruption and diagnose why traffic does not recover as quickly as expected.
A network is not resilient simply because it has redundant links or multiple routing protocols. Resilience depends on how effectively the network detects a failure, communicates the change, calculates an alternative path, and installs the resulting forwarding decision. Until those operations complete, packets may be lost, delayed, or sent along a path that is no longer usable.
Different protocols approach these tasks in different ways. OSPF and IS-IS distribute link-state information and calculate paths using SPF algorithms. EIGRP can use a feasible successor when its loop-free conditions are satisfied. BGP processes path attributes and routing updates, with its behaviour influenced by policy, session state, and route propagation. Their convergence characteristics therefore depend on more than the protocol name alone.
This engineering guide breaks convergence into measurable stages and connects routing-protocol behaviour to the forwarding plane. Through interactive experiments, you will investigate failure detection, topology changes, route selection, alternate paths, and the installation of updated forwarding entries.
The key engineering principle
Convergence time is an outcome of several processes, not one timer. Separate failure detection, control-plane processing, route calculation, and forwarding recovery when measuring performance or troubleshooting an outage.
STAGE 01
Detect
Identify the failed link, neighbour, or forwarding path.
STAGE 02
Propagate
Communicate the topology or reachability change.
STAGE 03
Calculate
Re-evaluate routes and select a usable alternative.
STAGE 04
Install & Forward
Update forwarding state and verify traffic recovery.
Network Insight · Engineering Guide
Routing Convergence: Table of Contents
Follow the complete convergence process, from initial failure detection to forwarding recovery and operational verification. Select a section to jump directly to its introduction.
Section 1 · Fundamentals
Understanding Routing Convergence
Before troubleshooting a routing failure, it is important to understand what convergence actually means. A network must detect a change, process routing information, select usable paths, and update forwarding behaviour. These activities are related, but they are not the same event.
In this section: Control plane versus forwarding plane, the four stages of convergence, and the factors that influence recovery time.
Section 01 · Foundations
Understanding Routing Convergence
Routing convergence is the process through which routing systems respond to a network change and establish usable, consistent routing and forwarding state. It connects topology information, route calculation, and packet delivery—and determines how much disruption users may experience when the network changes.
What does it mean for a network to converge?
Routers build their view of reachable destinations from directly connected networks, static configuration, and information learned through routing protocols. When a link fails or a neighbour becomes unavailable, some of that information may no longer describe the current network. Routing systems must detect the change, process the resulting information, and recalculate affected routes.
A network is considered converged for a particular routing domain when the relevant protocol state has settled and routers have processed the changes needed to make their routing decisions. This does not necessarily mean every router has an identical routing table. Different routers can have different perspectives, policies, and next hops while still operating correctly. The important requirement is that their routing and forwarding behaviour is consistent with the intended network design.
Control plane versus forwarding plane
Understanding convergence starts with separating two related functions. The control plane determines reachability and selects routes. The forwarding plane uses installed forwarding entries to move packets toward their destinations. A routing protocol can detect a failure and calculate an alternative route before every relevant forwarding entry has been updated.
Control plane
Determine the route
Maintains protocol state, exchanges routing information, evaluates route preferences, and selects paths. Examples include OSPF SPF calculations, EIGRP route selection, and BGP best-path processing.
Forwarding plane
Move the packet
Looks up packet destinations in forwarding state, selects an outgoing interface or next hop, and transmits traffic. Hardware and software implementation details vary by platform.
The Routing Information Base (RIB) contains selected routing information. The Forwarding Information Base (FIB) contains destination-to-forwarding information used by the forwarding process. The FIB is derived from routing information and may be programmed into hardware, software, or a combination of both. It is therefore more accurate to describe the FIB as derived forwarding state than simply as a copy of the RIB.
The four stages of convergence
A useful troubleshooting model divides a convergence event into four stages. The stages are closely related and may overlap; they are not always four strictly sequential operations.
01 · DETECT
Recognise failure
Identify a failed link, lost neighbour, or unreachable next hop.
02 · PROPAGATE
Share the change
Notify relevant routing processes and distribute updated information.
03 · CALCULATE
Select a path
Re-evaluate available routes and choose a valid replacement.
04 · INSTALL
Restore forwarding
Update forwarding state and verify that traffic can use the path.
Why the distinction matters
Fast failure detection does not guarantee fast application recovery. Route propagation, route calculation, forwarding-table programming, and the availability of a genuinely independent alternate path can all affect the result.
Convergence is more than a protocol timer
Convergence performance depends on the topology, routing information being processed, protocol configuration, implementation, processor load, and forwarding hardware. A precomputed backup path may allow local repair before the entire routing domain has completed its response. Conversely, a route that is theoretically available may not provide a usable repair if it violates routing policy or loop-free requirements.
For this reason, engineers should avoid treating a single advertised convergence time as a guarantee. Measurements must specify the event being timed, the start and end points, the traffic being tested, and the conditions under which the result was obtained.
What you should be able to explain
- Why routing convergence and forwarding recovery are related but distinct.
- How a topology change affects routing information and selected paths.
- Why the RIB and FIB serve different purposes.
- Why failure detection, route calculation, and forwarding updates should be measured separately.
Engineering takeaway: treat convergence as a chain of observable events. Establish what failed, when the routing system recognised it, when a replacement route became available, and when traffic successfully resumed.
Network Insight · Interactive Lab 01
Routing Convergence: Follow a Failure to Recovery
Inject a link failure and follow the sequence from detection to route recalculation and forwarding recovery. Step through the events to see why convergence is more than a single timer.
PRIMARY PATH ACTIVE
Destination prefix: 10.40.40.0/24
Network topology
All links operational
Primary path
Alternate path
Failed link
Convergence event
Stage 0 / 7
READY
Primary route is forwarding
R1 forwards traffic towards R4 through R2. The alternate path through R3 is available in this conceptual topology.
Selected next hopR2
Forwarding statePrimary path
Failure detectionNot triggered
Route statusStable
This is a staged teaching model, not a vendor-specific timer or a real packet-forwarding test. Event order and convergence time vary by protocol and implementation.
Route and forwarding state
R1 · destination prefix
| State | Next hop | Interpretation |
|---|---|---|
| Selected route | R2 | Primary route selected |
| Forwarding entry | R2 | Traffic uses primary path |
| Alternate route | R3 | Available as the backup path |
Event timeline
0 / 7 events
1
Failure occursPrimary R2–R4 link goes down
2
Failure detectedR2 recognises the adjacency failure
3
Routing information propagatesR1 learns that the primary route is affected
4
Route recalculationR1 selects the eligible alternate via R3
5
RIB selection changesThe selected route points to R3
6
FIB updateForwarding state is updated
7
Forwarding verifiedTraffic can use R1 → R3 → R4
Engineering takeaway: Routing convergence includes failure detection, propagation, route selection and forwarding-state update. A changed route in the control plane does not, by itself, prove that traffic is already forwarding over the new path.
Section 2 · Failure Detection
Failure Detection and Propagation
Routing recovery begins when a router recognises that something has changed. Physical interface events, protocol timers, and mechanisms such as Bidirectional Forwarding Detection (BFD) can help identify failures. The routing protocol must then communicate the resulting change to the routers that need to react.
In this section: Failure-detection mechanisms, protocol notifications, convergence timelines, and the trade-offs involved in timer tuning.
Section 02 · Failure Mechanics
Failure Detection and Propagation
Before a routing protocol can select a replacement path, the network must recognise that something has changed. How quickly a failure is detected—and how reliably that information reaches the routing processes that need it—strongly influences convergence and traffic disruption.
Failure detection is the starting point
A network failure is not always immediately obvious to the routing protocol. A physical interface going down can generate a direct local event. A remote router becoming unreachable, a silent packet drop, or a one-way communication failure may require additional detection mechanisms before the system can declare a neighbour or path unavailable.
The detection method depends on the failure being considered. Physical carrier state, routing-protocol hello or keepalive messages, hold timers, and Bidirectional Forwarding Detection (BFD) can all contribute to failure detection. They are not interchangeable, and each has different assumptions and operational trade-offs.
Interface or link state
A router may learn immediately that its local interface has lost carrier or been disabled. Detection is less direct when the underlying fault does not change the interface state.
Neighbour messages
Routing protocols use periodic messages and associated timers to establish or maintain adjacency and session state. A missed message is meaningful only in the context of the protocol's detection rules.
Bidirectional Forwarding Detection
BFD can provide rapid detection of failures along a monitored forwarding path. A routing protocol or other registered client can use the resulting state change when the platform and configuration support it.
Unidirectional link detection
Mechanisms such as UDLD on supported Ethernet platforms can help identify certain one-way link conditions. Their behaviour and configuration are platform-dependent.
Detection, notification, and recovery are different events
Engineers often describe a network as having a particular failure-detection time. That is only one part of the recovery timeline. A failure must be detected, the relevant state change must be acted upon, routing information must be processed, and usable forwarding state must become available.
-
Failure occurs
A link, device, interface, or forwarding path stops behaving as expected. The fault may or may not be visible as an immediate interface-down event.
-
The fault is detected
A physical event, protocol timer, BFD session, or another supported mechanism identifies the condition.
-
Routing state changes
The routing process or registered client reacts to the event and updates neighbour, adjacency, or next-hop state as appropriate.
-
Updated information is processed
Relevant routing information is flooded, advertised, withdrawn, or otherwise propagated according to the protocol and its rules.
-
Traffic recovers
A valid path is selected and forwarding state is updated—or a previously installed repair path is activated where supported.
Important measurement distinction
A BFD session going down is evidence that the monitored session detected a problem. It does not, by itself, prove that the routing table, forwarding hardware, or application traffic has recovered.
How routing protocols propagate a failure
Once a failure is recognised, the control plane must determine which routes and neighbours are affected. Different protocol families communicate and recalculate reachability in different ways.
| Protocol or mechanism | Typical response | Engineering consideration |
|---|---|---|
| OSPF / IS-IS | Topology information is updated and distributed; affected routers calculate routes from their resulting link-state database. | Flooding, SPF scheduling, topology size, and implementation behaviour influence recovery. |
| EIGRP | Neighbour and route state is updated; feasible alternatives may be used where available and eligible. | An eligible feasible successor can reduce recovery work, but an alternative is not guaranteed for every destination. |
| BGP | Session or next-hop changes can trigger route withdrawals, updates, and new best-path decisions. | Policy, path selection, route scale, session behaviour, and propagation all affect the outcome. |
| BFD | Signals the state of a monitored forwarding path to configured clients. | BFD is a detection mechanism, not a routing protocol or a complete recovery solution by itself. |
These are typical behaviours, not fixed performance guarantees. Configuration, software release, platform capabilities, topology, and the precise failure scenario can change the observed response.
Timer tuning: faster is not always safer
Shorter detection intervals can reduce the time needed to identify certain failures. However, aggressive settings also reduce the margin for packet loss, scheduling delays, congestion, or temporary processing spikes. If a healthy neighbour is declared down incorrectly, the resulting adjacency reset or route withdrawal can create avoidable disruption.
Timer values should therefore be selected in line with the network design, supported platform limits, expected path behaviour, and operational testing. The objective is not simply to make every timer as small as possible; it is to achieve dependable detection with an acceptable recovery time.
- Validate the failure type: determine whether the issue is a physical outage, a silent forwarding failure, or a control-plane/session problem.
- Check both ends: compare interface state, neighbour state, counters, and logs where available.
- Verify dependencies: confirm that routing protocols are configured to act on the relevant detection mechanism.
- Test under realistic conditions: include packet loss, load, and recovery behaviour rather than relying only on configured timer values.
What to observe during a failure investigation
A useful troubleshooting record distinguishes the time of the original fault from the time each router detected it, the time routing state changed, and the time traffic recovered. Comparing these events helps identify whether the delay comes from detection, propagation, route calculation, forwarding installation, or a missing alternate path.
Engineering takeaway: failure detection starts convergence, but it does not finish it. Diagnose each stage independently, and confirm recovery by observing the resulting forwarding behaviour—not merely by checking that a neighbour has gone down.
Network Insight · Interactive Lab 02
Failure Detection: How Fast Does the Network Notice?
Compare an immediate interface-down event with timer-based detection. Observe the difference between a failure occurring, a router detecting it and routing information being propagated.
NETWORK HEALTHY
Test prefix: 10.40.40.0/24
Failure detection topologyLink operational
Operational primary linkFailed test linkDestination
Detection controlsStep 0 / 4
READY
Waiting for a failure
Choose a detection model, then inject the failure. This simulation uses ordered stages rather than claiming a universal detection time.
Failure stateNone
Detection stateNot started
Propagation stateNot started
Primary adjacencyUp
The timer option represents a detection interval conceptually. It does not run a real protocol timer or emulate vendor-specific BFD, OSPF, IS-IS, EIGRP or BGP behaviour.
Detection sequence0 / 4 events
1
Failure occursThe R2–R4 link becomes unavailable
2
Local detectionR2 recognises the link or adjacency failure
3
Routing process reactsThe protocol processes the changed neighbour or link state
4
Information propagatesOther routers can begin their own recalculation
Event logConceptual sequence
READYWaiting for a failure injection.
Engineering takeaway: Failure detection identifies that a neighbour, interface or path is no longer usable. Propagation and route recalculation follow; they are separate stages.
Section 3 · Route Computation
Route Calculation and Protocol Behaviour
Once a routing change is known, the router must determine which paths remain usable. Link-state protocols calculate routes from topology information, EIGRP uses its DUAL algorithm, and BGP evaluates eligible paths using its selection process and routing policies. Each protocol responds differently to a change.
In this section: SPF recalculation, EIGRP feasible successors, BGP best-path selection, and why an alternate route is not always immediately usable.
Section 03 · Routing Intelligence
Route Calculation and Protocol Behaviour
After a failure is detected and the relevant routing information changes, the network must determine which paths remain usable. Routing protocols solve this problem in different ways: some calculate routes from a shared topology, while others compare advertised paths, metrics, policies, and protocol-specific conditions.
Why route calculation matters
Detecting a failure does not automatically identify the best replacement route. A router must process the new information, determine which destinations are affected, and decide whether an alternate path is available. The amount of work depends on the protocol, the network design, the number of affected routes, and the platform's implementation.
Some changes have a limited scope. Others alter a large part of the topology or affect many advertised routes. The resulting convergence behaviour can therefore vary significantly even within the same routing protocol.
Stage 1
Receive changed information
Stage 2
Evaluate affected routes
Stage 3
Select a usable path
Stage 4
Prepare forwarding update
Link-state protocols: topology and SPF
OSPF and IS-IS distribute link-state information within their respective flooding scopes. Routers build and maintain a link-state database representing the topology they have learned. When relevant information changes, a router can run a Shortest Path First (SPF) calculation to determine routes based on that topology and the protocol's metric rules.
A topology change does not necessarily require every router to perform identical work at exactly the same time. Routers may receive updates at different times, and implementations can schedule, pace, or optimise SPF calculations. The effect depends on the change, the topology, and the implementation.
Link-state database
Represents the link-state information learned within the relevant area or level. Accurate and current topology information is a prerequisite for a meaningful SPF result.
Shortest Path First
Calculates shortest paths according to the protocol's metric model. The resulting routes are candidates for installation, subject to route selection and platform processing.
Distance-vector behaviour: EIGRP example
EIGRP uses neighbour information and route metrics to maintain reachability. Its Diffusing Update Algorithm (DUAL) supports loop-free route computation. When a primary route becomes unavailable, a router may be able to use a feasible successor—a qualifying alternate route already known to satisfy the feasibility condition.
If a suitable feasible successor is available, the router can often change to that route without conducting a full diffusing computation for the destination. If no feasible successor exists, additional query-and-reply processing may be needed. The outcome depends on the available topology and route state.
Key distinction
A backup path is not automatically a feasible successor. The feasibility condition matters because it helps EIGRP preserve loop-free behaviour during route changes.
Path-vector behaviour: BGP best-path selection
BGP exchanges reachability information between peers using UPDATE messages. When a route is withdrawn, a session changes state, or a new path is advertised, the router evaluates the affected routes using its import policies, available path attributes, and best-path decision process.
BGP selection can involve attributes such as local preference, AS path, origin, MED, and implementation-specific tie-breakers. The precise decision sequence varies with vendor implementation and configuration. A route that appears shorter or has a lower MED is not necessarily selected if an earlier decision criterion favours another path.
After the best path changes, BGP may advertise a changed route to eligible peers according to export policy and protocol rules. Route propagation and the decisions made by other routers can extend the overall recovery process beyond the first router's local calculation.
| Protocol family | How it evaluates change | What can affect convergence |
|---|---|---|
| OSPF / IS-IS | Updates link-state information and calculates routes from the resulting topology. | Flooding scope, topology size, SPF scheduling, route installation, and implementation. |
| EIGRP | Evaluates successor and alternate routes using its metric and feasibility rules. | Availability of a feasible successor, query scope, and route state. |
| BGP | Evaluates candidate paths using policy and path attributes, then propagates eligible changes. | Policy, peer state, path diversity, route scale, and propagation behaviour. |
Why a replacement path may not be available
A routing protocol can only select from the paths it knows about and is permitted to use. A physically connected alternate may not be advertised, may fail a protocol's eligibility rules, may be rejected by policy, or may lead to a destination that is no longer reachable.
- Reachability: does the candidate path still lead to the destination?
- Protocol rules: is the path eligible under the protocol's selection and loop-prevention mechanisms?
- Policy: has an import, export, filtering, or preference rule changed which path can be used?
- Next-hop resolution: can the selected route's next hop be resolved and used by the forwarding system?
- Installation: can the resulting forwarding state be programmed successfully?
Fast convergence is not simply fast computation
A route calculation may finish quickly while traffic remains disrupted because the required update has not propagated, the selected next hop is unresolved, or forwarding hardware has not yet been updated. Conversely, a precomputed repair path can sometimes restore forwarding before the full control-plane process has completed.
For this reason, convergence investigations should distinguish the time when a protocol receives changed information, the time when it reaches a new route decision, and the time when the forwarding plane can successfully use that decision.
Engineering takeaway: protocols use different algorithms and selection rules to respond to topology and reachability changes. Understand the protocol's decision process, verify that a valid alternate path exists, and measure the forwarding outcome rather than assuming route calculation alone equals recovery.
Network Insight · Interactive Lab 03
Route Calculation: Which Path Wins?
Break the primary path, compare the remaining candidates and follow the distinction between route calculation, route selection and installation. The protocol selector changes the explanation, not a full implementation of each routing protocol.
PRIMARY ROUTE SELECTED
Destination: 10.40.40.0/24
Candidate path topologyPrimary path available
Primary candidateAlternate candidateFailed link
Decision engineStage 0 / 5
NORMAL OPERATION
Primary route is selected
R1 currently uses R2 as the next hop to reach 10.40.40.0/24. The route via R3 is an alternate candidate in this teaching topology.
Selected next hopR2
Route calculationStable
Alternate routeAvailable
Installation stateInstalled
Protocol behaviour is represented conceptually. The topology does not run SPF, DUAL, BGP best-path rules or real routing updates.
Candidate route evaluationR1 · 10.40.40.0/24
Candidate A · via R2Primary path is reachable and selected.
Selected
Candidate B · via R3Alternate path is available but not selected.
Standby
Selection principle: A router can use only routes it has learned or calculated and considers eligible. The selected route depends on protocol-specific rules and configuration.
Calculation timeline0 / 5 stages
1
Failure recognisedThe primary path is no longer usable
2
Routing information updatedThe protocol processes the changed topology or route
3
Alternatives evaluatedEligible remaining candidates are considered
4
New route selectedR1 selects the route via R3 in this model
5
Route installedThe selected route is ready for forwarding installation
Engineering takeaway: Route calculation, best-path selection and route installation are related but distinct. A protocol's method of finding or choosing a route depends on its design.
Section 4 · Forwarding Recovery
RIB, FIB and Fast Reroute
A router can calculate a replacement route without immediately restoring traffic. The selected route must be reflected in forwarding state, and packets must be able to use the resulting next hop. Techniques such as Equal-Cost Multipath (ECMP) and Loop-Free Alternates (LFA) can help maintain or restore forwarding, depending on the topology and platform.
In this section: How the RIB and FIB differ, forwarding installation, alternate paths, and fast-reroute techniques.
Section 04 · Forwarding Recovery
RIB, FIB and Fast Reroute
A routing protocol can calculate a replacement path, but packets recover only when the forwarding system can use a valid next hop. Understanding the Routing Information Base (RIB), Forwarding Information Base (FIB), and fast-reroute mechanisms explains why routing convergence and traffic recovery are related—but not identical—events.
From route information to packet forwarding
Routers maintain control-plane information about reachable destinations and use it to determine which routes should be selected. The selected routing information is then used to program the forwarding system. On many platforms, this involves software processes coordinating with hardware tables or other forwarding resources.
The RIB and FIB serve different purposes. The RIB stores routing information and supports route selection. The FIB contains the forwarding entries used to make packet-forwarding decisions, often with resolved next-hop and adjacency information appropriate to the platform.
01 · Learn
Receive routes and topology information
02 · Select
Choose eligible routes for the RIB
03 · Program
Resolve and install forwarding entries
04 · Forward
Send packets using the active FIB
Routing Information Base
Contains routing information learned from connected networks, static configuration, and routing protocols. Route selection determines which route or routes are eligible to contribute to forwarding.
Forwarding Information Base
Supports destination lookups for packet forwarding. Entries may reference resolved next hops, interfaces, adjacency information, and other platform-specific forwarding structures.
Why the RIB and FIB can differ in timing
A new best route in the control plane does not prove that every forwarding resource has already been updated. The system may need to resolve the next hop, reconcile competing routes, allocate forwarding resources, and program software or hardware tables. The timing and architecture differ across platforms.
During a failure, the old forwarding entry may point to an unusable next hop until the forwarding state changes. Alternatively, a platform may already have an eligible alternate or repair entry that can be activated quickly. Observing both control-plane route state and data-plane behaviour helps establish whether recovery is complete.
Operational rule
Do not treat “the route is present in the routing table” as conclusive evidence that packets are successfully reaching the destination. Verify next-hop resolution, forwarding entries, interface state, and end-to-end traffic where possible.
Equal-Cost Multi-Path forwarding (ECMP)
ECMP allows a router to forward traffic over multiple eligible paths that have equal cost according to the applicable routing decision. The forwarding system commonly uses a hash of packet fields to select a path, helping keep flows on a consistent path rather than distributing each packet independently.
If one ECMP next hop fails, the router may remove or avoid that next hop and continue using the remaining eligible paths. This can reduce disruption, but the exact effect depends on the implementation, hash behaviour, flow distribution, and whether the surviving paths have sufficient capacity.
- Path availability: the remaining next hops must still be valid and reachable.
- Flow impact: traffic may be rehashed when the set of next hops changes, potentially affecting packet ordering for some flows.
- Capacity: surviving links must be able to handle the shifted traffic load.
- Installation: the forwarding plane must apply the updated next-hop set successfully.
Fast reroute and precomputed repair paths
Fast reroute techniques aim to reduce traffic loss by making a repair path available before a failure occurs. Rather than waiting for every step of the normal control-plane reconvergence process, a router may be able to redirect affected traffic onto a precomputed alternate when the protected failure is detected.
Examples include Loop-Free Alternates (LFA) and, where supported and configured, remote LFA or Segment Routing-based repair mechanisms. Their applicability depends on topology, protocol support, platform capabilities, and the failure being protected against.
| Mechanism | Primary purpose | Important limitation |
|---|---|---|
| ECMP | Use multiple equal-cost forwarding paths. | Does not guarantee that every failure has a surviving path or that traffic remains evenly distributed. |
| LFA | Provide an eligible loop-free alternate next hop for protected destinations. | Topology may not provide a valid LFA for every destination and failure. |
| Remote LFA / Segment Routing repair | Extend repair options where supported, including cases not covered by a direct LFA. | Requires suitable topology, configuration, protocol support, and forwarding capabilities. |
| Normal reconvergence | Calculate and install the new forwarding state after the routing system processes the change. | Traffic disruption may persist while detection, propagation, calculation, and installation occur. |
Forwarding recovery and control-plane convergence
A repair path can restore traffic before the routing protocol has completed all of its normal calculations and advertisements. In that case, the data plane may be forwarding successfully while the control plane continues to converge. The reverse can also happen: a new route may appear in the RIB while traffic still fails because of unresolved next hops, stale forwarding state, or a separate downstream fault.
These are not necessarily contradictions. They are signs that different components have different responsibilities and update timelines. Reliable testing should record both routing-state changes and packet-level results.
How to verify forwarding recovery
- Inspect the selected route and its next-hop resolution.
- Check the relevant FIB entry and forwarding adjacency where the platform exposes them.
- Confirm that failed interfaces or next hops are no longer used for affected traffic.
- Run suitable reachability or application tests across the affected path.
- Compare the observed traffic recovery time with the routing and forwarding event timeline.
Engineering takeaway: the RIB represents selected routing information, while the FIB supports actual packet forwarding. ECMP and fast reroute can reduce disruption when valid alternatives exist, but successful recovery must be verified in the forwarding plane—not inferred from a route calculation alone.
Network Insight · Interactive Lab 04
RIB to FIB: When Does Forwarding Recover?
Fail the primary link and step through route selection and forwarding installation. Compare normal convergence with a conceptual pre-installed fast-reroute repair path.
PRIMARY FORWARDING ACTIVE
Destination: 10.40.40.0/24
Forwarding topologyPrimary path active
Primary routeAlternate routeFailed link
Recovery controlsStage 0 / 5
Forwarding strategy
NORMAL OPERATION
Primary forwarding entry is active
R1 forwards the destination prefix through R2. The alternate path through R3 exists in the topology but is not currently selected.
RIB next hopR2
FIB next hopR2
Forwarding statusPrimary active
Repair stateStandby
Fast reroute is represented conceptually. Real repair behaviour depends on the technology, topology, precomputed repair availability and platform support.
Routing and forwarding stateR1 · 10.40.40.0/24
| Plane / entry | Current next hop | What it tells you |
|---|---|---|
| Selected route (RIB) | R2 | Primary route selected |
| Forwarding entry (FIB) | R2 | Packets use primary path |
| Repair candidate | R3 | Available as alternate path |
Recovery sequence0 / 5 stages
1
Primary link failsThe R2–R4 path becomes unavailable
2
Failure response beginsNormal convergence or a repair mechanism reacts
3
RIB selection changesThe control plane selects the route via R3
4
FIB is updatedForwarding state reflects the selected next hop
5
Forwarding verifiedCheck packet delivery over the alternate path
Engineering takeaway: The RIB represents selected routing information; the FIB is the forwarding structure used to send packets. Verify both, and test traffic, before declaring recovery.
Section 5 · Operational Verification
Convergence Measurement and Troubleshooting
Reliable convergence analysis depends on evidence, not assumptions. A routing log may show when a change was detected, while protocol state, RIB and FIB inspection, packet tests, and application monitoring reveal different parts of the recovery process. A meaningful result requires a clearly defined start event and end condition.
In this section: Convergence metrics, repeatable failure tests, route flapping, telemetry, and a structured troubleshooting workflow.
Section 05 · Operational Evidence
Convergence Measurement and Troubleshooting
A network can appear healthy in a routing table while traffic is still being lost—or it can restore forwarding before every control-plane process has settled. Reliable convergence analysis requires a clear measurement boundary, timestamped evidence, and a methodical way to separate detection, route calculation, forwarding recovery, and application impact.
Define what convergence means for the test
A convergence measurement is meaningful only when its start and end points are explicit. Measuring from the moment a cable is disconnected to the moment a monitoring probe succeeds is different from measuring the interval between a routing adjacency failure and a new route appearing in the RIB.
Before testing, identify the failure being introduced, the destination or traffic flow being observed, and the event that will count as recovery. Use the same definitions across repeated tests so results can be compared fairly.
Routing convergence
Measures a defined routing-state transition, such as the time from a topology event to a stable, expected route selection. Specify exactly which event marks the beginning and end.
Traffic recovery
Measures when the tested packets or flows can successfully use the recovered path. This may occur before or after the routing system reaches its final stable state.
Application recovery
Measures when the application meets its service criteria again, such as successful transactions or acceptable latency. Application recovery can include dependencies beyond routing.
Time to stable state
Checks whether the route and traffic remain healthy after recovery, rather than counting a brief successful probe as proof that the incident is over.
Do not mix measurement boundaries
A result such as “convergence took 200 ms” is incomplete unless the failure trigger, observation method, destination, and recovery criterion are stated. Different test methods can produce different—and individually valid—measurements.
Build an evidence timeline
The most useful troubleshooting timeline combines control-plane events with forwarding and traffic observations. Synchronised clocks make it easier to correlate logs across routers, although timestamp precision and collection delays should also be considered.
-
Record the trigger
Capture when the link, interface, session, or simulated fault was introduced. Record the test conditions and affected destination or flow.
-
Capture detection
Record interface transitions, neighbour-state changes, BFD events, or protocol timer expirations as applicable.
-
Capture route processing
Observe topology updates, route withdrawals or advertisements, SPF or other route calculations, and best-path changes where available.
-
Verify forwarding state
Check the selected route, next-hop resolution, FIB or adjacency changes, and any active fast-reroute behaviour.
-
Confirm traffic recovery
Use repeated probes or representative application tests to determine when service returns and whether it remains stable.
Evidence to collect during an investigation
| Evidence source | What it helps establish | What it cannot prove alone |
|---|---|---|
| Interface counters and logs | Physical state transitions, errors, drops, and link-related events. | That all routing and forwarding state has recovered. |
| Neighbour and session state | Whether the relevant protocol adjacency or session is established. | That every destination has a usable route. |
| RIB and route details | Which route is selected and why a candidate path may have been preferred or rejected. | That the forwarding hardware is already using that path. |
| FIB and next-hop details | Which forwarding entries and next hops are programmed, where observable. | That end-to-end traffic is successful beyond the local forwarding point. |
| Packet probes and application telemetry | Observed loss, latency, reachability, and service recovery for the tested flow. | The exact internal routing event responsible for every symptom. |
Recognise route flapping and unstable recovery
Route flapping occurs when route availability or attributes repeatedly change. Repeated interface transitions, unstable sessions, intermittent reachability, or changing path attributes can prevent the network from settling into a stable state.
Flapping can increase control-plane work and cause repeated forwarding changes. In BGP, route-flap damping may be available in some environments, but its use requires care: suppressing a route can extend unreachability, and the behaviour depends on configuration and operational policy.
- Correlate repeated events: compare interface logs, neighbour resets, route changes, and traffic-loss windows.
- Look for a shared cause: investigate common links, power, optics, congestion, software events, or unstable downstream dependencies.
- Inspect policy and attributes: for BGP, determine whether changing advertisements or policy inputs are altering best-path decisions.
- Check the final state: confirm the selected route and forwarding path remain stable after the last observed event.
Use repeatable tests instead of assumptions
A single successful run may hide timing variation. Repeat controlled tests, keep the topology and traffic conditions consistent, and record the observed recovery interval for each run. Where the test is intended to represent production behaviour, include realistic traffic load and relevant failure types.
Report the measurement method alongside the results. Depending on the objective, useful summaries may include minimum, median, maximum, and a high percentile of recovery times. Preserve individual run data as well: aggregate values can conceal an occasional long outage.
A practical troubleshooting sequence
- Confirm the fault and its scope before changing configuration.
- Identify whether detection was delayed or incorrect.
- Determine whether routing information propagated as expected.
- Inspect route selection and verify that a usable alternative exists.
- Check FIB installation, next-hop resolution, and fast-reroute state where supported.
- Test end-to-end traffic and application behaviour.
- Repeat the test after remediation and compare evidence against the original baseline.
Engineering takeaway: measure convergence from a defined event to a defined recovery condition. Correlate logs, routing state, forwarding state, and traffic evidence to locate the delay, then repeat the test to verify that the fix improves both recovery time and stability.
Network Insight · Interactive Lab 05
Convergence Incident: Measure, Diagnose, Verify
Investigate a simulated routing incident from first failure evidence to verified forwarding recovery. Step through the event timeline, inspect the evidence and complete the engineering checklist.
INCIDENT READY
Incident RC-05 · Prefix 10.50.50.0/24
Incident scenarioStage 0 / 5
Reported symptom: intermittent packet loss
A primary path has failed. The operations team needs to establish when failure was detected, when routing state changed, when forwarding recovered and whether traffic actually resumed.
INITIAL STATE
Begin the incident investigation
Start the incident to simulate a link failure. Then step through detection, route recalculation, forwarding installation and verification. The sequence is illustrative, not a live router measurement.
Failure evidenceNot observed
Control-plane stateStable
Forwarding statePrimary path
Traffic verificationNot tested
No real timers or millisecond values are generated. Use actual event timestamps and defined start/end criteria when measuring a production network.
Event evidenceIllustrative records
InterfaceAwaiting failure event
RoutingNo route change observed yet
ForwardingPrimary path currently selected
TrafficEnd-to-end test not run
Evidence rule: Do not infer the entire convergence interval from a single log. Correlate interface, protocol, route, forwarding and traffic observations.
Convergence timeline0 / 5 events
1
Failure occursPhysical or logical path event
2
Failure is detectedInterface state, timer, BFD or other evidence
3
Route selection changesControl plane selects an eligible alternative
4
Forwarding state updatesFIB and relevant forwarding hardware are checked
5
Traffic recovery is verifiedProbe or application test confirms delivery
Engineer’s verification checklist0 / 5 checked
Complete the checklist as you investigate. A checked item records your assessment; it does not query a live network.
Measurement principle: Define the start event and recovery endpoint before calculating convergence time. Control-plane convergence, forwarding recovery and application recovery are related but distinct outcomes.
Conclusion · Engineering Practice
Routing Convergence: From Failure to Recovery
Routing convergence is the process through which a network detects a change, distributes the relevant information, recalculates usable paths, and updates forwarding behaviour. Understanding each stage helps engineers distinguish a routing-protocol problem from a forwarding, detection, or service-recovery problem.
01
Detect
Recognise a failure or topology change through interface events, protocol timers, or mechanisms such as BFD.
02
Propagate
Distribute the changed reachability or topology information to the routers that need to react.
03
Calculate
Run the relevant protocol logic and determine which routes or alternate paths remain eligible.
04
Forward
Install the resulting forwarding state and verify that traffic can use the intended path.
The key engineering principle
A routing-table change does not, by itself, prove that an application has recovered. Control-plane processing, forwarding installation, packet delivery, and service recovery must be measured separately.
Routing convergence troubleshooting checklist
When investigating a slow or incomplete recovery, work through the following questions in sequence:
- When did the failure occur, and when was it first detected?
- Did the relevant routing protocol propagate the change correctly?
- Was the expected replacement route eligible and selected?
- Did the RIB and FIB update as expected?
- Did packets actually traverse the replacement path?
- Did application traffic recover, and was the result repeatable?
Explore · Trace · Diagnose
Put routing convergence into practice
Move from theory to an interactive BGP environment. Explore route propagation, inspect best-path decisions, follow routing-state changes, and investigate how network behaviour changes when a link or route becomes unavailable.
Launch the BGP Core SimulatorOpens the existing Network Insight simulator in a new tab.
Continue learning with Network Insight
Use these learning resources to connect routing theory with practical forwarding and fault investigation.
BGP Learning Centre
Explore BGP architecture, route propagation, best-path selection, and fault investigation with the BGP Learning Centre and simulator .
IP Forwarding
Follow how routers use routing and forwarding information to make packet-by-packet forwarding decisions. Use the IP forwarding guide alongside this article.
BGP Route Selection and Propagation
Extend your understanding of path attributes, route eligibility, route advertisement, and how routing changes affect reachability.
Failure Detection and Fast Reroute
Study the difference between detecting a failure, selecting an alternate path, installing forwarding state, and restoring traffic.
Support Network Insight
Help support the development of interactive networking labs,
BGP simulations, and educational content.
How-to: Fabric Extenders & VPC
Topology Diagram
The topology diagram depicts two Nexus 5K acting as parent switches with physical connections to two downstream Nexus 2k (FEX) acting as the 10G physical termination points for the connected server.
Part1. Connecting the FEX to the Parent switch:
The FEX and the parent switch use Satellite Discovery Protocol (SDP) periodic messages to discovery and register with one another.
When you initially log on to the Nexus 5K you can see that the OS does not recognise the FEX even though there are two FEXs that are cabled correctly to parent switch. As the FEX is recognised as a remote line card you would expect to see it with a “show module” command.
| N5K3# sh module Mod Ports Module-Type Model Status — —– ——————————– ———————- ———— 1 40 40x10GE/Supervisor N5K-C5020P-BF-SUP active * 2 8 8×1/2/4G FC Module N5K-M1008 ok Mod Sw Hw World-Wide-Name(s) (WWN) — ————– —— ————————————————– 1 5.1(3)N2(1c) 1.3 — 2 5.1(3)N2(1c) 1.0 93:59:41:08:5a:0c:08:08 to 00:00:00:00:00:00:00:00 Mod MAC-Address(es) Serial-Num — ————————————– ———- 1 0005.9b1e.82c8 to 0005.9b1e.82ef JAF1419BLMA 2 0005.9b1e.82f0 to 0005.9b1e.82f7 JAF1411AQBJ |
We issue the “feature fex” command we observe the FEX sending SDP messages to the parent switch i.e. RX but we don’t see the parent switch sending SDP messages to the FEX i.e. TX.
Notice in the output below there is only “fex:Sdp-Rx” messages.
| N5K3# debug fex pkt-trace N5K3# 2014 Aug 21 09:51:57.410701 fex: Sdp-Rx: Interface: Eth1/11, Fex Id: 0, Ctrl Vntag: -1, Ctrl Vlan: 1 2014 Aug 21 09:51:57.410729 fex: Sdp-Rx: Refresh Intvl: 3000ms, Uid: 0x4000ff2929f0, device: Fex, Remote link: 0x20000080 2014 Aug 21 09:51:57.410742 fex: Sdp-Rx: Vendor: Cisco Systems Model: N2K-C2232PP-10GE Serial: FOC17100NHX 2014 Aug 21 09:51:57.821776 fex: Sdp-Rx: Interface: Eth1/10, Fex Id: 0, Ctrl Vntag: -1, Ctrl Vlan: 1 2014 Aug 21 09:51:57.821804 fex: Sdp-Rx: Refresh Intvl: 3000ms, Uid: 0x2ff2929f0, device: Fex, Remote link: 0x20000080 2014 Aug 21 09:51:57.821817 fex: Sdp-Rx: Vendor: Cisco Systems Model: N2K-C2232PP-10GE Serial: FOC17100NHU |
The FEX appears as “DISCOVERED” but no additional FEX host interfaces appear when you issue a “show interface brief“.
Command: show fex [chassid_id [detail]]: Displays information about a specific Fabric Extender Chassis ID
Command: show interface brief: Display interface information and connection status for each interface.
| N5K3# sh fex FEX FEX FEX FEX Number Description State Model Serial ———————————————————————— — ——– Discovered N2K-C2232PP-10GE SSI16510AWF — ——– Discovered N2K-C2232PP-10GE SSI165204YC N5K3# N5K3# show interface brief ——————————————————————————– Ethernet VLAN Type Mode Status Reason Speed Port Interface Ch # ——————————————————————————– Eth1/1 1 eth access down SFP validation failed 10G(D) — Eth1/2 1 eth access down SFP validation failed 10G(D) — Eth1/3 1 eth access up none 10G(D) — Eth1/4 1 eth access up none 10G(D) — Eth1/5 1 eth access up none 10G(D) — Eth1/6 1 eth access down Link not connected 10G(D) — Eth1/7 1 eth access down Link not connected 10G(D) — Eth1/8 1 eth access down Link not connected 10G(D) — Eth1/9 1 eth access down Link not connected 10G(D) — Eth1/10 1 eth fabric down FEX not configured 10G(D) — Eth1/11 1 eth fabric down FEX not configured 10G(D) — Eth1/12 1 eth access down Link not connected 10G(D) — snippet removed |
The Fabric interface Ethernet1/10 show as DOWN with a “FEX not configured” statement.
| N5K3# sh int Ethernet1/10 Ethernet1/10 is down (FEX not configured) Hardware: 1000/10000 Ethernet, address: 0005.9b1e.82d1 (bia 0005.9b1e.82d1) MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA Port mode is fex-fabric auto-duplex, 10 Gb/s, media type is 10G Beacon is turned off Input flow-control is off, output flow-control is off Rate mode is dedicated Switchport monitor is off EtherType is 0x8100 snippet removed |
To enable the parent switch to fully discover the FEX we need to issue the “switchport mode fex-fabric” under the connected interface. As you can see we are still not sending any SDP messages but we are discovering the FEX.
The next step is to enable the FEX logical numbering under the interface so we can start to configure the FEX host interfaces. Once this is complete we run the “debug fex pkt-trace” and we are not sending TX and receiving RX SDP messages.
Command:”fex associate chassis_id“: Associates a Fabric Extender (FEX) to a fabric interface. To disassociate the Fabric Extender, use the “no” form of this command.
From the “debug fexpkt-race” you can see the parent switch is now sending TX SDP messages to the fully discovered FEX.
| N5K3(config)# int Ethernet1/10 N5K3(config-if)# fex associate 101 N5K3# debug fex pkt-trace N5K3# 2014 Aug 21 10:00:33.674605 fex: Sdp-Tx: Interface: Eth1/10, Fex Id: 101, Ctrl Vntag: 0, Ctrl Vlan: 4042 2014 Aug 21 10:00:33.674633 fex: Sdp-Tx: Refresh Intvl: 3000ms, Uid: 0xc0821e9b0500, device: Switch, Remote link: 0x1a009000 2014 Aug 21 10:00:33.674646 fex: Sdp-Tx: Vendor: Model: Serial: ———- 2014 Aug 21 10:00:33.674718 fex: Sdp-Rx: Interface: Eth1/10, Fex Id: 0, Ctrl Vntag: 0, Ctrl Vlan: 4042 2014 Aug 21 10:00:33.674733 fex: Sdp-Rx: Refresh Intvl: 3000ms, Uid: 0x2ff2929f0, device: Fex, Remote link: 0x20000080 2014 Aug 21 10:00:33.674746 fex: Sdp-Rx: Vendor: Cisco Systems Model: N2K-C2232PP-10GE Serial: FOC17100NHU 2014 Aug 21 10:00:33.836774 fex: Sdp-Rx: Interface: Eth1/11, Fex Id: 0, Ctrl Vntag: -1, Ctrl Vlan: 1 2014 Aug 21 10:00:33.836803 fex: Sdp-Rx: Refresh Intvl: 3000ms, Uid: 0x4000ff2929f0, device: Fex, Remote link: 0x20000080 2014 Aug 21 10:00:33.836816 fex: Sdp-Rx: Vendor: Cisco Systems Model: N2K-C2232PP-10GE Serial: FOC17100NHX 2014 Aug 21 10:00:36.678624 fex: Sdp-Tx: Interface: Eth1/10, Fex Id: 101, Ctrl Vntag: 0, Ctrl Vlan: 4042 2014 Aug 21 10:00:36.678664 fex: Sdp-Tx: Refresh Intvl: 3000ms, Uid: 0xc0821e9b0500, device: Switch, Remote snippet removed |
Now the 101 FEX status changes from “DISCOVERED” to “ONLINE”. You may also see an additional FEX with serial number SSI165204YC as “DISCOVERED” and not “ONLINE”. This is due to the fact that we have not explicitly configured it under the other Fabric interface.
| N5K3# sh fex FEX FEX FEX FEX Number Description State Model Serial ———————————————————————— 101 FEX0101 Online N2K-C2232PP-10GE SSI16510AWF — ——– Discovered N2K-C2232PP-10GE SSI165204YC N5K3# N5K3# show module fex 101 FEX Mod Ports Card Type Model Status. — — —– ———————————- —————— ———– 101 1 32 Fabric Extender 32x10GE + 8x10G Module N2K-C2232PP-10GE present FEX Mod Sw Hw World-Wide-Name(s) (WWN) — — ————– —— ———————————————– 101 1 5.1(3)N2(1c) 4.4 — FEX Mod MAC-Address(es) Serial-Num — — ————————————– ———- 101 1 f029.29ff.0200 to f029.29ff.021f SSI16510AWF |
Issuing the “show interface brief” we see new interfaces, specifically host interfaces for the FEX. The syntax below shows that only one interface is up; interface labelled Eth101/1/1. Reason for this is that only one end host (server) is connected to the FEX
| N5K3# show interface brief ——————————————————————————– Ethernet VLAN Type Mode Status Reason Speed Port Interface Ch # ——————————————————————————– Eth1/1 1 eth access down SFP validation failed 10G(D) — Eth1/2 1 eth access down SFP validation failed 10G(D) — snipped removed ——————————————————————————– Port VRF Status IP Address Speed MTU ——————————————————————————– mgmt0 — up 192.168.0.53 100 1500 ——————————————————————————– Ethernet VLAN Type Mode Status Reason Speed Port Interface Ch # ——————————————————————————– Eth101/1/1 1 eth access up none 10G(D) — Eth101/1/2 1 eth access down SFP not inserted 10G(D) — Eth101/1/3 1 eth access down SFP not inserted 10G(D) — Eth101/1/4 1 eth access down SFP not inserted 10G(D) — Eth101/1/5 1 eth access down SFP not inserted 10G(D) — Eth101/1/6 1 eth access down SFP not inserted 10G(D) — snipped removed |
| N5K3# sh run int eth1/10 interface Ethernet1/10 switchport mode fex-fabric fex associate 101 |
The Fabric Interfaces do not run a Spanning tree instance while the host interfaces do run BPDU guard and BPDU filter by default. The reason why the fabric interfaces do not run spanning tree is because they are backplane point to point interfaces.
By default, the FEX interfaces will send out a couple of BPDU’s on start-up.
| N5K3# sh spanning-tree interface Ethernet1/10 No spanning tree information available for Ethernet1/10 N5K3# N5K3# N5K3# sh spanning-tree interface Eth101/1/1Vlan Role Sts Cost Prio.Nbr Type —————- —- — ——— ——– ——————————– VLAN0001 Desg FWD 2 128.1153 Edge P2p N5K3# N5K3# sh spanning-tree interface Eth101/1/1 detail Port 1153 (Ethernet101/1/1) of VLAN0001 is designated forwarding Port path cost 2, Port priority 128, Port Identifier 128.1153 Designated root has priority 32769, address 0005.9b1e.82fc Designated bridge has priority 32769, address 0005.9b1e.82fc Designated port id is 128.1153, designated path cost 0 Timers: message age 0, forward delay 0, hold 0 Number of transitions to forwarding state: 1 The port type is edge Link type is point-to-point by default Bpdu guard is enabled Bpdu filter is enabled by default BPDU: sent 11, received 0 |
| N5K3# sh spanning-tree interface Ethernet1/10 No spanning tree information available for Ethernet1/10 N5K3# N5K3# N5K3# sh spanning-tree interface Eth101/1/1Vlan Role Sts Cost Prio.Nbr Type—————- —- — ——— ——– ——————————– VLAN0001 Desg FWD 2 128.1153 Edge P2p N5K3# N5K3# sh spanning-tree interface Eth101/1/1 detail Port 1153 (Ethernet101/1/1) of VLAN0001 is designated forwarding Port path cost 2, Port priority 128, Port Identifier 128.1153 Designated root has priority 32769, address 0005.9b1e.82fc Designated bridge has priority 32769, address 0005.9b1e.82fc Designated port id is 128.1153, designated path cost 0 Timers: message age 0, forward delay 0, hold 0 Number of transitions to forwarding state: 1 The port type is edge Link type is point-to-point by default Bpdu guard is enabled Bpdu filter is enabled by default BPDU: sent 11, received 0 |
Issue the commands below to determine the transceiver type for the fabric ports and also the hosts ports for each fabric interface.
Command: “show interface fex-fabric“: displays all the Fabric Extender interfaces
Command: “show fex detail“: Shows detailed information about all FEXs. Including more recent log messages related to the FEX.
| N5K3# show interface fex-fabric Fabric Fabric Fex FEX Fex Port Port State Uplink Model Serial ————————————————————— 101 Eth1/10 Active 3 N2K-C2232PP-10GE SSI16510AWF — Eth1/11 Discovered 3 N2K-C2232PP-10GE SSI165204YC N5K3# N5K3# N5K3# show interface Ethernet1/10 fex-intf Fabric FEX Interface Interfaces ————————————————— Eth1/10 Eth101/1/1 N5K3# N5K3# show interface Ethernet1/10 transceiver Ethernet1/10 transceiver is present type is SFP-H10GB-CU3M name is CISCO-TYCO part number is 1-2053783-2 revision is N serial number is TED1530B11W nominal bitrate is 10300 MBit/sec Link length supported for copper is 3 m cisco id is — cisco extended id number is 4 N5K3# show fex detail FEX: 101 Description: FEX0101 state: Online FEX version: 5.1(3)N2(1c) [Switch version: 5.1(3)N2(1c)] FEX Interim version: 5.1(3)N2(1c) Switch Interim version: 5.1(3)N2(1c) Extender Serial: SSI16510AWF Extender Model: N2K-C2232PP-10GE, Part No: 73-12533-05 Card Id: 82, Mac Addr: f0:29:29:ff:02:02, Num Macs: 64 Module Sw Gen: 12594 [Switch Sw Gen: 21] post level: complete pinning-mode: static Max-links: 1 Fabric port for control traffic: Eth1/10 FCoE Admin: false FCoE Oper: true FCoE FEX AA Configured: false Fabric interface state: Eth1/10 – Interface Up. State: Active Fex Port State Fabric Port Eth101/1/1 Up Eth1/10 Eth101/1/2 Down None Eth101/1/3 Down None Eth101/1/4 Down None snippet removed Logs: 08/21/2014 10:00:06.107783: Module register received 08/21/2014 10:00:06.109935: Registration response sent 08/21/2014 10:00:06.239466: Module Online S |
Now we quickly enable the second FEX connected to fabric interface E1/11.
| N5K3(config)# int et1/11 N5K3(config-if)# switchport mode fex-fabric N5K3(config-if)# fex associate 102 N5K3(config-if)# end N5K3# sh fex FEX FEX FEX FEX Number Description State Model Serial ———————————————————————— 101 FEX0101 Online N2K-C2232PP-10GE SSI16510AWF 102 FEX0102 Online N2K-C2232PP-10GE SSI165204YC N5K3# show fex detail FEX: 101 Description: FEX0101 state: Online FEX version: 5.1(3)N2(1c) [Switch version: 5.1(3)N2(1c)] FEX Interim version: 5.1(3)N2(1c) Switch Interim version: 5.1(3)N2(1c) Extender Serial: SSI16510AWF Extender Model: N2K-C2232PP-10GE, Part No: 73-12533-05 Card Id: 82, Mac Addr: f0:29:29:ff:02:02, Num Macs: 64 Module Sw Gen: 12594 [Switch Sw Gen: 21] post level: complete pinning-mode: static Max-links: 1 Fabric port for control traffic: Eth1/10 FCoE Admin: false FCoE Oper: true FCoE FEX AA Configured: false Fabric interface state: Eth1/10 – Interface Up. State: Active Fex Port State Fabric Port Eth101/1/1 Up Eth1/10 Eth101/1/2 Down None Eth101/1/3 Down None Eth101/1/4 Down None Eth101/1/5 Down None Eth101/1/6 Down None snippet removed Logs: 08/21/2014 10:00:06.107783: Module register received 08/21/2014 10:00:06.109935: Registration response sent 08/21/2014 10:00:06.239466: Module Online Sequence 08/21/2014 10:00:09.621772: Module Online FEX: 102 Description: FEX0102 state: Online FEX version: 5.1(3)N2(1c) [Switch version: 5.1(3)N2(1c)] FEX Interim version: 5.1(3)N2(1c) Switch Interim version: 5.1(3)N2(1c) Extender Serial: SSI165204YC Extender Model: N2K-C2232PP-10GE, Part No: 73-12533-05 Card Id: 82, Mac Addr: f0:29:29:ff:00:42, Num Macs: 64 Module Sw Gen: 12594 [Switch Sw Gen: 21] post level: complete pinning-mode: static Max-links: 1 Fabric port for control traffic: Eth1/11 FCoE Admin: false FCoE Oper: true FCoE FEX AA Configured: false Fabric interface state: Eth1/11 – Interface Up. State: Active Fex Port State Fabric Port Eth102/1/1 Up Eth1/11 Eth102/1/2 Down None Eth102/1/3 Down None Eth102/1/4 Down None Eth102/1/5 Down None snippet removed Logs: 08/21/2014 10:12:13.281018: Module register received 08/21/2014 10:12:13.283215: Registration response sent 08/21/2014 10:12:13.421037: Module Online Sequence 08/21/2014 10:12:16.665624: Module Online |
Part 2. Fabric Interfaces redundancy
Static Pinning is when you pin a number of host ports to a fabric port. If the fabric port goes down so do the host ports that are pinned to it. This is useful when you want no oversubscription in the network.
Once the host port shut down due to a fabric port down event, the server if configured correctly should revert to the secondary NIC.
The “pinning max-link” divides the number specified in the command by the number of host interfaces to determine how many host interfaces go down if there is a fabric interface failure.
Now we shut down fabric interface E1/10, you can see that Eth101/1/1 has changed its operation mode to DOWN. The FEX has additional connectivity with E1/11 which remains up.
| Enter configuration commands, one per line. End with CNTL/Z. N5K3(config)# int et1/10 N5K3(config-if)# shu N5K3(config-if)# N5K3(config-if)# end N5K3# sh fex detail FEX: 101 Description: FEX0101 state: Offline FEX version: 5.1(3)N2(1c) [Switch version: 5.1(3)N2(1c)] FEX Interim version: 5.1(3)N2(1c) Switch Interim version: 5.1(3)N2(1c) Extender Serial: SSI16510AWF Extender Model: N2K-C2232PP-10GE, Part No: 73-12533-05 Card Id: 82, Mac Addr: f0:29:29:ff:02:02, Num Macs: 64 Module Sw Gen: 12594 [Switch Sw Gen: 21] post level: complete pinning-mode: static Max-links: 1 Fabric port for control traffic: FCoE Admin: false FCoE Oper: true FCoE FEX AA Configured: false Fabric interface state: Eth1/10 – Interface Down. State: Configured Fex Port State Fabric Port Eth101/1/1 Down Eth1/10 Eth101/1/2 Down None Eth101/1/3 Down None snippet removed Logs: 08/21/2014 10:00:06.107783: Module register received 08/21/2014 10:00:06.109935: Registration response sent 08/21/2014 10:00:06.239466: Module Online Sequence 08/21/2014 10:00:09.621772: Module Online 08/21/2014 10:13:20.50921: Deleting route to FEX 08/21/2014 10:13:20.58158: Module disconnected 08/21/2014 10:13:20.61591: Offlining Module 08/21/2014 10:13:20.62686: Module Offline Sequence 08/21/2014 10:13:20.797908: Module Offline FEX: 102 Description: FEX0102 state: Online FEX version: 5.1(3)N2(1c) [Switch version: 5.1(3)N2(1c)] FEX Interim version: 5.1(3)N2(1c) Switch Interim version: 5.1(3)N2(1c) Extender Serial: SSI165204YC Extender Model: N2K-C2232PP-10GE, Part No: 73-12533-05 Card Id: 82, Mac Addr: f0:29:29:ff:00:42, Num Macs: 64 Module Sw Gen: 12594 [Switch Sw Gen: 21] post level: complete pinning-mode: static Max-links: 1 Fabric port for control traffic: Eth1/11 FCoE Admin: false FCoE Oper: true FCoE FEX AA Configured: false Fabric interface state: Eth1/11 – Interface Up. State: Active Fex Port State Fabric Port Eth102/1/1 Up Eth1/11 Eth102/1/2 Down None Eth102/1/3 Down None Eth102/1/4 Down None snippet removed Logs: 08/21/2014 10:12:13.281018: Module register received 08/21/2014 10:12:13.283215: Registration response sent 08/21/2014 10:12:13.421037: Module Online Sequence 08/21/2014 10:12:16.665624: Module Online |
Port Channels can be used instead of static pinning between parent switch and FEX so in the event of a fabric interface failure all hosts ports remain active. However, the remaining bandwidth on the parent switch will be shared by all the host ports resulting in an increase in oversubscription.
Part 3. Fabric Extender Topologies
Straight-Through: The FEX is connected to a single parent switch. The servers connecting to the FEX can leverage active-active data plane by using host vPC.
Shutting down the Peer link results in ALL vPC member ports in the secondary peer become disabled. For this reason it is better to use a Dual Homed.
Dual Homed: Connecting a single FEX to two parent switches.
In active – active, a single parent switch failure does not affect the host’s interfaces because both vpc peers have separate control planes and manage the FEX separately.
For the remainder of the post we are going to look at Dual Homed FEX connectivity with host Vpc.
Full configuration:
| N5K1: feature lacp feature vpc feature fex ! vlan 10 ! vpc domain 1 peer-keepalive destination 192.168.0.52 ! interface port-channel1 switchport mode trunk spanning-tree port type network vpc peer-link ! interface port-channel10 switchport access vlan 10 vpc 10 ! interface Ethernet1/1 switchport access vlan 10 spanning-tree port type edge speed 1000 ! interface Ethernet1/3 – 5 switchport mode trunk spanning-tree port type network channel-group 1 mode active ! interface Ethernet1/10 switchport mode fex-fabric fex associate 101 ! interface Ethernet101/1/1 switchport access vlan 10 channel-group 10 mode on N5K2: feature lacp feature vpc feature fex ! vlan 10 ! vpc domain 1 peer-keepalive destination 192.168.0.51 ! interface port-channel1 switchport mode trunk spanning-tree port type network vpc peer-link ! interface port-channel10 switchport access vlan 10 vpc 10 ! interface Ethernet1/2 switchport access vlan 10 spanning-tree port type edge speed 1000 ! interface Ethernet1/3 – 5 switchport mode trunk spanning-tree port type network channel-group 1 mode active ! interface Ethernet1/11 switchport mode fex-fabric fex associate 102 ! interface Ethernet102/1/1 switchport access vlan 10 channel-group 10 mode on |
The FEX do not support LACP so configure the port-channel mode to “ON”
The first step is to check the VPC peer link and general VPC parameters.
Command: “show vpc brief“. Displays the vPC domain ID, the peer-link status, the keepalive message status, whether the configuration consistency is successful, and whether peer-link formed or the failure to form
Command: “show vpc peer-keepalive” Displays the destination IP of the peer keepalive message for the vPC. The command also displays the send and receives status as well as the last update from the peer in seconds and milliseconds
| N5K3# sh vpc brief Legend: (*) – local vPC is down, forwarding via vPC peer-link vPC domain id : 1 Peer status : peer adjacency formed ok vPC keep-alive status : peer is alive Configuration consistency status: success Per-vlan consistency status : success Type-2 consistency status : success vPC role : primary Number of vPCs configured : 1 Peer Gateway : Disabled Dual-active excluded VLANs : – Graceful Consistency Check : Enabled vPC Peer-link status ——————————————————————— id Port Status Active vlans — —- —— ————————————————– 1 Po1 up 1,10 vPC status —————————————————————————- id Port Status Consistency Reason Active vlans —— ———– —— ———– ————————– ———– 10 Po10 up success success 10 N5K3# show vpc peer-keepalive vPC keep-alive status : peer is alive –Peer is alive for : (1753) seconds, (536) msec –Send status : Success –Last send at : 2014.08.21 10:52:30 130 ms –Sent on interface : mgmt0 –Receive status : Success –Last receive at : 2014.08.21 10:52:29 925 ms –Received on interface : mgmt0 –Last update from peer : (0) seconds, (485) msec vPC Keep-alive parameters –Destination : 192.168.0.54 –Keepalive interval : 1000 msec –Keepalive timeout : 5 seconds –Keepalive hold timeout : 3 seconds –Keepalive vrf : management –Keepalive udp port : 3200 –Keepalive tos : 192 |
The trunk interface should be forwarding on the peer link and VLAN 10 must be forwarding and active on the trunk link. Take note if any vlans are on err-disable mode on the trunk.
| N5K3# sh interface trunk ——————————————————————————- Port Native Status Port Vlan Channel ——————————————————————————– Eth1/3 1 trnk-bndl Po1 Eth1/4 1 trnk-bndl Po1 Eth1/5 1 trnk-bndl Po1 Po1 1 trunking — ——————————————————————————– Port Vlans Allowed on Trunk ——————————————————————————– Eth1/3 1-3967,4048-4093 Eth1/4 1-3967,4048-4093 Eth1/5 1-3967,4048-4093 Po1 1-3967,4048-4093 ——————————————————————————– Port Vlans Err-disabled on Trunk ——————————————————————————– Eth1/3 none Eth1/4 none Eth1/5 none Po1 none ——————————————————————————– Port STP Forwarding ——————————————————————————– Eth1/3 none Eth1/4 none Eth1/5 none Po1 1,10 ——————————————————————————– Port Vlans in spanning tree forwarding state and not pruned ——————————————————————————– Eth1/3 — Eth1/4 — Eth1/5 — Po1 — ——————————————————————————– Port Vlans Forwarding on FabricPath ——————————————————————————– N5K3# sh spanning-tree vlan 10 VLAN0010 Spanning tree enabled protocol rstp Root ID Priority 32778 Address 0005.9b1e.82fc This bridge is the root Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec Bridge ID Priority 32778 (priority 32768 sys-id-ext 10) Address 0005.9b1e.82fc Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec Interface Role Sts Cost Prio.Nbr Type —————- —- — ——— ——– ——————————– Po1 Desg FWD 1 128.4096 (vPC peer-link) Network P2p Po10 Desg FWD 1 128.4105 (vPC) Edge P2p Eth1/1 Desg FWD 4 128.129 Edge P2p |
Check the Port Channel database and determine the status of the port channel
| N5K3# show port-channel database port-channel1 Last membership update is successful 3 ports in total, 3 ports up First operational port is Ethernet1/3 Age of the port-channel is 0d:00h:13m:22s Time since last bundle is 0d:00h:13m:18s Last bundled member is Ethernet1/5 Ports: Ethernet1/3 [active ] [up] * Ethernet1/4 [active ] [up] Ethernet1/5 [active ] [up] port-channel10 Last membership update is successful 1 ports in total, 1 ports up First operational port is Ethernet101/1/1 Age of the port-channel is 0d:00h:13m:20s Time since last bundle is 0d:00h:02m:42s Last bundled member is Ethernet101/1/1 Time since last unbundle is 0d:00h:02m:46s Last unbundled member is Ethernet101/1/1 Ports: Ethernet101/1/1 [on] [up] * |
To execute reachability tests, create an SVI on the first parent switch and run ping tests. You must first enable the feature set “feature interface-vlan”. The reason we create an SVI in VLAN 10 is because we need an interfaces to source our pings.
| N5K3# conf t Enter configuration commands, one per line. End with CNTL/Z. N5K3(config)# fea feature feature-set N5K3(config)# feature interface-vlan N5K3(config)# int vlan 10 N5K3(config-if)# ip address 10.0.0.3 255.255.255.0 N5K3(config-if)# no shu N5K3(config-if)# N5K3(config-if)# N5K3(config-if)# end N5K3# ping 10.0.0.3 PING 10.0.0.3 (10.0.0.3): 56 data bytes 64 bytes from 10.0.0.3: icmp_seq=0 ttl=255 time=0.776 ms 64 bytes from 10.0.0.3: icmp_seq=1 ttl=255 time=0.504 ms 64 bytes from 10.0.0.3: icmp_seq=2 ttl=255 time=0.471 ms 64 bytes from 10.0.0.3: icmp_seq=3 ttl=255 time=0.473 ms 64 bytes from 10.0.0.3: icmp_seq=4 ttl=255 time=0.467 ms — 10.0.0.3 ping statistics — 5 packets transmitted, 5 packets received, 0.00% packet loss round-trip min/avg/max = 0.467/0.538/0.776 ms N5K3# ping 10.0.0.10 PING 10.0.0.10 (10.0.0.10): 56 data bytes Request 0 timed out 64 bytes from 10.0.0.10: icmp_seq=1 ttl=127 time=1.874 ms 64 bytes from 10.0.0.10: icmp_seq=2 ttl=127 time=0.896 ms 64 bytes from 10.0.0.10: icmp_seq=3 ttl=127 time=1.023 ms 64 bytes from 10.0.0.10: icmp_seq=4 ttl=127 time=0.786 ms — 10.0.0.10 ping statistics — 5 packets transmitted, 4 packets received, 20.00% packet loss round-trip min/avg/max = 0.786/1.144/1.874 ms N5K3# |
Do the same tests on the second Nexus 5K.
| N5K4(config)# int vlan 10 N5K4(config-if)# ip address 10.0.0.4 255.255.255.0 N5K4(config-if)# no shu N5K4(config-if)# end N5K4# ping 10.0.0.10 PING 10.0.0.10 (10.0.0.10): 56 data bytes Request 0 timed out 64 bytes from 10.0.0.10: icmp_seq=1 ttl=127 time=1.49 ms 64 bytes from 10.0.0.10: icmp_seq=2 ttl=127 time=1.036 ms 64 bytes from 10.0.0.10: icmp_seq=3 ttl=127 time=0.904 ms 64 bytes from 10.0.0.10: icmp_seq=4 ttl=127 time=0.889 ms — 10.0.0.10 ping statistics — 5 packets transmitted, 4 packets received, 20.00% packet loss round-trip min/avg/max = 0.889/1.079/1.49 ms N5K4# ping 10.0.0.13 PING 10.0.0.13 (10.0.0.13): 56 data bytes Request 0 timed out Request 1 timed out Request 2 timed out Request 3 timed out Request 4 timed out — 10.0.0.13 ping statistics — 5 packets transmitted, 0 packets received, 100.00% packet loss N5K4# ping 10.0.0.3 PING 10.0.0.3 (10.0.0.3): 56 data bytes Request 0 timed out 64 bytes from 10.0.0.3: icmp_seq=1 ttl=254 time=1.647 ms 64 bytes from 10.0.0.3: icmp_seq=2 ttl=254 time=1.298 ms 64 bytes from 10.0.0.3: icmp_seq=3 ttl=254 time=1.332 ms 64 bytes from 10.0.0.3: icmp_seq=4 ttl=254 time=1.24 ms — 10.0.0.3 ping statistics — 5 packets transmitted, 4 packets received, 20.00% packet loss round-trip min/avg/max = 1.24/1.379/1.647 ms |
Shut down one of the FEX links to the parent and you see that the FEX is still reachable via the other link that is in the port channel bundle.
| N5K3# conf t Enter configuration commands, one per line. End with CNTL/Z. N5K3(config)# int Eth101/1/1 N5K3(config-if)# shu N5K3(config-if)# end N5K3# N5K3# N5K3# ping 10.0.0.3 PING 10.0.0.3 (10.0.0.3): 56 data bytes 64 bytes from 10.0.0.3: icmp_seq=0 ttl=255 time=0.659 ms 64 bytes from 10.0.0.3: icmp_seq=1 ttl=255 time=0.515 ms 64 bytes from 10.0.0.3: icmp_seq=2 ttl=255 time=0.471 ms 64 bytes from 10.0.0.3: icmp_seq=3 ttl=255 time=0.466 ms 64 bytes from 10.0.0.3: icmp_seq=4 ttl=255 time=0.465 ms — 10.0.0.3 ping statistics — 5 packets transmitted, 5 packets received, 0.00% packet loss round-trip min/avg/max = 0.465/0.515/0.659 ms |
If you would like to futher your knowlege on VPC and how it relates to Data Center toplogies and more specifically, Cisco’s Application Centric Infrastructure (ACI), you can check out my following training courses on Cisco ACI. Course 1: Design and Architect Cisco ACI, Course 2: Implement Cisco ACI, and Course 3: Troubleshooting Cisco ACI,
Data Center Design with Active Active design
Network Insight · High Availability Architecture
Active-Active Data Center Design: Building for Continuous Availability
What happens when one data center can no longer serve its workloads? An active-active architecture aims to keep multiple sites operational at the same time, distributing traffic and capacity while preparing services to withstand component, network and site failures.
Modern applications rarely depend on a single network, server or storage system. They operate across interconnected infrastructure, distributed services and, increasingly, multiple locations. Active-active design brings these components together into an architecture where more than one site can actively serve workloads. But operating two sites simultaneously is not simply a matter of duplicating hardware. Routing, application behavior, data consistency, traffic distribution and failure detection must work together.
Data Center A
Primary active site in this illustration
Applications
Compute
Storage
Serving workloads
Data Center B
Second active site in this illustration
Applications
Compute
Storage
Serving workloads
01 · Connectivity
Route around failures
Use diverse network paths and appropriate routing policies to preserve reachability when links or devices fail.
02 · Consistency
Keep application state safe
Design replication, quorum and application behavior around the consistency requirements of the service.
03 · Recovery
Prove service continuity
Test site isolation, traffic shifts and recovery behavior instead of assuming that redundancy guarantees availability.
i
The essential distinction: both sites can be active without every application being able to survive every failure. True service resilience depends on the complete design—including data consistency, health detection, routing, capacity and application recovery behavior.
Multi-site networking
BGP and DCI
Data consistency
High availability
Failure engineering
Section 01 · Foundations
Understanding Active-Active Data Center Design
Explore how two or more data center sites can serve workloads concurrently, how this differs from active-passive resilience, and why application behaviour, dependency design and failure handling matter as much as network redundancy.
Active vs. passiveAvailabilityFailure domains
Section 01 · Architecture Fundamentals
Understanding Active-Active Data Center Design
Active-active design allows two or more data center sites to serve production workloads concurrently. The objective is to use distributed capacity while maintaining service continuity when a component, network path or site becomes unavailable.
Traditional active-passive designs generally keep one site serving traffic while another is prepared to take over during a failure. In an active-active design, both sites perform useful production work during normal operation. Traffic may be distributed through global load balancing, application-aware routing, anycast services or other mechanisms appropriate to the application and network architecture.
This can improve resource utilisation and provide additional capacity options. It also introduces design questions that cannot be solved by routing alone: where application state lives, how data is replicated, how clients reach healthy services, and what happens when the sites lose contact with one another.
Active-active operating model
Conceptual architecture · Not a deployment blueprint
Data Center A
Serving production traffic
Local compute and services
Data Center B
Serving production traffic
Local compute and services
Both sites are active under normal conditions. The actual traffic split, replication method,
routing policy and failover behaviour depend on the application and implementation.
Three principles that shape the design
Concurrent workload
Both sites contribute to normal service delivery. Capacity planning must account for traffic distribution and whether surviving infrastructure can absorb additional demand after a failure.
Consistent service behaviour
Applications must handle sessions, shared state, data replication and concurrent updates according to their consistency and availability requirements.
Designed failure response
Health checks, routing policies, load balancers and application recovery logic must work together to direct clients towards services that can actually respond.
Important distinction: two active sites do not automatically provide application resilience. If both sites depend on the same database, identity service, WAN path or control-plane component, that shared dependency may still become a single point of failure.
Questions to answer before choosing active-active
- Can the application safely run across multiple sites at the same time?
- How much data loss, if any, is acceptable following a site failure?
- What recovery time is required, and how will client traffic move to healthy services?
- Can one surviving site handle the required workload during a major failure?
- How will split-brain conditions, network partitions and inconsistent state be detected and handled?
Engineering takeaway
Active-active is an end-to-end service architecture, not simply a pair of interconnected data centers. Reliable outcomes depend on coordinated network design, application behaviour, data consistency, capacity planning and tested recovery procedures.
Interactive Lab 01 · Architecture explorer
What Happens When a Data Center Fails?
Switch the deployment model or simulate a site failure. Observe the illustrative traffic paths, site states and engineering implications. This is a conceptual model, not a live network simulator.
Choose a scenario
Data Center A
Active · serving traffic
Data Center B
Active · serving traffic
SCENARIO: ACTIVE-ACTIVE
Both sites serve production traffic
Workloads at both sites are available to serve requests. Traffic distribution depends on application placement, routing, load balancing and health checks. Shared dependencies and data consistency still need deliberate design.
Engineering checks
- Confirm application and data consistency.
- Verify each site's independent capacity.
- Test routing and service health during failure.
Section 02 · Routing architecture
Multi-Site Network Architecture and BGP
Follow traffic between data centers and external networks. Understand how BGP exchanges reachability, how routing policy and multipath influence path selection, and how engineers avoid asymmetric traffic, route leaks and unexpected failover behaviour.
DCI designBGP policyTraffic engineering
Section 02 · Network Architecture
Multi-Site Network Architecture and BGP
An active-active design needs more than two functioning data centers. It needs predictable routing, resilient inter-site connectivity and a clear policy for how users and applications reach services when paths or entire sites fail.
The network provides the connectivity between data centers, users, applications and external services. In a multi-site architecture, Data Center Interconnect (DCI) links connect the sites, while routing and traffic-management systems determine which paths are available and where traffic should be sent.
Border Gateway Protocol (BGP) is commonly used to exchange reachability information between sites, WAN networks and external networks. Its policy controls can influence path selection and traffic distribution. However, BGP does not by itself verify that an application is healthy or that its data is consistent.
Two-site network architecture
Illustrative topology · Not to scale
Data Center A
Applications and local services
Leaf
Spine
Border
Local routing · BGP policy
Data Center B
Applications and local services
Leaf
Spine
Border
Local routing · BGP policy
Diverse links only provide meaningful resilience when they avoid relevant shared risks,
such as the same conduit, carrier equipment, power source or upstream failure domain.
The three network design layers
Data Center Interconnect
DCI may use routed IP connectivity, an optical transport service or an overlay architecture. Choose bandwidth, latency and path diversity according to application traffic and replication requirements—not simply the number of physical links.
Routing and reachability
BGP can advertise site prefixes and apply routing policy. Multipath forwarding may use multiple eligible routes when supported and correctly configured. Route selection and forwarding behaviour must be validated for the actual platform and topology.
Service traffic steering
DNS-based global load balancing, anycast, application load balancers or other steering mechanisms can direct clients towards available services. Health checks must reflect genuine service readiness, not just network reachability.
BGP: control the path, then validate the outcome
A common design advertises service or site prefixes from more than one location. The routing policy determines which advertisements are accepted and how paths are preferred. Depending on the design, engineers may use attributes such as LOCAL_PREF within an autonomous system, AS_PATH for inter-AS policy, or multipath features to support selected equal-cost paths.
The important distinction is between route availability and service availability. A router may have a valid route to a site whose application is unhealthy. Conversely, an application may be healthy while a routing or traffic-steering policy prevents clients from reaching it. These systems need coordinated health signals and carefully tested failure behaviour.
Design warning — avoid accidental traffic tromboning. If a client reaches a service in one site but the application retrieves data or calls a dependency in the other site, traffic may cross the DCI unnecessarily. This can increase latency, consume inter-site bandwidth and complicate failure handling. Keep traffic local where practical, while preserving the required consistency and availability guarantees.
What should happen during a failure?
- DCI link failure: traffic should use a surviving path where one exists, with convergence and capacity validated under load.
- Border router failure: alternate routing paths should remain available without creating a forwarding loop or black hole.
- Site failure: route withdrawal and service-health mechanisms should steer new traffic away from unavailable services.
- Application failure: health checks should stop advertising or selecting an endpoint that cannot deliver the required service.
- Network partition: routing and application policies must avoid unsafe assumptions about the remote site's state or data ownership.
Architecture checks before deployment
- Identify shared physical and logical failure domains across DCI paths.
- Document prefix advertisements, routing policy, route filtering and intended path preferences.
- Check convergence behaviour and whether surviving links can carry the expected load.
- Validate inbound and outbound traffic steering independently; they may follow different paths.
- Test application health checks alongside routing changes and site-isolation scenarios.
Engineering takeaway
BGP and DCI establish and control network reachability; they do not guarantee that an application is usable. A resilient active-active design coordinates routing, traffic steering, application health, capacity and data consistency—and verifies the result through controlled failure tests.
Interactive Lab 02 · Multi-site routing
Trace the Route Between Data Centers
Explore how inter-site connectivity, BGP reachability and routing policy influence traffic. Simulate a DCI link failure or route withdrawal, then select a router or network to inspect its role.
Select a routing scenario
Illustrative preferred path
Illustrative alternate path
Failed or withdrawn path
STATE: NORMAL ROUTING
Reachability is available through the configured paths
The diagram illustrates an available inter-site path and route advertisement. Actual forwarding depends on the installed route, routing policy, next-hop reachability and data-plane health.
Select a network component
Select either border router, an application network or the external AS in the diagram to explore its role in route propagation and traffic forwarding.
DCI statusAvailable
Illustrative routeAdvertised
Engineering focusValidate policy
| Item | Illustrative state | What to verify |
|---|---|---|
| Inter-site path | Available | Independent physical paths, capacity and forwarding health. |
| Route advertisement | Present | Accepted prefixes, next hop, policy and route installation. |
| Service reachability | Check required | End-to-end application health, not only control-plane state. |
Section 03 · Overlays and mobility
Layer 2 Extension, Overlays and Workload Mobility
Compare stretched Layer 2 networks with routed Layer 3 designs, and examine how VXLAN/EVPN overlays can support multi-site connectivity. Consider workload mobility, broadcast scope, failure propagation and the risk of traffic tromboning before extending networks between sites.
VXLAN / EVPNLayer 2 vs. Layer 3Workload mobility
Section 03 · Layer 2 and Overlay Design
Layer 2 Extension, Overlays and Workload Mobility
Workload mobility across data centers can simplify some application migrations and recovery scenarios. The challenge is deciding how much network continuity is actually required—and whether extending a Layer 2 domain is the right way to achieve it.
In a multi-site environment, applications may need to move between sites for maintenance, capacity balancing, disaster recovery or infrastructure changes. Some workloads can move while retaining their IP addresses; others can reconnect through a new endpoint or service address. The application’s requirements should determine the network design, rather than assuming that every workload needs a stretched subnet.
A Layer 2 extension carries an Ethernet broadcast domain across multiple locations. This can preserve subnet membership and support certain migration workflows, but it also extends the domain’s failure and operational characteristics. A Layer 3 design keeps subnets local to each site and uses routing to connect them, often reducing the scope of Layer 2 failure propagation.
Approach A
Stretched Layer 2
The same VLAN or subnet is available across sites, subject to the chosen extension technology and design constraints.
- Can support address-preserving workload mobility.
- May carry broadcast and unknown-unicast traffic across the DCI.
- Requires careful control of loops, failure domains and inter-site latency.
- Can make partial failures harder to diagnose.
Approach B
Routed Layer 3
Each site uses its own routed subnets, and connectivity between sites is provided through routing and application-level service discovery or traffic steering.
- Provides clearer Layer 3 failure boundaries.
- Supports route summarisation and explicit traffic policy.
- May require application, DNS or load-balancer changes during migration.
- Often simplifies fault isolation and scaling.
VXLAN and EVPN: separating the overlay from the underlay
VXLAN (Virtual Extensible LAN) encapsulates Layer 2 frames in UDP packets so that virtual networks can be carried across an IP underlay. VXLAN Tunnel Endpoints (VTEPs) perform encapsulation and decapsulation. EVPN (Ethernet VPN), commonly used as a control plane for VXLAN fabrics, distributes MAC and IP reachability information and supports policy-driven tenant connectivity.
This separation allows an IP-routed fabric to carry multiple logical networks without requiring the physical underlay itself to be one large Layer 2 domain. EVPN can support distributed gateway designs and multi-homing, depending on the implementation. It does not, however, remove the need to design inter-site routing, failure handling and application consistency deliberately.
Overlay network across two sites
Conceptual VXLAN/EVPN model
Data Center A
IP underlay · Local VTEP
Workload A1
Workload A2
Data Center B
IP underlay · Local VTEP
Workload B1
Workload B2
The diagram is illustrative. Actual VTEP placement, EVPN route exchange, gateway placement and whether an overlay spans both sites depend on the design.
Overlay connectivity can provide flexibility, but stretching a tenant network across sites
should be a deliberate requirement—not the default for every application.
Where do OTV, FabricPath and LISP fit?
OTV
Overlay Transport Virtualization was designed to extend selected Layer 2 connectivity between data centers while limiting some traditional Layer 2 extension risks. Treat it as a platform- and lifecycle-dependent technology, not a universal modern default.
FabricPath
Cisco FabricPath introduced a Layer 2 fabric approach using a fabric control plane rather than relying on conventional spanning tree for all forwarding decisions. It is mainly relevant when understanding existing or legacy environments.
LISP
Locator/ID Separation Protocol separates endpoint identity from routing location. It can support mobility and mapping use cases in appropriate architectures, but it is not interchangeable with VXLAN or EVPN.
Design principle: do not extend Layer 2 simply because an application currently uses a fixed IP address. First check whether the application can tolerate a different subnet, use a stable service endpoint, or reconnect through DNS or a load balancer. A routed design may meet the requirement with a smaller failure domain.
Workload mobility and traffic tromboning
Preserving a workload’s IP address does not guarantee that its traffic path remains optimal. After a workload moves, clients or dependent services may still reach it through a remote site. This can create traffic tromboning: packets travel across the DCI to reach a service and may cross back again to reach another dependency.
The result can be increased latency, unnecessary bandwidth consumption and a more complicated failure path. Mobility planning should therefore include endpoint discovery, routing updates, load-balancer behaviour, dependency locality and the time needed for existing sessions to recover.
Questions to resolve before extending a network
- Does the workload genuinely require the same subnet at both sites?
- What are the latency, bandwidth and MTU requirements across the DCI?
- How are MAC/IP reachability and gateway behaviour handled after a move or failure?
- Can a Layer 2 loop, broadcast event or control-plane issue affect both sites?
- Will traffic remain local to the active workload and its dependencies?
- What happens to established sessions when the workload or site becomes unavailable?
Engineering takeaway
VXLAN and EVPN can provide scalable overlay connectivity, while routed designs can keep site boundaries clearer. Choose Layer 2 extension only when the application and operational requirements justify it, and test mobility, failure isolation and traffic locality before relying on it for resilience.
Interactive design lab · 03
Layer 2 Extension, Overlays and Workload Mobility
Compare three ways to connect data-centre sites. Change the design, simulate an inter-site failure, and examine what happens to addressing, mobility and the fault domain.
1. Choose a connectivity design
2. Test the inter-site connection
Shared segment is available
Both sites use the same example subnet. This can support address continuity, but it also extends Layer 2 dependencies across the sites.
Addressing
Same subnet at both sites
Mobility approach
Layer 2 continuity
Failure consideration
Broadcast and loop control
Engineering trade-off
A stretched VLAN can make subnet continuity easier, but the design must deliberately control loop prevention, broadcast scope, gateway behaviour and inter-site failure handling.
- Define the Layer 2 extension boundary
- Test behaviour during link and site failure
- Validate gateway redundancy and convergence
- Confirm application recovery requirements
Educational topology only — not a live network simulator. VXLAN is an encapsulation technology; EVPN can provide a BGP-based control plane. Actual mobility, redundancy and failure behaviour depend on the chosen platform, configuration and application design.
Section 04 · Application resilience
Application, Storage and State Consistency
Network redundancy cannot guarantee application availability on its own. Examine stateless and stateful services, session continuity, synchronous and asynchronous replication, quorum, split-brain risk, and the trade-offs between recovery time and potential data loss.
Application stateReplicationRPO / RTO
Section 04 · Application and Data Resilience
Application, Storage and State Consistency
A network can keep routes available while an application remains unable to serve requests. Active-active resilience therefore depends on how application instances, user sessions and persistent data behave across sites—not just on redundant links and routers.
In an active-active environment, application instances in different data centers may process requests concurrently. That arrangement works well only when the application can manage shared state, concurrent operations and failures consistently. A stateless service is often easier to distribute because each request can be handled by any healthy instance, provided its dependencies are also available.
Stateful applications introduce more complexity. They may depend on databases, distributed caches, message queues, session stores or file systems. Each dependency needs an explicit availability and consistency strategy. Replicating the application tier alone does not make the whole service resilient if its database or identity provider remains a single point of failure.
Application and data layers across two sites
Conceptual architecture
Data Center A
Active application instance
Application service A Local connection pool Data replica / access pathData Center B
Active application instance
Application service B Local connection pool Data replica / access path
Replication may be synchronous, asynchronous or managed by a distributed data platform.
The correct model depends on the database technology, latency, failure assumptions and
application consistency requirements.
The drawing represents logical relationships, not a recommendation to permit unrestricted
concurrent writes to every replicated data store.
Synchronous versus asynchronous replication
Replication determines how updates made at one site become available at another. The key trade-off is between acknowledgement latency, consistency guarantees and the amount of data that may be lost during a failure. These characteristics vary by platform and configuration; the terms alone do not guarantee a particular recovery outcome.
Replication model A
Synchronous replication
A write is acknowledged only after the configured synchronous durability or commit conditions have been met.
- Can reduce the recovery point exposure for acknowledged writes.
- Inter-site latency can affect write response times.
- A link or site failure may affect write availability, depending on quorum and product design.
- Requires careful planning for network partitions and failover.
Replication model B
Asynchronous replication
A write can be acknowledged before the remote replica has received or committed the update.
- Can reduce the impact of remote-site latency on write acknowledgement.
- Replication lag may leave the remote site temporarily behind.
- A sudden failure can result in loss of updates not yet replicated.
- Recovery procedures must account for lag and data reconciliation.
Important: synchronous replication does not automatically mean zero data loss in every failure scenario, and asynchronous replication does not always imply unacceptable loss. The real outcome depends on commit semantics, quorum, durable storage, topology and the failure being considered. Define the required Recovery Point Objective (RPO) and Recovery Time Objective (RTO), then validate them against the actual platform.
Three application-level challenges
Session continuity
If a client moves between sites, its session may depend on local memory, a shared session store, a signed token or another mechanism. Design for reconnects and retries instead of assuming a transport path change preserves every session.
Concurrent updates
Two active instances may attempt to modify the same logical record. Transactions, conflict resolution, idempotency and application-specific consistency rules must prevent duplicated or conflicting operations.
Dependency availability
An application instance may be healthy while its database, authentication service, queue or external API is not. Service health should reflect the dependencies required to complete real user operations.
Quorum, network partitions and split-brain
A network partition can leave two sites running while preventing them from communicating. If both sides independently accept conflicting writes or claim ownership of the same resource, the system may enter a split-brain condition. Restoring connectivity does not necessarily reconcile the resulting state safely.
Distributed systems often use quorum, leases, fencing tokens, leader election or other coordination mechanisms to decide which operations are permitted during a partition. A two-site deployment may need a third failure-independent voting participant or witness, depending on the platform. A witness can help establish a decision, but it does not replace data replication or application-level recovery design.
Network partition is not the same as site failure. If the DCI fails, both sites may still be running and serving local clients. The architecture must decide whether each site can safely continue, whether one side must stop writes, and how service ownership is re-established when communication returns.
Questions to answer before production
- Can application instances safely process requests concurrently in both sites?
- Where is session state stored, and what happens when a client is redirected?
- What RPO and RTO are required for each critical data store?
- How are quorum, ownership and split-brain conditions handled during a partition?
- Can the surviving site sustain the required transaction rate and connection load?
- How will replication lag, failed writes, stale reads and recovery conflicts be detected?
- Have failover and failback been tested with real application transactions, not just ping tests?
Engineering takeaway
Active-active application resilience requires an explicit strategy for state, replication, coordination and recovery. Network redundancy helps services communicate, but only the combined behaviour of the application, data platform and network determines whether service can continue safely during a failure.
Interactive design lab · 04
Application, Storage and State Consistency
A network can remain connected while an application becomes unsafe or unavailable. Explore replication modes, simulate an inter-site partition, and examine why two active sites need a deliberate write-ownership strategy.
1. Choose a data replication model
2. Choose a scenario
Replication is operating normally
In this conceptual synchronous model, a write waits for the configured remote acknowledgement before it is confirmed. Real guarantees depend on the data platform and commit policy.
Write behaviour
Waits for required acknowledgement
Data risk
Depends on commit and quorum rules
Recovery priority
Validate quorum and ownership
Engineering checks
A healthy network path does not prove that a transaction is durable or that both replicas agree. Check the application's consistency contract and the data platform's failure behaviour.
- Confirm what a successful commit guarantees
- Define quorum and write ownership
- Measure replication delay and transaction errors
- Test failover, fencing and safe recovery
Educational model only — no real transactions or replication occur. The diagram illustrates design trade-offs, not a guarantee that a particular database or storage platform behaves this way. Synchronous replication does not automatically prevent data loss in every failure; asynchronous replication may have a non-zero recovery point.
Section 05 · Operations and recovery
Failure Scenarios, Observability and Recovery
Move beyond theoretical redundancy and test how the architecture behaves under real failure conditions. Trace link loss, BGP changes, application dependency failures, replication lag and site isolation using telemetry, incident procedures and measured recovery objectives.
Failure testingTelemetryFailover and failback
Section 05 · Observability and Recovery
Failure Scenarios, Observability and Recovery
Active-active architecture must be designed around failure, not just normal operation. Reliable recovery depends on detecting what has failed, identifying the affected services, understanding the data state and directing traffic towards capacity that can safely serve it.
Redundant links and duplicated infrastructure reduce some risks, but they do not eliminate failures. A site can remain reachable while its applications are unhealthy; a routing session can remain established while packets are being dropped; and a database replica can be online while lagging behind the active data set. Effective operations require end-to-end observability across network, application and data layers.
Identify the real failure
Use telemetry and health checks to distinguish a failed link, an unreachable site, a degraded application and an unhealthy data dependency.
Choose a safe response
Determine whether traffic should move, whether writes can continue, and whether the surviving site has enough capacity to take additional work.
Prove service recovery
Confirm that users can complete meaningful transactions, not merely that interfaces are up or routing adjacencies have recovered.
Five failure scenarios to test
Inter-data-center link failure
One DCI path fails while both sites remain operational. Check route convergence, alternate-path capacity, packet loss and whether replication or application dependencies are affected.
Validate: path failover · latency · loss · capacityBorder router or routing failure
A border device fails, a route is withdrawn or a policy change directs traffic incorrectly. Verify that alternate routes are usable and that recovery does not create a forwarding loop or black hole.
Validate: BGP state · route changes · forwarding pathApplication or dependency failure
Network connectivity remains available, but an application instance, database or identity service cannot complete requests. Check whether health signals remove the affected endpoint from service selection.
Validate: synthetic transactions · dependency healthReplication lag or data inconsistency
The remote data copy falls behind or a replication process stops. Establish whether the secondary site is safe to promote, what data could be missing and how consistency will be verified before normal service resumes.
Validate: replication lag · RPO · recovery procedureComplete site loss or network partition
A site becomes unavailable, or both sites remain running but lose communication. Confirm the failover decision, capacity at the surviving site, data ownership and safeguards against split-brain.
Validate: site isolation · quorum · service continuityA practical incident investigation sequence
During an incident, investigate from the observed user impact towards the underlying dependency. Correlating timestamps across routing events, interface counters, application logs and replication metrics helps distinguish the initiating failure from symptoms that appear later.
From alert to verified recovery
Operational workflow
1
Scope impact
Identify affected users, services, sites and transactions.
2
Correlate signals
Compare network, application and data telemetry.
3
Apply safe recovery
Follow tested routing, failover and data procedures.
4
Verify service
Test transactions, data integrity and performance.
This is a general investigation pattern. Follow your incident procedures and service-specific
recovery runbooks when making production changes.
What should you monitor?
| Layer | Useful signals | What they help reveal |
|---|---|---|
| Network | Interface errors, packet loss, latency, BGP changes, path utilisation | Transport degradation, route changes and capacity pressure |
| Application | Request success rate, latency, error rate, synthetic transactions | Whether users can actually complete operations |
| Data | Replication lag, commit failures, consistency checks, storage health | Whether the remote site has a safe and sufficiently current data set |
| Platform | CPU, memory, connection pools, queue depth, storage throughput | Whether the surviving site can absorb additional workload |
| Recovery | Failover duration, RTO/RPO achievement, recovery test results | Whether the design meets its agreed recovery objectives |
Do not confuse an infrastructure health check with a service health check. An interface can be up, a BGP session can be established and a server can respond to ping while a real transaction fails. Use application-aware probes and representative synthetic transactions wherever practical.
Failover, failback and recovery objectives
Recovery Time Objective (RTO) defines the target time to restore a service after disruption. Recovery Point Objective (RPO) defines the acceptable data-loss window, expressed as time or an equivalent recovery-point requirement. Different applications and data stores may need different objectives; a single pair of numbers should not automatically be applied to every service.
Failover is only half of the recovery story. Failback—returning service to its intended operating arrangement—can introduce another period of risk. Before returning workloads, confirm data consistency, replication health, routing policy, capacity and the correct ownership of stateful services. Avoid automatic failback if it could trigger repeated site switching or disrupt active sessions.
Operational readiness checklist
- Document the expected behaviour for link, device, application, data and full-site failures.
- Ensure alerts identify the affected service and provide enough context for investigation.
- Test routing convergence and verify actual packet forwarding after route changes.
- Confirm that the surviving site has enough compute, storage, network and application capacity.
- Define who can authorise failover, data promotion and failback during an incident.
- Record measured recovery times and data outcomes during controlled exercises.
- Review failures and near misses to improve runbooks, monitoring and architecture.
Engineering takeaway
Resilience must be measured through observed service outcomes. Combine network telemetry, application health, data replication metrics and tested recovery procedures to determine whether the active-active architecture can withstand the failures it was designed to handle.
Interactive design lab · 05
Failure Scenarios, Observability and Recovery
Simulate a router failure, an inter-site partition or a complete site outage. Observe the illustrative network state, monitoring signals and recovery checks that matter before declaring an active-active service healthy.
1. Select a failure scenario
Both sites are operational
The conceptual topology shows active services at both sites. Monitoring should confirm end-to-end transaction success, route health, replication status and available capacity.
Network reachability
Primary and inter-site paths available
Application service
Both sites serving
Recovery decision
Monitor and validate health
Observability panel
Routing adjacencyHealthy
End-to-end transactionsHealthy
Inter-site connectivityHealthy
Replication / coordinationHealthy
Recovery checklist
A green interface or established BGP session is only one signal. Correlate network telemetry with application transactions, data health and the capacity of the surviving site.
- Confirm the failure and its scope
- Validate the alternate forwarding path
- Check application and data integrity
- Verify recovery objectives are met
Educational simulation only. No live devices, BGP sessions or monitoring systems are contacted. The alternate path is illustrative; real failover depends on topology, routing policy, detection timers, capacity, application dependencies and the platform's recovery behaviour.
Section 06 · Hybrid architecture
Cloud Connectivity and Modern Design Choices
Explore how data centers connect to cloud networks using private connectivity, VPNs, dynamic routing, VPC peering and hub-based services such as Google Cloud Network Connectivity Center. Compare resilience, route control, security, latency and cost without assuming one cloud model fits every deployment.
Hybrid cloudVPC and BGPConnectivity choices
Active-Active Data Center Design · Section 06
Cloud Connectivity and Modern Design Choices
Active-active design does not stop at the data center perimeter. Applications may depend on private cloud services, public cloud workloads, shared identity systems, or data platforms in another environment. The challenge is to connect these resources without creating hidden dependencies, unnecessary latency, or new single points of failure.
1. Separate data center interconnect from hybrid cloud connectivity
A data center interconnect (DCI) connects separate data center sites. Hybrid cloud connectivity connects an enterprise network or data center to cloud networks and services. These designs can overlap, but they solve different problems and should be assessed independently.
For example, two active data centers may exchange application traffic over a dedicated DCI while each site also has its own connection to a cloud provider. Another design may route both sites through a shared cloud connectivity hub. The right choice depends on application placement, traffic patterns, security boundaries, and failure-domain requirements.
Illustrative hybrid connectivity model
Data Center A
Active workloads
Local routing and security
Private DCI or routed inter-site path
Data Center B
Active workloads
Independent failure domain
Enterprise / DCI edge
Routing policy and controlled egress
Private connectivity or VPN
Cloud network
VPCs, cloud workloads and services
This is a conceptual view, not a required physical topology. Connectivity can be direct, hub-and-spoke, or provider-mediated. Redundancy must be verified across the complete path, not just at the visible connection.
2. Choose the connectivity model for the workload
Cloud connectivity is not one universal design. Dedicated private links, provider-managed private connectivity, and IPsec VPNs offer different trade-offs in capacity, latency, encryption, deployment time, and cost.
01
Private connectivity
Often suited to predictable, sustained traffic and workloads with demanding bandwidth or latency requirements. Validate provider availability, port and circuit diversity, routing hand-offs, and the actual end-to-end service commitment.
02
Site-to-site VPN
IPsec VPN can provide encrypted connectivity over an IP underlay. High-availability VPN designs use redundant tunnels or endpoints, but the underlying internet paths and provider failure domains still matter.
03
Cloud routing
Cloud routers and BGP sessions can exchange routes dynamically where supported. Define which prefixes are advertised, accepted, preferred, or filtered, and test behaviour when a peer or path fails.
04
Hub-and-spoke connectivity
A cloud transit or connectivity hub can simplify connections between multiple networks. It also becomes an important policy and operational component that needs its own resilience and capacity planning.
3. Understand VPCs, peering and cloud connectivity hubs
In Google Cloud, a Virtual Private Cloud (VPC) provides a logical network boundary for cloud resources. VPC connectivity, Cloud VPN, Cloud Router, and Network Connectivity Center (NCC) address different parts of hybrid and multi-network design. Other providers have comparable concepts, but their capabilities and terminology differ.
| Building block | Design role | Important consideration |
|---|---|---|
| VPC network | Provides a logical network environment for cloud resources. | Plan address ranges, segmentation, route policy, and access controls. |
| VPC peering | Connects eligible VPC networks through a peering relationship. | Do not assume peering provides transitive routing through another VPC. |
| Cloud VPN | Provides IPsec-based connectivity between networks. | Plan tunnel redundancy, routing, encryption requirements, and failover testing. |
| Cloud Router / BGP | Supports dynamic route exchange for compatible connectivity services. | Validate advertised prefixes, learned routes, route priorities, and withdrawal behaviour. |
| Network Connectivity Center | Provides a hub-based framework for supported network connectivity patterns. | Check supported resources, topology constraints, route propagation, and configuration requirements. |
Do not confuse connectivity with end-to-end resilience
A cloud hub, a pair of VPN tunnels, or two physical circuits does not automatically make an application highly available. Shared carrier infrastructure, common provider endpoints, route-policy errors, cloud-region dependencies, DNS, identity, and application state can still create a common failure mode.
4. Design the routing and failure behaviour
Once connectivity is established, define what happens when a link, tunnel, router, data center, or cloud attachment becomes unavailable. Routing must support the intended failover behaviour without creating loops, accidental transit, or unintended access between security zones.
Route control
Advertise only required prefixes. Apply explicit import and export policy, avoid overlapping address spaces, and understand how route preference changes during failure.
Path diversity
Check whether redundant connections use independent devices, facilities, carriers, cloud endpoints, and regions where the availability objective requires it.
Security boundaries
Apply segmentation, firewall policy, identity controls, and inspection requirements to both primary and failover paths. A backup route should not bypass required controls.
Application dependencies
Validate DNS, authentication, secrets, databases, monitoring, and other shared services. Network reachability alone does not prove that the service can operate.
5. Compare the design options before committing
Select connectivity according to measured workload requirements and operational constraints—not simply because a particular product or topology is available.
| Design question | What to evaluate |
|---|---|
| Traffic profile | Peak and sustained throughput, burst behaviour, packet size, and growth expectations. |
| Performance | Latency, jitter, packet loss, regional distance, and application sensitivity. |
| Availability | Independent paths, redundant endpoints, provider failure domains, and tested recovery behaviour. |
| Routing | Address planning, BGP policy, route propagation, failover timing, and prevention of unintended transit. |
| Security | Encryption, segmentation, inspection, identity, and compliance requirements. |
| Operations and cost | Provisioning lead time, monitoring, support ownership, data transfer charges, and ongoing operational effort. |
6. Validate the architecture with failure tests
Test the complete application path, including the network and the dependencies it uses. Record what users experience, how quickly the system detects a failure, which routes change, and whether recovery occurs within the required objective.
- Disconnect one connectivity path and verify traffic moves as designed.
- Withdraw a route and confirm that only intended prefixes change reachability.
- Test a cloud attachment or VPN endpoint failure where the environment permits.
- Verify that security policy remains enforced on the alternate path.
- Measure application recovery, not just routing convergence.
- Document ownership, alerts, escalation, and restoration procedures.
Engineering takeaway
Active-active data center design and hybrid cloud connectivity should be designed together where workloads depend on both—but they are not the same problem. Choose private connectivity, VPN, peering, or a hub-based model according to workload needs, then validate route policy, failure-domain independence, security, and application recovery end to end.
Interactive design explorer · Section 6
Cloud Connectivity and Modern Design Choices
Choose a connectivity pattern to explore how an on-premises network can connect to cloud networks. Compare the conceptual architecture, routing considerations and security decisions before selecting a design for a real environment.
01 / Select a connectivity pattern
02 / Architecture view
Encrypted tunnel
Conceptual diagram, not a provider-specific reference architecture. Redundant tunnels, gateways and route exchange must be designed and tested explicitly.
03 / Design assessment
Site-to-site VPN
Build an encrypted connection between the enterprise network and the cloud provider's VPN endpoint. The internet may carry the tunnel traffic, while routing controls which prefixes can traverse it.
Connection model
Encrypted tunnel
Routing focus
Prefixes and failover
Security focus
IKE / IPsec policy
Resilience focus
Multiple tunnels
Engineering checks
- Confirm tunnel encryption, authentication and key-management requirements.
- Validate route advertisements, allowed prefixes and return-path symmetry.
- Test failure detection, alternate paths, throughput and maximum packet size.
Watch-outs
- A tunnel being established does not prove application reachability.
- One tunnel or gateway can remain a single point of failure.
High availability is an end-to-end property
Consider independent circuits, gateways, cloud regions or zones where supported, route convergence, capacity under failure, and the application's own dependency model. Two connections that share the same failure domain may not provide meaningful redundancy.
Routing and security are separate decisions
Route exchange determines reachability; firewall policy, segmentation, identity and encryption determine what is permitted and protected. Validate both directions and test the application path rather than relying on a successful routing adjacency alone.
Network Insight · Article Summary
Active-Active Data Center Design: Key Takeaways
A practical recap of the architecture, routing, application dependencies, failure planning and cloud connectivity decisions behind a resilient multi-site data center.
Active-active data center design aims to use multiple sites to serve workloads concurrently while maintaining service continuity when links, network devices, applications or entire sites fail. Achieving this requires more than duplicating infrastructure: routing, application state, data consistency, security and recovery procedures must work together.
01
Architecture and workload placement
Define which services operate at each site, how traffic is distributed, and what happens when one location becomes unavailable. Active-active is a service and workload design decision, not simply a topology with two data centers.
02
Multi-site routing and BGP
Use explicit routing policy, controlled prefix advertisement and deliberate path selection. Validate convergence, asymmetric paths, route withdrawal and the capacity of the surviving site under failure conditions.
03
Layer 2 extension and overlays
Choose between routed inter-site connectivity, selective Layer 2 extension and overlay-based approaches according to workload requirements. Avoid extending broadcast domains across sites without a clear operational and failure-containment justification.
04
Application and data consistency
Understand replication lag, transaction behaviour, quorum, write ownership and recovery objectives. Network reachability alone cannot guarantee consistent data or prevent conflicting writes during a partition.
05
Failure detection and recovery
Monitor routing and infrastructure alongside application transactions, replication health and user-visible service availability. Test realistic failures and document safe failover, failback and recovery procedures.
06
Cloud connectivity and modern choices
Select VPN, private connectivity, network peering or hub-based connectivity based on reachability, routing, security and resilience requirements. Confirm provider-specific route propagation and service limitations before implementation.
Six engineering principles to carry into your design
- Design around service availability, not device availability alone.
- Ensure the surviving site has sufficient capacity for its intended failure role.
- Keep routing policy, application behaviour and data consistency aligned.
- Use Layer 2 extension only where the workload genuinely requires it.
- Measure end-to-end service health, not just link and routing status.
- Prove failover and recovery through repeatable tests and documented outcomes.
Final perspective: resilience must be engineered and tested
The strongest active-active designs make failure behaviour explicit. They define how traffic moves, how stateful applications continue safely, how data is protected, and how operators recognise and resolve degraded service. They also account for partial failures, where a site remains reachable but can no longer safely perform its intended role.
No single technology guarantees resilience. BGP, overlays, replication, cloud connectivity and redundant infrastructure are building blocks whose effectiveness depends on the requirements, implementation and operational testing around them.
The central takeaway
Build for the failure you need to survive, define the service outcome you must preserve, and test that outcome before relying on the architecture in production.
```
Support Network Insight
```
Support Network Insight
Help support independent engineering guides, interactive
networking tools, and future learning labs.