Anycast HSRP Archives

EVPN – MPLS-based Layer 2 VPN

December 16, 2015

by Matt Conran Blog

Network Insight · Network Virtualisation

EVPN Explained: A Control Plane for Modern Ethernet VPNs

Ethernet VPN (EVPN) provides a scalable way to distribute Ethernet endpoint reachability through a BGP-based control plane. It allows network devices to learn where remote MAC addresses and, where applicable, associated IP addresses are reachable without relying exclusively on data-plane learning across the entire network.

EVPN is used in service-provider VPNs, data-centre fabrics and data-centre interconnects. It can operate over different transport technologies, including Multiprotocol Label Switching (MPLS) and Virtual Extensible LAN (VXLAN). Understanding EVPN therefore requires separating three related but distinct functions: how endpoint reachability is advertised, how traffic is transported, and how packets are forwarded at the network edge.

The central engineering principle

MP-BGP EVPN distributes reachability information. MPLS or VXLAN provides the data-plane transport for the service. The underlay provides the connectivity required to reach the relevant provider-edge routers or VXLAN tunnel endpoints. These roles work together, but they are not interchangeable.

Three concepts to understand first

01 / CONTROL PLANE

Endpoint reachability

MP-BGP EVPN advertises endpoint information using defined EVPN route types. Remote devices use this information to build the forwarding state needed to reach endpoints across the network.

02 / TRANSPORT

MPLS or VXLAN

MPLS-based EVPN uses label-based transport, while VXLAN-based EVPN encapsulates Ethernet frames in UDP/IP packets between tunnel endpoints. The choice depends on the network architecture and service.

03 / RESILIENCE

Multihoming and recovery

EVPN supports defined mechanisms for multihoming, endpoint mobility and efficient forwarding. Correct operation still depends on the topology, configuration, platform support and failure behaviour.

What this guide will help you investigate

This guide moves from the limitations of traditional Layer 2 VPNs to EVPN route distribution, MPLS and VXLAN transport, BGP-based data-centre fabrics, multihoming and operational troubleshooting. Interactive models will help you trace endpoint reachability, compare transport choices and diagnose a missing remote MAC advertisement.

Engineering takeaway: EVPN is not a transport protocol in itself. It is a BGP-based control-plane technology that can support different Ethernet VPN services and data-plane transports. Keeping control-plane learning separate from transport and forwarding is the foundation for understanding its architecture.

Section 1 · Foundations

Traditional Layer 2 VPNs: Connecting Ethernet Networks Across a Provider

A Layer 2 VPN allows geographically separated customer networks to communicate as though they share an Ethernet service, even when their traffic crosses a service provider's infrastructure. Before EVPN, providers commonly built these services using technologies such as Virtual Private LAN Service (VPLS), point-to-point pseudowires, and related Ethernet transport mechanisms.

The traditional service model

Customer Site A Ethernet frames from local devices

Provider Network Frames carried over a provisioned Layer 2 service

Customer Site B Frames delivered to the remote Ethernet segment

The provider network supplies the connectivity between customer sites. Depending on the service, it may carry traffic between two endpoints or provide multipoint Ethernet connectivity between several locations.

Point-to-point

Pseudowires

A pseudowire emulates a Layer 2 connection across a packet network. It is well suited to connecting two endpoints, such as two customer sites, but a service with many sites can require additional service instances and careful provisioning.

Multipoint Ethernet

VPLS

Virtual Private LAN Service provides multipoint Ethernet connectivity, allowing multiple customer sites to participate in a bridged service. The provider must still handle MAC-address learning, forwarding state, and the distribution of traffic across the service.

Where the traditional model becomes difficult

These technologies can deliver reliable Layer 2 connectivity, but scaling the service introduces operational challenges. The exact limitations depend on the architecture and implementation; they are not identical for every pseudowire or VPLS deployment.

MAC-address learning: Traditional bridged services often rely on data-plane learning, so the network must learn which customer-facing interface leads to each MAC address.
Flooding and replication: Unknown unicast, broadcast, and multicast traffic may need to be replicated to multiple service endpoints, increasing the importance of controlling unnecessary traffic.
Multihoming: Connecting a customer site to multiple provider edges for redundancy can introduce additional requirements for loop prevention, designated forwarding, and consistent forwarding behaviour.
Service growth: As the number of sites, endpoints, and services increases, provisioning and troubleshooting can become more complex.
Endpoint mobility: When a device moves between locations, the network must update forwarding information and avoid stale or conflicting reachability state.

The key distinction: connectivity is not the same as reachability distribution

A traditional Layer 2 VPN provides the service that carries Ethernet frames. A separate question is how the network learns where MAC addresses are located, distributes that information, and responds when endpoints move or links fail.

Why this led to EVPN

Ethernet VPN (EVPN) introduces a BGP-based control plane for advertising endpoint reachability. Rather than relying solely on data-plane MAC learning to populate forwarding information across the service, EVPN can distribute MAC and associated IP reachability information through Multiprotocol BGP (MP-BGP).

EVPN does not replace every part of the underlying transport. It can operate with MPLS or VXLAN, while the transport carries the encapsulated traffic. This separation between control-plane reachability and data-plane transport is central to understanding the architecture.

Engineering takeaway

Traditional Layer 2 VPNs established the connectivity model. EVPN builds on that model by providing a scalable control plane for endpoint reachability, with support for designs that require mobility, multihoming, and more structured control over forwarding behaviour.

Network Insight · EVPN Incident Lab

Incident Challenge: The Remote MAC Is Missing

A host behind PE1 cannot reach a host behind PE2. The local MAC is present, but the remote endpoint is missing from the expected forwarding state. Investigate one layer at a time and identify the likely fault.

Incident topology

Incident briefing

Symptom confirmed

Remote MAC missing on PE1

Host B is reachable locally from PE2, but PE1 has no expected remote MAC entry for Host B. The cause is not yet confirmed.

Service EVPN Layer 2 service · VNI 10100

Known observation Host B's MAC is learned on PE2.

Unknown Is the route absent, rejected by policy, or unable to reach the forwarding table?

Choose your next diagnostic check

Select a check to reveal its result. Start with control-plane evidence before assuming the underlay is broken. This scenario uses a simulated fault; the results are educational, not live device output.

Investigation not started

Choose a diagnostic check above. Each result narrows the fault domain. Use the evidence to distinguish route distribution from transport and forwarding problems.

Checks completed: 0 of 6

Troubleshooting principle: follow the evidence through local learning → EVPN route distribution and import → overlay/service mapping → underlay reachability → forwarding state. For VXLAN, the underlay must provide VTEP reachability and the network must permit the required encapsulated traffic. For MPLS EVPN, inspect the applicable label-switched transport instead. CLI commands and table names vary by platform.

Network Insight • EVPN Architecture

How Does EVPN Turn Endpoint Learning Into a Scalable Control Plane?

EVPN separates the question of where an endpoint is from the act of forwarding the packet. MP-BGP distributes endpoint reachability information, while an MPLS or VXLAN transport carries the resulting traffic across the network.

Transport Model

EVPN Service

Network Event

Use MP-BGP EVPN control-plane distribution Advertise MAC/IP endpoint correlation Enable active-active multihoming model

Control Plane → Transport → Forwarding Ready

EVPN
Control Plane

Endpoint MAC / IP

PE / VTEP PE1

MP-BGP EVPN NLRI

PE / VTEP PE2

Data Plane

Ingress Frame Arrives

Transport MPLS Core

Egress PE2 / VTEP

Destination Remote Endpoint

Control Plane MP-BGP EVPN

Endpoint State Distributed

Transport MPLS

Forwarding Model Control-Plane Guided

EVPN Trace

Waiting for topology trace...

Configure the EVPN model and run the trace.

Engineering Insight

EVPN uses BGP as a control plane to distribute endpoint reachability information. The transport beneath it can differ: EVPN can be deployed over MPLS or used with VXLAN in data-center fabrics. The control-plane concept remains distinct from the underlying packet-transport mechanism.

Illustrative architecture model. EVPN route types, multihoming behaviour, transport encapsulation and implementation details vary by deployment and platform.

Interactive Lab · Section 1

Traditional Layer 2 VPNs: How Does the Service Connect Sites?

Choose a traditional Layer 2 VPN model, then step through a simplified Ethernet forwarding example. Compare a point-to-point pseudowire with a multipoint VPLS service before exploring why EVPN introduced a different control-plane approach.

1 · Select a service model

2 · Explore the forwarding path

Point-to-point pseudowire

A Layer 2 service connects Site A directly to Site B through a provider network.

Frame ready

Service model Point-to-point

Forwarding scope One connected remote endpoint

Learning approach Depends on service design

Step 1 — The customer sends an Ethernet frame

Site A sends a frame toward a destination at Site B. The provider edge applies the configured Layer 2 service and determines how to carry the frame across the provider network.

Step 1 of 3

NETWORK INSIGHT · TRADITIONAL LAYER 2 VPNs · CONCEPTUAL FORWARDING MODEL

Section 2 · Control Plane

How the EVPN Control Plane Works

EVPN separates endpoint reachability distribution from the transport that carries Ethernet traffic. Using Multiprotocol BGP (MP-BGP), provider edge devices or data-centre VTEPs can advertise information about customer MAC addresses and, where applicable, associated IP addresses. Other participating devices use these advertisements to build the control-plane knowledge needed to make forwarding decisions.

From local learning to remote reachability

1. Learn locally A device learns a MAC address from a locally attached endpoint or Ethernet segment.

2. Advertise with BGP The EVPN control plane distributes the relevant reachability information in an EVPN route.

3. Build remote state Remote devices process the route and program appropriate forwarding information.

This does not mean all local learning disappears. Devices may still learn endpoints locally, while MP-BGP distributes reachability between participating network nodes. The control plane and the data plane perform related but distinct jobs.

What does an EVPN advertisement tell the network?

EVPN uses specific BGP Network Layer Reachability Information (NLRI) formats called EVPN route types. Each type communicates a different kind of reachability or service information. The receiving device interprets the route in the context of the relevant EVPN instance and its import policies.

Type 2

MAC/IP Advertisement

Advertises a MAC address and may also include an associated IP address. This is a key route type for communicating endpoint reachability between EVPN nodes.

Type 3

Inclusive Multicast Ethernet Tag

Identifies participating nodes for a broadcast domain or related EVPN service so that broadcast, unknown-unicast, and multicast traffic can be handled using the applicable replication model.

Type 1

Ethernet Auto-Discovery

Supports Ethernet-segment and multihoming procedures, including aliasing and mass withdrawal behaviours where applicable to the design.

Type 4

Ethernet Segment

Advertises Ethernet-segment information used by EVPN multihoming procedures, including discovery of the PEs connected to a shared Ethernet segment.

Type 5

IP Prefix

Advertises IP prefixes for EVPN integrated routing and bridging designs. It is especially relevant when EVPN is used to distribute Layer 3 reachability.

Route types are not interchangeable

Type 2 is central to MAC/IP endpoint reachability. Types 1 and 4 support multihoming procedures; Type 3 supports inclusive multicast membership signalling; and Type 5 carries IP-prefix reachability. Type 2 is not a replacement for every other EVPN route type.

Three identifiers and policies to understand

Several BGP attributes and EVPN service fields help keep routes unique and determine where they are imported. They serve different purposes and should not be treated as synonyms.

Element	Purpose	Engineering question
Route Distinguisher (RD)	Makes otherwise overlapping VPN route identities distinguishable in BGP.	How can similar prefixes or MAC-related routes remain unique across services?
Route Target (RT)	Controls which advertised VPN routes are eligible for import into a service or routing context.	Which EVPN participants should receive this route?
Ethernet Tag / service identifiers	Help identify the relevant Ethernet broadcast domain or service context within EVPN NLRI.	Which Ethernet service does this route refer to?

How does the route become usable for forwarding?

Receiving an EVPN route is not, by itself, the final forwarding action. The device checks route validity and import policy, determines the relevant remote endpoint or service, resolves the required transport reachability, and installs forwarding state as appropriate. The exact processing depends on the EVPN architecture and implementation.

In an EVPN-VXLAN fabric, for example, the remote VTEP must be reachable through the IP underlay before traffic can be encapsulated and sent to it. In an EVPN-MPLS network, the relevant MPLS transport and label forwarding must be available. MP-BGP distributes control-plane information; it does not replace the underlay or transport forwarding system.

Engineering takeaway

EVPN uses MP-BGP to distribute structured endpoint and service reachability. To troubleshoot it effectively, check the advertised route, the RD and RT policy, the EVPN service context, and the transport path separately. A valid BGP advertisement does not guarantee that the data plane can successfully forward the traffic.

Interactive Lab · Section 2

How Does EVPN Distribute MAC Reachability?

Follow a host MAC address from local learning to a BGP EVPN advertisement, then see how the remote PE or VTEP uses the imported route to build forwarding state.

1 · Choose the control-plane event

2 · Step through the route lifecycle

Step 1 — Learn the local MAC

The local PE learns a host MAC address from its attached Ethernet segment or bridge domain.

Local state

Route type Type 2 · MAC/IP Advertisement

Route state Locally learned

Next dependency Create and advertise EVPN route

Local learning happens first

PE-A learns the endpoint MAC from the local Ethernet service. EVPN can then advertise that reachability using a Type 2 route. If an IP binding is available and advertised, the route can carry that information too.

Step 1 of 4

Important: route visibility is not the same as forwarding readiness.

A Type 2 route must be accepted by the relevant policy, associated with the correct service, and usable for forwarding. RD values distinguish route identities; route targets govern import and export policy. The example values above are illustrative, not a required configuration.

NETWORK INSIGHT · EVPN CONTROL PLANE · MP-BGP / TYPE 2 MAC-IP ROUTES

EVPN Transport Experiment

EVPN over MPLS vs EVPN over VXLAN

Change the transport and service model to see what actually moves through the network. EVPN remains the BGP-based control plane; MPLS or VXLAN provides the underlying data-plane transport.

ILLUSTRATIVE MODEL

Transport / Data Plane

Service Model

Network Event

Multi-Homing Model

EVPN Forwarding Path Ready

Endpoint A MAC / IP

PE / VTEP 1 EVPN edge

IP UNDERLAY

VXLAN Tunnel VTEP → VTEP · VNI encapsulation

PE / VTEP 2 EVPN edge

Endpoint B MAC / IP

Control Plane MP-BGP EVPN

Data Plane VXLAN / UDP / IP

Service State MAC Reachability

Multi-Homing Single-Homed

Experiment Result

Transport VXLAN over IP

EVPN Function Endpoint Reachability

Encapsulation VXLAN VNI

Failure / Event Normal

Flooding Dependency Reduced by control-plane learning

EVPN Event Trace

12:00:00 Waiting for experiment...

12:00:01 MP-BGP EVPN control plane ready

Engineering insight: EVPN separates the control-plane job from the transport technology. MP-BGP advertises endpoint reachability, while VXLAN or MPLS carries the resulting traffic across the network.

Conceptual EVPN model. Exact route types, encapsulation, VTEP/PE behaviour, multi-homing mechanisms and vendor implementations vary.

Section 3 · Transport Architectures

EVPN over MPLS vs EVPN over VXLAN

EVPN is not tied to a single transport technology. The same broad BGP-based control-plane approach can support Ethernet services carried across an MPLS network or across an IP fabric using VXLAN encapsulation. The right choice depends on the network environment, existing infrastructure, operational requirements, and the services being delivered.

Separate the control plane from the data plane

MP-BGP EVPN distributes reachability and service information. MPLS or VXLAN carries the encapsulated traffic across the network. The underlay provides the connectivity needed to reach the relevant transport endpoints.

Provider WANs

EVPN over MPLS

Commonly used in service-provider and large routed networks where MPLS already provides a transport foundation for VPN services.

Uses MPLS labels to carry traffic across the provider network.
Can support multipoint Ethernet services and EVPN integrated routing and bridging designs.
Fits environments with established MPLS forwarding, traffic-engineering, and operational practices.
Requires the relevant MPLS transport and label-forwarding state to be working.

Data-Centre Fabrics

EVPN over VXLAN

Widely used in data-centre fabrics where an IP underlay connects Virtual Tunnel Endpoints (VTEPs) and VXLAN carries the Ethernet overlay.

Encapsulates Ethernet frames in VXLAN over UDP/IP.
Uses a 24-bit VXLAN Network Identifier (VNI) to identify the overlay segment.
Supports scalable Layer 2 segments across a routed IP fabric.
Requires IP reachability between VTEPs and correct overlay configuration.

The three functional layers

Control plane · MP-BGP EVPN Advertises endpoint reachability and EVPN service information

Transport · MPLS or VXLAN Carries traffic between the participating network nodes

Underlay connectivity Provides the paths needed to reach transport endpoints

This is a functional view rather than a strict packet-processing sequence. The control plane and forwarding system interact, but each has distinct responsibilities. A healthy EVPN BGP session alone does not prove that the transport path is usable.

Comparing the design considerations

Consideration	EVPN over MPLS	EVPN over VXLAN
Transport	MPLS label-switched transport	VXLAN encapsulation over IP
Typical environment	Provider networks and MPLS-based enterprise WANs	Data-centre and campus fabrics built on IP reachability
Service identifier	Depends on the EVPN service and MPLS label allocation	VNI identifies the VXLAN overlay segment
Key transport dependency	MPLS forwarding and required label reachability	IP reachability between VTEPs and correct VXLAN configuration
Operational focus	EVPN routes, labels, provider transport and service configuration	EVPN routes, VTEP reachability, VNI mapping and IP underlay

How to choose between them

If a provider already operates an MPLS network, EVPN over MPLS can extend established transport capabilities with EVPN's control-plane model. If a data centre is built around a routed IP fabric, EVPN over VXLAN can provide Ethernet overlay services without requiring MPLS in the underlay.

Neither option is universally superior. Consider scale, existing routing and transport infrastructure, multihoming requirements, operations tooling, hardware support, and the team's ability to troubleshoot both the control plane and the forwarding path.

Engineering takeaway

When troubleshooting EVPN, verify the layers independently: confirm that the EVPN route is present and imported, then check the transport and underlay needed to reach the remote endpoint. A control-plane success and a data-plane success are related, but they are not the same test.

Interactive Lab · Section 3

EVPN Transport Lab: MPLS or VXLAN?

The EVPN control plane can advertise endpoint reachability over different transport architectures. Switch the transport model and trace how a customer Ethernet frame crosses the network.

1 · Select the transport

2 · Inspect the forwarding sequence

Step 1 — EVPN provides endpoint reachability

MP-BGP EVPN distributes the remote MAC reachability needed by the service.

Route available

Control plane MP-BGP EVPN

Transport mechanism MPLS label switching

Underlay dependency Provider transport reachability

Step 1 — EVPN provides endpoint reachability

MP-BGP EVPN advertises the remote endpoint’s reachability. The receiving edge uses that information to select the service and forwarding next hop. EVPN is the control plane, not the transport itself.

Step 1 of 4

Functional stack

Control plane MP-BGP EVPN — advertises endpoint reachability and service information.

Service EVPN service — associates learned reachability with the appropriate Layer 2 or Layer 3 service.

Transport MPLS — carries traffic through a label-switched provider network.

Underlay Provider transport — supplies the reachability required by the chosen MPLS design.

Do not confuse the EVPN route with the transport encapsulation.

Both designs can use MP-BGP EVPN. MPLS carries traffic using labels; VXLAN encapsulates an inner Ethernet frame in VXLAN over UDP/IP between VTEPs. The underlay must support the selected transport, and the service and forwarding configuration must agree at both ends.

NETWORK INSIGHT · EVPN TRANSPORT LAB · MPLS / VXLAN / MP-BGP

Section 4 · Data-Centre Fabrics

BGP, VXLAN and Modern Data-Centre Fabrics

Modern data-centre networks often combine a routed IP underlay with an Ethernet overlay. In an EVPN-VXLAN fabric, the underlay provides IP connectivity between network devices, VXLAN carries overlay Ethernet frames, and MP-BGP EVPN distributes endpoint and service reachability. Keeping these roles separate makes the architecture easier to design, operate and troubleshoot.

How the fabric fits together

MP-BGP EVPN · Control plane Advertises endpoint reachability and EVPN service information between participating nodes

VXLAN overlay · Data plane Encapsulates Ethernet frames and identifies the overlay segment with a VNI

Routed IP underlay Provides reachability between the source and destination VTEPs

This is a functional model, not a strict sequence in which one layer operates only after another. BGP control-plane information, overlay forwarding state and underlay routing work together, but a fault in one layer can produce symptoms in another.

1. The underlay: a routed IP foundation

The underlay is the network of physical links and routed interfaces that connects the fabric devices. It is commonly built using Layer 3 links and an interior gateway protocol such as OSPF or IS-IS, or using BGP in designs that use BGP for underlay routing. The protocol choice depends on the architecture.

Reachability

Route between VTEPs

The underlay must provide a working IP path to the remote VTEP address. If that destination is unreachable, correctly advertised EVPN information may still fail to deliver user traffic.

Resilience

ECMP and alternate paths

Equal-Cost Multipath (ECMP) can allow traffic to use multiple eligible paths through the routed fabric. Its operation depends on the routing design, topology, hashing and device capabilities.

2. The overlay: VTEPs and VXLAN

A Virtual Tunnel Endpoint (VTEP) performs VXLAN encapsulation and decapsulation. When a host sends an Ethernet frame to a remote endpoint, the ingress VTEP maps the frame to the appropriate overlay service and encapsulates it in VXLAN over UDP/IP. The outer IP header identifies the destination VTEP.

The routed underlay forwards the encapsulated packet to that destination. The receiving VTEP removes the outer encapsulation and forwards the original Ethernet frame within the destination service.

What does the VNI do?

The 24-bit VXLAN Network Identifier distinguishes overlay segments. It is not an IP address, a BGP autonomous system number, or a replacement for an EVPN route. The VNI must be mapped consistently to the intended service at the relevant VTEPs.

3. MP-BGP EVPN: distributing endpoint reachability

EVPN allows participating nodes to advertise endpoint information through BGP. For example, an EVPN Type 2 route can advertise a host's MAC address and may include its IP address. Remote devices process the route according to their EVPN configuration and policy, then build the forwarding state needed for the service.

This provides a control-plane alternative to relying exclusively on data-plane MAC learning across the fabric. It can improve endpoint discovery and mobility handling, but local learning, broadcast traffic and other data-plane behaviours still depend on the design.

Following one packet across the fabric

Example · Host A sends to Host B

1. Host A Sends an Ethernet frame to its local gateway or destination MAC.

2. Ingress VTEP Uses forwarding state to select the overlay service and remote VTEP, then encapsulates the frame.

3. Egress VTEP Decapsulates the packet and forwards the original frame into the destination segment.

MP-BGP EVPN helps distribute the reachability used to build forwarding state. The actual packet is transported by VXLAN over the routed underlay; it is not carried inside the BGP update itself.

Endpoint mobility and flooding

When a host moves to another attachment point, the network needs to update its reachability information. EVPN advertisements can help remote nodes learn the new location, while implementation-specific mechanisms and policy determine how competing or stale information is handled.

EVPN can also reduce dependence on flood-and-learn behaviour by distributing endpoint information through the control plane. However, broadcast, unknown-unicast and multicast (BUM) traffic may still require replication according to the service and replication model. EVPN does not eliminate all flooding in every deployment.

What to check when traffic fails

Layer	Questions to investigate
EVPN control plane	Is the BGP EVPN session established? Is the expected route advertised, received and imported into the correct service?
Overlay configuration	Does the endpoint have the expected forwarding entry? Are the VNI, bridge-domain or service mappings correct?
VTEP reachability	Can the ingress VTEP reach the remote VTEP address using the expected source and routing context?
Underlay forwarding	Are the relevant routes, next hops, interfaces and ECMP paths healthy? Are MTU and encapsulation requirements satisfied?
Host and gateway	Are the endpoint's VLAN or access-port settings, gateway behaviour, MAC learning and IP configuration correct?

Engineering takeaway

An EVPN-VXLAN fabric has three distinct concerns: BGP distributes reachability, VXLAN carries overlay Ethernet frames, and the IP underlay transports the encapsulated packets. Troubleshoot each layer independently before concluding that the entire fabric or control plane is faulty.

Interactive Lab · Section 4

Inside a VXLAN EVPN Fabric

Inspect the control plane, overlay and underlay separately. Then introduce a fault and identify which layer to investigate first.

1 · Select the layer to inspect

Two endpoints, two VTEPs

Dashed purple links represent EVPN control-plane exchange.

Control plane VXLAN overlay IP underlay

Layer selected

Control plane

MP-BGP EVPN distributes endpoint reachability so a VTEP can learn how to reach remote MAC addresses.

Primary function Advertise endpoint reachability

What to verify BGP session, EVPN route and import policy

Failure clue Remote MAC route is absent

Engineering principle

A working BGP session alone does not prove that the required EVPN route has been advertised, imported and installed for forwarding.

2 · Fabric fault challenge

Choose a symptom and identify the most relevant first checks.

Start with the EVPN control plane

Check the MP-BGP EVPN session, confirm the remote Type 2 MAC/IP Advertisement route exists, and verify route-target import policy and forwarding installation.

Check 1 of 3

Keep the three layers separate during troubleshooting.

For a missing remote MAC, begin with EVPN route distribution and import. For an unreachable VTEP, inspect the routed underlay. For a VNI mismatch, compare service mappings at both ends. For large packets that fail, investigate end-to-end MTU and encapsulation overhead. These are starting points, not proof of a single root cause.

NETWORK INSIGHT · VXLAN EVPN FABRIC LAB · CONTROL PLANE / OVERLAY / UNDERLAY

Section 5 · High Availability

EVPN Multihoming and Resilience

Multihoming connects a customer device or Ethernet segment to more than one provider edge (PE) device. The goal is to maintain connectivity if a link or network device fails, and, where the design supports it, to use multiple attachment paths for forwarding. EVPN defines control-plane procedures that help participating PEs coordinate this behaviour without treating every redundant connection as an unrelated Ethernet service.

A multihomed Ethernet segment

Customer device or Ethernet segment Redundant attachment to the provider network

PE1 EVPN participant

PE2 EVPN participant

In EVPN, the shared attachment is represented by an Ethernet Segment (ES), identified by an Ethernet Segment Identifier (ESI). The participating PEs exchange EVPN information to support multihoming functions such as designated-forwarder election, aliasing where applicable, and withdrawal of reachability after failures.

Single-active and all-active multihoming

EVPN supports different multihoming modes. The appropriate mode depends on the service, the customer-facing technology, the platform capabilities and the desired traffic behaviour.

Single-active

One active forwarder

For a given Ethernet segment and service, one PE forwards traffic toward the multihomed segment at a time under the applicable forwarding procedures. Another PE can take over when the active path fails. This can simplify certain forwarding and loop-prevention requirements, but it does not mean that only one PE is involved in every direction or every aspect of the service.

All-active

Multiple active paths

Multiple PEs can forward traffic to and from the multihomed segment, subject to the service and platform design. EVPN procedures help remote devices use eligible paths and reduce unnecessary disruption when an attachment or PE fails.

All-active does not mean every packet is duplicated

All-active describes the availability of multiple forwarding paths. Actual traffic distribution depends on forwarding state, hashing, service configuration and platform behaviour. EVPN also uses split-horizon procedures to help prevent traffic from looping back toward the same multihomed Ethernet segment.

The role of the Designated Forwarder

For a given Ethernet segment and Ethernet tag or service context, EVPN elects a Designated Forwarder (DF) among the participating PEs. The DF is responsible for forwarding certain traffic, particularly relevant BUM traffic from the provider network toward the multihomed Ethernet segment, according to the applicable procedures.

DF election is not a universal rule that only one PE may forward all traffic. In all-active designs, multiple PEs may forward eligible known unicast traffic, while the DF role helps coordinate traffic such as broadcast, unknown unicast and multicast toward the segment.

How EVPN responds to failures

Failure or event	EVPN mechanism or design consideration	What to verify
Customer-facing link fails	Local detection and Ethernet Segment state can trigger relevant reachability updates.	Link state, ESI membership, local service state and route withdrawal.
A PE becomes unavailable	Other participating PEs may continue forwarding; mass-withdrawal procedures can help remote nodes remove affected reachability efficiently.	BGP session state, Ethernet Segment routes and remote forwarding entries.
Remote endpoint is reachable through multiple PEs	Aliasing may allow traffic to use eligible PEs advertising reachability for the same multihomed Ethernet segment.	Type 1 routes, ESI consistency, forwarding next hops and platform support.
BUM traffic reaches the segment	DF election and split-horizon procedures help coordinate forwarding and prevent loops.	DF state, service context, replication behaviour and duplicate traffic symptoms.

Why Ethernet Segment routes matter

EVPN route types provide the information needed for multihoming. Type 4 Ethernet Segment routes support discovery of PEs attached to the same Ethernet segment and contribute to DF election. Type 1 Ethernet Auto-Discovery routes support procedures including aliasing and mass withdrawal, depending on the route form and deployment.

These mechanisms work together with the Ethernet Segment Identifier and the service configuration. If the ESI is inconsistent, a route is missing, or the PEs disagree about the segment, expected redundancy behaviour may not occur.

Designing for predictable recovery

Redundancy should be tested rather than assumed. A resilient design considers link failure, PE failure, BGP session loss, forwarding convergence, traffic duplication, and the effect of failures on both unicast and BUM traffic. Recovery times depend on failure detection, routing and EVPN convergence, hardware programming, and the service configuration.

Engineering takeaway

EVPN multihoming combines Ethernet Segment identification, route exchange, DF election and forwarding procedures to coordinate redundant attachments. When investigating a failure, verify the ESI, Type 1 and Type 4 routes, DF state, BGP health and actual forwarding behaviour. The presence of two physical links alone does not guarantee resilient service.

Network Insight · Interactive EVPN Lab

EVPN Multihoming: Explore Resilience

Connect one Ethernet Segment to two Provider Edge routers. Change the multihoming mode, inspect the Designated Forwarder role, and simulate a PE failure to see how forwarding responsibilities can change.

Ethernet Segment · Dual-homed customer

Available path Ethernet Segment Failed path

What is happening?

Both PEs available

All-active multihoming

Both PE1 and PE2 can forward eligible traffic for the Ethernet Segment. EVPN procedures help coordinate reachability and prevent loops; the exact forwarding behaviour depends on the service and implementation.

Forwarding model Both PEs may forward eligible known-unicast traffic.

Designated Forwarder The DF role coordinates forwarding of relevant BUM traffic toward the Ethernet Segment.

EVPN control-plane clues Type 1 Ethernet A-D and Type 4 Ethernet Segment routes support multihoming procedures.

Engineering observation

All-active is not the same as sending every frame twice. DF election coordinates relevant BUM forwarding toward the segment, while known-unicast forwarding and load-sharing depend on the EVPN design.

Learning model: this is a conceptual illustration, not a live protocol simulator. It does not calculate DF election from a real ESI, Ethernet Tag, preference, or vendor-specific configuration. Actual convergence and forwarding behaviour depend on the implementation and service design.

Network Insight · EVPN Operations

Troubleshooting EVPN: Trace the Failure by Layer

When a remote endpoint becomes unreachable, the cause may sit in the EVPN control plane, the overlay service, the transport underlay or the multihoming design. A structured investigation helps isolate the fault instead of treating every symptom as a BGP problem.

Start with the observed failure, identify which forwarding or control-plane state is missing, and then test the relevant dependency. Avoid changing several layers at once: each check should narrow the fault domain.

First principle: establish the scope. Determine whether the problem affects one endpoint, one bridge domain or VNI, one PE/VTEP, one direction, or the entire EVPN service. The scope is often the first clue to the failing component.

A step-by-step investigation

Follow the path from the local attachment through route distribution to remote forwarding.

Confirm the local endpoint and service mapping

Check the access interface, VLAN, bridge domain and local MAC table. Verify that the endpoint is learned on the expected attachment and that the local service is mapped to the intended EVPN instance or VNI. If local learning is wrong, remote route checks may be premature.

Verify the EVPN control-plane session

Confirm that the relevant BGP session is established and that the EVPN address family is enabled and exchanging routes. A session can be established while the required address family, neighbor policy or route exchange is still incorrect.

Trace the advertised and imported EVPN route

For a missing remote MAC, inspect the Type 2 MAC/IP Advertisement route. Check the advertised MAC and optional IP binding, route next hop, route target import/export policy and relevant route distinguisher. Confirm the route is both received and eligible for installation—not merely present on a peer.

Check the overlay service and forwarding state

Verify that the remote MAC is installed in the expected bridge or forwarding table, and that the correct VNI or service identifier is associated with it. For VXLAN, confirm the remote VTEP and encapsulation parameters. For MPLS, confirm the expected service and transport labels are programmed.

Validate the transport underlay

For VXLAN, test IP reachability between VTEPs, inspect routing and ECMP, and check MTU and any ACL or firewall rules affecting VXLAN traffic (commonly UDP port 4789). For MPLS, verify the label-switched transport path and reachability to the relevant provider edge.

Investigate multihoming when the service uses an Ethernet Segment

Check the Ethernet Segment Identifier (ESI), Ethernet Auto-Discovery routes (Type 1), Ethernet Segment routes (Type 4), Designated Forwarder election where applicable, and the expected aliasing or mass-withdrawal behavior. After a failure, confirm that stale paths have been withdrawn and forwarding has converged.

Separate known-unicast failures from BUM behavior

Test a destination with a known remote MAC separately from broadcast, unknown-unicast and multicast traffic. If known unicast works but BUM traffic fails, focus on replication, multicast or ingress-replication configuration and the relevant Type 3 routes—not only Type 2 endpoint routes.

Symptom-to-check reference

Use the observed behavior to prioritize the next verification step.

Observed symptom	First checks	Likely fault area
Remote MAC is missing	Local MAC learning, Type 2 route advertisement and receipt, RT policy, route installation	Endpoint learning or EVPN control plane
Remote MAC exists, but traffic fails	Forwarding entry, service/VNI mapping, remote next hop, VTEP or label reachability	Overlay forwarding or transport
One direction works only	Reverse route, return-path forwarding, asymmetric policy, MTU and packet drops	Return path or directional configuration
Known unicast works; BUM fails	Type 3 routes, replication mode, multicast state and service membership	BUM replication and distribution
Traffic fails after a link or PE failure	ESI state, Type 1/Type 4 routes, DF state where applicable, withdrawal and convergence	Multihoming or failure recovery
Intermittent loss or unexpected flooding	MAC movement, duplicate endpoint learning, route churn, service mapping and replication	Endpoint mobility or inconsistent state

Keep the fault domains separate

EVPN troubleshooting becomes clearer when each layer is checked for the state it owns.

Layer 01 Control plane

BGP session, EVPN address family, route advertisement, route import policy and best-path selection.

Layer 02 Overlay service

Bridge domain, VNI or service identifier, MAC/IP state, replication and forwarding entries.

Layer 03 Transport underlay

IP reachability and MTU for VXLAN, or label-switched reachability for MPLS transport.

Layer 04 Multihoming

Ethernet Segment state, DF election, path withdrawal and convergence after failure.

Do not assume a healthy BGP session means the service is healthy.

Control-plane reachability is only one dependency. A route may be absent, rejected by policy, not installed in forwarding, mapped to the wrong service, or point to an unreachable transport next hop.

Engineering takeaway

Trace the failure from the endpoint outward: local learning → EVPN route exchange → remote forwarding state → overlay mapping → transport reachability. If multihoming is involved, verify the Ethernet Segment state and recovery behavior as a separate step. Use vendor-specific commands to validate each state, then change only the layer supported by the evidence.

EVPN Incident Challenge

EVPN Incident: The Remote MAC Is Missing

A workload is reachable locally but cannot be reached from the remote EVPN edge. Use the evidence to identify where the control-plane learning process has failed.

DIAGNOSE THE FAILURE

Incident Evidence

Alert: Endpoint B is active on PE/VTEP 2, but PE/VTEP 1 has no usable remote MAC entry.

Endpoint B local attachment UP

Local MAC learning PRESENT

MP-BGP EVPN session DOWN

VNI / service mapping MATCH

IP underlay reachability UP

Remote MAC state MISSING

Control Plane → Data Plane Investigation ready

Endpoint B Local MAC

PE / VTEP 2 MAC learned

MP-BGP EVPN Session DOWN

PE / VTEP 1 MAC missing

What Should You Investigate First?

ENGINEERING SCORE

0 / 4

Investigation Trace

12:00:00 Incident loaded

12:00:01 Waiting for engineer diagnosis...

Engineer hint: EVPN endpoint reachability is distributed through the control plane. If the local PE/VTEP has learned the MAC but the remote device has not, investigate the EVPN control-plane exchange before changing the transport.

Conceptual troubleshooting scenario. Real EVPN deployments may involve additional checks such as route targets, VRF/VLAN/VNI mapping, Ethernet Segments, route-type state, next-hop reachability and vendor-specific control-plane behaviour.

☕

Support Network Insight

Help support the development of interactive networking labs, BGP simulations, and educational content.

Support the Project

Data Center Topologies

August 14, 2014

by Matt Conran Blog

Network Architecture / Data Center Design

Data Center Topology: From Three-Tier Networks to Leaf-Spine Fabrics

Data center topology defines how switches, servers, storage systems and external networks connect—and ultimately how application traffic moves through the infrastructure. The architecture you choose influences latency, available bandwidth, resilience, operational complexity and the ability to scale as workloads grow.

Traditional hierarchical networks remain relevant in many environments, but modern cloud platforms and distributed applications increasingly rely on scalable Layer 3 fabrics, leaf-spine architectures and equal-cost multipath routing. Understanding these designs means looking beyond the network diagram to examine routing behaviour, traffic patterns, failure domains and the control plane that keeps the fabric connected.

Design principle: build around application requirements

A good topology is not simply the one with the most links or the most switches. It is the one that delivers the required capacity, predictable forwarding, appropriate redundancy and manageable failure behaviour at an acceptable cost.

01 / STRUCTURE

Topology and Fabric Design

Compare three-tier, centralized, top-of-rack, end-of-row and leaf-spine architectures.

02 / ROUTING

BGP and ECMP

Explore routing adjacencies, route selection, multipath forwarding and convergence during failures.

03 / OVERLAYS

VXLAN and EVPN

Understand how Layer 3 underlays support virtual networks and distributed endpoint reachability.

04 / OPERATIONS

Capacity and Resilience

Evaluate oversubscription, traffic distribution, observability and failure-domain behaviour.

In this guide: Work from fundamental topology choices to modern data center fabrics, BGP-based routing, traffic engineering considerations and practical troubleshooting.

NETWORK INSIGHT · FIELD GUIDE

Explore Data Center Topology

Navigate the architecture, routing, traffic flows and resilience principles behind modern data center networks.

06 sections

01 / ARCHITECTURE FUNDAMENTALS

Understanding Data Center Architectures

Data center topology defines how servers, storage systems, switches and external networks connect. The architecture determines how traffic moves through the infrastructure and influences available bandwidth, latency, redundancy, scalability and operational complexity.

This section introduces traditional three-tier networks, centralized switching, top-of-rack (ToR), end-of-row (EoR) and leaf-spine architectures. Compare their connectivity models and trade-offs to understand which designs best match different workload and infrastructure requirements.

Engineering focus Choose topology according to traffic patterns, growth expectations, capacity, failure domains and the cost of operating the network.

Section 01 / Architecture Fundamentals

The Main Data Center Architectures

Data center networks can be built using hierarchical switching layers, centralized switching platforms, distributed access switches or fabric-based designs. Each approach creates a different balance between cabling, port utilization, bandwidth, failure isolation and operational complexity.

The right starting point is the workload. A smaller environment with limited server-to-server traffic may not need the same architecture as a cloud platform supporting thousands of virtual machines, containers or data-intensive applications. Topology selection should reflect traffic patterns, growth expectations, availability targets and the cost of operating the network over time.

Common architectural approaches

Hierarchical

Traditional Three-Tier Architecture

A traditional enterprise design separates access, aggregation or distribution, and core functions. Access switches connect servers, the aggregation layer concentrates traffic and may enforce policy, and the core provides high-speed connectivity between network blocks.

This model remains useful where existing infrastructure, established operational practices or clear policy boundaries make a hierarchical design appropriate. However, multiple switching tiers can increase path length and create bottlenecks when east-west traffic grows.

Centralized

Centralized Switching

In a centralized design, servers or server racks connect back to a central switching location. Concentrating switching resources can simplify management and make port allocation efficient in smaller environments.

As the facility grows, cable lengths, pathway congestion, centralized capacity limits and the consequences of equipment failure must be evaluated carefully.

Distributed Access

Top-of-Rack and End-of-Row

Top-of-rack (ToR) places access switches in individual server racks. End-of-row (EoR) places switching equipment in a dedicated rack serving a row of servers. Both approaches can provide redundant connectivity when designed with suitable independent paths and devices.

ToR generally shortens server cabling and can isolate access failures to a rack. EoR can consolidate switching resources, but may require more structured cabling. Neither approach automatically determines whether the wider network uses a three-tier or leaf-spine architecture.

Fabric-Based

Leaf-Spine Architecture

A leaf-spine fabric connects each leaf switch to multiple spine switches. Servers and other endpoints connect to the leaf layer, while the spine layer provides paths between different leaves. A Layer 3 underlay commonly uses equal-cost multipath routing to distribute eligible traffic across available paths.

This architecture provides predictable path structure and supports incremental expansion. Its capacity and resilience depend on link speeds, the number of spine switches, oversubscription, routing design and failure behaviour.

Important distinction: ToR and EoR describe where access switching is placed. Three-tier and leaf-spine describe broader network architectures. A data center can use ToR switches as its leaf layer in a leaf-spine fabric.

Compare the design trade-offs

Design consideration	Three-tier / centralized	Leaf-spine fabric
Traffic paths	May traverse several hierarchical layers.	Typically uses a consistent leaf-to-spine-to-leaf path.
Scaling	Expansion depends on available tier capacity and design.	Can expand by adding leafs and, when needed, spine capacity.
Multipath routing	Depends on the routing and switching design.	Layer 3 ECMP is commonly used between leafs and spines.
Operational considerations	May suit established environments and existing investments.	Requires deliberate underlay, routing, capacity and failure planning.

Before selecting a topology

Traffic patterns

Determine whether workloads primarily communicate with external networks or generate substantial east-west traffic between servers.

Capacity and growth

Estimate endpoint growth, uplink bandwidth, oversubscription limits and the cost of adding future capacity.

Failure domains

Identify the impact of losing a switch, link, rack or entire network block, including the capacity that remains.

Operations and cost

Balance equipment and cabling costs against monitoring, troubleshooting, automation and lifecycle requirements.

NETWORK INSIGHT · INTERACTIVE ARCHITECTURE LAB

Explore Data Center Architectures

Compare four common ways to connect compute, access and switching. Select an architecture, then click a device in the topology to inspect its role in the network.

Three-tier architecture Illustrative topology

Selectable device Network connection Highlighted path

Selected device

Core switch

The core provides high-speed connectivity between distribution blocks and upstream networks in a traditional three-tier design.

Architecture insight

Separate layers make roles easy to understand, but traffic may traverse multiple tiers and scaling can increase operational complexity.

Layered Design structure

Multi-hop Typical path shape

Layer by layer Common scaling approach

Design strengths

Clear separation of network roles
Familiar operational model
Can suit hierarchical traffic patterns

Design considerations

Potentially longer paths between servers
Capacity planning across tiers
Additional layers to operate and troubleshoot

These diagrams illustrate common design patterns, not vendor-specific configurations. Actual forwarding paths, redundancy and oversubscription depend on the implementation. ToR means Top of Rack; EoR means End of Row; Clos describes a multistage network fabric.

02 / FABRIC ARCHITECTURE

Leaf-Spine and Clos Fabrics

Leaf-spine architecture connects each leaf switch to multiple spine switches, creating a consistent path structure between different server racks. It is widely used in modern data centers where distributed applications, virtualization and cloud workloads generate substantial east-west traffic.

Explore how leaf and spine layers work together, how two-tier fabrics relate to Clos network designs, and how adding switches can expand connectivity. The section also examines oversubscription, available path diversity and the limits of relying on topology alone to guarantee performance.

Engineering focus Understand fabric stages, leaf-to-spine connectivity, expansion options and the capacity that remains when a device or link fails.

Section 02 / Fabric Architecture

Leaf-Spine and Clos Fabrics

Leaf-spine architecture is a common foundation for modern data center networks. Instead of sending traffic through a succession of hierarchical switching layers, it connects access-layer leaf switches to a set of spine switches. This creates a fabric with consistent paths between leaf switches and provides a repeatable way to expand connectivity as workloads grow.

The architecture is particularly useful for environments with substantial east-west traffic, including virtualized platforms, container infrastructure, distributed storage and cloud-native applications. Its performance depends on the actual link speeds, number of paths, traffic distribution and capacity provisioned throughout the fabric.

The physical fabric

Spine layer

Spine 1

Spine 2

Spine 3

Every leaf connects to every spine

Leaf layer

Leaf 1

Leaf 2

Leaf 3

Leaf 4

Server and workload connectivity

Servers

Storage

Compute

Services

Conceptual topology, not a physical cabling diagram. Each leaf has links to each spine; server-facing connections are shown separately. The simplified graphic illustrates connectivity rather than every individual link.

1. The role of leaf switches

Leaf switches provide connectivity for servers, storage, firewalls, load balancers and other data center endpoints. They may also perform Layer 3 routing, enforce network policy and act as VXLAN tunnel endpoints when an overlay is deployed.

In a Layer 3 fabric, endpoints in different subnets can communicate through routed paths. When VXLAN EVPN is used, the leaf layer can additionally provide overlay connectivity and distribute endpoint reachability through the EVPN control plane.

2. The role of spine switches

Spine switches interconnect the leaf layer and provide transit paths between leaf switches. In a conventional two-tier leaf-spine fabric, spine switches do not connect directly to servers, and leaf switches do not connect directly to other leaf switches.

Adding spine capacity can increase the number of available paths and aggregate fabric bandwidth, provided the leaf uplinks, interfaces, routing configuration and downstream capacity are designed to use it. A spine failure can reduce available capacity even when connectivity survives.

Why path predictability matters: Traffic between endpoints attached to different leaf switches normally traverses one spine switch. This creates a consistent switching-hop structure, although actual latency still depends on link speeds, device processing, congestion and the forwarding path.

How the architecture scales

01 / ADD LEAF CAPACITY

Expand endpoint connectivity

Add leaf switches with uplinks to the fabric's spine layer. Confirm that the existing spine and uplink capacity can support the additional traffic.

02 / ADD SPINE CAPACITY

Increase fabric bandwidth

Additional spine switches can create more equal-cost paths. Validate routing limits, optics, cabling, interface capacity and platform support before expanding.

03 / MANAGE OVERSUBSCRIPTION

Plan for real traffic demand

Compare aggregate server-facing bandwidth with available uplink bandwidth. Size the fabric according to workload behaviour and acceptable contention.

04 / CONTROL FAILURE IMPACT

Test the remaining capacity

Evaluate whether surviving spines and links can carry traffic after a failure, not simply whether alternate paths exist.

Two-tier and multi-tier Clos designs

Leaf-spine is a practical implementation of a two-tier Clos-style fabric. Larger environments may introduce a super-spine or additional fabric stages to interconnect multiple pods, rooms or data halls. These designs extend the architecture while requiring careful consideration of path diversity, capacity and failure boundaries.

Design	Typical purpose	Key consideration
Two-tier leaf-spine	Connect endpoints across a single fabric.	Spine capacity, leaf uplinks and ECMP behaviour.
Multi-pod fabric	Connect modular groups of racks or infrastructure.	Inter-pod connectivity and failure isolation.
Super-spine / multi-stage Clos	Scale connectivity across larger fabrics or multiple data halls.	Additional stages, path design and end-to-end capacity.

NETWORK INSIGHT · FABRIC EXPLORER

Explore Leaf-Spine and Clos Fabrics

Investigate how leaf and spine switches connect, trace an illustrative server-to-server path, simulate a spine-link failure, and see why uplink capacity and oversubscription matter.

Explore:

Two-tier leaf-spine fabric All illustrated links up

Network device Available link Illustrative selected path Failed link

Topology focus

Leaf switches

Leaf switches connect servers and provide uplinks into the spine layer. In a conventional two-tier fabric, each leaf connects to every spine.

Fabric principle

A full-mesh leaf-to-spine design provides multiple potential paths between different leaves. Routing and ECMP determine which next hops are installed and used.

Trace a flow

Source leaf

Destination leaf

2 Illustrated spine paths between selected leaves

8 Leaf-to-spine links currently available

Healthy Illustrative fabric state

Explore leaf uplink oversubscription 3:1 example

1:1 · equal nominal capacity 5:1 · more aggregate server capacity

A 3:1 illustrative oversubscription ratio means aggregate nominal server-facing bandwidth is three times the uplink bandwidth. Actual throughput depends on traffic patterns, link rates and congestion.

Why use a Clos-style fabric?

A multistage design can provide scalable connectivity and multiple paths without requiring every leaf to connect directly to every other leaf.

What should engineers verify?

Check spine and leaf capacity, uplink speed, ECMP behavior, failure convergence and the bandwidth remaining after a link or device failure.

This is an educational model, not a live network or packet-level simulator. The four-leaf, two-spine diagram is illustrative. ECMP distributes eligible flows across available next hops; it does not guarantee that each physical link carries exactly the same load.

03 / TRAFFIC ENGINEERING

Data Center Traffic Patterns

A data center network must accommodate the traffic generated by applications, storage systems, virtual machines and distributed services. Understanding where traffic originates and where it travels is essential for designing a fabric with sufficient capacity and predictable performance.

This section distinguishes east-west traffic between internal workloads from north-south traffic entering or leaving the data center. It explores how flow size, concurrent demand, ECMP hashing, uplink capacity and congestion influence the actual behaviour of a leaf-spine network.

Engineering focus Size the fabric around real workload demand, identify potential bottlenecks and validate performance during normal operation and failure conditions.

Section 03 / Traffic Engineering

Data Center Traffic Patterns

A data center topology must be designed around the traffic it carries, not simply the number of switches and links it contains. Application placement, storage access, replication, service dependencies and external connectivity all influence where traffic flows and where congestion can occur.

Leaf-spine fabrics are particularly suited to distributed east-west communication, but north-south traffic, large data transfers and uneven workload demand can still create bottlenecks. Understanding these patterns helps engineers size uplinks, evaluate redundancy and choose appropriate routing and forwarding designs.

Two fundamental traffic directions

01 / EAST-WEST

Workload-to-workload traffic

Traffic between servers, virtual machines, containers, storage systems and distributed application components. In modern environments, this can represent a substantial share of total network demand.

02 / NORTH-SOUTH

Traffic entering or leaving the fabric

Traffic between internal workloads and external networks, users, WANs, internet services or other data centers. Its path may include border leaf switches, firewalls, routers and load balancers.

Example / east-west traffic path

Communication between two server racks

Server A
Rack 1

Leaf 1

Spine

Leaf 2

Server B
Rack 2

Simplified logical path. In a conventional two-tier leaf-spine fabric, inter-leaf traffic normally traverses a spine. The forwarding plane selects an available path according to the installed routes and forwarding configuration.

Why east-west traffic affects fabric design

Distributed applications frequently exchange data across multiple servers. Examples include database replication, distributed storage, service-to-service requests, cluster coordination and virtual machine communication. If these flows cross leaf switches, the spine layer and leaf uplinks must provide enough capacity for their combined demand.

Traffic demand is rarely perfectly uniform. Some flows are short and latency-sensitive, while others transfer large volumes of data. Multiple simultaneous flows may also compete for the same links. Consequently, aggregate bandwidth alone does not reveal every congestion risk.

Important distinction: A fabric can have several equal-cost paths and still experience congestion. ECMP provides a mechanism for distributing eligible flows across paths; it does not guarantee that every flow uses a different link or that traffic is perfectly balanced.

North-south traffic and border connectivity

External traffic commonly enters or leaves through designated border leaf switches or dedicated edge devices. These connect the fabric to WAN routers, internet edge infrastructure, firewalls, load balancers and other networks.

The design must consider both bandwidth and resilience. Concentrating external connectivity on a small number of devices may simplify policy enforcement, but it can also create capacity constraints or larger failure domains. Distributing border functions can improve scale, provided routing policy and security controls remain consistent.

Traffic patterns and engineering decisions

Traffic pattern	Potential pressure point	Design response
Server-to-server	Leaf uplinks and spine capacity.	Assess aggregate demand, path diversity and ECMP.
Storage replication	Sustained high-throughput flows.	Measure peak demand and validate link utilization.
Internet or WAN access	Border links, security appliances and external routing.	Plan edge capacity, policy and redundant paths.
Large data transfers	Flow imbalance and transient congestion.	Evaluate hashing behaviour, flow sizes and telemetry.
Failure or maintenance	Reduced capacity on surviving paths.	Validate traffic behaviour after losing a link or device.

Measure the traffic, not just the topology

Traffic engineering should use interface counters, utilization trends, flow telemetry, packet drops, latency measurements and application performance data. These observations help distinguish a routing issue from a capacity problem or an application-generated traffic burst.

Engineers should also examine behaviour during failure conditions. A fabric that operates comfortably during normal traffic may become congested when a spine or uplink is unavailable and flows are redistributed across fewer remaining paths.

NETWORK INSIGHT · TRAFFIC EXPLORER

Explore Data Center Traffic Patterns

Follow traffic between servers, across the fabric and toward external networks. Adjust the illustrative load to understand why workload patterns, path capacity and congestion matter when designing a data center.

East-west: server to server Illustrative traffic flow

Network link East-west flow North-south flow Potential congestion

Traffic focus

East-west traffic

Traffic between workloads inside the data center. In a leaf-spine fabric, communication between servers on different leaves normally traverses a spine.

Engineering insight

Distributed applications, storage replication and service-to-service calls can generate substantial east-west traffic. Design for workload demand rather than assuming all traffic exits the data center.

Example flow

East-West Selected traffic direction

Leaf → Spine → Leaf Illustrative forwarding path

Moderate Illustrative link demand

Simulate traffic demand 45% illustrative load

Low demand High demand

At this illustrative load, the selected links have moderate demand. Real congestion depends on offered traffic, available link capacity, buffering, queueing and the distribution of flows.

Why traffic direction matters

East-west traffic can stay inside the fabric and use leaf uplinks and spine capacity. North-south traffic crosses the boundary toward external networks or services.

What engineers should measure

Compare interface utilization, drops, queue depth, flow records and application latency. A high average utilization can hide short bursts that cause packet loss or delay.

This widget is a conceptual teaching model, not a live traffic generator or telemetry system. The load slider represents illustrative demand, not measured utilization. Actual paths and capacity depend on topology, routing, link rates, hashing and workload behavior.

04 / ROUTING CONTROL PLANE

BGP in Data Center Fabrics

Border Gateway Protocol (BGP) provides a scalable way to distribute network reachability information across modern data center fabrics. In a routed leaf-spine design, BGP can establish the routes that allow traffic to move between endpoints while supporting controlled route propagation and multipath forwarding.

Explore eBGP and iBGP design approaches, route advertisements, best-path selection, ECMP and convergence. The section also introduces the relationship between a routed underlay and a VXLAN EVPN overlay, showing why control-plane state must be checked alongside the actual forwarding table.

Engineering focus Trace route propagation, validate next-hop reachability, understand multipath installation and troubleshoot routing changes without confusing BGP state with end-to-end connectivity.

Section 04 / Routing Control Plane

BGP in Data Center Fabrics

Border Gateway Protocol (BGP) is widely used to exchange reachability information in modern data center networks. In a routed leaf-spine fabric, it can provide scalable route distribution, controlled reachability, multipath forwarding and predictable responses to topology changes.

BGP does not replace the physical fabric. It supplies control-plane information that routers use to decide which network destinations are reachable and which paths can be installed for forwarding. The resulting behaviour depends on the routing design, policy, platform capabilities and operational configuration.

Keep the layers distinct: the physical topology defines available links; the routing control plane distributes reachability; the forwarding plane moves packets using installed forwarding entries. A healthy BGP session alone does not prove that the intended traffic path is working.

01 / Choosing a BGP fabric design

DESIGN A

eBGP leaf-spine

Leaf and spine switches use different autonomous system numbers (ASNs), and BGP sessions between them operate as external BGP. This is a common design for routed Clos fabrics because it can keep routing relationships explicit and support repeatable deployment patterns.

DESIGN B

iBGP-based fabric

Routers exchange BGP routes within the same autonomous system. Larger designs may use route reflectors to distribute routes without requiring a full mesh of iBGP sessions between every participating router.

DESIGN C

Hybrid routing

A fabric may combine an IGP or static routing for underlay reachability with BGP for other routing functions. The exact division depends on the design; there is no requirement that every data center use one universal routing model.

DESIGN D

Border and external routing

Border leaf switches or edge routers exchange routes with external networks. Import and export policies determine which prefixes can enter or leave the fabric and how they are represented to other routing domains.

Example: eBGP in a leaf-spine fabric

In this simplified example, each leaf has an eBGP session to each spine. Leaf-to-spine routing sessions exchange the reachability needed to forward traffic across the fabric. The example illustrates the relationship rather than prescribing a specific ASN plan or configuration.

Illustrative two-tier routed fabric

SPINE LAYER

Spine 1 AS 65101

Spine 2 AS 65102

eBGP sessions between each leaf and both spines

LEAF LAYER

Leaf 1 AS 65001

Leaf 2 AS 65002

Conceptual relationship only; individual physical links and separate BGP sessions are not drawn. ASNs are illustrative. This design assumes the chosen BGP configuration permits the required route exchange between the relevant devices.

Design caveat: eBGP does not automatically guarantee a working fabric or multipath forwarding. ASN allocation, next-hop reachability, route policy, prefix limits, supported BGP features and ECMP configuration must all be validated.

02 / Underlay and overlay routing

The BGP underlay

The underlay is the routed IP infrastructure that connects the switches and provides IP reachability across the fabric. In a BGP-routed underlay, BGP can advertise the infrastructure prefixes needed to reach loopbacks and other routed destinations. These routes provide the foundation for control-plane sessions and overlay tunnel endpoints where applicable.

The VXLAN EVPN overlay

In a VXLAN EVPN design, VXLAN encapsulates tenant traffic across the IP fabric, while EVPN uses BGP to distribute overlay reachability information. This can include information about MAC addresses, IP addresses and tenant segments, depending on the EVPN route type and deployment.

Component	Primary responsibility	Example question
Underlay routing	IP reachability between routed fabric endpoints.	Can this VTEP reach the remote VTEP address?
BGP EVPN control plane	Advertises overlay reachability and related information.	Has the expected MAC/IP or prefix route been learned?
VXLAN data plane	Encapsulates tenant packets across the IP underlay.	Can the encapsulated packet traverse the fabric?
Forwarding and policy	Applies installed forwarding entries and applicable policies.	Does traffic reach the correct destination and tenant?

These functions are related but not interchangeable. A healthy underlay does not prove that EVPN routes are correct, and receiving an EVPN route does not by itself prove that end-to-end forwarding succeeds.

03 / How BGP selects routes

From route advertisement to forwarding

Receive: a BGP neighbor advertises a prefix and its associated path attributes. The receiving router evaluates the update against session state, import policy and protocol rules.

Validate: the route must be acceptable under policy and protocol requirements. Its next hop must also be resolvable for the route to be usable for forwarding.

Select: the router evaluates eligible paths using its BGP decision process and implementation-specific policy. Attributes such as local preference, AS path, origin and MED can influence the outcome; their exact effect depends on the decision context and configuration.

Install: the selected route or eligible set of multipaths may be installed into the routing table and forwarding hardware, subject to platform limits, configuration and interactions with other routing sources.

Forward: packets are sent using the installed forwarding entries. The actual path depends on the forwarding implementation, ECMP eligibility, hashing and current topology.

Best path is not the same as every installed path. BGP implementations can select a best path while also installing multiple eligible paths for multipath forwarding. Multipath rules and supported attributes vary by platform and configuration.

Why BGP policy matters

BGP exchanges reachability, but policy determines which routes are accepted, preferred, modified or advertised onward. In a data center, an incorrect policy can suppress a required prefix, permit an unintended route or cause traffic to prefer an unexpected exit.

Engineers should explicitly document intended import and export behaviour, allowed prefixes, ASN expectations, next-hop handling and any route attributes used to influence path selection. Reusable policy templates can reduce configuration drift across large fabrics.

04 / ECMP and BGP multipath

How equal-cost paths support fabric capacity

Equal-Cost Multipath (ECMP) allows a router to forward traffic across multiple eligible next hops for a destination. BGP can supply candidate paths, but the routing implementation and configuration determine whether multiple paths are considered equivalent and installed.

Once multiple next hops are installed, forwarding hardware commonly uses a hash of packet or flow fields to select a path. This helps distribute traffic while typically preserving packet ordering within a flow. It does not mean every individual packet is sent down a different path, nor does it guarantee perfectly even utilization.

Mechanism	Role	Common misconception
BGP best-path selection	Chooses a preferred route according to policy and decision rules.	It does not necessarily install every received path.
BGP multipath	Allows multiple eligible BGP paths to be installed.	It is not automatic on every platform or configuration.
ECMP forwarding	Distributes eligible flows across equal-cost next hops.	It does not guarantee perfect load balance.
LACP	Negotiates and manages a Layer 2 link aggregation group.	It is not the same mechanism as Layer 3 ECMP.

ECMP and LACP can both use hashing in forwarding or distribution decisions, but they operate at different levels. ECMP selects among Layer 3 next hops; LACP negotiates membership in a Layer 2 aggregated link. A design may use one or both, depending on its physical and logical architecture.

05 / Convergence and failure handling

What happens when a path fails?

When a link or device fails, the fabric must detect the event, update the relevant routing state and adjust forwarding. BGP may withdraw affected routes or advertise revised reachability. The forwarding plane then changes its installed entries as the control plane and hardware respond.

Convergence time depends on failure detection, session behaviour, routing policy, protocol timers, implementation, hardware programming and the scope of the change. BFD may help detect some failures faster when supported and correctly configured, but it is not a universal guarantee of a particular recovery time.

FAILURE 01

Single link loss

If alternate paths remain available and the routing design recognizes the failure, traffic may be redirected through surviving paths. Check both route changes and actual forwarding.

FAILURE 02

Spine or peer loss

Routes or next hops associated with the failed device may be removed. Surviving paths carry the remaining demand, potentially increasing utilization and exposing oversubscription.

FAILURE 03

Policy or route withdrawal

A prefix may disappear even though physical links and BGP sessions remain up. Investigate advertisements, filters, next-hop resolution and the routing table.

FAILURE 04

Session instability

Repeated BGP resets can cause route churn and traffic disruption. Check interface errors, reachability, timers, logs and control-plane resource conditions.

Resilience must be tested under load. Redundant links are useful only if surviving paths have sufficient capacity and forwarding converges as intended. Test representative link and device failures, and measure packet loss, recovery time and post-failure utilization.

06 / Practical BGP troubleshooting

Work from session state to packet forwarding

When a prefix is missing or traffic takes an unexpected path, avoid starting with a broad configuration change. Follow the route through the control plane and forwarding plane, checking each stage before moving to the next.

Verify the session. Confirm the intended BGP neighbor is Established, and inspect logs, reset reasons, interface status and reachability between peer addresses.

Check received routes. Confirm the expected prefix is being advertised by the peer and received locally. Use the platform's received-route or neighbor-route commands where supported.

Inspect policy. Review prefix lists, route maps, import/export policies, community handling, maximum-prefix controls and any policy that could reject or modify the route.

Inspect best path and next hop. Identify why a candidate route was selected or rejected. Confirm that the selected next hop is resolvable and reachable.

Verify multipath installation. If several paths are expected, check the routing table for multiple installed next hops and confirm that multipath settings and platform capabilities permit them.

Validate forwarding. Inspect the forwarding table, interface counters, flow telemetry and packet path. For VXLAN EVPN, also verify underlay reachability, relevant EVPN routes, VTEP reachability and tenant forwarding state.

Example diagnostic commands

These are illustrative Cisco IOS/IOS XE-style commands. Exact syntax and available detail vary by operating system and platform; use the equivalent commands for your switches.

show ip bgp summary
show ip bgp
show ip route
show ip cef
show running-config | section router bgp

Start with the neighbor summary, then inspect the target prefix and its candidate paths. Check the routing table to determine which route was selected, and inspect the forwarding table to verify the next hop used for packet forwarding. Review the BGP configuration and policies when the observed result differs from the intended design.

The relevant BGP neighbor is Established.
The expected prefix is received and passes import policy.
The selected route has a valid, reachable next hop.
Expected ECMP next hops are installed when multipath is intended.
Export policy allows the required route to reach downstream peers.
Forwarding state and traffic telemetry agree with the expected path.
Failure tests confirm acceptable convergence and surviving capacity.

Continue with the BGP Learning Centre

Explore route propagation, best-path selection and fault investigation through the Network Insight BGP Learning Centre. It complements this architecture section with a practical, interactive way to examine BGP behaviour.

Open the BGP Learning Centre: Explore. Trace. Diagnose. →

NETWORK INSIGHT · BGP FABRIC EXPLORER

Explore BGP in a Leaf-Spine Fabric

Select a router, inspect the illustrative BGP sessions, trace how a prefix is learned, and simulate a path failure. See how routing information and forwarding paths work together in a data center network.

Illustrative BGP fabric Sessions established

iBGP eBGP Failed path

Selected element

Spine 1

A spine switch provides fabric transit. Depending on the design, it may establish eBGP sessions with leaf switches and exchange underlay reachability.

Engineering insight

BGP distributes reachability information. The forwarding plane uses the installed routes to move packets; a BGP session being established does not alone prove that an application path is working.

Interactive controls

6Illustrative BGP sessions

2Illustrative available paths

EstablishedSelected scenario state

Follow a route advertisement 10.10.10.0/24

1Prefix learnedA peer advertises reachability to the prefix.

2BGP evaluatesAttributes and policy influence route selection.

3Route installedAn eligible route is installed for forwarding.

4Packet forwardedThe forwarding table directs traffic to a next hop.

Underlay reachability

The underlay provides IP reachability between fabric devices and their loopbacks or next hops. eBGP is often used between leaf and spine switches in routed fabrics, though designs vary.

Route selection and forwarding

BGP applies policy and path-selection rules to choose eligible routes. ECMP may install multiple equal-cost next hops when supported by the routing design and platform.

Educational model only. This topology and its route progression are illustrative, not a live BGP implementation. Real session counts, ASNs, route attributes, ECMP eligibility, convergence and forwarding behavior depend on the configured network and platform.

05 / Observe

Network Visibility, Traffic Intelligence and BGP

You cannot reliably protect a network if you cannot establish what is happening within it. Flow records, packet captures, encrypted-traffic metadata and security logs reveal different aspects of network behaviour and provide evidence for troubleshooting and threat detection.

This section connects traffic analysis with BGP route visibility, helping engineers investigate unexpected communication patterns, routing changes and reachability problems using evidence from multiple layers.

05 / Observe

Network Visibility: From Traffic Flows to BGP Intelligence

Security controls determine what should be allowed. Observability helps engineers establish what is actually happening. By correlating flow records, packet-level evidence, encrypted traffic metadata and routing information, network teams can identify suspicious behaviour, investigate connectivity failures and understand how an incident moves across the network.

A firewall log may show that a connection was permitted, but it rarely explains the entire conversation. A packet capture may reveal a protocol exchange, yet only represent a short interval or one observation point. Flow telemetry can expose communication patterns across a longer period, while routing telemetry can explain which paths were available and why traffic may have taken an unexpected route.

Engineering principle: No single telemetry source provides the complete picture. Strong network investigations correlate evidence from multiple layers rather than treating an alert, a flow record or a routing change as conclusive proof on its own.

5.1 — Choose the right observation method

Flow telemetry

NetFlow and IPFIX

Summarise conversations using fields such as source and destination addresses, ports, protocol, packet counts, byte counts and timestamps. IPFIX provides an extensible framework for exporting structured flow information.

Sampling

sFlow

Uses statistical sampling of packets and interface counters to provide scalable visibility into traffic patterns, particularly in large switched and routed networks. Sampling can miss individual events.

Packet evidence

SPAN and packet capture

SPAN mirrors selected switch traffic to an observation port. Packet captures can reveal protocol headers, retransmissions, handshakes and other details that summaries cannot show. Mirror oversubscription can cause packet loss.

Encrypted traffic

Metadata and behaviour

Connection timing, volume, direction, destination reputation, DNS context where available, TLS handshake metadata and endpoint telemetry can help detect anomalies without assuming that encrypted payloads are readable.

Method	Best used for	Important limitation
NetFlow / IPFIX	Traffic accounting, unusual communication patterns and historical investigation	Typically lacks full packet payload and detailed packet-by-packet sequence
sFlow	High-volume traffic trends and scalable network sampling	Sampling may omit short-lived or low-volume activity
SPAN / packet capture	Protocol troubleshooting and detailed session analysis	Limited observation point, capture duration and possible mirror-port loss
Endpoint and security logs	Process attribution, identity, host events and security outcomes	Coverage and reliability depend on instrumentation and log integrity
BGP and routing telemetry	Route changes, path selection, reachability and control-plane anomalies	A valid route does not prove that the application or forwarding path works

5.2 — Build an evidence chain

Consider a server that suddenly transfers much more data than usual. A single bandwidth graph might identify the spike, but it cannot establish whether the cause is a legitimate backup, a compromised service or an unexpected change in traffic routing. A structured investigation combines several sources.

DetectBaseline deviation

→

CorrelateFlows and identities

→

InspectPackets and logs

→

ValidateRoutes and forwarding

First, establish when the anomaly began. Next, identify the communicating endpoints, destination ports, volume and duration. Compare the behaviour against the normal baseline. Then inspect available DNS, endpoint, firewall and packet evidence. Finally, check whether a route change, next-hop failure or policy update coincided with the event.

Interactive investigation explorer

Select an observation to see what it can establish and what should be checked next.

INVESTIGATION VIEW Flow records: establish who communicated

Compare source and destination addresses, ports, protocol, bytes, packets and timestamps. Identify unusual destinations, repeated connections or a sudden change in traffic volume. Then correlate the flow with endpoint and security logs.

5.3 — Encrypted traffic: visibility without payload inspection

Encryption protects confidentiality, but it does not make every aspect of a connection invisible. Depending on the protocol, configuration and observation point, engineers may still have access to connection duration, byte counts, packet sizes, timing, DNS information, destination addresses and some handshake metadata.

Behavioural baselines

Detect changes in connection frequency, data volume, session duration and communication direction compared with a service's established pattern.

Identity correlation

Combine network observations with endpoint, workload and authentication records to associate a connection with a device, user or application where the evidence supports it.

Protocol-aware analysis

Use available TLS, DNS and transport metadata carefully. Encrypted Client Hello and encrypted DNS can limit visibility into some fields, while QUIC changes what traditional passive inspection can observe.

Important distinction: A large encrypted transfer to an unfamiliar destination is an investigation lead, not proof of data exfiltration. Validate the destination, expected application behaviour, endpoint context and any applicable organisational policy before drawing conclusions.

5.4 — BGP visibility: understand the route behind the traffic

Border Gateway Protocol (BGP) exchanges reachability information between autonomous systems and distributes routes within an autonomous system through iBGP. Its control plane influences the paths that IP packets may follow. For network security and troubleshooting, this means traffic anomalies should sometimes be investigated alongside BGP updates, routing-table changes, next-hop resolution and forwarding state.

Routing intelligence

Three autonomous systems, one destination prefix

Illustrative topology for correlating route propagation with observed traffic.

AS65002 External BGP peer

AS65001 Internal BGP and routing policy

AS65003 External BGP peer

Example destination: 10.10.10.0/24

# Illustrative route observation Prefix: 10.10.10.0/24 Protocol: BGP AS_PATH: 65002 64496 NEXT_HOP: 192.0.2.2 Observation: Route present in BGP table Validation: Confirm best path, next-hop reachability and FIB

The values above are illustrative. They do not describe a live network or establish that this route is selected, reachable or safe.

What should engineers observe?

UPDATE activity: identify announcements and withdrawals, including the prefix affected and the time of the change.
AS_PATH changes: investigate unexpected path changes, unusual origins and route attributes that may affect selection.
Best-path selection: compare competing routes, local preference, AS_PATH length and other attributes in the context of the vendor's decision process and policy.
Next-hop resolution: verify that the selected next hop is reachable through the relevant routing information.
Forwarding state: confirm that the selected route is programmed into the forwarding information base (FIB) and that traffic can actually traverse the path.
Session health: correlate BGP neighbour state transitions, resets, timers and logs with the time the traffic anomaly occurred.

Security note: BGP route leaks and route hijacks can affect reachability or redirect traffic. Monitoring is only one defence. Apply appropriate prefix filtering, route-policy controls, peer validation and, where supported and operationally suitable, RPKI-based Route Origin Validation. ROV validates route-origin authorisation; it does not by itself validate the entire AS_PATH or prevent every routing attack.

5.5 — BGP and traffic telemetry must be correlated

A route appearing in the BGP table is not the same as a successful application connection. BGP provides control-plane reachability information, while the forwarding plane handles packets. An investigation must distinguish a route that has been received from one that has been selected, installed and successfully used.

Observation	Possible interpretation	Next validation step
Traffic volume drops suddenly	Application failure, link problem, filtering or a route change	Correlate interface counters, flow records, route events and endpoint health
AS_PATH changes unexpectedly	Legitimate policy change, upstream change or suspicious announcement	Check the source peer, prefix origin, route policy and trusted routing evidence
BGP route exists, but traffic fails	Unresolved next hop, FIB inconsistency, ACL, MTU issue or downstream failure	Check route selection, next-hop recursion, FIB, interface state and packet path
Repeated BGP session resets	Transport instability, peer configuration, resource pressure or timer expiry	Inspect neighbour logs, TCP connectivity, interface errors and control-plane load
Unexpected outbound traffic	New workload behaviour, misconfiguration or potentially malicious activity	Correlate destination, workload identity, DNS, security events and route history

5.6 — Operational requirements for reliable visibility

Observability systems are only as useful as the quality and consistency of their data. Time synchronisation is essential when correlating events across routers, firewalls, switches, hosts and collectors. Retention policies should preserve enough history to establish a baseline and investigate delayed reports. Telemetry collectors should be monitored for gaps, overload and export failures.

Synchronise timestamps across network devices, endpoints and telemetry platforms.
Document flow sampling rates, SPAN limitations and which interfaces are actually observed.
Monitor telemetry loss, collector health, storage capacity and data-retention windows.
Restrict access to sensitive flow records, packet captures and routing logs.
Protect management and telemetry channels against unauthorised access or tampering.
Establish normal traffic and routing baselines before treating deviations as incidents.

Section takeaway: Effective visibility connects traffic behaviour, security events and routing decisions. Flow records show communication patterns; packet captures provide deeper protocol evidence; encrypted-traffic metadata supports behavioural analysis; and BGP telemetry helps explain changes in network reachability. Correlation turns those separate signals into a defensible engineering investigation.

NETWORK INSIGHT · VISIBILITY WORKBENCH

Investigate the Data Center with Network Telemetry

Explore how flow records, interface counters, packet captures and BGP telemetry reveal different parts of a network problem. Select a diagnostic view to correlate traffic behavior with routing and forwarding state.

Traffic and telemetry sources Illustrative telemetry

Network connection Selected diagnostic path Warning signal Potential fault

Selected source

Flow telemetry

Flow records summarize conversations between endpoints. They help identify major talkers, traffic direction and volume, but do not normally contain every packet's full contents.

Engineering insight

No single telemetry source explains every fault. Correlating traffic records, interface counters, packet evidence and routing state can narrow the cause much faster.

Diagnostic actions

Flow recordsSelected evidence source

Traffic patternWhat it can reveal

PartialEvidence coverage

Illustrative diagnostic evidence

Signal	Example observation	Interpretation	Status
Flow records	Large app-to-database flow	Identify traffic source and volume	Useful
Interface counters	Output drops increasing	Investigate congestion or queue pressure	Check
Packet capture	Retransmissions observed	Correlate loss, delay and endpoint behavior	Check
BGP telemetry	Next-hop route changed	Verify route selection and forwarding state	Useful

What the data tells you

Flow telemetry helps locate large or unexpected conversations. It is valuable for traffic engineering and capacity planning, but its detail depends on the exporter, sampling and collection interval.

Correlate before concluding

Compare timestamps and affected paths across telemetry sources. A counter increase, a packet symptom and a routing change are stronger evidence when they align in time and relate to the same flow.

Educational demonstration only. The evidence values are illustrative examples, not live measurements. Available telemetry, sampling, counter definitions, packet visibility and BGP monitoring capabilities vary by platform and deployment.

06 / Respond

Incident Response, Threat Detection and Network Forensics

When suspicious activity is detected, the priority shifts from observation to controlled investigation and response. Engineers must validate the evidence, determine the scope of the incident, preserve relevant records and contain the threat without creating unnecessary operational disruption.

Explore the incident response lifecycle, practical forensic techniques, containment decisions and BGP incident investigation, from the first alert through recovery and lessons learned.

06 / Respond

Incident Response, Threat Detection and Network Forensics

Detecting suspicious activity is only the beginning. Effective incident response requires engineers to validate the alert, establish the scope of the incident, preserve evidence, contain the threat and restore services safely. Network telemetry, endpoint records, security controls and routing information provide complementary evidence that helps teams understand what happened and determine the appropriate response.

A security alert does not automatically establish that a system has been compromised. A sudden increase in outbound traffic, an unfamiliar destination, repeated authentication failures or an unexpected BGP route change may indicate a security incident, a configuration error or legitimate operational activity. The objective is to investigate systematically rather than act on an assumption.

Core principle: Incident response must balance containment, evidence preservation, service availability and business risk. Every action should have a clear purpose, an authorised scope and a way to verify its outcome.

6.1 — The incident response lifecycle

A practical response process moves from preparation through detection, analysis, containment, eradication and recovery. Lessons learned then feed back into monitoring, configuration standards and security controls. In a serious incident, these activities may overlap, and responders may need to revisit earlier stages as new evidence emerges.

PrepareTools, roles and playbooks

→

DetectAlert and validate

→

ContainLimit further impact

→

EradicateRemove root cause

→

RecoverRestore and monitor

Phase 1

Preparation

Define incident ownership, escalation paths, approved containment actions, logging requirements, backup procedures and communication channels before an incident occurs.

Phase 2

Detection and analysis

Validate the alert, establish a timeline, identify affected assets and correlate evidence from network, endpoint, identity and security systems.

Phase 3

Containment

Reduce the opportunity for further harm using proportionate, authorised controls. Consider dependencies and the operational consequences of isolating a device or blocking a communication path.

Phase 4

Eradication and recovery

Remove the confirmed cause, remediate affected systems, restore trusted configurations and verify that services are operating correctly before returning to normal operations.

6.2 — Detect and validate suspicious network activity

Detection systems compare observed activity against rules, threat intelligence, behavioural baselines and known indicators. The resulting alerts must be evaluated in context. An IP address associated with a threat feed, for example, is a useful investigation lead but may be outdated, shared by multiple services or unrelated to the activity under examination.

Signal	Potential concern	Validation approach
Unusual outbound data volume	Possible data transfer or unexpected application behaviour	Check flow records, destination, workload identity, scheduled jobs and endpoint events.
Repeated authentication failures	Possible credential guessing or account misuse	Correlate identity logs, source addresses, successful logins and account activity.
Repeated DNS queries	Possible misconfiguration, automated activity or suspicious domain generation	Inspect query patterns, resolver logs, domain context and the originating endpoint.
Unexpected east-west traffic	Possible lateral movement or a newly introduced dependency	Compare with workload communication baselines, segmentation policy and process telemetry.
Unexpected BGP announcement	Possible route leak, misconfiguration or unauthorised route propagation	Verify the peer, prefix, origin, policy, route history and expected maintenance activity.
Repeated connection resets	Possible service instability, filtering or hostile connection activity	Correlate packet captures, firewall decisions, server logs and interface counters.

6.3 — Interactive incident investigation

Select a scenario to explore the evidence an engineer should collect before choosing a response. These are illustrative investigation paths, not automated diagnoses.

Investigation workbench

Select a scenario to reveal its evidence requirements and recommended next steps.

INVESTIGATION SCENARIO Unusual outbound data transfer

Establish which host initiated the transfer, the destination, data volume, timing and expected application behaviour. Correlate flow records with endpoint activity, DNS context and available security logs before deciding whether containment is necessary.

6.4 — Network forensics and evidence preservation

Network forensics reconstructs relevant events from available evidence. Depending on the environment, that evidence may include packet captures, NetFlow or IPFIX records, DNS logs, firewall events, authentication records, endpoint telemetry, configuration changes and BGP update history.

Packet captures

Preserve the relevant capture interval and observation point. Record any capture filters, truncation settings, packet loss indicators and limitations affecting interpretation.

Centralised logs

Preserve original records where possible, document collection methods and correlate events using consistent timestamps and asset identifiers.

Configuration history

Compare approved and current firewall rules, ACLs, routing policies, interface configuration and relevant changes made before or during the incident.

Evidence handling: Follow organisational policy and applicable legal requirements. Restrict access to sensitive records, preserve provenance, document collection and handling, and use cryptographic hashes where appropriate to help demonstrate that collected files have not changed.

# Example: calculate a SHA-256 hash of a collected evidence file # Run on an authorised system with access to the evidence file. shasum -a 256 incident-capture.pcap # Record the hash, filename, collection time, collector and storage location. # A hash supports integrity verification; it does not prove the original # capture was complete or that its contents are accurate.

6.5 — Containment: reduce risk without creating another outage

Containment aims to limit the impact of an incident while preserving enough operational capability to investigate and recover. The right action depends on confidence in the evidence, the criticality of the affected service, the likely threat and the available alternatives.

Containment option	Potential benefit	Risk to consider
Isolate an endpoint	Limits some network communications from a suspected host	May interrupt critical services or remove access needed for investigation.
Apply a targeted firewall rule	Restricts a verified malicious flow or exposure	An overly broad rule may block legitimate dependencies.
Disable compromised credentials	Reduces the opportunity for further unauthorised access	Can interrupt automated workloads or legitimate administrative activity.
Change routing policy	May mitigate an established routing threat or isolate an affected path	Incorrect changes can cause route withdrawal, traffic blackholing or wider reachability loss.
Increase targeted monitoring	Improves evidence collection when immediate isolation is not justified	Monitoring alone may be insufficient when active harm is continuing.

Operational safeguard: High-impact containment actions should follow the incident response plan and appropriate authorisation. In a critical network, coordinate changes with service owners, assess failover behaviour and maintain a documented rollback plan wherever feasible.

6.6 — BGP incidents: distinguish control-plane anomalies from forwarding failures

BGP-related incidents require particular care because an incorrect response can affect reachability well beyond a single router. Unexpected route advertisements, route leaks, session instability and policy errors may alter the paths available to traffic. However, a route change alone does not establish malicious intent.

# Illustrative BGP incident triage checklist 1. Identify the affected prefix and first observed change. 2. Determine which peer advertised or withdrew the route. 3. Compare AS_PATH, origin and relevant route attributes. 4. Verify import/export policy and expected maintenance changes. 5. Check the selected route and next-hop resolution. 6. Confirm the FIB and test the affected forwarding path. 7. Apply an authorised mitigation and verify the result.

In an interdomain routing incident, useful safeguards may include carefully maintained prefix filters, explicit import and export policies, maximum-prefix controls, appropriate peer authentication and RPKI Route Origin Validation where supported. ROV helps validate whether an origin AS is authorised to originate a prefix; it does not validate every aspect of the route or prevent every possible BGP attack.

Within an autonomous system, engineers should also check iBGP propagation, route-reflector behaviour, next-hop reachability, IGP dependencies and forwarding state. A route may be visible in one routing table but absent from another, or selected by BGP without being usable for forwarding.

6.7 — Recovery, verification and lessons learned

Recovery is more than removing a block or restarting a service. Before returning affected systems to normal operation, teams should verify that the original cause has been addressed, required security controls are working and relevant traffic is following expected paths.

Confirm the affected systems, accounts, prefixes or services have been identified.
Remove the confirmed cause and remediate the relevant configuration or vulnerability.
Restore systems from trusted sources where required and verify their security posture.
Validate routing, next-hop reachability, forwarding state and application connectivity.
Maintain heightened monitoring for recurrence and unexpected changes.
Document the timeline, decisions, evidence, impact, recovery steps and outstanding risks.
Update detection rules, runbooks, access controls and training based on lessons learned.

6.8 — Measure incident response effectiveness

Response metrics help organisations identify gaps and improve operational readiness. They should be interpreted alongside incident severity, service impact and investigation quality rather than treated as isolated performance targets.

Detection

Mean time to detect

Measures the elapsed time between a defined incident starting point and detection. The starting point must be consistently defined for comparisons to be meaningful.

Response

Mean time to contain

Measures how long it takes to implement an effective containment measure after the relevant response clock starts.

Quality

Recurrence and false positives

Review recurring incident causes, unnecessary escalations, detection coverage and whether mitigations actually reduce risk without unacceptable disruption.

Section takeaway: Effective incident response joins detection, evidence preservation, containment, remediation and recovery into a controlled engineering process. Network telemetry and BGP visibility provide important context, but reliable decisions depend on corroborating evidence, understanding operational dependencies and verifying that corrective actions work.

NETWORK INSIGHT · RESILIENCE LAB

What Happens When a Data Center Fabric Fails?

Simulate illustrative link and spine failures in a leaf-spine fabric. Follow the surviving paths, inspect the remaining connectivity and identify what engineers should verify before declaring the fabric resilient.

Leaf-spine fabric connectivity Normal operation

Available connection Traffic path Failed component or link Illustrative alternate path

Scenario analysis

Normal operation

Both spine switches are available. Each leaf has two uplinks, allowing traffic to use multiple fabric paths when routing and forwarding are configured accordingly.

Engineering lesson

Redundancy is only useful when alternate paths are operational, routing converges correctly and the remaining links can carry the offered load.

Test controls

8 / 8Available leaf-to-spine links

2 / 2Available spine switches

2Illustrative paths between selected leaves

Failure investigation checklist

Check	Question	Why it matters	Status
Physical state	Are the links and devices operational?	Confirms the failure boundary	Baseline
Routing	Have paths been withdrawn or recalculated?	Control-plane state influences path selection	Baseline
Forwarding	Are alternate next hops installed?	Confirms traffic has a usable path	Baseline
Capacity	Can surviving links carry the load?	Prevents congestion after failover	Baseline

Resilience is more than redundancy

Multiple links and switches create alternate paths, but they do not guarantee uninterrupted service. Detection time, routing convergence, forwarding updates, hashing and remaining capacity all affect the outcome.

Validate the recovery

After a failure, verify reachability and application behavior, inspect routing and forwarding state, compare interface counters and confirm that the surviving fabric is not overloaded.

Educational model only. This is a simplified two-spine, four-leaf fabric. The diagram illustrates connectivity and possible alternate paths; it does not calculate live routing convergence, ECMP hashing, throughput, packet loss or application availability. Actual behavior depends on topology, routing policy, hardware and configuration.

Network Insight · Architecture Briefing

Designing Data Center Networks for Scale, Visibility and Resilience

Data center topology is more than a diagram of switches and links. It determines how workloads communicate, how capacity scales, how routing adapts to change and how engineers isolate faults when the network behaves unexpectedly.

Six ideas to take away

From architectural foundations to operational troubleshooting, each design decision affects the next.

Choose the right architecture

Three-tier designs separate access, aggregation and core functions. Leaf-spine fabrics provide a more consistent path structure for many modern, east-west-heavy workloads.

Understand the fabric

Leaf switches connect endpoints; spine switches interconnect the leaves. Clos-style designs can scale horizontally, subject to port count, link speed and oversubscription choices.

Design for real traffic

North-south and east-west flows create different demands. Capacity planning must consider traffic concentration, ECMP distribution, burstiness and workload communication patterns.

Make routing part of the design

BGP can provide scalable underlay routing in a leaf-spine fabric. Route policy, next-hop reachability, ECMP and convergence behavior all influence the forwarding outcome.

Make the network observable

Flow telemetry, interface counters, packet captures and routing telemetry reveal different parts of the story. Correlating them helps connect symptoms to causes.

Prove resilience under failure

Redundant links are only the beginning. Test failover, verify routing and forwarding state, check surviving capacity and confirm that affected applications recover.

A simplified fabric mental model

SPINE LAYER Path diversity

Leaf 1Leaf 2Leaf 3Leaf 4

ComputeStorageApplications

Think in layers, paths and failure domains.

A useful fabric design connects physical topology, routing behavior and application requirements. Each layer answers a different engineering question: what is connected, where can traffic travel and what happens when a component is lost?

✓Topology: Are there enough paths between the required endpoints?

✓Routing: Are the intended next hops available and installed?

✓Capacity: Can the remaining fabric carry the expected load?

✓Operations: Can telemetry identify the fault and verify recovery?

The engineering takeaway

A resilient data center is not simply one with redundant links. It is one whose behavior can be understood, measured and tested.

Build around the workload, understand the trade-offs in the chosen topology, validate routing and forwarding, and exercise failure scenarios before they occur in production. The goal is not just connectivity—it is predictable performance and recoverable service.

```

☕

Support Network Insight

Help support independent engineering guides, interactive networking tools, and future learning labs.

Support Network Insight ```

EVPN Explained: A Control Plane for Modern Ethernet VPNs

Three concepts to understand first

Endpoint reachability

MPLS or VXLAN

Multihoming and recovery

What this guide will help you investigate

Traditional Layer 2 VPNs: Connecting Ethernet Networks Across a Provider

Pseudowires

VPLS

Where the traditional model becomes difficult

Why this led to EVPN

Incident Challenge: The Remote MAC Is Missing

Remote MAC missing on PE1

Choose your next diagnostic check

Investigation not started

Root-cause analysis

How Does EVPN Turn Endpoint Learning Into a Scalable Control Plane?

Traditional Layer 2 VPNs: How Does the Service Connect Sites?

Point-to-point pseudowire

Step 1 — The customer sends an Ethernet frame

How the EVPN Control Plane Works

What does an EVPN advertisement tell the network?

MAC/IP Advertisement

Inclusive Multicast Ethernet Tag

Ethernet Auto-Discovery

Ethernet Segment

IP Prefix

Three identifiers and policies to understand

How does the route become usable for forwarding?

How Does EVPN Distribute MAC Reachability?

Step 1 — Learn the local MAC

Local learning happens first

EVPN over MPLS vs EVPN over VXLAN

Experiment Result

EVPN over MPLS vs EVPN over VXLAN

EVPN over MPLS

EVPN over VXLAN

Comparing the design considerations

How to choose between them

EVPN Transport Lab: MPLS or VXLAN?

Step 1 — EVPN provides endpoint reachability

Step 1 — EVPN provides endpoint reachability

BGP, VXLAN and Modern Data-Centre Fabrics

1. The underlay: a routed IP foundation

Route between VTEPs

ECMP and alternate paths

2. The overlay: VTEPs and VXLAN

3. MP-BGP EVPN: distributing endpoint reachability

Following one packet across the fabric

Endpoint mobility and flooding

What to check when traffic fails

Inside a VXLAN EVPN Fabric

Two endpoints, two VTEPs

Control plane

Engineering principle

2 · Fabric fault challenge

EVPN Multihoming and Resilience

Single-active and all-active multihoming

One active forwarder

Multiple active paths

The role of the Designated Forwarder

How EVPN responds to failures

Why Ethernet Segment routes matter

Designing for predictable recovery

EVPN Multihoming: Explore Resilience

All-active multihoming

Engineering observation

A step-by-step investigation

Confirm the local endpoint and service mapping

Verify the EVPN control-plane session

Trace the advertised and imported EVPN route

Check the overlay service and forwarding state

Validate the transport underlay

Investigate multihoming when the service uses an Ethernet Segment

Separate known-unicast failures from BUM behavior

Symptom-to-check reference

Keep the fault domains separate

EVPN Incident: The Remote MAC Is Missing

Data Center Topology: From Three-Tier Networks to Leaf-Spine Fabrics

Design principle: build around application requirements