Data Center Topology: From Three-Tier Networks to Leaf-Spine Fabrics
Data center topology defines how switches, servers, storage systems and external networks connect—and ultimately how application traffic moves through the infrastructure. The architecture you choose influences latency, available bandwidth, resilience, operational complexity and the ability to scale as workloads grow.
Traditional hierarchical networks remain relevant in many environments, but modern cloud platforms and distributed applications increasingly rely on scalable Layer 3 fabrics, leaf-spine architectures and equal-cost multipath routing. Understanding these designs means looking beyond the network diagram to examine routing behaviour, traffic patterns, failure domains and the control plane that keeps the fabric connected.
Design principle: build around application requirements
A good topology is not simply the one with the most links or the most switches. It is the one that delivers the required capacity, predictable forwarding, appropriate redundancy and manageable failure behaviour at an acceptable cost.
Topology and Fabric Design
Compare three-tier, centralized, top-of-rack, end-of-row and leaf-spine architectures.
BGP and ECMP
Explore routing adjacencies, route selection, multipath forwarding and convergence during failures.
VXLAN and EVPN
Understand how Layer 3 underlays support virtual networks and distributed endpoint reachability.
Capacity and Resilience
Evaluate oversubscription, traffic distribution, observability and failure-domain behaviour.
In this guide: Work from fundamental topology choices to modern data center fabrics, BGP-based routing, traffic engineering considerations and practical troubleshooting.
Explore Data Center Topology
Navigate the architecture, routing, traffic flows and resilience principles behind modern data center networks.
Understanding Data Center Architectures
Data center topology defines how servers, storage systems, switches and external networks connect. The architecture determines how traffic moves through the infrastructure and influences available bandwidth, latency, redundancy, scalability and operational complexity.
This section introduces traditional three-tier networks, centralized switching, top-of-rack (ToR), end-of-row (EoR) and leaf-spine architectures. Compare their connectivity models and trade-offs to understand which designs best match different workload and infrastructure requirements.
The Main Data Center Architectures
Data center networks can be built using hierarchical switching layers, centralized switching platforms, distributed access switches or fabric-based designs. Each approach creates a different balance between cabling, port utilization, bandwidth, failure isolation and operational complexity.
The right starting point is the workload. A smaller environment with limited server-to-server traffic may not need the same architecture as a cloud platform supporting thousands of virtual machines, containers or data-intensive applications. Topology selection should reflect traffic patterns, growth expectations, availability targets and the cost of operating the network over time.
Traditional Three-Tier Architecture
A traditional enterprise design separates access, aggregation or distribution, and core functions. Access switches connect servers, the aggregation layer concentrates traffic and may enforce policy, and the core provides high-speed connectivity between network blocks.
This model remains useful where existing infrastructure, established operational practices or clear policy boundaries make a hierarchical design appropriate. However, multiple switching tiers can increase path length and create bottlenecks when east-west traffic grows.
Centralized Switching
In a centralized design, servers or server racks connect back to a central switching location. Concentrating switching resources can simplify management and make port allocation efficient in smaller environments.
As the facility grows, cable lengths, pathway congestion, centralized capacity limits and the consequences of equipment failure must be evaluated carefully.
Top-of-Rack and End-of-Row
Top-of-rack (ToR) places access switches in individual server racks. End-of-row (EoR) places switching equipment in a dedicated rack serving a row of servers. Both approaches can provide redundant connectivity when designed with suitable independent paths and devices.
ToR generally shortens server cabling and can isolate access failures to a rack. EoR can consolidate switching resources, but may require more structured cabling. Neither approach automatically determines whether the wider network uses a three-tier or leaf-spine architecture.
Leaf-Spine Architecture
A leaf-spine fabric connects each leaf switch to multiple spine switches. Servers and other endpoints connect to the leaf layer, while the spine layer provides paths between different leaves. A Layer 3 underlay commonly uses equal-cost multipath routing to distribute eligible traffic across available paths.
This architecture provides predictable path structure and supports incremental expansion. Its capacity and resilience depend on link speeds, the number of spine switches, oversubscription, routing design and failure behaviour.
Important distinction: ToR and EoR describe where access switching is placed. Three-tier and leaf-spine describe broader network architectures. A data center can use ToR switches as its leaf layer in a leaf-spine fabric.
| Design consideration | Three-tier / centralized | Leaf-spine fabric |
|---|---|---|
| Traffic paths | May traverse several hierarchical layers. | Typically uses a consistent leaf-to-spine-to-leaf path. |
| Scaling | Expansion depends on available tier capacity and design. | Can expand by adding leafs and, when needed, spine capacity. |
| Multipath routing | Depends on the routing and switching design. | Layer 3 ECMP is commonly used between leafs and spines. |
| Operational considerations | May suit established environments and existing investments. | Requires deliberate underlay, routing, capacity and failure planning. |
Traffic patterns
Determine whether workloads primarily communicate with external networks or generate substantial east-west traffic between servers.
Capacity and growth
Estimate endpoint growth, uplink bandwidth, oversubscription limits and the cost of adding future capacity.
Failure domains
Identify the impact of losing a switch, link, rack or entire network block, including the capacity that remains.
Operations and cost
Balance equipment and cabling costs against monitoring, troubleshooting, automation and lifecycle requirements.
Explore Data Center Architectures
Compare four common ways to connect compute, access and switching. Select an architecture, then click a device in the topology to inspect its role in the network.
Core switch
The core provides high-speed connectivity between distribution blocks and upstream networks in a traditional three-tier design.
Architecture insight
Separate layers make roles easy to understand, but traffic may traverse multiple tiers and scaling can increase operational complexity.
Design strengths
- Clear separation of network roles
- Familiar operational model
- Can suit hierarchical traffic patterns
Design considerations
- Potentially longer paths between servers
- Capacity planning across tiers
- Additional layers to operate and troubleshoot
These diagrams illustrate common design patterns, not vendor-specific configurations. Actual forwarding paths, redundancy and oversubscription depend on the implementation. ToR means Top of Rack; EoR means End of Row; Clos describes a multistage network fabric.
Leaf-Spine and Clos Fabrics
Leaf-spine architecture connects each leaf switch to multiple spine switches, creating a consistent path structure between different server racks. It is widely used in modern data centers where distributed applications, virtualization and cloud workloads generate substantial east-west traffic.
Explore how leaf and spine layers work together, how two-tier fabrics relate to Clos network designs, and how adding switches can expand connectivity. The section also examines oversubscription, available path diversity and the limits of relying on topology alone to guarantee performance.
Leaf-Spine and Clos Fabrics
Leaf-spine architecture is a common foundation for modern data center networks. Instead of sending traffic through a succession of hierarchical switching layers, it connects access-layer leaf switches to a set of spine switches. This creates a fabric with consistent paths between leaf switches and provides a repeatable way to expand connectivity as workloads grow.
The architecture is particularly useful for environments with substantial east-west traffic, including virtualized platforms, container infrastructure, distributed storage and cloud-native applications. Its performance depends on the actual link speeds, number of paths, traffic distribution and capacity provisioned throughout the fabric.
Conceptual topology, not a physical cabling diagram. Each leaf has links to each spine; server-facing connections are shown separately. The simplified graphic illustrates connectivity rather than every individual link.
1. The role of leaf switches
Leaf switches provide connectivity for servers, storage, firewalls, load balancers and other data center endpoints. They may also perform Layer 3 routing, enforce network policy and act as VXLAN tunnel endpoints when an overlay is deployed.
In a Layer 3 fabric, endpoints in different subnets can communicate through routed paths. When VXLAN EVPN is used, the leaf layer can additionally provide overlay connectivity and distribute endpoint reachability through the EVPN control plane.
2. The role of spine switches
Spine switches interconnect the leaf layer and provide transit paths between leaf switches. In a conventional two-tier leaf-spine fabric, spine switches do not connect directly to servers, and leaf switches do not connect directly to other leaf switches.
Adding spine capacity can increase the number of available paths and aggregate fabric bandwidth, provided the leaf uplinks, interfaces, routing configuration and downstream capacity are designed to use it. A spine failure can reduce available capacity even when connectivity survives.
Why path predictability matters: Traffic between endpoints attached to different leaf switches normally traverses one spine switch. This creates a consistent switching-hop structure, although actual latency still depends on link speeds, device processing, congestion and the forwarding path.
Expand endpoint connectivity
Add leaf switches with uplinks to the fabric's spine layer. Confirm that the existing spine and uplink capacity can support the additional traffic.
Increase fabric bandwidth
Additional spine switches can create more equal-cost paths. Validate routing limits, optics, cabling, interface capacity and platform support before expanding.
Plan for real traffic demand
Compare aggregate server-facing bandwidth with available uplink bandwidth. Size the fabric according to workload behaviour and acceptable contention.
Test the remaining capacity
Evaluate whether surviving spines and links can carry traffic after a failure, not simply whether alternate paths exist.
Leaf-spine is a practical implementation of a two-tier Clos-style fabric. Larger environments may introduce a super-spine or additional fabric stages to interconnect multiple pods, rooms or data halls. These designs extend the architecture while requiring careful consideration of path diversity, capacity and failure boundaries.
| Design | Typical purpose | Key consideration |
|---|---|---|
| Two-tier leaf-spine | Connect endpoints across a single fabric. | Spine capacity, leaf uplinks and ECMP behaviour. |
| Multi-pod fabric | Connect modular groups of racks or infrastructure. | Inter-pod connectivity and failure isolation. |
| Super-spine / multi-stage Clos | Scale connectivity across larger fabrics or multiple data halls. | Additional stages, path design and end-to-end capacity. |
Explore Leaf-Spine and Clos Fabrics
Investigate how leaf and spine switches connect, trace an illustrative server-to-server path, simulate a spine-link failure, and see why uplink capacity and oversubscription matter.
Leaf switches
Leaf switches connect servers and provide uplinks into the spine layer. In a conventional two-tier fabric, each leaf connects to every spine.
Fabric principle
A full-mesh leaf-to-spine design provides multiple potential paths between different leaves. Routing and ECMP determine which next hops are installed and used.
Why use a Clos-style fabric?
A multistage design can provide scalable connectivity and multiple paths without requiring every leaf to connect directly to every other leaf.
What should engineers verify?
Check spine and leaf capacity, uplink speed, ECMP behavior, failure convergence and the bandwidth remaining after a link or device failure.
This is an educational model, not a live network or packet-level simulator. The four-leaf, two-spine diagram is illustrative. ECMP distributes eligible flows across available next hops; it does not guarantee that each physical link carries exactly the same load.
Data Center Traffic Patterns
A data center network must accommodate the traffic generated by applications, storage systems, virtual machines and distributed services. Understanding where traffic originates and where it travels is essential for designing a fabric with sufficient capacity and predictable performance.
This section distinguishes east-west traffic between internal workloads from north-south traffic entering or leaving the data center. It explores how flow size, concurrent demand, ECMP hashing, uplink capacity and congestion influence the actual behaviour of a leaf-spine network.
Data Center Traffic Patterns
A data center topology must be designed around the traffic it carries, not simply the number of switches and links it contains. Application placement, storage access, replication, service dependencies and external connectivity all influence where traffic flows and where congestion can occur.
Leaf-spine fabrics are particularly suited to distributed east-west communication, but north-south traffic, large data transfers and uneven workload demand can still create bottlenecks. Understanding these patterns helps engineers size uplinks, evaluate redundancy and choose appropriate routing and forwarding designs.
Workload-to-workload traffic
Traffic between servers, virtual machines, containers, storage systems and distributed application components. In modern environments, this can represent a substantial share of total network demand.
Traffic entering or leaving the fabric
Traffic between internal workloads and external networks, users, WANs, internet services or other data centers. Its path may include border leaf switches, firewalls, routers and load balancers.
Rack 1
Rack 2
Simplified logical path. In a conventional two-tier leaf-spine fabric, inter-leaf traffic normally traverses a spine. The forwarding plane selects an available path according to the installed routes and forwarding configuration.
Why east-west traffic affects fabric design
Distributed applications frequently exchange data across multiple servers. Examples include database replication, distributed storage, service-to-service requests, cluster coordination and virtual machine communication. If these flows cross leaf switches, the spine layer and leaf uplinks must provide enough capacity for their combined demand.
Traffic demand is rarely perfectly uniform. Some flows are short and latency-sensitive, while others transfer large volumes of data. Multiple simultaneous flows may also compete for the same links. Consequently, aggregate bandwidth alone does not reveal every congestion risk.
Important distinction: A fabric can have several equal-cost paths and still experience congestion. ECMP provides a mechanism for distributing eligible flows across paths; it does not guarantee that every flow uses a different link or that traffic is perfectly balanced.
North-south traffic and border connectivity
External traffic commonly enters or leaves through designated border leaf switches or dedicated edge devices. These connect the fabric to WAN routers, internet edge infrastructure, firewalls, load balancers and other networks.
The design must consider both bandwidth and resilience. Concentrating external connectivity on a small number of devices may simplify policy enforcement, but it can also create capacity constraints or larger failure domains. Distributing border functions can improve scale, provided routing policy and security controls remain consistent.
| Traffic pattern | Potential pressure point | Design response |
|---|---|---|
| Server-to-server | Leaf uplinks and spine capacity. | Assess aggregate demand, path diversity and ECMP. |
| Storage replication | Sustained high-throughput flows. | Measure peak demand and validate link utilization. |
| Internet or WAN access | Border links, security appliances and external routing. | Plan edge capacity, policy and redundant paths. |
| Large data transfers | Flow imbalance and transient congestion. | Evaluate hashing behaviour, flow sizes and telemetry. |
| Failure or maintenance | Reduced capacity on surviving paths. | Validate traffic behaviour after losing a link or device. |
Measure the traffic, not just the topology
Traffic engineering should use interface counters, utilization trends, flow telemetry, packet drops, latency measurements and application performance data. These observations help distinguish a routing issue from a capacity problem or an application-generated traffic burst.
Engineers should also examine behaviour during failure conditions. A fabric that operates comfortably during normal traffic may become congested when a spine or uplink is unavailable and flows are redistributed across fewer remaining paths.
Explore Data Center Traffic Patterns
Follow traffic between servers, across the fabric and toward external networks. Adjust the illustrative load to understand why workload patterns, path capacity and congestion matter when designing a data center.
East-west traffic
Traffic between workloads inside the data center. In a leaf-spine fabric, communication between servers on different leaves normally traverses a spine.
Engineering insight
Distributed applications, storage replication and service-to-service calls can generate substantial east-west traffic. Design for workload demand rather than assuming all traffic exits the data center.
At this illustrative load, the selected links have moderate demand. Real congestion depends on offered traffic, available link capacity, buffering, queueing and the distribution of flows.
Why traffic direction matters
East-west traffic can stay inside the fabric and use leaf uplinks and spine capacity. North-south traffic crosses the boundary toward external networks or services.
What engineers should measure
Compare interface utilization, drops, queue depth, flow records and application latency. A high average utilization can hide short bursts that cause packet loss or delay.
This widget is a conceptual teaching model, not a live traffic generator or telemetry system. The load slider represents illustrative demand, not measured utilization. Actual paths and capacity depend on topology, routing, link rates, hashing and workload behavior.
BGP in Data Center Fabrics
Border Gateway Protocol (BGP) provides a scalable way to distribute network reachability information across modern data center fabrics. In a routed leaf-spine design, BGP can establish the routes that allow traffic to move between endpoints while supporting controlled route propagation and multipath forwarding.
Explore eBGP and iBGP design approaches, route advertisements, best-path selection, ECMP and convergence. The section also introduces the relationship between a routed underlay and a VXLAN EVPN overlay, showing why control-plane state must be checked alongside the actual forwarding table.
BGP in Data Center Fabrics
Border Gateway Protocol (BGP) is widely used to exchange reachability information in modern data center networks. In a routed leaf-spine fabric, it can provide scalable route distribution, controlled reachability, multipath forwarding and predictable responses to topology changes.
BGP does not replace the physical fabric. It supplies control-plane information that routers use to decide which network destinations are reachable and which paths can be installed for forwarding. The resulting behaviour depends on the routing design, policy, platform capabilities and operational configuration.
Keep the layers distinct: the physical topology defines available links; the routing control plane distributes reachability; the forwarding plane moves packets using installed forwarding entries. A healthy BGP session alone does not prove that the intended traffic path is working.
eBGP leaf-spine
Leaf and spine switches use different autonomous system numbers (ASNs), and BGP sessions between them operate as external BGP. This is a common design for routed Clos fabrics because it can keep routing relationships explicit and support repeatable deployment patterns.
iBGP-based fabric
Routers exchange BGP routes within the same autonomous system. Larger designs may use route reflectors to distribute routes without requiring a full mesh of iBGP sessions between every participating router.
Hybrid routing
A fabric may combine an IGP or static routing for underlay reachability with BGP for other routing functions. The exact division depends on the design; there is no requirement that every data center use one universal routing model.
Border and external routing
Border leaf switches or edge routers exchange routes with external networks. Import and export policies determine which prefixes can enter or leave the fabric and how they are represented to other routing domains.
Example: eBGP in a leaf-spine fabric
In this simplified example, each leaf has an eBGP session to each spine. Leaf-to-spine routing sessions exchange the reachability needed to forward traffic across the fabric. The example illustrates the relationship rather than prescribing a specific ASN plan or configuration.
Conceptual relationship only; individual physical links and separate BGP sessions are not drawn. ASNs are illustrative. This design assumes the chosen BGP configuration permits the required route exchange between the relevant devices.
Design caveat: eBGP does not automatically guarantee a working fabric or multipath forwarding. ASN allocation, next-hop reachability, route policy, prefix limits, supported BGP features and ECMP configuration must all be validated.
The BGP underlay
The underlay is the routed IP infrastructure that connects the switches and provides IP reachability across the fabric. In a BGP-routed underlay, BGP can advertise the infrastructure prefixes needed to reach loopbacks and other routed destinations. These routes provide the foundation for control-plane sessions and overlay tunnel endpoints where applicable.
The VXLAN EVPN overlay
In a VXLAN EVPN design, VXLAN encapsulates tenant traffic across the IP fabric, while EVPN uses BGP to distribute overlay reachability information. This can include information about MAC addresses, IP addresses and tenant segments, depending on the EVPN route type and deployment.
| Component | Primary responsibility | Example question |
|---|---|---|
| Underlay routing | IP reachability between routed fabric endpoints. | Can this VTEP reach the remote VTEP address? |
| BGP EVPN control plane | Advertises overlay reachability and related information. | Has the expected MAC/IP or prefix route been learned? |
| VXLAN data plane | Encapsulates tenant packets across the IP underlay. | Can the encapsulated packet traverse the fabric? |
| Forwarding and policy | Applies installed forwarding entries and applicable policies. | Does traffic reach the correct destination and tenant? |
These functions are related but not interchangeable. A healthy underlay does not prove that EVPN routes are correct, and receiving an EVPN route does not by itself prove that end-to-end forwarding succeeds.
From route advertisement to forwarding
Receive: a BGP neighbor advertises a prefix and its associated path attributes. The receiving router evaluates the update against session state, import policy and protocol rules.
Validate: the route must be acceptable under policy and protocol requirements. Its next hop must also be resolvable for the route to be usable for forwarding.
Select: the router evaluates eligible paths using its BGP decision process and implementation-specific policy. Attributes such as local preference, AS path, origin and MED can influence the outcome; their exact effect depends on the decision context and configuration.
Install: the selected route or eligible set of multipaths may be installed into the routing table and forwarding hardware, subject to platform limits, configuration and interactions with other routing sources.
Forward: packets are sent using the installed forwarding entries. The actual path depends on the forwarding implementation, ECMP eligibility, hashing and current topology.
Best path is not the same as every installed path. BGP implementations can select a best path while also installing multiple eligible paths for multipath forwarding. Multipath rules and supported attributes vary by platform and configuration.
Why BGP policy matters
BGP exchanges reachability, but policy determines which routes are accepted, preferred, modified or advertised onward. In a data center, an incorrect policy can suppress a required prefix, permit an unintended route or cause traffic to prefer an unexpected exit.
Engineers should explicitly document intended import and export behaviour, allowed prefixes, ASN expectations, next-hop handling and any route attributes used to influence path selection. Reusable policy templates can reduce configuration drift across large fabrics.
How equal-cost paths support fabric capacity
Equal-Cost Multipath (ECMP) allows a router to forward traffic across multiple eligible next hops for a destination. BGP can supply candidate paths, but the routing implementation and configuration determine whether multiple paths are considered equivalent and installed.
Once multiple next hops are installed, forwarding hardware commonly uses a hash of packet or flow fields to select a path. This helps distribute traffic while typically preserving packet ordering within a flow. It does not mean every individual packet is sent down a different path, nor does it guarantee perfectly even utilization.
| Mechanism | Role | Common misconception |
|---|---|---|
| BGP best-path selection | Chooses a preferred route according to policy and decision rules. | It does not necessarily install every received path. |
| BGP multipath | Allows multiple eligible BGP paths to be installed. | It is not automatic on every platform or configuration. |
| ECMP forwarding | Distributes eligible flows across equal-cost next hops. | It does not guarantee perfect load balance. |
| LACP | Negotiates and manages a Layer 2 link aggregation group. | It is not the same mechanism as Layer 3 ECMP. |
ECMP and LACP can both use hashing in forwarding or distribution decisions, but they operate at different levels. ECMP selects among Layer 3 next hops; LACP negotiates membership in a Layer 2 aggregated link. A design may use one or both, depending on its physical and logical architecture.
What happens when a path fails?
When a link or device fails, the fabric must detect the event, update the relevant routing state and adjust forwarding. BGP may withdraw affected routes or advertise revised reachability. The forwarding plane then changes its installed entries as the control plane and hardware respond.
Convergence time depends on failure detection, session behaviour, routing policy, protocol timers, implementation, hardware programming and the scope of the change. BFD may help detect some failures faster when supported and correctly configured, but it is not a universal guarantee of a particular recovery time.
Single link loss
If alternate paths remain available and the routing design recognizes the failure, traffic may be redirected through surviving paths. Check both route changes and actual forwarding.
Spine or peer loss
Routes or next hops associated with the failed device may be removed. Surviving paths carry the remaining demand, potentially increasing utilization and exposing oversubscription.
Policy or route withdrawal
A prefix may disappear even though physical links and BGP sessions remain up. Investigate advertisements, filters, next-hop resolution and the routing table.
Session instability
Repeated BGP resets can cause route churn and traffic disruption. Check interface errors, reachability, timers, logs and control-plane resource conditions.
Resilience must be tested under load. Redundant links are useful only if surviving paths have sufficient capacity and forwarding converges as intended. Test representative link and device failures, and measure packet loss, recovery time and post-failure utilization.
Work from session state to packet forwarding
When a prefix is missing or traffic takes an unexpected path, avoid starting with a broad configuration change. Follow the route through the control plane and forwarding plane, checking each stage before moving to the next.
Verify the session. Confirm the intended BGP neighbor is Established, and inspect logs, reset reasons, interface status and reachability between peer addresses.
Check received routes. Confirm the expected prefix is being advertised by the peer and received locally. Use the platform's received-route or neighbor-route commands where supported.
Inspect policy. Review prefix lists, route maps, import/export policies, community handling, maximum-prefix controls and any policy that could reject or modify the route.
Inspect best path and next hop. Identify why a candidate route was selected or rejected. Confirm that the selected next hop is resolvable and reachable.
Verify multipath installation. If several paths are expected, check the routing table for multiple installed next hops and confirm that multipath settings and platform capabilities permit them.
Validate forwarding. Inspect the forwarding table, interface counters, flow telemetry and packet path. For VXLAN EVPN, also verify underlay reachability, relevant EVPN routes, VTEP reachability and tenant forwarding state.
Example diagnostic commands
These are illustrative Cisco IOS/IOS XE-style commands. Exact syntax and available detail vary by operating system and platform; use the equivalent commands for your switches.
show ip bgp summary show ip bgp show ip route show ip cef show running-config | section router bgp
Start with the neighbor summary, then inspect the target prefix and its candidate paths. Check the routing table to determine which route was selected, and inspect the forwarding table to verify the next hop used for packet forwarding. Review the BGP configuration and policies when the observed result differs from the intended design.
- The relevant BGP neighbor is Established.
- The expected prefix is received and passes import policy.
- The selected route has a valid, reachable next hop.
- Expected ECMP next hops are installed when multipath is intended.
- Export policy allows the required route to reach downstream peers.
- Forwarding state and traffic telemetry agree with the expected path.
- Failure tests confirm acceptable convergence and surviving capacity.
Continue with the BGP Learning Centre
Explore route propagation, best-path selection and fault investigation through the Network Insight BGP Learning Centre. It complements this architecture section with a practical, interactive way to examine BGP behaviour.
Open the BGP Learning Centre: Explore. Trace. Diagnose. →Explore BGP in a Leaf-Spine Fabric
Select a router, inspect the illustrative BGP sessions, trace how a prefix is learned, and simulate a path failure. See how routing information and forwarding paths work together in a data center network.
Spine 1
A spine switch provides fabric transit. Depending on the design, it may establish eBGP sessions with leaf switches and exchange underlay reachability.
Engineering insight
BGP distributes reachability information. The forwarding plane uses the installed routes to move packets; a BGP session being established does not alone prove that an application path is working.
Underlay reachability
The underlay provides IP reachability between fabric devices and their loopbacks or next hops. eBGP is often used between leaf and spine switches in routed fabrics, though designs vary.
Route selection and forwarding
BGP applies policy and path-selection rules to choose eligible routes. ECMP may install multiple equal-cost next hops when supported by the routing design and platform.
Educational model only. This topology and its route progression are illustrative, not a live BGP implementation. Real session counts, ASNs, route attributes, ECMP eligibility, convergence and forwarding behavior depend on the configured network and platform.
Network Visibility, Traffic Intelligence and BGP
You cannot reliably protect a network if you cannot establish what is happening within it. Flow records, packet captures, encrypted-traffic metadata and security logs reveal different aspects of network behaviour and provide evidence for troubleshooting and threat detection.
This section connects traffic analysis with BGP route visibility, helping engineers investigate unexpected communication patterns, routing changes and reachability problems using evidence from multiple layers.
Network Visibility: From Traffic Flows to BGP Intelligence
Security controls determine what should be allowed. Observability helps engineers establish what is actually happening. By correlating flow records, packet-level evidence, encrypted traffic metadata and routing information, network teams can identify suspicious behaviour, investigate connectivity failures and understand how an incident moves across the network.
A firewall log may show that a connection was permitted, but it rarely explains the entire conversation. A packet capture may reveal a protocol exchange, yet only represent a short interval or one observation point. Flow telemetry can expose communication patterns across a longer period, while routing telemetry can explain which paths were available and why traffic may have taken an unexpected route.
5.1 — Choose the right observation method
NetFlow and IPFIX
Summarise conversations using fields such as source and destination addresses, ports, protocol, packet counts, byte counts and timestamps. IPFIX provides an extensible framework for exporting structured flow information.
sFlow
Uses statistical sampling of packets and interface counters to provide scalable visibility into traffic patterns, particularly in large switched and routed networks. Sampling can miss individual events.
SPAN and packet capture
SPAN mirrors selected switch traffic to an observation port. Packet captures can reveal protocol headers, retransmissions, handshakes and other details that summaries cannot show. Mirror oversubscription can cause packet loss.
Metadata and behaviour
Connection timing, volume, direction, destination reputation, DNS context where available, TLS handshake metadata and endpoint telemetry can help detect anomalies without assuming that encrypted payloads are readable.
| Method | Best used for | Important limitation |
|---|---|---|
| NetFlow / IPFIX | Traffic accounting, unusual communication patterns and historical investigation | Typically lacks full packet payload and detailed packet-by-packet sequence |
| sFlow | High-volume traffic trends and scalable network sampling | Sampling may omit short-lived or low-volume activity |
| SPAN / packet capture | Protocol troubleshooting and detailed session analysis | Limited observation point, capture duration and possible mirror-port loss |
| Endpoint and security logs | Process attribution, identity, host events and security outcomes | Coverage and reliability depend on instrumentation and log integrity |
| BGP and routing telemetry | Route changes, path selection, reachability and control-plane anomalies | A valid route does not prove that the application or forwarding path works |
5.2 — Build an evidence chain
Consider a server that suddenly transfers much more data than usual. A single bandwidth graph might identify the spike, but it cannot establish whether the cause is a legitimate backup, a compromised service or an unexpected change in traffic routing. A structured investigation combines several sources.
First, establish when the anomaly began. Next, identify the communicating endpoints, destination ports, volume and duration. Compare the behaviour against the normal baseline. Then inspect available DNS, endpoint, firewall and packet evidence. Finally, check whether a route change, next-hop failure or policy update coincided with the event.
Interactive investigation explorer
Select an observation to see what it can establish and what should be checked next.
Compare source and destination addresses, ports, protocol, bytes, packets and timestamps. Identify unusual destinations, repeated connections or a sudden change in traffic volume. Then correlate the flow with endpoint and security logs.
5.3 — Encrypted traffic: visibility without payload inspection
Encryption protects confidentiality, but it does not make every aspect of a connection invisible. Depending on the protocol, configuration and observation point, engineers may still have access to connection duration, byte counts, packet sizes, timing, DNS information, destination addresses and some handshake metadata.
Behavioural baselines
Detect changes in connection frequency, data volume, session duration and communication direction compared with a service's established pattern.
Identity correlation
Combine network observations with endpoint, workload and authentication records to associate a connection with a device, user or application where the evidence supports it.
Protocol-aware analysis
Use available TLS, DNS and transport metadata carefully. Encrypted Client Hello and encrypted DNS can limit visibility into some fields, while QUIC changes what traditional passive inspection can observe.
5.4 — BGP visibility: understand the route behind the traffic
Border Gateway Protocol (BGP) exchanges reachability information between autonomous systems and distributes routes within an autonomous system through iBGP. Its control plane influences the paths that IP packets may follow. For network security and troubleshooting, this means traffic anomalies should sometimes be investigated alongside BGP updates, routing-table changes, next-hop resolution and forwarding state.
Three autonomous systems, one destination prefix
Illustrative topology for correlating route propagation with observed traffic.
Example destination: 10.10.10.0/24
The values above are illustrative. They do not describe a live network or establish that this route is selected, reachable or safe.
What should engineers observe?
- UPDATE activity: identify announcements and withdrawals, including the prefix affected and the time of the change.
- AS_PATH changes: investigate unexpected path changes, unusual origins and route attributes that may affect selection.
- Best-path selection: compare competing routes, local preference, AS_PATH length and other attributes in the context of the vendor's decision process and policy.
- Next-hop resolution: verify that the selected next hop is reachable through the relevant routing information.
- Forwarding state: confirm that the selected route is programmed into the forwarding information base (FIB) and that traffic can actually traverse the path.
- Session health: correlate BGP neighbour state transitions, resets, timers and logs with the time the traffic anomaly occurred.
5.5 — BGP and traffic telemetry must be correlated
A route appearing in the BGP table is not the same as a successful application connection. BGP provides control-plane reachability information, while the forwarding plane handles packets. An investigation must distinguish a route that has been received from one that has been selected, installed and successfully used.
| Observation | Possible interpretation | Next validation step |
|---|---|---|
| Traffic volume drops suddenly | Application failure, link problem, filtering or a route change | Correlate interface counters, flow records, route events and endpoint health |
| AS_PATH changes unexpectedly | Legitimate policy change, upstream change or suspicious announcement | Check the source peer, prefix origin, route policy and trusted routing evidence |
| BGP route exists, but traffic fails | Unresolved next hop, FIB inconsistency, ACL, MTU issue or downstream failure | Check route selection, next-hop recursion, FIB, interface state and packet path |
| Repeated BGP session resets | Transport instability, peer configuration, resource pressure or timer expiry | Inspect neighbour logs, TCP connectivity, interface errors and control-plane load |
| Unexpected outbound traffic | New workload behaviour, misconfiguration or potentially malicious activity | Correlate destination, workload identity, DNS, security events and route history |
5.6 — Operational requirements for reliable visibility
Observability systems are only as useful as the quality and consistency of their data. Time synchronisation is essential when correlating events across routers, firewalls, switches, hosts and collectors. Retention policies should preserve enough history to establish a baseline and investigate delayed reports. Telemetry collectors should be monitored for gaps, overload and export failures.
- Synchronise timestamps across network devices, endpoints and telemetry platforms.
- Document flow sampling rates, SPAN limitations and which interfaces are actually observed.
- Monitor telemetry loss, collector health, storage capacity and data-retention windows.
- Restrict access to sensitive flow records, packet captures and routing logs.
- Protect management and telemetry channels against unauthorised access or tampering.
- Establish normal traffic and routing baselines before treating deviations as incidents.
Investigate the Data Center with Network Telemetry
Explore how flow records, interface counters, packet captures and BGP telemetry reveal different parts of a network problem. Select a diagnostic view to correlate traffic behavior with routing and forwarding state.
Flow telemetry
Flow records summarize conversations between endpoints. They help identify major talkers, traffic direction and volume, but do not normally contain every packet's full contents.
Engineering insight
No single telemetry source explains every fault. Correlating traffic records, interface counters, packet evidence and routing state can narrow the cause much faster.
| Signal | Example observation | Interpretation | Status |
|---|---|---|---|
| Flow records | Large app-to-database flow | Identify traffic source and volume | Useful |
| Interface counters | Output drops increasing | Investigate congestion or queue pressure | Check |
| Packet capture | Retransmissions observed | Correlate loss, delay and endpoint behavior | Check |
| BGP telemetry | Next-hop route changed | Verify route selection and forwarding state | Useful |
What the data tells you
Flow telemetry helps locate large or unexpected conversations. It is valuable for traffic engineering and capacity planning, but its detail depends on the exporter, sampling and collection interval.
Correlate before concluding
Compare timestamps and affected paths across telemetry sources. A counter increase, a packet symptom and a routing change are stronger evidence when they align in time and relate to the same flow.
Educational demonstration only. The evidence values are illustrative examples, not live measurements. Available telemetry, sampling, counter definitions, packet visibility and BGP monitoring capabilities vary by platform and deployment.
Incident Response, Threat Detection and Network Forensics
When suspicious activity is detected, the priority shifts from observation to controlled investigation and response. Engineers must validate the evidence, determine the scope of the incident, preserve relevant records and contain the threat without creating unnecessary operational disruption.
Explore the incident response lifecycle, practical forensic techniques, containment decisions and BGP incident investigation, from the first alert through recovery and lessons learned.
Incident Response, Threat Detection and Network Forensics
Detecting suspicious activity is only the beginning. Effective incident response requires engineers to validate the alert, establish the scope of the incident, preserve evidence, contain the threat and restore services safely. Network telemetry, endpoint records, security controls and routing information provide complementary evidence that helps teams understand what happened and determine the appropriate response.
A security alert does not automatically establish that a system has been compromised. A sudden increase in outbound traffic, an unfamiliar destination, repeated authentication failures or an unexpected BGP route change may indicate a security incident, a configuration error or legitimate operational activity. The objective is to investigate systematically rather than act on an assumption.
6.1 — The incident response lifecycle
A practical response process moves from preparation through detection, analysis, containment, eradication and recovery. Lessons learned then feed back into monitoring, configuration standards and security controls. In a serious incident, these activities may overlap, and responders may need to revisit earlier stages as new evidence emerges.
Preparation
Define incident ownership, escalation paths, approved containment actions, logging requirements, backup procedures and communication channels before an incident occurs.
Detection and analysis
Validate the alert, establish a timeline, identify affected assets and correlate evidence from network, endpoint, identity and security systems.
Containment
Reduce the opportunity for further harm using proportionate, authorised controls. Consider dependencies and the operational consequences of isolating a device or blocking a communication path.
Eradication and recovery
Remove the confirmed cause, remediate affected systems, restore trusted configurations and verify that services are operating correctly before returning to normal operations.
6.2 — Detect and validate suspicious network activity
Detection systems compare observed activity against rules, threat intelligence, behavioural baselines and known indicators. The resulting alerts must be evaluated in context. An IP address associated with a threat feed, for example, is a useful investigation lead but may be outdated, shared by multiple services or unrelated to the activity under examination.
| Signal | Potential concern | Validation approach |
|---|---|---|
| Unusual outbound data volume | Possible data transfer or unexpected application behaviour | Check flow records, destination, workload identity, scheduled jobs and endpoint events. |
| Repeated authentication failures | Possible credential guessing or account misuse | Correlate identity logs, source addresses, successful logins and account activity. |
| Repeated DNS queries | Possible misconfiguration, automated activity or suspicious domain generation | Inspect query patterns, resolver logs, domain context and the originating endpoint. |
| Unexpected east-west traffic | Possible lateral movement or a newly introduced dependency | Compare with workload communication baselines, segmentation policy and process telemetry. |
| Unexpected BGP announcement | Possible route leak, misconfiguration or unauthorised route propagation | Verify the peer, prefix, origin, policy, route history and expected maintenance activity. |
| Repeated connection resets | Possible service instability, filtering or hostile connection activity | Correlate packet captures, firewall decisions, server logs and interface counters. |
6.3 — Interactive incident investigation
Select a scenario to explore the evidence an engineer should collect before choosing a response. These are illustrative investigation paths, not automated diagnoses.
Investigation workbench
Select a scenario to reveal its evidence requirements and recommended next steps.
Establish which host initiated the transfer, the destination, data volume, timing and expected application behaviour. Correlate flow records with endpoint activity, DNS context and available security logs before deciding whether containment is necessary.
6.4 — Network forensics and evidence preservation
Network forensics reconstructs relevant events from available evidence. Depending on the environment, that evidence may include packet captures, NetFlow or IPFIX records, DNS logs, firewall events, authentication records, endpoint telemetry, configuration changes and BGP update history.
Packet captures
Preserve the relevant capture interval and observation point. Record any capture filters, truncation settings, packet loss indicators and limitations affecting interpretation.
Centralised logs
Preserve original records where possible, document collection methods and correlate events using consistent timestamps and asset identifiers.
Configuration history
Compare approved and current firewall rules, ACLs, routing policies, interface configuration and relevant changes made before or during the incident.
6.5 — Containment: reduce risk without creating another outage
Containment aims to limit the impact of an incident while preserving enough operational capability to investigate and recover. The right action depends on confidence in the evidence, the criticality of the affected service, the likely threat and the available alternatives.
| Containment option | Potential benefit | Risk to consider |
|---|---|---|
| Isolate an endpoint | Limits some network communications from a suspected host | May interrupt critical services or remove access needed for investigation. |
| Apply a targeted firewall rule | Restricts a verified malicious flow or exposure | An overly broad rule may block legitimate dependencies. |
| Disable compromised credentials | Reduces the opportunity for further unauthorised access | Can interrupt automated workloads or legitimate administrative activity. |
| Change routing policy | May mitigate an established routing threat or isolate an affected path | Incorrect changes can cause route withdrawal, traffic blackholing or wider reachability loss. |
| Increase targeted monitoring | Improves evidence collection when immediate isolation is not justified | Monitoring alone may be insufficient when active harm is continuing. |
6.6 — BGP incidents: distinguish control-plane anomalies from forwarding failures
BGP-related incidents require particular care because an incorrect response can affect reachability well beyond a single router. Unexpected route advertisements, route leaks, session instability and policy errors may alter the paths available to traffic. However, a route change alone does not establish malicious intent.
In an interdomain routing incident, useful safeguards may include carefully maintained prefix filters, explicit import and export policies, maximum-prefix controls, appropriate peer authentication and RPKI Route Origin Validation where supported. ROV helps validate whether an origin AS is authorised to originate a prefix; it does not validate every aspect of the route or prevent every possible BGP attack.
Within an autonomous system, engineers should also check iBGP propagation, route-reflector behaviour, next-hop reachability, IGP dependencies and forwarding state. A route may be visible in one routing table but absent from another, or selected by BGP without being usable for forwarding.
6.7 — Recovery, verification and lessons learned
Recovery is more than removing a block or restarting a service. Before returning affected systems to normal operation, teams should verify that the original cause has been addressed, required security controls are working and relevant traffic is following expected paths.
- Confirm the affected systems, accounts, prefixes or services have been identified.
- Remove the confirmed cause and remediate the relevant configuration or vulnerability.
- Restore systems from trusted sources where required and verify their security posture.
- Validate routing, next-hop reachability, forwarding state and application connectivity.
- Maintain heightened monitoring for recurrence and unexpected changes.
- Document the timeline, decisions, evidence, impact, recovery steps and outstanding risks.
- Update detection rules, runbooks, access controls and training based on lessons learned.
6.8 — Measure incident response effectiveness
Response metrics help organisations identify gaps and improve operational readiness. They should be interpreted alongside incident severity, service impact and investigation quality rather than treated as isolated performance targets.
Mean time to detect
Measures the elapsed time between a defined incident starting point and detection. The starting point must be consistently defined for comparisons to be meaningful.
Mean time to contain
Measures how long it takes to implement an effective containment measure after the relevant response clock starts.
Recurrence and false positives
Review recurring incident causes, unnecessary escalations, detection coverage and whether mitigations actually reduce risk without unacceptable disruption.
What Happens When a Data Center Fabric Fails?
Simulate illustrative link and spine failures in a leaf-spine fabric. Follow the surviving paths, inspect the remaining connectivity and identify what engineers should verify before declaring the fabric resilient.
Normal operation
Both spine switches are available. Each leaf has two uplinks, allowing traffic to use multiple fabric paths when routing and forwarding are configured accordingly.
Engineering lesson
Redundancy is only useful when alternate paths are operational, routing converges correctly and the remaining links can carry the offered load.
| Check | Question | Why it matters | Status |
|---|---|---|---|
| Physical state | Are the links and devices operational? | Confirms the failure boundary | Baseline |
| Routing | Have paths been withdrawn or recalculated? | Control-plane state influences path selection | Baseline |
| Forwarding | Are alternate next hops installed? | Confirms traffic has a usable path | Baseline |
| Capacity | Can surviving links carry the load? | Prevents congestion after failover | Baseline |
Resilience is more than redundancy
Multiple links and switches create alternate paths, but they do not guarantee uninterrupted service. Detection time, routing convergence, forwarding updates, hashing and remaining capacity all affect the outcome.
Validate the recovery
After a failure, verify reachability and application behavior, inspect routing and forwarding state, compare interface counters and confirm that the surviving fabric is not overloaded.
Educational model only. This is a simplified two-spine, four-leaf fabric. The diagram illustrates connectivity and possible alternate paths; it does not calculate live routing convergence, ECMP hashing, throughput, packet loss or application availability. Actual behavior depends on topology, routing policy, hardware and configuration.
Network Insight · Architecture Briefing
Designing Data Center Networks for Scale, Visibility and Resilience
Data center topology is more than a diagram of switches and links. It determines how workloads communicate, how capacity scales, how routing adapts to change and how engineers isolate faults when the network behaves unexpectedly.
Six ideas to take away
From architectural foundations to operational troubleshooting, each design decision affects the next.
Choose the right architecture
Three-tier designs separate access, aggregation and core functions. Leaf-spine fabrics provide a more consistent path structure for many modern, east-west-heavy workloads.
Understand the fabric
Leaf switches connect endpoints; spine switches interconnect the leaves. Clos-style designs can scale horizontally, subject to port count, link speed and oversubscription choices.
Design for real traffic
North-south and east-west flows create different demands. Capacity planning must consider traffic concentration, ECMP distribution, burstiness and workload communication patterns.
Make routing part of the design
BGP can provide scalable underlay routing in a leaf-spine fabric. Route policy, next-hop reachability, ECMP and convergence behavior all influence the forwarding outcome.
Make the network observable
Flow telemetry, interface counters, packet captures and routing telemetry reveal different parts of the story. Correlating them helps connect symptoms to causes.
Prove resilience under failure
Redundant links are only the beginning. Test failover, verify routing and forwarding state, check surviving capacity and confirm that affected applications recover.
A simplified fabric mental model
Think in layers, paths and failure domains.
A useful fabric design connects physical topology, routing behavior and application requirements. Each layer answers a different engineering question: what is connected, where can traffic travel and what happens when a component is lost?
The engineering takeaway
A resilient data center is not simply one with redundant links. It is one whose behavior can be understood, measured and tested.
Build around the workload, understand the trade-offs in the chosen topology, validate routing and forwarding, and exercise failure scenarios before they occur in production. The goal is not just connectivity—it is predictable performance and recoverable service.
- BGP Solutions and Insights - October 8, 2026
- DMVPN - May 20, 2023
- Computer Networking: Building a Strong Foundation for Success - April 7, 2023