data center design

Open Networking

ENGINEERING GUIDE · OPEN NETWORKING

Open Networking — From Disaggregated Hardware to Programmable Fabrics

Open networking is not a single product or protocol. It is an architectural approach that separates hardware, network software, control, telemetry and automation so infrastructure can become more programmable, interoperable and adaptable.

The important engineering shift is not simply replacing a proprietary platform. It is separating the components of the network so that hardware, software, control and operations can evolve independently.
01 · DISAGGREGATE

Separate Hardware

Treat switching hardware and network software as independently selectable components.

02 · PROGRAM

Expose Control

Use protocols, APIs and software interfaces to make network behaviour programmable.

03 · AUTOMATE

Operate at Fabric Scale

Move from box-by-box configuration toward repeatable desired-state workflows.

04 · OBSERVE

Measure Network State

Use telemetry and analytics to understand actual network behaviour continuously.

ENGINEERING MODEL
HARDWARE → NETWORK OS → CONTROL → AUTOMATION → TELEMETRY → ANALYTICS
ENGINEERING QUESTION

Can the network change without requiring every layer to be redesigned or every device to be configured independently?

Open networking moves intelligence toward software, automation and common interfaces.
White-Box Networking Open NOS SONiC FRRouting BGP EVPN VXLAN Leaf-Spine OpenConfig gNMI Automation Telemetry
Interactive Open Networking Architecture

Open Networking Architecture Explorer

Explore how open networking separates hardware, network software, control, telemetry and automation to create a more flexible, programmable and vendor-neutral infrastructure.
From hardware dependency to programmable fabric
Hardware
→
Network Software
→
Control
→
Operations
Disaggregation creates independence
DATA PLANE Forward packets
CONTROL PLANE Build routing state
MANAGEMENT Automate & observe
HW
White-Box Hardware
Disaggregated infrastructure

Open networking separates the physical switching platform from the software controlling it, allowing organizations greater freedom in hardware selection.

Layer Data plane
Principle Hardware / software separation
Benefit Vendor flexibility
Result Reduced lock-in
OPEN NETWORKING PRINCIPLE
Hardware becomes a replaceable platform rather than the sole source of network intelligence.

Why disaggregation matters

Traditional networking commonly couples hardware, operating system and features into a single vendor stack. Open networking separates these functions so each layer can evolve independently.

OPEN STANDARDS Common protocols and models allow different components to interoperate.
OPEN SOFTWARE Network operating systems and routing software can evolve independently of hardware.
OPEN APIs Automation systems can program and integrate the network.
OPEN TELEMETRY Network state becomes observable through streaming data and common models.
Open Networking · Interactive Use Case
How the Open Fabric Works
See how a modern leaf-spine fabric separates the routed underlay from the VXLAN overlay while BGP and EVPN provide the control plane.
L3 UNDERLAY
VXLAN OVERLAY
WORKLOADS
BGP EVPN CONTROL PLANE
VNI 10100 · VXLAN SEGMENT
SPINE 1
S1
Transit
SPINE 2
S2
Transit
LEAF 1
L1
VTEP
LEAF 2
L2
VTEP
VM-A
Host A
10.10.1.10
VM-B
Host B
10.10.1.20
VM-C
Host C
10.10.1.30
Open Fabric
Click a device to explore its role in the open fabric.
Underlay
IP + BGP
Control Plane
BGP EVPN
Overlay
VXLAN VNI 10100
Architecture
Leaf / Spine
BGP Underlay Routed IP connectivity between leaf and spine switches.
EVPN Control Plane BGP distributes MAC/IP and endpoint reachability information.
VXLAN Overlay VXLAN carries logical network segments across the IP fabric.
VTEP Leaf switches encapsulate and decapsulate VXLAN traffic.
01 · DEFINE · ARCHITECTURE

What Does “Open” Actually Mean?

Open networking begins by separating the components that traditionally arrive as one integrated platform: hardware, network operating system, control protocols and operational tooling. The engineering objective is reduced coupling, greater choice and programmable interfaces.

Engineering Question Can each layer evolve without forcing the entire network architecture to change with it?
01 · DEFINE · OPEN NETWORKING ARCHITECTURE

Open Networking Starts With Separation

Open networking begins by separating the functions that traditional networking often bundles together. Hardware provides forwarding capability, network software provides operating and routing functions, control protocols build network state, and automation and telemetry provide the operational feedback loop.

ARCHITECTURAL LAYERS
HARDWARE Switching silicon, ports, memory and physical forwarding resources.
NETWORK OS Operating system, routing software, interfaces and platform services.
CONTROL BGP, OSPF, IS-IS, EVPN and other mechanisms that establish network state.
OPERATIONS APIs, automation, configuration management, telemetry and analytics.
OPEN NETWORKING PRINCIPLES
Hardware Independence

Hardware can become a replaceable platform rather than defining the entire software stack.

Software Choice

Network operating systems and routing software can be selected according to operational requirements.

Open Interfaces

Standards, APIs and models allow systems to exchange state and participate in common workflows.

Operational Automation

Configuration and lifecycle operations can move from individual devices toward repeatable fabric-wide processes.

ENGINEERING PRINCIPLE

Open networking is not simply “using open source.” The architectural objective is to reduce unnecessary coupling between hardware, software, control and operations so that each layer can evolve without forcing the entire network to change with it.

DATA PLANE
Forward the packet

The forwarding infrastructure moves traffic according to the state programmed into the network.

CONTROL PLANE
Build network state

Routing protocols and control systems determine reachability and distribute network information.

MANAGEMENT PLANE
Operate the fabric

APIs, automation and telemetry provide the mechanisms for configuring, observing and validating the network.

02 · DISAGGREGATE · PLATFORM

Separate the Platform From the Network Software

Disaggregation separates the physical switching platform from the network software that controls it. White-box hardware, open network operating systems and routing software can therefore become independent architectural choices rather than a single fixed product.

Engineering Question Which parts of the network platform actually need to come from the same supplier?
ENGINEERING LAB · 01 · DEFINE

Open Networking Architecture Explorer

Explore the four architectural layers that turn a traditional integrated network platform into a more modular and programmable system. Select a layer or use the architecture tabs to inspect what becomes separated and why that separation matters.

OPEN NETWORKING STACK
● MODULAR
LAYER INSPECTOR
SELECTED
01 · PLATFORM

Hardware

Open networking separates the physical switching platform from the software that operates it. This creates greater freedom in choosing the underlying hardware platform, but also introduces responsibility for validating compatibility and lifecycle support.

Network software
Platform choice
Integration & support
White-box switching
OPENNESS SIGNAL

The hardware platform no longer has to dictate which network software is used.

02 · DISAGGREGATE · HARDWARE, NOS & CONTROL PLANE

Separate the Platform From the Network Software

Disaggregation separates the physical switching platform from the network operating system and the control protocols running above it. This creates greater architectural choice, but it also makes lifecycle, integration, compatibility and operational ownership more important.

THE DISAGGREGATED STACK
WHITE-BOX HARDWARE Switching ASICs, interfaces, buffers and physical resources. The platform provides forwarding capability without necessarily defining the complete software stack.
NETWORK OS Software responsible for operating the device, exposing interfaces and integrating routing and forwarding functions.
ROUTING SOFTWARE Protocol implementations such as BGP, OSPF or IS-IS build control-plane state and calculate reachability.
MANAGEMENT APIs, configuration models and automation systems provide the operational interface to the device or fabric.
CONTROL-PLANE EXAMPLES
BGP

Exchanges reachability information between routing peers and is widely used for data-centre and service-provider fabrics.

OSPF

Link-state routing protocol commonly used to build internal IP reachability within an autonomous system.

IS-IS

Link-state protocol used extensively in large-scale provider and data-centre network designs.

EVPN

A BGP-based control-plane technology commonly used to distribute MAC and IP reachability information for modern overlays.

01 · HARDWARE
Forwarding Resources

ASICs and interfaces determine what the platform can physically forward at line rate.

02 · NOS
Operating Environment

The network OS provides the software environment that connects the platform to routing and operational functions.

03 · CONTROL
Network Intelligence

Routing protocols exchange information and construct the state used to make forwarding decisions.

04 · OPERATE
Automation Interface

APIs, models and automation turn individual device configuration into repeatable operational workflows.

ENGINEERING REALITY

Disaggregation increases choice, but choice introduces responsibility. Hardware compatibility, NOS support, routing features, lifecycle management, observability and operational skills must all be considered as part of the architecture.

ENGINEERING LAB · 02 · DISAGGREGATE

Disaggregation Decision Lab

Build an open networking platform by selecting the hardware, network operating system and control model. The objective is not simply to choose “open” components — it is to understand the engineering trade-offs created when the layers are separated.

PLATFORM BUILDER
MODULARITY 82%
01 · Hardware Choose platform
02 · Network OS Choose software
03 · Control Choose model
ARCHITECTURE RESULT
CONFIGURATION READY
Selected Architecture
Hardware White-Box Switching Platform
Network Operating System Open Network OS
Control Open Standards & APIs
Operations Automation + Telemetry
Engineering Assessment
Platform choice
88
Software freedom
86
Integration effort
42
Vendor coupling
24
High platform and software choice
More integration responsibility
ENGINEERING DECISION

Disaggregation increases architectural choice, but the responsibility for validating hardware, software, interoperability and lifecycle support also increases.

03 · FABRIC · BGP EVPN

Build the Fabric With a Routed Underlay

Once the hardware and software layers are separated, the network can be designed as a programmable fabric. A routed IP underlay provides predictable transport while BGP EVPN and VXLAN can provide the control and overlay mechanisms required for scalable workload connectivity.

Engineering Question Can the physical fabric remain simple while the overlay provides the required logical connectivity?
03 · FABRIC · LEAF-SPINE & BGP EVPN

Build the Fabric With a Routed Underlay and Programmable Overlay

Open networking becomes especially powerful when disaggregated components are assembled into a fabric. A routed leaf-spine underlay provides predictable IP connectivity, while BGP and EVPN can distribute the control-plane information required for scalable workload and overlay connectivity.

FABRIC ARCHITECTURE
SPINE
High-speed routed transit between leaf switches. The spine provides fabric connectivity rather than directly hosting workloads.
LEAF
Connects servers, appliances and external networks while providing the edge of the fabric.
UNDERLAY
Routed IP transport between fabric devices. BGP, OSPF or IS-IS can provide the reachability required by the design.
OVERLAY
VXLAN can provide virtual network segmentation while EVPN distributes endpoint reachability through the control plane.
SERVER
→
LEAF
→
SPINE
→
LEAF
→
SERVER
CONTROL-PLANE MODEL
IP UNDERLAY

Provides simple routed reachability between the devices that make up the fabric.

BGP

Can provide scalable routing between leaf and spine devices and establish the fabric's IP reachability.

EVPN

Uses BGP to distribute MAC and IP reachability information for modern Ethernet VPN designs.

VXLAN

Encapsulates tenant traffic across the IP underlay and identifies virtual networks using a VNI.

01 · UNDERLAY
Transport Layer

The underlay answers a fundamental question: can the fabric devices reach one another reliably?

02 · CONTROL
Reachability State

Routing protocols exchange information so devices can calculate paths through the fabric.

03 · OVERLAY
Virtual Networks

VXLAN separates logical networks from the physical topology and provides scalable segmentation.

04 · EVPN
Endpoint Knowledge

EVPN distributes endpoint reachability through the control plane, reducing reliance on traditional flood-and-learn behaviour.

ENGINEERING PRINCIPLE

Keep the underlay simple and predictable. Put the complexity required for tenant connectivity, segmentation and endpoint distribution into the overlay and its control plane. This separation makes the fabric easier to reason about, troubleshoot and automate.

ENGINEERING LAB · 03 · FABRIC

Open Fabric Path Lab

Follow a packet through a modern open networking fabric. Select each stage to expose the engineering evidence behind the underlay, BGP control plane, EVPN and VXLAN overlay.

Stage 1 of 6
Fabric Path
Host A → Host B
Packet context: Host A · 10.10.10.21 → Host B · 10.10.20.31 · VNI 10100
Underlay
Keep Transport Simple
The routed fabric provides IP reachability between loopbacks and VTEPs. The underlay should not need to understand tenant networks.
Overlay
Move Tenant State Into EVPN
EVPN distributes endpoint and MAC/IP reachability information through the control plane rather than relying only on data-plane flooding.
Live Trace
Path Evidence
00:00 Path investigation initialised
00:01 Ingress leaf selected
04 · PROGRAM · AUTOMATION

Turn Network Operations Into a Programmable System

Open interfaces become valuable when they can be used consistently by automation. Models such as OpenConfig, interfaces such as gNMI, and tools such as Ansible and Terraform allow network intent to become repeatable change rather than a sequence of manual device commands.

Engineering Question Can the same intended network state be deployed repeatedly and then verified automatically?
04 · PROGRAM · APIs, AUTOMATION & DESIRED STATE

Turn Network Operations Into a Programmable System

Once hardware and network software are separated, the next engineering challenge is operating the infrastructure consistently. APIs, configuration models and automation allow engineers to describe, deploy and validate network state without relying on repetitive device-by-device configuration.

FROM INTENT TO NETWORK STATE
01
INTENT

Define what the network should provide.

02
MODEL

Represent configuration using structured data and models.

03
AUTOMATE

Apply repeatable changes across the infrastructure.

04
DEPLOY

Push validated state into network devices and services.

05
VERIFY

Compare the resulting state with the intended state.

06
REPEAT

Treat network change as an operational lifecycle.

OPEN PROGRAMMABILITY TOOLBOX
OpenConfig

Open, vendor-neutral data models commonly used to represent network configuration and operational state.

gNMI

A management interface commonly used to retrieve and modify structured configuration and operational data.

Ansible

Automation and configuration-management tooling suited to repeatable network changes and operational workflows.

Terraform

Infrastructure-as-code tooling commonly used to provision and manage infrastructure through declarative configuration.

DAY 0 → DAY 1 → DAY 2
DAY 0 · BOOTSTRAP Establish the initial platform, management connectivity, credentials and baseline configuration.
DAY 1 · DEPLOY Introduce production services such as routing, VLANs, VRFs, interfaces and fabric configuration.
DAY 2 · OPERATE Continue monitoring, changing, validating and remediating the network throughout its lifecycle.
ENGINEERING PRINCIPLE

Automation should not simply make configuration faster. The stronger objective is repeatability: the same intent should produce a predictable result, and the resulting network state should be measurable and verifiable.

API
Expose the Network

APIs provide a programmable interface instead of forcing every operation through manual CLI interaction.

MODEL
Structure the State

Models provide a consistent representation of configuration and operational information.

AUTOMATION
Scale the Change

Automation turns individual operations into repeatable workflows that can be applied across the fabric.

ENGINEERING LAB · 04 · PROGRAM

Desired-State Automation Lab

Define the network you want, generate the intended configuration, deploy the change, then verify that the fabric actually reached the desired state.

Intent not deployed
Define Intent
Desired state
Fabric Protocol
Overlay
Automation
Validation
Automation Pipeline
Awaiting plan
01
Intent
Defined
02
Model
Pending
03
Deploy
Pending
04
Observe
Pending
05
Verify
Pending
Generated Plan
Configuration Intent
# intent fabric.protocol = BGP overlay = EVPN/VXLAN automation = model-driven validation = telemetry
Configuration Diff
Proposed Change
+ enable bgp fabric + enable evpn overlay + configure telemetry
PENDING
Config State
PENDING
Control Plane
PENDING
Data Plane
PENDING
Telemetry
05 · OBSERVE · TELEMETRY

You Cannot Automate What You Cannot Observe

Programmable infrastructure needs equally strong visibility. Interface state, routing information, traffic flows and system health provide the evidence required to understand what the network is actually doing. Telemetry turns network state into operational data.

Engineering Question Can the network prove what changed, what is happening now and whether the resulting state is healthy?
05 · OBSERVE · TELEMETRY & NETWORK VISIBILITY

Expose the State of the Network

Programmability is only useful when engineers can also observe the resulting network state. Open networking therefore depends on telemetry that exposes interfaces, routing state, traffic behaviour and system health so that automation can be validated against what the network is actually doing.

NETWORK TELEMETRY SIGNALS
INTERFACES

Counters, errors, link state and utilisation reveal the health of physical and logical interfaces.

ROUTING STATE

BGP, OSPF, IS-IS and EVPN state shows whether the control plane is building the expected reachability.

TRAFFIC FLOWS

Flow records and sampled telemetry provide visibility into communication patterns and traffic volumes.

SYSTEM HEALTH

CPU, memory, buffers, process state and platform health provide additional operational context.

OBSERVABILITY PIPELINE
NETWORK STATE Devices generate operational and protocol information.
TELEMETRY State is exposed through streaming telemetry, APIs or sampled visibility mechanisms.
COLLECT Data is transported into monitoring, analytics or observability systems.
CORRELATE Signals are combined to understand relationships and anomalies.
ACT Engineers or automation can investigate, validate and respond.
STREAMING TELEMETRY
Continuous State

Streaming telemetry can deliver operational state continuously rather than relying entirely on periodic polling.

OPEN MODELS
Structured Data

OpenConfig and interfaces such as gNMI help expose structured configuration and operational information.

FLOW VISIBILITY
Traffic Behaviour

Technologies such as sFlow provide sampled visibility into traffic patterns without requiring full packet capture.

ENGINEERING PRINCIPLE

Automation without telemetry creates blind change. Telemetry without automation creates visibility without scale. The stronger architecture connects both: change the network, observe the result, compare it with the intended state, and investigate any deviation.

ENGINEERING LAB · 05 · OBSERVE

Telemetry Investigation Console

Select the signals you would use to investigate a fabric event, correlate them against the timeline, and build enough evidence to distinguish a control-plane problem from a forwarding or workload problem.

Select evidence
Telemetry Signals
Select multiple
Correlation View
No correlation yet
Signals Selected
0 / 5
Evidence Quality
LOW
Likely Domain
UNKNOWN
Confidence
0%
10:42:01 Baseline stable Leaf and spine interfaces operating normally.
10:42:17 EVPN state changed Remote endpoint reachability changed on Leaf-02.
10:42:18 Traffic drop observed Flow telemetry reports loss toward remote workload.
10:42:21 Interface remains up No physical link failure detected.
10:42:24 System health normal No resource saturation detected.
Investigation Finding
0 evidence sources
Select telemetry sources, then correlate them with the event timeline. The objective is to identify which layer changed first.
06 · OVERLAY · VXLAN

Separate Logical Networks From the Physical Fabric

Modern fabrics often carry many logical networks across the same physical infrastructure. VXLAN provides the encapsulation mechanism, while EVPN can provide the control-plane information required to distribute endpoint and reachability state across the fabric.

Engineering Question Can multiple logical networks share the same physical fabric without coupling tenant connectivity to the physical topology?
06 · OVERLAY · VXLAN & NETWORK VIRTUALISATION

Separate Logical Networks From the Physical Fabric

Network virtualisation allows logical connectivity to span a physical fabric without requiring every physical link to represent a separate tenant or broadcast domain. VXLAN provides the encapsulation mechanism, while EVPN can provide the control-plane information needed to discover and distribute endpoint reachability.

UNDERLAY → OVERLAY MODEL
PHYSICAL FABRIC

Switches, links and interfaces provide the physical transport between fabric nodes.

IP UNDERLAY

Routed connectivity provides the transport path between VTEPs across the fabric.

VXLAN OVERLAY

Tenant Ethernet frames can be encapsulated inside IP packets for transport across the underlay.

EVPN CONTROL PLANE

BGP EVPN distributes endpoint information so VTEPs can make informed forwarding decisions.

VTEP A Workload / VLAN
VNI 10100
VTEP B Workload / VLAN
CONTROL VS DATA PLANE
FUNCTION ROLE
UNDERLAY Provides IP reachability between the physical fabric devices.
VTEP Encapsulates and decapsulates VXLAN traffic at the edge of the overlay.
VNI Identifies a VXLAN virtual network and separates logical connectivity within the overlay.
EVPN Distributes MAC and IP reachability information through the control plane.
VXLAN Carries the overlay traffic across the IP underlay using encapsulation.
01 · VNI
Logical Identifier

VXLAN uses a 24-bit VNI, providing a large identifier space for logical networks.

02 · VTEP
Overlay Edge

VTEPs connect local workloads to the VXLAN overlay and perform encapsulation and decapsulation.

03 · EVPN
Reachability

EVPN provides control-plane distribution of endpoint information rather than relying entirely on flood-and-learn behaviour.

04 · MOBILITY
Physical Independence

Logical networks can span different physical locations and paths without changing the underlying fabric topology.

ENGINEERING PRINCIPLE

The underlay should answer “How do I reach the VTEP?” while the overlay answers “Which logical network and endpoint am I trying to reach?” Keeping those responsibilities separate makes large fabrics easier to scale, troubleshoot and automate.

ENGINEERING LAB · 06 · OVERLAY

VXLAN Overlay Packet Trace

Trace an Ethernet frame across a shared IP fabric and see where the logical network, VNI, VTEPs and VXLAN encapsulation fit into the forwarding path.

Ready to trace
Overlay Parameters
Packet context
H
Host A
10.10.10.21
Ethernet frame
V
VTEP A
Leaf-01
Encapsulation
I
IP Fabric
Underlay
ECMP
V
VTEP B
Leaf-02
Decapsulation
H
Host B
10.10.20.31
Ethernet frame
Current Packet Representation
ETH SRC=H-A DST=H-B
VNI
Logical Network Identity
The VNI identifies the logical VXLAN segment carried across the shared physical fabric. Different VNIs can use the same underlay.
VTEP
Where Encapsulation Happens
The VTEP sits at the edge of the overlay. It encapsulates traffic entering VXLAN and decapsulates traffic leaving the overlay.
Underlay
Transport Does Not Need Tenant State
The IP fabric forwards toward the remote VTEP. The physical transport can remain largely independent from the logical tenant network.
07 · OPERATE · WORKLOAD NETWORKING

Extend Open Networking Into the Host

The network does not end at the top-of-rack switch. Linux networking, virtual switching and container networking extend connectivity into servers and workloads. This creates another programmable layer that must be automated, observed and secured alongside the physical fabric.

Engineering Question Where does the network boundary actually end when applications run inside virtualised or containerised workloads?
07 · OPERATE · LINUX, VIRTUAL SWITCHING & CONTAINERS

Extend Open Networking Into the Workload Layer

Open networking does not stop at the physical switch. Linux networking, virtual switches and container networking extend the same programmable principles into the systems and workloads connected to the fabric. The result is a network model that spans physical interfaces, virtual interfaces, workloads and applications.

SOFTWARE NETWORKING STACK
PHYSICAL NIC Provides host connectivity into the physical network and participates in the server's forwarding path.
LINUX NETWORKING Interfaces, routing, namespaces, bridges, filtering and other kernel networking functions provide the host network layer.
VIRTUAL SWITCH Software such as Open vSwitch can provide programmable switching between physical interfaces, virtual interfaces and workloads.
WORKLOAD NETWORK Containers and virtual workloads consume the networking services provided by the host and its virtual switching layer.
OPEN SOFTWARE COMPONENTS
Linux Networking

Provides interfaces, routing, namespaces, bridges, firewalling and other host-level networking capabilities.

Open vSwitch

A software switch designed for programmable virtual networking and integration with physical and virtual infrastructure.

Network Namespaces

Provide isolated network views that can separate interfaces, routing tables and network resources.

Container Networking

Connects containers to host and external networks through bridges, virtual interfaces or additional networking systems.

01
PHYSICAL

Server connects to the fabric.

02
HOST

Linux provides the local network stack.

03
VIRTUAL

Software switching connects logical interfaces.

04
WORKLOAD

Containers or VMs consume network connectivity.

05
FABRIC

Traffic exits through the physical network.

LINUX
Programmable Host

Linux provides a powerful networking environment that can be automated alongside the physical fabric.

OPEN VSWITCH
Virtual Forwarding

OVS provides software-based switching between virtual and physical network interfaces.

CONTAINERS
Workload Connectivity

Containers consume network connectivity through host and container networking mechanisms rather than directly owning the physical fabric.

ENGINEERING PRINCIPLE

The physical fabric and workload network should be treated as connected layers rather than one giant control plane. The fabric provides transport, while Linux, virtual switching and workload networking provide connectivity closer to the application.

ENGINEERING LAB · 07 · OPERATE

Host Networking Stack Inspector

Follow connectivity from an application workload through the Linux network namespace, virtual switch, physical NIC and finally into the upstream fabric.

Select a host layer
Host Networking Path
Click any layer
Compute Host · worker-07
Linux
External Network Fabric
BGP / EVPN
HOST VTEP 192.0.2.11
↔
FABRIC Leaf / Spine
Linux
Host Networking Is Its Own Layer
Interfaces, namespaces, routing tables and policy exist below the application but above the physical fabric.
Virtual Switching
Forwarding Can Move Into Software
Open vSwitch and other virtual switching mechanisms connect workload interfaces to host uplinks and can apply forwarding policy locally.
Engineering Boundary
Do Not Collapse Every Layer
The physical fabric transports traffic while host networking provides connectivity closer to the workload. Automation and observability need visibility across both.
08 · VALIDATE · CLOSED LOOP

The Network Becomes a Continuous Feedback System

The final step is connecting intent, automation and telemetry into a controlled feedback loop. The network should not simply accept changes; it should provide evidence that the resulting state matches the design, detect meaningful drift and support controlled remediation.

Engineering Question Can the network detect when reality has diverged from the intended state and safely guide it back into compliance?
08 · VALIDATE · CLOSED-LOOP OPEN NETWORKING

Turn Open Networking Into a Closed-Loop System

The real advantage of an open network appears when programmability, automation and telemetry operate as one system. Desired state defines the intended configuration, automation applies it, the fabric delivers connectivity, telemetry exposes the resulting state and validation determines whether the network still matches the design.

01 · INTENT
DESIRED STATE

Define what the network should look like.

02 · CHANGE
AUTOMATION

Translate intent into repeatable configuration.

03 · FORWARD
NETWORK FABRIC

The physical and virtual network delivers connectivity.

04 · OBSERVE
TELEMETRY

Collect evidence about the resulting network state.

05 · DETECT
DRIFT

Compare observed state with intended state.

06 · ACT
REMEDIATE

Correct validated deviations and verify again.

CLOSED-LOOP OPERATING CYCLE
01
Define Intent Specify topology, policy, connectivity and operational requirements.
02
Automate Change Use models, APIs and automation to produce consistent changes.
03
Observe State Gather configuration, routing, interface and traffic evidence.
04
Compare Determine whether observed state matches the intended design.
05
Validate & Remediate Correct confirmed drift and return the system to the desired state.
ENGINEERING SAFETY GATES
01 · INTENT
Define Before Change

Automation should operate against an explicit desired state rather than an undocumented collection of commands.

02 · EVIDENCE
Observe Before Acting

Telemetry and state data provide the evidence required to determine whether a deviation is real.

03 · CONTROL
Validate Before Remediation

Not every difference should trigger an automatic change. Context and policy must determine the response.

04 · RECOVERY
Verify After Change

A successful configuration deployment is not the final result. The resulting network state must also be verified.

CHECK 01
Configuration State

Does the deployed configuration match the intended model?

CHECK 02
Control-Plane State

Are routing adjacencies and learned paths behaving as expected?

CHECK 03
Data-Plane State

Is traffic actually following the expected forwarding path?

CHECK 04
Operational State

Does telemetry confirm that the network remains healthy over time?

ENGINEERING PRINCIPLE

Open networking becomes strategically valuable when openness is combined with automation and evidence. The goal is not simply to make individual devices programmable, but to create a network that can be defined, changed, observed, validated and safely corrected as a system.

ENGINEERING LAB · 08 · VALIDATE

Closed-Loop Network Validation

Introduce configuration drift into a network, collect evidence, identify the affected state and safely drive the system back toward the intended design.

Network converged
Closed-Loop Cycle
STATE · CONVERGED
01
Desired State
Declared network intent and policy baseline
→
02
Deploy
Automation applies intended configuration
→
03
Observe
Telemetry exposes operational state
→
04
Detect Drift
Compare intended and observed state
→
05
Remediate
Controlled correction and verification
Network Evidence
Configuration State MATCH
Control Plane HEALTHY
Data Plane FORWARDING
Telemetry NORMAL
Evidence
Observe Before You Act
Telemetry provides evidence of what the network is actually doing. A desired-state system should not remediate solely because a configuration differs.
Safety Gate
Validate the Failure Domain
Compare configuration, control-plane, data-plane and operational evidence before deciding that automated remediation is appropriate.
Convergence
Verify After Remediation
Closed-loop automation is incomplete until the resulting state is checked and confirmed against the original intent.
Open Networking · Interactive Use Case
Automation & Observability Lab
Follow the operational lifecycle from desired state and Day 0 provisioning through Day 1/2 services, telemetry, drift detection and remediation.
DESIRED STATE
AUTOMATION
NETWORK FABRIC
OBSERVABILITY / FEEDBACK
Desired State
Intent
Policy
Inventory
Devices
Source of Truth
Templates
Config Model
Reusable
Automation
Pipeline
Ansible / Terraform
Day 0
Provision
Base Fabric
Day 1 / 2
Services
VRF / EVPN
Leaf / Spine
Fabric
BGP / EVPN
Network State
Running State
Actual Config
Telemetry
gNMI / gRPC
Streaming
Analytics
Visibility
Health / KPIs
Drift Detection
Compare
Desired ≠ Actual
Remediation
Closed Loop
Restore State
Closed-Loop Networking
Modern open networking connects desired state, automation, network operation and telemetry into a continuous lifecycle.
Automation
Desired State
Network
Leaf / Spine
Telemetry
Streaming Telemetry
State
In Sync
● FABRIC OPERATING NORMALLY
Day 0 Provision the base fabric using repeatable infrastructure-as-code workflows.
Day 1 / 2 Automate services such as VRFs, EVPN, VXLAN and network policy.
Telemetry Continuously observe network state using streaming telemetry.
Closed Loop Detect drift and, where policy permits, restore the intended state.
```
☕
Support Network Insight
Help support independent engineering guides, interactive networking tools, and future learning labs.
Support Network Insight ```
Cisco ACI

ACI Cisco

Cisco ACI Components

Cisco Application Centric Infrastructure (ACI) represents a different way of approaching data centre networking. Rather than treating the network primarily as a collection of individually configured switches and interfaces, ACI uses application requirements and policy as the foundation for how connectivity, segmentation, and network behaviour are defined. This becomes particularly valuable as organizations operate physical servers, virtual machines, containers, and workloads across increasingly dynamic environments.

Cisco ACI brings the physical data centre fabric and software-defined policy model together through a centralized operational framework. The underlying network still provides the forwarding infrastructure, but administrators can define how applications and workloads should communicate without having to express every requirement as a series of device-by-device configurations. This separation between intent and implementation is central to the ACI model.

A major benefit is policy consistency. Instead of relying entirely on individual switch configurations, teams can define application relationships, network connectivity, and security requirements through policy. Those policies can then be applied across the fabric as workloads are deployed or changed, helping reduce configuration drift and making large environments easier to operate.

Security is another important part of the architecture. ACI supports segmentation and policy-based control between application tiers and workloads, allowing organizations to restrict communication according to defined requirements rather than relying solely on broad network boundaries. Integration with external security and operational platforms can further extend visibility and enforcement across the environment.

The model is also designed for scale. Modern data centres may contain large numbers of applications, virtual workloads, containers, and rapidly changing endpoints. A policy-driven fabric allows network operations to respond to those changes through automation and orchestration rather than requiring every network change to be handled manually. This can improve deployment consistency while reducing operational overhead.

ACI also provides a framework for connecting traditional physical infrastructure with virtualized and container-based workloads. This is important because most enterprise environments are not completely homogeneous: legacy applications, modern distributed applications, databases, virtualization platforms, and containerized services often coexist within the same broader architecture.

Hybrid and multi-cloud environments add another dimension. Organizations increasingly need consistent connectivity and policy across on-premises infrastructure and external cloud platforms. ACI can participate in these architectures through its integration and multi-site capabilities, helping organizations extend network and application policy across environments while recognizing that each cloud still has its own native networking and security model.

Cisco ACI is therefore best understood as a policy-driven data centre networking architecture, rather than simply a replacement for traditional switching. Its value comes from combining a scalable fabric with centralized policy, automation, segmentation, and application-aware operational models. For organizations managing complex data centre environments, this approach can provide a more consistent way to connect applications, enforce policy, and adapt the network as workloads evolve.

Cisco ACI • Interactive Architecture
ACI Fabric Architecture Explorer
Explore how APIC policy, the spine–leaf fabric and endpoint connectivity work together to create a policy-driven data-center network.
Learning objective: Understand how Cisco ACI separates policy and management from distributed forwarding — APIC defines intent while leaf and spine switches enforce and transport that policy.
MANAGEMENT / POLICY PLANE SPINE FABRIC LEAF / DATA-PLANE EDGE ENDPOINTS / WORKLOADS APIC CLUSTER Policy • Management • Orchestration SPINE 1 Fabric Transit SPINE 2 Fabric Transit LEAF 1 Endpoint Edge LEAF 2 Endpoint Edge LEAF 3 Endpoint Edge SERVER A EPG: WEB SERVER B EPG: APP SERVER C EPG: DB MULTI-PATH FABRIC / ECMP POLICY ENFORCEMENT AT THE LEAF EDGE
Fabric
Policy
Endpoint
Interactive Inspector
Cisco ACI Fabric
Fabric concept
Select a device or connection in the diagram to see how that component participates in the ACI architecture.
Selected component
Architecture overview
ACI concept
Policy-driven spine–leaf fabric
Why it matters
ACI separates centralized policy definition from distributed packet forwarding.

Highlights: Cisco ACI Components

The ACI Fabric

Cisco ACI is a software-defined networking (SDN) solution that integrates with software and hardware. With the ACI, we can create software policies and use hardware for forwarding, an efficient and highly scalable approach offering better performance. The hardware for ACI is based on the Cisco Nexus 9000 platform product line. The APIC centralized policy controller drives the software, which stores all configuration and statistical data.

–The Cisco Nexus Family–

To build the ACI underlay, you must exclusively use the Nexus 9000 family of switches. You can choose from modular Nexus 9500 switches or fixed 1U to 2U Nexus 9300 models. Specific models and line cards are dedicated to the spine function in ACI fabric; others can be used as leaves, and some can be used for both purposes. You can combine various leaf switches inside one fabric without any limitations.

a) Cisco ACI Fabric: Cisco ACI’s foundation lies in its fabric, which forms the backbone of the entire infrastructure. The ACI fabric comprises leaf switches, spine switches, and the application policy infrastructure controller (APIC). Each component ensures a scalable, agile, and resilient network.

b) Leaf Switches: Leaf switches serve as the access points for endpoints within the ACI fabric. They provide connectivity to servers, storage devices, and other network devices. With their high port density and advanced features, such as virtual port channels (vPCs) and fabric extenders (FEX), leaf switches enable efficient and flexible network designs.

c) Spine Switches: Spine switches serve as the core of the ACI fabric, providing high-bandwidth connectivity between the leaf switches. They use a non-blocking, multipath forwarding mechanism to ensure optimal traffic flow and eliminate bottlenecks. With their modular design and support for advanced protocols like Ethernet VPN (EVPN), spine switches offer scalability and resiliency.

d) Application Policy Infrastructure Controller (APIC): At the heart of Cisco ACI is the APIC, a centralized management and policy control plane. The APIC acts as a single control point, simplifying network operations and enabling policy-based automation. It provides a comprehensive view of the entire fabric, allowing administrators to define and enforce policies across the network.

e) Integration with Virtualization and Cloud Environments: Cisco ACI seamlessly integrates with virtualization platforms such as VMware vSphere and Microsoft Hyper-V and cloud environments like Amazon Web Services (AWS) and Microsoft Azure. This integration enables consistent policy enforcement and visibility across physical, virtual, and cloud infrastructures, enhancing agility and simplifying operations.

Cisco ACI — Policy‑Driven, Application‑Centric Networking

Cisco ACI (Application Centric Infrastructure) is Cisco’s flagship data‑center fabric built around policy‑driven automation, centralized control, and application‑aware networking. Instead of configuring individual switches, ACI abstracts the network into application policies, endpoint groups, and contracts. This creates a programmable, intent‑based fabric where the network automatically aligns with application requirements. Data analytics strengthens ACI by providing visibility into fabric health, policy behavior, endpoint movement, and application flows, ensuring that the fabric remains consistent, secure, and optimized.

Policy Model — Endpoint Groups, Contracts, and Intent

ACI’s core innovation is its policy model:

  • Endpoint Groups (EPGs) define logical groupings of workloads
  • Contracts define how EPGs communicate
  • Application Network Profiles (ANPs) map application tiers to policies
  • Tenants provide multi‑domain isolation

This model replaces traditional VLANs, ACLs, and manual segmentation with intent‑based policies. Analytics enhances this by identifying unused contracts, misaligned EPGs, shadow policies, and unexpected east‑west flows. With analytics, ACI’s policy model becomes adaptive and identity‑aware rather than static.

Fabric Architecture — Spine‑Leaf, VXLAN, and Distributed Control

ACI uses a spine‑leaf architecture with VXLAN overlays and a centralized policy controller (APIC). The fabric provides:

  • consistent forwarding across all leaf switches
  • distributed anycast gateways
  • scalable multi‑tenant segmentation
  • automated endpoint learning

Analytics correlates fabric telemetry—routing updates, VXLAN behavior, endpoint mobility, and link performance—to detect anomalies such as asymmetric routing, endpoint flapping, or misconfigured policies. This ensures the fabric remains stable and predictable even under dynamic workloads.

Security and Microsegmentation in ACI

ACI provides built‑in microsegmentation through EPGs and contracts. Security becomes application‑centric rather than IP‑centric. Analytics enhances ACI security by monitoring:

  • unusual endpoint movement
  • abnormal contract usage
  • unexpected east‑west communication
  • privilege escalation inside application tiers

This transforms ACI into a Zero Trust‑aligned data‑center fabric where segmentation is continuously validated and enforced.

Operational Visibility — Telemetry, Health Scores, and Assurance

ACI includes extensive telemetry and health scoring across:

  • fabric nodes
  • policies
  • endpoints
  • application flows

Analytics consumes this telemetry to provide deeper insights into performance, security, and policy alignment. Tools like ACI AppDynamics integration, Nexus Dashboard Insights, and fabric assurance engines use analytics to detect drift, highlight anomalies, and recommend remediation. ACI becomes a self‑optimizing fabric driven by real‑time intelligence.

ACI in Hybrid and Multi‑Cloud Architectures

ACI extends beyond the data center into cloud environments through ACI Anywhere and Cloud ACI. Policies can be pushed to AWS, Azure, and GCP, creating a unified segmentation and security model across hybrid environments. Analytics ensures consistency across on‑prem and cloud fabrics, detecting mismatches, drift, and cross‑cloud anomalies. This creates a seamless, policy‑driven hybrid architecture.

–ACI Architecture: Spine and Leaf–

To be used as ACI spines or leaves, Nexus 9000 switches must be equipped with powerful Cisco CloudScale ASICs manufactured using 16-nm technology. The following figure shows the Cisco ACI based on the Nexus 9000 series. Cisco Nexus 9300 and 9500 platform switches support Cisco ACI. As a result, organizations can use them as spines or leaves to utilize an automated, policy-based systems management approach fully. 

Cisco ACI Components
Diagram: Cisco ACI Components. Source is Cisco

**Hardware-based Underlay**

Server virtualization helped by decoupling workloads from the hardware, making the compute platform more scalable and agile. However, the server is not the main interconnection point for network traffic. So, we need to look at how we could virtualize the network infrastructure similarly to the agility gained from server virtualization.

**Mapping Network Endpoints**

This is carried out with software-defined networking and overlays that could map network endpoints and be spun up and down as needed without human intervention. In addition, the SDN architecture includes an SDN controller and an SDN network that enables an entirely new data center topology.

**Specialized Forwarding Chips**

In ACI, hardware-based underlay switching offers a significant advantage over software-only solutions due to specialized forwarding chips. Furthermore, thanks to Cisco’s ASIC development, ACI brings many advanced features, including security policy enforcement, microsegmentation, dynamic policy-based redirect (inserting external L4-L7 service devices into the data path), or detailed flow analytics—besides the vast performance and flexibility.

 

Cisco ACI • Fabric Mechanics

ACI Overlay & Underlay Lab

See how ACI builds the fabric: IS-IS provides the underlay, VXLAN carries the overlay, VTEPs connect endpoints, and COOP helps the fabric locate endpoints.
ACI FABRIC MECHANICS OVERLAY UNDERLAY ENDPOINTS VXLAN OVERLAY Leaf 1 VTEP Leaf 2 VTEP IS-IS UNDERLAY Spine Fabric transport Leaf 1 Underlay node Leaf 2 Underlay node COOP Endpoint database Server A Endpoint Server B Endpoint FABRIC TRANSPORT ATTACHMENT ATTACHMENT
IS-IS Underlay
VXLAN Overlay
COOP Control Plane
Endpoint Attachment
Selected Component
ACI Fabric Flow
Start here. ACI separates the physical transport network from the logical VXLAN overlay. Click a tab, node, or connection to explore how the pieces work together.
Technology
IS-IS + VXLAN + VTEP + COOP
Role
Build transport, carry overlay traffic, and locate endpoints.
Why it matters
This separation allows ACI to provide scalable fabric connectivity without making applications depend on the physical topology.

Cisco ACI Components

 Introduction to Leaf and Spine

The Cisco SDN ACI works with a Clos architecture, a fully meshed ACI network. Based on a spine leaf architecture. As a result, every Leaf is physically connected to every Spine, enabling traffic forwarding through non-blocking links. Physically, a leaf switch set creates a leaf layer attached to the spines in a full BIPARTITE graph. This means that each Leaf is connected to each Spine, and each Spine is connected to each Leaf. 

The ACI uses a horizontally elongated Leaf and Spine architecture with one hop to every host in an entirely messed ACI fabric, offering good throughput and convergence needed for today’s applications.

The ACI fabric: Does Not Aggregate Traffic

A key point in the spine-and-leaf design is the fabric concept, like a stretch network. One of the core ideas around a fabric is that it does not aggregate traffic. This does increase data center performance along with a non-blocking architecture. With the spine-leaf topology, we are spreading a fabric across multiple devices.

Required: Increased Bandwidth Available

The result of the fabric is that each edge device has the total bandwidth of the fabric available to every other edge device. This is one big difference from traditional data center designs; we aggregate the traffic by either stacking multiple streams onto a single link or carrying the streams serially.

Challenge: Oversubscription

With the traditional 3-tier design, we aggregate everything at the core, leading to oversubscription ratios that degrade performance. With the ACI Leaf and Spine design, we spread the load across all devices with equidistant endpoints, allowing us to carry the streams parallel.

Required: Routed Multipathing

Then, we have horizontal scaling load balancing.  Load balancing with this topology uses multipathing to achieve the desired bandwidth between the nodes. Even though this forwarding paradigm can be based on Layer 2 forwarding ( bridging) or Layer 3 forwarding ( routing), the ACI leverages a routed approach to the Leaf and Spine design, and we have Equal Cost Multi-Path (ECMP) for both Layer 2 and Layer 3 traffic. 

**Overlay and Underlay Design**

Mapping Traffic:

So you may be asking how we can have Layer 3 routed core and pass Layer 2 traffic. This is done using the overlay, which can map different traffic types to other overlays. So, we can have Layer 2 traffic mapped to an overlay over a routed core.

L3 active-active links: ACI links between the Leaf and the Spine switches are L3 active-active links. Therefore, we can intelligently load balance and traffic steer to avoid issues. We don’t need to rely on STP to block links or involve STP in fixing the topology.

Challenge: IP – Identity & Location

When networks were first developed, there was no such thing as an application moving from one place to another while it was in use. So, the original architects of IP, the communication protocol used between computers, used the IP address to indicate both the identity of a device connected to the network and its location on the network. Today, in the modern data center, we need to be able to communicate with an application or application tier, no matter where it is.

Required: Overlay Encapsulation

One day, it may be in location A and the next in location B, but its identity, which we communicate with, is the same on both days. An overlay is when we encapsulate an application’s original message with the location to which it needs to be delivered before sending it through the network. Once it arrives at its final destination, we unwrap it and deliver the original message as desired.

The identities of the devices (applications) communicating are in the original message, and the locations are in the encapsulation, thus separating the place from the identity. This wrapping and unwrapping is done on a per-packet basis and, therefore, must be done quickly and efficiently.

**Overlay and Underlay Components**

The Cisco SDN ACI has an overlay and underlay concept, which forms a virtual overlay solution. The role of the underlay is to glue together devices so the overlay can work and be built on top. So, the overlay, which is VXLAN, runs on top of the underlay, which is IS-IS. In the ACI, the IS-IS protocol provides the routing for the overlay, which is why we can provide ECMP from the Leaf to the Spine nodes. The routed underlay provides an ECMP network where all leaves can access Spine and have the same cost links. 

ACI overlay
Diagram: Overlay. Source Cisco

Underlay & Overlay Interaction

Example: 

Let’s take a simple example to illustrate how this is done. Imagine that application App-A wants to send a packet to App-B. App-A is located on a server attached to switch S1, and App-B is initially on switch S2. When App-A creates the message, it will put App-B as the destination and send it to the network; when the message is received at the edge of the network, whether a virtual edge in a hypervisor or a physical edge in a switch, the network will look up the location of App-B in a “mapping” database and see that it is attached to switch S2.

It will then put the address of S2 outside of the original message. So, we now have a new message addressed to switch S2. The network will forward this new message to S2 using traditional networking mechanisms. Note that the location of S2 is very static, i.e., it does not move, so using traditional mechanisms works just fine.

Upon receiving the new message, S2 will remove the outer address and thus recover the original message. Since App-B is directly connected to S2, it can easily forward the message to App-B. App-A never had to know where App-B was located, nor did the network’s core. Only the edge of the network, specifically the mapping database, had to know the location of App-B. The rest of the network only had to see the location of switch S2, which does not change.

Let’s now assume App-B moves to a new location switch S3. Now, when App-A sends a message to App-B, it does the same thing it did before, i.e., it addresses the message to App-B and gives the packet to the network. The network then looks up the location of App-B and finds that it is now attached to switch S3. So, it puts S3’s address on the message and forwards it accordingly. At S3, the message is received, the outer address is removed, and the original message is delivered as desired.

App-A did not track App-B’s movement at all. App-B’s address identified It, while the switch’s address, S2 or S3, identified its location. App-A can communicate freely with App-B no matter where It is located, allowing the system administrator to place App-B in any area and move it as desired, thus achieving the flexibility needed in the data center.

Multicast Distribution Tree (MDT)

We have a Multicast Distribution Tree MDT tree on top that is used to forward multi-destination traffic without having loops. The Multicast distribution tree is dynamically built to send flood traffic for specific protocols. Again, it does this without creating loops in the overlay network. The tunnels created for the endpoints to communicate will have tunnel endpoints. The tunnel endpoints are known as the VTEP. The VTEP addresses are assigned to each Leaf switch from a pool that you specify in the ACI startup and discovery process.

Normalize the transports

VXLAN tunnels in the ACI fabric normalize the transports in the ACI network. Therefore, traffic between endpoints can be delivered using the VXLAN tunnel, resulting in any transport network regardless of the device connecting to the fabric. 

So, using VXLAN in the overlay enables any network, and you don’t need to configure anything special on the endpoints for this to happen. The endpoints that connect to the ACI fabric do not need special software or hardware. The endpoints send regular packets to the leaf nodes they are connected to directly or indirectly. As endpoints come online, they send traffic to reach a destination.

Bridge Domains and VRF

Therefore, the Cisco SDN ACI under the hood will automatically start to build the VXLAN overlay network for you. The VXLAN network is based on the Bridge Domain (BD), or VRF ACI constructs deployed to the leaf switches. The Bridge Domain is for Layer 2, and the VRF is for Layer 3. So, as devices come online and send traffic to each other, the overlay will grow in reachability in the Bridge Domain or the VRF. 

Direct host routing for endoints

Routing within each tenant, VRF is based on host routing for endpoints directly connected to the Cisco ACI fabric. For IPv4, the host routing is based on the /32, giving the ACI a very accurate picture of the endpoints. Therefore, we have exact routing in the ACI.  In conjunction, we have a COOP database that runs on the Spines and offers remarkably optimized fabric to know where all the endpoints are located.

To facilitate this, every node in the fabric has a TEP address, and we have different types of TEPs depending on the device’s role. The Spine and the Leaf will have TEP addresses but will differ from each other.

COOP database
Diagram: COOP database

The VTEP and PTEP

The Leaf’s nodes are the Virtual Tunnel Endpoints (VTEP), which are also known as the physical tunnel endpoints (PTEP) in ACI. These PTEP addresses represent the “WHERE” in the ACI fabric where an endpoint lives. Cisco ACI uses a dedicated VRF and a subinterface of the uplinks from the Leaf to the Spines as the infrastructure to carry VXLAN traffic. In Cisco ACI terminology, the transport infrastructure for VXLAN traffic is known as Overlay-1, which is part of the tenant “infra.” 

**The Spine TEP**

The Spines also have a PTEP and an additional proxy TEP, which are used for forwarding lookups into the mapping database. The Spines have a global view of where everything is, which is held in the COOP database synchronized across all Spine nodes. All of this is done automatically for you.

**Anycast IP Addressing**

For this to work, the Spines have an Anycast IP address known as the Proxy TEP. The Leaf can use this address if they do not know where an endpoint is, so they ask the Spine for any unknown endpoints, and then the Spine checks the COOP database. This brings many benefits to the ACI solution, especially for traffic optimizations and reducing flooded traffic in the ACI. Now, we have an optimized fabric for better performance.

The ACI optimizations

**Mouse and elephant flow**

This provides better performance for load balancing different flows. For example, in most data centers, we have latency-sensitive flows, known as mouse flows, and long-lived bandwidth-intensive flows, known as elephant flows. 

The ACI has more precisely load-balanced traffic using algorithms that optimize mouse and elephant flows and distribute traffic based on flow lets: flow let load-balancing. Within a Leaf, Spine latency is low and consistent from port to port.

The max latency of a packet from one port to another in the architecture is the same regardless of the network size. So you can scale the network without degrading performance. Scaling is often done on a POD-by-POD basis. For more extensive networks, each POD would be its Leaf and Spine network.

**ARP optimizations: Anycast gateways**

The ACI comes by default with a lot of traffic optimizations. Firstly, instead of using an ARP and broadcasting across the network, that can hamper performance. The Leaf can assume that the Spine will know where the destination is ( and it does via the COOP database ), so there is no need to broadcast to everyone to find a destination.

If the Spine knows where the endpoint is, it will forward the traffic to the other Leaf. If not, it will drop it.

**Fabric anycast addressing**

This again adds performance benefits to the ACI solution as the table sizes on the Leaf switches can be kept smaller than they would if they needed to know where all the destinations were, even if they were not or never needed to communicate with them. On the Leaf, we have an Anycast address too.

These fabric Anycast addresses are available for Layer 3 interfaces. On the Leaf ToR, we can establish an SVI that uses the same MAC address on every ToR; therefore, when an endpoint needs to route to a ToR, it doesn’t matter which ToR you use. The Anycast Address is spread across all ToR leaf switches. 

**Pervasive gateway**

Now we have predictable latency to the first hop, and you will use the local route VRF table within that ToR instead of traversing the fabric to a different ToR. This is the Pervasive Gateway feature that is used on all Leaf switches. The Cisco ACI has many advanced networking features, but the pervasive gateway is my favorite. It does take away all the configuration mess we had in the past.

ACI Cisco: Integrations

  • Routing Control Platform

Then came along Cisco SDN ACI, the ACI Cisco, which operates differently from the traditional data center with an application-centric infrastructure. The Cisco application-centric infrastructure achieves resource elasticity with automation through standard policies for data center operations and consistent policy management across multiple on-premises and cloud instances.

  • Extending & Integrating the fabric

What makes the Cisco ACI interesting is its several vital integrations. I’m not talking about extending the data center with multi-pod and multi-site, for example, with AlgoSec, Cisco AppDynamics, and SD-WAN. AlgoSec enables secure application delivery and policy across hybrid network estates, while AppDynamic lives in a world of distributed systems Observability. SD-WAN enabled path performance per application with virtual WANs.

Cisco Multi-Pod Design

Cisco ACI Multi-Pod is part of the “Single APIC Cluster / Single Domain” family of solutions, as a single APIC cluster is deployed to manage all the interconnected ACI networks. These separate ACI networks are named “pods,” Each looks like a regular two-tier spine-leaf topology. The same APIC cluster can manage several pods, and to increase the resiliency of the solution, the various controller nodes that make up the cluster can be deployed across different pods.

ACI Multi-Pod
Diagram: Cisco ACI Multi-Pod. Source Cisco.

ACI Cisco and AlgoSec

With AlgoSec integrated with the Cisco ACI, we can now provide automated security policy change management for multi-vendor devices and risk and compliance analysis. The AlgoSec Security Management Solution for Cisco ACI extends ACI’s policy-driven automation to secure various endpoints connected to the Cisco SDN ACI fabric.

These simplify network security policy management across on-premises firewalls, SDNs, and cloud environments. They also provide visibility into ACI’s security posture, even across multi-cloud environments. 

ACI Cisco and AppDynamics 

Then, with AppDynamics, we are heading into Observability and controllability. Now, we can correlate app health and network for optimal performance, deep monitoring, and fast root-cause analysis across complex distributed systems with numbers of business transactions that need to be tracked.

This will give your teams complete visibility of your entire technology stack, from your database servers to cloud-native and hybrid environments. In addition, AppDynamics works with agents that monitor application behavior in several ways. We will examine the types of agents and how they work later in this post.

ACI Cisco and SD-WAN 

SD-WAN brings a software-defined approach to the WAN. These enable a virtual WAN architecture to leverage transport services such as MPLS, LTE, and broadband internet services. So, SD-WAN is not a new technology; its benefits are well known, including improving application performance, increasing agility, and, in some cases, reducing costs.

The Cisco ACI and SD-WAN integration makes active-active data center design less risky than in the past. The following figures give a high-level overview of the Cisco ACI and SD-WAN integration. For pre-information generic to SD-WAN, go here: SD-WAN Tutorial

SD WAN integration
Diagram: Cisco ACI and SD-WAN integration

The Cisco SDN ACI with SD-WAN integration helps ensure an excellent application experience by defining application Service-Level Agreement (SLA) parameters. Cisco ACI releases 4.1(1i) and adds support for WAN SLA policies. This feature enables admins to apply pre-configured policies to specify the packet loss, jitter, and latency levels for the tenant traffic over the WAN.

When you apply a WAN SLA policy to the tenant traffic, the Cisco APIC sends the pre-configured policies to a vManage controller. The vManage controller, configured as an external device manager that provides SDWAN capability, chooses the best WAN link that meets the loss, jitter, and latency parameters specified in the SLA policy.

Openshift and Cisco SDN ACI

OpenShift Container Platform (formerly known as OpenShift Enterprise) or OCP is Red Hat’s offering for the on-premises private platform as a service (PaaS). OpenShift is based on the Origin open-source project and is a Kubernetes distribution, the defacto for container-based virtualization. The foundation of the OpenShift networking SDN is based on Kubernetes and, therefore, shares some of the same networking technology along with some enhancements, such as the OpenShift route construct.

Other data center integrations

Cisco SDN ACI has another integration with Cisco DNA Center/ISE that maps user identities consistently to endpoints and apps across the network, from campus to the data center. Cisco Software-Defined Access (SD-Access) provides policy-based automation from the edge to the data center and the cloud.

Cisco SD-Access provides automated end-to-end segmentation to separate user, device, and application traffic without redesigning the network. This integration will enable customers to use standard policies across Cisco SD-Access and Cisco ACI, simplifying customer policy management using Cisco technology in different operational domains.

OpenShift and Cisco ACI

OpenShift does this with an SDN layer and enhances Kubernetes networking to create a virtual network across all the nodes. It is made with the Open Switch standard. For OpenShift SDN, this pod network is established and maintained by the OpenShift SDN, configuring an overlay network using a virtual switch called the OVS bridge. This forms an OVS network that gets programmed with several OVS rules. The OVS is a popular open-source solution for virtual switching.

OpenShift SDN plugin

We mentioned that you could tailor the virtual network topology to suit your networking requirements, which can be determined by the OpenShift SDN plugin and the SDN model you select. With the default OpenShift SDN, several modes are available. This level of SDN mode you choose is concerned with managing connectivity between applications and providing external access to them. Some modes are more fine-grained than others. The Cisco ACI plugins offer the most granular.

Integrating ACI and OpenShift platform

The Cisco ACI CNI plugin for the OpenShift Container Platform provides a single, programmable network infrastructure, enterprise-grade security, and flexible micro-segmentation possibilities. The APIC can provide all networking needs for the workloads in the cluster. Kubernetes workloads become fabric endpoints, like Virtual Machines or Bare Metal endpoints.

Cisco ACI CNI Plugin

The Cisco ACI CNI plugin extends the ACI fabric capabilities to OpenShift clusters to provide IP Address Management, networking, load balancing, and security functions for OpenShift workloads. In addition, the Cisco ACI CNI plugin connects all OpenShift Pods to the integrated VXLAN overlay provided by Cisco ACI.

Cisco SDN ACI and AppDynamics

AppDynamis overview

So, an application requires multiple steps or services to work. These services may include logging in and searching to add something to a shopping cart. These services invoke various applications, web services, third-party APIs, and databases, known as business transactions.

The user’s critical path

A business transaction is the essential user interaction with the system and is the customer’s critical path. Therefore, business transactions are the things you care about. If they start to go, your system will degrade. So, you need ways to discover your business transactions and determine if there are any deviations from baselines. This should also be automated, as learning baseline and business transitions in deep systems is nearly impossible using the manual approach.

So, how do you discover all these business transactions?

AppDynamics automatically discovers business transactions and builds an application topology map of how the traffic flows. A topology map can view usage patterns and hidden flows, acting as a perfect feature for an Observability platform.

AppDynamic topology

AppDynamics will automatically discover the topology for all of your application components. It can then build a performance baseline by capturing metrics and traffic patterns. This allows you to highlight issues when services and components are slower than usual.

AppDynamics uses agents to collect all the information it needs. The agent monitors and records the calls that are made to a service. This is from the entry point and follows executions along its path through the call stack. 

Types of Agents for Infrastructure Visibility

If the agent is installed on all critical parts, you can get information about that specific instance, which can help you build a global picture. So we have an Application Agent, Network Agent, and Machine Agent for Server visibility and Hardware/OS.

  • App Agent: This agent will monitor apps and app servers, and example metrics will be slow transitions, stalled transactions, response times, wait times, block times, and errors.  
  • Network Agent: This agent monitors the network packets, TCP connection, and TCP socket. Example metrics include performance impact Events, Packet loss, and retransmissions, RTT for data transfers, TCP window size, and connection setup/teardown.
  • Machine Agent Server Visibility: This agent monitors the number of processes, services, caching, swapping, paging, and querying. Example Metrics include hardware/software interrupts, virtual memory/swapping, process faults, and CPU/DISK/Memory utilization by the process.
  • Machine Agent: Hardware/OS – disks, volumes, partitions, memory, CPU. Example metrics: CPU busy time, MEM utilization, and pieces file.

Automatic establishment of the baseline

A baseline is essential, a critical step in your monitoring strategy. Doing this manually is hard, if not impossible, with complex applications. It is much better to have this done automatically. You must automatically establish the baseline and alert yourself about deviations from it.

This will help you pinpoint the issue faster and resolve it before it can be affected. Platforms such as AppDynamics can help you here. Any malicious activity can be seen from deviations from the security baseline and performance issues from the network baseline.

Cisco ACI • Policy & Application Flow

ACI Policy & Application Flow Lab

Follow how ACI turns application requirements into policy: Tenant → VRF → Bridge Domain → EPG → Contract → Application Flow.
ACI POLICY MODEL TENANT & NETWORK CONTEXT APPLICATION GROUPS POLICY APPLICATION FLOW Tenant Administrative container NETWORK CONTEXT VRF Layer 3 routing context Bridge Domain Endpoint network Web EPG Web application endpoints App EPG Application endpoints CONTRACT ALLOWED APPLICATION FLOW Web Servers Endpoint group App Servers Endpoint group
Tenant / VRF
Network
EPG
Contract
Application Flow
Selected Component
ACI Policy Model
ACI expresses application connectivity through a hierarchy of objects. Start at the Tenant and follow the model down to the application flow.
Object
Tenant → VRF → Bridge Domain → EPG
Policy
Contracts define which EPGs can communicate.
Why it matters
ACI lets administrators describe application connectivity as policy rather than configuring every individual switch interface.

Summary: Cisco ACI Components

In the ever-evolving world of networking, organizations are constantly seeking ways to enhance their infrastructure’s performance, security, and scalability. Cisco ACI (Application Centric Infrastructure) presents a cutting-edge solution to these challenges. By unifying physical and virtual environments and leveraging network automation, Cisco ACI revolutionizes how networks are built and managed.

Understanding Cisco ACI Architecture

At the core of Cisco ACI lies a robust architecture that enables seamless integration between applications and the underlying network infrastructure. The architecture comprises three key components:

1. Application Policy Infrastructure Controller (APIC):

The APIC serves as the centralized management and policy engine of Cisco ACI. It provides a single point of control for configuring and managing the entire network fabric. Through its intuitive graphical user interface (GUI), administrators can define policies, allocate resources, and monitor network performance.

2. Nexus Switches:

Cisco Nexus switches form the backbone of the ACI fabric. These high-performance switches deliver ultra-low latency and high throughput, ensuring optimal data transfer between applications and the network. Nexus switches provide the necessary connectivity and intelligence to enable the automation and programmability features of Cisco ACI.

3. Application Network Profiles:

Application Network Profiles (ANPs) are a fundamental aspect of Cisco ACI. ANPs define the policies and characteristics required for specific applications or application groups. By encapsulating network, security, and quality of service (QoS) policies within ANPs, administrators can streamline the deployment and management of applications.

The Power of Network Automation

One of the most compelling aspects of Cisco ACI is its ability to automate network provisioning, configuration, and monitoring. Through the APIC’s powerful automation capabilities, network administrators can eliminate manual tasks, reduce human errors, and accelerate the deployment of applications. With Cisco ACI, organizations can achieve greater agility and operational efficiency, enabling them to rapidly adapt to evolving business needs.

Security and Microsegmentation with Cisco ACI

Security is a paramount concern for every organization. Cisco ACI addresses this by providing robust security features and microsegmentation capabilities. With microsegmentation, administrators can create granular security policies at the application level, effectively isolating workloads and preventing lateral movement of threats. Cisco ACI also integrates with leading security solutions, enabling seamless network enforcement and threat intelligence sharing.

Conclusion

Cisco ACI is a game-changer in the realm of network automation and infrastructure management. Its innovative architecture, coupled with powerful automation capabilities, empowers organizations to build agile, secure, and scalable networks. By leveraging Cisco ACI’s components, businesses can unlock new levels of efficiency, flexibility, and performance, ultimately driving growth and success in today’s digital landscape.

SDN Data Center

SDN Data Center

NETWORK ARCHITECTURE · SOFTWARE-DEFINED NETWORKING

SDN Data Center: Architecture, Fabrics and Programmable Networks

Modern data centers must connect increasingly distributed applications, virtual machines, containers and services while maintaining predictable performance, security and availability. Software-Defined Networking (SDN) introduces a programmable approach to network control, enabling operators to express policies, automate configuration and coordinate network behaviour across a shared infrastructure.

The shift is not simply about replacing the command-line interface with software. It is about changing how network intent is translated into forwarding behaviour, how infrastructure responds to workload changes, and how engineers verify that the network is operating as designed. Understanding this distinction is essential when evaluating SDN architectures and deciding where automation and centralised control provide real operational value.

Traditional data center designs often relied on hierarchical switching, Layer 2 extension and Spanning Tree Protocol (STP) to manage redundant connectivity. As east-west traffic, virtualisation and workload mobility increased, operators adopted routed leaf-spine fabrics, Equal-Cost Multi-Path (ECMP) forwarding and overlay technologies such as VXLAN. SDN can coordinate these mechanisms through policy and automation, but the exact implementation depends on the platform and architecture.

From network infrastructure to programmable fabric

01 · ARCHITECTURE Separate responsibilities Understand the relationship between applications, control systems and the forwarding infrastructure.
02 · FABRIC Build resilient paths Explore leaf-spine connectivity, Layer 3 routing, ECMP and failure-aware forwarding.
03 · OVERLAY Connect virtual networks Examine VXLAN, network segmentation, virtual network identifiers and workload connectivity.
04 · OPERATIONS Verify network intent Use policy, telemetry and fault analysis to compare expected behaviour with observed network state.

The SDN control model

SDN separates the definition and coordination of network behaviour from packet forwarding. A controller or control system can translate application requirements and network policies into device configuration, forwarding rules or other platform-specific actions. The forwarding devices continue to process packets using their programmed state.

Conceptual SDN architecture
Applications Workload needs and services
→
Policy and APIs Intent and desired outcomes
→
Controller Coordination and configuration
→
Fabric Forwarding and enforcement

This is a conceptual model, not a requirement that every SDN deployment use one physical controller or a single centralised control process. Production architectures may distribute controller functions, retain routing protocols on network devices, and combine central policy with local forwarding decisions.

Why data center design matters

A programmable network is only as effective as the fabric underneath it. The physical topology, routing design, link capacity, failure domains and overlay configuration all influence how applications perform. Centralised policy cannot compensate for every physical bottleneck or incorrect forwarding assumption.

For this reason, SDN must be considered alongside data center architecture. A resilient leaf-spine fabric can provide multiple routed paths, while ECMP distributes eligible flows across equal-cost routes. VXLAN can provide virtual Layer 2 segments across a Layer 3 underlay, and policy systems can help coordinate tenant isolation and service connectivity.

Engineering principle

Separate desired network intent from actual forwarding state. A controller may report that a policy has been deployed, but operational verification must still establish reachability, path availability, segmentation, resource utilisation and application connectivity.

What this engineering guide covers

This guide develops the subject from the foundations of SDN through the design and operation of a modern data center fabric. It connects the architectural concepts to the protocols, forwarding mechanisms and operational evidence that engineers use to troubleshoot real networks.

Architecture and control Understand controllers, programmability, policy abstraction and the relationship between control and forwarding.
Fabric and overlays Relate leaf-spine topology, ECMP, VXLAN, routing and virtual network segmentation.
Operations and resilience Investigate convergence, link failures, telemetry, policy consistency and observable forwarding outcomes.

From theory to engineering evidence

Throughout the guide, the emphasis is on understanding cause and effect. When a link fails, which paths remain available? When a virtual workload moves, what must change in the network? When a controller applies a new policy, how can an engineer confirm that the intended result has reached the forwarding infrastructure?

These are operational questions, not just architectural definitions. Answering them requires an understanding of topology, routing, encapsulation, policy and telemetry—and the ability to correlate evidence from different parts of the network.

Engineering Guide · Introduction
Start with the architecture

The first section examines what Software-Defined Networking actually changes, how the application, control and infrastructure responsibilities relate to one another, and why SDN should not be treated as synonymous with OpenFlow or a single controller product.

Engineering Guide · Navigation

Explore the SDN Data Center

Follow the engineering path from programmable network architecture through resilient fabrics and overlays to automation, troubleshooting and operational security. Select a topic to jump directly to its introduction.

Section 01 · Architecture Fundamentals

Understanding SDN Architecture and Programmability

Software-Defined Networking separates network policy and control from packet forwarding through programmable interfaces and coordinated management. Explore the application, control and infrastructure layers, understand how they interact, and examine why SDN is an architectural approach rather than a single protocol or controller product.

Control vs. data plane SDN architecture Programmable interfaces
01 · ARCHITECTURE · CONTROL · PROGRAMMABILITY

SDN Architecture: The Control Plane and Network Programmability

Software-Defined Networking changes how network behaviour is configured, coordinated and managed. Rather than requiring engineers to configure every device independently for each network-wide change, SDN introduces software-based control and policy mechanisms that can coordinate infrastructure through programmable interfaces.

The central architectural idea is the separation of responsibilities between the systems that express network requirements, the control functions that translate those requirements into network behaviour, and the devices that forward packets. This separation can simplify operations and enable automation, but it does not mean that every routing decision moves into one central controller.

To understand an SDN data center, engineers must distinguish the control plane from the data plane, understand how controllers interact with network devices, and establish how a desired policy becomes observable forwarding behaviour.

The three conceptual layers of SDN

LAYER 01 Application layer Expresses application connectivity requirements, network services and higher-level policies.
LAYER 02 Control layer Coordinates network state, policy translation, topology information and infrastructure programming.
LAYER 03 Infrastructure layer Contains switches, routers and other forwarding elements that process traffic.
CROSS-LAYER Feedback and telemetry Reports configuration, device health, reachability and operational state to management and control systems.

These layers provide a useful architectural model rather than a mandatory physical deployment. Some platforms combine functions, distribute controller services across multiple systems, or retain substantial control-plane intelligence on individual switches and routers.

Control plane versus data plane

The control plane determines or coordinates how traffic should be forwarded. Depending on the architecture, it can include routing protocols, topology discovery, path calculation, policy processing and controller logic.

The data plane performs the actual packet-handling work. It examines packet headers, consults forwarding information and applies actions such as forwarding, encapsulation, decapsulation, marking or dropping. High-speed switches commonly execute these operations in specialised hardware.

From intent to packet forwarding
Network intent Desired connectivity and policy
→
Control logic Calculate or coordinate behaviour
→
Forwarding state Routes, rules and tables
→
Packet action Forward, modify or drop

Consider a workload that needs connectivity to an application in another network segment. The management system may express the required communication policy. A controller or orchestration system translates it into platform-specific configuration. Switches and routers then apply the resulting forwarding and enforcement state to real packets.

Each stage introduces a separate verification requirement. A policy can be accepted by the management system but fail during deployment; configuration can be installed while a route is missing; and a valid route can exist while a security rule prevents the application flow.

Engineering distinction

A controller's intended network state, a device's installed configuration and the forwarding behaviour observed by a packet capture are three different forms of evidence. Reliable troubleshooting correlates them instead of assuming that success at one layer proves success at every other layer.

What does an SDN controller actually do?

An SDN controller provides software-based mechanisms for coordinating network behaviour. Its responsibilities depend on the product and architecture, but can include maintaining a topology view, translating policy, calculating paths, managing forwarding rules, programming devices and collecting operational information.

Some controllers directly install forwarding entries in compatible switches. Others configure devices through management APIs, coordinate existing routing protocols, or distribute policy through agents and platform-specific components. An SDN controller should therefore be understood by the functions it performs, not simply by its product name.

Controller responsibilities
Topology and network state Build or maintain a view of devices, links, connectivity and relevant operational information.
Policy translation Convert higher-level requirements into configuration or forwarding instructions supported by the platform.
Infrastructure programming Communicate with network devices through supported protocols, APIs or management interfaces.
Verification and feedback Compare expected outcomes with reported configuration, connectivity and operational telemetry.

Northbound and southbound interfaces

SDN architectures commonly describe interfaces according to the direction in which information or control is exchanged. Northbound interfaces allow applications and orchestration systems to interact with controller services. Southbound interfaces connect control functions to network infrastructure.

Interface Purpose Examples
Northbound Expose network services, intent or operational information to applications and orchestration systems. REST APIs, controller SDKs and policy interfaces.
Southbound Exchange configuration, control information or forwarding instructions with infrastructure. OpenFlow, NETCONF, RESTCONF and platform-specific APIs, depending on the implementation.
Telemetry and feedback Return device state, counters, events and performance information for monitoring and verification. Streaming telemetry, counters, logs and event notifications.

These categories describe interface roles, not a requirement that every protocol be used in every SDN design. NETCONF and RESTCONF, for example, are management protocols rather than equivalents of OpenFlow's flow-programming model. The correct interface depends on the device capabilities and the operation being performed.

SDN and OpenFlow are not the same thing

OpenFlow is a protocol historically associated with SDN that allows a controller to communicate with compatible forwarding devices and manage flow-table behaviour. It helped demonstrate how packet forwarding could be programmed through an external control system.

SDN is the broader architectural approach. A network can use SDN principles without using OpenFlow. Modern data center platforms may combine controller-based policy, routing protocols, management APIs, overlays and distributed forwarding intelligence.

Open Networking Foundation initiatives helped advance the development and standardisation of SDN concepts and OpenFlow. In practice, engineers should evaluate the interfaces, supported features and operational model of the specific implementation rather than assume that every SDN product uses the same protocol stack.

Centralised policy does not mean one central forwarding engine

The term centralised control can be misleading if interpreted too literally. A controller may provide a global policy view while switches continue to forward packets locally. Distributed routing protocols can also operate alongside a controller, exchanging reachability information and maintaining forwarding tables.

This combination is particularly important in data centers. A controller might define tenant segmentation and application connectivity, while the underlay uses BGP or an IGP to establish IP reachability between leaf and spine switches. ECMP can then distribute eligible traffic across available equal-cost paths.

Separating these responsibilities allows policy orchestration and packet forwarding to work together without requiring every packet to pass through the controller.

Automation, consistency and failure handling

Programmability enables engineers to express repeatable network changes in software. Rather than manually configuring every device, an automation workflow can generate and deploy configuration, validate prerequisites and check the resulting state.

Automation does not automatically guarantee correctness. Incorrect intent, incomplete device support, stale topology data, failed deployment or partial updates can produce inconsistent outcomes. A robust implementation needs validation, access controls, change tracking, rollback or recovery procedures, and independent operational verification.

What to verify during an SDN change
Desired state Is the requested connectivity or segmentation policy valid and consistent with the intended design?
Deployment state Did all relevant controllers and devices accept and install the required changes?
Forwarding state Do the installed routes, next hops, forwarding entries and policy rules match the expected outcome?
Observed behaviour Can the required application traffic pass, and do telemetry and tests confirm the result?

Applying the architecture to a data center

Imagine a tenant needs a new application segment that can communicate with a database but must remain isolated from another tenant. The application or orchestration layer defines the connectivity intent. The control system translates it into the supported policy and configuration. The data center fabric provides the actual transport, while security enforcement ensures that only permitted communication is allowed.

The engineering task is to trace this chain end to end. Confirm that the policy exists, that the relevant devices received it, that the underlay can reach the required destinations, and that the overlay and security configuration support the intended traffic.

Section 01 takeaway

SDN provides programmable coordination of network behaviour. The control plane determines or coordinates how the network should operate, while the data plane performs packet forwarding. Successful operation depends on translating intent into valid device state and verifying the resulting traffic behaviour.

Engineering Lab · Section 01
SDN Architecture and Intent Verification

In the interactive lab, inspect a conceptual SDN data center and follow a policy change from application intent through controller processing to infrastructure configuration and packet forwarding. Examine the control and data planes separately, identify the role of northbound and southbound interfaces, and investigate what happens when intended policy and observed network state disagree.

Network Insight / Interactive Engineering Lab 01

SDN Architecture & Programmability

Trace how a programmable network separates decision-making from packet forwarding. Step through a simulated packet-in event, controller decision, flow-rule installation and data-plane forwarding. Then break a link and investigate the recovery.
Ready
Programmable Network Topology
Event 0 / 8
Primary path Alternate path Ethernet Control channel SDN CONTROLLER Policy / topology / rules HOST A 10.1.0.10 SWITCH 1 Ingress / flow match No rule yet SWITCH 2 Forwarding No rule yet SWITCH 3 Alternate route Standby HOST B 10.1.0.20 DATA PLANE Packet forwarding across switch links CONTROL PLANE Controller communicates with switches
Active forwarding path Controller channel Alternate path Failed link
Topology note: the controller makes logical forwarding decisions and programs the switches. User packets normally travel through the data plane; they do not pass through the controller.
Two Logical Planes
Architecture
Control Plane Decides

Maintains network state, applies policy and determines which forwarding behaviour should be installed.

Data Plane Forwards

Matches packets against forwarding entries and performs actions such as output, drop or header modification.

Controller stateInitialising
Installed flow rules0
Forwarding pathNot selected
Link conditionAll links available
Flow Table
Simulated entries
MatchActionState
10.1.0.10 → .20Pending—
Important: this table represents illustrative flow entries in the lab. Real switch rule syntax and controller protocols depend on the SDN platform.
Packet & Controller Event Timeline
Step-by-step trace
01Ingress Host A sends a packet to Host B; Switch 1 receives it.
02Match Switch 1 checks its flow table and finds no matching entry in this scenario.
03Packet-in The switch reports the table miss to the controller using the configured control mechanism.
04Decision The controller evaluates topology and policy, then selects a forwarding path.
05Install The controller sends flow-modification instructions to the relevant switches.
06Forward The packet is forwarded along the installed path toward Host B.
07Failure A link fails; the network must detect the change and determine whether an alternate route is usable.
08Recovery The lab models a controller-directed update to use the alternate path through Switch 3.
Live Engineering Readout
Lab telemetry
Current Event
Ready to begin
Select Step Forward to trace how a controller and programmable switches coordinate packet delivery.
Simulation Progress

Engineer Challenge

Run the packet trace, inject a link failure, and continue the simulation. Can the network deliver traffic without using the failed primary link?

Challenge status: not yet attempted.
Engineering takeaway: separating the control plane from the data plane makes forwarding behaviour programmable, but recovery still depends on detection, topology knowledge and valid forwarding updates.
Section 02 · Fabric Engineering

Leaf–Spine Fabrics, ECMP and Network Resilience

Modern data centers need predictable connectivity, usable redundant paths and reliable recovery from failures. Discover how leaf–spine topologies, routed underlays, BGP and Equal-Cost Multipath work together, and learn why routing convergence, forwarding state and available capacity must all be considered during an incident.

Leaf–spine topology BGP and ECMP Failure recovery
02 · FABRIC · ROUTING · RESILIENCE

SDN Data Center Fabrics: Leaf–Spine, ECMP and Resilient Forwarding

A programmable network needs a dependable transport foundation. In modern data centers, that foundation is often a routed leaf–spine fabric: a predictable topology designed to provide multiple paths, scalable east–west connectivity and fast recovery when links or devices fail.

Software-defined control and the physical fabric solve different problems. SDN provides mechanisms to coordinate policy and network configuration. The fabric provides the paths over which packets travel. Understanding how these parts work together is essential when designing, automating or troubleshooting a data center.

Why data centers moved beyond large Layer 2 domains

Traditional data center designs often relied on extensive Layer 2 connectivity. Spanning Tree Protocol (STP) prevents Layer 2 loops by blocking selected redundant paths. This protects the network from looping Ethernet frames, but some physical links may not forward traffic while the topology is stable.

As east–west traffic between servers, application tiers and storage increased, engineers looked for designs that could use redundant links more effectively. Routed leaf–spine fabrics provide multiple Layer 3 paths and can use Equal-Cost Multipath (ECMP) forwarding where supported and correctly configured.

This does not make STP obsolete. Layer 2 still exists at access edges and within particular designs. The key architectural choice is to limit where Layer 2 must extend and use routed connectivity for the scalable fabric wherever the design permits.

Understanding the leaf–spine topology

Conceptual routed data center fabric
Spine 1 Layer 3 transit
Spine 2 Layer 3 transit
Leaf 1 Compute rack A
Leaf 2 Compute rack B
Leaf 3 Storage / services
Leaf 4 Application tier

Simplified illustration: production fabrics generally connect each leaf to multiple spines. The actual number of devices, link speeds, redundancy and routing design depend on capacity and availability requirements.

Leaf switches connect directly to servers, hypervisors, storage systems or other network edges. Spine switches provide transit between leaves. In a conventional two-tier design, leaf switches connect to the spine layer rather than to one another.

Leaf layer

Provides workload attachment, Layer 3 gateways where required, access policy and connectivity to the fabric. It is commonly the point where server-facing links enter the network.

Spine layer

Provides high-capacity transit between leaves. Multiple spines create alternate paths and help scale fabric capacity without requiring every leaf to connect directly to every other leaf.

Underlay

The routed IP network connecting the fabric devices. It provides reachability between relevant interfaces and endpoints used by the routing and overlay systems.

Overlay

A logical network built over the underlay, often using VXLAN to carry tenant or segment traffic between tunnel endpoints. The overlay does not remove the need for working underlay reachability.

ECMP: using multiple equal-cost paths

Equal-Cost Multipath routing allows a router to install multiple eligible next hops for a destination when the routing system determines that those paths have the same cost. Instead of selecting only one next hop, the device can distribute eligible flows across the available paths.

In a leaf–spine fabric, a leaf may have a route to a remote subnet through several spine switches. If the routes qualify for ECMP, the leaf can use those next hops to distribute traffic across the fabric.

Example: two eligible next hops

Leaf 1 has two equal-cost routes to a remote destination: one through Spine 1 and one through Spine 2. If both next hops are installed and usable, eligible flows can be spread across the two paths. If a path becomes unavailable, routing and forwarding mechanisms can remove or stop using that next hop.

How ECMP distributes traffic

Many implementations use a hash calculated from packet or flow fields to select a next hop. This commonly keeps packets belonging to the same flow on the same path, reducing packet reordering. The exact hash inputs and resilience behaviour vary by platform.

ECMP does not necessarily divide every individual flow evenly. A large flow can dominate one path, and uneven flow sizes can create unequal utilisation even when the hashing mechanism is working correctly. Some platforms support additional techniques to improve distribution or reduce disruption when the set of next hops changes.

How BGP and other routing protocols support the fabric

A routed data center fabric needs a way to exchange reachability information. BGP is widely used in data center underlays because it provides policy controls, scalable route exchange and well-defined operational mechanisms. Other routing protocols, such as OSPF or IS-IS, can also be used depending on the design.

With an eBGP underlay, adjacent fabric devices in different autonomous systems exchange routes using external BGP sessions. Some designs use iBGP with a suitable route-reflection or control-plane architecture. The choice depends on the topology, operational model and requirements.

BGP multipath allows multiple paths to be installed when they satisfy the implementation's multipath eligibility rules and relevant configuration. Equal-cost forwarding is not guaranteed simply because BGP has learned more than one route: engineers must verify which paths were selected, which next hops were installed and how the forwarding hardware uses them.

Component Primary role What to verify
BGP or IGP Exchanges routes and establishes network reachability. Neighbour state, learned routes, route selection and next hops.
ECMP Uses multiple eligible equal-cost forwarding paths. Installed next hops, path eligibility and traffic distribution.
Forwarding hardware Applies the programmed forwarding decision to packets. Hardware programming, counters, drops and interface state.
SDN or automation layer Coordinates policy, configuration and operational workflows. Deployment status, device consistency and observed results.

Link failures and convergence

Redundant physical links do not automatically guarantee uninterrupted service. When a link or spine fails, the network must detect the failure, update relevant state and move traffic onto usable paths. The speed and impact of that process depend on the failure-detection method, routing protocol behaviour, hardware support and topology.

01
Detect the failure An interface may go down, a neighbour session may fail, or a configured liveness mechanism may detect loss of connectivity.
02
Update routing and forwarding state The device withdraws or invalidates affected reachability and adjusts the usable next-hop set according to its control-plane and forwarding implementation.
03
Use surviving paths Traffic can use remaining eligible paths if those paths have sufficient capacity and the destination remains reachable.
04
Validate service recovery Check route state, next-hop installation, packet loss, application connectivity and relevant telemetry rather than relying only on the interface status.

Some platforms support fast reroute or other mechanisms that can reduce interruption before full routing convergence completes. These features have specific design and support requirements. They should be validated under realistic failure conditions instead of assumed to be active by default.

Capacity planning and oversubscription

Fabric resilience is also a capacity question. A design may have multiple paths but still experience congestion if surviving links cannot carry the offered load after a failure.

Oversubscription compares the potential aggregate bandwidth offered by downstream links with the available upstream capacity. The acceptable ratio depends on the workloads. Traffic patterns dominated by short server-to-server exchanges can have different needs from storage traffic, distributed analytics or GPU clusters with heavy all-to-all communication.

Engineers should size the fabric for normal operation and consider what happens when one or more links or devices are unavailable. Monitoring utilisation, queue drops, latency and flow distribution helps establish whether the fabric behaves as designed.

Where SDN fits into fabric operations

SDN does not replace the physical fabric or make routing protocols unnecessary. Instead, programmable systems can standardise device provisioning, apply configuration templates, coordinate policy, collect telemetry and validate expected state.

For example, an automation system may deploy a new leaf switch with consistent interface settings, routing configuration and monitoring. The routing protocol still establishes reachability, while the automation layer checks that the device joins the fabric and receives the intended configuration.

This division of responsibilities supports repeatability while preserving the importance of routing fundamentals. If a workload cannot reach a destination, engineers still need to inspect the route, next hop, interface state, forwarding entry and any relevant policy or overlay dependency.

Section 02 takeaway

A routed leaf–spine fabric provides a scalable transport foundation. Routing establishes reachability, ECMP can use multiple eligible paths, and failure handling determines how traffic recovers. SDN and automation help coordinate and verify the design, but correct forwarding, capacity and resilience must still be proven through operational evidence.

Engineering Lab · Section 02
Leaf–Spine Path Selection and Link-Failure Investigation

In the interactive lab, trace traffic between two leaf switches through multiple spines. Inspect route selection and ECMP next hops, simulate a link failure, and compare the expected path with the surviving forwarding state. The investigation will distinguish routing convergence from application recovery and highlight how to verify whether remaining paths have sufficient capacity.

Section 03 · Network Virtualisation

VXLAN Overlays and Logical Network Segmentation

Virtualised networks allow multiple logical environments to share a physical fabric while maintaining defined connectivity boundaries. Follow the packet through the VXLAN encapsulation process, examine VTEPs and VNIs, and learn how underlay reachability, endpoint discovery, MTU and security policy affect end-to-end communication.

Underlay vs. overlay VXLAN and VTEPs Tenant isolation
03 · VIRTUALISATION · VXLAN · SEGMENTATION

Network Virtualisation: VXLAN, Overlays and Tenant Segmentation

Network virtualisation separates logical network design from the physical links and switches that carry traffic. It allows an infrastructure team to create multiple isolated network segments over a shared IP fabric, supporting multi-tenant environments, flexible workload placement and consistent policy.

Software-defined networking can help automate how these logical networks are created and managed. In many data centers, VXLAN provides the overlay encapsulation, while the routed IP underlay supplies connectivity between the devices that terminate those tunnels.

Underlay and overlay: two different responsibilities

The underlay is the physical or routed transport network. It provides IP reachability between fabric endpoints, including Virtual Tunnel Endpoints (VTEPs) when VXLAN is used.

The overlay is a logical network carried across that transport. VXLAN encapsulates an inner Ethernet frame inside an outer transport packet, allowing Layer 2 segments to extend across an IP network without requiring the underlay to behave as one large Layer 2 domain.

How an overlay packet travels
Source workload Inner Ethernet frame
→
Source VTEP Adds VXLAN and outer headers
→
IP underlay Routes outer packet
→
Destination VTEP Removes encapsulation
→
Destination workload Receives inner frame

Conceptual packet path. The underlay forwards using the outer IP header; the overlay identifies the logical segment and carries the original Ethernet frame.

What VXLAN contributes

VXLAN is an encapsulation technology, not a complete network automation or policy system by itself. It carries Ethernet frames over an IP network and uses a 24-bit VXLAN Network Identifier (VNI) to identify a logical Layer 2 segment.

VTEP

A VXLAN Tunnel Endpoint performs encapsulation and decapsulation. It maps local traffic to the appropriate VXLAN segment and sends encapsulated traffic toward the remote VTEP.

VNI

The VXLAN Network Identifier distinguishes logical VXLAN segments. A VNI is not the same thing as a VLAN ID, even though implementations can map VLANs to VNIs.

Outer IP header

Identifies the tunnel endpoints and enables the underlay routers to forward the encapsulated packet toward the destination VTEP.

Inner Ethernet frame

Carries the original workload traffic through the overlay. The destination VTEP decapsulates the packet and forwards the inner frame within the correct logical segment.

Engineering distinction

VXLAN provides encapsulation and segment identification. It does not, by itself, determine every endpoint location, distribute all required forwarding information, enforce a complete security policy or guarantee underlay reachability. Those responsibilities depend on the control-plane design, platform and surrounding network services.

How endpoints and VTEPs discover each other

A VTEP needs sufficient forwarding information to reach the correct remote VTEP and deliver traffic to the intended endpoint. Different VXLAN designs use different control-plane approaches.

Control-plane learning

Some deployments use data-plane learning, where information about remote endpoints is learned from traffic. Other designs use a control plane to distribute endpoint reachability before traffic needs to be forwarded. In BGP EVPN designs, BGP EVPN distributes MAC and IP reachability information and supports additional services according to the configured architecture.

A control plane can reduce dependence on flood-and-learn behaviour and improve how endpoint information is distributed across a fabric. However, correct operation still depends on consistent configuration, route exchange, VNI membership, endpoint learning and the availability of the underlying transport.

Tenant segmentation and policy

Virtual networks allow different applications, customers or organisational units to share physical infrastructure while maintaining logical separation. For example, production services, development workloads and shared infrastructure services can occupy different logical segments.

Segmentation is not automatically equivalent to a complete security boundary. Engineers must determine which communication is allowed between segments and implement the appropriate routing, gateway controls, distributed firewall rules, access policies or other enforcement mechanisms.

Design element Question it answers Typical verification
VLAN / VNI mapping Which logical segment does this traffic belong to? VLAN configuration, VNI mapping and segment membership.
VTEP reachability Can the source tunnel endpoint reach the destination VTEP? Underlay routes, next hops and IP connectivity.
Endpoint information Where should the destination MAC or IP be forwarded? MAC tables, ARP/ND state and control-plane advertisements.
Inter-segment policy Is communication between these segments permitted? Gateway routes, ACLs, firewall rules and policy counters.
MTU Can encapsulated packets traverse the underlay without unwanted fragmentation or drops? Interface MTU, encapsulation overhead and packet-drop counters.

Why the underlay MTU matters

VXLAN adds outer headers around the original Ethernet frame. This increases packet size. If the underlay MTU is not designed to accommodate the encapsulated packet, traffic may be fragmented where permitted or dropped, depending on the packet size, configuration and forwarding behaviour.

MTU problems can be deceptive: small packets may work while larger transfers fail or perform poorly. A reliable deployment validates the intended MTU end to end and checks counters and packet behaviour rather than assuming that successful ping tests prove every packet size will work.

How SDN and automation manage virtual networks

An SDN controller or automation platform can provide a consistent workflow for provisioning logical networks, mapping segments, distributing configuration and applying policy. Depending on the implementation, it may also integrate with virtualisation platforms, IP address management, inventory and telemetry systems.

The implementation model varies. Some systems use a central orchestration layer to coordinate configuration across devices; others rely on distributed control-plane protocols, local device decisions or a combination of both. A central management interface does not necessarily mean all packet-forwarding decisions are made by one central process.

Overlay troubleshooting: follow the packet in layers

1. Verify the workload Confirm the source and destination addressing, local interface, VLAN or segment attachment, and the intended security policy.
2. Verify the underlay Check that the source VTEP can reach the destination VTEP through the routed fabric.
3. Verify the overlay Inspect VNI membership, remote endpoint information, MAC learning and relevant EVPN routes where applicable.
4. Verify forwarding and policy Check tunnel state, forwarding entries, MTU, packet drops and any gateway or firewall rules affecting the flow.
Section 03 takeaway

Network virtualisation creates logical networks over shared transport. VXLAN carries Layer 2 frames across an IP underlay, while VTEPs, VNIs, endpoint reachability and policy determine how traffic is delivered. Troubleshooting should separate underlay reachability, overlay state and security enforcement instead of treating the virtual network as one opaque system.

Engineering Lab · Section 03
VXLAN Overlay Reachability and Tenant Isolation

The planned interactive lab will trace a packet from a source workload through its VTEP and routed underlay to a remote endpoint. Investigate a missing route, incorrect segment mapping or policy restriction, then identify which evidence proves whether the fault belongs to the underlay, overlay or security layer.

Network Insight / Interactive Engineering Lab 02

Leaf–Spine Fabrics, ECMP & Resilience

Explore a two-spine data centre fabric. Trace traffic from one leaf to another, inspect equal-cost paths, then disable a spine-facing link and observe how the available forwarding paths change.
Fabric healthy
Leaf–Spine Topology
Packets delivered: 0
Path A Path B SPINE LAYER LEAF LAYER SPINE 1 ECMP next hop Available SPINE 2 ECMP next hop Available LEAF 1 Source rack 2 uplinks 10.10.1.0/24 LEAF 2 Destination rack 2 uplinks 10.10.2.0/24 HOST A 10.10.1.10 HOST B 10.10.2.20 Conceptual topology: each leaf connects to each spine.
Selected path Remaining / alternate path Failed link
Fabric principle: every leaf connects to both spines. This creates multiple routed paths between racks without requiring leaf-to-leaf links.
Fabric Telemetry
Live model
Available Paths
2
Leaf 1 to Leaf 2
Delivered Packets
0
Simulated traffic
ECMP modeRound Robin
Current next hopNot selected
Spine 1 linkUp
Spine 2 linkUp
Last packetNot sent
ECMP Path Options
Next-hop candidates
Path A · Spine 1AVAILABLE
Host A → Leaf 1 → Spine 1 → Leaf 2 → Host B
Path B · Spine 2AVAILABLE
Host A → Leaf 1 → Spine 2 → Leaf 2 → Host B
ECMP: Equal-Cost Multi-Path allows routing to use multiple paths with equal routing cost. Actual traffic distribution depends on the forwarding platform and its hashing or load-sharing implementation.
Fabric Event Timeline
Packet and failure trace
01IngressHost A sends traffic to Host B through Leaf 1.
02RouteLeaf 1 has two equal-cost next-hop candidates: Spine 1 and Spine 2.
03ECMPThe simulated ECMP policy selects one available spine path.
04ForwardTraffic crosses the selected spine and reaches Leaf 2.
05FailureA spine-facing link is taken out of service in the lab.
06RebalanceThe failed candidate is excluded and the remaining path is selected.
Engineering Readout
Interpret the result
Current Event
Ready to send traffic
Send a packet to see the ECMP path selection model in action.
Trace Progress

Engineer Challenge

Send several packets, fail Spine 1's path, then send more traffic. Check whether the selected path avoids the failed link.

Challenge status: not yet attempted.
Engineering takeaway: redundant physical links provide path choices, but routing and forwarding must recognise failures before traffic can use the remaining path.
Section 04 · Programmable Operations

SDN Controllers, Automation and Policy-Based Operations

Network automation should deliver a verified outcome, not merely execute configuration commands. Explore the relationship between intent, controllers, APIs and distributed routing, then examine idempotency, controlled deployments, access security and the evidence required to confirm that a network change has succeeded.

Controllers and APIs Desired-state workflows Change validation
04 · CONTROLLERS · AUTOMATION · POLICY

SDN Controllers, Network Automation and Policy-Based Operations

The operational value of Software-Defined Networking is not simply that a network has a controller. It is the ability to express intent, translate that intent into consistent configuration or policy, and verify that the network behaves as expected. Controllers and automation systems can help make these workflows repeatable, observable and easier to manage at scale.

The exact implementation varies by platform. Some architectures use a controller to calculate or distribute network state; others combine central orchestration with distributed routing protocols and local forwarding decisions. Understanding that division is essential when designing resilient automation.

From network intent to verified forwarding

A useful operational model separates what the organisation wants the network to do from the configuration used to achieve it. An engineer might request connectivity between two application tiers, isolation for a tenant, or a resilient path to a service. The management system then translates the requirement into configuration, policy and validation checks.

STEP 01
Define intent
Specify connectivity, segmentation, performance and security requirements.
STEP 02
Translate policy
Map the requested outcome to device configuration, routes, services and rules.
STEP 03
Deploy changes
Apply configuration through supported APIs, protocols or management interfaces.
STEP 04
Verify outcome
Compare observed state with intended state and investigate any discrepancy.

What an SDN controller actually does

The term controller covers several different roles. A controller may maintain topology information, calculate paths, manage policy, coordinate provisioning or expose an API to orchestration tools. Not every controller performs every role, and not every SDN architecture relies on one controller for all control-plane decisions.

A simplified management and forwarding relationship
Applications and operators Business intent, workflows and service requirements
→
Controller / orchestrator Policy, coordination and supported APIs
→
Network devices Routing, local decisions and packet forwarding

Conceptual model only. Production systems may separate orchestration, control-plane services, device management and telemetry into several components. Data-plane forwarding normally continues locally even when a management interface is temporarily unavailable, subject to platform design.

Centralised management does not mean every decision is centralised

A common misunderstanding is that SDN always removes distributed control protocols. In practice, many systems use a central management layer while routing protocols such as BGP, OSPF or IS-IS continue to exchange reachability information between network devices.

Logical centralisation means that policy or control can be coordinated through a common abstraction. The underlying implementation may still be distributed across multiple controller instances and network devices for scale and resilience. The design must define which component owns each decision and what happens when components lose connectivity.

Central orchestration

Coordinates workflows, inventory, templates and service deployment across the infrastructure. It can help maintain consistent configuration but must handle partial failures and incomplete deployments.

Distributed routing

Network devices exchange reachability and make routing decisions according to protocol rules and local state. The routing system remains responsible for the behaviour defined by its configuration.

Local forwarding

Forwarding hardware or software applies installed forwarding entries to packets. A loss of management-plane access does not necessarily stop existing forwarding.

Telemetry and assurance

Monitoring systems collect operational evidence, detect anomalies and help compare actual network behaviour with intended service outcomes.

Programmable interfaces: APIs, models and device configuration

Programmability gives software a supported way to interact with network infrastructure. Depending on the platform, this may include REST APIs, NETCONF, RESTCONF, gNMI, YANG data models, command-line interfaces or vendor-specific management systems. These technologies are not interchangeable; each has its own capabilities, data models, transport and operational requirements.

A reliable automation workflow should avoid relying on fragile assumptions about device output or configuration syntax. Where supported, structured data models and APIs can make it easier to validate input, detect errors and compare configuration with expected state.

Mechanism Typical purpose Engineering consideration
REST API Integrate applications with controller or device services. Authentication, API versioning, error handling and request limits.
NETCONF / RESTCONF Access structured configuration and operational data where supported. Data-model support, transaction behaviour and device compatibility.
YANG Define structured data models for configuration and state. Model revisions and implementation-specific support.
gNMI Provide model-driven management and telemetry in supported systems. Subscription design, data volume and operational-state interpretation.
CLI automation Automate devices where command-line workflows are required. Prompt handling, parsing, idempotency and platform-specific syntax.

Configuration management and idempotency

An automation task should ideally be safe to run more than once. This property is known as idempotency: repeating an operation should not create unintended duplicate state or progressively change the result.

For example, an automation workflow should check whether a required VLAN, routing neighbour or policy object already exists before creating or modifying it. It should also distinguish between the configuration it intends to apply and the operational state the device actually reports.

Desired state is not proof of operational state

A successful API response can indicate that a request was accepted, but it does not by itself prove that the intended service is working. Confirm the resulting device configuration, protocol state, installed forwarding entries and end-to-end connectivity where appropriate.

Change control: validate before and after deployment

Network automation increases the speed and consistency of change, but it can also propagate an incorrect policy or configuration across many devices. Production workflows should therefore include validation, controlled rollout, logging and recovery procedures.

BEFORE
Pre-checks
Confirm inventory, dependencies, baseline state and permitted change scope.
CHANGE
Controlled rollout
Apply a limited change, with safeguards and clear failure handling.
AFTER
Post-checks
Verify configuration, routing, forwarding and service reachability.
RECOVERY
Rollback or repair
Restore a known-good state or apply a documented corrective action if checks fail.

Security and resilience of the management plane

Controllers and automation systems can have broad privileges. Their compromise or misconfiguration may affect a large part of the network. Protecting these systems is therefore part of the network architecture, not an optional operational detail.

Least privilege Give services and operators only the permissions required for their specific tasks.
Authentication and secrets Use supported authentication, protect credentials and rotate secrets according to operational policy.
Management-plane isolation Restrict management access to approved networks, identities and administrative paths.
Audit and recovery Record changes, monitor controller health and maintain tested recovery procedures for failed deployments.

How to troubleshoot an automation failure

When an automated change fails, avoid assuming that the controller is the only possible source of the problem. Work through the workflow and identify the first point where observed behaviour differs from the intended result.

Check the input request, policy translation, API response, device configuration, routing state, forwarding state and service-level result. This sequence helps distinguish a rejected request from a partially deployed change or a network that is configured correctly but cannot provide the expected service.

Section 04 takeaway

Controllers and automation systems coordinate intent, policy and configuration. Reliable SDN operations require more than issuing commands: they require idempotent workflows, access controls, change validation, operational telemetry and proof that the intended network outcome has been achieved.

Engineering Lab · Section 04
Policy Deployment, State Validation and Change Recovery

The planned interactive lab will follow a policy change from intent through deployment to observed network state. Inspect a failed or incomplete change, identify where the workflow diverged from the expected result, and determine which configuration, routing and forwarding evidence is needed before declaring the service healthy.

Network Insight / Interactive Engineering Lab 04

Fabric Capacity, Congestion & Resilience

Increase traffic demand, observe per-uplink utilisation, then remove an uplink and investigate how the remaining path absorbs the load.
Nominal capacity
Leaf–Spine Traffic Topology
Demand: 0 units
SPINE LAYER Redundant uplinks HOST ACCESS SPINE 1 Path A Available SPINE 2 Path B Available LEAF 1 Ingress 2 uplinks Rack A LEAF 2 Egress 2 uplinks Rack B HOST A Traffic source HOST B Traffic sink Illustrative capacity model: 100 units per uplink
Active traffic path Available alternate High utilisation Failed uplink
Model assumption: each leaf-to-spine uplink has a teaching capacity of 100 traffic units. Traffic is split across available paths for this exercise; this is a simplified capacity model, not a packet-level simulator or a benchmark of real hardware.
Capacity Dashboard
Illustrative units
Offered demand
0
Traffic units
Active capacity
200
Traffic units
Forwarded
0
Traffic units
Excess demand
0
Unserved in model
Path A · Spine 10 / 100
Path B · Spine 20 / 100
Overall utilisation0%
No traffic has been offered. Add demand to establish a baseline.
Operations Event Log
Latest event first
READYBoth spine paths are available. Aggregate illustrative capacity is 200 units.
Engineer Diagnosis
Observe · Compare · Explain
Current finding
The fabric is ready for a traffic test.
Increase demand in stages. Compare forwarded traffic with the available capacity, then repeat the experiment after removing an uplink.

Challenge: Survive the Uplink Failure

Add 60 units of demand, fail Leaf 1–Spine 1, then add another 60 units. Compare the available capacity before and after the failure.

Status: awaiting the initial traffic test.
Key concept: redundancy provides an alternate path, but it does not guarantee the remaining path has enough capacity to carry the full traffic load without congestion or drops.
Section 05 · Assurance and Troubleshooting

SDN Operations, Security and Intent-Based Networking

Reliable network operations depend on evidence that the service is working as intended. Learn how to correlate telemetry and events, isolate faults across physical, control and forwarding layers, secure programmable infrastructure, and use intent-based workflows to compare expected network behaviour with observed results.

Telemetry and assurance Incident investigation Intent-based networking
05 · OPERATIONS · ASSURANCE · SECURITY

SDN Operations, Troubleshooting, Security and Intent-Based Networking

A network is not successful simply because it has a controller, programmable interfaces or automated configuration. It is successful when it delivers the required connectivity, performance and security—and when engineers can prove that it continues to do so during changes and failures.

Effective SDN operations connect configuration management with observability, troubleshooting, security controls and service validation. This is where network automation becomes an operational discipline rather than just a way to issue commands faster.

From device monitoring to network assurance

Traditional monitoring often starts with individual device metrics: interface state, CPU, memory, routing neighbours and packet drops. These remain important, but they do not always explain whether an application can communicate with its intended destination.

Network assurance connects those individual signals to a broader question: does the network meet its intended service requirements? A useful assurance process compares expected state with observed state and gathers enough evidence to identify the source of a discrepancy.

01 · OBSERVE
Collect telemetry
Gather interface counters, routes, events, logs and performance measurements.
02 · CORRELATE
Build context
Relate device events to topology, services, policy and affected workloads.
03 · DIAGNOSE
Find the fault
Test hypotheses against routing, forwarding and end-to-end evidence.
04 · VALIDATE
Prove recovery
Confirm the intended service has recovered and check for remaining impact.

Build a layered troubleshooting method

When a service fails, begin with the observed symptom and follow the actual packet path. Avoid changing several configuration elements at once: doing so can hide the original cause and make recovery more difficult.

Physical and interface state

Check link status, errors, discards, optical levels where relevant, speed and duplex settings, and the health of connected devices.

Control-plane state

Verify routing neighbours, learned routes, selected paths, route policies and whether the expected prefixes are being advertised and received.

Forwarding state

Inspect the installed next hops, forwarding entries, adjacency resolution, tunnel state and counters to confirm that the data plane matches the routing intent.

Service and policy state

Test source-to-destination reachability, application ports, segmentation rules, firewall policy, DNS and other service dependencies relevant to the failure.

Operational principle

A healthy routing neighbour does not prove that a specific application flow will succeed. A route in the routing table does not prove that the correct next hop is installed in forwarding hardware. Each layer needs evidence appropriate to the question being investigated.

Telemetry, logs and event correlation

Telemetry can expose changes in interface utilisation, queue drops, route state, device health and other operational signals. Logs provide event context, while flow records and packet captures can help show where traffic is going or being lost.

These sources become more useful when they share reliable timestamps and are correlated with the topology and configuration in effect at the time. For example, a burst of packet loss may align with a link failure, a routing change or an automated policy deployment.

Correlation helps narrow the investigation, but timing alone does not prove causation. Engineers should verify the suspected relationship using route changes, interface counters, forwarding state or controlled reproduction where possible.

Security boundaries in an SDN environment

Programmability creates opportunities to enforce consistent policy, but it also increases the importance of securing the management plane and validating policy changes. An incorrectly scoped rule or compromised automation credential can have consequences across many devices or tenants.

Identity and access Authenticate users and services, apply least privilege and review administrative permissions.
Tenant separation Validate segment boundaries, permitted routes and the controls governing inter-tenant communication.
Change auditing Record who or what initiated a change, what was modified and whether post-change checks succeeded.
Resilience and recovery Protect controller access, monitor critical services and test recovery procedures before they are needed.

Intent-based networking: intent, policy and assurance

Intent-based networking aims to express desired network outcomes at a higher level than individual device commands. Instead of manually configuring every relevant device, an operator defines an outcome—such as approved connectivity between application tiers—and a system translates it into supported configuration and policy.

A more complete intent-based workflow also checks whether the requested outcome has been achieved. It may detect a mismatch, explain the affected components and recommend or perform a corrective action within approved safeguards.

The label alone does not guarantee autonomous correction or closed-loop operation. Capabilities vary by platform, and remediation should be bounded by policy, testing, approval requirements and a clear rollback strategy.

INTENT
Define the outcome
Specify the service, security and performance requirements.
TRANSLATE
Derive the policy
Convert the outcome into supported configuration and network controls.
ASSURE
Compare actual state
Measure operational evidence against the intended outcome.
IMPROVE
Correct safely
Investigate deviations and apply authorised corrective action when appropriate.

Common failure patterns and the evidence to collect

Symptom Possible causes Evidence to inspect
Configuration deployment fails Authentication, schema mismatch, invalid input or unsupported feature. API response, controller logs, validation output and device configuration.
Route exists but traffic fails Incorrect next hop, unresolved adjacency, ACL, MTU or forwarding issue. Routing table, forwarding entries, counters and packet-path tests.
Service fails after a change Policy mismatch, partial deployment or unexpected dependency. Change audit, before-and-after state, affected routes and service tests.
Traffic loss after link failure Convergence delay, insufficient remaining capacity or missing alternate path. Interface events, neighbour state, route updates, ECMP next hops and drops.
Unexpected cross-segment access Incorrect segmentation or overly permissive policy. VLAN/VNI mapping, gateway routing, firewall rules and access-policy counters.

Measure operational success

Automation success should be measured by more than the number of commands executed or devices configured. Useful measures include deployment success rate, change failure rate, recovery time, configuration drift, service availability and the time needed to diagnose a fault.

Baselines matter. Record normal behaviour before introducing a new workflow, then compare results over a meaningful period. A reduction in manual effort is valuable, but not if it comes at the cost of less reliable change or weaker security.

Section 05 takeaway

Mature SDN operations combine programmability with observability, layered troubleshooting, security controls and measurable assurance. Intent-based workflows can help close the loop between desired and observed network behaviour, but reliable outcomes still depend on sound architecture, trustworthy evidence and controlled remediation.

Engineering Lab · Section 05
Network Incident Investigation and Assurance

The planned interactive lab will present a network service incident with multiple possible causes. Inspect topology, event history, routing state, forwarding evidence and policy before selecting a diagnosis. The objective is to build a defensible explanation of the failure, identify the corrective action and verify recovery using observed network state.

Network Insight / Interactive Engineering Lab 05

ECMP Path Selection & Flow Distribution

Explore how multiple flows are assigned to equal-cost paths. Compare deterministic flow hashing with a simplified round-robin model, then remove a spine path and inspect the reassignment.
Two paths available
ECMP Forwarding Topology
0 active flows
SPINE LAYER Equal-cost paths across the fabric SPINE 1 ECMP Path A Available SPINE 2 ECMP Path B Available LEAF 1 Ingress ECMP next hops 2 available LEAF 2 Egress Destination prefix 10.20.0.0/24 HOST A Flow source HOST B Flow destination Conceptual flow assignment · not packet-level emulation
Path A · Spine 1 Path B · Spine 2 Failed path
Important: ECMP normally distributes flows using a hash over selected packet-header fields. A flow generally remains on one next hop to avoid excessive reordering. The round-robin option here is a teaching comparison, not a claim that production ECMP typically schedules individual packets this way.
Distribution Summary
Current flow assignments
Active flows
0
Generated sessions
Available paths
2
Equal-cost next hops
Path A flows
0
Via Spine 1
Path B flows
0
Via Spine 2
Add flows to observe how the selected distribution method assigns next hops.
Engineering note: an even flow count does not guarantee equal bandwidth. A single high-volume flow can dominate a path even when the number of assigned flows is balanced.
Flow Assignment Table
Up to 24 flows displayed
No flows yet. Select “Add 4 Flows” to begin the experiment.
Engineer Diagnosis
Inspect · Compare · Explain
Current finding
The fabric has two available equal-cost paths.
Generate several flows and compare their next-hop assignments. Then fail one path and observe the simplified reassignment behaviour.

Challenge: Explain ECMP Rebalancing

Generate at least 8 flows using flow hashing. Fail Spine 1's path and inspect the assignments. Switch to round-robin and repeat. Explain why a path failure can cause flows to move and why production devices may use resilient hashing to reduce unnecessary remapping.

Status: generate at least 8 flows to begin.
Remember: ECMP chooses among eligible equal-cost next hops. Routing determines which paths are available; forwarding hardware or software performs the per-flow selection.
Network Insight · Engineering Conclusion

Conclusion: Building and Operating an SDN Data Centre

Software-defined networking is more than a programmable network interface. It is an architectural approach that brings together scalable forwarding, network virtualisation, centralised policy, automation and evidence-driven operations.

Throughout this guide, we have examined how SDN changes the way data centre networks are designed, controlled and operated. The central engineering lesson is that programmability creates value only when it is supported by sound architecture, predictable forwarding, reliable state, and effective verification.

Five engineering principles to take forward

01
Design the architecture first

Separate forwarding responsibilities from control and management functions where appropriate. Define clear interfaces, failure boundaries and operational ownership before introducing automation.

02
Engineer for scale and resilience

Leaf–spine fabrics, ECMP and multiple paths can provide scalable east–west connectivity. Validate capacity, flow distribution, convergence and the impact of link or device failures.

03
Make virtual networks understandable

VXLAN overlays and tenant segmentation provide logical separation over shared infrastructure. Correct VTEP reachability, VNI mappings, endpoint learning and policy enforcement remain essential.

04
Automate with explicit intent

Controllers, APIs and configuration pipelines should translate defined requirements into controlled changes. Validate inputs, manage dependencies, check results and maintain a clear rollback path.

05
Operate from evidence

Telemetry, logs, counters, flow data and state inspection help engineers distinguish a control-plane issue from a forwarding, overlay, policy or application problem.

06
Secure the complete system

Protect controller access, APIs, credentials and automation pipelines. Apply least privilege, segment management traffic, audit changes and test failure recovery as part of routine operations.

From design principles to operational practice

A successful SDN deployment should be evaluated against measurable outcomes: predictable forwarding, adequate capacity, reliable convergence, effective segmentation, repeatable changes and recoverable failures. A topology diagram or successful configuration alone does not prove that the network is behaving as intended.

Engineers should test both normal operation and failure conditions. Introduce controlled faults, observe how state changes across the fabric, verify the resulting forwarding path and compare the outcome with the original design intent. This turns SDN from a collection of technologies into an operational system that can be understood and validated.

The engineering takeaway

Build for predictable behaviour, automate with guardrails, and verify every important outcome. The strongest SDN data centre is not simply the one with the most automation; it is the one whose architecture, policies, forwarding and recovery behaviour can be explained and tested.

Practical review checklist

  • Can you explain how control, management and forwarding responsibilities interact?
  • Have you tested ECMP distribution, available capacity and single-link or device failures?
  • Can you trace an endpoint across the overlay and verify its tenant and policy context?
  • Are automated changes validated, observable, auditable and reversible?
  • Can the operations team diagnose a fault using evidence rather than assumptions?

SDN continues to evolve alongside cloud-native infrastructure, network APIs, intent-based operations and AI-assisted data centre management. The underlying engineering disciplines remain the same: understand the architecture, observe the system, test the failure modes and verify that the network delivers the behaviour its users depend on.

Use these principles as a foundation for evaluating new designs, investigating production incidents and developing more reliable, programmable network infrastructure.

```
☕
Support Network Insight
Help support independent engineering guides, interactive networking tools, and future learning labs.
Support Network Insight ```