container based virtualization

Cisco Switch Virtualization Nexus 1000v

Cisco Switch Virtualization Nexus 1000v

Virtualization has become integral to modern data centers in today's digital landscape. With the increasing demand for agility, flexibility, and scalability, organizations are turning to virtual networking solutions to meet their evolving needs. One such solution is the Nexus 1000v, a virtual network switch offering comprehensive features and functionalities. In this blog post, we will delve into the world of the Nexus 1000v, exploring its key features, benefits, and use cases.

The Nexus 1000v is a distributed virtual switch that operates at the hypervisor level, providing advanced networking capabilities for virtual machines (VMs). It is designed to integrate seamlessly with VMware vSphere, offering enhanced network visibility, control, and security.

Cisco Switch Virtualization is a revolutionary concept that allows network administrators to create multiple virtual switches on a single physical switch. By abstracting the network functions from the hardware, it provides enhanced flexibility, scalability, and efficiency. With Cisco Switch Virtualization, businesses can maximize resource utilization and simplify network management.

At the forefront of Cisco's Switch Virtualization portfolio is the Nexus 1000v. This powerful platform brings the benefits of virtualization to the data center, enabling seamless integration between virtual and physical networks. By extending Cisco's renowned networking capabilities into the virtual environment, Nexus 1000v empowers organizations to achieve consistent policy enforcement, enhanced security, and simplified operations.

The Nexus 1000v boasts a wide range of features that make it a compelling choice for network administrators. From advanced network segmentation and traffic isolation to granular policy control and deep visibility, this platform has it all. By leveraging the power of Cisco's Virtual Network Services (VNS), organizations can optimize their network infrastructure, streamline operations, and deliver superior performance.

Deploying Cisco Switch Virtualization, specifically the Nexus 1000v, requires careful planning and consideration. Organizations must evaluate their network requirements, ensure compatibility with existing infrastructure, and adhere to best practices. From designing a scalable architecture to implementing proper security measures, attention to detail is crucial to achieve a successful deployment.

To truly understand the impact of Cisco Switch Virtualization, it's essential to explore real-world use cases and success stories. From large enterprises to service providers, organizations across various industries have leveraged the power of Nexus 1000v to transform their networks. This section will highlight a few compelling examples, showcasing the versatility and value that Cisco Switch Virtualization brings to the table.

Highlights: Cisco Switch Virtualization Nexus 1000v

Hypervisor and vSphere Introduction

An operating system can run multiple operating systems on a single hardware host using a hypervisor, also known as a virtual machine manager. Operating systems use the host’s processor, memory, and other resources. Hypervisors control the host processor, memory, and other resources and allocate what each operating system needs. Hypervisors run guest operating systems or virtual machines on top of them.

Designed specifically for integration with VMware vSphere environments, the Cisco Nexus 1000V Series Switch runs Cisco NX-OS software. Enterprise-class performance, scalability, and scalability are delivered by VMware vSphere 2.0 across multiple platforms. Within the VMware ESX hypervisor, the Nexus 1000V runs. With the Cisco Nexus 1000V Series, you can take advantage of Cisco VN-Link server virtualization technology

• Policy-based virtual machine (VM) connectivity

• Mobile VM security

• Network policy

• Nondisruptive operational model for your server virtualization and networking teams

As with physical servers, virtual servers can be configured with the same network configuration, security policy, diagnostic tools, and operational models as physical servers. The Cisco Nexus 1000V Series is also compatible with VMware vSphere, vCenter, ESX, and ESXi.

A brief overview of the Nexus 1000V system

There are two primary components of the Cisco Nexus 1000V Series switch:

  • VEM (Virtual Ethernet Module): Executes inside hypervisors
  • VSM (External Virtual Supervisor Module): Manages VEMs

Nexus 1000v implements a generic concept of Cisco Distributed Virtual Switch (DVS). VMware ESX or ESXi executes the Cisco Nexus 1000V Virtual Ethernet Module (VEM). The VEM’s application programming interface (API) is VMware vNetwork Distributed Switch (vDS).

By integrating the API with VMware VMotion and Distributed Resource Scheduler (DRS), advanced networking capabilities can be provided to virtual machines. In the VEM, Layer 2 switching and advanced networking functions are performed based on configuration information from the VSM:

Nexus Switch Virtualization

**Virtual routing and forwarding**

Virtual routing and forwarding form the basis of this stack. Firstly, network virtualization comes with two primary methods: 1) One too many and 2) Many to one.  The “one too many” network virtualization method means you segment one physical network into multiple logical segments. Conversely, the “many to one” network virtualization method consolidates numerous physical devices into one logical entity. By definition, they seem to be opposites, but they fall under the same umbrella in network virtualization.

Before you proceed, you may find the following posts helpful:

  1. Container Based Virtualization
  2. Virtual Switch
  3. What is VXLAN
  4. Redundant Links
  5. WAN Virtualization
  6. What Is FabricPath

Network virtualization

Before we get stuck in Cisco virtualization, let us address some basics. For example, if you have multiple virtual endpoints share a physical network. Still, different virtual endpoints belong to various customers, and the communication between these endpoints also needs to be isolated. In other words, the network is a resource, too, and network virtualization is the technology that enables the sharing of a standard physical network infrastructure.

Virtualization uses software to simulate traditional hardware platforms and create virtual software-based systems. For example, virtualization allows specialists to construct a single virtual network or partition a physical network into multiple virtual networks.

Cisco Switch Virtualization: Logical segmentation: One too many

We have one-to-many network virtualization for the Cisco switch virtualization design; a single physical network is logically segmented into multiple virtual networks. For example, each virtual network could correspond to a user group or a specific security function.

End-to-end path isolation requires the virtualization of networking devices and their interconnecting links. VLANs have been traditionally used, and hosts from one user group are mapped to a single VLAN. To extend the path across multiple switches at Layer 2, VLAN tagging (802.1Q) can carry VLAN information between switches. These VLAN trunks were created to transport multiple VLANs over a single Ethernet interface.

The diagram below displays two independent VLANs, VLAN201 and VLAN101. These VLANs can share one physical wire to provide L2 reachability between hosts connected to Switch B and Switch A via Switch C, but they remain separate entities.

Nexus1000v
Nexus1000v: The operation

VLANs are sufficient for small Layer 2 segments. However, today’s networks will likely have a mix of Layer 2 and 3 routed networks. In this case, Layer 2 VLANs alone are insufficient because you must extend the Layer 2 isolation over a Layer 3 device. This can be achieved by using Virtual Routing and Forwarding ( VRF ), the next step in the Cisco switch virtualization. A virtual routing and forwarding instance logically carves a Layer 3 device into several isolated independent L3 devices. The virtual routing and forwarding configured locally cannot communicate directly.

The diagram below displays one physical Layer 3 router with three VRFs: VRF Yellow, VRF Red, and VRF Blue. These virtual routing and forwarding instances are completely separated; without explicit configuration, routes in one virtual routing and forwarding instance cannot be leaked to another.

Virtual Routing and Forwarding

virtual routing and forwarding

The virtualization of the interconnecting links depends on how the virtual routers are connected. If they are physically ( directly ) connected, you could use a technology known as VRF-lite to separate traffic and 802.1Q to label the data plane. This is known as hop-by-hop virtualization.

However, it’s possible to run into scalability issues when the number of devices grows. This design is typically used when you connect virtual routing and forwarding back to back, i.e., no more than two devices.

When the virtual routers are connected over multiple hops through an IP cloud, you can use generic routing encapsulation ( GRE ) or Multiprotocol Label Switching ( MPLS ) virtual private networks.

GRE is probably the simpler of the Layer 3 methods, and it can work over any IP core. GRE can encapsulate the contents and transport them over a network with the network unaware of the packet contents. Instead, the core will see the GRE header, virtualizing the network path.

Cisco Switch Virtualization: The additional overhead

When designing Cisco switch virtualization, you need to consider the additional overhead. There are a further 24 bytes overhead for the GRE header, so it may be the case that the forwarding router may break the datagram into two fragments, so the packet may not be larger than the outgoing interface MTU. To resolve the fragmentation issue, you can correctly configure MTU, MSS, and Path MTU parameters on the outgoing and intermediate routers.

The GRE standard is typically static. You only need to configure tunnel endpoints, and the tunnel will be up as long as you can reach those endpoints. However, recent designs can establish a dynamic GRE tunnel.

GRE over IPsec

MPLS/VPN, on the other hand, is a different beast. It requires signaling to distribute labels and build an end-to-end Label Switched Path ( LSP ). The label distribution can be done with BGP+label, LDP, and RSVP. Unlike GRE tunnels, MPLS VPNs do not have to manage multiple point-to-point tunnels to provide a full mesh of connectivity. Instead, they are used for connectivity, and packets’ labels provide traffic separation.

Cisco switch virtualization: Many to one

Many-to-one network consolidation refers to grouping two or more physical devices into one. Examples of this Cisco switch virtualization technology include a Virtual Switching System ( VSS ), Stackable switches, and Nexus VPC. Combining many physicals into one logical entity allows STP to view the logical group as one, allowing all ports to be active. By default, STP will block the redundant path.

Software-defined networking takes this concept further; it completely abstracts the entire network into a single virtual switch. The control and data planes are on the same device on traditional routers, yet they are decoupled with SDN. The control plan is now on a policy-driven controller, and the data plane is local on the OpenFlow-enabled switch.

Network Virtualization

Server and network virtualization presented the challenge of multiple VMs sharing a single network physical port, such as a network interface controller ( NIC ). The question then arises: how do I link multiple VMs to the same uplink? How do I provide path separation? Today’s networks need to virtualize the physical port and allow the configuration of policies per port.

Nexus 1000

NIC-per-VM design

One way to do this is to have a NIC-per-VM design where each VM is assigned a single physical NIC, and the NIC is not shared with any other VM. The hypervisor, aka virtualization layer, would be bypassed, and the VM would access the I/O device directly. This is known as VMDirectPath.

This direct path or pass-through can improve performance for hosts that utilize high-speed I/O devices, such as 10 Gigabit Ethernet. However, the lack of flexibility and the ability to move VMs offset higher performance benefits.  

Virtual-NIC-per-VM in Cisco UCS (Adapter FEX)

Another way to do this is to create multiple logical NICs on the same physical NIC, such as Virtual-NIC-per-VM in Cisco UCS (Adapter FEX). These logical NICs are assigned directly to VMs, and traffic gets marked with a vNIC-specific tag on the hardware (VN-Tag/802.1ah).

The actual VN-Tag tagging is implemented in the server NICs so that you can clone the physical NIC in the server to multiple virtual NICs. This technology provides faster switching and enables you to apply a rich set of management features to local and remote traffic.

Software Virtual Switch

The third option is to implement a virtual software switch in the hypervisor. For example, VMware introduced virtual switching compatibility with its vSphere ( ESXi ) hypervisor, called vSphere Distributed Switch ( VDS ). Initially, they introduced a local L2 software switch, which was soon phased out due to a lack of distributed architecture.

Data physically moves between the servers through the external network, but the control plane abstracts this movement to look like one large distributed switch spanning multiple servers. This approach has a single management and configuration point, similar to stackable switches – one control plane with many physical data forwarding paths.

The data does not move through a parent partition but logically connects directly to the network interface through local vNICs associated with each VM.

Network virtualization and Nexus 1000v ( Nexus 1000 )

The VDS introduced by VMware lacked any good networking features, which led Cisco to introduce the Nexus 1000V software-based switch. The Nexus 1000v is a multi-cloud, multi-hypervisor, and multi-services distributed virtual switch. Its function is to enable communication between VMs.

Nexus1000v
Nexus1000v: Virtual Distributed Switch.

**Nexus 1000 components: VEM and VSM**

The Nexus 1000v has two essential components:

  1. The Virtual Supervisor Module ( VSM )
  2. The Virtual Ethernet Module ( VEM ).

Compared to a physical switch, the VSM could be viewed as the supervisor, setting up the control plane functions for the data plane to forward efficiently, and the VEM as the physical line cards that do all the packet forwarding. The VEM is the software component that runs within the hypervisor kernel. It handles all VM traffic, including inter-VM frames and Ethernet traffic between a VM and external resources.

The VSM runs its NX-OS code and controls the control and management planes, which integrate into a cloud manager, such as a VMware vCenter. You can have two VSMs for redundancy. Both modules remain constantly synchronized with unicast VSM-to-VSM heartbeats to provide stateful failover in the event of an active VSM failure.

The two available communication options for VSM to VEM are:

  1. Layer 2 control mode: The VSM control interface shares the same VLAN with the VEM.
  2. Layer 3 control mode: The VEM and the VSM are in different IP subnets.

The VSM also uses heartbeat messages to detect a loss of connectivity between it and the VEM. However, the VEM does not depend on connectivity to the VSM to perform its data plane functions and will continue forwarding packets if the VSM fails.

With Layer 3 control mode, the heartbeat messages are encapsulated in a GRE envelope.

Nexus 1000 and VSM best practices

  • L2 control is recommended for new installations.
  • Use MAC pinning instead of LACP.
  • Packet, Control, and Management in the same VLAN.
  • Do not use VLAN 1 for Control and Packet.
  • Use 2 x VSM for redundancy. 

The max latency between VSM and VEM is ten milliseconds. Therefore, a VSM can be placed outside the data center if you have a high-quality DCI link, and the VEM can still be controlled.

Nexus 1000v InterCloud – Cisco switch virtualization

A vital element of the Nexus 1000 is its use case for hybrid cloud deployments and its ability to place workloads in private and public environments via a single pane of glass. In addition, the Nexus 1000v interCloud addresses the main challenges with hybrid cloud deployments, such as security concerns and control/visibility challenges within the public cloud.

The Nexus 1000 interCloud works with Cisco Prime Service Controller to create a secure L2 extension between the private data center and the public cloud.

This L2 extension is based on Datagram Transport Layer Security ( DTLS ) protocol and allows you to securely transfer VMs and Network services over a public IP backbone. DTLS derives the SSL protocol and provides communications privacy for datagram protocols, so all data in motion is cryptographically isolated and encrypted.

Nexus 1000
Nexus 1000 and Hybrid Cloud.

 

Nexus 1000v Hybrid Cloud Components 

Cisco Prime Network Service Controller for InterCloud **A VM that provides a single pane of glass to manage all functions of the inter clouds
InterCloud VSMManage port profiles for VMs in the InterCloud infrastructure
InterCloud ExtenderProvides secure connectivity to the InterCloud Switch in the provider cloud. Install in the private data center.
InterCloud SwitchVirtual Machine in the provider data center has secure connectivity to the InterCloud Extender in the enterprise cloud and secure connectivity to the Virtual Machines in the provider cloud.
Cloud Virtual MachinesVMs in the public cloud running workloads.

Prerequisites

Port 80HTTP access from PNSC for AWS calls and communicating with InterCloud VMs in the provider cloud
Port 443HTTPS access from PNSC for AWS calls and communicating with InterCloud VMs in the provider cloud
Port 22SSH from PNSC to InterCloud VMs in the provider cloud
UDP 6644DTLS data tunnel
TCP 6644DTLS control tunnel

VXLAN – Virtual Extensible LAN

The requirement for applications on demand has led to an increased number of required VLANs for cloud providers. The standard 12-bit identifier, which provided 4000 VLANs, proved to be a limiting factor in multi-tier, multi-tenant environments, and engineers started to run out of isolation options.

This has introduced a 24-bit VXLAN identifier, offering 16 million logical networks. Now, we can cross Layer 3 boundaries. The MAC in UDP encapsulation uses switch hashing to analyze UDP packets and efficiently distribute all packets in a port channel.

nexus 1000
VXLAN operations

VXLAN works like a layer 2 bridge ( Flood and Learn ); the VEM learn does all the heavy lifting, learns all the VM source MAC and Host VXLAN IPs, and encapsulates the traffic according to the port profile to which the VM belongs. Broadcast, Multicast, and unknown unicast traffic are sent as Multicast.

At the same time, unicast traffic is encapsulated and shipped directly to the destination host’s VXLAN IP, aka destination VEM. Enhanced VXLAN offers VXLAN MAC distribution and ARP termination, making it more optional. 

VXLAN Mode Packet Functions

PacketVXLAN(multicast mode)Enhanced VXLAN(unicast mode)Enhanced VXLANMAC DistributionEnhanced VXLANARP Termination
Broadcast /MulticastMulticast EncapsulationReplication plus Unicast EncapReplication plus Unicast EncapReplication plus Unicast Encap
Unknown UnicastMulticast EncapsulationReplication plus Unicast EncapDropDrop
Known UnicastUnicast EncapsulationUnicast EncapUnicast EncapUnicast Encap
ARPMulticast EncapsulationReplication plus Unicast EncapReplication plus Unicast EncapVEM ARP Reply

vPath – Service chaining

Intelligent Policy-based traffic steering through multiple network services.

vPath allows you to intelligently traffic steer VM traffic to virtualized devices. It intercepts and redirects the initial traffic to the service node. Once the service node performs its policy function, the result is cached, and the local virtual switch treats the subsequent packets accordingly. In addition, it enables you to tie services together to push the VM through each service as required. Previously, if you wanted to tie services together in a data center, you needed to stitch the VLANs together, which was limited by design and scale.

Cisco virtualization
Nexus and service chaining

vPath 3.0 is now submitted to the IETF for standardization, allowing service chaining with vPath and non-vpath network services. It enables you to use vpath service chaining between multiple physical devices and supporting multiple hypervisors.

License Options 

Nexus 1000 Essential EditionNexus 1000 Advanced Edition
Full Layer-2 Feature SetAll Features of Essential Edition
Security, QoS PoliciesVSG firewall
VXLAN virtual overlaysVXLAN Gateway
vPath enabled Virtual ServicesTrustSec SGA
Full monitoring and management capabilitiesA platform for other Cisco DC Extensions in the Future
Free$695 per CPU MSRP

Nexus 1000 features and benefits

SwitchingL2 Switching, 802.1Q Tagging, VLAN, Rate Limiting (TX), VXLAN
IGMP Snooping, QoS Marking (COS & DSCP), Class-based WFQ
SecurityPolicy Mobility, Private VLANs w/ local PVLAN Enforcement
Access Control Lists, Port Security, Cisco TrustSec Support
Dynamic ARP inspection, IP Source Guard, DHCP Snooping
Network ServicesVirtual Services Datapath (vPath) support for traffic steering & fast-path off-load[leveraged by Virtual Security Gateway (VSG), vWAAS, ASA1000V]
ProvisioningPort Profiles, Integration with vC, vCD, SCVMM*, BMC CLM
Optimized NIC Teaming with Virtual Port Channel – Host Mode
VisibilityVM Migration Tracking, VC Plugin, NetFlow v.9 w/ NDE, CDP v.2
VM-Level Interface Statistics, vTrackerSPAN & ERSPAN (policy-based)
ManagementVirtual Centre VM Provisioning, vCenter Plugin, Cisco LMS, DCNM
Cisco CLI, Radius, TACACs, Syslog, SNMP (v.1, 2, 3)
Hitless upgrade, SW Installer

Advantages and disadvantages of the Nexus 1000

AdvantagesDisadvantages
The Standard edition is FREE; you can upgrade to an enhanced version when needed.VEM and VSM internal communication is very sensitive to latency. Due to their chatty nature, they may not be good for inter-DC deployments.
Easy and Quick to deployVSM – VEM, VSM (active) – VSM (standby) heartbeat time of 6 seconds makes it sensitive to network failures and congestion.
It offers you many rich network features unavailable on other distributed software switches.VEM over-dependency on VSM reduces resiliency.
Hypervisor agnosticVSM is required for vSphere HA, FT, and VMotion to work.
Hybrid Cloud functionality 

**Closing Points on Cisco Nexus 1000v**

Virtual Ethernet Module (VEM):

The Nexus 1000v employs the Virtual Ethernet Module (VEM), which runs as a module inside the hypervisor. This allows for efficient and direct communication between VMs, bypassing the traditional reliance on the hypervisor networking stack.

Virtual Supervisor Module (VSM):

The Virtual Supervisor Module (VSM) serves as the control plane for the Nexus 1000v, providing centralized management and configuration. It enables network administrators to define policies, manage virtual ports, and monitor network traffic.

Policy-Based Virtual Network Management:

With the Nexus 1000v, administrators can define policies to manage virtual networks. These policies ensure consistent network configurations across multiple hosts, simplifying network management and reducing the risk of misconfigurations.

Advanced Security and Monitoring Capabilities:

The Nexus 1000v offers granular security controls, including access control lists (ACLs), port security, and dynamic host configuration protocol (DHCP) snooping. Additionally, it provides comprehensive visibility into network traffic, enabling administrators to monitor and troubleshoot network issues effectively.

Benefits of the Nexus 1000v:

Enhanced Network Performance:

By offloading network processing to the VEM, the Nexus 1000v minimizes the impact on the hypervisor, resulting in improved network performance and reduced latency.

Increased Scalability:

The distributed architecture of the Nexus 1000v allows for seamless scalability, ensuring that organizations can meet the growing demands of their virtualized environments.

Simplified Network Management:

With its policy-based approach, the Nexus 1000v simplifies network management tasks, enabling administrators to provision and manage virtual networks more efficiently.

Use Cases:

Data Centers:

The Nexus 1000v is particularly beneficial in data center environments where virtualization is prevalent. It provides a robust and scalable networking solution, ensuring optimal performance and security for virtualized workloads.

Cloud Service Providers:

Cloud service providers can leverage the Nexus 1000v to enhance their network virtualization capabilities, offering customers more flexibility and control over their virtual networks.

The Nexus 1000v is a powerful virtual network switch that provides advanced networking capabilities for virtualized environments. Its rich features, policy-based management approach, and seamless integration with VMware vSphere allow organizations to achieve enhanced network performance, scalability, and management efficiency. As virtualization continues to shape the future of data centers, the Nexus 1000v remains a valuable tool for optimizing virtual network infrastructures.

Summary: Cisco Switch Virtualization Nexus 1000v

Welcome to our blog post, where we dive into the world of Cisco Switch Virtualization, explicitly focusing on the Nexus 1000v. In this article, we will unravel the complexities surrounding switch virtualization, explore the key features of the Nexus 1000v, and understand its significance in modern networking environments.

Understanding Switch Virtualization

Switch virtualization is a technique that allows for creating multiple virtual switches on a single physical switch, enabling greater flexibility and efficiency in network management. Organizations can consolidate their infrastructure, reduce costs, and streamline network operations by virtualizing switches.

Introducing the Nexus 1000v

The Cisco Nexus 1000v is a powerful switch virtualization solution that extends the functionality of VMware environments. Unlike traditional virtual switches, it provides a more comprehensive set of features and advanced network control. It seamlessly integrates with VMware vSphere, offering enhanced visibility, security, and policy management.

Key Features of the Nexus 1000v

– Distributed Virtual Switch: The Nexus 1000v operates as a distributed virtual switch, distributing network intelligence across all hosts in the virtualized environment. This ensures consistent policies, simplified troubleshooting, and improved performance.

– Virtual Port Profiles: With virtual port profiles, administrators can define consistent network policies for virtual machines, irrespective of their physical location. This simplifies network provisioning and reduces the chances of misconfigurations.

– Network Analysis Module (NAM): The Nexus 1000v incorporates NAM, a robust monitoring and analysis tool that provides deep visibility into virtual network traffic. This enables administrators to identify and resolve network issues, ensuring optimal performance quickly.

Deployment Considerations

When planning to deploy the Nexus 1000v, it is essential to consider factors such as network architecture, compatibility with existing infrastructure, and scalability requirements. It is advisable to consult with Cisco experts or certified partners to ensure a smooth and successful implementation.

Conclusion:

In conclusion, the Cisco Nexus 1000v is a game-changer in switch virtualization. Its advanced features, seamless integration with VMware environments, and extensive network control make it an ideal choice for organizations seeking to optimize their network infrastructure. By understanding the fundamentals of switch virtualization and exploring Nexus 1000v’s capabilities, network administrators can unlock a world of possibilities in network management and performance.

SDN Data Center

Redundant links with Virtual PortChannels

Data Centre Networking · Resilience · Routing

Redundant Links with Virtual PortChannels

Modern data centres need more than a backup connection. They need resilient paths, effective bandwidth utilisation and predictable behaviour when links or switches fail. Virtual PortChannels (vPC) help achieve active-active Layer 2 connectivity by allowing a downstream device to form a port-channel across two compatible Cisco Nexus switches.

Traditional designs can leave redundant paths blocked by Spanning Tree Protocol (STP), while a poorly planned aggregation design can create uneven traffic distribution or unexpected failure conditions. Port channels, LACP and multi-chassis link aggregation provide different mechanisms for building resilient connectivity—but each has its own operational requirements and failure modes.

This guide examines how these mechanisms work, how to verify them, and how to investigate what happens when a member link, peer-link or switch becomes unavailable. It also connects Layer 2 redundancy to Layer 3 forwarding and BGP, where routing decisions and equal-cost multipath (ECMP) can provide additional resilience across a data-centre fabric.

What you will learn

01 / FOUNDATIONS

Port-channels and LACP

Understand physical member links, logical interfaces, negotiation modes and link-level redundancy.

02 / ARCHITECTURE

vPC operation

Explore peer-links, keepalive connectivity, consistency checks, primary and secondary roles, and orphan ports.

03 / TRAFFIC

Hashing and load balancing

Learn why multiple active links do not guarantee perfectly even traffic distribution and how to identify polarization.

04 / ROUTING

BGP and ECMP

Connect resilient Layer 2 access with routed leaf-spine fabrics, BGP path selection, next-hop reachability and multipath forwarding.

05 / OPERATIONS

Failure investigation

Trace link failures, peer-link loss, routing changes and recovery using a structured troubleshooting workflow.

06 / PRACTICE

Design and verification labs

Apply the concepts to topology walkthroughs, failure scenarios, configuration checks and design decisions.

Key design principle: vPC and BGP solve different problems. vPC provides multi-chassis Layer 2 link aggregation; BGP exchanges routing information and can support multipath forwarding in routed designs. A resilient architecture must account for both the data path and the control-plane behaviour during failures.

Cisco vPC EtherChannel LACP STP BGP ECMP Leaf-Spine Fabrics Network Troubleshooting

Section 01 · Foundations

Redundant Links, EtherChannel and vPC Fundamentals

Redundancy is a core requirement in data-centre networking. Applications, servers and network services need connectivity to continue when an individual cable, interface or switch fails. However, adding physical links alone does not guarantee that all available capacity will be used efficiently or that every failure will be handled correctly.

A robust design must combine physical connectivity with mechanisms that control forwarding, distribute traffic and respond to failures. EtherChannel groups multiple Ethernet links into one logical interface, while Cisco Virtual PortChannel (vPC) extends the port-channel concept across a pair of supported switches.

Why a single link is not enough

A single connection between a server and a switch creates a potential single point of failure. If the cable, interface or switch port becomes unavailable, traffic using that connection may be interrupted until connectivity is restored or an alternative path is selected.

Multiple links can improve capacity and resilience, but their behaviour depends on how they are configured. Independent parallel Layer 2 links can create loops, while designs relying on Spanning Tree Protocol (STP) may place some redundant links into a non-forwarding state for a given VLAN. Link aggregation provides another approach by treating compatible physical links as one logical connection.

Physical redundancy

Multiple cables and interfaces reduce dependence on a single physical connection. Diversity of line cards, paths and power domains can provide further protection when designed appropriately.

Logical aggregation

A port-channel presents a bundle of compatible member links as one logical interface to supported protocols and connected devices.

Multi-switch resilience

vPC allows a supported downstream device to connect its port-channel to two separate Cisco Nexus peers, subject to the platform's design and configuration requirements.

EtherChannel versus vPC

These terms are related, but they describe different parts of the design. A conventional port-channel aggregates links between two endpoints. A vPC allows the two endpoints of a downstream logical port-channel to reside on separate switches within a configured vPC domain.

Technology Primary purpose Design consideration
EtherChannel / port-channel Aggregate multiple physical links into one logical interface. Member links must meet compatibility requirements and share a supported aggregation configuration.
LACP Negotiate and maintain a link aggregation group. Active/passive modes and configuration compatibility affect whether the bundle forms.
vPC Provide a multi-switch port-channel to a connected device. Peer-link health, keepalive, consistency and failure handling must be designed and verified.
STP Prevent Layer 2 forwarding loops in bridged networks. It remains relevant to the broader topology; vPC does not remove every STP requirement.

How a port-channel uses multiple links

When a port-channel is operational, the switch selects a physical member for each eligible traffic flow using a platform-dependent hashing algorithm. Depending on the configured inputs, the hash may use source and destination MAC addresses, IP addresses or transport-layer ports. Different flows can therefore use different members.

  1. Configure the member interfaces. Ensure the physical interfaces have compatible settings and are eligible to join the same aggregation group.
  2. Establish the logical bundle. Configure the port-channel and, where used, compatible LACP settings. Verify that the expected members become active.
  3. Distribute traffic. The switch hashes eligible flows onto member links. This is normally flow-based distribution, not a guarantee that every packet takes the next available link.
  4. Handle a member failure. Traffic can use surviving active members if the bundle remains operational, although capacity is reduced and some flows may be temporarily disrupted.

Engineering insight: Aggregating four links does not mean every individual flow receives four times the bandwidth. A particular flow is typically assigned to one member at a time, and actual throughput depends on hashing, traffic patterns, link speeds and the capabilities of the platform.

Where BGP enters the picture

Port-channels and vPC primarily address link aggregation and Layer 2 connectivity. In routed data-centre fabrics, BGP can provide a separate layer of resilience by exchanging reachability information between routers and supporting equal-cost multipath (ECMP) where the design, configuration and platform permit it.

This distinction matters during failures. A physical member failure may be handled within a port-channel without the routing protocol seeing a change to the logical interface. By contrast, a routed link or next-hop failure may trigger routing updates and a new forwarding decision. The recovery process depends on which component failed and how the control and data planes are configured.

Key takeaway: Redundant links, EtherChannel, LACP, vPC and BGP are complementary tools—not interchangeable technologies. Start by identifying the failure domain you need to protect, then select the appropriate mechanism and verify its actual failure behaviour.

Interactive learning · Module 01

From Redundant Links to vPC

Explore three ways to connect a server or downstream switch to the network. Compare a standalone physical link, a conventional EtherChannel and a virtual PortChannel across two switches.

Member A Member B Peer-link Server / Host NIC teaming / attachment Switch A vPC peer Switch B vPC peer vPC DOMAIN Two switches present a coordinated multi-switch attachment
Switch attachment Two switches
Logical link One coordinated vPC
Resilience model Multi-switch

vPC: a multi-switch port-channel

A downstream device can form a port-channel across two vPC peers. The switches coordinate through the vPC architecture, allowing supported topologies to use links to both switches without treating them as unrelated standalone connections.

Important: The design still depends on peer-link health, keepalive behavior, configuration consistency, and the platform's documented failure handling.

Knowledge check

What is the key architectural difference between a conventional EtherChannel and a vPC?

Educational illustration only. The link-failure control changes the diagram, not a real network device. Exact behavior depends on the vendor platform, topology and configuration.

Section 02 · Architecture

Understanding the vPC Peer-Link, Keepalive and Consistency

A vPC domain consists of two cooperating switches that maintain separate control planes while presenting a coordinated multi-chassis port-channel to connected devices. This arrangement enables active forwarding across supported links, but it also introduces important synchronization and failure-handling requirements.

Three concepts are central to understanding vPC operation: the peer-link, the peer-keepalive link and consistency checks. They have different purposes. The peer-link carries required inter-peer traffic and synchronization; the keepalive helps peers determine whether the other switch is alive; and consistency checks identify configuration differences that could compromise correct operation.

1. The main components of a vPC domain

Data and synchronization

Peer-link

A logical link between the two vPC peers. It carries synchronization and selected traffic required by vPC operation. Its capacity, redundancy and failure behaviour must be planned carefully.

Peer liveness

Peer-keepalive

A separate heartbeat path used to help distinguish a peer failure from a peer-link failure. It should be designed to remain available independently of the peer-link where practical.

Configuration safety

Consistency checks

Checks compare relevant configuration parameters across peers. Incompatible settings can prevent a vPC from coming up or cause a member to be suspended, depending on the parameter and platform.

2. How the peer-link supports the domain

The peer-link is not simply another downstream traffic connection. It supports communication between the vPC peers and provides a path for certain traffic when the normal forwarding path is unavailable. The network should not depend on it as the routine path for all end-host traffic.

vPC Peer A Independent control plane
Peer-link Synchronization and selected inter-peer traffic
vPC Peer B Independent control plane

Conceptual representation only. Actual peer-link requirements and forwarding behaviour depend on the Cisco Nexus platform, software release and configuration.

Under normal conditions, traffic from a downstream device should use its active vPC member links. If a peer loses its local member links for a particular vPC, the surviving peer and peer-link may be involved in forwarding traffic, subject to the applicable vPC forwarding rules. This is one reason peer-link capacity and failure scenarios matter.

Design principle: Do not treat the peer-link as a substitute for correctly designed downstream and routed connectivity. Understand which traffic can cross it, how much capacity is required, and what the platform does when peer-link members fail.

3. Why the peer-keepalive is separate

The keepalive mechanism helps the peers determine whether the other switch is reachable when peer-link communication is interrupted. That distinction is important: losing the peer-link does not automatically mean that the other switch has failed.

Observed condition What it may indicate Engineering response
Peer-link up, keepalive reachable Normal peer communication, assuming the domain is otherwise healthy. Verify peer-link members, consistency status and vPC state.
Peer-link down, keepalive reachable The peers may still be alive, but their main synchronization path is unavailable. Investigate peer-link interfaces, port-channel state and the platform's documented failure response.
Peer-link down, keepalive also unavailable The peers may be unable to distinguish a peer failure from a communication partition. Check peer reachability, keepalive path independence and the documented dual-active protection behaviour.
One peer is genuinely down The surviving peer may continue forwarding supported traffic, depending on the failure and configuration. Verify vPC state, downstream connectivity and recovery after the peer returns.

These are diagnostic patterns, not a universal state machine. The exact response to peer-link and keepalive failures is platform- and configuration-dependent, so operational decisions must be checked against the relevant vendor documentation.

4. Consistency checks: preventing incompatible configurations

Because both switches participate in a coordinated logical topology, important configuration must be compatible. Differences in VLAN membership, port-channel parameters, trunk settings or other vPC-relevant configuration can affect operation. Not every parameter is treated identically: some inconsistencies are more serious than others, and the resulting action depends on the specific check.

  • Confirm both peers belong to the intended vPC domain.
  • Verify peer-link status and its member interfaces.
  • Check vPC consistency output for global and per-vPC issues.
  • Compare VLANs, allowed VLAN lists, MTU and relevant interface settings.
  • Confirm port-channel membership and LACP state on both peers.
  • Review platform-specific requirements before changing a live domain.

5. Primary and secondary roles

vPC peers have primary and secondary roles for particular operational decisions. These roles do not mean that the secondary peer is simply an idle standby switch: both peers can forward traffic over their supported active vPC links. Role election, role changes and behavior during failure must be understood in the context of the platform and release.

The design should document the intended role priorities and the expected behavior during reloads, peer-link loss, keepalive loss and recovery. Avoid assuming that a role label alone predicts which traffic will be forwarded after every possible failure.

6. Orphan ports and single-homed devices

An orphan port is typically a port connected to a device through only one vPC peer rather than through a vPC to both peers. Such devices deserve special attention because a peer or peer-link failure can leave the device with different connectivity from a dual-homed server.

Identify orphan-connected devices, determine which peer they depend on, and test their reachability under the failure conditions relevant to the design. Where necessary, use supported platform-specific protection mechanisms and documented operational procedures.

Key takeaway: Reliable vPC operation depends on more than bringing up a port-channel. The peer-link, keepalive, consistency state, role behavior and orphan-port exposure must all be understood together. These are the foundations for the failure investigations covered later in this guide.

Interactive learning · Module 02

Inside the vPC Peer Relationship

The peer-link and peer-keepalive serve different purposes. Explore how their states differ and why you must investigate both before drawing conclusions about a vPC failure.

Peer-link · operational Peer-keepalive · separate path Switch A vPC peer Peer state: healthy Role depends on design Switch B vPC peer Peer state: healthy Role depends on design vPC member interfaces vPC member interfaces Both peer-link and keepalive paths are operational
Peer-link Operational
Keepalive Operational
Investigation priority Routine monitoring

Normal operation

The peer-link supports the vPC peer relationship, while the keepalive path provides a separate means of checking peer liveness. Confirm that both paths and the vPC consistency status are healthy.

What to investigate: Peer status, peer-link members, keepalive reachability and consistency checks.

Investigation checklist

Use these checks to guide your diagnosis for the selected condition.

Example verification commands

For a Cisco NX-OS-style environment, relevant commands may include:

show vpc
show port-channel summary
show interface port-channel 10
show logging

Command availability and output vary by platform and release. Use the appropriate vendor documentation for peer-keepalive and consistency verification.

Knowledge check

Why is it important to check the peer-keepalive path separately from the peer-link?

This diagram is a conceptual teaching aid, not a device-state simulator. A real peer-link or keepalive failure can have platform-specific consequences. Never infer the exact forwarding outcome from a diagram alone.

Section 03 · Traffic Engineering

Load Balancing, Hashing and Traffic Polarization

A port-channel can provide multiple active physical links, but those links will not necessarily carry equal amounts of traffic. Switches commonly use a hashing algorithm to select a member link for each eligible flow. The result depends on the traffic pattern, the selected hash inputs and the implementation supported by the platform.

Understanding this distinction is essential when designing redundant data-centre connections. A bundle may be operational and have all its members up, yet one link can become heavily utilized while other members remain comparatively idle. This uneven distribution is often described as traffic polarization or hash imbalance.

1. How port-channel hashing works

When a frame or packet is eligible for forwarding through an aggregated interface, the switch evaluates selected header fields. It uses those values to calculate a hash and map the result to one of the available physical members.

Incoming flow Traffic arrives at the switch
Hash inputs Selected MAC, IP or transport fields
Hash result Algorithm maps the flow to a member
Physical link Selected member forwards the traffic

Simplified conceptual model. Exact hash algorithms, fields, seed values and member-selection behavior vary by platform and software release.

2. Common hash inputs

Depending on the platform and configuration, the hashing algorithm can use Layer 2, Layer 3 or Layer 4 information. A richer set of inputs can help distinguish traffic flows that share the same source or destination address.

Hash input What it identifies Potential limitation
Source MAC The originating Layer 2 address. Many flows from one source can share the same value.
Destination MAC The destination Layer 2 address. Traffic directed to one next-hop MAC can concentrate on a subset of members.
Source and destination IP The Layer 3 endpoints of a flow. A workload with a small number of IP endpoint pairs may not spread evenly.
TCP/UDP ports Transport-layer conversations between endpoints. Availability and use of these fields depend on the platform, packet type and configured hashing policy.
Combined fields A broader flow identity using multiple header values. More inputs can improve distribution but cannot guarantee perfectly balanced utilization.

3. Why traffic polarization occurs

Hashing is not the same as sending each new packet to whichever link is least busy. It is generally designed to keep a flow on a consistent member, helping preserve packet ordering. As a result, the number of flows, their sizes and the distribution of their hash values influence the load on each member.

Uneven flow sizes

A few high-volume transfers can consume most of one member's capacity, even when the total number of flows appears balanced.

Limited hash diversity

If many flows share the same relevant header fields, they may map to the same member or a narrow subset of members.

Multiple hashing stages

Traffic crossing several aggregation or ECMP stages can experience correlated distribution effects, leaving some paths busier than others.

Important: A four-member port-channel does not guarantee that every flow receives one quarter of the bandwidth. Nor does a 25% share of the flow count guarantee a 25% share of bytes. Always evaluate actual traffic volume and member utilization.

4. Choosing and validating a hashing policy

Hashing configuration should reflect the traffic that the network actually carries. A policy suitable for many server-to-server flows may perform poorly when most traffic is concentrated on a small number of destinations. The available choices and their effects must be verified against the switch platform's documentation.

  • Identify the main traffic sources, destinations and application flow patterns.
  • Check which hash inputs and load-balancing options the platform supports.
  • Measure traffic rates and errors on individual physical members, not only the logical port-channel.
  • Compare flow distribution with byte counts, packet counts and interface utilization.
  • Test whether the selected policy improves distribution without creating unexpected forwarding behavior.
  • Recheck the distribution after link failures, workload changes or configuration updates.

5. The relationship between port-channel hashing and BGP ECMP

Port-channel hashing and BGP ECMP both distribute traffic across multiple forwarding choices, but they operate at different levels. Port-channel hashing chooses a physical member of an aggregated link. ECMP chooses between eligible equal-cost next hops in the routing and forwarding system.

In a routed leaf-spine fabric, a packet might be assigned to an ECMP next hop and then traverse an aggregated physical connection. Each stage may apply its own hashing behavior. Good distribution at one stage does not guarantee good distribution end to end.

Mechanism Selection being made What to investigate
Port-channel hashing Which physical member carries a flow. Member utilization, hash inputs, link capacity and traffic skew.
BGP ECMP Which eligible equal-cost next hop is used. Installed routes, eligible paths, next-hop resolution and forwarding-table programming.
End-to-end forwarding The combined path through the network. Traffic distribution at each stage, congestion points and the effect of failures.

6. Diagnosing an imbalanced bundle

Begin by confirming that the port-channel and its expected members are operational. Then examine per-member traffic counters over the same measurement interval. Look for sustained utilization differences, errors, drops or unexpected changes after a member or path failure.

If the physical links are healthy but the load remains uneven, compare the observed distribution with the workload's flow sizes and the configured hashing inputs. Where the design also uses routed ECMP, inspect route and next-hop state separately rather than assuming that the port-channel explains every imbalance.

Key takeaway: Multiple active links create forwarding choices, not automatic load equality. Effective load balancing comes from understanding hash behavior, measuring real traffic, and verifying distribution at both the link-aggregation and routing layers.

Interactive learning · Module 03

Explore Port-Channel Hashing

A port-channel usually selects an egress member using a hash of configured packet fields. The goal is to distribute flows across available links while keeping packets in a flow on a consistent path. Change the inputs and observe this illustrative model.

Traffic flows Input fields IP address hash Member 1 Active Member 2 Active Member 3 Active Member 4 Active Destination Network or host
Active members 4 / 4
Flow distribution Relatively spread
Model status Normal

Illustrative load by member

The bars represent a modelled share of traffic, not measured link utilisation. A real switch uses platform-specific hash algorithms and forwarding details.

Why hashing matters

Diverse flows can spread across multiple members. The distribution is not guaranteed to be perfectly even, and a single flow is generally not split packet-by-packet across every member.

Knowledge check

Why can a port-channel show uneven utilisation even when all member links are up?

Educational model only: member selection and load percentages are illustrative, not a reproduction of a specific vendor's hashing algorithm. Hash fields, resilient hashing, member selection and failure behavior depend on platform and configuration.

Section 04 · BGP and Routed Resilience

BGP and Layer 3 Routing in the Data Centre

Virtual PortChannels provide resilient Layer 2 connectivity by allowing a device to form a logical port-channel across two cooperating switches. BGP addresses a different requirement: exchanging routing information between routers and selecting eligible paths through a Layer 3 network. In modern data centres, both technologies may be used in the same architecture, but they solve different problems.

Understanding the boundary between Layer 2 redundancy and Layer 3 routing is important when designing leaf-spine fabrics, connecting server networks to routed infrastructure, or investigating a failure that affects only part of the forwarding path.

1. A leaf-spine fabric using BGP

In a common routed leaf-spine design, leaf switches connect to servers, storage or other endpoint networks. Spine switches provide routed connectivity between the leaves. BGP can exchange reachability information across these links and support multiple eligible paths through the fabric.

With multiple spine switches, a leaf may learn multiple eligible routes to a destination. If the routing policy and platform permit equal-cost multipath forwarding, the switch can install multiple next hops and distribute traffic across them.

2. BGP, ECMP and next-hop selection

BGP route exchange

BGP advertises network reachability and path attributes between peers. Policy and best-path processing determine which paths are selected or made eligible for multipath forwarding.

ECMP forwarding

Equal-Cost Multipath can use multiple eligible next hops for a destination when the routing system and forwarding platform support and install those paths.

Next-hop resolution

A route is useful only when its next hop can be resolved and programmed into the forwarding system. Routing and interface state must be checked together.

Important distinction: BGP does not automatically mean that all available paths will carry traffic. Route policy, path attributes, next-hop reachability, multipath configuration and platform capabilities all influence the forwarding result.

3. vPC and BGP solve different problems

A vPC pair presents a multi-chassis Layer 2 aggregation capability to a connected device. BGP exchanges Layer 3 routes between routing peers. One technology does not replace the other, and their relationship depends on the overall network design.

Design concern vPC / port-channel BGP / ECMP
Primary purpose Redundant Layer 2 connectivity and aggregated links. Route exchange and Layer 3 path selection.
Forwarding choice Selects a physical member within a logical link. Selects among eligible routed next hops.
Failure investigation Check member state, peer-link state, consistency and port-channel counters. Check BGP sessions, routes, next hops, route policy and forwarding-table state.
Design trade-off Requires careful multi-chassis coordination and failure handling. Requires sound routing policy, path design and convergence planning.

For example, a server may connect redundantly to a vPC pair while those switches participate in a separate routed fabric. Alternatively, a design may use routed server-facing links and ECMP without relying on a vPC pair for that connection. The appropriate choice depends on the desired failure domains, operational model and platform support.

4. What happens when a link or path fails?

Resilience depends on which component fails and how that failure is detected. A physical member failure inside a port-channel is different from the loss of a routed leaf-to-spine link, a BGP session failure, or the loss of an entire switch.

Failure event What may happen What to verify
One port-channel member fails Traffic may continue over remaining eligible members, with reduced aggregate capacity. Member state, errors, capacity and traffic redistribution.
One routed fabric link fails Routing and forwarding may use another eligible path if one is available and installed. Interface state, route changes, next-hop programming and packet loss.
A BGP session fails Routes learned through that session may be withdrawn or otherwise changed according to protocol state and policy. Session state, route withdrawals, alternate paths and convergence time.
A switch fails Recovery depends on the topology, remaining paths, gateway design and failure-detection mechanisms. Reachability, redundant paths, gateway behavior and application impact.

These are general outcomes, not guarantees. Actual behavior depends on the topology, platform, configuration, failure-detection timers and the availability of alternate forwarding paths.

5. A practical BGP and ECMP verification workflow

When traffic does not follow the expected path, separate the investigation into routing control-plane checks and forwarding data-plane checks. A BGP session can be established while a particular destination is missing, rejected by policy or not installed for multipath forwarding.

  • Check adjacency: Confirm the expected BGP peers are established and inspect recent session changes.
  • Check route reception: Verify that the expected prefix is received and accepted by policy.
  • Check best-path and multipath eligibility: Determine why paths are selected, rejected or excluded from ECMP.
  • Check next-hop reachability: Confirm that each intended next hop is resolvable.
  • Check forwarding state: Verify which next hops are actually installed in the routing or forwarding table.
  • Measure traffic: Inspect per-interface counters, drops, utilization and the effects of a controlled failure test.

Engineering takeaway: vPC provides one form of Layer 2 resilience; BGP and ECMP provide routed reachability and multipath forwarding. Reliable data-centre design requires engineers to understand each layer, test its failure behavior and verify the complete forwarding path.

Interactive learning · Module 04

BGP Route Propagation and ECMP

Follow an illustrative route advertisement through a data-centre fabric. Then withdraw one route and inspect how the available paths change. BGP selects routes; ECMP can allow multiple eligible next hops to participate in forwarding.

Illustrative fabric · eBGP or iBGP depends on design Leaf A AS 65001 · client side Spine 1 Routing node Leaf B AS 65002 · destination Spine 2 Alternate routing node Illustrative end-to-end route context Prefix: 10.10.10.0/24 · awaiting route walkthrough Blue links = available conceptual paths · red dashed = unavailable path
Prefix 10.10.10.0/24
Available paths 2 illustrative paths
Current stage Route origin

Step 1 — A prefix is originated

A network advertises 10.10.10.0/24 into the routing system. The receiving router must learn the route before it can evaluate the path and install an eligible next hop.

Engineering focus: Confirm the route exists at the origin and that the intended BGP session is established.

Path status

Path A · via Spine 1Available in this illustrative scenario
Path B · via Spine 2Available in this illustrative scenario

What to verify on a real network

Use commands appropriate to your router OS and design:

show ip bgp summary
show ip bgp 10.10.10.0/24
show ip route 10.10.10.0/24
show ip cef 10.10.10.0/24

These are illustrative Cisco IOS-style examples; exact syntax and ECMP inspection commands vary by platform.

Knowledge check

What is the key difference between BGP path selection and ECMP forwarding?

This is a conceptual learning model, not a live BGP implementation. ECMP depends on the routes accepted, next-hop resolution, routing policy, platform capabilities and configuration. The diagram does not imply that every displayed link is a BGP adjacency in every design.

Section 05 · Resilience Engineering

Failure Scenarios, Redundancy and Network Convergence

Redundancy is only valuable when the network can respond correctly to a failure. A vPC design may provide multiple physical connections, while BGP and ECMP provide alternative routed paths. However, each mechanism detects different events and follows its own recovery behavior. Engineers must understand how those mechanisms interact during a real outage.

The goal is not simply to keep every link green. It is to preserve reachability, maintain acceptable application performance and restore the intended forwarding state within an acceptable period.

1. Understand the failure domain

Before troubleshooting, identify which component has failed. A single physical member, an entire port-channel, a vPC peer-link, a peer switch or a routed BGP adjacency represents a different failure domain. The observed symptom alone may not identify the root cause.

Physical-link failure

One member stops forwarding. If the logical port-channel remains operational, traffic may continue over the remaining active members, but total capacity is reduced.

vPC peer-link failure

The two peers lose their inter-peer connection. Peer-keepalive status and platform-specific protection behavior become important when determining whether forwarding should continue.

Routed-path failure

A routed interface, next hop or BGP session may become unavailable. Traffic can use an alternate path only if one is eligible, reachable and installed for forwarding.

2. What happens during common failures?

Failure scenario Expected consideration Engineering checks
One member link fails Remaining port-channel members may continue carrying traffic, with less available bandwidth. Member state, interface errors, utilization and port-channel status.
All members of a logical link fail The logical connection may go down, triggering another forwarding or routing response depending on the topology. Logical interface state, alternate paths, route changes and endpoint reachability.
vPC peer-link fails Peer behavior depends on keepalive reachability, peer roles, configuration and platform protection mechanisms. Peer-link state, keepalive status, consistency state and relevant system logs.
One vPC peer fails Surviving connectivity depends on the remaining peer, attached links, gateway design and failure handling. Peer status, server-facing links, gateway reachability and application traffic.
BGP adjacency fails Routes learned through that session may be withdrawn or changed; alternate forwarding depends on available paths and convergence. BGP session state, route table, next hops and forwarding-table entries.
Uneven traffic after recovery Traffic may redistribute differently because of changed members, paths or flow hashes. Per-member counters, ECMP next hops, drops and traffic distribution.

Design warning: Do not assume that every failure triggers immediate, lossless failover. Detection timers, protocol behavior, hardware programming, topology and configuration all influence recovery. Validate the behavior on the actual platform.

3. Control-plane convergence versus data-plane forwarding

The control plane detects and communicates changes in network state. The data plane forwards packets using the currently programmed forwarding information. These processes are related, but they are not identical and may not complete at exactly the same time.

1
Failure detection

A physical interface, peer relationship or routing session detects a change through its configured detection mechanism.

2
State and route updates

The affected protocol or subsystem updates its state. A routing change may cause paths to be withdrawn, recalculated or selected.

3
Forwarding adjustment

Hardware or software forwarding state is updated to use the remaining eligible links or next hops.

4
Traffic and application recovery

Traffic resumes along the available path, but packet loss, latency or reduced capacity may remain during or after the transition.

This sequence is a conceptual troubleshooting model. Some systems react locally before routing convergence, and implementations can overlap or reorder these stages.

4. Why alternate paths do not guarantee zero packet loss

A redundant path may exist without being immediately usable. The failure must be detected, the relevant forwarding decision must change, and the alternate path must be capable of carrying the traffic. A routing protocol can also have a valid alternate route that is not eligible for installation under the current policy or multipath rules.

  • Capacity: Remaining links must be able to accommodate the shifted traffic load.
  • Reachability: Alternate next hops and destinations must remain reachable.
  • Convergence: Detection and forwarding updates must complete within the application's tolerance.
  • Consistency: vPC peers and connected devices must maintain a valid forwarding arrangement.
  • Asymmetric paths: Forward and return traffic may follow different paths, which can affect stateful devices or troubleshooting.
  • Failure correlation: Redundant links should not share an avoidable single point of failure, such as one cable path or power domain.

5. A structured failure-testing workflow

Controlled testing is more reliable than assuming redundancy works because the configuration appears correct. Establish a baseline, introduce one planned failure at a time, measure the impact and restore the original state before testing the next scenario.

1
Capture the baseline

Record interface state, port-channel membership, vPC health, BGP sessions, installed routes, traffic counters and application reachability.

2
Introduce a controlled failure

In an approved test window or lab, disable one selected link or simulate a planned adjacency failure. Avoid changing multiple components at once.

3
Measure the impact

Track packet loss, interruption duration, latency, route changes, member utilization and any application errors.

4
Restore and compare

Confirm that the failed component recovers, the intended paths return, and traffic distribution returns to an acceptable state.

6. Failure testing checklist

  • Confirm the intended redundant links and alternate routes are present before testing.
  • Verify that the failure is detected by the expected mechanism.
  • Measure both control-plane changes and actual data-plane traffic.
  • Check that remaining links do not become overloaded.
  • Confirm gateway and endpoint reachability throughout recovery.
  • Review logs and counters for errors that remain after the failure is cleared.
  • Document the measured recovery time and compare it with the service requirement.

Key takeaway: Resilience must be demonstrated, not assumed. A robust design combines independent failure domains, valid alternate forwarding paths, predictable convergence and repeatable tests that measure the impact on real traffic.

Interactive learning · Module 05

Failure Investigation and Convergence

Redundant paths help a network survive failures, but they do not guarantee zero packet loss. Explore common failures, distinguish physical and routing symptoms, and follow a structured investigation before deciding whether service has recovered.

vPC peer-link Switch A vPC peer Operational Switch B vPC peer Operational Route A Available Route B Available Conceptual failure map · not a live device simulation
Selected fault Member link
Expected concern Traffic shift
Recovery status Needs verification

Member-link failure

One physical member of a port-channel fails. If other members remain operational, the port-channel may continue forwarding, but capacity falls and some flows may be disrupted during detection and remapping.

First investigation: Check physical interface state, port-channel membership, errors and traffic counters.

Investigation checklist

Work through these checks for the selected failure. Tick them off as you investigate.

Example verification commands

show interface status
show port-channel summary
show vpc
show interface counters errors
show ip bgp summary
show ip route

These are illustrative commands; syntax and output vary by platform and operating-system version.

Knowledge check

After a redundant path fails, what is the strongest evidence that service has recovered?

Failure behavior depends on the platform, topology, configuration, failure-detection timers and routing design. This teaching widget illustrates investigation priorities; it does not predict exact convergence time or guarantee a particular forwarding outcome.

Section 06 · Architecture Decisions

vPC vs Layer 3 Multipathing and BGP ECMP

Both Virtual PortChannels and routed multipathing can improve network resilience, but they achieve it in different ways. vPC allows a connected device to use a logical Layer 2 port-channel across two cooperating switches. Layer 3 multipathing uses multiple eligible routed next hops to forward traffic through the network.

The design decision is not simply which technology is better. It is where the network needs Layer 2 adjacency, where routing boundaries should exist, and how the design should respond when links, switches or routing adjacencies fail.

1. Two different approaches to redundancy

Approach A — vPC

A connected device forms one logical port-channel using physical links to two cooperating switches. The vPC peers coordinate relevant forwarding and consistency behavior.

Approach B — Layer 3 ECMP

A routed device has multiple eligible next hops for a destination. The forwarding system distributes flows across installed paths according to its multipath and hashing implementation.

2. Compare the forwarding models

Design dimension vPC / Layer 2 aggregation Layer 3 multipathing / ECMP
Forwarding model Aggregated Layer 2 connectivity across two cooperating switches. Routing and forwarding across multiple eligible next hops.
Primary dependency vPC peer coordination, peer-link health, consistency and member state. Route availability, next-hop reachability and multipath eligibility.
Failure handling Depends on port-channel membership and vPC-specific protection behavior. Depends on failure detection, route changes and the availability of alternate installed paths.
Traffic distribution Hashing selects a physical port-channel member. Hashing selects among eligible routed next hops.
Layer 2 scope Can support a Layer 2 segment spanning the vPC-connected infrastructure. Routed boundaries limit the need to extend a Layer 2 segment across the fabric.
Operational focus Peer state, port-channel health and Layer 2 forwarding consistency. BGP state, route policy, next-hop resolution and forwarding-table programming.

3. Where BGP ECMP fits in a leaf-spine fabric

In a routed leaf-spine architecture, each leaf commonly has routed connections to multiple spine switches. BGP can exchange reachability information over those connections. Where multiple paths are eligible and ECMP is supported and configured, the forwarding system can use multiple next hops to reach a destination.

Path diversity

Multiple routed paths can provide alternatives when a link or spine becomes unavailable, provided the remaining topology offers valid reachability.

Routing boundaries

Routed links can reduce dependence on extending the same Layer 2 segment throughout the fabric.

Independent verification

Engineers can inspect BGP routes, installed next hops and interface counters to identify where forwarding behavior differs from the intended design.

Remember: BGP is the routing protocol; ECMP is a forwarding capability that may use multiple eligible equal-cost paths. Running BGP alone does not guarantee that multiple paths will be installed or used.

4. When might each approach be appropriate?

vPC may be appropriate when a server, appliance or downstream switch needs redundant Layer 2 connectivity to a pair of switches. Routed multipathing may be attractive when the network can use Layer 3 links and routing to provide path diversity across a fabric.

Requirement Design direction to evaluate Key question
Endpoint requires Layer 2 adjacency across two switches Evaluate vPC or another supported multi-chassis aggregation design. Can the peer relationship and failure behavior meet the requirement?
Fabric can use routed point-to-point links Evaluate a routed leaf-spine architecture with BGP and ECMP. Are alternate paths available, eligible and correctly installed?
Application requires high availability Evaluate the complete path, including server NICs, switches, routing and upstream services. Does the design survive the relevant failure without unacceptable impact?
Uneven traffic utilization Inspect port-channel hashing and routed ECMP distribution separately. At which forwarding stage is traffic concentrating?

5. Avoid treating vPC and ECMP as interchangeable

A vPC port-channel and routed ECMP may both distribute traffic, but they operate at different forwarding layers. They also have different state, failure mechanisms and troubleshooting commands. A design can use both technologies in different parts of the same network, provided their interaction is understood and supported by the platforms.

  • Document the Layer 2 and Layer 3 boundaries.
  • Identify which component detects each expected failure.
  • Verify the alternate forwarding path exists before testing a failure.
  • Measure convergence, packet loss and remaining link capacity.
  • Check port-channel members and ECMP next hops independently.
  • Validate the design against the vendor's platform and software documentation.

Key takeaway: Choose vPC when multi-chassis Layer 2 aggregation is required; evaluate routed BGP ECMP when the design calls for multiple Layer 3 paths. The strongest architecture is the one whose failure domains, forwarding behavior and operational requirements have been deliberately designed and tested.

Interactive learning · Module 06

Choose the Right Multipathing Model

vPC and BGP ECMP solve different networking problems. Compare a Layer 2 multi-switch attachment with a Layer 3 routed fabric, then simulate a path failure to see which part of the design must be investigated.

Layer 2 attachment · vPC Server / Host Traffic source vPC Peer A Layer 2 switch vPC Peer B Layer 2 switch Peer-link Network Conceptual model · not to scale
Primary purpose Dual-switch Layer 2 attachment
Forwarding context EtherChannel / vPC
Failure to investigate Member and peer state

Layer 2 attachment with vPC

A supported downstream device can form a port-channel across two vPC peers, presenting a logical attachment while using links to both switches. Peer-link health, consistency and platform-specific failure behavior remain important.

Engineering check: Verify vPC state, peer-link, member interfaces, consistency and traffic counters.

vPC · Layer 2 attachment

  • Provides a multi-switch port-channel attachment for supported devices.
  • Depends on peer coordination and platform-specific vPC behavior.
  • Does not itself provide BGP route selection or Layer 3 ECMP.

BGP ECMP · Layer 3 paths

  • Uses multiple eligible equal-cost next hops when supported and configured.
  • Depends on routing policy, route selection, next-hop reachability and forwarding installation.
  • Does not require a vPC peer-link between the routed next hops.

Verification command examples

show vpc
show port-channel summary
show ip bgp summary
show ip route
show interface counters errors

These are illustrative Cisco-style commands. Availability and syntax vary by platform and software version.

Knowledge check

Which statement best describes the relationship between vPC and BGP ECMP?

These designs can coexist in a larger network, but one does not automatically configure or guarantee the other. The drawings are conceptual and do not represent a vendor-specific production topology.

Section 07 · Operational Troubleshooting

vPC Troubleshooting and Verification

A resilient design needs more than redundant links and correctly configured switches. Engineers must be able to verify the operational state of the vPC domain, identify inconsistencies, isolate failures and confirm that traffic is using the expected forwarding paths.

The most effective troubleshooting approach is to work from the underlying physical interfaces towards the logical port-channel, vPC peer relationship and end-to-end forwarding path. This helps distinguish a physical connectivity problem from a peer coordination, Layer 2 or routing issue.

1. Begin with the health of the vPC domain

Start by confirming that both peer switches are available and that their peer relationship is functioning as expected. Check the peer-link, peer-keepalive mechanism, vPC member interfaces and consistency status. A single healthy interface does not prove that the complete vPC arrangement is healthy.

Peer relationship

Verify that the peers recognize each other and that the expected vPC domain state is present.

Peer-link

Check physical members, logical interface state, errors and capacity. Confirm that the peer-link is operating as designed.

Keepalive

Verify the configured keepalive path and inspect logs for reachability changes or missed messages.

Consistency

Review vPC consistency results and any reported type-1 or type-2 configuration mismatches.

Safety first: Peer-link and keepalive failures can affect vPC forwarding and peer protection behavior. Do not troubleshoot by blindly shutting interfaces or changing peer roles. Understand the current state and follow the platform's documented recovery procedure.

2. Verify the physical members and logical port-channel

A port-channel can have a logical interface that appears configured while one or more expected physical members are down, suspended or not bundled correctly. Compare the configured membership with the operational state.

On Cisco Nexus platforms, the following NX-OS commands are common starting points. Exact command availability and output vary by platform and software release.

show port-channel summary show interface port-channel 10 show interface ethernet 1/1 show vpc

Replace the example interface and port-channel identifiers with those used in your environment. Check whether the physical members are bundled, whether the logical interface is up, and whether counters indicate errors or drops.

3. Troubleshooting symptoms and likely causes

Observed symptom Possible causes Next checks
Port-channel is down Member links are down, members are not bundled, or configuration does not match. Physical state, member configuration, LACP state and port-channel summary.
Member is suspended Negotiation, compatibility or configuration problems. LACP details, interface logs, speed, duplex and channel-group settings.
vPC consistency warning Relevant configuration differs between peers. Consistency output, VLAN membership, port-channel settings and platform guidance.
Peer-link is down Physical failure, member bundle failure or a configuration issue. Peer-link members, interface errors, cabling and peer state.
Traffic is unevenly distributed Hash distribution, flow size differences or reduced available capacity. Per-member traffic counters, hashing policy and workload patterns.
Connectivity fails after a routing change Route withdrawal, missing next hop, policy issue or unavailable alternate path. BGP session state, route table, next-hop reachability and forwarding entries.

4. Investigate vPC consistency issues

Consistency checks help identify configuration differences between peers that could affect forwarding. The exact checks and the impact of a mismatch depend on the platform, interface type and software version.

Do not assume that every warning has the same severity. Some inconsistencies can suspend a vPC or prevent it from operating as intended; others may generate warnings without immediately disabling forwarding.

  • Compare VLAN membership and allowed VLAN lists.
  • Check port-channel and LACP configuration on both peers.
  • Verify relevant trunk, access and spanning-tree settings.
  • Review vPC-specific consistency output and system logs.
  • Make corrective changes using the platform's documented procedure, then verify the resulting operational state.

5. Use counters to investigate traffic problems

When connectivity works but performance is poor, examine counters rather than relying only on link status. A physical link can remain up while experiencing errors, congestion or uneven utilization.

show interface counters errors show interface ethernet 1/1 show port-channel summary

These are illustrative NX-OS commands. Verify the supported syntax for your specific device and software release.

  • Compare traffic rates across all members of the same port-channel.
  • Look for CRC errors, input errors, output drops and unexpected counter changes.
  • Check whether one large flow is dominating a member link.
  • Confirm that the remaining members have enough capacity during a failure.
  • Compare the observed behavior with the configured hash policy.

6. Follow the traffic across the Layer 3 boundary

If the vPC domain appears healthy but remote destinations are unreachable, continue the investigation into the routed network. A working port-channel does not guarantee that a route exists or that the destination is reachable through the expected next hop.

show ip route show ip bgp summary

These examples are intended for supported Cisco routing contexts; syntax and command availability vary. Check the routing table, BGP adjacency state and next-hop reachability before concluding that vPC is responsible for the outage.

7. A repeatable troubleshooting workflow

1
Define the impact

Determine which endpoints, VLANs, links or destinations are affected and whether the issue is total loss, intermittent connectivity or reduced performance.

2
Check physical and logical state

Verify interface status, port-channel membership, LACP state and relevant error counters.

3
Check the vPC domain

Inspect peer status, peer-link, keepalive and consistency results. Review logs for state transitions around the time of the problem.

4
Verify the forwarding path

Confirm VLAN and Layer 2 reachability where applicable, then inspect routing, next hops and forwarding entries for remote destinations.

5
Validate the correction

Confirm that the original symptom has cleared, counters are stable and the network has returned to its intended state. Document the root cause and evidence.

Key takeaway: Troubleshoot from the physical interface through the port-channel and vPC domain, then follow the traffic into the routed network where necessary. Use operational state, consistency results, counters and routing evidence together rather than relying on a single command or status indicator.

Interactive troubleshooting · Module 07

Diagnose the Fault Before Changing the Network

Choose a symptom to build a focused investigation plan. Work from the observed fault toward the relevant control-plane and data-plane evidence. The checks below are a learning aid, not live device output.

1. Select the symptom

Selected investigation

Port-channel member investigation

Determine whether the port-channel is operational, which members are active, and whether traffic loss is linked to a physical interface, a flow-hashing pattern or a downstream fault.

2. Work through the checks

0 of 4 checks completed

Start by establishing the current operational state before changing configuration.

3. Example verification commands

Use commands appropriate to the device and software release. Compare outputs before and after a controlled test.

show port-channel summary
show interface port-channel 10
show interface counters errors
Interpretation: a member being up does not alone prove that every flow is distributed evenly or that the end-to-end path is healthy.

Knowledge check

You observe that a BGP session is established, but the expected prefix is missing from the routing table. What is the best next step?

Command syntax and available fields vary by platform and software version. Follow the vendor's documentation and your operational procedures. Record the baseline, make one controlled change at a time, and verify both recovery and end-to-end traffic.

Section 08 · Design and Deployment

vPC Design Best Practices and Deployment Checklist

A successful vPC deployment starts with a clear redundancy objective, a supported topology and a documented failure model. Link aggregation alone does not guarantee end-to-end resilience. The peer relationship, physical infrastructure, Layer 2 configuration, routing design and operational procedures must work together.

Use the following principles to review a proposed design or assess an existing data-centre deployment. Platform-specific implementation details must always be checked against the relevant vendor documentation.

1. Build redundancy into the physical design

Separate failure domains

Where practical, diversify cable paths, line cards, power sources and upstream connectivity so that one local incident does not remove every redundant path.

Plan peer-link capacity

Size the peer-link for its expected control and data traffic, including relevant failure scenarios. Do not assume normal utilization represents the worst-case requirement.

Protect the keepalive path

Design peer-keepalive connectivity according to platform guidance. It should provide useful peer-liveness information independently of the peer-link failure it helps diagnose.

Preserve usable capacity

Confirm that the surviving links and switches can carry the expected load after a failure, rather than merely keeping endpoints connected.

2. Keep the configuration consistent

vPC peers need compatible configurations for the features and interfaces involved in the design. Consistency checking can detect important differences, but it should not replace a deliberate configuration review.

  • Verify the vPC domain, peer-link and peer-keepalive configuration.
  • Review VLAN membership, trunk settings and spanning-tree requirements.
  • Confirm port-channel and LACP settings match the intended design.
  • Check relevant MTU, speed, interface and feature compatibility.
  • Document the treatment of orphan ports and single-homed devices.
  • Review supported software versions, known limitations and upgrade procedures.

Design principle: Do not copy a configuration between peers without validating which settings must match and which settings are intentionally different. Follow the consistency requirements for the specific platform and release.

3. Decide where Layer 2 should end

One of the most important architectural decisions is determining which networks need Layer 2 adjacency and where Layer 3 routing should begin. Extending Layer 2 can be necessary for some workloads, but it also increases the scope of the Layer 2 failure domain.

Design question What to evaluate
Does the endpoint need Layer 2 adjacency? Determine whether vPC-based connectivity is required or whether a routed attachment is suitable.
Can the fabric use routed links? Evaluate a Layer 3 leaf-spine design with routing and ECMP where multiple paths are appropriate.
Where is the gateway located? Document gateway redundancy, first-hop behavior and the forwarding path during peer or link failure.
How are failures contained? Identify which devices, VLANs, routes and applications are affected by each failure domain.

4. Integrate BGP and ECMP deliberately

Where the data centre uses BGP, review the routing architecture independently from the vPC configuration. BGP route exchange, best-path selection and ECMP forwarding have their own operational requirements. A healthy vPC domain does not prove that the routed network has a valid path to every destination.

  • Document BGP peers, address families, route policies and intended advertisements.
  • Verify next-hop reachability and which routes are installed for forwarding.
  • Confirm that multiple paths are eligible for ECMP where multipath forwarding is expected.
  • Inspect both ECMP next-hop distribution and port-channel member utilization.
  • Test how link, peer and BGP-session failures affect traffic and convergence.

Remember: vPC hashing chooses a physical member of a logical link. ECMP chooses among eligible routed next hops. If traffic is unevenly distributed, investigate each forwarding stage rather than assuming one mechanism explains the entire path.

5. Interactive deployment readiness checklist

Use this checklist during a design review or before a controlled deployment. Tick each item only after the relevant configuration, documentation or test evidence has been verified.

0 of 10 checks completed

Checklist progress is held in this page view only. It is not saved to an account or network device. Completing the checklist is not a substitute for platform-specific validation or an approved change plan.

6. Final design review

Before declaring the design ready, confirm that the team can explain how traffic flows during normal operation, which links carry it, what happens when each important component fails, and how the resulting behavior will be verified. A design that can be tested, monitored and recovered consistently is easier to operate than one that relies on assumptions about redundancy.

Key takeaway: Build vPC resilience around supported peer behavior, consistent configuration, adequate capacity and independent failure domains. Where BGP and ECMP are also used, validate the routing and forwarding layers separately, then test the complete end-to-end design.

Design readiness · Module 08

vPC Design Readiness Assessment

Use this interactive checklist to review redundancy, peer coordination, traffic behavior, routing boundaries and operational recovery before deploying or changing a vPC-based design. Tick a control only when it has been verified for your actual platform and topology.

Controls reviewed 0 / 10
Completion 0%
Readiness indicator Not assessed

1. Physical and logical redundancy

Check that the design can tolerate the failures it is intended to survive.

2. Configuration and Layer 2 behavior

Confirm that the peers and attached devices behave as designed.

3. Routing and traffic distribution

Separate Layer 2 attachment resilience from Layer 3 routing decisions.

4. Operations and recovery

A resilient design also needs a tested operational process.

Assessment not complete

Review each control and mark it only when verified. Completion measures checklist progress, not an automatic guarantee that the network is production-ready.

Design review summary

    This checklist is an educational planning aid. A checked box records your review; it does not validate device configuration or prove resilience. Apply the requirements and failure behavior documented for your specific platform, release and design.

    Section 09 · Practical Learning

    Interactive vPC and BGP Labs

    Understanding redundancy means more than knowing which links exist. You need to predict what happens when a component fails, identify which control-plane or forwarding state changes, and verify whether traffic can continue to reach its destination.

    Explore these five guided scenarios. Each lab gives you a problem, an investigation sequence, verification commands and the observations you should look for. Select a lab to switch between scenarios.

    Lab safety: These are illustrative exercises, not live-device simulations. Commands and outputs depend on platform and software version. Perform failure tests only in an authorized lab or maintenance window with a documented recovery plan.

    LAYER 2 ETHERCHANNEL

    Lab 1: Member-link failure

    Scenario

    A server connects through a port-channel whose physical members provide redundant bandwidth. One member link fails, but the logical port-channel may remain operational.

    Your investigation

    1. Record the original port-channel state and member status.
    2. Identify which physical interface has failed.
    3. Check whether the port-channel remains up and has fewer active members.
    4. Inspect interface errors, utilization and application connectivity.
    Example verification commands
    show port-channel summary
    show interface port-channel 10
    show interface ethernet 1/1
    show interface counters errors
    Expected observation If a surviving member remains active and the port-channel stays up, traffic may continue over the remaining capacity. Existing flows can be redistributed or affected depending on platform behavior and the hashing implementation.

    Engineering question: Does the remaining capacity support the expected load?

    Test your reasoning

    Reveal each answer after considering the scenario. These questions check the design principles behind the labs.

    1. A single port-channel member fails, but the logical port-channel remains up. Must BGP necessarily reset?

    No. If the routed interface and relevant adjacency remain operational, a member failure may be handled below the routing layer. The actual result depends on where the failed link sits and how the platform reports and handles the failure.

    2. Does a healthy vPC peer relationship prove that every BGP route is valid?

    No. vPC health and BGP route validity are separate checks. Inspect routing adjacencies, route selection, next-hop reachability and forwarding state independently.

    3. If a route has two eligible ECMP next hops, will every flow use both paths equally?

    Not necessarily. Hashing, flow count, flow sizes and traffic patterns influence distribution. Two installed next hops do not guarantee equal utilization.

    How to use these labs in a real environment

    1. Write down the expected behavior before making a change.
    2. Capture baseline interface, vPC, routing and traffic information.
    3. Change only one controlled failure condition at a time.
    4. Compare the observed result with the prediction and record any packet loss or convergence delay.
    5. Restore the original state and verify end-to-end connectivity before proceeding.

    Key takeaway: Troubleshoot from the physical interface through the port-channel and vPC state, then inspect Layer 3 routing and end-to-end forwarding. This layered method helps isolate whether a fault affects link redundancy, peer behavior, route selection or the final traffic path.

    Commands shown are examples for Cisco NX-OS-style environments. Syntax and availability vary by platform and release. The interactive panels illustrate learning scenarios and do not execute commands, change device state or persist lab progress.

    Hands-on learning · Module 09

    vPC and BGP Troubleshooting Lab Console

    Choose a scenario, examine the conceptual topology and work through a guided investigation. Each lab focuses on evidence gathering, interpreting network state and identifying what to verify next. These are simulated learning exercises, not live device sessions.

    Layer 2 · Port-channel

    Lab 01 — Member-link failure

    A port-channel has lost one physical member. Determine whether the bundle remains operational and whether the remaining capacity is sufficient for the observed traffic.

    SIMULATED TOPOLOGY Downstream device Port-channel Switch A Member active Switch B Member active Peer-link Network Illustrative only · not to scale

    Investigation progress

    Begin by confirming the operational state of the port-channel and its physical members.

    Expected observations will appear here after the scenario is simulated.

    Lab objective

    Confirm member state, assess remaining capacity and verify end-to-end forwarding.

    Commands to investigate

    Use these examples as a starting point and adapt them to the platform.

    show port-channel summary
    show interface port-channel 10
    show interface counters errors

    What good evidence looks like

    Confirm that the port-channel is up, identify active members and compare traffic and error counters before and after the event.

    Lab knowledge check

    One physical port-channel member fails. What should you verify first?

    All topology changes and observations in this widget are illustrative. They do not emulate a particular switch operating system, predict exact convergence time or guarantee lossless recovery. Follow platform documentation and approved change procedures when working on real devices.

    ```
    ☕
    Support Network Insight
    Help support independent engineering guides, interactive networking tools, and future learning labs.
    Support Network Insight ```
    Green data center with eco friendly electricity usage tiny person concept. Database server technology for file storage hosting with ecological and carbon neutral power source vector illustration.

    Data Center Design with Active Active design

    Network Insight · High Availability Architecture

    Active-Active Data Center Design: Building for Continuous Availability

    What happens when one data center can no longer serve its workloads? An active-active architecture aims to keep multiple sites operational at the same time, distributing traffic and capacity while preparing services to withstand component, network and site failures.

    Modern applications rarely depend on a single network, server or storage system. They operate across interconnected infrastructure, distributed services and, increasingly, multiple locations. Active-active design brings these components together into an architecture where more than one site can actively serve workloads. But operating two sites simultaneously is not simply a matter of duplicating hardware. Routing, application behavior, data consistency, traffic distribution and failure detection must work together.

    Data Center A

    Primary active site in this illustration

    Applications Compute Storage
    Serving workloads

    Data Center B

    Second active site in this illustration

    Applications Compute Storage
    Serving workloads

    01 · Connectivity

    Route around failures

    Use diverse network paths and appropriate routing policies to preserve reachability when links or devices fail.

    02 · Consistency

    Keep application state safe

    Design replication, quorum and application behavior around the consistency requirements of the service.

    03 · Recovery

    Prove service continuity

    Test site isolation, traffic shifts and recovery behavior instead of assuming that redundancy guarantees availability.

    i

    The essential distinction: both sites can be active without every application being able to survive every failure. True service resilience depends on the complete design—including data consistency, health detection, routing, capacity and application recovery behavior.

    Multi-site networking BGP and DCI Data consistency High availability Failure engineering
    Section 01 · Foundations

    Understanding Active-Active Data Center Design

    Explore how two or more data center sites can serve workloads concurrently, how this differs from active-passive resilience, and why application behaviour, dependency design and failure handling matter as much as network redundancy.

    Active vs. passiveAvailabilityFailure domains
    Section 01 · Architecture Fundamentals

    Understanding Active-Active Data Center Design

    Active-active design allows two or more data center sites to serve production workloads concurrently. The objective is to use distributed capacity while maintaining service continuity when a component, network path or site becomes unavailable.

    Traditional active-passive designs generally keep one site serving traffic while another is prepared to take over during a failure. In an active-active design, both sites perform useful production work during normal operation. Traffic may be distributed through global load balancing, application-aware routing, anycast services or other mechanisms appropriate to the application and network architecture.

    This can improve resource utilisation and provide additional capacity options. It also introduces design questions that cannot be solved by routing alone: where application state lives, how data is replicated, how clients reach healthy services, and what happens when the sites lose contact with one another.

    Three principles that shape the design

    01

    Concurrent workload

    Both sites contribute to normal service delivery. Capacity planning must account for traffic distribution and whether surviving infrastructure can absorb additional demand after a failure.

    02

    Consistent service behaviour

    Applications must handle sessions, shared state, data replication and concurrent updates according to their consistency and availability requirements.

    03

    Designed failure response

    Health checks, routing policies, load balancers and application recovery logic must work together to direct clients towards services that can actually respond.

    Important distinction: two active sites do not automatically provide application resilience. If both sites depend on the same database, identity service, WAN path or control-plane component, that shared dependency may still become a single point of failure.

    Questions to answer before choosing active-active

    • Can the application safely run across multiple sites at the same time?
    • How much data loss, if any, is acceptable following a site failure?
    • What recovery time is required, and how will client traffic move to healthy services?
    • Can one surviving site handle the required workload during a major failure?
    • How will split-brain conditions, network partitions and inconsistent state be detected and handled?
    Engineering takeaway

    Active-active is an end-to-end service architecture, not simply a pair of interconnected data centers. Reliable outcomes depend on coordinated network design, application behaviour, data consistency, capacity planning and tested recovery procedures.

    Interactive Lab 01 · Architecture explorer

    What Happens When a Data Center Fails?

    Switch the deployment model or simulate a site failure. Observe the illustrative traffic paths, site states and engineering implications. This is a conceptual model, not a live network simulator.

    Choose a scenario
    Two-site data center architecture Two data center sites connect to a shared user traffic entry point. The selected scenario changes the illustrative traffic paths and site status. Users / Traffic Entry DNS · GSLB · Load balancing Data Center A Data Center B Serving workloads Serving workloads Application stack A ACTIVE Application stack B ACTIVE Inter-site connectivity / replication Both sites serve traffic concurrently
    Data Center A
    Active · serving traffic
    Data Center B
    Active · serving traffic
    SCENARIO: ACTIVE-ACTIVE

    Both sites serve production traffic

    Workloads at both sites are available to serve requests. Traffic distribution depends on application placement, routing, load balancing and health checks. Shared dependencies and data consistency still need deliberate design.

    Engineering checks

    • Confirm application and data consistency.
    • Verify each site's independent capacity.
    • Test routing and service health during failure.
    Illustrative topology · No live devices, routing protocols or production traffic are controlled.
    Section 02 · Routing architecture

    Multi-Site Network Architecture and BGP

    Follow traffic between data centers and external networks. Understand how BGP exchanges reachability, how routing policy and multipath influence path selection, and how engineers avoid asymmetric traffic, route leaks and unexpected failover behaviour.

    DCI designBGP policyTraffic engineering
    Section 02 · Network Architecture

    Multi-Site Network Architecture and BGP

    An active-active design needs more than two functioning data centers. It needs predictable routing, resilient inter-site connectivity and a clear policy for how users and applications reach services when paths or entire sites fail.

    The network provides the connectivity between data centers, users, applications and external services. In a multi-site architecture, Data Center Interconnect (DCI) links connect the sites, while routing and traffic-management systems determine which paths are available and where traffic should be sent.

    Border Gateway Protocol (BGP) is commonly used to exchange reachability information between sites, WAN networks and external networks. Its policy controls can influence path selection and traffic distribution. However, BGP does not by itself verify that an application is healthy or that its data is consistent.

    The three network design layers

    Layer 01

    Data Center Interconnect

    DCI may use routed IP connectivity, an optical transport service or an overlay architecture. Choose bandwidth, latency and path diversity according to application traffic and replication requirements—not simply the number of physical links.

    Layer 02

    Routing and reachability

    BGP can advertise site prefixes and apply routing policy. Multipath forwarding may use multiple eligible routes when supported and correctly configured. Route selection and forwarding behaviour must be validated for the actual platform and topology.

    Layer 03

    Service traffic steering

    DNS-based global load balancing, anycast, application load balancers or other steering mechanisms can direct clients towards available services. Health checks must reflect genuine service readiness, not just network reachability.

    BGP: control the path, then validate the outcome

    A common design advertises service or site prefixes from more than one location. The routing policy determines which advertisements are accepted and how paths are preferred. Depending on the design, engineers may use attributes such as LOCAL_PREF within an autonomous system, AS_PATH for inter-AS policy, or multipath features to support selected equal-cost paths.

    The important distinction is between route availability and service availability. A router may have a valid route to a site whose application is unhealthy. Conversely, an application may be healthy while a routing or traffic-steering policy prevents clients from reaching it. These systems need coordinated health signals and carefully tested failure behaviour.

    Design warning — avoid accidental traffic tromboning. If a client reaches a service in one site but the application retrieves data or calls a dependency in the other site, traffic may cross the DCI unnecessarily. This can increase latency, consume inter-site bandwidth and complicate failure handling. Keep traffic local where practical, while preserving the required consistency and availability guarantees.

    What should happen during a failure?

    • DCI link failure: traffic should use a surviving path where one exists, with convergence and capacity validated under load.
    • Border router failure: alternate routing paths should remain available without creating a forwarding loop or black hole.
    • Site failure: route withdrawal and service-health mechanisms should steer new traffic away from unavailable services.
    • Application failure: health checks should stop advertising or selecting an endpoint that cannot deliver the required service.
    • Network partition: routing and application policies must avoid unsafe assumptions about the remote site's state or data ownership.

    Architecture checks before deployment

    • Identify shared physical and logical failure domains across DCI paths.
    • Document prefix advertisements, routing policy, route filtering and intended path preferences.
    • Check convergence behaviour and whether surviving links can carry the expected load.
    • Validate inbound and outbound traffic steering independently; they may follow different paths.
    • Test application health checks alongside routing changes and site-isolation scenarios.
    Engineering takeaway

    BGP and DCI establish and control network reachability; they do not guarantee that an application is usable. A resilient active-active design coordinates routing, traffic steering, application health, capacity and data consistency—and verifies the result through controlled failure tests.

    Interactive Lab 02 · Multi-site routing

    Trace the Route Between Data Centers

    Explore how inter-site connectivity, BGP reachability and routing policy influence traffic. Simulate a DCI link failure or route withdrawal, then select a router or network to inspect its role.

    Select a routing scenario
    Illustrative multi-site BGP routing topology Two data centers with border routers connect through an inter-site link. An external network advertises a prefix. Select a node to inspect its role. SITE A · AS 65001 SITE B · AS 65001 Application A 10.10.10.0/24 Border Router A BGP / route policy Border Router B BGP / route policy Application B Service destination External Network AS 65002 Prefix advertisement DCI · inter-site path eBGP iBGP / local routing iBGP / local routing Illustrative route: 198.51.100.0/24 Conceptual diagram · BGP behaviour depends on topology and policy
    Illustrative preferred path Illustrative alternate path Failed or withdrawn path
    STATE: NORMAL ROUTING

    Reachability is available through the configured paths

    The diagram illustrates an available inter-site path and route advertisement. Actual forwarding depends on the installed route, routing policy, next-hop reachability and data-plane health.

    Select a network component

    Select either border router, an application network or the external AS in the diagram to explore its role in route propagation and traffic forwarding.

    DCI statusAvailable
    Illustrative routeAdvertised
    Engineering focusValidate policy
    ItemIllustrative stateWhat to verify
    Inter-site pathAvailableIndependent physical paths, capacity and forwarding health.
    Route advertisementPresentAccepted prefixes, next hop, policy and route installation.
    Service reachabilityCheck requiredEnd-to-end application health, not only control-plane state.
    Educational model only · No live BGP sessions or production routes are changed.
    Section 03 · Overlays and mobility

    Layer 2 Extension, Overlays and Workload Mobility

    Compare stretched Layer 2 networks with routed Layer 3 designs, and examine how VXLAN/EVPN overlays can support multi-site connectivity. Consider workload mobility, broadcast scope, failure propagation and the risk of traffic tromboning before extending networks between sites.

    VXLAN / EVPNLayer 2 vs. Layer 3Workload mobility
    Section 03 · Layer 2 and Overlay Design

    Layer 2 Extension, Overlays and Workload Mobility

    Workload mobility across data centers can simplify some application migrations and recovery scenarios. The challenge is deciding how much network continuity is actually required—and whether extending a Layer 2 domain is the right way to achieve it.

    In a multi-site environment, applications may need to move between sites for maintenance, capacity balancing, disaster recovery or infrastructure changes. Some workloads can move while retaining their IP addresses; others can reconnect through a new endpoint or service address. The application’s requirements should determine the network design, rather than assuming that every workload needs a stretched subnet.

    A Layer 2 extension carries an Ethernet broadcast domain across multiple locations. This can preserve subnet membership and support certain migration workflows, but it also extends the domain’s failure and operational characteristics. A Layer 3 design keeps subnets local to each site and uses routing to connect them, often reducing the scope of Layer 2 failure propagation.

    Approach A

    Stretched Layer 2

    The same VLAN or subnet is available across sites, subject to the chosen extension technology and design constraints.

    • Can support address-preserving workload mobility.
    • May carry broadcast and unknown-unicast traffic across the DCI.
    • Requires careful control of loops, failure domains and inter-site latency.
    • Can make partial failures harder to diagnose.
    Approach B

    Routed Layer 3

    Each site uses its own routed subnets, and connectivity between sites is provided through routing and application-level service discovery or traffic steering.

    • Provides clearer Layer 3 failure boundaries.
    • Supports route summarisation and explicit traffic policy.
    • May require application, DNS or load-balancer changes during migration.
    • Often simplifies fault isolation and scaling.

    VXLAN and EVPN: separating the overlay from the underlay

    VXLAN (Virtual Extensible LAN) encapsulates Layer 2 frames in UDP packets so that virtual networks can be carried across an IP underlay. VXLAN Tunnel Endpoints (VTEPs) perform encapsulation and decapsulation. EVPN (Ethernet VPN), commonly used as a control plane for VXLAN fabrics, distributes MAC and IP reachability information and supports policy-driven tenant connectivity.

    This separation allows an IP-routed fabric to carry multiple logical networks without requiring the physical underlay itself to be one large Layer 2 domain. EVPN can support distributed gateway designs and multi-homing, depending on the implementation. It does not, however, remove the need to design inter-site routing, failure handling and application consistency deliberately.

    Where do OTV, FabricPath and LISP fit?

    Legacy DCI

    OTV

    Overlay Transport Virtualization was designed to extend selected Layer 2 connectivity between data centers while limiting some traditional Layer 2 extension risks. Treat it as a platform- and lifecycle-dependent technology, not a universal modern default.

    Fabric technology

    FabricPath

    Cisco FabricPath introduced a Layer 2 fabric approach using a fabric control plane rather than relying on conventional spanning tree for all forwarding decisions. It is mainly relevant when understanding existing or legacy environments.

    Identifier / locator

    LISP

    Locator/ID Separation Protocol separates endpoint identity from routing location. It can support mobility and mapping use cases in appropriate architectures, but it is not interchangeable with VXLAN or EVPN.

    Design principle: do not extend Layer 2 simply because an application currently uses a fixed IP address. First check whether the application can tolerate a different subnet, use a stable service endpoint, or reconnect through DNS or a load balancer. A routed design may meet the requirement with a smaller failure domain.

    Workload mobility and traffic tromboning

    Preserving a workload’s IP address does not guarantee that its traffic path remains optimal. After a workload moves, clients or dependent services may still reach it through a remote site. This can create traffic tromboning: packets travel across the DCI to reach a service and may cross back again to reach another dependency.

    The result can be increased latency, unnecessary bandwidth consumption and a more complicated failure path. Mobility planning should therefore include endpoint discovery, routing updates, load-balancer behaviour, dependency locality and the time needed for existing sessions to recover.

    Questions to resolve before extending a network

    • Does the workload genuinely require the same subnet at both sites?
    • What are the latency, bandwidth and MTU requirements across the DCI?
    • How are MAC/IP reachability and gateway behaviour handled after a move or failure?
    • Can a Layer 2 loop, broadcast event or control-plane issue affect both sites?
    • Will traffic remain local to the active workload and its dependencies?
    • What happens to established sessions when the workload or site becomes unavailable?
    Engineering takeaway

    VXLAN and EVPN can provide scalable overlay connectivity, while routed designs can keep site boundaries clearer. Choose Layer 2 extension only when the application and operational requirements justify it, and test mobility, failure isolation and traffic locality before relying on it for resilience.

    Interactive design lab · 03

    Layer 2 Extension, Overlays and Workload Mobility

    Compare three ways to connect data-centre sites. Change the design, simulate an inter-site failure, and examine what happens to addressing, mobility and the fault domain.

    1. Choose a connectivity design
    2. Test the inter-site connection
    Conceptual topology Shared Layer 2 segment
    Two-site data-centre connectivity diagram Two data-centre sites with workloads and network gateways connected by an inter-site path. The selected mode changes the path labels and the illustrative failure state. DATA CENTRE A Primary site · example workload DATA CENTRE B Secondary site · example workload Application workload A VLAN 120 · 10.20.120.0/24 Application workload B VLAN 120 · 10.20.120.0/24 Site A gateway Local forwarding Site B gateway Local forwarding L2 extension One broadcast domain Inter-site path operational
    ✓
    Shared segment is available

    Both sites use the same example subnet. This can support address continuity, but it also extends Layer 2 dependencies across the sites.

    Addressing Same subnet at both sites
    Mobility approach Layer 2 continuity
    Failure consideration Broadcast and loop control

    Engineering trade-off

    A stretched VLAN can make subnet continuity easier, but the design must deliberately control loop prevention, broadcast scope, gateway behaviour and inter-site failure handling.

    • Define the Layer 2 extension boundary
    • Test behaviour during link and site failure
    • Validate gateway redundancy and convergence
    • Confirm application recovery requirements
    Educational topology only — not a live network simulator. VXLAN is an encapsulation technology; EVPN can provide a BGP-based control plane. Actual mobility, redundancy and failure behaviour depend on the chosen platform, configuration and application design.
    Section 04 · Application resilience

    Application, Storage and State Consistency

    Network redundancy cannot guarantee application availability on its own. Examine stateless and stateful services, session continuity, synchronous and asynchronous replication, quorum, split-brain risk, and the trade-offs between recovery time and potential data loss.

    Application stateReplicationRPO / RTO
    Section 04 · Application and Data Resilience

    Application, Storage and State Consistency

    A network can keep routes available while an application remains unable to serve requests. Active-active resilience therefore depends on how application instances, user sessions and persistent data behave across sites—not just on redundant links and routers.

    In an active-active environment, application instances in different data centers may process requests concurrently. That arrangement works well only when the application can manage shared state, concurrent operations and failures consistently. A stateless service is often easier to distribute because each request can be handled by any healthy instance, provided its dependencies are also available.

    Stateful applications introduce more complexity. They may depend on databases, distributed caches, message queues, session stores or file systems. Each dependency needs an explicit availability and consistency strategy. Replicating the application tier alone does not make the whole service resilient if its database or identity provider remains a single point of failure.

    Synchronous versus asynchronous replication

    Replication determines how updates made at one site become available at another. The key trade-off is between acknowledgement latency, consistency guarantees and the amount of data that may be lost during a failure. These characteristics vary by platform and configuration; the terms alone do not guarantee a particular recovery outcome.

    Replication model A

    Synchronous replication

    A write is acknowledged only after the configured synchronous durability or commit conditions have been met.

    • Can reduce the recovery point exposure for acknowledged writes.
    • Inter-site latency can affect write response times.
    • A link or site failure may affect write availability, depending on quorum and product design.
    • Requires careful planning for network partitions and failover.
    Replication model B

    Asynchronous replication

    A write can be acknowledged before the remote replica has received or committed the update.

    • Can reduce the impact of remote-site latency on write acknowledgement.
    • Replication lag may leave the remote site temporarily behind.
    • A sudden failure can result in loss of updates not yet replicated.
    • Recovery procedures must account for lag and data reconciliation.

    Important: synchronous replication does not automatically mean zero data loss in every failure scenario, and asynchronous replication does not always imply unacceptable loss. The real outcome depends on commit semantics, quorum, durable storage, topology and the failure being considered. Define the required Recovery Point Objective (RPO) and Recovery Time Objective (RTO), then validate them against the actual platform.

    Three application-level challenges

    Challenge 01

    Session continuity

    If a client moves between sites, its session may depend on local memory, a shared session store, a signed token or another mechanism. Design for reconnects and retries instead of assuming a transport path change preserves every session.

    Challenge 02

    Concurrent updates

    Two active instances may attempt to modify the same logical record. Transactions, conflict resolution, idempotency and application-specific consistency rules must prevent duplicated or conflicting operations.

    Challenge 03

    Dependency availability

    An application instance may be healthy while its database, authentication service, queue or external API is not. Service health should reflect the dependencies required to complete real user operations.

    Quorum, network partitions and split-brain

    A network partition can leave two sites running while preventing them from communicating. If both sides independently accept conflicting writes or claim ownership of the same resource, the system may enter a split-brain condition. Restoring connectivity does not necessarily reconcile the resulting state safely.

    Distributed systems often use quorum, leases, fencing tokens, leader election or other coordination mechanisms to decide which operations are permitted during a partition. A two-site deployment may need a third failure-independent voting participant or witness, depending on the platform. A witness can help establish a decision, but it does not replace data replication or application-level recovery design.

    Network partition is not the same as site failure. If the DCI fails, both sites may still be running and serving local clients. The architecture must decide whether each site can safely continue, whether one side must stop writes, and how service ownership is re-established when communication returns.

    Questions to answer before production

    • Can application instances safely process requests concurrently in both sites?
    • Where is session state stored, and what happens when a client is redirected?
    • What RPO and RTO are required for each critical data store?
    • How are quorum, ownership and split-brain conditions handled during a partition?
    • Can the surviving site sustain the required transaction rate and connection load?
    • How will replication lag, failed writes, stale reads and recovery conflicts be detected?
    • Have failover and failback been tested with real application transactions, not just ping tests?
    Engineering takeaway

    Active-active application resilience requires an explicit strategy for state, replication, coordination and recovery. Network redundancy helps services communicate, but only the combined behaviour of the application, data platform and network determines whether service can continue safely during a failure.

    Interactive design lab · 04

    Application, Storage and State Consistency

    A network can remain connected while an application becomes unsafe or unavailable. Explore replication modes, simulate an inter-site partition, and examine why two active sites need a deliberate write-ownership strategy.

    1. Choose a data replication model
    2. Choose a scenario
    Application and data flow Synchronous data replication
    Two-site application and data replication model Two data-centre sites contain application instances and data services. A solid path represents client and application connectivity, while a dashed path represents replication or inter-site coordination. DATA CENTRE A Serving requests · replica A DATA CENTRE B Serving requests · replica B Application instance A Accepting requests Application instance B Accepting requests Data replica A Commit acknowledgement Data replica B Remote copy Commit coordination Remote acknowledgement Inter-site coordination available
    ✓
    Replication is operating normally

    In this conceptual synchronous model, a write waits for the configured remote acknowledgement before it is confirmed. Real guarantees depend on the data platform and commit policy.

    Write behaviour Waits for required acknowledgement
    Data risk Depends on commit and quorum rules
    Recovery priority Validate quorum and ownership

    Engineering checks

    A healthy network path does not prove that a transaction is durable or that both replicas agree. Check the application's consistency contract and the data platform's failure behaviour.

    • Confirm what a successful commit guarantees
    • Define quorum and write ownership
    • Measure replication delay and transaction errors
    • Test failover, fencing and safe recovery
    Educational model only — no real transactions or replication occur. The diagram illustrates design trade-offs, not a guarantee that a particular database or storage platform behaves this way. Synchronous replication does not automatically prevent data loss in every failure; asynchronous replication may have a non-zero recovery point.
    Section 05 · Operations and recovery

    Failure Scenarios, Observability and Recovery

    Move beyond theoretical redundancy and test how the architecture behaves under real failure conditions. Trace link loss, BGP changes, application dependency failures, replication lag and site isolation using telemetry, incident procedures and measured recovery objectives.

    Failure testingTelemetryFailover and failback
    Section 05 · Observability and Recovery

    Failure Scenarios, Observability and Recovery

    Active-active architecture must be designed around failure, not just normal operation. Reliable recovery depends on detecting what has failed, identifying the affected services, understanding the data state and directing traffic towards capacity that can safely serve it.

    Redundant links and duplicated infrastructure reduce some risks, but they do not eliminate failures. A site can remain reachable while its applications are unhealthy; a routing session can remain established while packets are being dropped; and a database replica can be online while lagging behind the active data set. Effective operations require end-to-end observability across network, application and data layers.

    Detect

    Identify the real failure

    Use telemetry and health checks to distinguish a failed link, an unreachable site, a degraded application and an unhealthy data dependency.

    Decide

    Choose a safe response

    Determine whether traffic should move, whether writes can continue, and whether the surviving site has enough capacity to take additional work.

    Validate

    Prove service recovery

    Confirm that users can complete meaningful transactions, not merely that interfaces are up or routing adjacencies have recovered.

    Five failure scenarios to test

    Inter-data-center link failure

    One DCI path fails while both sites remain operational. Check route convergence, alternate-path capacity, packet loss and whether replication or application dependencies are affected.

    Validate: path failover · latency · loss · capacity

    Border router or routing failure

    A border device fails, a route is withdrawn or a policy change directs traffic incorrectly. Verify that alternate routes are usable and that recovery does not create a forwarding loop or black hole.

    Validate: BGP state · route changes · forwarding path

    Application or dependency failure

    Network connectivity remains available, but an application instance, database or identity service cannot complete requests. Check whether health signals remove the affected endpoint from service selection.

    Validate: synthetic transactions · dependency health

    Replication lag or data inconsistency

    The remote data copy falls behind or a replication process stops. Establish whether the secondary site is safe to promote, what data could be missing and how consistency will be verified before normal service resumes.

    Validate: replication lag · RPO · recovery procedure

    Complete site loss or network partition

    A site becomes unavailable, or both sites remain running but lose communication. Confirm the failover decision, capacity at the surviving site, data ownership and safeguards against split-brain.

    Validate: site isolation · quorum · service continuity

    A practical incident investigation sequence

    During an incident, investigate from the observed user impact towards the underlying dependency. Correlating timestamps across routing events, interface counters, application logs and replication metrics helps distinguish the initiating failure from symptoms that appear later.

    From alert to verified recovery Operational workflow
    1 Scope impact

    Identify affected users, services, sites and transactions.

    2 Correlate signals

    Compare network, application and data telemetry.

    3 Apply safe recovery

    Follow tested routing, failover and data procedures.

    4 Verify service

    Test transactions, data integrity and performance.

    This is a general investigation pattern. Follow your incident procedures and service-specific recovery runbooks when making production changes.

    What should you monitor?

    Layer Useful signals What they help reveal
    Network Interface errors, packet loss, latency, BGP changes, path utilisation Transport degradation, route changes and capacity pressure
    Application Request success rate, latency, error rate, synthetic transactions Whether users can actually complete operations
    Data Replication lag, commit failures, consistency checks, storage health Whether the remote site has a safe and sufficiently current data set
    Platform CPU, memory, connection pools, queue depth, storage throughput Whether the surviving site can absorb additional workload
    Recovery Failover duration, RTO/RPO achievement, recovery test results Whether the design meets its agreed recovery objectives

    Do not confuse an infrastructure health check with a service health check. An interface can be up, a BGP session can be established and a server can respond to ping while a real transaction fails. Use application-aware probes and representative synthetic transactions wherever practical.

    Failover, failback and recovery objectives

    Recovery Time Objective (RTO) defines the target time to restore a service after disruption. Recovery Point Objective (RPO) defines the acceptable data-loss window, expressed as time or an equivalent recovery-point requirement. Different applications and data stores may need different objectives; a single pair of numbers should not automatically be applied to every service.

    Failover is only half of the recovery story. Failback—returning service to its intended operating arrangement—can introduce another period of risk. Before returning workloads, confirm data consistency, replication health, routing policy, capacity and the correct ownership of stateful services. Avoid automatic failback if it could trigger repeated site switching or disrupt active sessions.

    Operational readiness checklist

    • Document the expected behaviour for link, device, application, data and full-site failures.
    • Ensure alerts identify the affected service and provide enough context for investigation.
    • Test routing convergence and verify actual packet forwarding after route changes.
    • Confirm that the surviving site has enough compute, storage, network and application capacity.
    • Define who can authorise failover, data promotion and failback during an incident.
    • Record measured recovery times and data outcomes during controlled exercises.
    • Review failures and near misses to improve runbooks, monitoring and architecture.
    Engineering takeaway

    Resilience must be measured through observed service outcomes. Combine network telemetry, application health, data replication metrics and tested recovery procedures to determine whether the active-active architecture can withstand the failures it was designed to handle.

    Interactive design lab · 05

    Failure Scenarios, Observability and Recovery

    Simulate a router failure, an inter-site partition or a complete site outage. Observe the illustrative network state, monitoring signals and recovery checks that matter before declaring an active-active service healthy.

    1. Select a failure scenario
    Illustrative topology and alternate path All paths operational
    Active-active data-centre failure scenario Two data-centre sites connect through border routers and an inter-site link. A separate illustrative alternate path is shown. Select a scenario to change the topology and monitoring indicators. DATA CENTRE A Site healthy DATA CENTRE B Site healthy Application service A Serving requests Application service B Serving requests Border router A BGP established Border router B BGP established Inter-site path Illustrative alternate path
    ✓
    Both sites are operational

    The conceptual topology shows active services at both sites. Monitoring should confirm end-to-end transaction success, route health, replication status and available capacity.

    Network reachability Primary and inter-site paths available
    Application service Both sites serving
    Recovery decision Monitor and validate health

    Observability panel

    Routing adjacencyHealthy
    End-to-end transactionsHealthy
    Inter-site connectivityHealthy
    Replication / coordinationHealthy

    Recovery checklist

    A green interface or established BGP session is only one signal. Correlate network telemetry with application transactions, data health and the capacity of the surviving site.

    • Confirm the failure and its scope
    • Validate the alternate forwarding path
    • Check application and data integrity
    • Verify recovery objectives are met
    Educational simulation only. No live devices, BGP sessions or monitoring systems are contacted. The alternate path is illustrative; real failover depends on topology, routing policy, detection timers, capacity, application dependencies and the platform's recovery behaviour.
    Section 06 · Hybrid architecture

    Cloud Connectivity and Modern Design Choices

    Explore how data centers connect to cloud networks using private connectivity, VPNs, dynamic routing, VPC peering and hub-based services such as Google Cloud Network Connectivity Center. Compare resilience, route control, security, latency and cost without assuming one cloud model fits every deployment.

    Hybrid cloudVPC and BGPConnectivity choices
    Active-Active Data Center Design · Section 06

    Cloud Connectivity and Modern Design Choices

    Active-active design does not stop at the data center perimeter. Applications may depend on private cloud services, public cloud workloads, shared identity systems, or data platforms in another environment. The challenge is to connect these resources without creating hidden dependencies, unnecessary latency, or new single points of failure.

    1. Separate data center interconnect from hybrid cloud connectivity

    A data center interconnect (DCI) connects separate data center sites. Hybrid cloud connectivity connects an enterprise network or data center to cloud networks and services. These designs can overlap, but they solve different problems and should be assessed independently.

    For example, two active data centers may exchange application traffic over a dedicated DCI while each site also has its own connection to a cloud provider. Another design may route both sites through a shared cloud connectivity hub. The right choice depends on application placement, traffic patterns, security boundaries, and failure-domain requirements.

    Illustrative hybrid connectivity model
    Data Center A Active workloads Local routing and security
    Data Center B Active workloads Independent failure domain
    Enterprise / DCI edge Routing policy and controlled egress
    Cloud network VPCs, cloud workloads and services
    This is a conceptual view, not a required physical topology. Connectivity can be direct, hub-and-spoke, or provider-mediated. Redundancy must be verified across the complete path, not just at the visible connection.

    2. Choose the connectivity model for the workload

    Cloud connectivity is not one universal design. Dedicated private links, provider-managed private connectivity, and IPsec VPNs offer different trade-offs in capacity, latency, encryption, deployment time, and cost.

    01
    Private connectivity

    Often suited to predictable, sustained traffic and workloads with demanding bandwidth or latency requirements. Validate provider availability, port and circuit diversity, routing hand-offs, and the actual end-to-end service commitment.

    02
    Site-to-site VPN

    IPsec VPN can provide encrypted connectivity over an IP underlay. High-availability VPN designs use redundant tunnels or endpoints, but the underlying internet paths and provider failure domains still matter.

    03
    Cloud routing

    Cloud routers and BGP sessions can exchange routes dynamically where supported. Define which prefixes are advertised, accepted, preferred, or filtered, and test behaviour when a peer or path fails.

    04
    Hub-and-spoke connectivity

    A cloud transit or connectivity hub can simplify connections between multiple networks. It also becomes an important policy and operational component that needs its own resilience and capacity planning.

    3. Understand VPCs, peering and cloud connectivity hubs

    In Google Cloud, a Virtual Private Cloud (VPC) provides a logical network boundary for cloud resources. VPC connectivity, Cloud VPN, Cloud Router, and Network Connectivity Center (NCC) address different parts of hybrid and multi-network design. Other providers have comparable concepts, but their capabilities and terminology differ.

    Building block Design role Important consideration
    VPC network Provides a logical network environment for cloud resources. Plan address ranges, segmentation, route policy, and access controls.
    VPC peering Connects eligible VPC networks through a peering relationship. Do not assume peering provides transitive routing through another VPC.
    Cloud VPN Provides IPsec-based connectivity between networks. Plan tunnel redundancy, routing, encryption requirements, and failover testing.
    Cloud Router / BGP Supports dynamic route exchange for compatible connectivity services. Validate advertised prefixes, learned routes, route priorities, and withdrawal behaviour.
    Network Connectivity Center Provides a hub-based framework for supported network connectivity patterns. Check supported resources, topology constraints, route propagation, and configuration requirements.
    Do not confuse connectivity with end-to-end resilience

    A cloud hub, a pair of VPN tunnels, or two physical circuits does not automatically make an application highly available. Shared carrier infrastructure, common provider endpoints, route-policy errors, cloud-region dependencies, DNS, identity, and application state can still create a common failure mode.

    4. Design the routing and failure behaviour

    Once connectivity is established, define what happens when a link, tunnel, router, data center, or cloud attachment becomes unavailable. Routing must support the intended failover behaviour without creating loops, accidental transit, or unintended access between security zones.

    Route control

    Advertise only required prefixes. Apply explicit import and export policy, avoid overlapping address spaces, and understand how route preference changes during failure.

    Path diversity

    Check whether redundant connections use independent devices, facilities, carriers, cloud endpoints, and regions where the availability objective requires it.

    Security boundaries

    Apply segmentation, firewall policy, identity controls, and inspection requirements to both primary and failover paths. A backup route should not bypass required controls.

    Application dependencies

    Validate DNS, authentication, secrets, databases, monitoring, and other shared services. Network reachability alone does not prove that the service can operate.

    5. Compare the design options before committing

    Select connectivity according to measured workload requirements and operational constraints—not simply because a particular product or topology is available.

    Design question What to evaluate
    Traffic profile Peak and sustained throughput, burst behaviour, packet size, and growth expectations.
    Performance Latency, jitter, packet loss, regional distance, and application sensitivity.
    Availability Independent paths, redundant endpoints, provider failure domains, and tested recovery behaviour.
    Routing Address planning, BGP policy, route propagation, failover timing, and prevention of unintended transit.
    Security Encryption, segmentation, inspection, identity, and compliance requirements.
    Operations and cost Provisioning lead time, monitoring, support ownership, data transfer charges, and ongoing operational effort.

    6. Validate the architecture with failure tests

    Test the complete application path, including the network and the dependencies it uses. Record what users experience, how quickly the system detects a failure, which routes change, and whether recovery occurs within the required objective.

    • Disconnect one connectivity path and verify traffic moves as designed.
    • Withdraw a route and confirm that only intended prefixes change reachability.
    • Test a cloud attachment or VPN endpoint failure where the environment permits.
    • Verify that security policy remains enforced on the alternate path.
    • Measure application recovery, not just routing convergence.
    • Document ownership, alerts, escalation, and restoration procedures.
    Engineering takeaway

    Active-active data center design and hybrid cloud connectivity should be designed together where workloads depend on both—but they are not the same problem. Choose private connectivity, VPN, peering, or a hub-based model according to workload needs, then validate route policy, failure-domain independence, security, and application recovery end to end.

    Design note: cloud networking products and capabilities change over time. Confirm current provider documentation, regional availability, quotas, and supported topology before implementation.
    Interactive design explorer · Section 6

    Cloud Connectivity and Modern Design Choices

    Choose a connectivity pattern to explore how an on-premises network can connect to cloud networks. Compare the conceptual architecture, routing considerations and security decisions before selecting a design for a real environment.

    01 / Select a connectivity pattern
    02 / Architecture view Encrypted tunnel
    Site-to-site VPN cloud connectivity An on-premises network connects through a VPN gateway to a cloud network. The tunnel is conceptual; actual routing and redundancy depend on configuration. ON-PREMISES Enterprise network Core router BGP / static Workloads Private LAN CLOUD Virtual network Gateway VPN endpoint Cloud apps Private subnets IPsec VPN tunnel Routes configured at both ends Tunnel redundancy depends on deployment

    Conceptual diagram, not a provider-specific reference architecture. Redundant tunnels, gateways and route exchange must be designed and tested explicitly.

    03 / Design assessment

    Site-to-site VPN

    Build an encrypted connection between the enterprise network and the cloud provider's VPN endpoint. The internet may carry the tunnel traffic, while routing controls which prefixes can traverse it.

    Connection model Encrypted tunnel
    Routing focus Prefixes and failover
    Security focus IKE / IPsec policy
    Resilience focus Multiple tunnels
    Engineering checks
    • Confirm tunnel encryption, authentication and key-management requirements.
    • Validate route advertisements, allowed prefixes and return-path symmetry.
    • Test failure detection, alternate paths, throughput and maximum packet size.
    Watch-outs
    • A tunnel being established does not prove application reachability.
    • One tunnel or gateway can remain a single point of failure.

    High availability is an end-to-end property

    Consider independent circuits, gateways, cloud regions or zones where supported, route convergence, capacity under failure, and the application's own dependency model. Two connections that share the same failure domain may not provide meaningful redundancy.

    Routing and security are separate decisions

    Route exchange determines reachability; firewall policy, segmentation, identity and encryption determine what is permitted and protected. Validate both directions and test the application path rather than relying on a successful routing adjacency alone.

    Network Insight · Article Summary

    Active-Active Data Center Design: Key Takeaways

    A practical recap of the architecture, routing, application dependencies, failure planning and cloud connectivity decisions behind a resilient multi-site data center.

    Active-active data center design aims to use multiple sites to serve workloads concurrently while maintaining service continuity when links, network devices, applications or entire sites fail. Achieving this requires more than duplicating infrastructure: routing, application state, data consistency, security and recovery procedures must work together.

    01

    Architecture and workload placement

    Define which services operate at each site, how traffic is distributed, and what happens when one location becomes unavailable. Active-active is a service and workload design decision, not simply a topology with two data centers.

    02

    Multi-site routing and BGP

    Use explicit routing policy, controlled prefix advertisement and deliberate path selection. Validate convergence, asymmetric paths, route withdrawal and the capacity of the surviving site under failure conditions.

    03

    Layer 2 extension and overlays

    Choose between routed inter-site connectivity, selective Layer 2 extension and overlay-based approaches according to workload requirements. Avoid extending broadcast domains across sites without a clear operational and failure-containment justification.

    04

    Application and data consistency

    Understand replication lag, transaction behaviour, quorum, write ownership and recovery objectives. Network reachability alone cannot guarantee consistent data or prevent conflicting writes during a partition.

    05

    Failure detection and recovery

    Monitor routing and infrastructure alongside application transactions, replication health and user-visible service availability. Test realistic failures and document safe failover, failback and recovery procedures.

    06

    Cloud connectivity and modern choices

    Select VPN, private connectivity, network peering or hub-based connectivity based on reachability, routing, security and resilience requirements. Confirm provider-specific route propagation and service limitations before implementation.

    Six engineering principles to carry into your design

    • Design around service availability, not device availability alone.
    • Ensure the surviving site has sufficient capacity for its intended failure role.
    • Keep routing policy, application behaviour and data consistency aligned.
    • Use Layer 2 extension only where the workload genuinely requires it.
    • Measure end-to-end service health, not just link and routing status.
    • Prove failover and recovery through repeatable tests and documented outcomes.

    Final perspective: resilience must be engineered and tested

    The strongest active-active designs make failure behaviour explicit. They define how traffic moves, how stateful applications continue safely, how data is protected, and how operators recognise and resolve degraded service. They also account for partial failures, where a site remains reachable but can no longer safely perform its intended role.

    No single technology guarantees resilience. BGP, overlays, replication, cloud connectivity and redundant infrastructure are building blocks whose effectiveness depends on the requirements, implementation and operational testing around them.

    The central takeaway

    Build for the failure you need to survive, define the service outcome you must preserve, and test that outcome before relying on the architecture in production.

    Network Insight · Data Center Architecture · Routing · High Availability · Cloud Networking
    ```
    ☕
    Support Network Insight
    Help support independent engineering guides, interactive networking tools, and future learning labs.
    Support Network Insight ```