Redundant Links with Virtual PortChannels
Modern data centres need more than a backup connection. They need resilient paths, effective bandwidth utilisation and predictable behaviour when links or switches fail. Virtual PortChannels (vPC) help achieve active-active Layer 2 connectivity by allowing a downstream device to form a port-channel across two compatible Cisco Nexus switches.
Traditional designs can leave redundant paths blocked by Spanning Tree Protocol (STP), while a poorly planned aggregation design can create uneven traffic distribution or unexpected failure conditions. Port channels, LACP and multi-chassis link aggregation provide different mechanisms for building resilient connectivity—but each has its own operational requirements and failure modes.
This guide examines how these mechanisms work, how to verify them, and how to investigate what happens when a member link, peer-link or switch becomes unavailable. It also connects Layer 2 redundancy to Layer 3 forwarding and BGP, where routing decisions and equal-cost multipath (ECMP) can provide additional resilience across a data-centre fabric.
What you will learn
Port-channels and LACP
Understand physical member links, logical interfaces, negotiation modes and link-level redundancy.
vPC operation
Explore peer-links, keepalive connectivity, consistency checks, primary and secondary roles, and orphan ports.
Hashing and load balancing
Learn why multiple active links do not guarantee perfectly even traffic distribution and how to identify polarization.
BGP and ECMP
Connect resilient Layer 2 access with routed leaf-spine fabrics, BGP path selection, next-hop reachability and multipath forwarding.
Failure investigation
Trace link failures, peer-link loss, routing changes and recovery using a structured troubleshooting workflow.
Design and verification labs
Apply the concepts to topology walkthroughs, failure scenarios, configuration checks and design decisions.
Key design principle: vPC and BGP solve different problems. vPC provides multi-chassis Layer 2 link aggregation; BGP exchanges routing information and can support multipath forwarding in routed designs. A resilient architecture must account for both the data path and the control-plane behaviour during failures.
Network Insight · Article Guide
Explore the vPC Learning Guide
Navigate from EtherChannel fundamentals to vPC architecture, BGP and ECMP integration, failure investigation and practical labs. Select any section to jump directly to its content.
Section 01 · Foundations
Redundant Links, EtherChannel and vPC Fundamentals
Redundancy is a core requirement in data-centre networking. Applications, servers and network services need connectivity to continue when an individual cable, interface or switch fails. However, adding physical links alone does not guarantee that all available capacity will be used efficiently or that every failure will be handled correctly.
A robust design must combine physical connectivity with mechanisms that control forwarding, distribute traffic and respond to failures. EtherChannel groups multiple Ethernet links into one logical interface, while Cisco Virtual PortChannel (vPC) extends the port-channel concept across a pair of supported switches.
Why a single link is not enough
A single connection between a server and a switch creates a potential single point of failure. If the cable, interface or switch port becomes unavailable, traffic using that connection may be interrupted until connectivity is restored or an alternative path is selected.
Multiple links can improve capacity and resilience, but their behaviour depends on how they are configured. Independent parallel Layer 2 links can create loops, while designs relying on Spanning Tree Protocol (STP) may place some redundant links into a non-forwarding state for a given VLAN. Link aggregation provides another approach by treating compatible physical links as one logical connection.
Physical redundancy
Multiple cables and interfaces reduce dependence on a single physical connection. Diversity of line cards, paths and power domains can provide further protection when designed appropriately.
Logical aggregation
A port-channel presents a bundle of compatible member links as one logical interface to supported protocols and connected devices.
Multi-switch resilience
vPC allows a supported downstream device to connect its port-channel to two separate Cisco Nexus peers, subject to the platform's design and configuration requirements.
EtherChannel versus vPC
These terms are related, but they describe different parts of the design. A conventional port-channel aggregates links between two endpoints. A vPC allows the two endpoints of a downstream logical port-channel to reside on separate switches within a configured vPC domain.
| Technology | Primary purpose | Design consideration |
|---|---|---|
| EtherChannel / port-channel | Aggregate multiple physical links into one logical interface. | Member links must meet compatibility requirements and share a supported aggregation configuration. |
| LACP | Negotiate and maintain a link aggregation group. | Active/passive modes and configuration compatibility affect whether the bundle forms. |
| vPC | Provide a multi-switch port-channel to a connected device. | Peer-link health, keepalive, consistency and failure handling must be designed and verified. |
| STP | Prevent Layer 2 forwarding loops in bridged networks. | It remains relevant to the broader topology; vPC does not remove every STP requirement. |
How a port-channel uses multiple links
When a port-channel is operational, the switch selects a physical member for each eligible traffic flow using a platform-dependent hashing algorithm. Depending on the configured inputs, the hash may use source and destination MAC addresses, IP addresses or transport-layer ports. Different flows can therefore use different members.
- Configure the member interfaces. Ensure the physical interfaces have compatible settings and are eligible to join the same aggregation group.
- Establish the logical bundle. Configure the port-channel and, where used, compatible LACP settings. Verify that the expected members become active.
- Distribute traffic. The switch hashes eligible flows onto member links. This is normally flow-based distribution, not a guarantee that every packet takes the next available link.
- Handle a member failure. Traffic can use surviving active members if the bundle remains operational, although capacity is reduced and some flows may be temporarily disrupted.
Engineering insight: Aggregating four links does not mean every individual flow receives four times the bandwidth. A particular flow is typically assigned to one member at a time, and actual throughput depends on hashing, traffic patterns, link speeds and the capabilities of the platform.
Where BGP enters the picture
Port-channels and vPC primarily address link aggregation and Layer 2 connectivity. In routed data-centre fabrics, BGP can provide a separate layer of resilience by exchanging reachability information between routers and supporting equal-cost multipath (ECMP) where the design, configuration and platform permit it.
This distinction matters during failures. A physical member failure may be handled within a port-channel without the routing protocol seeing a change to the logical interface. By contrast, a routed link or next-hop failure may trigger routing updates and a new forwarding decision. The recovery process depends on which component failed and how the control and data planes are configured.
Key takeaway: Redundant links, EtherChannel, LACP, vPC and BGP are complementary tools—not interchangeable technologies. Start by identifying the failure domain you need to protect, then select the appropriate mechanism and verify its actual failure behaviour.
Interactive learning · Module 01
From Redundant Links to vPC
Explore three ways to connect a server or downstream switch to the network. Compare a standalone physical link, a conventional EtherChannel and a virtual PortChannel across two switches.
vPC: a multi-switch port-channel
A downstream device can form a port-channel across two vPC peers. The switches coordinate through the vPC architecture, allowing supported topologies to use links to both switches without treating them as unrelated standalone connections.
Important: The design still depends on peer-link health, keepalive behavior, configuration consistency, and the platform's documented failure handling.
Knowledge check
What is the key architectural difference between a conventional EtherChannel and a vPC?
Educational illustration only. The link-failure control changes the diagram, not a real network device. Exact behavior depends on the vendor platform, topology and configuration.
Section 02 · Architecture
Understanding the vPC Peer-Link, Keepalive and Consistency
A vPC domain consists of two cooperating switches that maintain separate control planes while presenting a coordinated multi-chassis port-channel to connected devices. This arrangement enables active forwarding across supported links, but it also introduces important synchronization and failure-handling requirements.
Three concepts are central to understanding vPC operation: the peer-link, the peer-keepalive link and consistency checks. They have different purposes. The peer-link carries required inter-peer traffic and synchronization; the keepalive helps peers determine whether the other switch is alive; and consistency checks identify configuration differences that could compromise correct operation.
1. The main components of a vPC domain
Peer-link
A logical link between the two vPC peers. It carries synchronization and selected traffic required by vPC operation. Its capacity, redundancy and failure behaviour must be planned carefully.
Peer-keepalive
A separate heartbeat path used to help distinguish a peer failure from a peer-link failure. It should be designed to remain available independently of the peer-link where practical.
Consistency checks
Checks compare relevant configuration parameters across peers. Incompatible settings can prevent a vPC from coming up or cause a member to be suspended, depending on the parameter and platform.
2. How the peer-link supports the domain
The peer-link is not simply another downstream traffic connection. It supports communication between the vPC peers and provides a path for certain traffic when the normal forwarding path is unavailable. The network should not depend on it as the routine path for all end-host traffic.
Conceptual representation only. Actual peer-link requirements and forwarding behaviour depend on the Cisco Nexus platform, software release and configuration.
Under normal conditions, traffic from a downstream device should use its active vPC member links. If a peer loses its local member links for a particular vPC, the surviving peer and peer-link may be involved in forwarding traffic, subject to the applicable vPC forwarding rules. This is one reason peer-link capacity and failure scenarios matter.
Design principle: Do not treat the peer-link as a substitute for correctly designed downstream and routed connectivity. Understand which traffic can cross it, how much capacity is required, and what the platform does when peer-link members fail.
3. Why the peer-keepalive is separate
The keepalive mechanism helps the peers determine whether the other switch is reachable when peer-link communication is interrupted. That distinction is important: losing the peer-link does not automatically mean that the other switch has failed.
| Observed condition | What it may indicate | Engineering response |
|---|---|---|
| Peer-link up, keepalive reachable | Normal peer communication, assuming the domain is otherwise healthy. | Verify peer-link members, consistency status and vPC state. |
| Peer-link down, keepalive reachable | The peers may still be alive, but their main synchronization path is unavailable. | Investigate peer-link interfaces, port-channel state and the platform's documented failure response. |
| Peer-link down, keepalive also unavailable | The peers may be unable to distinguish a peer failure from a communication partition. | Check peer reachability, keepalive path independence and the documented dual-active protection behaviour. |
| One peer is genuinely down | The surviving peer may continue forwarding supported traffic, depending on the failure and configuration. | Verify vPC state, downstream connectivity and recovery after the peer returns. |
These are diagnostic patterns, not a universal state machine. The exact response to peer-link and keepalive failures is platform- and configuration-dependent, so operational decisions must be checked against the relevant vendor documentation.
4. Consistency checks: preventing incompatible configurations
Because both switches participate in a coordinated logical topology, important configuration must be compatible. Differences in VLAN membership, port-channel parameters, trunk settings or other vPC-relevant configuration can affect operation. Not every parameter is treated identically: some inconsistencies are more serious than others, and the resulting action depends on the specific check.
- Confirm both peers belong to the intended vPC domain.
- Verify peer-link status and its member interfaces.
- Check vPC consistency output for global and per-vPC issues.
- Compare VLANs, allowed VLAN lists, MTU and relevant interface settings.
- Confirm port-channel membership and LACP state on both peers.
- Review platform-specific requirements before changing a live domain.
5. Primary and secondary roles
vPC peers have primary and secondary roles for particular operational decisions. These roles do not mean that the secondary peer is simply an idle standby switch: both peers can forward traffic over their supported active vPC links. Role election, role changes and behavior during failure must be understood in the context of the platform and release.
The design should document the intended role priorities and the expected behavior during reloads, peer-link loss, keepalive loss and recovery. Avoid assuming that a role label alone predicts which traffic will be forwarded after every possible failure.
6. Orphan ports and single-homed devices
An orphan port is typically a port connected to a device through only one vPC peer rather than through a vPC to both peers. Such devices deserve special attention because a peer or peer-link failure can leave the device with different connectivity from a dual-homed server.
Identify orphan-connected devices, determine which peer they depend on, and test their reachability under the failure conditions relevant to the design. Where necessary, use supported platform-specific protection mechanisms and documented operational procedures.
Key takeaway: Reliable vPC operation depends on more than bringing up a port-channel. The peer-link, keepalive, consistency state, role behavior and orphan-port exposure must all be understood together. These are the foundations for the failure investigations covered later in this guide.
Interactive learning · Module 02
Inside the vPC Peer Relationship
The peer-link and peer-keepalive serve different purposes. Explore how their states differ and why you must investigate both before drawing conclusions about a vPC failure.
Normal operation
The peer-link supports the vPC peer relationship, while the keepalive path provides a separate means of checking peer liveness. Confirm that both paths and the vPC consistency status are healthy.
What to investigate: Peer status, peer-link members, keepalive reachability and consistency checks.
Investigation checklist
Use these checks to guide your diagnosis for the selected condition.
Example verification commands
For a Cisco NX-OS-style environment, relevant commands may include:
show vpc show port-channel summary show interface port-channel 10 show logging
Command availability and output vary by platform and release. Use the appropriate vendor documentation for peer-keepalive and consistency verification.
Knowledge check
Why is it important to check the peer-keepalive path separately from the peer-link?
This diagram is a conceptual teaching aid, not a device-state simulator. A real peer-link or keepalive failure can have platform-specific consequences. Never infer the exact forwarding outcome from a diagram alone.
Section 03 · Traffic Engineering
Load Balancing, Hashing and Traffic Polarization
A port-channel can provide multiple active physical links, but those links will not necessarily carry equal amounts of traffic. Switches commonly use a hashing algorithm to select a member link for each eligible flow. The result depends on the traffic pattern, the selected hash inputs and the implementation supported by the platform.
Understanding this distinction is essential when designing redundant data-centre connections. A bundle may be operational and have all its members up, yet one link can become heavily utilized while other members remain comparatively idle. This uneven distribution is often described as traffic polarization or hash imbalance.
1. How port-channel hashing works
When a frame or packet is eligible for forwarding through an aggregated interface, the switch evaluates selected header fields. It uses those values to calculate a hash and map the result to one of the available physical members.
Simplified conceptual model. Exact hash algorithms, fields, seed values and member-selection behavior vary by platform and software release.
2. Common hash inputs
Depending on the platform and configuration, the hashing algorithm can use Layer 2, Layer 3 or Layer 4 information. A richer set of inputs can help distinguish traffic flows that share the same source or destination address.
| Hash input | What it identifies | Potential limitation |
|---|---|---|
| Source MAC | The originating Layer 2 address. | Many flows from one source can share the same value. |
| Destination MAC | The destination Layer 2 address. | Traffic directed to one next-hop MAC can concentrate on a subset of members. |
| Source and destination IP | The Layer 3 endpoints of a flow. | A workload with a small number of IP endpoint pairs may not spread evenly. |
| TCP/UDP ports | Transport-layer conversations between endpoints. | Availability and use of these fields depend on the platform, packet type and configured hashing policy. |
| Combined fields | A broader flow identity using multiple header values. | More inputs can improve distribution but cannot guarantee perfectly balanced utilization. |
3. Why traffic polarization occurs
Hashing is not the same as sending each new packet to whichever link is least busy. It is generally designed to keep a flow on a consistent member, helping preserve packet ordering. As a result, the number of flows, their sizes and the distribution of their hash values influence the load on each member.
Uneven flow sizes
A few high-volume transfers can consume most of one member's capacity, even when the total number of flows appears balanced.
Limited hash diversity
If many flows share the same relevant header fields, they may map to the same member or a narrow subset of members.
Multiple hashing stages
Traffic crossing several aggregation or ECMP stages can experience correlated distribution effects, leaving some paths busier than others.
Important: A four-member port-channel does not guarantee that every flow receives one quarter of the bandwidth. Nor does a 25% share of the flow count guarantee a 25% share of bytes. Always evaluate actual traffic volume and member utilization.
4. Choosing and validating a hashing policy
Hashing configuration should reflect the traffic that the network actually carries. A policy suitable for many server-to-server flows may perform poorly when most traffic is concentrated on a small number of destinations. The available choices and their effects must be verified against the switch platform's documentation.
- Identify the main traffic sources, destinations and application flow patterns.
- Check which hash inputs and load-balancing options the platform supports.
- Measure traffic rates and errors on individual physical members, not only the logical port-channel.
- Compare flow distribution with byte counts, packet counts and interface utilization.
- Test whether the selected policy improves distribution without creating unexpected forwarding behavior.
- Recheck the distribution after link failures, workload changes or configuration updates.
5. The relationship between port-channel hashing and BGP ECMP
Port-channel hashing and BGP ECMP both distribute traffic across multiple forwarding choices, but they operate at different levels. Port-channel hashing chooses a physical member of an aggregated link. ECMP chooses between eligible equal-cost next hops in the routing and forwarding system.
In a routed leaf-spine fabric, a packet might be assigned to an ECMP next hop and then traverse an aggregated physical connection. Each stage may apply its own hashing behavior. Good distribution at one stage does not guarantee good distribution end to end.
| Mechanism | Selection being made | What to investigate |
|---|---|---|
| Port-channel hashing | Which physical member carries a flow. | Member utilization, hash inputs, link capacity and traffic skew. |
| BGP ECMP | Which eligible equal-cost next hop is used. | Installed routes, eligible paths, next-hop resolution and forwarding-table programming. |
| End-to-end forwarding | The combined path through the network. | Traffic distribution at each stage, congestion points and the effect of failures. |
6. Diagnosing an imbalanced bundle
Begin by confirming that the port-channel and its expected members are operational. Then examine per-member traffic counters over the same measurement interval. Look for sustained utilization differences, errors, drops or unexpected changes after a member or path failure.
If the physical links are healthy but the load remains uneven, compare the observed distribution with the workload's flow sizes and the configured hashing inputs. Where the design also uses routed ECMP, inspect route and next-hop state separately rather than assuming that the port-channel explains every imbalance.
Key takeaway: Multiple active links create forwarding choices, not automatic load equality. Effective load balancing comes from understanding hash behavior, measuring real traffic, and verifying distribution at both the link-aggregation and routing layers.
Interactive learning · Module 03
Explore Port-Channel Hashing
A port-channel usually selects an egress member using a hash of configured packet fields. The goal is to distribute flows across available links while keeping packets in a flow on a consistent path. Change the inputs and observe this illustrative model.
Illustrative load by member
The bars represent a modelled share of traffic, not measured link utilisation. A real switch uses platform-specific hash algorithms and forwarding details.
Why hashing matters
Diverse flows can spread across multiple members. The distribution is not guaranteed to be perfectly even, and a single flow is generally not split packet-by-packet across every member.
Knowledge check
Why can a port-channel show uneven utilisation even when all member links are up?
Educational model only: member selection and load percentages are illustrative, not a reproduction of a specific vendor's hashing algorithm. Hash fields, resilient hashing, member selection and failure behavior depend on platform and configuration.
Section 04 · BGP and Routed Resilience
BGP and Layer 3 Routing in the Data Centre
Virtual PortChannels provide resilient Layer 2 connectivity by allowing a device to form a logical port-channel across two cooperating switches. BGP addresses a different requirement: exchanging routing information between routers and selecting eligible paths through a Layer 3 network. In modern data centres, both technologies may be used in the same architecture, but they solve different problems.
Understanding the boundary between Layer 2 redundancy and Layer 3 routing is important when designing leaf-spine fabrics, connecting server networks to routed infrastructure, or investigating a failure that affects only part of the forwarding path.
1. A leaf-spine fabric using BGP
In a common routed leaf-spine design, leaf switches connect to servers, storage or other endpoint networks. Spine switches provide routed connectivity between the leaves. BGP can exchange reachability information across these links and support multiple eligible paths through the fabric.
Conceptual topology only. A production fabric normally has routed links from each leaf to each spine. The diagram does not represent every physical connection.
With multiple spine switches, a leaf may learn multiple eligible routes to a destination. If the routing policy and platform permit equal-cost multipath forwarding, the switch can install multiple next hops and distribute traffic across them.
2. BGP, ECMP and next-hop selection
BGP route exchange
BGP advertises network reachability and path attributes between peers. Policy and best-path processing determine which paths are selected or made eligible for multipath forwarding.
ECMP forwarding
Equal-Cost Multipath can use multiple eligible next hops for a destination when the routing system and forwarding platform support and install those paths.
Next-hop resolution
A route is useful only when its next hop can be resolved and programmed into the forwarding system. Routing and interface state must be checked together.
Important distinction: BGP does not automatically mean that all available paths will carry traffic. Route policy, path attributes, next-hop reachability, multipath configuration and platform capabilities all influence the forwarding result.
3. vPC and BGP solve different problems
A vPC pair presents a multi-chassis Layer 2 aggregation capability to a connected device. BGP exchanges Layer 3 routes between routing peers. One technology does not replace the other, and their relationship depends on the overall network design.
| Design concern | vPC / port-channel | BGP / ECMP |
|---|---|---|
| Primary purpose | Redundant Layer 2 connectivity and aggregated links. | Route exchange and Layer 3 path selection. |
| Forwarding choice | Selects a physical member within a logical link. | Selects among eligible routed next hops. |
| Failure investigation | Check member state, peer-link state, consistency and port-channel counters. | Check BGP sessions, routes, next hops, route policy and forwarding-table state. |
| Design trade-off | Requires careful multi-chassis coordination and failure handling. | Requires sound routing policy, path design and convergence planning. |
For example, a server may connect redundantly to a vPC pair while those switches participate in a separate routed fabric. Alternatively, a design may use routed server-facing links and ECMP without relying on a vPC pair for that connection. The appropriate choice depends on the desired failure domains, operational model and platform support.
4. What happens when a link or path fails?
Resilience depends on which component fails and how that failure is detected. A physical member failure inside a port-channel is different from the loss of a routed leaf-to-spine link, a BGP session failure, or the loss of an entire switch.
| Failure event | What may happen | What to verify |
|---|---|---|
| One port-channel member fails | Traffic may continue over remaining eligible members, with reduced aggregate capacity. | Member state, errors, capacity and traffic redistribution. |
| One routed fabric link fails | Routing and forwarding may use another eligible path if one is available and installed. | Interface state, route changes, next-hop programming and packet loss. |
| A BGP session fails | Routes learned through that session may be withdrawn or otherwise changed according to protocol state and policy. | Session state, route withdrawals, alternate paths and convergence time. |
| A switch fails | Recovery depends on the topology, remaining paths, gateway design and failure-detection mechanisms. | Reachability, redundant paths, gateway behavior and application impact. |
These are general outcomes, not guarantees. Actual behavior depends on the topology, platform, configuration, failure-detection timers and the availability of alternate forwarding paths.
5. A practical BGP and ECMP verification workflow
When traffic does not follow the expected path, separate the investigation into routing control-plane checks and forwarding data-plane checks. A BGP session can be established while a particular destination is missing, rejected by policy or not installed for multipath forwarding.
- Check adjacency: Confirm the expected BGP peers are established and inspect recent session changes.
- Check route reception: Verify that the expected prefix is received and accepted by policy.
- Check best-path and multipath eligibility: Determine why paths are selected, rejected or excluded from ECMP.
- Check next-hop reachability: Confirm that each intended next hop is resolvable.
- Check forwarding state: Verify which next hops are actually installed in the routing or forwarding table.
- Measure traffic: Inspect per-interface counters, drops, utilization and the effects of a controlled failure test.
Engineering takeaway: vPC provides one form of Layer 2 resilience; BGP and ECMP provide routed reachability and multipath forwarding. Reliable data-centre design requires engineers to understand each layer, test its failure behavior and verify the complete forwarding path.
Interactive learning · Module 04
BGP Route Propagation and ECMP
Follow an illustrative route advertisement through a data-centre fabric. Then withdraw one route and inspect how the available paths change. BGP selects routes; ECMP can allow multiple eligible next hops to participate in forwarding.
Step 1 — A prefix is originated
A network advertises 10.10.10.0/24 into the routing system. The receiving router must learn the route before it can evaluate the path and install an eligible next hop.
Engineering focus: Confirm the route exists at the origin and that the intended BGP session is established.
Path status
What to verify on a real network
Use commands appropriate to your router OS and design:
show ip bgp summary show ip bgp 10.10.10.0/24 show ip route 10.10.10.0/24 show ip cef 10.10.10.0/24
These are illustrative Cisco IOS-style examples; exact syntax and ECMP inspection commands vary by platform.
Knowledge check
What is the key difference between BGP path selection and ECMP forwarding?
This is a conceptual learning model, not a live BGP implementation. ECMP depends on the routes accepted, next-hop resolution, routing policy, platform capabilities and configuration. The diagram does not imply that every displayed link is a BGP adjacency in every design.
Section 05 · Resilience Engineering
Failure Scenarios, Redundancy and Network Convergence
Redundancy is only valuable when the network can respond correctly to a failure. A vPC design may provide multiple physical connections, while BGP and ECMP provide alternative routed paths. However, each mechanism detects different events and follows its own recovery behavior. Engineers must understand how those mechanisms interact during a real outage.
The goal is not simply to keep every link green. It is to preserve reachability, maintain acceptable application performance and restore the intended forwarding state within an acceptable period.
1. Understand the failure domain
Before troubleshooting, identify which component has failed. A single physical member, an entire port-channel, a vPC peer-link, a peer switch or a routed BGP adjacency represents a different failure domain. The observed symptom alone may not identify the root cause.
Physical-link failure
One member stops forwarding. If the logical port-channel remains operational, traffic may continue over the remaining active members, but total capacity is reduced.
vPC peer-link failure
The two peers lose their inter-peer connection. Peer-keepalive status and platform-specific protection behavior become important when determining whether forwarding should continue.
Routed-path failure
A routed interface, next hop or BGP session may become unavailable. Traffic can use an alternate path only if one is eligible, reachable and installed for forwarding.
2. What happens during common failures?
| Failure scenario | Expected consideration | Engineering checks |
|---|---|---|
| One member link fails | Remaining port-channel members may continue carrying traffic, with less available bandwidth. | Member state, interface errors, utilization and port-channel status. |
| All members of a logical link fail | The logical connection may go down, triggering another forwarding or routing response depending on the topology. | Logical interface state, alternate paths, route changes and endpoint reachability. |
| vPC peer-link fails | Peer behavior depends on keepalive reachability, peer roles, configuration and platform protection mechanisms. | Peer-link state, keepalive status, consistency state and relevant system logs. |
| One vPC peer fails | Surviving connectivity depends on the remaining peer, attached links, gateway design and failure handling. | Peer status, server-facing links, gateway reachability and application traffic. |
| BGP adjacency fails | Routes learned through that session may be withdrawn or changed; alternate forwarding depends on available paths and convergence. | BGP session state, route table, next hops and forwarding-table entries. |
| Uneven traffic after recovery | Traffic may redistribute differently because of changed members, paths or flow hashes. | Per-member counters, ECMP next hops, drops and traffic distribution. |
Design warning: Do not assume that every failure triggers immediate, lossless failover. Detection timers, protocol behavior, hardware programming, topology and configuration all influence recovery. Validate the behavior on the actual platform.
3. Control-plane convergence versus data-plane forwarding
The control plane detects and communicates changes in network state. The data plane forwards packets using the currently programmed forwarding information. These processes are related, but they are not identical and may not complete at exactly the same time.
A physical interface, peer relationship or routing session detects a change through its configured detection mechanism.
The affected protocol or subsystem updates its state. A routing change may cause paths to be withdrawn, recalculated or selected.
Hardware or software forwarding state is updated to use the remaining eligible links or next hops.
Traffic resumes along the available path, but packet loss, latency or reduced capacity may remain during or after the transition.
This sequence is a conceptual troubleshooting model. Some systems react locally before routing convergence, and implementations can overlap or reorder these stages.
4. Why alternate paths do not guarantee zero packet loss
A redundant path may exist without being immediately usable. The failure must be detected, the relevant forwarding decision must change, and the alternate path must be capable of carrying the traffic. A routing protocol can also have a valid alternate route that is not eligible for installation under the current policy or multipath rules.
- Capacity: Remaining links must be able to accommodate the shifted traffic load.
- Reachability: Alternate next hops and destinations must remain reachable.
- Convergence: Detection and forwarding updates must complete within the application's tolerance.
- Consistency: vPC peers and connected devices must maintain a valid forwarding arrangement.
- Asymmetric paths: Forward and return traffic may follow different paths, which can affect stateful devices or troubleshooting.
- Failure correlation: Redundant links should not share an avoidable single point of failure, such as one cable path or power domain.
5. A structured failure-testing workflow
Controlled testing is more reliable than assuming redundancy works because the configuration appears correct. Establish a baseline, introduce one planned failure at a time, measure the impact and restore the original state before testing the next scenario.
Record interface state, port-channel membership, vPC health, BGP sessions, installed routes, traffic counters and application reachability.
In an approved test window or lab, disable one selected link or simulate a planned adjacency failure. Avoid changing multiple components at once.
Track packet loss, interruption duration, latency, route changes, member utilization and any application errors.
Confirm that the failed component recovers, the intended paths return, and traffic distribution returns to an acceptable state.
6. Failure testing checklist
- Confirm the intended redundant links and alternate routes are present before testing.
- Verify that the failure is detected by the expected mechanism.
- Measure both control-plane changes and actual data-plane traffic.
- Check that remaining links do not become overloaded.
- Confirm gateway and endpoint reachability throughout recovery.
- Review logs and counters for errors that remain after the failure is cleared.
- Document the measured recovery time and compare it with the service requirement.
Key takeaway: Resilience must be demonstrated, not assumed. A robust design combines independent failure domains, valid alternate forwarding paths, predictable convergence and repeatable tests that measure the impact on real traffic.
Interactive learning · Module 05
Failure Investigation and Convergence
Redundant paths help a network survive failures, but they do not guarantee zero packet loss. Explore common failures, distinguish physical and routing symptoms, and follow a structured investigation before deciding whether service has recovered.
Member-link failure
One physical member of a port-channel fails. If other members remain operational, the port-channel may continue forwarding, but capacity falls and some flows may be disrupted during detection and remapping.
First investigation: Check physical interface state, port-channel membership, errors and traffic counters.
Investigation checklist
Work through these checks for the selected failure. Tick them off as you investigate.
Example verification commands
show interface status show port-channel summary show vpc show interface counters errors show ip bgp summary show ip route
These are illustrative commands; syntax and output vary by platform and operating-system version.
Knowledge check
After a redundant path fails, what is the strongest evidence that service has recovered?
Failure behavior depends on the platform, topology, configuration, failure-detection timers and routing design. This teaching widget illustrates investigation priorities; it does not predict exact convergence time or guarantee a particular forwarding outcome.
Section 06 · Architecture Decisions
vPC vs Layer 3 Multipathing and BGP ECMP
Both Virtual PortChannels and routed multipathing can improve network resilience, but they achieve it in different ways. vPC allows a connected device to use a logical Layer 2 port-channel across two cooperating switches. Layer 3 multipathing uses multiple eligible routed next hops to forward traffic through the network.
The design decision is not simply which technology is better. It is where the network needs Layer 2 adjacency, where routing boundaries should exist, and how the design should respond when links, switches or routing adjacencies fail.
1. Two different approaches to redundancy
Approach A — vPC
A connected device forms one logical port-channel using physical links to two cooperating switches. The vPC peers coordinate relevant forwarding and consistency behavior.
Approach B — Layer 3 ECMP
A routed device has multiple eligible next hops for a destination. The forwarding system distributes flows across installed paths according to its multipath and hashing implementation.
Conceptual comparison, not a single combined topology. Actual connectivity and routing behavior depend on the chosen design.
2. Compare the forwarding models
| Design dimension | vPC / Layer 2 aggregation | Layer 3 multipathing / ECMP |
|---|---|---|
| Forwarding model | Aggregated Layer 2 connectivity across two cooperating switches. | Routing and forwarding across multiple eligible next hops. |
| Primary dependency | vPC peer coordination, peer-link health, consistency and member state. | Route availability, next-hop reachability and multipath eligibility. |
| Failure handling | Depends on port-channel membership and vPC-specific protection behavior. | Depends on failure detection, route changes and the availability of alternate installed paths. |
| Traffic distribution | Hashing selects a physical port-channel member. | Hashing selects among eligible routed next hops. |
| Layer 2 scope | Can support a Layer 2 segment spanning the vPC-connected infrastructure. | Routed boundaries limit the need to extend a Layer 2 segment across the fabric. |
| Operational focus | Peer state, port-channel health and Layer 2 forwarding consistency. | BGP state, route policy, next-hop resolution and forwarding-table programming. |
3. Where BGP ECMP fits in a leaf-spine fabric
In a routed leaf-spine architecture, each leaf commonly has routed connections to multiple spine switches. BGP can exchange reachability information over those connections. Where multiple paths are eligible and ECMP is supported and configured, the forwarding system can use multiple next hops to reach a destination.
Path diversity
Multiple routed paths can provide alternatives when a link or spine becomes unavailable, provided the remaining topology offers valid reachability.
Routing boundaries
Routed links can reduce dependence on extending the same Layer 2 segment throughout the fabric.
Independent verification
Engineers can inspect BGP routes, installed next hops and interface counters to identify where forwarding behavior differs from the intended design.
Remember: BGP is the routing protocol; ECMP is a forwarding capability that may use multiple eligible equal-cost paths. Running BGP alone does not guarantee that multiple paths will be installed or used.
4. When might each approach be appropriate?
vPC may be appropriate when a server, appliance or downstream switch needs redundant Layer 2 connectivity to a pair of switches. Routed multipathing may be attractive when the network can use Layer 3 links and routing to provide path diversity across a fabric.
| Requirement | Design direction to evaluate | Key question |
|---|---|---|
| Endpoint requires Layer 2 adjacency across two switches | Evaluate vPC or another supported multi-chassis aggregation design. | Can the peer relationship and failure behavior meet the requirement? |
| Fabric can use routed point-to-point links | Evaluate a routed leaf-spine architecture with BGP and ECMP. | Are alternate paths available, eligible and correctly installed? |
| Application requires high availability | Evaluate the complete path, including server NICs, switches, routing and upstream services. | Does the design survive the relevant failure without unacceptable impact? |
| Uneven traffic utilization | Inspect port-channel hashing and routed ECMP distribution separately. | At which forwarding stage is traffic concentrating? |
5. Avoid treating vPC and ECMP as interchangeable
A vPC port-channel and routed ECMP may both distribute traffic, but they operate at different forwarding layers. They also have different state, failure mechanisms and troubleshooting commands. A design can use both technologies in different parts of the same network, provided their interaction is understood and supported by the platforms.
- Document the Layer 2 and Layer 3 boundaries.
- Identify which component detects each expected failure.
- Verify the alternate forwarding path exists before testing a failure.
- Measure convergence, packet loss and remaining link capacity.
- Check port-channel members and ECMP next hops independently.
- Validate the design against the vendor's platform and software documentation.
Key takeaway: Choose vPC when multi-chassis Layer 2 aggregation is required; evaluate routed BGP ECMP when the design calls for multiple Layer 3 paths. The strongest architecture is the one whose failure domains, forwarding behavior and operational requirements have been deliberately designed and tested.
Interactive learning · Module 06
Choose the Right Multipathing Model
vPC and BGP ECMP solve different networking problems. Compare a Layer 2 multi-switch attachment with a Layer 3 routed fabric, then simulate a path failure to see which part of the design must be investigated.
Layer 2 attachment with vPC
A supported downstream device can form a port-channel across two vPC peers, presenting a logical attachment while using links to both switches. Peer-link health, consistency and platform-specific failure behavior remain important.
Engineering check: Verify vPC state, peer-link, member interfaces, consistency and traffic counters.
vPC · Layer 2 attachment
- Provides a multi-switch port-channel attachment for supported devices.
- Depends on peer coordination and platform-specific vPC behavior.
- Does not itself provide BGP route selection or Layer 3 ECMP.
BGP ECMP · Layer 3 paths
- Uses multiple eligible equal-cost next hops when supported and configured.
- Depends on routing policy, route selection, next-hop reachability and forwarding installation.
- Does not require a vPC peer-link between the routed next hops.
Verification command examples
show vpc show port-channel summary show ip bgp summary show ip route show interface counters errors
These are illustrative Cisco-style commands. Availability and syntax vary by platform and software version.
Knowledge check
Which statement best describes the relationship between vPC and BGP ECMP?
These designs can coexist in a larger network, but one does not automatically configure or guarantee the other. The drawings are conceptual and do not represent a vendor-specific production topology.
Section 07 · Operational Troubleshooting
vPC Troubleshooting and Verification
A resilient design needs more than redundant links and correctly configured switches. Engineers must be able to verify the operational state of the vPC domain, identify inconsistencies, isolate failures and confirm that traffic is using the expected forwarding paths.
The most effective troubleshooting approach is to work from the underlying physical interfaces towards the logical port-channel, vPC peer relationship and end-to-end forwarding path. This helps distinguish a physical connectivity problem from a peer coordination, Layer 2 or routing issue.
1. Begin with the health of the vPC domain
Start by confirming that both peer switches are available and that their peer relationship is functioning as expected. Check the peer-link, peer-keepalive mechanism, vPC member interfaces and consistency status. A single healthy interface does not prove that the complete vPC arrangement is healthy.
Peer relationship
Verify that the peers recognize each other and that the expected vPC domain state is present.
Peer-link
Check physical members, logical interface state, errors and capacity. Confirm that the peer-link is operating as designed.
Keepalive
Verify the configured keepalive path and inspect logs for reachability changes or missed messages.
Consistency
Review vPC consistency results and any reported type-1 or type-2 configuration mismatches.
Safety first: Peer-link and keepalive failures can affect vPC forwarding and peer protection behavior. Do not troubleshoot by blindly shutting interfaces or changing peer roles. Understand the current state and follow the platform's documented recovery procedure.
2. Verify the physical members and logical port-channel
A port-channel can have a logical interface that appears configured while one or more expected physical members are down, suspended or not bundled correctly. Compare the configured membership with the operational state.
On Cisco Nexus platforms, the following NX-OS commands are common starting points. Exact command availability and output vary by platform and software release.
show port-channel summary show interface port-channel 10 show interface ethernet 1/1 show vpcReplace the example interface and port-channel identifiers with those used in your environment. Check whether the physical members are bundled, whether the logical interface is up, and whether counters indicate errors or drops.
3. Troubleshooting symptoms and likely causes
| Observed symptom | Possible causes | Next checks |
|---|---|---|
| Port-channel is down | Member links are down, members are not bundled, or configuration does not match. | Physical state, member configuration, LACP state and port-channel summary. |
| Member is suspended | Negotiation, compatibility or configuration problems. | LACP details, interface logs, speed, duplex and channel-group settings. |
| vPC consistency warning | Relevant configuration differs between peers. | Consistency output, VLAN membership, port-channel settings and platform guidance. |
| Peer-link is down | Physical failure, member bundle failure or a configuration issue. | Peer-link members, interface errors, cabling and peer state. |
| Traffic is unevenly distributed | Hash distribution, flow size differences or reduced available capacity. | Per-member traffic counters, hashing policy and workload patterns. |
| Connectivity fails after a routing change | Route withdrawal, missing next hop, policy issue or unavailable alternate path. | BGP session state, route table, next-hop reachability and forwarding entries. |
4. Investigate vPC consistency issues
Consistency checks help identify configuration differences between peers that could affect forwarding. The exact checks and the impact of a mismatch depend on the platform, interface type and software version.
Do not assume that every warning has the same severity. Some inconsistencies can suspend a vPC or prevent it from operating as intended; others may generate warnings without immediately disabling forwarding.
- Compare VLAN membership and allowed VLAN lists.
- Check port-channel and LACP configuration on both peers.
- Verify relevant trunk, access and spanning-tree settings.
- Review vPC-specific consistency output and system logs.
- Make corrective changes using the platform's documented procedure, then verify the resulting operational state.
5. Use counters to investigate traffic problems
When connectivity works but performance is poor, examine counters rather than relying only on link status. A physical link can remain up while experiencing errors, congestion or uneven utilization.
show interface counters errors show interface ethernet 1/1 show port-channel summaryThese are illustrative NX-OS commands. Verify the supported syntax for your specific device and software release.
- Compare traffic rates across all members of the same port-channel.
- Look for CRC errors, input errors, output drops and unexpected counter changes.
- Check whether one large flow is dominating a member link.
- Confirm that the remaining members have enough capacity during a failure.
- Compare the observed behavior with the configured hash policy.
6. Follow the traffic across the Layer 3 boundary
If the vPC domain appears healthy but remote destinations are unreachable, continue the investigation into the routed network. A working port-channel does not guarantee that a route exists or that the destination is reachable through the expected next hop.
show ip route show ip bgp summaryThese examples are intended for supported Cisco routing contexts; syntax and command availability vary. Check the routing table, BGP adjacency state and next-hop reachability before concluding that vPC is responsible for the outage.
7. A repeatable troubleshooting workflow
Determine which endpoints, VLANs, links or destinations are affected and whether the issue is total loss, intermittent connectivity or reduced performance.
Verify interface status, port-channel membership, LACP state and relevant error counters.
Inspect peer status, peer-link, keepalive and consistency results. Review logs for state transitions around the time of the problem.
Confirm VLAN and Layer 2 reachability where applicable, then inspect routing, next hops and forwarding entries for remote destinations.
Confirm that the original symptom has cleared, counters are stable and the network has returned to its intended state. Document the root cause and evidence.
Key takeaway: Troubleshoot from the physical interface through the port-channel and vPC domain, then follow the traffic into the routed network where necessary. Use operational state, consistency results, counters and routing evidence together rather than relying on a single command or status indicator.
Interactive troubleshooting · Module 07
Diagnose the Fault Before Changing the Network
Choose a symptom to build a focused investigation plan. Work from the observed fault toward the relevant control-plane and data-plane evidence. The checks below are a learning aid, not live device output.
1. Select the symptom
Port-channel member investigation
Determine whether the port-channel is operational, which members are active, and whether traffic loss is linked to a physical interface, a flow-hashing pattern or a downstream fault.
2. Work through the checks
0 of 4 checks completed
3. Example verification commands
Use commands appropriate to the device and software release. Compare outputs before and after a controlled test.
show port-channel summary show interface port-channel 10 show interface counters errors
Knowledge check
You observe that a BGP session is established, but the expected prefix is missing from the routing table. What is the best next step?
Command syntax and available fields vary by platform and software version. Follow the vendor's documentation and your operational procedures. Record the baseline, make one controlled change at a time, and verify both recovery and end-to-end traffic.
Section 08 · Design and Deployment
vPC Design Best Practices and Deployment Checklist
A successful vPC deployment starts with a clear redundancy objective, a supported topology and a documented failure model. Link aggregation alone does not guarantee end-to-end resilience. The peer relationship, physical infrastructure, Layer 2 configuration, routing design and operational procedures must work together.
Use the following principles to review a proposed design or assess an existing data-centre deployment. Platform-specific implementation details must always be checked against the relevant vendor documentation.
1. Build redundancy into the physical design
Separate failure domains
Where practical, diversify cable paths, line cards, power sources and upstream connectivity so that one local incident does not remove every redundant path.
Plan peer-link capacity
Size the peer-link for its expected control and data traffic, including relevant failure scenarios. Do not assume normal utilization represents the worst-case requirement.
Protect the keepalive path
Design peer-keepalive connectivity according to platform guidance. It should provide useful peer-liveness information independently of the peer-link failure it helps diagnose.
Preserve usable capacity
Confirm that the surviving links and switches can carry the expected load after a failure, rather than merely keeping endpoints connected.
2. Keep the configuration consistent
vPC peers need compatible configurations for the features and interfaces involved in the design. Consistency checking can detect important differences, but it should not replace a deliberate configuration review.
- Verify the vPC domain, peer-link and peer-keepalive configuration.
- Review VLAN membership, trunk settings and spanning-tree requirements.
- Confirm port-channel and LACP settings match the intended design.
- Check relevant MTU, speed, interface and feature compatibility.
- Document the treatment of orphan ports and single-homed devices.
- Review supported software versions, known limitations and upgrade procedures.
Design principle: Do not copy a configuration between peers without validating which settings must match and which settings are intentionally different. Follow the consistency requirements for the specific platform and release.
3. Decide where Layer 2 should end
One of the most important architectural decisions is determining which networks need Layer 2 adjacency and where Layer 3 routing should begin. Extending Layer 2 can be necessary for some workloads, but it also increases the scope of the Layer 2 failure domain.
| Design question | What to evaluate |
|---|---|
| Does the endpoint need Layer 2 adjacency? | Determine whether vPC-based connectivity is required or whether a routed attachment is suitable. |
| Can the fabric use routed links? | Evaluate a Layer 3 leaf-spine design with routing and ECMP where multiple paths are appropriate. |
| Where is the gateway located? | Document gateway redundancy, first-hop behavior and the forwarding path during peer or link failure. |
| How are failures contained? | Identify which devices, VLANs, routes and applications are affected by each failure domain. |
4. Integrate BGP and ECMP deliberately
Where the data centre uses BGP, review the routing architecture independently from the vPC configuration. BGP route exchange, best-path selection and ECMP forwarding have their own operational requirements. A healthy vPC domain does not prove that the routed network has a valid path to every destination.
- Document BGP peers, address families, route policies and intended advertisements.
- Verify next-hop reachability and which routes are installed for forwarding.
- Confirm that multiple paths are eligible for ECMP where multipath forwarding is expected.
- Inspect both ECMP next-hop distribution and port-channel member utilization.
- Test how link, peer and BGP-session failures affect traffic and convergence.
Remember: vPC hashing chooses a physical member of a logical link. ECMP chooses among eligible routed next hops. If traffic is unevenly distributed, investigate each forwarding stage rather than assuming one mechanism explains the entire path.
5. Interactive deployment readiness checklist
Use this checklist during a design review or before a controlled deployment. Tick each item only after the relevant configuration, documentation or test evidence has been verified.
0 of 10 checks completed
Checklist progress is held in this page view only. It is not saved to an account or network device. Completing the checklist is not a substitute for platform-specific validation or an approved change plan.
6. Final design review
Before declaring the design ready, confirm that the team can explain how traffic flows during normal operation, which links carry it, what happens when each important component fails, and how the resulting behavior will be verified. A design that can be tested, monitored and recovered consistently is easier to operate than one that relies on assumptions about redundancy.
Key takeaway: Build vPC resilience around supported peer behavior, consistent configuration, adequate capacity and independent failure domains. Where BGP and ECMP are also used, validate the routing and forwarding layers separately, then test the complete end-to-end design.
Design readiness · Module 08
vPC Design Readiness Assessment
Use this interactive checklist to review redundancy, peer coordination, traffic behavior, routing boundaries and operational recovery before deploying or changing a vPC-based design. Tick a control only when it has been verified for your actual platform and topology.
1. Physical and logical redundancy
Check that the design can tolerate the failures it is intended to survive.
2. Configuration and Layer 2 behavior
Confirm that the peers and attached devices behave as designed.
3. Routing and traffic distribution
Separate Layer 2 attachment resilience from Layer 3 routing decisions.
4. Operations and recovery
A resilient design also needs a tested operational process.
Assessment not complete
Review each control and mark it only when verified. Completion measures checklist progress, not an automatic guarantee that the network is production-ready.
Design review summary
This checklist is an educational planning aid. A checked box records your review; it does not validate device configuration or prove resilience. Apply the requirements and failure behavior documented for your specific platform, release and design.
Section 09 · Practical Learning
Interactive vPC and BGP Labs
Understanding redundancy means more than knowing which links exist. You need to predict what happens when a component fails, identify which control-plane or forwarding state changes, and verify whether traffic can continue to reach its destination.
Explore these five guided scenarios. Each lab gives you a problem, an investigation sequence, verification commands and the observations you should look for. Select a lab to switch between scenarios.
Lab safety: These are illustrative exercises, not live-device simulations. Commands and outputs depend on platform and software version. Perform failure tests only in an authorized lab or maintenance window with a documented recovery plan.
Lab 1: Member-link failure
Scenario
A server connects through a port-channel whose physical members provide redundant bandwidth. One member link fails, but the logical port-channel may remain operational.
Your investigation
- Record the original port-channel state and member status.
- Identify which physical interface has failed.
- Check whether the port-channel remains up and has fewer active members.
- Inspect interface errors, utilization and application connectivity.
show port-channel summary show interface port-channel 10 show interface ethernet 1/1 show interface counters errors
Engineering question: Does the remaining capacity support the expected load?
Test your reasoning
Reveal each answer after considering the scenario. These questions check the design principles behind the labs.
1. A single port-channel member fails, but the logical port-channel remains up. Must BGP necessarily reset?
2. Does a healthy vPC peer relationship prove that every BGP route is valid?
3. If a route has two eligible ECMP next hops, will every flow use both paths equally?
How to use these labs in a real environment
- Write down the expected behavior before making a change.
- Capture baseline interface, vPC, routing and traffic information.
- Change only one controlled failure condition at a time.
- Compare the observed result with the prediction and record any packet loss or convergence delay.
- Restore the original state and verify end-to-end connectivity before proceeding.
Key takeaway: Troubleshoot from the physical interface through the port-channel and vPC state, then inspect Layer 3 routing and end-to-end forwarding. This layered method helps isolate whether a fault affects link redundancy, peer behavior, route selection or the final traffic path.
Commands shown are examples for Cisco NX-OS-style environments. Syntax and availability vary by platform and release. The interactive panels illustrate learning scenarios and do not execute commands, change device state or persist lab progress.
Hands-on learning · Module 09
vPC and BGP Troubleshooting Lab Console
Choose a scenario, examine the conceptual topology and work through a guided investigation. Each lab focuses on evidence gathering, interpreting network state and identifying what to verify next. These are simulated learning exercises, not live device sessions.
Lab 01 — Member-link failure
A port-channel has lost one physical member. Determine whether the bundle remains operational and whether the remaining capacity is sufficient for the observed traffic.
Investigation progress
Begin by confirming the operational state of the port-channel and its physical members.
Lab objective
Confirm member state, assess remaining capacity and verify end-to-end forwarding.
Commands to investigate
Use these examples as a starting point and adapt them to the platform.
show port-channel summary show interface port-channel 10 show interface counters errors
What good evidence looks like
Confirm that the port-channel is up, identify active members and compare traffic and error counters before and after the event.
Lab knowledge check
One physical port-channel member fails. What should you verify first?
All topology changes and observations in this widget are illustrative. They do not emulate a particular switch operating system, predict exact convergence time or guarantee lossless recovery. Follow platform documentation and approved change procedures when working on real devices.