BGP Multipath

BGP Multipath

NETWORK INSIGHT · BGP ENGINEERING GUIDE
BGP Multipath: From Route Selection to Forwarding

A BGP router can learn several routes to the same destination, select a preferred path, and still install multiple next hops for forwarding. Understanding how these decisions interact is essential when designing resilient, load-sharing networks and troubleshooting unexpected traffic behaviour.

BGP Multipath allows a router to install multiple eligible paths for a prefix, subject to its configuration, path-selection rules and platform capabilities. Instead of relying on a single forwarding next hop, the router can use a set of next hops to distribute traffic and maintain forwarding options when a path becomes unavailable.

The important distinction is that routes learned by BGP are not automatically routes installed in the forwarding table. A route may be present in the BGP table yet fail a multipath eligibility check. Even when several paths are installed, the actual traffic distribution depends on the forwarding implementation and its hashing behaviour.

The engineering question

If a router learns two or more paths to the same prefix, what determines which next hops reach the FIB, how is traffic distributed across them, and what happens when one path fails?

01 · SEE

Understand the path set

Follow candidate routes from BGP selection through multipath eligibility to the next hops installed in the forwarding table.

02 · OBSERVE

Verify traffic distribution

Compare installed paths with forwarding state, interface counters, traffic demand and path characteristics.

03 · INVESTIGATE

Find the missing path

Use routing evidence to identify why a candidate path is excluded and validate the result after a change or failure.

This guide combines routing concepts with interactive engineering labs. You will examine candidate paths, change the permitted multipath count, investigate a missing forwarding next hop, and explore how Route Reflectors and BGP Add-Path affect the visibility and advertisement of alternative routes.

Engineering guide · Navigation

4 investigation stages

BGP Multipath: Engineering Navigation

Follow the route from BGP selection to real forwarding behaviour. Each stage builds on the previous one, moving from architecture and visibility to fault isolation and verified recovery.

SECTION 01 · SEE THE FORWARDING PATH

When Does a BGP Route Become a Forwarding Next Hop?

Learning multiple routes does not mean installing every route. A router evaluates candidate paths, applies its BGP decision process and checks which alternatives qualify for multipath forwarding. Follow the decision chain below.

STAGE 01 Learn routes Receive candidate paths from BGP peers.
STAGE 02 Evaluate paths Compare policy attributes and route eligibility.
STAGE 03 Check multipath Apply maximum-paths and implementation-specific rules.
STAGE 04 Install next hops Program eligible paths into forwarding, if installation succeeds.
Engineering question: Which candidate paths qualify for the multipath set, and how can you prove that their next hops were actually installed in the FIB?

Section 1 — From BGP Paths to Forwarding

BGP multipath is the ability to install multiple eligible routes to the same destination into a router's forwarding state, rather than relying on a single next hop. It can improve link utilisation and provide additional forwarding options, but only when the routes satisfy the platform's multipath rules and the forwarding hardware can use them.

The important engineering distinction is between routes BGP knows about and next hops the router actually uses to forward packets. A router may learn several paths for a prefix yet select only one for installation. Seeing multiple paths in a BGP table is therefore not proof that multipath forwarding is active.

Engineering question: For a destination prefix, which candidate paths qualify for multipath, which next hops are installed in the forwarding information base (FIB), and what evidence proves packets can use those next hops?

1.1 The path from route learning to packet forwarding

BGP receives route advertisements from peers and evaluates the attributes and policies associated with each path. The routing process then determines the eligible route or routes. If multipath is configured and the relevant eligibility conditions are met, the router may install multiple next hops for the same prefix. The data plane uses the installed forwarding state to direct packets.

Stage 01 Learn routes

Receive BGP UPDATEs and maintain paths learned from peers.

Stage 02 Evaluate paths

Apply policy, compare attributes and assess route eligibility.

Stage 03 Check multipath

Determine whether additional paths satisfy configured rules.

Stage 04 Install next hops

Program usable forwarding entries and verify the resulting FIB.

These stages describe the conceptual workflow; exact implementation details differ between vendors and operating systems. Some platforms expose a separate multipath set, while others present the selected route and additional forwarding next hops through different commands.

1.2 BGP RIB versus FIB: two different views

The routing information base (RIB) represents route information maintained by the control plane. The forwarding information base (FIB) contains the entries used by the forwarding plane. The two are related, but they answer different operational questions.

Evidence source What it tells you What it does not prove alone
BGP route table Which BGP paths are known, their attributes and their selection status. That every visible path is installed for forwarding.
Routing table / RIB Which route or routes the routing process has selected for the destination. That all intended next hops are programmed and usable in hardware.
Forwarding table / FIB Which next-hop entries the forwarding plane uses for matching packets. That traffic is distributed evenly or that every path is healthy under load.
Interface and traffic counters Whether interfaces carry traffic and how utilisation changes. That BGP multipath is the only cause of the observed traffic pattern.

1.3 What makes a path eligible for multipath?

Enabling a maximum-paths setting does not mean that any route to the same prefix can be installed alongside the current best path. The router must still apply its route-selection process and its implementation-specific multipath criteria. Those criteria can include path attributes, peer type, next-hop reachability, routing policy and other platform-specific constraints.

Best-path selection and multipath eligibility are related, but not identical

BGP first evaluates routes according to its decision process. Multipath then permits additional paths that satisfy the implementation's requirements to be installed alongside the selected path. Depending on the platform and configuration, some attributes that matter to best-path selection may need to match or meet specific conditions for multipath. Do not assume identical requirements across vendors.

Path attributes

Compare attributes such as Local Preference, AS_PATH, origin and MED where relevant to the platform's multipath rules.

Next-hop reachability

Verify that each candidate next hop resolves through the routing table and has a usable forwarding path.

Configuration and limits

Check the applicable maximum-paths command, address family, peer type, policy and platform limits.

1.4 Configuring multipath: verify the scope, not just the command

Many BGP implementations provide a command such as maximum-paths to control how many eligible paths can be installed. The precise command, supported path types and default behaviour depend on the vendor, software release and address family. eBGP and iBGP multipath may use separate configuration.

Treat the following as an illustrative configuration pattern, not a universal command sequence:

Illustrative BGP configuration router bgp 65001 address-family ipv4 unicast maximum-paths 2

Before applying configuration, confirm the syntax and placement in your platform's documentation. After making a change, verify the operational result in both the routing table and the FIB. A configured limit of two paths is a ceiling, not a guarantee that two paths will qualify.

1.5 Worked example: three candidate paths, two installed next hops

Consider router R1 learning prefix 203.0.113.0/24 through three candidate paths. Assume Paths A and B meet the configured multipath criteria, while Path C does not. In this example, the router installs two next hops for the destination.

Candidate Control-plane observation Forwarding result
Path A Selected best path and eligible for multipath. Installed
Path B Meets the configured additional-path eligibility rules. Installed
Path C Visible as a candidate but fails a required eligibility condition in this scenario. Not installed

The router can now forward matching traffic using the installed next-hop set. How traffic is divided depends on the forwarding platform, hashing algorithm, flow characteristics and configuration. Two installed next hops do not guarantee a 50/50 traffic split, and packet ordering or per-flow behaviour depends on the implementation.

1.6 How to prove multipath is operational

Validate the state at three levels. First, inspect BGP to confirm the candidate routes and their attributes. Second, inspect the routing and forwarding tables to confirm that the expected next hops are installed. Third, observe interface counters or traffic telemetry under an appropriate test load to see whether traffic is actually using the available paths.

Vendor-neutral verification checklist 1. Identify the destination prefix. 2. Inspect all relevant BGP paths and attributes. 3. Check the selected route and multipath eligibility. 4. Inspect the installed FIB next hops. 5. Compare interface counters before and during traffic. 6. Repeat after a controlled path failure and recovery.

Command names differ across platforms, so use the equivalent BGP route, routing-table, forwarding-table and interface-counter commands for your environment. Establish a baseline before testing, and avoid injecting a failure into production without an approved maintenance or test procedure.

Section 1 · Engineering takeaway

Multipath is proven by more than seeing two BGP routes. Establish which paths are eligible, verify the installed next-hop set in the FIB, and then measure forwarding behaviour. This separates control-plane visibility from actual packet forwarding.

BGP Routing Lab

How Does BGP Multipath Turn Several Routes into One Forwarding Decision?

BGP normally identifies a single best path. With multipath enabled, multiple paths that satisfy the implementation's multipath eligibility rules can be installed and used for forwarding. Explore the difference between candidate paths and installed next hops.

Candidate paths for 203.0.113.0/24
Path A · PE1
Installed
LocalPref 200
AS Path 65010
MED 50
Next-hop 10.0.1.1
Path B · PE2
Installed
LocalPref 200
AS Path 65010
MED 50
Next-hop 10.0.2.1
Path C · PE3
Not eligible
LocalPref 200
AS Path 65020 65010
MED 50
Next-hop 10.0.3.1
BGP RIB candidate routes
→
Multipath Check eligibility rules
→
FIB multiple next hops
10.0.1.1 · PE1
10.0.2.1 · PE2
Forwarding decision
Destination 203.0.113.0/24
Best policy value LocalPref 200
Eligible paths 2
Forwarding model ECMP / Multipath
Engineering insight: Paths A and B have matching policy and path characteristics for this simplified example, so both can be installed. Path C has a longer AS path and is outside the multipath set.
BGP UPDATE received for 203.0.113.0/24
→ compare candidate attributes
→ Path A eligible for multipath
→ Path B eligible for multipath
→ Path C excluded: path attributes differ
FIB programmed with 2 next hops
SECTION 02 · OBSERVE THE EVIDENCE

Are Multiple Paths Carrying Traffic as Intended?

A multipath configuration is not proof of successful forwarding. Engineers need to compare routing state with the forwarding table and real traffic measurements. Each evidence source answers a different operational question.

EVIDENCE 01
BGP and RIB state

Confirm which candidate routes are learned, which paths qualify for multipath, and whether the expected number of next hops is selected for installation.

EVIDENCE 02
FIB and interface counters

Verify that multiple next hops are programmed and inspect interface packet and byte counters to see whether traffic traverses the expected links.

EVIDENCE 03
Traffic and utilisation

Compare per-link utilisation and traffic over time. Uneven distribution is not automatically a fault: flow hashing, traffic patterns and link capacity all matter.

BGP route table FIB next hops Interface counters Per-link utilisation
Engineering question: Do the installed next hops match the expected forwarding design, and does measured traffic provide evidence that those paths are being used?

Section 2 — Verify Traffic Distribution

Installing multiple BGP next hops creates the possibility of multipath forwarding; it does not prove that traffic is being distributed as expected. Engineers must correlate the control-plane view with forwarding entries, interface counters and measured traffic to establish what the network is actually doing.

A routing table can show two installed next hops while one physical link carries substantially more traffic than another. That observation is not automatically a fault. Traffic distribution depends on the forwarding implementation, hash inputs, flow sizes, path capacity, traffic direction and other network conditions.

Engineering question: Are all intended next hops installed and healthy, and does observed traffic behaviour match the forwarding design and expected workload?

2.1 Three evidence sources, one operational picture

Reliable validation combines evidence from different layers. No single command or counter provides the complete picture.

Evidence 01 BGP and routing state

Confirm the destination prefix, candidate paths, selected route, path attributes and the router's multipath status.

Evidence 02 FIB and next hops

Verify the actual installed forwarding entries, resolved next hops and outgoing interfaces for the prefix.

Evidence 03 Traffic and utilisation

Compare interface counters, flow telemetry, drops, errors and utilisation over a consistent measurement interval.

2.2 Why two paths rarely mean an exact 50/50 split

Many routers use a hash to map flows or packets to next hops. Depending on the platform and configuration, hash inputs may include source and destination addresses, transport ports, protocol identifiers or other fields. This helps preserve packet ordering for a flow, but it does not guarantee equal byte counts across links.

Imagine two installed next hops. If a few high-volume flows hash to the first path while many small flows hash to the second, their utilisation can differ considerably. The paths may both be functioning correctly.

Observation Possible explanation Next check
Both next hops are installed, but one link is busier. Uneven flow sizes or hash distribution. Compare flow-level telemetry, byte counters and the hashing policy.
One expected next hop is absent from the FIB. Multipath eligibility, route resolution, policy or programming issue. Recheck the BGP paths, routing table and forwarding entries.
A link's utilisation rises while drops also increase. Congestion, insufficient capacity, errors or a downstream bottleneck. Inspect queue drops, interface errors and downstream links.
Traffic changes after a path failure. Traffic has moved to surviving paths, possibly with a temporary convergence effect. Check route state, FIB changes, loss, latency and remaining capacity.

2.3 Measure counters correctly

Interface counters are cumulative, so compare changes over a defined interval rather than relying only on their absolute values. Record the starting and ending byte or packet counts, the elapsed time, and whether the interface counters reset during the test.

Measurement method 1. Record interface counters at time T1. 2. Generate or observe representative traffic. 3. Record counters again at time T2. 4. Calculate counter deltas over T2 − T1. 5. Compare utilisation, drops and errors across paths. 6. Repeat under comparable traffic conditions.

For byte counters, the approximate average bit rate over the interval is:

Average bit rate (bits/s)
\((\text{byte delta} \times 8) \div \text{elapsed seconds}\)

Compare this value with the interface's usable capacity. Account for units, counter width, measurement interval and traffic direction when interpreting the result.

2.4 Confirm the measurement point

A router's egress interface counters show traffic transmitted through that interface; they do not necessarily describe what the remote endpoint received. Conversely, ingress counters measure traffic arriving at that interface. Drops, encapsulation overhead, congestion and measurement placement can make these values differ.

When investigating an imbalance, identify where each measurement was taken and which traffic direction it represents. If possible, correlate interface counters with flow records, queue statistics and application-level observations. This helps distinguish a routing issue from a congestion, capacity or measurement issue.

2.5 Validate behaviour during a controlled failure

Multipath can provide alternate forwarding options, but it does not guarantee uninterrupted service. A failed link may trigger interface-state changes, route withdrawal, next-hop resolution changes and FIB reprogramming. The observed impact depends on the topology, protocol convergence, forwarding hardware and available capacity on the surviving paths.

In a controlled test, establish a baseline first. Then remove or fail one path using an approved method, observe the routing and forwarding changes, and measure traffic on the surviving path. Restore the path and verify that the expected operational state returns.

Before Capture baseline

Record installed next hops, link utilisation, packet loss, latency and interface health.

During Observe the transition

Correlate the failed path with route changes, FIB updates, traffic movement and any loss.

After Verify recovery

Confirm the restored path is eligible, installed as expected and carrying traffic when appropriate.

2.6 Operational verification checklist

Before declaring multipath healthy, verify each of the following against the intended design:

Control plane: The required paths are learned and the expected route state is present.

Forwarding plane: The intended next hops appear in the FIB and resolve to usable interfaces.

Traffic plane: Counters or flow telemetry demonstrate actual traffic movement, with no unexplained drops or errors.

Resilience: A controlled path failure produces the expected transition, and recovery restores the intended state.
Section 2 · Engineering takeaway

Multiple FIB next hops prove that forwarding options are installed; they do not prove equal load sharing, sufficient capacity or zero packet loss. Correlate route state, forwarding entries and time-based traffic measurements to determine whether multipath is operating as designed.

BGP Multipath Experiment

How Many Paths Actually Reach the FIB?

Multipath is not simply “use every route.” Change the number of permitted paths, traffic demand and path similarity to see how the forwarding set changes. The model below illustrates the engineering decision rather than a vendor-specific command syntax.

Network controls
Maximum installed paths 2
Traffic demand 60%
Path similarity High
Forwarding policy
Candidate paths
Path A · 10.0.1.1
Excluded
Direct path · 20 ms · 100 Gbps
Path B · 10.0.2.1
Excluded
Alternate path · 28 ms · 140 Gbps
Path C · 10.0.3.1
Excluded
Long-haul path · 41 ms · 200 Gbps
Path D · 10.0.4.1
Excluded
Diverse path · 47 ms · 240 Gbps
Installed
0 / 4
Aggregate capacity
0 Gbps
Average latency
—
FIB state
Single path
Engineering insight: Increase the maximum path count to allow more eligible paths into the forwarding set.
SECTION 03 · INVESTIGATE THE FAILURE

Two Routes Are Visible. Why Is Only One Installed?

When a second path is missing from the FIB, do not assume that the BGP neighbour is down or that multipath is broken. Start with the evidence and work through the routing and forwarding decision chain until the exclusion point is identified.

01
Confirm route availability

Check the peer session, received prefix, route validity and next-hop reachability. Establish whether the candidate path is actually usable.

02
Compare path attributes

Examine Local Preference, AS_PATH, origin, MED where applicable, and other attributes required by the implementation's multipath rules.

03
Verify multipath configuration

Check the relevant maximum-paths settings, peer-type requirements, address family and platform-specific eligibility conditions.

04
Inspect forwarding installation

Compare the BGP path set with the installed FIB next hops. If a path qualifies but is absent from forwarding, investigate programming errors and resource constraints.

Investigation principle: Identify the first point where observed state differs from expected state. A route missing from the FIB can have several causes, so validate each layer before changing configuration.

Section 3 — Investigate: Why Is the Second Path Missing?

When a router learns two BGP paths but installs only one forwarding next hop, the correct response is not to increase the maximum-paths value blindly. The task is to discover where the expected state diverges from the actual state, using evidence from route learning, path selection, multipath eligibility and forwarding installation.

A missing path can result from route policy, a difference in path attributes, an unresolved next hop, an incompatible peer type, an address-family configuration issue, a platform limitation or a forwarding-programming problem. These causes occur at different stages and require different remedies.

Engineering question: At which stage does the second path stop qualifying, and what observable evidence identifies the cause?

3.1 Troubleshoot in dependency order

Follow the route through the system in sequence. This prevents an engineer from changing a downstream setting before confirming that the upstream prerequisite is satisfied.

01
Confirm the route exists

Verify the prefix is learned from the expected peer and has not been rejected, filtered or withdrawn.

02
Compare path attributes

Compare the candidate paths and identify differences that affect best-path selection or multipath eligibility.

03
Check configuration and policy

Verify the correct address family, peer type, maximum-paths setting, import policy and platform-specific requirements.

04
Inspect next-hop resolution

Confirm each next hop resolves through a valid route and is usable by the forwarding plane.

05
Inspect routing and forwarding tables

Determine whether the path is excluded during route selection, omitted from the multipath set or missing from the FIB.

06
Validate the correction

Repeat the checks after a targeted change and confirm both forwarding state and traffic behaviour.

3.2 Read the evidence before changing the configuration

A useful incident investigation records what was expected, what was observed and what the evidence rules out. The following examples illustrate how to narrow the search without assuming that a single attribute explains every multipath failure.

Observed evidence Likely investigation area Next verification
Only one path appears in the received routes. Advertisement, route policy, session or route visibility. Check peer state, received routes where supported, filters and upstream advertisement.
Two paths are known, but one is rejected by policy. Import policy, route map, prefix list or community-based filtering. Inspect policy evaluation and the resulting accepted route attributes.
Both paths are known but differ in a relevant attribute. Best-path decision or platform-specific multipath criteria. Compare the full attribute set and the vendor's documented eligibility rules.
Attributes appear compatible, but one next hop is unresolved. Reachability, recursive resolution or IGP dependency. Check the route to the next-hop address and the associated outgoing interface.
The routing table shows multipath, but the FIB has one next hop. Forwarding installation, hardware constraints or implementation-specific state. Inspect platform forwarding diagnostics, resource limits and programming status.
The FIB has multiple next hops but traffic looks uneven. Traffic hashing, flow distribution, capacity or measurement placement. Compare time-based counters, flow telemetry, errors and drops.

3.3 Do not confuse a best-path difference with a multipath failure

BGP attributes affect route selection, but not every attribute difference automatically makes multipath impossible on every platform. For example, the treatment of AS_PATH, MED, IGP cost to the next hop and other attributes can depend on the implementation and configuration. Use the router's best-path explanation and documented multipath rules rather than relying on a generic list of attributes that must match.

Local Preference is an important example. A higher Local Preference normally favours a path in the BGP decision process. If two routes have different Local Preference values, do not assume they will be installed together as equal multipath alternatives. Establish which path wins, then verify whether the platform supports any relevant relaxation or special configuration.

Evidence discipline

Record the complete relevant path attributes, the selected best path, the platform's multipath eligibility result and the installed next hops. Matching two visible attributes is not proof that two paths meet all eligibility requirements.

3.4 Distinguish three different failure points

Failure point A — The path never reaches the router's usable candidate set

The route may not be advertised, may be filtered, or may be unavailable because the peer or address family is not functioning as expected. Investigate the sender, session, policy and received-route visibility first.

Failure point B — The path is learned but not selected for multipath

The router knows the route, but its selection process or multipath rules do not include it in the installed set. Investigate attributes, next-hop reachability, peer type, configuration scope and vendor-specific criteria.

Failure point C — The path qualifies, but forwarding state is unexpected

If the control plane reports the expected multipath set but the FIB does not reflect it, examine route installation status, hardware capabilities, resource constraints and platform diagnostics. Do not assume a control-plane configuration change is the right fix until the discrepancy is understood.

3.5 A disciplined CLI investigation

Command syntax varies by vendor and release. Use the equivalent commands for your platform to collect the following evidence before modifying configuration.

Investigation sequence 1. Inspect the BGP session and address-family state. 2. Display all available paths for the destination prefix. 3. Compare attributes and the best-path decision. 4. Inspect import policy and multipath configuration. 5. Verify recursive next-hop reachability. 6. Display the routing-table and FIB entries. 7. Review platform diagnostics and interface counters.

Save the initial outputs and note the time of each observation. This creates a baseline for comparing state after a targeted correction. In production, use approved change controls and avoid clearing BGP sessions or withdrawing routes as a first-line diagnostic action.

3.6 Confirm that the problem is actually fixed

A configuration command completing successfully is not the same as a successful operational outcome. Re-run the same checks used to establish the original fault and compare them with the expected state.

A
Control-plane proof

The expected candidate routes are present, and the multipath decision is consistent with the intended design.

B
Forwarding proof

The FIB contains the intended usable next hops, without unexplained installation errors.

C
Traffic proof

Appropriate counters or flow telemetry confirm forwarding, and the original symptom no longer occurs.

D
Resilience proof

Where required, a controlled failure and restoration test confirms the intended alternate-path behaviour.

Section 3 · Engineering takeaway

Find the first point where the expected route state diverges from reality. Separate route visibility, multipath eligibility and FIB installation, then verify the fix at both the control plane and forwarding plane. This is more reliable than changing maximum-paths or comparing only a subset of BGP attributes.

Engineering workflow · Correlate & operate

Can You Prove the Alternate Path Is Visible, Installed and Ready?

A route advertised by a peer is not automatically installed as an additional forwarding path. Correlate what the router receives, which paths qualify for local multipath, what reaches the FIB, and what happens when a link fails.

01
Route visibility

Check which paths the upstream router or Route Reflector advertises and which paths the receiving router actually learns.

BGP updates · Adj-RIB-In
02
Local eligibility

Verify multipath configuration, path compatibility, next-hop reachability and platform-specific selection rules.

Best path · Multipath rules
03
Forwarding state

Inspect the FIB and next-hop entries, then compare interface counters and traffic measurements to confirm actual forwarding.

FIB · Counters · Utilisation
04
Failover & recovery

Withdraw or fail one path in a controlled test. Confirm the remaining path forwards traffic and the expected state returns after recovery.

Failure · Convergence · Validation
Key distinction Add-Path is a negotiated BGP capability that can advertise multiple paths; it does not itself install multiple next hops in the FIB. Route advertisement, local multipath eligibility and forwarding installation must be verified separately.

Section 4 — Correlate & Operate: Route Reflectors, Add-Path and Failover

Multiple paths can exist in the network without all of them reaching the router that needs to forward traffic. To understand the complete forwarding outcome, an engineer must correlate what upstream peers advertise, what route reflectors propagate, what the local router accepts, and what the forwarding plane actually installs.

This is the operational difference between path visibility and multipath forwarding. A route may be available somewhere in the BGP system but hidden from a particular router by route-reflection behaviour, policy, or the capabilities negotiated between peers. Even when multiple routes reach the router, local eligibility and next-hop resolution still determine whether multiple next hops can be installed.

The end-to-end evidence chain

Peer advertisement → Route Reflector decision → Receiving router's BGP table → Multipath eligibility → RIB/FIB installation → Measured traffic.

Each stage answers a different question. If a path disappears between two stages, investigate that boundary before changing configuration elsewhere.

4.1 — Route Reflectors: Which Paths Reach the Client?

Route Reflectors reduce the need for a full mesh of iBGP sessions. In a conventional route-reflection design, a reflector generally advertises its selected best path to a client for a given prefix, subject to its routing policy and implementation. A client may therefore receive fewer paths than the reflector itself knows about.

This matters when investigating multipath. If a router has only one usable candidate route, enabling a local multipath setting cannot create a second route that was never advertised or learned. First establish whether the alternate path is available at the point where the forwarding decision is made.

What the reflector knows

The reflector may learn several paths from different peers but select one path for normal advertisement to a client. Inspect its received routes, best-path decision, export policy, and advertised-route state.

What the client receives

The client can only evaluate routes available to it. Confirm its BGP table and next-hop reachability before concluding that local multipath eligibility is the problem.

Engineering rule: distinguish “the network has another path” from “this router has another eligible path.” These are different statements and require different evidence.

4.2 — Add-Path: Advertise More Than One Path

BGP Add-Path extends BGP so that a peer can advertise multiple paths for the same Network Layer Reachability Information (NLRI), rather than being limited to a single path in the relevant advertisement context. The capability must be supported and negotiated between the peers, and the implementation and configuration must permit the intended paths to be sent and accepted.

Add-Path can improve path visibility in designs where a single advertised best path would otherwise hide useful alternatives. It can be relevant to route-reflector topologies, redundant exits, and scenarios where downstream routers need more candidate paths to make their own routing decisions.

Mechanism Primary purpose What it does not guarantee
Multipath Allows multiple eligible paths to be installed for forwarding, subject to platform and configuration rules. It does not guarantee that upstream peers advertise every alternate path.
Add-Path Allows multiple paths for a prefix to be advertised to a peer when negotiated and configured. It does not, by itself, make the receiver install multiple next hops in its FIB.
Best External Can advertise an eligible external path in addition to the selected best path, where supported and configured. It is not a general replacement for Add-Path or local multipath.
Optimal Route Reflection (ORR) Can help a route reflector select paths that better reflect a client's IGP perspective in supported designs. It does not itself install multiple forwarding paths on the client.

When Add-Path is involved, verify both ends of the relationship: the negotiated capability and the actual routes advertised and received. Then inspect the receiver's own best-path and multipath decisions. A capability being enabled in configuration is not, by itself, proof that the desired paths are reaching the destination router.

4.3 — Next-Hop Reachability, IGP Cost and Egress Choice

A BGP path is useful for forwarding only if its next hop can be resolved. In many designs, that resolution depends on the IGP and the local routing table. A path can be present in BGP but unusable for forwarding if its next hop is unreachable or cannot be resolved according to the platform's rules.

IGP cost can also influence which exit a router prefers. With hot-potato routing, a network commonly prefers to hand traffic to an external network at the closest eligible exit, often based on IGP cost after higher-priority BGP decision criteria have been considered. A topology or IGP metric change can therefore alter the selected egress even if the externally learned route attributes have not changed.

Check next-hop resolution

Confirm the BGP next hop resolves through the expected route, interface, tunnel, or recursive path. Compare the BGP next-hop value with the route used to reach it.

Check the forwarding exit

Compare IGP costs to candidate exits, installed FIB next hops, interface state, and traffic counters. A change in egress may reflect normal routing behaviour rather than a BGP session failure.

Understand the design before diagnosing instability

Interactions between MED comparison rules, route-reflection design, hot-potato decisions, topology changes, and policy can contribute to route instability in some networks. MED is not necessarily compared globally across every candidate path in the same way on every implementation or configuration, and route reflectors do not inevitably cause oscillation.

If a prefix repeatedly changes its selected path, examine the sequence of received updates, the attributes considered at each decision point, the relevant IGP metrics, and the policy applied by each router. Look for a repeatable feedback pattern rather than attributing the problem to one feature based on the symptom alone.

4.4 — Correlate Control Plane, Forwarding Plane and Telemetry

A reliable diagnosis joins several evidence sources. No single BGP command proves that traffic is using all intended paths. Likewise, interface counters alone cannot explain why a path was selected or excluded.

Evidence source Question answered What to correlate next
Peer and update state Are sessions established, and are route updates being exchanged? Received prefixes, advertised routes, withdrawals, and session events.
Route Reflector Which candidate paths are learned, selected, and advertised to the client? Export policy, Add-Path negotiation, path identifiers where applicable, and client receive state.
Local BGP table and RIB Which paths are available and eligible at the destination router? Path attributes, multipath rules, next-hop resolution, and routing policy.
FIB and adjacency state Which next hops are programmed for forwarding? Installed interfaces, recursive resolution, hardware or software programming status.
Interface counters and telemetry Is traffic actually using the intended links? Counter deltas, traffic direction, hash behaviour, utilisation, drops, and latency.
Evidence standard: capture a baseline and timestamps. Align routing events, FIB changes, interface counters, and traffic measurements to the same interval. Otherwise, normal update delays can make unrelated observations look like one failure.

4.5 — Run a Controlled Failover and Recovery Test

A second path is operationally valuable only if the network can use it under the conditions the design is intended to tolerate. Validate the failure and recovery sequence in a controlled maintenance window or lab, with a defined rollback plan and measurable acceptance criteria.

  1. Record the baseline Capture BGP paths, selected route, FIB next hops, peer state, interface counters, and representative traffic measurements.
  2. Trigger one controlled failure Disable or isolate the intended path in the test environment. Avoid changing several variables at once.
  3. Observe convergence Record the failure signal, route withdrawal or selection change, FIB update, traffic movement, packet loss, and recovery time.
  4. Restore and verify Re-enable the path, confirm session and route recovery, inspect the resulting FIB, and compare traffic against the original baseline.
Operational acceptance checklist
  • The intended alternate path is visible at the router that needs it.
  • Its next hop resolves and it satisfies the platform's multipath eligibility rules.
  • The forwarding table contains the expected next hops before the test.
  • A single controlled failure causes the expected route and forwarding changes.
  • Traffic moves to surviving paths within the defined convergence objective.
  • After restoration, routing and forwarding state return to an acceptable condition.
  • Logs, timestamps, counters, and test results provide evidence for the outcome.
Engineering takeaway

Multipath is an end-to-end outcome, not a single configuration command. Prove that alternate routes are advertised and received, eligible locally, resolved through valid next hops, installed in the FIB, and used as intended. Then test both failure and recovery to establish that the design behaves correctly under operational conditions.

Conclusion — Proving BGP Multipath Works

BGP multipath is about more than learning two routes. It is about determining whether multiple paths are available, eligible for simultaneous installation, programmed into the forwarding plane, and used correctly when traffic flows through the network.

Effective troubleshooting follows the evidence from the control plane to the data plane. Start with the routes the router receives, establish why each path is accepted or excluded, verify the installed next hops, and measure what happens to real traffic.

01 · SEE

Understand the path

Identify the candidate routes, their attributes, next hops, and the decisions that determine which paths are eligible.

02 · OBSERVE

Prove forwarding behaviour

Compare the routing table and FIB with interface counters and traffic measurements. Multiple installed paths do not guarantee equal traffic distribution.

03 · INVESTIGATE

Find the first failure point

Determine whether the alternate route is missing, ineligible for multipath, unresolved, or absent from the forwarding table.

04 · CORRELATE & OPERATE

Validate resilience

Correlate route advertisements, route-reflector behaviour, next-hop resolution, FIB changes, and telemetry during failure and recovery.

The key engineering principle: a route visible in BGP is not necessarily installed for forwarding, and a path installed in the FIB is not proof that traffic is distributed as intended. Each stage requires its own evidence.

From configuration to operational confidence

Use the interactive labs in this guide to explore path eligibility, experiment with the maximum number of paths, and investigate why a second route might not be installed. Change one condition at a time, observe the resulting state, and use the evidence to support your diagnosis.

A resilient BGP design is not proven by configuration alone. It is proven when the expected paths are visible, the intended next hops are installed, traffic behaves as designed, and controlled failure-and-recovery testing confirms the result.

BGP Incident Investigation

Why Isn't the Second BGP Path Being Installed?

Two routes appear to reach the router, but only one next hop is present in the forwarding table. Investigate the evidence and identify the most likely reason the second path is not entering the multipath set.

Incident evidence
!
Forwarding imbalance detected Prefix 203.0.113.0/24 is reachable through two eBGP peers, but only one next hop is installed.
Path A · Local Preference 200
Path B · Local Preference 200
Path A · AS Path 65010
Path B · AS Path 65010
Configured maximum paths 2
FIB next hops 1 / 2
Observed forwarding state
Router 203.0.113.0/24
→
PE1 10.0.1.1 · FIB
×
PE2 10.0.2.1 · not installed
Investigation score
0 / 1
What is the most likely diagnosis?
Investigation required.
Review the evidence before selecting a diagnosis.
Engineer takeaway: Matching Local Preference and AS Path does not automatically guarantee multipath installation. Implementations can require additional attributes and next-hop or path conditions to match before multiple paths become eligible.
BGP prefix 203.0.113.0/24 received from PE1 and PE2
→ LocalPref comparison: equal
→ AS_PATH comparison: equal
→ Multipath eligibility requires further validation
→ FIB currently contains next-hop 10.0.1.1 only
☕
Support Network Insight
Help support the development of interactive networking labs, BGP simulations, and educational content.
Support the Project
ip routing

Advances of IP routing and Cloud

 

ip routing

 

With the introduction and hype around Software Defined Networking ( SDN ) and Cloud Computing, one would assume that there has been little or no work with the advances in IP routing. You could say that the cloud has clouded the mind of the market. Regardless of the hype around this transformation, routing is still very much alive and makes up a vital part of the main internet statistics you can read. Packets still need to get to their destinations.

 

Advanced in IP Routing

The Internet Engineering Task Force (IETF) develops and promotes voluntary internet standards, particularly those that comprise the Internet Protocol Suite (TCP/IP). The IETF shapes what comes next, and this is where all the routing takes place. It focuses on anything between the physical layer and the application layer. It doesn’t focus on the application itself, but on the technologies used to transport it, for example, HTTP.

In the IETF, no one is in charge, anyone can contribute, and everyone can benefit. As you can see from the chart below, that routing ( RTG ) has over 250 active drafts and is the most popular working group within the IETF.

 

 

IP routinng
Diagram: IETF Work Distribution.

 

The routing area is responsible for ensuring the continuous operation of the Internet routing system by maintaining the scalability and stability characteristics of the existing routing protocols and developing new protocols, extensions, and bug fixes promptly

The following table illustrates the subgroups of the RTG working group:

Bidirectional Forwarding Detection (BFD) Open Shortest Path First IGP (OSPF)
Forwarding and Control Element Separation (forces) Routing Over Low power and Lossy networks (roll)
Interface to the Routing System (i2rs) Routing Area Working Group (RTGW)
Inter-Domain Routing (IDR) Secure Inter-Domain Routing (SCIDR)
IS-IS for IP Internets (isis) Source Packet Routing in Networking (spring)
Keying and Authentication for Routing Protocols (Karp)
Mobile Ad-hoc Networks (manet)

The chart below displays the number of drafts per subgroup of the routing area. There has been a big increase in the subgroup “roll,” which is second to BGP. “Roll” relates to “Routing Over Low power and Lossy networks” and is driven by the Internet of Everything and Machine-to-Machine communication.

 

IP ROUTING
Diagram: RTG Ongoing Work.

 

OSPF Enhancements

OSPF is a link-state protocol that uses a common database to determine the shortest path to any destination.

Two main areas of interest in the Open Shortest Path First IGP (OSPF) subgroups are OSPFv3 LSA Extendibility and Remote Loop-Free Alternatives ( LFAs ). One benefit IS-IS has over OSPF is its ability to easily introduce new features with the inclusion of Type Length Values ( TLVs ) and sub-TLVs. The IETF draft-IETF-OSPF-ospfv3-lsa-extend extends the LSA format by allowing the optional inclusion of TLVs, making OSPF more flexible and extensible. For example, OSPFv3 uses a new TLV to support intra-area Traffic Engineering ( TE ), while OSPFv2 uses an opaque LSA.

 

TLV for OSPFv3
Diagram: TLV for OSPFv3.

 

Another shortcoming of OSPF is that it does not install a backup route in the routing table by default. Having a pre-installed backup up path greatly improves convergence time. With pre-calculated backup routes already installed in the routing table, the router process does not need to go through the convergence process’s LSA propagation and SPF calculation steps.

 

Loop-free alternatives (LFA)

Loop-Free Alternatives ( LFA ), known as Feasible Successors in EIGRP, are local router decisions to pre-install a backup path.
In the diagram below:

-Router A has a primary ( A-C) and secondary ( A-B-C) path to 10.1.1.0/24
-Link State allows Router A to know the entire topology
-Router A should know that Router B is an alternative path. Router B is a Loop-Free Alternate for destination 10.1.1.0/24

OSPF LFA
Diagram: OSPF LFA.

 

This is not done with any tunneling, and the backup route needs to exist for it to be used by the RIB. If the second path doesn’t exist in the first place, the OSPF process cannot install a Loop-Free Alternative. The LFA process does not create backup routes if they don’t already exist. An LFA is simply an alternative loop-free route calculated at any network router.

A drawback of LFA is that it cannot work in all topologies. This is most notable in RING topologies. The answer is to tunnel and to get the traffic past the point where it will loop. This effectively makes the RING topology a MESH topology. For example, the diagram below recognizes that we must tunnel traffic from A to C. The tunnel type doesn’t matter – it could be a GRE tunnel, an MPLS tunnel, an IP-in-IP tunnel, or just about any other encapsulation.

 

In this network:

-Router A’s best path through E
-Routers C’s best path is through D
-Router A must forward traffic directly to C to prevent packets from looping back.

Remote LFA
Diagram: Remote LFA.

 

In the preceding example, we will look at “Remote LFA,” which leverages an MPLS network and Label Distribution Protocol ( LDP ) for label distribution. If you use Traffic Engineering ( TE ), it’s called “TE Fast ReRoute” and not “Remote LFA.” There is also a hybrid model combining Remote LFA and TE Fast ReRoute, and is used only when the above cannot work efficiently due to a complex topology or corner case scenario.

Remote LFAs extend the LFA space to “tunneled neighbors”.

– Router A runs a constrained SPF and finds C is a valid LFA

– Since C is not directly connected, Router A must tunnel to C

a) Router A uses LDP to configure an MPLS path to C

b) Installs this alternate path as an LFA in the CEF table

– If the A->E link fails.

a) Router A begins forwarding traffic along the LDP path

The total time for convergence usually takes 10ms.

Remote LFA has some topology constraints. For example, they cannot be calculated across a flooding domain boundary, i.e., an ABR in OSPF or L1/L2 boundary is IS-IS. However, they work in about 80% of all possible topologies and 90% of production topologies.

 

BGP Enhancements

BGP is a scalable distance vector protocol that runs on top of TCP. It uses a path algorithm to determine the best path to install in the IP routing table and for IP forwarding.

 

Recap BGP route advertisement:

  • RR client can send to a RR client.
  • RR client can send to a non RR client.
  • A non-RR client cannot send to a non-RR client.

One drawback to the default BGP behavior is that it only advertises the best route. When a BGP Route Reflector receives multiple paths to the same destination, it will advertise only one of those routes to its neighbors.

This can limit the visibility in the network and affect the best path selection used for hot potato routing when you want traffic to leave your AS as quickly as possible. In addition, all paths to exit an AS are not advertised to all peers, basically hiding ( not advertising ) some paths to exit the network.

The diagram below displays default BGP behavior; the RR receives two routes from PE2 and PE3 about destination Z; due to the BGP best path mechanism, it only advertises one of those paths to PE1. 

Route Reflector - Default
Diagram: Route Reflector – Default.

 

In certain designs, you could advertise the destination CE with different Route Distinguishers (RDs), creating two instances for the same destination prefix. This would allow the RR to send two paths to PE.

 

Diverse BGP path distribution

Another new feature is diverse BGP Path distribution, where you can create a shadow BGP session to the RR. It is easy to deploy, and the diverse iBGP session will announce the 2nd best path. Shadow BGP sessions are especially useful in virtualized deployments, where you can create another BGP session to a VM acting as a Route-Reflector. The VM can then be scaled out in a virtualized environment creating numerous BGP sessions. You are allowing the advertisements of multiple paths for each destination prefix.

Route Reflector - Shadow Sessions
Diagram: Route Reflector – Shadow Sessions.

 

BGP Add-path 

A special extension to BGP known as “Add Paths” allows BGP speakers to propagate and accept multiple paths for the same prefix. The BGP Add-Path feature will signal diverse paths, so you don’t need to create shadow BGP sessions. There is a requirement that all Add-Path receiver BGP routers must support the Add-Path capability.

There are two flavors of the Add-Path capability, Add-n-path, and Add-all-path. The “Add-n-path” will add 2 or 3 paths depending on the IOS version. With “Add-all-path,” the route reflector will do the primary best path computation (only on the first path) and then send all paths to the BR/PE. This is useful for large ECMP load balancing, where you need multiple existing paths for hot potato routing.

BGP Add Path
Diagram: BGP Add Path

 

Source packet routing

Another interesting draft the IETF is working on is Source Packet Routing ( spring ). Source Packet Routing is the ability of a node to specify a forwarding path. As the packet arrives in the network, the edge device looks at the application, determines what it needs, and predicts its path throughout the network. Segment routing leverages the MPLS data plane, i.e., push, swap, and pop controls, without needing LDP or RSVP-TE. This avoids millions of labels in the LDP database or TE LSPs in the networks.

 

Application Controls - Network DeliversDiagram: Application Controls – Network Delivers 

The complexity and state are now isolated to the network’s edges, and the middle nodes are only swapping labels. The source routing is based on the notion of a 32-bit segment that can represent any instructions, such as service, context, or IGP-based forwarding construct. This results in an ordered chain of topological and service instructions where the ingress node pushes the segment list on the packet.

 

Prefix Hijacking in BGP

BGP hijacking revolves around locating an ISP that is not filtering advertisements, or its misconfiguration makes it susceptible to a man-in-the-middle attack. Once located, an attacker can advertise any prefix they want, causing some or all traffic to be diverted from the real source towards the attacker.

In February 2008, a large portion of YouTube’s address space was redirected to Pakistan when the Pakistan Telecommunication Authority ( PTA ) decided to restrict access to YouTube.com inside the country but accidentally blackholed the route in the global BGP table.

These events and others have led the Secure-Inter Domain Routing Group ( SIDR ) to address the following two vulnerabilities in BGP:

-Is the AS authorized to originate an IP prefix?

-Is the AS-Path represented in the route the same as the path through which the NLRI traveled?

This lockdown of BGP has three solution components:

 

RPKI Infrastructure Offline repository of verifiable secure objects based on public-key cryptography
Follows resources (IPv4/v6 + ASN) allocation hierarchy to provide “right of use”
BGP Secure Origin AS You only validate the Origin AS of a BGP UPDATE
Solves most frequent incidents (*)
No changes to BGP nor the router’s hardware impact
Standardization is almost finished and running code
BGP PATH Validation BGPSEC proposal under development at IETF
Requires forward signing AS-PATH attribute
Changes in BGP and possible routers

The roll-out and implementation should be gradual and create islands of trust worldwide. These islands of trust will eventually interconnect together, making BGP more secure.

The table below displays the RPKI Deployment State;

RIR Total Valid Invalid Unknown Accuracy RPKI Adoption Rate
AFRINIC 100% .44% .42% 99.14% 51.49% .86%
APNIC 100% .22% .24% 99.5% 48.32% .46%
ARIN 100% .4% .14% 99.46% 74.65% .54%
LACNIC 100% 17.84% 2.01% 80.15% 89.87% 19.85%
RIPE NCC 100% 6.7% 0.59% 92.71% 91.92% 7.29%

Cloud Enhancements – The Intercloud

Today’s clouds have crossed well beyond the initial hype, and applications are now offered as on-demand services ( anything-as-a-service [XaaS] ). These services are making significant cost savings, and the cloud transition is shaping up to be as powerful as the previous one – the Internet. The Intercloud and the Internet of Things are the two new big clouds of the future.

Currently, the cloud market is driven by two spaces – the public cloud ( off-premise ) and the private cloud (on-premise). The intercloud takes the concept of cloud much further and attempts to connect multiple public clouds. A single application that could integrate services and workloads from ten or more clouds would create opportunities and potentially alter the cloud market landscape significantly. Hence, it is important to know and understand the need for cloud migration and its related problems.

We are already beginning to see signs of this in the current market. Various applications, such as Spotify and Google Maps, authenticate unregistered users with their Facebook credentials. Another use case is a cloud IaaS provider could divert incoming workload to another provider if it doesn’t have the resources to serve the incoming requests, essentially cloud bursting from provider to provider. It would also make economic sense to move workload and services between cloud providers based on cooling costs ( follow the moon ). Or maybe dynamically move workloads between providers, so they are closest to the active user base ( follow the sun )

The following diagram displays a Dynamic Workload Migration between two Cloud companies.

 

Intercloud
Diagram: Intercloud.

 

A: Cloud 1 finds Cloud 2 -Naming, Presence
B: Cloud 1 Trusts Cloud 2 -Certificates, Trustsec
C: Both Cloud 1 and 2 negotiate -Security, Policy
D: Cloud 1 sets up Cloud 2 -Placement, Deployment
E: Cloud 1 sends to Cloud 2 -VM runs in cloud-Addressing, configurations

The concept of Intercloud was difficult to achieve with the previous version of vSphere based on the restriction of latency for VMotion to operate efficiently. Now vSphere v6 can tolerate 100 msec of RTT.

InterCloud is still a conceptual framework, and the following questions must be addressed before it can be moved from concept to production.

1) Intercloud security

2) Intercloud SLA management

3) Interoperability across cloud providers.

 

Cisco’s One Platform Kit (onePK)

The One Platform Kit is Cisco’s answer to Software Defined Networking. It aims to provide simplicity and agility to a programmatic network. It’s a set of APIs driven by programming languages, such as C and Java, that are used to program the network. We currently have existing ways to program the network with EEM applets but lack an automation tool that can program multiple devices simultaneously. It’s the same with Performance Routing ( PfR ). PfR can program and traffic engineer the network by remotely changing metrics, but the decisions are still local and not controller-based.

 

Traffic engineering

One useful element of Cisco’s One Platform Kit is its ability to perform “Off box” traffic engineering, i.e., the computation is made outside the local routing device. It allows you to create route paths throughout the network without relying on default routing protocol behavior. For example, the cost is the default metric for route selection for equal-length routes in OSPF. This cannot be changed, which makes the routing decisions very static. In addition, Cisco’s One Platform Kit (onePK) allows you to calculate routes using different variables you set, giving you complete path control.

 

ip routing