fabric extender Archives

Physical diagram - Fabric Extenders and Parent switches

How-to: Fabric Extenders & VPC

August 22, 2014

by Matt Conran Blog

Topology Diagram

The topology diagram depicts two Nexus 5K acting as parent switches with physical connections to two downstream Nexus 2k (FEX) acting as the 10G physical termination points for the connected server.

Part1. Connecting the FEX to the Parent switch:

The FEX and the parent switch use Satellite Discovery Protocol (SDP) periodic messages to discovery and register with one another.

When you initially log on to the Nexus 5K you can see that the OS does not recognise the FEX even though there are two FEXs that are cabled correctly to parent switch. As the FEX is recognised as a remote line card you would expect to see it with a “show module” command.

N5K3# sh module
Mod Ports Module-Type Model Status
— —– ——————————– ———————- ————
1 40 40x10GE/Supervisor N5K-C5020P-BF-SUP active *
2 8 8×1/2/4G FC Module N5K-M1008 ok
Mod Sw Hw World-Wide-Name(s) (WWN)
— ————– —— ————————————————–
1 5.1(3)N2(1c) 1.3 —

2 5.1(3)N2(1c) 1.0 93:59:41:08:5a:0c:08:08 to 00:00:00:00:00:00:00:00

Mod MAC-Address(es) Serial-Num

— ————————————– ———-

1 0005.9b1e.82c8 to 0005.9b1e.82ef JAF1419BLMA

2 0005.9b1e.82f0 to 0005.9b1e.82f7 JAF1411AQBJ

We issue the “feature fex” command we observe the FEX sending SDP messages to the parent switch i.e. RX but we don’t see the parent switch sending SDP messages to the FEX i.e. TX.

Notice in the output below there is only “fex:Sdp-Rx” messages.

N5K3# debug fex pkt-trace
N5K3# 2014 Aug 21 09:51:57.410701 fex: Sdp-Rx: Interface: Eth1/11, Fex Id: 0, Ctrl Vntag: -1, Ctrl Vlan: 1
2014 Aug 21 09:51:57.410729 fex: Sdp-Rx: Refresh Intvl: 3000ms, Uid: 0x4000ff2929f0, device: Fex, Remote link: 0x20000080
2014 Aug 21 09:51:57.410742 fex: Sdp-Rx: Vendor: Cisco Systems Model: N2K-C2232PP-10GE Serial: FOC17100NHX
2014 Aug 21 09:51:57.821776 fex: Sdp-Rx: Interface: Eth1/10, Fex Id: 0, Ctrl Vntag: -1, Ctrl Vlan: 1
2014 Aug 21 09:51:57.821804 fex: Sdp-Rx: Refresh Intvl: 3000ms, Uid: 0x2ff2929f0, device: Fex, Remote link: 0x20000080
2014 Aug 21 09:51:57.821817 fex: Sdp-Rx: Vendor: Cisco Systems Model: N2K-C2232PP-10GE Serial: FOC17100NHU

The FEX appears as “DISCOVERED” but no additional FEX host interfaces appear when you issue a “show interface brief“.

Command: show fex [chassid_id [detail]]: Displays information about a specific Fabric Extender Chassis ID

Command: show interface brief: Display interface information and connection status for each interface.

N5K3# sh fex
FEX FEX FEX FEX
Number Description State Model Serial
————————————————————————
— ——– Discovered N2K-C2232PP-10GE SSI16510AWF
— ——– Discovered N2K-C2232PP-10GE SSI165204YC
N5K3#
N5K3# show interface brief

——————————————————————————–

Ethernet VLAN Type Mode Status Reason Speed Port

Interface Ch #

——————————————————————————–

Eth1/1 1 eth access down SFP validation failed 10G(D) —

Eth1/2 1 eth access down SFP validation failed 10G(D) —

Eth1/3 1 eth access up none 10G(D) —

Eth1/4 1 eth access up none 10G(D) —

Eth1/5 1 eth access up none 10G(D) —

Eth1/6 1 eth access down Link not connected 10G(D) —

Eth1/7 1 eth access down Link not connected 10G(D) —

Eth1/8 1 eth access down Link not connected 10G(D) —

Eth1/9 1 eth access down Link not connected 10G(D) —

Eth1/10 1 eth fabric down FEX not configured 10G(D) —

Eth1/11 1 eth fabric down FEX not configured 10G(D) —

Eth1/12 1 eth access down Link not connected 10G(D) —

snippet removed

The Fabric interface Ethernet1/10 show as DOWN with a “FEX not configured” statement.

N5K3# sh int Ethernet1/10
Ethernet1/10 is down (FEX not configured)
Hardware: 1000/10000 Ethernet, address: 0005.9b1e.82d1 (bia 0005.9b1e.82d1)
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA
Port mode is fex-fabric
auto-duplex, 10 Gb/s, media type is 10G

Beacon is turned off

Input flow-control is off, output flow-control is off

Rate mode is dedicated

Switchport monitor is off

EtherType is 0x8100

snippet removed

To enable the parent switch to fully discover the FEX we need to issue the “switchport mode fex-fabric” under the connected interface. As you can see we are still not sending any SDP messages but we are discovering the FEX.

The next step is to enable the FEX logical numbering under the interface so we can start to configure the FEX host interfaces. Once this is complete we run the “debug fex pkt-trace” and we are not sending TX and receiving RX SDP messages.

Command:”fex associate chassis_id“: Associates a Fabric Extender (FEX) to a fabric interface. To disassociate the Fabric Extender, use the “no” form of this command.

From the “debug fexpkt-race” you can see the parent switch is now sending TX SDP messages to the fully discovered FEX.

N5K3(config)# int Ethernet1/10
N5K3(config-if)# fex associate 101
N5K3# debug fex pkt-trace
N5K3# 2014 Aug 21 10:00:33.674605 fex: Sdp-Tx: Interface: Eth1/10, Fex Id: 101, Ctrl Vntag: 0, Ctrl Vlan: 4042
2014 Aug 21 10:00:33.674633 fex: Sdp-Tx: Refresh Intvl: 3000ms, Uid: 0xc0821e9b0500, device: Switch, Remote link: 0x1a009000
2014 Aug 21 10:00:33.674646 fex: Sdp-Tx: Vendor: Model: Serial: ———-

2014 Aug 21 10:00:33.674718 fex: Sdp-Rx: Interface: Eth1/10, Fex Id: 0, Ctrl Vntag: 0, Ctrl Vlan: 4042

2014 Aug 21 10:00:33.674733 fex: Sdp-Rx: Refresh Intvl: 3000ms, Uid: 0x2ff2929f0, device: Fex, Remote link: 0x20000080

2014 Aug 21 10:00:33.674746 fex: Sdp-Rx: Vendor: Cisco Systems Model: N2K-C2232PP-10GE Serial: FOC17100NHU

2014 Aug 21 10:00:33.836774 fex: Sdp-Rx: Interface: Eth1/11, Fex Id: 0, Ctrl Vntag: -1, Ctrl Vlan: 1

2014 Aug 21 10:00:33.836803 fex: Sdp-Rx: Refresh Intvl: 3000ms, Uid: 0x4000ff2929f0, device: Fex, Remote link: 0x20000080

2014 Aug 21 10:00:33.836816 fex: Sdp-Rx: Vendor: Cisco Systems Model: N2K-C2232PP-10GE Serial: FOC17100NHX

2014 Aug 21 10:00:36.678624 fex: Sdp-Tx: Interface: Eth1/10, Fex Id: 101, Ctrl Vntag: 0, Ctrl Vlan: 4042

2014 Aug 21 10:00:36.678664 fex: Sdp-Tx: Refresh Intvl: 3000ms, Uid: 0xc0821e9b0500, device: Switch, Remote snippet removed

Now the 101 FEX status changes from “DISCOVERED” to “ONLINE”. You may also see an additional FEX with serial number SSI165204YC as “DISCOVERED” and not “ONLINE”. This is due to the fact that we have not explicitly configured it under the other Fabric interface.

N5K3# sh fex
FEX FEX FEX FEX
Number Description State Model Serial
————————————————————————
101 FEX0101 Online N2K-C2232PP-10GE SSI16510AWF
— ——– Discovered N2K-C2232PP-10GE SSI165204YC
N5K3#
N5K3# show module fex 101

FEX Mod Ports Card Type Model Status.

— — —– ———————————- —————— ———–

101 1 32 Fabric Extender 32x10GE + 8x10G Module N2K-C2232PP-10GE present

FEX Mod Sw Hw World-Wide-Name(s) (WWN)

— — ————– —— ———————————————–

101 1 5.1(3)N2(1c) 4.4 —

FEX Mod MAC-Address(es) Serial-Num

— — ————————————– ———-

101 1 f029.29ff.0200 to f029.29ff.021f SSI16510AWF

Issuing the “show interface brief” we see new interfaces, specifically host interfaces for the FEX. The syntax below shows that only one interface is up; interface labelled Eth101/1/1. Reason for this is that only one end host (server) is connected to the FEX

N5K3# show interface brief
——————————————————————————–
Ethernet VLAN Type Mode Status Reason Speed Port
Interface Ch #
——————————————————————————–
Eth1/1 1 eth access down SFP validation failed 10G(D) —
Eth1/2 1 eth access down SFP validation failed 10G(D) —
snipped removed

——————————————————————————–

Port VRF Status IP Address Speed MTU

——————————————————————————–

mgmt0 — up 192.168.0.53 100 1500

——————————————————————————–

Ethernet VLAN Type Mode Status Reason Speed Port

Interface Ch #

——————————————————————————–

Eth101/1/1 1 eth access up none 10G(D) —

Eth101/1/2 1 eth access down SFP not inserted 10G(D) —

Eth101/1/3 1 eth access down SFP not inserted 10G(D) —

Eth101/1/4 1 eth access down SFP not inserted 10G(D) —

Eth101/1/5 1 eth access down SFP not inserted 10G(D) —

Eth101/1/6 1 eth access down SFP not inserted 10G(D) —

snipped removed

N5K3# sh run int eth1/10
interface Ethernet1/10
switchport mode fex-fabric
fex associate 101

The Fabric Interfaces do not run a Spanning tree instance while the host interfaces do run BPDU guard and BPDU filter by default. The reason why the fabric interfaces do not run spanning tree is because they are backplane point to point interfaces.

By default, the FEX interfaces will send out a couple of BPDU’s on start-up.

N5K3# sh spanning-tree interface Ethernet1/10
No spanning tree information available for Ethernet1/10
N5K3#
N5K3#
N5K3# sh spanning-tree interface Eth101/1/1Vlan Role Sts Cost Prio.Nbr Type
—————- —- — ——— ——– ——————————–
VLAN0001 Desg FWD 2 128.1153 Edge P2p

N5K3#

N5K3# sh spanning-tree interface Eth101/1/1 detail

Port 1153 (Ethernet101/1/1) of VLAN0001 is designated forwarding

Port path cost 2, Port priority 128, Port Identifier 128.1153

Designated root has priority 32769, address 0005.9b1e.82fc

Designated bridge has priority 32769, address 0005.9b1e.82fc

Designated port id is 128.1153, designated path cost 0

Timers: message age 0, forward delay 0, hold 0

Number of transitions to forwarding state: 1

The port type is edge

Link type is point-to-point by default

Bpdu guard is enabled

Bpdu filter is enabled by default

BPDU: sent 11, received 0

N5K3# sh spanning-tree interface Ethernet1/10
No spanning tree information available for Ethernet1/10
N5K3#
N5K3#
N5K3# sh spanning-tree interface Eth101/1/1Vlan Role Sts Cost Prio.Nbr Type—————- —- — ——— ——– ——————————–
VLAN0001 Desg FWD 2 128.1153 Edge P2p

N5K3#

N5K3# sh spanning-tree interface Eth101/1/1 detail

Port 1153 (Ethernet101/1/1) of VLAN0001 is designated forwarding

Port path cost 2, Port priority 128, Port Identifier 128.1153

Designated root has priority 32769, address 0005.9b1e.82fc

Designated bridge has priority 32769, address 0005.9b1e.82fc

Designated port id is 128.1153, designated path cost 0

Timers: message age 0, forward delay 0, hold 0

Number of transitions to forwarding state: 1

The port type is edge

Link type is point-to-point by default

Bpdu guard is enabled

Bpdu filter is enabled by default

BPDU: sent 11, received 0

Issue the commands below to determine the transceiver type for the fabric ports and also the hosts ports for each fabric interface.

Command: “show interface fex-fabric“: displays all the Fabric Extender interfaces

Command: “show fex detail“: Shows detailed information about all FEXs. Including more recent log messages related to the FEX.

N5K3# show interface fex-fabric
Fabric Fabric Fex FEX
Fex Port Port State Uplink Model Serial
—————————————————————
101 Eth1/10 Active 3 N2K-C2232PP-10GE SSI16510AWF
— Eth1/11 Discovered 3 N2K-C2232PP-10GE SSI165204YC
N5K3#
N5K3#

N5K3# show interface Ethernet1/10 fex-intf

Fabric FEX

Interface Interfaces

—————————————————

Eth1/10 Eth101/1/1

N5K3#

N5K3# show interface Ethernet1/10 transceiver

Ethernet1/10

transceiver is present

type is SFP-H10GB-CU3M

name is CISCO-TYCO

part number is 1-2053783-2

revision is N

serial number is TED1530B11W

nominal bitrate is 10300 MBit/sec

Link length supported for copper is 3 m

cisco id is —

cisco extended id number is 4

N5K3# show fex detail

FEX: 101 Description: FEX0101 state: Online

FEX version: 5.1(3)N2(1c) [Switch version: 5.1(3)N2(1c)]

FEX Interim version: 5.1(3)N2(1c)

Switch Interim version: 5.1(3)N2(1c)

Extender Serial: SSI16510AWF

Extender Model: N2K-C2232PP-10GE, Part No: 73-12533-05

Card Id: 82, Mac Addr: f0:29:29:ff:02:02, Num Macs: 64

Module Sw Gen: 12594 [Switch Sw Gen: 21]

post level: complete

pinning-mode: static Max-links: 1

Fabric port for control traffic: Eth1/10

FCoE Admin: false

FCoE Oper: true

FCoE FEX AA Configured: false

Fabric interface state:

Eth1/10 – Interface Up. State: Active

Fex Port State Fabric Port

Eth101/1/1 Up Eth1/10

Eth101/1/2 Down None

Eth101/1/3 Down None

Eth101/1/4 Down None

snippet removed

Logs:

08/21/2014 10:00:06.107783: Module register received

08/21/2014 10:00:06.109935: Registration response sent

08/21/2014 10:00:06.239466: Module Online S

Now we quickly enable the second FEX connected to fabric interface E1/11.

N5K3(config)# int et1/11
N5K3(config-if)# switchport mode fex-fabric
N5K3(config-if)# fex associate 102
N5K3(config-if)# end
N5K3# sh fex
FEX FEX FEX FEX
Number Description State Model Serial

————————————————————————

101 FEX0101 Online N2K-C2232PP-10GE SSI16510AWF

102 FEX0102 Online N2K-C2232PP-10GE SSI165204YC

N5K3# show fex detail

FEX: 101 Description: FEX0101 state: Online

FEX version: 5.1(3)N2(1c) [Switch version: 5.1(3)N2(1c)]

FEX Interim version: 5.1(3)N2(1c)

Switch Interim version: 5.1(3)N2(1c)

Extender Serial: SSI16510AWF

Extender Model: N2K-C2232PP-10GE, Part No: 73-12533-05

Card Id: 82, Mac Addr: f0:29:29:ff:02:02, Num Macs: 64

Module Sw Gen: 12594 [Switch Sw Gen: 21]

post level: complete

pinning-mode: static Max-links: 1

Fabric port for control traffic: Eth1/10

FCoE Admin: false

FCoE Oper: true

FCoE FEX AA Configured: false

Fabric interface state:

Eth1/10 – Interface Up. State: Active

Fex Port State Fabric Port

Eth101/1/1 Up Eth1/10

Eth101/1/2 Down None

Eth101/1/3 Down None

Eth101/1/4 Down None

Eth101/1/5 Down None

Eth101/1/6 Down None

snippet removed

Logs:

08/21/2014 10:00:06.107783: Module register received

08/21/2014 10:00:06.109935: Registration response sent

08/21/2014 10:00:06.239466: Module Online Sequence

08/21/2014 10:00:09.621772: Module Online

FEX: 102 Description: FEX0102 state: Online

FEX version: 5.1(3)N2(1c) [Switch version: 5.1(3)N2(1c)]

FEX Interim version: 5.1(3)N2(1c)

Switch Interim version: 5.1(3)N2(1c)

Extender Serial: SSI165204YC

Extender Model: N2K-C2232PP-10GE, Part No: 73-12533-05

Card Id: 82, Mac Addr: f0:29:29:ff:00:42, Num Macs: 64

Module Sw Gen: 12594 [Switch Sw Gen: 21]

post level: complete

pinning-mode: static Max-links: 1

Fabric port for control traffic: Eth1/11

FCoE Admin: false

FCoE Oper: true

FCoE FEX AA Configured: false

Fabric interface state:

Eth1/11 – Interface Up. State: Active

Fex Port State Fabric Port

Eth102/1/1 Up Eth1/11

Eth102/1/2 Down None

Eth102/1/3 Down None

Eth102/1/4 Down None

Eth102/1/5 Down None

snippet removed

Logs:

08/21/2014 10:12:13.281018: Module register received

08/21/2014 10:12:13.283215: Registration response sent

08/21/2014 10:12:13.421037: Module Online Sequence

08/21/2014 10:12:16.665624: Module Online

Part 2. Fabric Interfaces redundancy

Static Pinning is when you pin a number of host ports to a fabric port. If the fabric port goes down so do the host ports that are pinned to it. This is useful when you want no oversubscription in the network.

Once the host port shut down due to a fabric port down event, the server if configured correctly should revert to the secondary NIC.

The “pinning max-link” divides the number specified in the command by the number of host interfaces to determine how many host interfaces go down if there is a fabric interface failure.

Now we shut down fabric interface E1/10, you can see that Eth101/1/1 has changed its operation mode to DOWN. The FEX has additional connectivity with E1/11 which remains up.

Enter configuration commands, one per line. End with CNTL/Z.
N5K3(config)# int et1/10
N5K3(config-if)# shu
N5K3(config-if)#
N5K3(config-if)# end
N5K3# sh fex detail
FEX: 101 Description: FEX0101 state: Offline
FEX version: 5.1(3)N2(1c) [Switch version: 5.1(3)N2(1c)]

FEX Interim version: 5.1(3)N2(1c)

Switch Interim version: 5.1(3)N2(1c)

Extender Serial: SSI16510AWF

Extender Model: N2K-C2232PP-10GE, Part No: 73-12533-05

Card Id: 82, Mac Addr: f0:29:29:ff:02:02, Num Macs: 64

Module Sw Gen: 12594 [Switch Sw Gen: 21]

post level: complete

pinning-mode: static Max-links: 1

Fabric port for control traffic:

FCoE Admin: false

FCoE Oper: true

FCoE FEX AA Configured: false

Fabric interface state:

Eth1/10 – Interface Down. State: Configured

Fex Port State Fabric Port

Eth101/1/1 Down Eth1/10

Eth101/1/2 Down None

Eth101/1/3 Down None

snippet removed

Logs:

08/21/2014 10:00:06.107783: Module register received

08/21/2014 10:00:06.109935: Registration response sent

08/21/2014 10:00:06.239466: Module Online Sequence

08/21/2014 10:00:09.621772: Module Online

08/21/2014 10:13:20.50921: Deleting route to FEX

08/21/2014 10:13:20.58158: Module disconnected

08/21/2014 10:13:20.61591: Offlining Module

08/21/2014 10:13:20.62686: Module Offline Sequence

08/21/2014 10:13:20.797908: Module Offline

FEX: 102 Description: FEX0102 state: Online

FEX version: 5.1(3)N2(1c) [Switch version: 5.1(3)N2(1c)]

FEX Interim version: 5.1(3)N2(1c)

Switch Interim version: 5.1(3)N2(1c)

Extender Serial: SSI165204YC

Extender Model: N2K-C2232PP-10GE, Part No: 73-12533-05

Card Id: 82, Mac Addr: f0:29:29:ff:00:42, Num Macs: 64

Module Sw Gen: 12594 [Switch Sw Gen: 21]

post level: complete

pinning-mode: static Max-links: 1

Fabric port for control traffic: Eth1/11

FCoE Admin: false

FCoE Oper: true

FCoE FEX AA Configured: false

Fabric interface state:

Eth1/11 – Interface Up. State: Active

Fex Port State Fabric Port

Eth102/1/1 Up Eth1/11

Eth102/1/2 Down None

Eth102/1/3 Down None

Eth102/1/4 Down None

snippet removed

Logs:

08/21/2014 10:12:13.281018: Module register received

08/21/2014 10:12:13.283215: Registration response sent

08/21/2014 10:12:13.421037: Module Online Sequence

08/21/2014 10:12:16.665624: Module Online

Port Channels can be used instead of static pinning between parent switch and FEX so in the event of a fabric interface failure all hosts ports remain active. However, the remaining bandwidth on the parent switch will be shared by all the host ports resulting in an increase in oversubscription.

Part 3. Fabric Extender Topologies

Straight-Through: The FEX is connected to a single parent switch. The servers connecting to the FEX can leverage active-active data plane by using host vPC.

Shutting down the Peer link results in ALL vPC member ports in the secondary peer become disabled. For this reason it is better to use a Dual Homed.

Dual Homed: Connecting a single FEX to two parent switches.

In active – active, a single parent switch failure does not affect the host’s interfaces because both vpc peers have separate control planes and manage the FEX separately.
For the remainder of the post we are going to look at Dual Homed FEX connectivity with host Vpc.

Full configuration:

N5K1:
feature lacp
feature vpc
feature fex
!
vlan 10
!

vpc domain 1

peer-keepalive destination 192.168.0.52

interface port-channel1

switchport mode trunk

spanning-tree port type network

vpc peer-link

interface port-channel10

switchport access vlan 10

vpc 10

interface Ethernet1/1

switchport access vlan 10

spanning-tree port type edge

speed 1000

interface Ethernet1/3 – 5

switchport mode trunk

spanning-tree port type network

channel-group 1 mode active

interface Ethernet1/10

switchport mode fex-fabric

fex associate 101

interface Ethernet101/1/1

switchport access vlan 10

channel-group 10 mode on

N5K2:

feature lacp

feature vpc

feature fex

vlan 10

vpc domain 1

peer-keepalive destination 192.168.0.51

interface port-channel1

switchport mode trunk

spanning-tree port type network

vpc peer-link

interface port-channel10

switchport access vlan 10

vpc 10

interface Ethernet1/2

switchport access vlan 10

spanning-tree port type edge

speed 1000

interface Ethernet1/3 – 5

switchport mode trunk

spanning-tree port type network

channel-group 1 mode active

interface Ethernet1/11

switchport mode fex-fabric

fex associate 102

interface Ethernet102/1/1

switchport access vlan 10

channel-group 10 mode on

The FEX do not support LACP so configure the port-channel mode to “ON”

The first step is to check the VPC peer link and general VPC parameters.

Command: “show vpc brief“. Displays the vPC domain ID, the peer-link status, the keepalive message status, whether the configuration consistency is successful, and whether peer-link formed or the failure to form

Command: “show vpc peer-keepalive” Displays the destination IP of the peer keepalive message for the vPC. The command also displays the send and receives status as well as the last update from the peer in seconds and milliseconds

N5K3# sh vpc brief
Legend:
(*) – local vPC is down, forwarding via vPC peer-link
vPC domain id : 1
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive

Configuration consistency status: success

Per-vlan consistency status : success

Type-2 consistency status : success

vPC role : primary

Number of vPCs configured : 1

Peer Gateway : Disabled

Dual-active excluded VLANs : –

Graceful Consistency Check : Enabled

vPC Peer-link status

———————————————————————

id Port Status Active vlans

— —- —— ————————————————–

1 Po1 up 1,10

vPC status

—————————————————————————-

id Port Status Consistency Reason Active vlans

—— ———– —— ———– ————————– ———–

10 Po10 up success success 10

N5K3# show vpc peer-keepalive

vPC keep-alive status : peer is alive

–Peer is alive for : (1753) seconds, (536) msec

–Send status : Success

–Last send at : 2014.08.21 10:52:30 130 ms

–Sent on interface : mgmt0

–Receive status : Success

–Last receive at : 2014.08.21 10:52:29 925 ms

–Received on interface : mgmt0

–Last update from peer : (0) seconds, (485) msec

vPC Keep-alive parameters

–Destination : 192.168.0.54

–Keepalive interval : 1000 msec

–Keepalive timeout : 5 seconds

–Keepalive hold timeout : 3 seconds

–Keepalive vrf : management

–Keepalive udp port : 3200

–Keepalive tos : 192

The trunk interface should be forwarding on the peer link and VLAN 10 must be forwarding and active on the trunk link. Take note if any vlans are on err-disable mode on the trunk.

N5K3# sh interface trunk
——————————————————————————-
Port Native Status Port
Vlan Channel
——————————————————————————–
Eth1/3 1 trnk-bndl Po1
Eth1/4 1 trnk-bndl Po1
Eth1/5 1 trnk-bndl Po1

Po1 1 trunking —

——————————————————————————–

Port Vlans Allowed on Trunk

——————————————————————————–

Eth1/3 1-3967,4048-4093

Eth1/4 1-3967,4048-4093

Eth1/5 1-3967,4048-4093

Po1 1-3967,4048-4093

——————————————————————————–

Port Vlans Err-disabled on Trunk

——————————————————————————–

Eth1/3 none

Eth1/4 none

Eth1/5 none

Po1 none

——————————————————————————–

Port STP Forwarding

——————————————————————————–

Eth1/3 none

Eth1/4 none

Eth1/5 none

Po1 1,10

——————————————————————————–

Port Vlans in spanning tree forwarding state and not pruned

——————————————————————————–

Eth1/3 —

Eth1/4 —

Eth1/5 —

Po1 —

——————————————————————————–

Port Vlans Forwarding on FabricPath

——————————————————————————–

N5K3# sh spanning-tree vlan 10

VLAN0010

Spanning tree enabled protocol rstp

Root ID Priority 32778

Address 0005.9b1e.82fc

This bridge is the root

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32778 (priority 32768 sys-id-ext 10)

Address 0005.9b1e.82fc

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type

—————- —- — ——— ——– ——————————–

Po1 Desg FWD 1 128.4096 (vPC peer-link) Network P2p

Po10 Desg FWD 1 128.4105 (vPC) Edge P2p

Eth1/1 Desg FWD 4 128.129 Edge P2p

Check the Port Channel database and determine the status of the port channel

N5K3# show port-channel database
port-channel1
Last membership update is successful
3 ports in total, 3 ports up
First operational port is Ethernet1/3
Age of the port-channel is 0d:00h:13m:22s
Time since last bundle is 0d:00h:13m:18s
Last bundled member is Ethernet1/5

Ports: Ethernet1/3 [active ] [up] *

Ethernet1/4 [active ] [up]

Ethernet1/5 [active ] [up]

port-channel10

Last membership update is successful

1 ports in total, 1 ports up

First operational port is Ethernet101/1/1

Age of the port-channel is 0d:00h:13m:20s

Time since last bundle is 0d:00h:02m:42s

Last bundled member is Ethernet101/1/1

Time since last unbundle is 0d:00h:02m:46s

Last unbundled member is Ethernet101/1/1

Ports: Ethernet101/1/1 [on] [up] *

To execute reachability tests, create an SVI on the first parent switch and run ping tests. You must first enable the feature set “feature interface-vlan”. The reason we create an SVI in VLAN 10 is because we need an interfaces to source our pings.

N5K3# conf t
Enter configuration commands, one per line. End with CNTL/Z.
N5K3(config)# fea
feature feature-set
N5K3(config)# feature interface-vlan
N5K3(config)# int vlan 10
N5K3(config-if)# ip address 10.0.0.3 255.255.255.0

N5K3(config-if)# no shu

N5K3(config-if)#

N5K3(config-if)# end

N5K3# ping 10.0.0.3

PING 10.0.0.3 (10.0.0.3): 56 data bytes

64 bytes from 10.0.0.3: icmp_seq=0 ttl=255 time=0.776 ms

64 bytes from 10.0.0.3: icmp_seq=1 ttl=255 time=0.504 ms

64 bytes from 10.0.0.3: icmp_seq=2 ttl=255 time=0.471 ms

64 bytes from 10.0.0.3: icmp_seq=3 ttl=255 time=0.473 ms

64 bytes from 10.0.0.3: icmp_seq=4 ttl=255 time=0.467 ms

— 10.0.0.3 ping statistics —

5 packets transmitted, 5 packets received, 0.00% packet loss

round-trip min/avg/max = 0.467/0.538/0.776 ms

N5K3# ping 10.0.0.10

PING 10.0.0.10 (10.0.0.10): 56 data bytes

Request 0 timed out

64 bytes from 10.0.0.10: icmp_seq=1 ttl=127 time=1.874 ms

64 bytes from 10.0.0.10: icmp_seq=2 ttl=127 time=0.896 ms

64 bytes from 10.0.0.10: icmp_seq=3 ttl=127 time=1.023 ms

64 bytes from 10.0.0.10: icmp_seq=4 ttl=127 time=0.786 ms

— 10.0.0.10 ping statistics —

5 packets transmitted, 4 packets received, 20.00% packet loss

round-trip min/avg/max = 0.786/1.144/1.874 ms

N5K3#

Do the same tests on the second Nexus 5K.

N5K4(config)# int vlan 10
N5K4(config-if)# ip address 10.0.0.4 255.255.255.0
N5K4(config-if)# no shu
N5K4(config-if)# end
N5K4# ping 10.0.0.10
PING 10.0.0.10 (10.0.0.10): 56 data bytes
Request 0 timed out

64 bytes from 10.0.0.10: icmp_seq=1 ttl=127 time=1.49 ms

64 bytes from 10.0.0.10: icmp_seq=2 ttl=127 time=1.036 ms

64 bytes from 10.0.0.10: icmp_seq=3 ttl=127 time=0.904 ms

64 bytes from 10.0.0.10: icmp_seq=4 ttl=127 time=0.889 ms

— 10.0.0.10 ping statistics —

5 packets transmitted, 4 packets received, 20.00% packet loss

round-trip min/avg/max = 0.889/1.079/1.49 ms

N5K4# ping 10.0.0.13

PING 10.0.0.13 (10.0.0.13): 56 data bytes

Request 0 timed out

Request 1 timed out

Request 2 timed out

Request 3 timed out

Request 4 timed out

— 10.0.0.13 ping statistics —

5 packets transmitted, 0 packets received, 100.00% packet loss

N5K4# ping 10.0.0.3

PING 10.0.0.3 (10.0.0.3): 56 data bytes

Request 0 timed out

64 bytes from 10.0.0.3: icmp_seq=1 ttl=254 time=1.647 ms

64 bytes from 10.0.0.3: icmp_seq=2 ttl=254 time=1.298 ms

64 bytes from 10.0.0.3: icmp_seq=3 ttl=254 time=1.332 ms

64 bytes from 10.0.0.3: icmp_seq=4 ttl=254 time=1.24 ms

— 10.0.0.3 ping statistics —

5 packets transmitted, 4 packets received, 20.00% packet loss

round-trip min/avg/max = 1.24/1.379/1.647 ms

Shut down one of the FEX links to the parent and you see that the FEX is still reachable via the other link that is in the port channel bundle.

N5K3# conf t
Enter configuration commands, one per line. End with CNTL/Z.
N5K3(config)# int Eth101/1/1
N5K3(config-if)# shu
N5K3(config-if)# end
N5K3#
N5K3#

N5K3# ping 10.0.0.3

PING 10.0.0.3 (10.0.0.3): 56 data bytes

64 bytes from 10.0.0.3: icmp_seq=0 ttl=255 time=0.659 ms

64 bytes from 10.0.0.3: icmp_seq=1 ttl=255 time=0.515 ms

64 bytes from 10.0.0.3: icmp_seq=2 ttl=255 time=0.471 ms

64 bytes from 10.0.0.3: icmp_seq=3 ttl=255 time=0.466 ms

64 bytes from 10.0.0.3: icmp_seq=4 ttl=255 time=0.465 ms

— 10.0.0.3 ping statistics —

5 packets transmitted, 5 packets received, 0.00% packet loss

round-trip min/avg/max = 0.465/0.515/0.659 ms

If you would like to futher your knowlege on VPC and how it relates to Data Center toplogies and more specifically, Cisco’s Application Centric Infrastructure (ACI), you can check out my following training courses on Cisco ACI. Course 1: Design and Architect Cisco ACI, Course 2: Implement Cisco ACI, and Course 3: Troubleshooting Cisco ACI,

Data Center Topologies

August 14, 2014

by Matt Conran Blog

Network Architecture / Data Center Design

Data Center Topology: From Three-Tier Networks to Leaf-Spine Fabrics

Data center topology defines how switches, servers, storage systems and external networks connect—and ultimately how application traffic moves through the infrastructure. The architecture you choose influences latency, available bandwidth, resilience, operational complexity and the ability to scale as workloads grow.

Traditional hierarchical networks remain relevant in many environments, but modern cloud platforms and distributed applications increasingly rely on scalable Layer 3 fabrics, leaf-spine architectures and equal-cost multipath routing. Understanding these designs means looking beyond the network diagram to examine routing behaviour, traffic patterns, failure domains and the control plane that keeps the fabric connected.

Design principle: build around application requirements

A good topology is not simply the one with the most links or the most switches. It is the one that delivers the required capacity, predictable forwarding, appropriate redundancy and manageable failure behaviour at an acceptable cost.

01 / STRUCTURE

Topology and Fabric Design

Compare three-tier, centralized, top-of-rack, end-of-row and leaf-spine architectures.

02 / ROUTING

BGP and ECMP

Explore routing adjacencies, route selection, multipath forwarding and convergence during failures.

03 / OVERLAYS

VXLAN and EVPN

Understand how Layer 3 underlays support virtual networks and distributed endpoint reachability.

04 / OPERATIONS

Capacity and Resilience

Evaluate oversubscription, traffic distribution, observability and failure-domain behaviour.

In this guide: Work from fundamental topology choices to modern data center fabrics, BGP-based routing, traffic engineering considerations and practical troubleshooting.

NETWORK INSIGHT · FIELD GUIDE

Explore Data Center Topology

Navigate the architecture, routing, traffic flows and resilience principles behind modern data center networks.

06 sections

01 / ARCHITECTURE FUNDAMENTALS

Understanding Data Center Architectures

Data center topology defines how servers, storage systems, switches and external networks connect. The architecture determines how traffic moves through the infrastructure and influences available bandwidth, latency, redundancy, scalability and operational complexity.

This section introduces traditional three-tier networks, centralized switching, top-of-rack (ToR), end-of-row (EoR) and leaf-spine architectures. Compare their connectivity models and trade-offs to understand which designs best match different workload and infrastructure requirements.

Engineering focus Choose topology according to traffic patterns, growth expectations, capacity, failure domains and the cost of operating the network.

Section 01 / Architecture Fundamentals

The Main Data Center Architectures

Data center networks can be built using hierarchical switching layers, centralized switching platforms, distributed access switches or fabric-based designs. Each approach creates a different balance between cabling, port utilization, bandwidth, failure isolation and operational complexity.

The right starting point is the workload. A smaller environment with limited server-to-server traffic may not need the same architecture as a cloud platform supporting thousands of virtual machines, containers or data-intensive applications. Topology selection should reflect traffic patterns, growth expectations, availability targets and the cost of operating the network over time.

Common architectural approaches

Hierarchical

Traditional Three-Tier Architecture

A traditional enterprise design separates access, aggregation or distribution, and core functions. Access switches connect servers, the aggregation layer concentrates traffic and may enforce policy, and the core provides high-speed connectivity between network blocks.

This model remains useful where existing infrastructure, established operational practices or clear policy boundaries make a hierarchical design appropriate. However, multiple switching tiers can increase path length and create bottlenecks when east-west traffic grows.

Centralized

Centralized Switching

In a centralized design, servers or server racks connect back to a central switching location. Concentrating switching resources can simplify management and make port allocation efficient in smaller environments.

As the facility grows, cable lengths, pathway congestion, centralized capacity limits and the consequences of equipment failure must be evaluated carefully.

Distributed Access

Top-of-Rack and End-of-Row

Top-of-rack (ToR) places access switches in individual server racks. End-of-row (EoR) places switching equipment in a dedicated rack serving a row of servers. Both approaches can provide redundant connectivity when designed with suitable independent paths and devices.

ToR generally shortens server cabling and can isolate access failures to a rack. EoR can consolidate switching resources, but may require more structured cabling. Neither approach automatically determines whether the wider network uses a three-tier or leaf-spine architecture.

Fabric-Based

Leaf-Spine Architecture

A leaf-spine fabric connects each leaf switch to multiple spine switches. Servers and other endpoints connect to the leaf layer, while the spine layer provides paths between different leaves. A Layer 3 underlay commonly uses equal-cost multipath routing to distribute eligible traffic across available paths.

This architecture provides predictable path structure and supports incremental expansion. Its capacity and resilience depend on link speeds, the number of spine switches, oversubscription, routing design and failure behaviour.

Important distinction: ToR and EoR describe where access switching is placed. Three-tier and leaf-spine describe broader network architectures. A data center can use ToR switches as its leaf layer in a leaf-spine fabric.

Compare the design trade-offs

Design consideration	Three-tier / centralized	Leaf-spine fabric
Traffic paths	May traverse several hierarchical layers.	Typically uses a consistent leaf-to-spine-to-leaf path.
Scaling	Expansion depends on available tier capacity and design.	Can expand by adding leafs and, when needed, spine capacity.
Multipath routing	Depends on the routing and switching design.	Layer 3 ECMP is commonly used between leafs and spines.
Operational considerations	May suit established environments and existing investments.	Requires deliberate underlay, routing, capacity and failure planning.

Before selecting a topology

Traffic patterns

Determine whether workloads primarily communicate with external networks or generate substantial east-west traffic between servers.

Capacity and growth

Estimate endpoint growth, uplink bandwidth, oversubscription limits and the cost of adding future capacity.

Failure domains

Identify the impact of losing a switch, link, rack or entire network block, including the capacity that remains.

Operations and cost

Balance equipment and cabling costs against monitoring, troubleshooting, automation and lifecycle requirements.

NETWORK INSIGHT · INTERACTIVE ARCHITECTURE LAB

Explore Data Center Architectures

Compare four common ways to connect compute, access and switching. Select an architecture, then click a device in the topology to inspect its role in the network.

Three-tier architecture Illustrative topology

Selectable device Network connection Highlighted path

Selected device

Core switch

The core provides high-speed connectivity between distribution blocks and upstream networks in a traditional three-tier design.

Architecture insight

Separate layers make roles easy to understand, but traffic may traverse multiple tiers and scaling can increase operational complexity.

Layered Design structure

Multi-hop Typical path shape

Layer by layer Common scaling approach

Design strengths

Clear separation of network roles
Familiar operational model
Can suit hierarchical traffic patterns

Design considerations

Potentially longer paths between servers
Capacity planning across tiers
Additional layers to operate and troubleshoot

These diagrams illustrate common design patterns, not vendor-specific configurations. Actual forwarding paths, redundancy and oversubscription depend on the implementation. ToR means Top of Rack; EoR means End of Row; Clos describes a multistage network fabric.

02 / FABRIC ARCHITECTURE

Leaf-Spine and Clos Fabrics

Leaf-spine architecture connects each leaf switch to multiple spine switches, creating a consistent path structure between different server racks. It is widely used in modern data centers where distributed applications, virtualization and cloud workloads generate substantial east-west traffic.

Explore how leaf and spine layers work together, how two-tier fabrics relate to Clos network designs, and how adding switches can expand connectivity. The section also examines oversubscription, available path diversity and the limits of relying on topology alone to guarantee performance.

Engineering focus Understand fabric stages, leaf-to-spine connectivity, expansion options and the capacity that remains when a device or link fails.

Section 02 / Fabric Architecture

Leaf-Spine and Clos Fabrics

Leaf-spine architecture is a common foundation for modern data center networks. Instead of sending traffic through a succession of hierarchical switching layers, it connects access-layer leaf switches to a set of spine switches. This creates a fabric with consistent paths between leaf switches and provides a repeatable way to expand connectivity as workloads grow.

The architecture is particularly useful for environments with substantial east-west traffic, including virtualized platforms, container infrastructure, distributed storage and cloud-native applications. Its performance depends on the actual link speeds, number of paths, traffic distribution and capacity provisioned throughout the fabric.

The physical fabric

Spine layer

Spine 1

Spine 2

Spine 3

Every leaf connects to every spine

Leaf layer

Leaf 1

Leaf 2

Leaf 3

Leaf 4

Server and workload connectivity

Servers

Storage

Compute

Services

Conceptual topology, not a physical cabling diagram. Each leaf has links to each spine; server-facing connections are shown separately. The simplified graphic illustrates connectivity rather than every individual link.

1. The role of leaf switches

Leaf switches provide connectivity for servers, storage, firewalls, load balancers and other data center endpoints. They may also perform Layer 3 routing, enforce network policy and act as VXLAN tunnel endpoints when an overlay is deployed.

In a Layer 3 fabric, endpoints in different subnets can communicate through routed paths. When VXLAN EVPN is used, the leaf layer can additionally provide overlay connectivity and distribute endpoint reachability through the EVPN control plane.

2. The role of spine switches

Spine switches interconnect the leaf layer and provide transit paths between leaf switches. In a conventional two-tier leaf-spine fabric, spine switches do not connect directly to servers, and leaf switches do not connect directly to other leaf switches.

Adding spine capacity can increase the number of available paths and aggregate fabric bandwidth, provided the leaf uplinks, interfaces, routing configuration and downstream capacity are designed to use it. A spine failure can reduce available capacity even when connectivity survives.

Why path predictability matters: Traffic between endpoints attached to different leaf switches normally traverses one spine switch. This creates a consistent switching-hop structure, although actual latency still depends on link speeds, device processing, congestion and the forwarding path.

How the architecture scales

01 / ADD LEAF CAPACITY

Expand endpoint connectivity

Add leaf switches with uplinks to the fabric's spine layer. Confirm that the existing spine and uplink capacity can support the additional traffic.

02 / ADD SPINE CAPACITY

Increase fabric bandwidth

Additional spine switches can create more equal-cost paths. Validate routing limits, optics, cabling, interface capacity and platform support before expanding.

03 / MANAGE OVERSUBSCRIPTION

Plan for real traffic demand

Compare aggregate server-facing bandwidth with available uplink bandwidth. Size the fabric according to workload behaviour and acceptable contention.

04 / CONTROL FAILURE IMPACT

Test the remaining capacity

Evaluate whether surviving spines and links can carry traffic after a failure, not simply whether alternate paths exist.

Two-tier and multi-tier Clos designs

Leaf-spine is a practical implementation of a two-tier Clos-style fabric. Larger environments may introduce a super-spine or additional fabric stages to interconnect multiple pods, rooms or data halls. These designs extend the architecture while requiring careful consideration of path diversity, capacity and failure boundaries.

Design	Typical purpose	Key consideration
Two-tier leaf-spine	Connect endpoints across a single fabric.	Spine capacity, leaf uplinks and ECMP behaviour.
Multi-pod fabric	Connect modular groups of racks or infrastructure.	Inter-pod connectivity and failure isolation.
Super-spine / multi-stage Clos	Scale connectivity across larger fabrics or multiple data halls.	Additional stages, path design and end-to-end capacity.

NETWORK INSIGHT · FABRIC EXPLORER

Explore Leaf-Spine and Clos Fabrics

Investigate how leaf and spine switches connect, trace an illustrative server-to-server path, simulate a spine-link failure, and see why uplink capacity and oversubscription matter.

Explore:

Two-tier leaf-spine fabric All illustrated links up

Network device Available link Illustrative selected path Failed link

Topology focus

Leaf switches

Leaf switches connect servers and provide uplinks into the spine layer. In a conventional two-tier fabric, each leaf connects to every spine.

Fabric principle

A full-mesh leaf-to-spine design provides multiple potential paths between different leaves. Routing and ECMP determine which next hops are installed and used.

Trace a flow

Source leaf

Destination leaf

2 Illustrated spine paths between selected leaves

8 Leaf-to-spine links currently available

Healthy Illustrative fabric state

Explore leaf uplink oversubscription 3:1 example

1:1 · equal nominal capacity 5:1 · more aggregate server capacity

A 3:1 illustrative oversubscription ratio means aggregate nominal server-facing bandwidth is three times the uplink bandwidth. Actual throughput depends on traffic patterns, link rates and congestion.

Why use a Clos-style fabric?

A multistage design can provide scalable connectivity and multiple paths without requiring every leaf to connect directly to every other leaf.

What should engineers verify?

Check spine and leaf capacity, uplink speed, ECMP behavior, failure convergence and the bandwidth remaining after a link or device failure.

This is an educational model, not a live network or packet-level simulator. The four-leaf, two-spine diagram is illustrative. ECMP distributes eligible flows across available next hops; it does not guarantee that each physical link carries exactly the same load.

03 / TRAFFIC ENGINEERING

Data Center Traffic Patterns

A data center network must accommodate the traffic generated by applications, storage systems, virtual machines and distributed services. Understanding where traffic originates and where it travels is essential for designing a fabric with sufficient capacity and predictable performance.

This section distinguishes east-west traffic between internal workloads from north-south traffic entering or leaving the data center. It explores how flow size, concurrent demand, ECMP hashing, uplink capacity and congestion influence the actual behaviour of a leaf-spine network.

Engineering focus Size the fabric around real workload demand, identify potential bottlenecks and validate performance during normal operation and failure conditions.

Section 03 / Traffic Engineering

Data Center Traffic Patterns

A data center topology must be designed around the traffic it carries, not simply the number of switches and links it contains. Application placement, storage access, replication, service dependencies and external connectivity all influence where traffic flows and where congestion can occur.

Leaf-spine fabrics are particularly suited to distributed east-west communication, but north-south traffic, large data transfers and uneven workload demand can still create bottlenecks. Understanding these patterns helps engineers size uplinks, evaluate redundancy and choose appropriate routing and forwarding designs.

Two fundamental traffic directions

01 / EAST-WEST

Workload-to-workload traffic

Traffic between servers, virtual machines, containers, storage systems and distributed application components. In modern environments, this can represent a substantial share of total network demand.

02 / NORTH-SOUTH

Traffic entering or leaving the fabric

Traffic between internal workloads and external networks, users, WANs, internet services or other data centers. Its path may include border leaf switches, firewalls, routers and load balancers.

Example / east-west traffic path

Communication between two server racks

Server A
Rack 1

Leaf 1

Spine

Leaf 2

Server B
Rack 2

Simplified logical path. In a conventional two-tier leaf-spine fabric, inter-leaf traffic normally traverses a spine. The forwarding plane selects an available path according to the installed routes and forwarding configuration.

Why east-west traffic affects fabric design

Distributed applications frequently exchange data across multiple servers. Examples include database replication, distributed storage, service-to-service requests, cluster coordination and virtual machine communication. If these flows cross leaf switches, the spine layer and leaf uplinks must provide enough capacity for their combined demand.

Traffic demand is rarely perfectly uniform. Some flows are short and latency-sensitive, while others transfer large volumes of data. Multiple simultaneous flows may also compete for the same links. Consequently, aggregate bandwidth alone does not reveal every congestion risk.

Important distinction: A fabric can have several equal-cost paths and still experience congestion. ECMP provides a mechanism for distributing eligible flows across paths; it does not guarantee that every flow uses a different link or that traffic is perfectly balanced.

North-south traffic and border connectivity

External traffic commonly enters or leaves through designated border leaf switches or dedicated edge devices. These connect the fabric to WAN routers, internet edge infrastructure, firewalls, load balancers and other networks.

The design must consider both bandwidth and resilience. Concentrating external connectivity on a small number of devices may simplify policy enforcement, but it can also create capacity constraints or larger failure domains. Distributing border functions can improve scale, provided routing policy and security controls remain consistent.

Traffic patterns and engineering decisions

Traffic pattern	Potential pressure point	Design response
Server-to-server	Leaf uplinks and spine capacity.	Assess aggregate demand, path diversity and ECMP.
Storage replication	Sustained high-throughput flows.	Measure peak demand and validate link utilization.
Internet or WAN access	Border links, security appliances and external routing.	Plan edge capacity, policy and redundant paths.
Large data transfers	Flow imbalance and transient congestion.	Evaluate hashing behaviour, flow sizes and telemetry.
Failure or maintenance	Reduced capacity on surviving paths.	Validate traffic behaviour after losing a link or device.

Measure the traffic, not just the topology

Traffic engineering should use interface counters, utilization trends, flow telemetry, packet drops, latency measurements and application performance data. These observations help distinguish a routing issue from a capacity problem or an application-generated traffic burst.

Engineers should also examine behaviour during failure conditions. A fabric that operates comfortably during normal traffic may become congested when a spine or uplink is unavailable and flows are redistributed across fewer remaining paths.

NETWORK INSIGHT · TRAFFIC EXPLORER

Explore Data Center Traffic Patterns

Follow traffic between servers, across the fabric and toward external networks. Adjust the illustrative load to understand why workload patterns, path capacity and congestion matter when designing a data center.

East-west: server to server Illustrative traffic flow

Network link East-west flow North-south flow Potential congestion

Traffic focus

East-west traffic

Traffic between workloads inside the data center. In a leaf-spine fabric, communication between servers on different leaves normally traverses a spine.

Engineering insight

Distributed applications, storage replication and service-to-service calls can generate substantial east-west traffic. Design for workload demand rather than assuming all traffic exits the data center.

Example flow

East-West Selected traffic direction

Leaf → Spine → Leaf Illustrative forwarding path

Moderate Illustrative link demand

Simulate traffic demand 45% illustrative load

Low demand High demand

At this illustrative load, the selected links have moderate demand. Real congestion depends on offered traffic, available link capacity, buffering, queueing and the distribution of flows.

Why traffic direction matters

East-west traffic can stay inside the fabric and use leaf uplinks and spine capacity. North-south traffic crosses the boundary toward external networks or services.

What engineers should measure

Compare interface utilization, drops, queue depth, flow records and application latency. A high average utilization can hide short bursts that cause packet loss or delay.

This widget is a conceptual teaching model, not a live traffic generator or telemetry system. The load slider represents illustrative demand, not measured utilization. Actual paths and capacity depend on topology, routing, link rates, hashing and workload behavior.

04 / ROUTING CONTROL PLANE

BGP in Data Center Fabrics

Border Gateway Protocol (BGP) provides a scalable way to distribute network reachability information across modern data center fabrics. In a routed leaf-spine design, BGP can establish the routes that allow traffic to move between endpoints while supporting controlled route propagation and multipath forwarding.

Explore eBGP and iBGP design approaches, route advertisements, best-path selection, ECMP and convergence. The section also introduces the relationship between a routed underlay and a VXLAN EVPN overlay, showing why control-plane state must be checked alongside the actual forwarding table.

Engineering focus Trace route propagation, validate next-hop reachability, understand multipath installation and troubleshoot routing changes without confusing BGP state with end-to-end connectivity.

Section 04 / Routing Control Plane

BGP in Data Center Fabrics

Border Gateway Protocol (BGP) is widely used to exchange reachability information in modern data center networks. In a routed leaf-spine fabric, it can provide scalable route distribution, controlled reachability, multipath forwarding and predictable responses to topology changes.

BGP does not replace the physical fabric. It supplies control-plane information that routers use to decide which network destinations are reachable and which paths can be installed for forwarding. The resulting behaviour depends on the routing design, policy, platform capabilities and operational configuration.

Keep the layers distinct: the physical topology defines available links; the routing control plane distributes reachability; the forwarding plane moves packets using installed forwarding entries. A healthy BGP session alone does not prove that the intended traffic path is working.

01 / Choosing a BGP fabric design

DESIGN A

eBGP leaf-spine

Leaf and spine switches use different autonomous system numbers (ASNs), and BGP sessions between them operate as external BGP. This is a common design for routed Clos fabrics because it can keep routing relationships explicit and support repeatable deployment patterns.

DESIGN B

iBGP-based fabric

Routers exchange BGP routes within the same autonomous system. Larger designs may use route reflectors to distribute routes without requiring a full mesh of iBGP sessions between every participating router.

DESIGN C

Hybrid routing

A fabric may combine an IGP or static routing for underlay reachability with BGP for other routing functions. The exact division depends on the design; there is no requirement that every data center use one universal routing model.

DESIGN D

Border and external routing

Border leaf switches or edge routers exchange routes with external networks. Import and export policies determine which prefixes can enter or leave the fabric and how they are represented to other routing domains.

Example: eBGP in a leaf-spine fabric

In this simplified example, each leaf has an eBGP session to each spine. Leaf-to-spine routing sessions exchange the reachability needed to forward traffic across the fabric. The example illustrates the relationship rather than prescribing a specific ASN plan or configuration.

Illustrative two-tier routed fabric

SPINE LAYER

Spine 1 AS 65101

Spine 2 AS 65102

eBGP sessions between each leaf and both spines

LEAF LAYER

Leaf 1 AS 65001

Leaf 2 AS 65002

Conceptual relationship only; individual physical links and separate BGP sessions are not drawn. ASNs are illustrative. This design assumes the chosen BGP configuration permits the required route exchange between the relevant devices.

Design caveat: eBGP does not automatically guarantee a working fabric or multipath forwarding. ASN allocation, next-hop reachability, route policy, prefix limits, supported BGP features and ECMP configuration must all be validated.

02 / Underlay and overlay routing

The BGP underlay

The underlay is the routed IP infrastructure that connects the switches and provides IP reachability across the fabric. In a BGP-routed underlay, BGP can advertise the infrastructure prefixes needed to reach loopbacks and other routed destinations. These routes provide the foundation for control-plane sessions and overlay tunnel endpoints where applicable.

The VXLAN EVPN overlay

In a VXLAN EVPN design, VXLAN encapsulates tenant traffic across the IP fabric, while EVPN uses BGP to distribute overlay reachability information. This can include information about MAC addresses, IP addresses and tenant segments, depending on the EVPN route type and deployment.

Component	Primary responsibility	Example question
Underlay routing	IP reachability between routed fabric endpoints.	Can this VTEP reach the remote VTEP address?
BGP EVPN control plane	Advertises overlay reachability and related information.	Has the expected MAC/IP or prefix route been learned?
VXLAN data plane	Encapsulates tenant packets across the IP underlay.	Can the encapsulated packet traverse the fabric?
Forwarding and policy	Applies installed forwarding entries and applicable policies.	Does traffic reach the correct destination and tenant?

These functions are related but not interchangeable. A healthy underlay does not prove that EVPN routes are correct, and receiving an EVPN route does not by itself prove that end-to-end forwarding succeeds.

03 / How BGP selects routes

From route advertisement to forwarding

Receive: a BGP neighbor advertises a prefix and its associated path attributes. The receiving router evaluates the update against session state, import policy and protocol rules.

Validate: the route must be acceptable under policy and protocol requirements. Its next hop must also be resolvable for the route to be usable for forwarding.

Select: the router evaluates eligible paths using its BGP decision process and implementation-specific policy. Attributes such as local preference, AS path, origin and MED can influence the outcome; their exact effect depends on the decision context and configuration.

Install: the selected route or eligible set of multipaths may be installed into the routing table and forwarding hardware, subject to platform limits, configuration and interactions with other routing sources.

Forward: packets are sent using the installed forwarding entries. The actual path depends on the forwarding implementation, ECMP eligibility, hashing and current topology.

Best path is not the same as every installed path. BGP implementations can select a best path while also installing multiple eligible paths for multipath forwarding. Multipath rules and supported attributes vary by platform and configuration.

Why BGP policy matters

BGP exchanges reachability, but policy determines which routes are accepted, preferred, modified or advertised onward. In a data center, an incorrect policy can suppress a required prefix, permit an unintended route or cause traffic to prefer an unexpected exit.

Engineers should explicitly document intended import and export behaviour, allowed prefixes, ASN expectations, next-hop handling and any route attributes used to influence path selection. Reusable policy templates can reduce configuration drift across large fabrics.

04 / ECMP and BGP multipath

How equal-cost paths support fabric capacity

Equal-Cost Multipath (ECMP) allows a router to forward traffic across multiple eligible next hops for a destination. BGP can supply candidate paths, but the routing implementation and configuration determine whether multiple paths are considered equivalent and installed.

Once multiple next hops are installed, forwarding hardware commonly uses a hash of packet or flow fields to select a path. This helps distribute traffic while typically preserving packet ordering within a flow. It does not mean every individual packet is sent down a different path, nor does it guarantee perfectly even utilization.

Mechanism	Role	Common misconception
BGP best-path selection	Chooses a preferred route according to policy and decision rules.	It does not necessarily install every received path.
BGP multipath	Allows multiple eligible BGP paths to be installed.	It is not automatic on every platform or configuration.
ECMP forwarding	Distributes eligible flows across equal-cost next hops.	It does not guarantee perfect load balance.
LACP	Negotiates and manages a Layer 2 link aggregation group.	It is not the same mechanism as Layer 3 ECMP.

ECMP and LACP can both use hashing in forwarding or distribution decisions, but they operate at different levels. ECMP selects among Layer 3 next hops; LACP negotiates membership in a Layer 2 aggregated link. A design may use one or both, depending on its physical and logical architecture.

05 / Convergence and failure handling

What happens when a path fails?

When a link or device fails, the fabric must detect the event, update the relevant routing state and adjust forwarding. BGP may withdraw affected routes or advertise revised reachability. The forwarding plane then changes its installed entries as the control plane and hardware respond.

Convergence time depends on failure detection, session behaviour, routing policy, protocol timers, implementation, hardware programming and the scope of the change. BFD may help detect some failures faster when supported and correctly configured, but it is not a universal guarantee of a particular recovery time.

FAILURE 01

Single link loss

If alternate paths remain available and the routing design recognizes the failure, traffic may be redirected through surviving paths. Check both route changes and actual forwarding.

FAILURE 02

Spine or peer loss

Routes or next hops associated with the failed device may be removed. Surviving paths carry the remaining demand, potentially increasing utilization and exposing oversubscription.

FAILURE 03

Policy or route withdrawal

A prefix may disappear even though physical links and BGP sessions remain up. Investigate advertisements, filters, next-hop resolution and the routing table.

FAILURE 04

Session instability

Repeated BGP resets can cause route churn and traffic disruption. Check interface errors, reachability, timers, logs and control-plane resource conditions.

Resilience must be tested under load. Redundant links are useful only if surviving paths have sufficient capacity and forwarding converges as intended. Test representative link and device failures, and measure packet loss, recovery time and post-failure utilization.

06 / Practical BGP troubleshooting

Work from session state to packet forwarding

When a prefix is missing or traffic takes an unexpected path, avoid starting with a broad configuration change. Follow the route through the control plane and forwarding plane, checking each stage before moving to the next.

Verify the session. Confirm the intended BGP neighbor is Established, and inspect logs, reset reasons, interface status and reachability between peer addresses.

Check received routes. Confirm the expected prefix is being advertised by the peer and received locally. Use the platform's received-route or neighbor-route commands where supported.

Inspect policy. Review prefix lists, route maps, import/export policies, community handling, maximum-prefix controls and any policy that could reject or modify the route.

Inspect best path and next hop. Identify why a candidate route was selected or rejected. Confirm that the selected next hop is resolvable and reachable.

Verify multipath installation. If several paths are expected, check the routing table for multiple installed next hops and confirm that multipath settings and platform capabilities permit them.

Validate forwarding. Inspect the forwarding table, interface counters, flow telemetry and packet path. For VXLAN EVPN, also verify underlay reachability, relevant EVPN routes, VTEP reachability and tenant forwarding state.

Example diagnostic commands

These are illustrative Cisco IOS/IOS XE-style commands. Exact syntax and available detail vary by operating system and platform; use the equivalent commands for your switches.

show ip bgp summary
show ip bgp
show ip route
show ip cef
show running-config | section router bgp

Start with the neighbor summary, then inspect the target prefix and its candidate paths. Check the routing table to determine which route was selected, and inspect the forwarding table to verify the next hop used for packet forwarding. Review the BGP configuration and policies when the observed result differs from the intended design.

The relevant BGP neighbor is Established.
The expected prefix is received and passes import policy.
The selected route has a valid, reachable next hop.
Expected ECMP next hops are installed when multipath is intended.
Export policy allows the required route to reach downstream peers.
Forwarding state and traffic telemetry agree with the expected path.
Failure tests confirm acceptable convergence and surviving capacity.

Continue with the BGP Learning Centre

Explore route propagation, best-path selection and fault investigation through the Network Insight BGP Learning Centre. It complements this architecture section with a practical, interactive way to examine BGP behaviour.

Open the BGP Learning Centre: Explore. Trace. Diagnose. →

NETWORK INSIGHT · BGP FABRIC EXPLORER

Explore BGP in a Leaf-Spine Fabric

Select a router, inspect the illustrative BGP sessions, trace how a prefix is learned, and simulate a path failure. See how routing information and forwarding paths work together in a data center network.

Illustrative BGP fabric Sessions established

iBGP eBGP Failed path

Selected element

Spine 1

A spine switch provides fabric transit. Depending on the design, it may establish eBGP sessions with leaf switches and exchange underlay reachability.

Engineering insight

BGP distributes reachability information. The forwarding plane uses the installed routes to move packets; a BGP session being established does not alone prove that an application path is working.

Interactive controls

6Illustrative BGP sessions

2Illustrative available paths

EstablishedSelected scenario state

Follow a route advertisement 10.10.10.0/24

1Prefix learnedA peer advertises reachability to the prefix.

2BGP evaluatesAttributes and policy influence route selection.

3Route installedAn eligible route is installed for forwarding.

4Packet forwardedThe forwarding table directs traffic to a next hop.

Underlay reachability

The underlay provides IP reachability between fabric devices and their loopbacks or next hops. eBGP is often used between leaf and spine switches in routed fabrics, though designs vary.

Route selection and forwarding

BGP applies policy and path-selection rules to choose eligible routes. ECMP may install multiple equal-cost next hops when supported by the routing design and platform.

Educational model only. This topology and its route progression are illustrative, not a live BGP implementation. Real session counts, ASNs, route attributes, ECMP eligibility, convergence and forwarding behavior depend on the configured network and platform.

05 / Observe

Network Visibility, Traffic Intelligence and BGP

You cannot reliably protect a network if you cannot establish what is happening within it. Flow records, packet captures, encrypted-traffic metadata and security logs reveal different aspects of network behaviour and provide evidence for troubleshooting and threat detection.

This section connects traffic analysis with BGP route visibility, helping engineers investigate unexpected communication patterns, routing changes and reachability problems using evidence from multiple layers.

05 / Observe

Network Visibility: From Traffic Flows to BGP Intelligence

Security controls determine what should be allowed. Observability helps engineers establish what is actually happening. By correlating flow records, packet-level evidence, encrypted traffic metadata and routing information, network teams can identify suspicious behaviour, investigate connectivity failures and understand how an incident moves across the network.

A firewall log may show that a connection was permitted, but it rarely explains the entire conversation. A packet capture may reveal a protocol exchange, yet only represent a short interval or one observation point. Flow telemetry can expose communication patterns across a longer period, while routing telemetry can explain which paths were available and why traffic may have taken an unexpected route.

Engineering principle: No single telemetry source provides the complete picture. Strong network investigations correlate evidence from multiple layers rather than treating an alert, a flow record or a routing change as conclusive proof on its own.

5.1 — Choose the right observation method

Flow telemetry

NetFlow and IPFIX

Summarise conversations using fields such as source and destination addresses, ports, protocol, packet counts, byte counts and timestamps. IPFIX provides an extensible framework for exporting structured flow information.

Sampling

sFlow

Uses statistical sampling of packets and interface counters to provide scalable visibility into traffic patterns, particularly in large switched and routed networks. Sampling can miss individual events.

Packet evidence

SPAN and packet capture

SPAN mirrors selected switch traffic to an observation port. Packet captures can reveal protocol headers, retransmissions, handshakes and other details that summaries cannot show. Mirror oversubscription can cause packet loss.

Encrypted traffic

Metadata and behaviour

Connection timing, volume, direction, destination reputation, DNS context where available, TLS handshake metadata and endpoint telemetry can help detect anomalies without assuming that encrypted payloads are readable.

Method	Best used for	Important limitation
NetFlow / IPFIX	Traffic accounting, unusual communication patterns and historical investigation	Typically lacks full packet payload and detailed packet-by-packet sequence
sFlow	High-volume traffic trends and scalable network sampling	Sampling may omit short-lived or low-volume activity
SPAN / packet capture	Protocol troubleshooting and detailed session analysis	Limited observation point, capture duration and possible mirror-port loss
Endpoint and security logs	Process attribution, identity, host events and security outcomes	Coverage and reliability depend on instrumentation and log integrity
BGP and routing telemetry	Route changes, path selection, reachability and control-plane anomalies	A valid route does not prove that the application or forwarding path works

5.2 — Build an evidence chain

Consider a server that suddenly transfers much more data than usual. A single bandwidth graph might identify the spike, but it cannot establish whether the cause is a legitimate backup, a compromised service or an unexpected change in traffic routing. A structured investigation combines several sources.

DetectBaseline deviation

→

CorrelateFlows and identities

→

InspectPackets and logs

→

ValidateRoutes and forwarding

First, establish when the anomaly began. Next, identify the communicating endpoints, destination ports, volume and duration. Compare the behaviour against the normal baseline. Then inspect available DNS, endpoint, firewall and packet evidence. Finally, check whether a route change, next-hop failure or policy update coincided with the event.

Interactive investigation explorer

Select an observation to see what it can establish and what should be checked next.

INVESTIGATION VIEW Flow records: establish who communicated

Compare source and destination addresses, ports, protocol, bytes, packets and timestamps. Identify unusual destinations, repeated connections or a sudden change in traffic volume. Then correlate the flow with endpoint and security logs.

5.3 — Encrypted traffic: visibility without payload inspection

Encryption protects confidentiality, but it does not make every aspect of a connection invisible. Depending on the protocol, configuration and observation point, engineers may still have access to connection duration, byte counts, packet sizes, timing, DNS information, destination addresses and some handshake metadata.

Behavioural baselines

Detect changes in connection frequency, data volume, session duration and communication direction compared with a service's established pattern.

Identity correlation

Combine network observations with endpoint, workload and authentication records to associate a connection with a device, user or application where the evidence supports it.

Protocol-aware analysis

Use available TLS, DNS and transport metadata carefully. Encrypted Client Hello and encrypted DNS can limit visibility into some fields, while QUIC changes what traditional passive inspection can observe.

Important distinction: A large encrypted transfer to an unfamiliar destination is an investigation lead, not proof of data exfiltration. Validate the destination, expected application behaviour, endpoint context and any applicable organisational policy before drawing conclusions.

5.4 — BGP visibility: understand the route behind the traffic

Border Gateway Protocol (BGP) exchanges reachability information between autonomous systems and distributes routes within an autonomous system through iBGP. Its control plane influences the paths that IP packets may follow. For network security and troubleshooting, this means traffic anomalies should sometimes be investigated alongside BGP updates, routing-table changes, next-hop resolution and forwarding state.

Routing intelligence

Three autonomous systems, one destination prefix

Illustrative topology for correlating route propagation with observed traffic.

AS65002 External BGP peer

AS65001 Internal BGP and routing policy

AS65003 External BGP peer

Example destination: 10.10.10.0/24

# Illustrative route observation Prefix: 10.10.10.0/24 Protocol: BGP AS_PATH: 65002 64496 NEXT_HOP: 192.0.2.2 Observation: Route present in BGP table Validation: Confirm best path, next-hop reachability and FIB

The values above are illustrative. They do not describe a live network or establish that this route is selected, reachable or safe.

What should engineers observe?

UPDATE activity: identify announcements and withdrawals, including the prefix affected and the time of the change.
AS_PATH changes: investigate unexpected path changes, unusual origins and route attributes that may affect selection.
Best-path selection: compare competing routes, local preference, AS_PATH length and other attributes in the context of the vendor's decision process and policy.
Next-hop resolution: verify that the selected next hop is reachable through the relevant routing information.
Forwarding state: confirm that the selected route is programmed into the forwarding information base (FIB) and that traffic can actually traverse the path.
Session health: correlate BGP neighbour state transitions, resets, timers and logs with the time the traffic anomaly occurred.

Security note: BGP route leaks and route hijacks can affect reachability or redirect traffic. Monitoring is only one defence. Apply appropriate prefix filtering, route-policy controls, peer validation and, where supported and operationally suitable, RPKI-based Route Origin Validation. ROV validates route-origin authorisation; it does not by itself validate the entire AS_PATH or prevent every routing attack.

5.5 — BGP and traffic telemetry must be correlated

A route appearing in the BGP table is not the same as a successful application connection. BGP provides control-plane reachability information, while the forwarding plane handles packets. An investigation must distinguish a route that has been received from one that has been selected, installed and successfully used.

Observation	Possible interpretation	Next validation step
Traffic volume drops suddenly	Application failure, link problem, filtering or a route change	Correlate interface counters, flow records, route events and endpoint health
AS_PATH changes unexpectedly	Legitimate policy change, upstream change or suspicious announcement	Check the source peer, prefix origin, route policy and trusted routing evidence
BGP route exists, but traffic fails	Unresolved next hop, FIB inconsistency, ACL, MTU issue or downstream failure	Check route selection, next-hop recursion, FIB, interface state and packet path
Repeated BGP session resets	Transport instability, peer configuration, resource pressure or timer expiry	Inspect neighbour logs, TCP connectivity, interface errors and control-plane load
Unexpected outbound traffic	New workload behaviour, misconfiguration or potentially malicious activity	Correlate destination, workload identity, DNS, security events and route history

5.6 — Operational requirements for reliable visibility

Observability systems are only as useful as the quality and consistency of their data. Time synchronisation is essential when correlating events across routers, firewalls, switches, hosts and collectors. Retention policies should preserve enough history to establish a baseline and investigate delayed reports. Telemetry collectors should be monitored for gaps, overload and export failures.

Synchronise timestamps across network devices, endpoints and telemetry platforms.
Document flow sampling rates, SPAN limitations and which interfaces are actually observed.
Monitor telemetry loss, collector health, storage capacity and data-retention windows.
Restrict access to sensitive flow records, packet captures and routing logs.
Protect management and telemetry channels against unauthorised access or tampering.
Establish normal traffic and routing baselines before treating deviations as incidents.

Section takeaway: Effective visibility connects traffic behaviour, security events and routing decisions. Flow records show communication patterns; packet captures provide deeper protocol evidence; encrypted-traffic metadata supports behavioural analysis; and BGP telemetry helps explain changes in network reachability. Correlation turns those separate signals into a defensible engineering investigation.

NETWORK INSIGHT · VISIBILITY WORKBENCH

Investigate the Data Center with Network Telemetry

Explore how flow records, interface counters, packet captures and BGP telemetry reveal different parts of a network problem. Select a diagnostic view to correlate traffic behavior with routing and forwarding state.

Traffic and telemetry sources Illustrative telemetry

Network connection Selected diagnostic path Warning signal Potential fault

Selected source

Flow telemetry

Flow records summarize conversations between endpoints. They help identify major talkers, traffic direction and volume, but do not normally contain every packet's full contents.

Engineering insight

No single telemetry source explains every fault. Correlating traffic records, interface counters, packet evidence and routing state can narrow the cause much faster.

Diagnostic actions

Flow recordsSelected evidence source

Traffic patternWhat it can reveal

PartialEvidence coverage

Illustrative diagnostic evidence

Signal	Example observation	Interpretation	Status
Flow records	Large app-to-database flow	Identify traffic source and volume	Useful
Interface counters	Output drops increasing	Investigate congestion or queue pressure	Check
Packet capture	Retransmissions observed	Correlate loss, delay and endpoint behavior	Check
BGP telemetry	Next-hop route changed	Verify route selection and forwarding state	Useful

What the data tells you

Flow telemetry helps locate large or unexpected conversations. It is valuable for traffic engineering and capacity planning, but its detail depends on the exporter, sampling and collection interval.

Correlate before concluding

Compare timestamps and affected paths across telemetry sources. A counter increase, a packet symptom and a routing change are stronger evidence when they align in time and relate to the same flow.

Educational demonstration only. The evidence values are illustrative examples, not live measurements. Available telemetry, sampling, counter definitions, packet visibility and BGP monitoring capabilities vary by platform and deployment.

06 / Respond

Incident Response, Threat Detection and Network Forensics

When suspicious activity is detected, the priority shifts from observation to controlled investigation and response. Engineers must validate the evidence, determine the scope of the incident, preserve relevant records and contain the threat without creating unnecessary operational disruption.

Explore the incident response lifecycle, practical forensic techniques, containment decisions and BGP incident investigation, from the first alert through recovery and lessons learned.

06 / Respond

Incident Response, Threat Detection and Network Forensics

Detecting suspicious activity is only the beginning. Effective incident response requires engineers to validate the alert, establish the scope of the incident, preserve evidence, contain the threat and restore services safely. Network telemetry, endpoint records, security controls and routing information provide complementary evidence that helps teams understand what happened and determine the appropriate response.

A security alert does not automatically establish that a system has been compromised. A sudden increase in outbound traffic, an unfamiliar destination, repeated authentication failures or an unexpected BGP route change may indicate a security incident, a configuration error or legitimate operational activity. The objective is to investigate systematically rather than act on an assumption.

Core principle: Incident response must balance containment, evidence preservation, service availability and business risk. Every action should have a clear purpose, an authorised scope and a way to verify its outcome.

6.1 — The incident response lifecycle

A practical response process moves from preparation through detection, analysis, containment, eradication and recovery. Lessons learned then feed back into monitoring, configuration standards and security controls. In a serious incident, these activities may overlap, and responders may need to revisit earlier stages as new evidence emerges.

PrepareTools, roles and playbooks

→

DetectAlert and validate

→

ContainLimit further impact

→

EradicateRemove root cause

→

RecoverRestore and monitor

Phase 1

Preparation

Define incident ownership, escalation paths, approved containment actions, logging requirements, backup procedures and communication channels before an incident occurs.

Phase 2

Detection and analysis

Validate the alert, establish a timeline, identify affected assets and correlate evidence from network, endpoint, identity and security systems.

Phase 3

Containment

Reduce the opportunity for further harm using proportionate, authorised controls. Consider dependencies and the operational consequences of isolating a device or blocking a communication path.

Phase 4

Eradication and recovery

Remove the confirmed cause, remediate affected systems, restore trusted configurations and verify that services are operating correctly before returning to normal operations.

6.2 — Detect and validate suspicious network activity

Detection systems compare observed activity against rules, threat intelligence, behavioural baselines and known indicators. The resulting alerts must be evaluated in context. An IP address associated with a threat feed, for example, is a useful investigation lead but may be outdated, shared by multiple services or unrelated to the activity under examination.

Signal	Potential concern	Validation approach
Unusual outbound data volume	Possible data transfer or unexpected application behaviour	Check flow records, destination, workload identity, scheduled jobs and endpoint events.
Repeated authentication failures	Possible credential guessing or account misuse	Correlate identity logs, source addresses, successful logins and account activity.
Repeated DNS queries	Possible misconfiguration, automated activity or suspicious domain generation	Inspect query patterns, resolver logs, domain context and the originating endpoint.
Unexpected east-west traffic	Possible lateral movement or a newly introduced dependency	Compare with workload communication baselines, segmentation policy and process telemetry.
Unexpected BGP announcement	Possible route leak, misconfiguration or unauthorised route propagation	Verify the peer, prefix, origin, policy, route history and expected maintenance activity.
Repeated connection resets	Possible service instability, filtering or hostile connection activity	Correlate packet captures, firewall decisions, server logs and interface counters.

6.3 — Interactive incident investigation

Select a scenario to explore the evidence an engineer should collect before choosing a response. These are illustrative investigation paths, not automated diagnoses.

Investigation workbench

Select a scenario to reveal its evidence requirements and recommended next steps.

INVESTIGATION SCENARIO Unusual outbound data transfer

Establish which host initiated the transfer, the destination, data volume, timing and expected application behaviour. Correlate flow records with endpoint activity, DNS context and available security logs before deciding whether containment is necessary.

6.4 — Network forensics and evidence preservation

Network forensics reconstructs relevant events from available evidence. Depending on the environment, that evidence may include packet captures, NetFlow or IPFIX records, DNS logs, firewall events, authentication records, endpoint telemetry, configuration changes and BGP update history.

Packet captures

Preserve the relevant capture interval and observation point. Record any capture filters, truncation settings, packet loss indicators and limitations affecting interpretation.

Centralised logs

Preserve original records where possible, document collection methods and correlate events using consistent timestamps and asset identifiers.

Configuration history

Compare approved and current firewall rules, ACLs, routing policies, interface configuration and relevant changes made before or during the incident.

Evidence handling: Follow organisational policy and applicable legal requirements. Restrict access to sensitive records, preserve provenance, document collection and handling, and use cryptographic hashes where appropriate to help demonstrate that collected files have not changed.

# Example: calculate a SHA-256 hash of a collected evidence file # Run on an authorised system with access to the evidence file. shasum -a 256 incident-capture.pcap # Record the hash, filename, collection time, collector and storage location. # A hash supports integrity verification; it does not prove the original # capture was complete or that its contents are accurate.

6.5 — Containment: reduce risk without creating another outage

Containment aims to limit the impact of an incident while preserving enough operational capability to investigate and recover. The right action depends on confidence in the evidence, the criticality of the affected service, the likely threat and the available alternatives.

Containment option	Potential benefit	Risk to consider
Isolate an endpoint	Limits some network communications from a suspected host	May interrupt critical services or remove access needed for investigation.
Apply a targeted firewall rule	Restricts a verified malicious flow or exposure	An overly broad rule may block legitimate dependencies.
Disable compromised credentials	Reduces the opportunity for further unauthorised access	Can interrupt automated workloads or legitimate administrative activity.
Change routing policy	May mitigate an established routing threat or isolate an affected path	Incorrect changes can cause route withdrawal, traffic blackholing or wider reachability loss.
Increase targeted monitoring	Improves evidence collection when immediate isolation is not justified	Monitoring alone may be insufficient when active harm is continuing.

Operational safeguard: High-impact containment actions should follow the incident response plan and appropriate authorisation. In a critical network, coordinate changes with service owners, assess failover behaviour and maintain a documented rollback plan wherever feasible.

6.6 — BGP incidents: distinguish control-plane anomalies from forwarding failures

BGP-related incidents require particular care because an incorrect response can affect reachability well beyond a single router. Unexpected route advertisements, route leaks, session instability and policy errors may alter the paths available to traffic. However, a route change alone does not establish malicious intent.

# Illustrative BGP incident triage checklist 1. Identify the affected prefix and first observed change. 2. Determine which peer advertised or withdrew the route. 3. Compare AS_PATH, origin and relevant route attributes. 4. Verify import/export policy and expected maintenance changes. 5. Check the selected route and next-hop resolution. 6. Confirm the FIB and test the affected forwarding path. 7. Apply an authorised mitigation and verify the result.

In an interdomain routing incident, useful safeguards may include carefully maintained prefix filters, explicit import and export policies, maximum-prefix controls, appropriate peer authentication and RPKI Route Origin Validation where supported. ROV helps validate whether an origin AS is authorised to originate a prefix; it does not validate every aspect of the route or prevent every possible BGP attack.

Within an autonomous system, engineers should also check iBGP propagation, route-reflector behaviour, next-hop reachability, IGP dependencies and forwarding state. A route may be visible in one routing table but absent from another, or selected by BGP without being usable for forwarding.

6.7 — Recovery, verification and lessons learned

Recovery is more than removing a block or restarting a service. Before returning affected systems to normal operation, teams should verify that the original cause has been addressed, required security controls are working and relevant traffic is following expected paths.

Confirm the affected systems, accounts, prefixes or services have been identified.
Remove the confirmed cause and remediate the relevant configuration or vulnerability.
Restore systems from trusted sources where required and verify their security posture.
Validate routing, next-hop reachability, forwarding state and application connectivity.
Maintain heightened monitoring for recurrence and unexpected changes.
Document the timeline, decisions, evidence, impact, recovery steps and outstanding risks.
Update detection rules, runbooks, access controls and training based on lessons learned.

6.8 — Measure incident response effectiveness

Response metrics help organisations identify gaps and improve operational readiness. They should be interpreted alongside incident severity, service impact and investigation quality rather than treated as isolated performance targets.

Detection

Mean time to detect

Measures the elapsed time between a defined incident starting point and detection. The starting point must be consistently defined for comparisons to be meaningful.

Response

Mean time to contain

Measures how long it takes to implement an effective containment measure after the relevant response clock starts.

Quality

Recurrence and false positives

Review recurring incident causes, unnecessary escalations, detection coverage and whether mitigations actually reduce risk without unacceptable disruption.

Section takeaway: Effective incident response joins detection, evidence preservation, containment, remediation and recovery into a controlled engineering process. Network telemetry and BGP visibility provide important context, but reliable decisions depend on corroborating evidence, understanding operational dependencies and verifying that corrective actions work.

NETWORK INSIGHT · RESILIENCE LAB

What Happens When a Data Center Fabric Fails?

Simulate illustrative link and spine failures in a leaf-spine fabric. Follow the surviving paths, inspect the remaining connectivity and identify what engineers should verify before declaring the fabric resilient.

Leaf-spine fabric connectivity Normal operation

Available connection Traffic path Failed component or link Illustrative alternate path

Scenario analysis

Normal operation

Both spine switches are available. Each leaf has two uplinks, allowing traffic to use multiple fabric paths when routing and forwarding are configured accordingly.

Engineering lesson

Redundancy is only useful when alternate paths are operational, routing converges correctly and the remaining links can carry the offered load.

Test controls

8 / 8Available leaf-to-spine links

2 / 2Available spine switches

2Illustrative paths between selected leaves

Failure investigation checklist

Check	Question	Why it matters	Status
Physical state	Are the links and devices operational?	Confirms the failure boundary	Baseline
Routing	Have paths been withdrawn or recalculated?	Control-plane state influences path selection	Baseline
Forwarding	Are alternate next hops installed?	Confirms traffic has a usable path	Baseline
Capacity	Can surviving links carry the load?	Prevents congestion after failover	Baseline

Resilience is more than redundancy

Multiple links and switches create alternate paths, but they do not guarantee uninterrupted service. Detection time, routing convergence, forwarding updates, hashing and remaining capacity all affect the outcome.

Validate the recovery

After a failure, verify reachability and application behavior, inspect routing and forwarding state, compare interface counters and confirm that the surviving fabric is not overloaded.

Educational model only. This is a simplified two-spine, four-leaf fabric. The diagram illustrates connectivity and possible alternate paths; it does not calculate live routing convergence, ECMP hashing, throughput, packet loss or application availability. Actual behavior depends on topology, routing policy, hardware and configuration.

Network Insight · Architecture Briefing

Designing Data Center Networks for Scale, Visibility and Resilience

Data center topology is more than a diagram of switches and links. It determines how workloads communicate, how capacity scales, how routing adapts to change and how engineers isolate faults when the network behaves unexpectedly.

Six ideas to take away

From architectural foundations to operational troubleshooting, each design decision affects the next.

Choose the right architecture

Three-tier designs separate access, aggregation and core functions. Leaf-spine fabrics provide a more consistent path structure for many modern, east-west-heavy workloads.

Understand the fabric

Leaf switches connect endpoints; spine switches interconnect the leaves. Clos-style designs can scale horizontally, subject to port count, link speed and oversubscription choices.

Design for real traffic

North-south and east-west flows create different demands. Capacity planning must consider traffic concentration, ECMP distribution, burstiness and workload communication patterns.

Make routing part of the design

BGP can provide scalable underlay routing in a leaf-spine fabric. Route policy, next-hop reachability, ECMP and convergence behavior all influence the forwarding outcome.

Make the network observable

Flow telemetry, interface counters, packet captures and routing telemetry reveal different parts of the story. Correlating them helps connect symptoms to causes.

Prove resilience under failure

Redundant links are only the beginning. Test failover, verify routing and forwarding state, check surviving capacity and confirm that affected applications recover.

A simplified fabric mental model

SPINE LAYER Path diversity

Leaf 1Leaf 2Leaf 3Leaf 4

ComputeStorageApplications

Think in layers, paths and failure domains.

A useful fabric design connects physical topology, routing behavior and application requirements. Each layer answers a different engineering question: what is connected, where can traffic travel and what happens when a component is lost?

✓Topology: Are there enough paths between the required endpoints?

✓Routing: Are the intended next hops available and installed?

✓Capacity: Can the remaining fabric carry the expected load?

✓Operations: Can telemetry identify the fault and verify recovery?

The engineering takeaway

A resilient data center is not simply one with redundant links. It is one whose behavior can be understood, measured and tested.

Build around the workload, understand the trade-offs in the chosen topology, validate routing and forwarding, and exercise failure scenarios before they occur in production. The goal is not just connectivity—it is predictable performance and recoverable service.

```

☕

Support Network Insight

Help support independent engineering guides, interactive networking tools, and future learning labs.

Support Network Insight ```