AI Networking Architecture Lab Guide

AI Networking Architecture: Ethernet, RoCEv2 and InfiniBand

Network Insight / Engineering Guide 11 / AI Infrastructure

AI Networking Architecture Explained: Ethernet, RoCEv2, InfiniBand and High-Performance GPU Fabrics

Artificial intelligence infrastructure introduces networking requirements that extend beyond traditional enterprise and data centre connectivity. Large GPU clusters must exchange enormous volumes of data during distributed training, synchronise model parameters, move tensors between accelerators and maintain high utilisation across thousands of parallel compute operations.

In these environments, network performance can directly influence the efficiency of expensive GPU resources. Congestion, packet loss, latency variation, inefficient collective communication and imbalanced fabric paths can cause accelerators to spend time waiting for data rather than performing useful computation.

AI GPU Clusters Ethernet RoCEv2 InfiniBand RDMA Leaf-Spine NCCL Congestion Control
DISTRIBUTED AI GPU CLUSTER Logical high-performance network architecture SCALE-OUT FABRIC Ethernet / RoCEv2 or InfiniBand GPU NODE A GPU NODE B GPU NODE C SCALE-UP COMMUNICATION NVLink / NVSwitch inside supported systems FRONT-END NETWORK Management · Storage · Services
Scale-out fabric GPU node connectivity High-speed data path Supporting services

Architecture interpretation: The diagram is conceptual. The fabric represents an Ethernet/RoCEv2 or InfiniBand deployment alternative, not a requirement to run both technologies on the same links. Scale-up interconnects operate within supported GPU systems, while scale-out fabrics connect separate compute nodes. Storage and management network placement varies by cluster design.

Why AI Networking Is Different

Traditional enterprise networks are often optimised for diverse client-server traffic, transaction processing, web applications and general-purpose connectivity.

AI training fabrics must additionally support communication patterns in which many GPUs exchange data at high rates and coordinate execution across distributed workloads.

01
Communication Intensity

Distributed training can create substantial east-west traffic between GPU nodes.

02
Low-Latency Operations

Collective communication can be sensitive to network delay and tail latency.

03
Congestion Sensitivity

Congestion and retransmission can reduce effective accelerator utilisation.

04
High Bandwidth

Large distributed workloads demand substantial aggregate network throughput.

05
Transport Engineering

RDMA and congestion control require careful NIC and fabric configuration.

06
Compute-Network Coupling

Communication stalls can leave GPU resources waiting for synchronisation.

Three Communication Domains in AI Infrastructure

Scale-Up Communication

Scale-up networking connects accelerators within a server or tightly coupled compute system. Technologies such as NVLink and NVSwitch enable high-bandwidth GPU-to-GPU communication on supported hardware.

Scale-Out Communication

Scale-out networking connects GPU servers across a data centre fabric. Ethernet with RoCEv2 and InfiniBand are important approaches for high-performance distributed communication.

Front-End and Supporting Connectivity

AI clusters also require management, orchestration, storage access, monitoring and application connectivity. These functions may use dedicated networks, shared fabrics or logically isolated network segments depending on design requirements.

Ethernet, RoCEv2 and InfiniBand: The Architectural Relationship

Technology Primary role Engineering consideration
Ethernet Layer 2 networking technology and switching fabric Topology, routing, QoS, congestion and operational tooling
RoCEv2 RDMA transport over UDP/IP on Ethernet networks NIC configuration, congestion control and packet-loss behaviour
InfiniBand Dedicated high-performance interconnect architecture Fabric management, routing, virtual lanes and adapter compatibility
NVLink / NVSwitch High-bandwidth GPU interconnect within supported scale-up systems Hardware topology, collective communication and locality

Important distinction: Ethernet and RoCEv2 are not competing technologies at the same architectural layer. RoCEv2 is an RDMA transport that uses Ethernet and IP networking. The more useful engineering comparison is often Ethernet with RoCEv2 versus InfiniBand for a particular scale-out GPU workload.

What Engineers Must Optimise

AI fabric design should be evaluated against the actual communication pattern of the workload, not only the advertised link speed of individual switches or network adapters.

Aggregate Throughput

How much useful data the cluster can exchange.

Tail Latency

How the slowest communication operations affect synchronisation.

Fabric Utilisation

How effectively available paths carry traffic.

Congestion Behaviour

How switches and endpoints react to pressure.

Failure Recovery

How the system responds to link, device and path failures.

GPU Efficiency

How much accelerator time is spent on useful computation.

Introduction — Engineering Perspective

AI networking is the engineering discipline that connects accelerator architecture, distributed communication, high-speed switching, RDMA transport, congestion control and production operations. The objective is not simply to build the fastest possible network, but to design a predictable, observable and resilient fabric that supports the communication requirements of large-scale AI workloads.

Network Insight / Engineering Guide 11 / Architecture Navigator

Explore the AI Networking Fabric

Follow the engineering journey from GPU cluster architecture and Ethernet fabrics to RoCEv2, InfiniBand, collective communication, performance observability and production troubleshooting. Select any section to jump directly to its detailed technical explanation.

Network Insight / AI Networking / Section 01

AI Networking Architecture Fundamentals: GPU Clusters, Scale-Up, Scale-Out and Distributed Communication

Modern AI infrastructure is built around accelerated compute systems that must communicate at extremely high bandwidth and with predictable latency. A distributed GPU cluster combines compute nodes, accelerator interconnects, high-performance network adapters, switching fabrics and supporting services into a coordinated computing platform.

The network is not simply a connectivity service between servers. For many distributed AI workloads, communication performance directly affects training efficiency, synchronisation time and overall accelerator utilisation.

1.1 — Layered AI GPU Cluster Architecture

An AI cluster can be understood as three connected architectural domains: the GPU scale-up interconnect, the scale-out high-performance fabric and the front-end supporting infrastructure.

DISTRIBUTED GPU CLUSTER SCALE-OUT NETWORK FABRIC Ethernet / RoCEv2 or InfiniBand GPU NODE A GPU NODE B GPU NODE C GPU 0 GPU 1 GPU 0 GPU 1 GPU 0 GPU 1 Scale-up Interconnect Scale-up Interconnect Scale-up Interconnect FRONT-END / STORAGE / MANAGEMENT SERVICES

Architecture interpretation: The GPU blocks represent accelerators within individual compute nodes. Scale-up communication operates inside supported systems, while scale-out communication crosses the network fabric between separate nodes. Actual clusters may contain many more GPUs per node and multiple network adapters per server.

1.2 — Core Components of an AI GPU Cluster

01
GPU Accelerators

Execute matrix operations and parallel AI workloads.

02
GPU Interconnect

Connect accelerators within supported scale-up systems.

03
Network Adapters

Provide high-speed communication between compute nodes.

04
Switching Fabric

Connect GPU servers across the cluster.

05
Storage Systems

Supply training data, checkpoints and model artifacts.

06
Cluster Orchestration

Schedule workloads and coordinate distributed execution.

1.3 — Scale-Up vs Scale-Out Networking

Characteristic Scale-Up Scale-Out
Communication scope Within a server or supported tightly coupled GPU system Between separate compute nodes
Typical technologies NVLink, NVSwitch and related accelerator interconnects Ethernet/RoCEv2 or InfiniBand
Primary objective High-bandwidth local accelerator communication High-performance distributed communication
Topology considerations GPU interconnect topology and locality Switching fabric, path diversity and oversubscription
Performance risks Local interconnect contention and topology mismatch Congestion, packet loss, imbalance and network failures

1.4 — Distributed Training Communication

Distributed training splits model computation across multiple accelerators or compute nodes. Communication is required to exchange gradients, model parameters, activations or other intermediate data depending on the parallelism strategy.

Data Parallelism

Workers process different data batches and synchronise relevant gradients.

Tensor Parallelism

Model tensor operations are partitioned across accelerators.

Pipeline Parallelism

Different model stages execute on separate accelerators or groups.

Engineering insight: The communication pattern depends on the parallelism strategy. A fabric optimised for one workload may perform differently under another. Network design should therefore consider the actual training framework and collective communication behaviour.

1.5 — Collective Communication Fundamentals

Collective operations coordinate data exchange between multiple accelerators. They are central to many distributed training workloads.

Collective Purpose Typical use
All-Reduce Combine values and distribute the result Gradient synchronisation
All-Gather Collect data from all participants Distributed tensor assembly
Reduce-Scatter Reduce data and distribute partitions Partitioned gradient processing
Broadcast Distribute data from one participant Parameter distribution
All-to-All Exchange different data partitions among participants Expert routing and distributed workloads

1.6 — GPU Efficiency and Network Communication Overhead

A GPU may complete a compute phase but remain unable to proceed until required communication has finished. This creates a relationship between network performance and useful accelerator execution time.

Illustrative training step Compute phase: 80 ms Communication phase: 20 ms ----------------------------- Total step duration: 100 ms Simplified compute fraction: 80 / 100 = 80% If communication increases to 40 ms: 80 / 120 = 66.7%

This simplified example assumes no overlap between computation and communication. Real distributed training systems may overlap these operations, and measured GPU utilisation depends on additional factors such as memory stalls, kernel scheduling and workload behaviour.

1.7 — Front-End and Back-End AI Network Separation

Back-End Compute Fabric

Carries latency-sensitive, high-bandwidth communication between GPU nodes. It may require specialised RDMA configuration, congestion management and detailed fabric telemetry.

Front-End Services Network

Supports workload orchestration, management access, application connectivity, monitoring and other supporting services. Storage traffic may use this network or a separate dedicated fabric.

1.8 — AI Cluster Network Design Considerations

Design area Engineering question
Bandwidth Can the fabric support expected collective traffic?
Topology Are paths balanced and sufficient for the workload?
Congestion How will the network respond to bursts and incast?
Reliability How will link or switch failures affect training jobs?
Observability Can engineers correlate GPU stalls with network behaviour?
Scalability Can additional compute nodes be added without unacceptable performance degradation?
Operations Are configuration, firmware and troubleshooting processes repeatable?
Section 1 — Engineering Takeaway

AI networking architecture combines GPU compute systems, scale-up interconnects, scale-out network fabrics, distributed communication, collective operations and supporting services. The network must be designed around the communication characteristics of the AI workload, because network congestion, latency and failures can directly affect distributed training efficiency and accelerator utilisation.

Network Insight · Engineering Lab 01 · AI Networking

AI GPU Network Architecture Explorer

Explore the relationship between GPU compute nodes, scale-up interconnects, scale-out fabrics and supporting networks. Change the cluster topology, communication pattern and network capacity to understand how distributed AI workloads interact with the underlying network architecture.

GPU cluster architecture configuration

32 Total GPUs
4 GPU servers
400G Per-server fabric
0 ms Idealised transfer time
Domain 01 · Scale-up GPU interconnect
Domain 02 · Scale-out AI network fabric
Fabric Switch A Ethernet / RoCEv2
Fabric Switch B Ethernet / RoCEv2
Domain 03 · Front-end and supporting networks
Storage Dataset / Checkpoints
Management Provisioning / Telemetry
Orchestration Scheduling / Control

Architecture engineering inspector

Scale-up vs scale-out

Scale-up communication connects GPUs within a tightly coupled system. Scale-out communication connects separate GPU servers through a network fabric.

Engineering check: identify which communication operations remain local and which cross the fabric.
Architecture diagnostics console

Engineering interpretation

Model assumptions: The transfer-time calculation is an idealised bandwidth-only estimate, not a measured collective completion time. It assumes each server can use the selected effective network bandwidth without contention or oversubscription. Actual results depend on collective algorithms, rank placement, GPU memory and interconnect performance, NIC topology, PCIe, congestion, protocol overhead and software implementation. Ethernet/TCP, RoCEv2 and InfiniBand are not assigned artificial performance multipliers in this model.
Network Insight / AI Networking / Section 02

Ethernet Fabric Design for AI Clusters: Leaf-Spine, Clos, ECMP, Bandwidth and Network Engineering

Ethernet has become an important foundation for large-scale AI infrastructure, providing a flexible switching architecture that can connect GPU servers through high-bandwidth, multi-path network fabrics.

For AI workloads, the engineering challenge extends beyond selecting high-speed Ethernet ports. The fabric must deliver sufficient aggregate capacity, balanced traffic distribution, predictable congestion behaviour and resilient connectivity between distributed compute nodes.

2.1 — Leaf-Spine Ethernet Fabric Architecture

A leaf-spine fabric uses a structured topology in which leaf switches connect GPU servers and spine switches provide multiple paths between leaves.

In a conventional two-tier Clos design, each leaf connects to every spine in its fabric group. This creates predictable path diversity and supports horizontal scaling through additional switching capacity.

AI ETHERNET LEAF-SPINE FABRIC SPINE 1 SPINE 2 High-Speed Ethernet High-Speed Ethernet LEAF 1 LEAF 2 LEAF 3 GPU Access GPU Access GPU Access GPU SERVER A GPU SERVER B GPU SERVER C

Architecture interpretation: The diagram shows a simplified two-tier Clos topology. Production AI fabrics may use additional tiers, multiple network interfaces per GPU server, separate rail architectures and more sophisticated failure-domain boundaries.

2.2 — Why Clos Fabrics Matter for AI Workloads

01
Path Diversity

Multiple fabric paths connect compute nodes across the cluster.

02
Horizontal Scaling

Capacity can increase through additional switching resources.

03
Predictable Structure

Regular topology simplifies design, automation and troubleshooting.

04
Resilience

Redundant links and switches reduce single failure dependencies.

05
High Aggregate Capacity

Parallel uplinks support substantial east-west traffic.

06
Operational Consistency

Repeatable designs support standardised configuration and telemetry.

2.3 — High-Speed Ethernet: 200G, 400G and 800G

High-performance AI fabrics may use 200GbE, 400GbE or 800GbE interfaces, depending on adapter capabilities, switch platforms, optical connectivity and deployment requirements.

Interface speed Nominal line rate Engineering consideration
200GbE 200 Gbit/s Adapter availability, port density and cost
400GbE 400 Gbit/s High-bandwidth GPU node and fabric connectivity
800GbE 800 Gbit/s Platform support, optics, cabling and power

Capacity distinction: Advertised interface speed is not the same as application-level throughput. Ethernet, IP, transport and RDMA overhead, congestion behaviour and workload characteristics affect effective data transfer rates.

2.4 — Oversubscription and Fabric Capacity Planning

Oversubscription describes the ratio between aggregate downlink capacity and aggregate uplink capacity at a network tier.

Example leaf switch capacity GPU-facing ports: 8 × 400 Gbit/s = 3.2 Tbit/s Spine-facing ports: 8 × 400 Gbit/s = 3.2 Tbit/s Oversubscription ratio: 3.2 / 3.2 = 1:1 Result: Nominally non-oversubscribed at the leaf uplink boundary

A 1:1 ratio means that nominal aggregate uplink bandwidth matches nominal downlink bandwidth. It does not guarantee congestion-free operation because traffic distribution, bursts, endpoint behaviour and other fabric bottlenecks still matter.

Design recommendation: For communication-intensive GPU training workloads, evaluate oversubscription against the expected collective traffic pattern rather than assuming traditional enterprise ratios are appropriate.

2.5 — Equal-Cost Multipath Routing

ECMP allows a routed fabric to install multiple equal-cost next hops toward the same destination. Traffic can then be distributed across available fabric paths.

Flow-Based Distribution

Many implementations select an ECMP path using a hash of packet header fields. Packets belonging to the same flow commonly follow the same path, helping limit reordering.

AI Traffic Considerations

Large RDMA flows may create uneven path utilisation when hash distribution is imperfect. Some platforms support additional load-balancing mechanisms, subject to hardware and transport compatibility.

ECMP limitation: ECMP does not automatically divide a single large flow evenly across every available path. Effective distribution depends on flow diversity, hash inputs and the specific switch implementation.

2.6 — Routing Architecture for AI Ethernet Fabrics

Routed leaf-spine fabrics commonly use Layer 3 connectivity between switching tiers. Routing protocols provide topology reachability and support multiple paths through the network.

Design component Engineering purpose
Layer 3 underlay Provide routed connectivity between fabric devices
ECMP Install multiple equal-cost next hops
BGP or supported IGP Exchange network reachability
Loopback addressing Provide stable router identities and management endpoints
Routing convergence Restore reachability after topology changes
Telemetry Monitor interface, routing and forwarding state

2.7 — Failure Domains and Resilience

AI training jobs may span many compute nodes, making the network failure domain an important architectural consideration.

Link Failure

Traffic may move to alternate paths if routing and available capacity permit.

Leaf Failure

Connected GPU nodes may lose fabric connectivity without redundant attachments.

Spine Failure

Remaining spines may continue forwarding with reduced aggregate capacity.

NIC Failure

Endpoint redundancy depends on the server and fabric design.

Optical Failure

Transceiver and cabling faults can produce errors or link loss.

Control-Plane Failure

Routing instability can affect reachability and path selection.

2.8 — Ethernet Fabric Engineering Checklist

Area Validation question
Topology Does the Clos design provide the required path diversity?
Capacity Can uplinks support the expected aggregate workload?
Routing Are all expected ECMP paths installed?
Load balancing Are traffic flows distributed acceptably?
Physical layer Are optics, cabling and interface errors within acceptable limits?
Resilience Does the design tolerate required failure scenarios?
Telemetry Can engineers identify congestion and path imbalance?
Automation Are configurations repeatable and auditable?
Section 2 — Engineering Takeaway

Ethernet AI fabric engineering combines leaf-spine and Clos topology design, high-speed Ethernet interfaces, ECMP routing, capacity planning, oversubscription analysis, failure-domain design and production telemetry. The objective is to provide a scalable and predictable network foundation for distributed GPU communication.

Network Insight · Engineering Lab 02 · AI Networking

AI Ethernet Fabric Design Workbench

Design a simplified leaf-spine Ethernet fabric for GPU clusters. Configure rack count, leaf and spine switches, link speeds and GPU network demand. Trace ECMP paths, inject a spine failure and investigate oversubscription and available uplink capacity.

Ethernet fabric design configuration

32 GPU servers
1600G Leaf uplink capacity
2:1 Oversubscription
4 Available ECMP paths
Spine Leaf GPU rack Failed link

Fabric engineering inspector

Leaf-Spine Architecture

Each leaf connects to each spine, creating multiple equal-cost paths between GPU racks.

Engineering check: compare rack downlink demand against available leaf uplink bandwidth.
Fabric diagnostics console

Fabric engineering assessment

Engineering assumptions: This model uses a simplified two-tier Clos topology: one leaf per rack and one equal-speed link from each leaf to each spine. Oversubscription compares nominal server-facing bandwidth with available uplink capacity for a leaf. ECMP path count assumes equal-cost routing and one eligible path per operational spine. Real implementations depend on routing, ECMP hashing, port breakout, NIC bonding, traffic patterns, switch ASIC capacity, buffering, congestion control and physical cabling. No actual packets are transmitted.
Network Insight / AI Networking / Section 03

RDMA and RoCEv2 Packet Architecture: Queue Pairs, Memory Registration, Encapsulation and GPU Data Transfer

Remote Direct Memory Access, or RDMA, enables network adapters to transfer data directly between registered memory regions with less involvement from the remote CPU than conventional socket-based data movement.

RoCEv2, or RDMA over Converged Ethernet version 2, carries RDMA transport traffic across UDP/IP networks using Ethernet infrastructure. It is particularly important in high-performance AI environments where distributed GPU workloads exchange large volumes of data.

3.1 — End-to-End RoCEv2 Architecture

A RoCEv2 transfer involves application software, registered memory, RDMA queue pairs, network adapters, the IP/Ethernet fabric and a receiving endpoint.

RoCEv2 END-TO-END DATA PATH SOURCE GPU NODE DESTINATION GPU NODE REGISTERED MEMORY Host / supported GPU buffer REGISTERED MEMORY Remote destination buffer QUEUE PAIR Send / Receive Queues QUEUE PAIR RDMA Transport State RDMA NIC RDMA NIC ETHERNET FABRIC IP / UDP / RoCEv2 Routed or switched paths RDMA DATA MOVEMENT Memory registration · Work requests · Packet transport · Completion

Engineering distinction: RDMA reduces CPU involvement in data movement, but it does not remove all software processing or eliminate the need for transport reliability. Applications still configure resources, register memory, post work requests and process completion information.

3.2 — RDMA Operations

01
RDMA Write

A requester places data into an authorised remote memory region.

02
RDMA Read

A requester retrieves data from an authorised remote memory region.

03
Send / Receive

A message is delivered to a receive buffer posted by the peer.

3.3 — Queue Pair Architecture

A queue pair typically contains a Send Queue and Receive Queue. Applications submit work requests through the RDMA verbs interface, while the network adapter processes transport operations.

Component Function
Send Queue (SQ) Holds submitted send-side work requests
Receive Queue (RQ) Holds receive buffers for applicable incoming operations
Completion Queue (CQ) Reports completion information for signalled operations and receives
Queue Pair Number (QPN) Identifies a queue pair within the RDMA transport context
Protection Domain (PD) Associates and isolates compatible RDMA resources
Memory Region (MR) Defines registered memory and access permissions

3.4 — Memory Registration and Access Keys

Before RDMA operations can access a memory buffer, the relevant memory must be registered with the RDMA subsystem. Registration establishes the translation and access information required by the network adapter.

Local Key — lkey

Used by the local adapter to validate access to registered memory for supported local operations.

Remote Key — rkey

Authorises permitted remote RDMA operations against a registered memory region when the transport and access permissions allow.

Security consideration: RDMA access permissions and remote keys are part of memory protection, but they are not a substitute for endpoint authentication, network isolation and broader infrastructure security controls.

3.5 — RoCEv2 Packet Encapsulation

RoCEv2 carries RDMA transport headers and payloads inside UDP/IP packets. The well-known RoCEv2 UDP destination port is 4791.

Illustrative RoCEv2 packet +--------------------------------------+ | Ethernet Header | | Destination MAC / Source MAC / VLAN | +--------------------------------------+ | IPv4 or IPv6 Header | | Source IP / Destination IP | | DSCP / ECN | +--------------------------------------+ | UDP Header | | Destination Port: 4791 | +--------------------------------------+ | RDMA Transport Headers | | Base Transport Header (BTH) | | Optional operation-specific headers | +--------------------------------------+ | RDMA Payload / Padding / ICRC | +--------------------------------------+ | Ethernet FCS | +--------------------------------------+

Packet engineering: The precise RDMA header sequence depends on the operation and transport service. Some packets include additional headers such as the RDMA Extended Transport Header (RETH). The invariant CRC (ICRC) protects the applicable RDMA packet contents independently of the Ethernet FCS.

3.6 — Reliable Connected Transport and Packet Sequence Numbers

Reliable Connected (RC) queue pairs provide transport mechanisms for reliable, ordered delivery of supported RDMA operations.

Packet Sequence Numbers (PSNs) help detect missing, duplicated or out-of-order transport packets. Acknowledgements, negative acknowledgements and retry behaviour support recovery from certain delivery failures.

Transport concept Engineering purpose
PSN Track packet ordering and transport progress
ACK Confirm successful transport progress
NAK Signal selected error or recovery conditions
Retry Recover from supported loss or timeout conditions
Completion Report work request status to the application

Important: RoCEv2 does not guarantee that packets will never be dropped. Loss and congestion can still occur, and transport recovery may introduce latency or cause operations to fail when retry limits are exhausted.

3.7 — RoCEv2 and GPUDirect RDMA

In supported GPU platforms, GPUDirect RDMA can allow compatible third-party devices, including network adapters, to access GPU memory without requiring the application to stage every transfer through a conventional host-memory copy.

This can reduce unnecessary data movement and CPU overhead, but practical support depends on GPU hardware, network adapters, PCIe topology, drivers, firmware and software configuration.

GPU Memory

Holds tensors and communication buffers used by AI workloads.

RDMA Adapter

Transfers data using supported registered memory resources.

System Topology

PCIe placement and device compatibility affect achievable performance.

3.8 — RoCEv2 Troubleshooting and Packet Analysis

Symptom Potential investigation area
Queue pair fails to connect Endpoint addressing, QP state and connection parameters
RDMA access error Memory registration, permissions, address and remote key
Unexpected transport retries Packet loss, congestion, MTU and fabric errors
Low effective throughput Link speed, congestion, PCIe topology and application behaviour
High tail latency Queueing, congestion control and competing traffic
Intermittent communication stalls NIC counters, transport timeouts and collective communication traces
Illustrative investigation workflow 1. Verify physical link health 2. Confirm IP reachability and MTU 3. Validate RoCEv2 traffic classification 4. Inspect RDMA device and QP state 5. Review NIC retry/error counters 6. Inspect ECN/PFC/congestion telemetry 7. Correlate failures with application logs 8. Test a controlled RDMA workload

Useful Linux diagnostic utilities may include rdma, ibv_devinfo, ibv_devices and vendor-specific NIC telemetry tools. Availability and output vary by operating system, drivers and adapter implementation.

Section 3 — Engineering Takeaway

RoCEv2 combines RDMA memory semantics, queue pairs, registered buffers, transport reliability and UDP/IP encapsulation to support high-performance data movement over Ethernet networks. Successful deployment requires coordinated engineering across the application, GPU, RDMA adapter, transport configuration and switching fabric.

Network Insight · Engineering Lab 03 · AI Networking

RoCEv2 Packet Journey Visualiser

Follow an RDMA operation through queue-pair preparation, memory registration, work request submission, RoCEv2 encapsulation, Ethernet forwarding, remote memory access and completion handling. Inspect packet headers and investigate transport failures.

RDMA transaction configuration

0 / 7 Transaction stage
RESET QP state
0 Packet sequence
Pending Completion status
GPU Server A
GPU
Registered source memory
Ethernet Fabric RoCEv2 / UDP/IP
GPU Server B
GPU
Registered destination memory
RoCEv2 packet encapsulation

Protocol engineering inspector

RoCEv2 Encapsulation

RoCEv2 transports InfiniBand transport packets inside UDP/IP over Ethernet.

Select a packet header to inspect its engineering role.
RDMA diagnostics console

Transaction engineering findings

Protocol scope: This widget models a simplified single-message Reliable Connected RoCEv2 transaction. Actual RDMA implementations may segment messages into multiple packets and use additional transport headers. RDMA Write and Read have different request/response and completion semantics. Send/Receive requires receive-side work requests. A local completion does not necessarily mean the remote application has processed the data. The visualisation is educational and does not implement a real RDMA stack.
Network Insight / AI Networking / Section 04

Lossless Ethernet and Congestion Control: PFC, ECN, DCQCN and AI Fabric Engineering

High-performance AI networking requires more than fast links. Distributed GPU workloads can generate synchronised bursts of network traffic, large data transfers and incast patterns that place significant pressure on switch buffers and endpoint network adapters.

When offered traffic exceeds the forwarding capacity of a network interface, packets accumulate in queues. If the congestion persists, the network may experience increased latency, packet drops, transport retries and reduced application throughput.

RoCEv2 deployments often combine Explicit Congestion Notification (ECN), Data Center Quantized Congestion Notification (DCQCN), Priority Flow Control (PFC) and QoS configuration to manage these conditions. However, these mechanisms must be engineered carefully because incorrect configuration can create additional performance problems.

4.1 — How Congestion Develops Inside an AI Ethernet Fabric

Congestion occurs when packets arrive at a forwarding resource faster than they can be transmitted. The immediate effect is queue growth, followed by increased queueing delay and potentially packet loss when available buffer space is exhausted.

AI FABRIC CONGESTION ARCHITECTURE GPU NODE A RDMA Traffic Source GPU NODE B RDMA Traffic Source ETHERNET SWITCH Congested Egress Queue GPU NODE C RDMA Receiver Shared Egress Destination BUFFER PRESSURE Queue occupancy increasing CONGESTION FEEDBACK ECN / CNP / Sender Rate Adaptation

Architecture interpretation: Multiple GPU senders are transmitting toward a common destination. The switch egress queue becomes congested. ECN marking can communicate congestion information to the receiving endpoint, which can generate congestion feedback for the sender. PFC operates separately through link-local pause frames when configured thresholds are reached.

4.2 — The Four Major Congestion Control Mechanisms

01
Switch Buffering

Absorbs temporary differences between packet arrival and departure rates.

02
ECN

Marks eligible packets to indicate congestion without necessarily dropping them.

03
DCQCN

Uses congestion feedback to adjust the sending rate of supported RoCEv2 traffic.

04
PFC

Pauses a selected Ethernet priority on an adjacent link to prevent buffer overflow.

05
QoS Classification

Maps traffic to appropriate priorities and forwarding queues.

06
Traffic Engineering

Uses topology, capacity and path distribution to reduce hotspots.

4.3 — Priority Flow Control: IEEE 802.1Qbb

Priority Flow Control is an Ethernet link-layer mechanism that allows a receiving device to request that its directly connected peer temporarily stop transmitting traffic belonging to specific priorities.

Unlike conventional Ethernet PAUSE, which pauses all traffic on a link, PFC supports selective pause behaviour for individual traffic priorities.

Property PFC behaviour
Scope Directly connected Ethernet peers
Classification Selected Ethernet priority values
Trigger Configured buffer or congestion thresholds
Action Request a temporary pause for the affected priority
Benefit Can reduce buffer-overflow loss for configured traffic
Risk Pause propagation, head-of-line blocking and possible deadlock

Important engineering distinction: PFC is not an end-to-end congestion control algorithm. It applies backpressure between neighbouring Ethernet devices. Excessive PFC activity can propagate congestion through the fabric and reduce performance for otherwise healthy traffic.

4.4 — Explicit Congestion Notification

ECN provides a mechanism for network devices to signal congestion by marking ECN-capable IP packets rather than necessarily discarding them.

The IP header contains a two-bit ECN field. A congested switch can mark eligible traffic with the Congestion Experienced (CE) codepoint.

ECN field in the IP header 00 = Not ECN-Capable Transport 01 = ECT(1) 10 = ECT(0) 11 = Congestion Experienced (CE) Illustrative packet journey: GPU Sender | | ECN-capable RoCEv2 packet v Ethernet Switch | | Queue threshold exceeded | Set ECN = CE v GPU Receiver | | Detect congestion indication v Congestion feedback to sender

For a typical DCQCN-based RoCEv2 deployment, the receiving NIC reacts to congestion-marked packets by generating a Congestion Notification Packet (CNP) toward the sending endpoint.

4.5 — DCQCN: Data Center Quantized Congestion Notification

DCQCN is an end-to-end congestion control mechanism commonly associated with RoCEv2 deployments. It uses network congestion marking and feedback to influence sender transmission rates.

1. Congestion Detection

A switch marks eligible packets when congestion thresholds are reached.

2. Receiver Feedback

The receiving endpoint generates congestion feedback according to its implementation.

3. Rate Reduction

The sender reduces its transmission rate in response to feedback.

4. Recovery

The sender may increase its rate as conditions improve.

5. Stabilisation

Rate adaptation seeks to limit persistent queue growth and congestion pressure.

6. Monitoring

Engineers inspect CNP, ECN and throughput counters to validate behaviour.

Design principle: ECN and DCQCN attempt to control congestion before prolonged buffer pressure requires PFC. Actual effectiveness depends on hardware implementation, threshold selection, traffic patterns and endpoint rate-control parameters.

4.6 — Switch Buffer Engineering

Switch buffers temporarily store packets when the output interface cannot transmit them immediately. Buffer capacity must accommodate short bursts, congestion feedback delay and traffic scheduling behaviour.

Illustrative Queue Growth Calculation
Ingress traffic rate: 800 Gbit/s Egress link capacity: 400 Gbit/s Excess offered rate: 800 - 400 = 400 Gbit/s If this imbalance persists for 10 µs: 400 Gbit/s × 10 µs = 4,000,000 bits = 500,000 bytes ≈ 488 KiB Approximate queue growth: 488 KiB in 10 microseconds

This simplified calculation assumes a sustained rate imbalance and no packet drops, pause events or sender rate adaptation during the interval. Real buffer behaviour depends on the switch architecture, queue allocation, shared memory, packet sizes and scheduling.

Engineering implication: Even very short congestion events can consume significant buffer space at 400G and 800G link speeds. Buffer sizing and congestion feedback timing must therefore be considered together.

4.7 — Head-of-Line Blocking and PFC Propagation

Head-of-line blocking occurs when queued traffic prevents other packets from progressing, even when some downstream resources may otherwise be available.

With PFC, a paused priority can affect multiple flows sharing that priority on an upstream link. A congested destination can therefore cause backpressure that extends beyond the original bottleneck.

Local Congestion

A switch egress queue experiences high occupancy and requests a pause from its upstream peer.

Congestion Propagation

The upstream device accumulates traffic and may issue additional pause requests, potentially affecting unrelated flows mapped to the same priority.

Operational warning: PFC can protect traffic against certain buffer-overflow conditions, but excessive or persistent pause activity is a sign that the fabric may be experiencing serious congestion or configuration problems.

4.8 — PFC Deadlock and Congestion Collapse Risks

PFC deadlock can occur when a cyclic dependency develops between paused traffic priorities and forwarding resources.

In such a condition, devices may be waiting for one another to resume transmission, preventing forward progress until the condition is broken.

Risk Potential impact Engineering response
PFC storm High pause activity across multiple links Inspect pause counters, congestion sources and QoS mapping
Head-of-line blocking Unrelated flows delayed within a paused priority Review traffic isolation and priority assignment
PFC deadlock Persistent forwarding stall Validate topology, deadlock detection and recovery features
Buffer exhaustion Packet loss and transport retries Review thresholds, capacity and traffic patterns
Incorrect ECN thresholds Late marking or unnecessary rate reduction Tune against platform guidance and measured workloads
DCQCN instability Throughput oscillation or persistent congestion Inspect endpoint rate-control configuration and CNP telemetry

4.9 — QoS and Traffic Classification

RoCEv2 traffic must be mapped consistently to the intended QoS treatment throughout the fabric.

This may involve IP DSCP markings, Ethernet priority code points, switch traffic classes, queue scheduling and PFC priority configuration.

Illustrative QoS design Application / RDMA NIC | | RoCEv2 traffic marking v IP DSCP Classification | v Switch Traffic Class | +--> ECN marking policy | +--> Buffer allocation | +--> Scheduling policy | +--> PFC priority (if enabled) | v Fabric Egress Port All fabric devices must apply compatible classification rules.

Production recommendation: Do not assume that a DSCP value automatically maps to the correct PFC priority. The mapping must be verified on each relevant switch and network adapter, including trust boundaries and remarking behaviour.

4.10 — Production Congestion Troubleshooting

Congestion troubleshooting should correlate switch queue telemetry, PFC pause counters, ECN markings, CNP generation, NIC transport counters and application-level performance.

Observation Likely investigation
High queue occupancy Offered load, egress bottlenecks and buffer allocation
Frequent ECN marks Persistent congestion and marking thresholds
High CNP activity Receiver feedback and sender rate adaptation
Persistent PFC pauses Congestion propagation and priority mapping
RDMA retries Packet loss, transport recovery and physical errors
Low GPU utilisation Collective communication stalls and network bottlenecks
Uneven switch utilisation ECMP distribution and large-flow placement
Production investigation sequence 1. Confirm GPU workload symptoms 2. Identify affected communication flows 3. Check physical link and NIC errors 4. Inspect switch egress queue occupancy 5. Review ECN marking counters 6. Review CNP generation and reception 7. Inspect PFC pause duration and frequency 8. Validate DSCP / PCP / queue mapping 9. Check ECMP and path distribution 10. Compare results against a healthy baseline

Change-control warning: Avoid disabling PFC or changing ECN/DCQCN thresholds blindly in a production RoCEv2 fabric. These changes can increase packet loss, transport retries or application failures. Validate proposed changes against vendor guidance and controlled workload testing.

Section 4 — Engineering Takeaway

High-performance RoCEv2 Ethernet fabrics require coordinated switch buffering, ECN marking, DCQCN sender rate adaptation, PFC configuration, QoS classification and traffic engineering. The objective is not simply to eliminate packet loss, but to maintain predictable throughput and latency without creating excessive backpressure, head-of-line blocking or fabric-wide congestion propagation.

Network Insight · Engineering Lab 04 · AI Networking

AI Fabric Congestion Laboratory

Investigate how offered load, link capacity, switch buffer occupancy, ECN marking, PFC pauses and simplified DCQCN feedback interact. Advance the simulation, inspect queue growth and compare congestion-control strategies under different traffic conditions.

Congestion engineering configuration

0 ms Simulation time
0% Buffer occupancy
0G Effective sender rate
0 Congestion events
GPU / RNIC Senders
GPU
RDMA traffic generation
Ethernet Switch Egress queue / congestion
0%
Queue empty
Destination GPU
GPU
RDMA receiving endpoint
Queue occupancy timeline

Congestion engineering inspector

AI Fabric Congestion

Queue occupancy increases when incoming traffic exceeds available egress capacity.

Engineering check: compare offered load, queue growth and congestion feedback thresholds.
Congestion diagnostics console

Congestion engineering findings

Simulation assumptions: This is an illustrative single-egress-queue model. It does not implement the full IEEE PFC state machine, ECN marking probability, CNP generation or DCQCN control equations. ECN/DCQCN is approximated as sender-rate reduction after a threshold event. PFC is approximated as temporary upstream admission suppression. Real PFC operates per priority and can propagate congestion upstream, while DCQCN depends on feedback timing and RNIC algorithms. Actual switch buffer architecture and QoS behavior vary by platform.
Network Insight / AI Networking / Section 05

InfiniBand Architecture Explained: HCAs, Subnet Managers, LIDs, Queue Pairs and GPU Fabric Engineering

InfiniBand is a high-performance networking architecture designed for low-latency, high-bandwidth communication between compute systems. It provides a native RDMA transport environment and is widely associated with high-performance computing and large-scale GPU infrastructure.

Unlike RoCEv2, which transports RDMA traffic across Ethernet and IP networks, InfiniBand defines its own fabric architecture, link-layer behaviour, addressing, management protocols and transport mechanisms.

Understanding InfiniBand requires examining not only its physical switches and host adapters, but also the subnet management system, forwarding tables, virtual lanes, credit-based flow control and RDMA transport resources.

5.1 — InfiniBand Fabric Architecture

An InfiniBand fabric connects compute nodes through Host Channel Adapters (HCAs), InfiniBand switches and managed forwarding paths.

A Subnet Manager discovers the fabric, assigns addresses and configures forwarding information within the InfiniBand subnet.

INFINIBAND GPU FABRIC SUBNET MANAGER Discovery / LIDs / Routing IB SPINE 1 IB SPINE 2 InfiniBand Switching InfiniBand Switching IB LEAF 1 IB LEAF 2 IB LEAF 3 Fabric Access Fabric Access Fabric Access GPU NODE + HCA GPU NODE + HCA GPU NODE + HCA

Architecture interpretation: The Subnet Manager is a logical fabric management role, not necessarily a dedicated physical appliance. InfiniBand fabrics can support redundant Subnet Managers with one active master for a subnet. The diagram shows a simplified two-tier topology; production deployments may use additional tiers and more complex routing policies.

5.2 — Core InfiniBand Components

01
Host Channel Adapter

Connects a compute host to the InfiniBand fabric and provides RDMA capabilities.

02
InfiniBand Switch

Forwards packets using configured fabric forwarding information.

03
Subnet Manager

Discovers fabric topology and configures subnet forwarding state.

04
Queue Pair

Provides send and receive work queues for RDMA operations.

05
Virtual Lane

Supports logical traffic separation and link-level flow-control resources.

06
Partitioning

Uses partition membership to control communication within the fabric.

5.3 — Subnet Manager and Fabric Discovery

The Subnet Manager plays a central role in InfiniBand fabric operation. It discovers nodes and switches, assigns local identifiers, configures forwarding tables and manages subnet-level configuration.

Simplified InfiniBand initialization 1. Physical links become active 2. Subnet Manager discovers topology 3. Ports receive configured LIDs 4. Switch forwarding tables are built 5. Partition information is distributed 6. Ports transition to usable states 7. RDMA endpoints establish communication

Actual initialization and recovery procedures depend on fabric topology, hardware support and Subnet Manager implementation.

5.4 — LIDs, GUIDs and InfiniBand Addressing

Identifier Purpose
GUID Globally unique identifier associated with fabric components or ports
LID Local Identifier used for forwarding within an InfiniBand subnet
GID Global Identifier used for addressing in relevant InfiniBand contexts
QPN Queue Pair Number identifying a transport endpoint
P_Key Partition key used for fabric partition membership

Forwarding distinction: Within an InfiniBand subnet, switch forwarding commonly uses destination LIDs and Subnet Manager programmed forwarding tables. This differs from conventional Ethernet MAC learning and IP routing behaviour.

5.5 — InfiniBand Packet Architecture

InfiniBand packets carry link-level and transport-level information required for forwarding and RDMA operations. The exact header sequence depends on the packet type and transport service.

Illustrative InfiniBand packet +----------------------------------+ | Local Routing Header (LRH) | | Destination LID / Source LID | | Service Level / Virtual Lane | +----------------------------------+ | Optional Global Routing Header | | Source GID / Destination GID | +----------------------------------+ | Base Transport Header (BTH) | | Opcode / QPN / PSN | +----------------------------------+ | Optional Transport Headers | | RETH / AETH / other headers | +----------------------------------+ | Payload / Padding | +----------------------------------+ | Invariant CRC (ICRC) | +----------------------------------+ | Variant CRC (VCRC) | +----------------------------------+

The Local Routing Header supports forwarding within the InfiniBand subnet. The Base Transport Header carries information such as the operation code, destination queue pair and packet sequence number.

5.6 — Virtual Lanes and Credit-Based Flow Control

InfiniBand uses link-level, credit-based flow control. A transmitting port tracks available receive buffer credits for the relevant virtual lane and must respect those credits when sending data.

Virtual Lanes

Virtual lanes provide logical traffic separation with associated buffering and flow-control resources. They can support differentiated traffic handling within the fabric.

Credits

Credits represent available receive buffer capacity. The sender must have sufficient credit before transmitting data on a credit-controlled link.

Engineering distinction: InfiniBand credit-based flow control is not the same mechanism as Ethernet Priority Flow Control. Both address link-level buffer management, but they use different protocol architectures and operational controls.

5.7 — InfiniBand Routing and Adaptive Path Selection

InfiniBand switches forward traffic according to routing information configured by the fabric management system.

Routing algorithms can be selected to suit the topology and workload. Some platforms also support adaptive routing, which can select among eligible paths based on network conditions.

Routing consideration Engineering impact
Topology awareness Routing should match the physical fabric structure
Path diversity Multiple available paths can improve resilience and distribution
Adaptive routing May improve utilisation under changing traffic conditions
Ordering requirements Transport and hardware constraints affect path selection
Failure recovery Fabric management and hardware features influence recovery behaviour

5.8 — InfiniBand Link Speed and Capacity Planning

InfiniBand link generations include HDR, NDR and XDR, with nominal aggregate link rates depending on the generation and lane configuration.

Generation Common nominal 4-lane rate Design consideration
HDR 200 Gbit/s Port density, topology and existing platform compatibility
NDR 400 Gbit/s High-bandwidth GPU fabric connectivity
XDR 800 Gbit/s Hardware generation, optics, cabling and ecosystem support
Illustrative capacity calculation 8 compute nodes 1 × 400 Gbit/s HCA link per node Aggregate nominal endpoint bandwidth: 8 × 400 Gbit/s = 3.2 Tbit/s If the switching fabric provides 3.2 Tbit/s of usable aggregate uplink capacity at the boundary: Nominal oversubscription = 1:1

This calculation describes nominal interface capacity. Actual effective application throughput depends on protocol overhead, traffic distribution, collective operations and fabric congestion behaviour.

5.9 — InfiniBand Partitioning and Security Boundaries

InfiniBand partitioning uses partition keys to control which ports can communicate within a configured fabric partition.

Partitioning provides an important fabric-level access-control mechanism, but it should not be treated as a complete security architecture for untrusted workloads.

Security consideration: Production InfiniBand environments should also consider host security, workload isolation, management access, firmware integrity and physical fabric boundaries.

5.10 — InfiniBand Production Troubleshooting

Symptom Investigation area
Port not active Physical connectivity, link negotiation and Subnet Manager state
Missing LID Subnet discovery and management configuration
Unexpected routing behaviour Switch forwarding tables and routing algorithm
Low throughput Link rate, congestion, topology and adapter configuration
RDMA operation failure Queue pair state, memory permissions and transport errors
Fabric instability Link errors, Subnet Manager events and topology changes
Communication isolation Partition membership and P_Key configuration
Illustrative InfiniBand checks ibstat ibv_devinfo ibnetdiscover iblinkinfo ibroute perfquery Investigate: 1. Physical port state 2. Active link width and speed 3. Subnet Manager availability 4. LID assignment and routing 5. Partition membership 6. Link error counters 7. Congestion and transport errors 8. Application collective performance

These commands are commonly associated with Linux InfiniBand tooling, but availability and supported options vary by installed packages, drivers and platform.

Section 5 — Engineering Takeaway

InfiniBand provides a dedicated high-performance fabric architecture built around HCAs, subnet management, LID-based forwarding, RDMA queue pairs, virtual lanes, credit-based flow control and fabric-aware routing. Successful deployment requires coordinated engineering across the physical topology, fabric management, endpoint configuration, traffic behaviour and operational telemetry.

Network Insight · Engineering Lab 05 · AI Networking

InfiniBand Fabric Architecture Explorer

Explore InfiniBand Host Channel Adapters, fabric switches, subnet management, local identifiers, forwarding tables, virtual lanes and path selection. Discover the fabric, trace packets and investigate the impact of a failed inter-switch link.

InfiniBand fabric configuration

4 HCAs discovered
4 Fabric switches
4 Assigned host LIDs
Ready Path state

InfiniBand engineering inspector

Subnet Management

The InfiniBand subnet manager discovers the fabric and configures subnet addressing and forwarding.

Engineering check: inspect LIDs, switch forwarding tables and path availability.
Fabric diagnostics console

InfiniBand engineering findings

Engineering scope: This widget provides a simplified InfiniBand subnet and forwarding model. Real subnet managers use management datagrams and configure LIDs, forwarding tables, partitioning and other fabric attributes. Virtual lane selection depends on service level mappings and switch configuration. Adaptive routing is implementation-dependent. The example does not implement actual InfiniBand packet forwarding, arbitration or subnet management protocols.
Network Insight / AI Networking / Section 06

Ethernet vs RoCEv2 vs InfiniBand: Architecture, Performance, Scalability and AI Fabric Design Decisions

Selecting a network architecture for an AI GPU cluster requires evaluating more than raw link bandwidth. The chosen fabric must support the communication patterns of distributed AI workloads while meeting operational, reliability, compatibility and cost requirements.

Ethernet, RoCEv2 and InfiniBand are frequently discussed together, but they do not represent three equivalent networking layers. Ethernet provides a general-purpose networking foundation, RoCEv2 enables RDMA over UDP/IP and Ethernet, while InfiniBand provides a dedicated high-performance interconnect architecture.

6.1 — Three AI Networking Approaches

The practical comparison is between conventional Ethernet/IP communication, Ethernet with RoCEv2 RDMA, and InfiniBand RDMA fabrics.

AI NETWORK TECHNOLOGY COMPARISON ETHERNET / IP RoCEv2 INFINIBAND AI APPLICATION AI APPLICATION AI APPLICATION TCP / UDP RDMA TRANSPORT RDMA TRANSPORT Socket-based communication Queue pairs / Memory access Queue pairs / Memory access IP NETWORK UDP / IP IB FABRIC Ethernet switching / routing Ethernet switching / routing LID forwarding / SM GENERAL PURPOSE ETHERNET RDMA NATIVE IB RDMA

Architecture distinction: RoCEv2 is not an alternative physical network to Ethernet. It is an RDMA transport that uses UDP/IP over Ethernet. InfiniBand provides its own link, management and transport architecture. Conventional Ethernet/IP traffic may also coexist with RoCEv2 traffic on appropriately engineered infrastructure.

6.2 — Core Technology Comparison

Characteristic Ethernet / IP Ethernet + RoCEv2 InfiniBand
Network foundation Ethernet and IP Ethernet and IP InfiniBand fabric
RDMA support Not inherent to conventional sockets Yes, with compatible RDMA adapters Native RDMA capabilities
Transport Typically TCP or UDP RDMA over UDP/IP InfiniBand transport
Routing IP routing and Ethernet switching IP routing and Ethernet switching Subnet-managed forwarding
Congestion management Transport control, QoS and network mechanisms ECN, DCQCN, PFC and other supported controls Credit-based flow control and fabric congestion mechanisms
Operational tooling Broad Ethernet/IP ecosystem Ethernet/IP plus RDMA-specific tools InfiniBand-specific fabric tooling
Typical consideration General connectivity and application traffic High-performance RDMA on Ethernet Dedicated high-performance RDMA fabric

6.3 — Latency and Throughput Engineering

Network performance must be evaluated using end-to-end workload measurements, not simply interface line rate or a single latency benchmark.

01
Transport Overhead

Software, adapter and protocol processing influence effective communication latency.

02
Fabric Latency

Switching, queueing and network hops contribute to end-to-end delay.

03
Tail Latency

Slow communication operations can delay collective synchronisation.

04
Aggregate Bandwidth

Available paths and topology determine usable cluster-wide capacity.

05
Congestion Behaviour

Rate adaptation, buffering and flow control affect sustained performance.

06
GPU Efficiency

Communication overhead affects useful accelerator execution time.

Performance warning: It is not technically accurate to claim that one fabric technology is always faster than another. Results depend on hardware generation, adapter configuration, topology, software stack, message size, congestion and workload communication patterns.

6.4 — Operational Complexity and Management

Ethernet and RoCEv2 Operations

Ethernet-based AI fabrics can reuse existing IP routing, network automation, telemetry and switching expertise. However, RoCEv2 adds specialised requirements for NIC configuration, QoS mapping, congestion control and RDMA troubleshooting.

InfiniBand Operations

InfiniBand requires expertise in subnet management, LID forwarding, fabric routing, HCAs, virtual lanes and InfiniBand-specific diagnostics. Its management architecture differs from conventional Ethernet/IP operations.

Neither approach is operationally simple by default. The better choice depends on existing engineering skills, automation maturity, hardware support and the expected scale of the deployment.

6.5 — Scalability and Fabric Expansion

Both Ethernet and InfiniBand can support large multi-switch fabrics. Scalability depends on switch radix, link capacity, topology, routing or forwarding algorithms, congestion management and physical infrastructure.

Scaling factor Engineering question
Switch radix How many high-speed ports are available per switch?
Topology Can additional nodes be added without excessive oversubscription?
Path diversity Are there enough independent paths between compute groups?
Optics and cabling Can the physical infrastructure support the required port speeds?
Management scale Can configuration and monitoring handle the full fabric?
Failure domains How many workloads are affected by a device or link failure?

6.6 — Compatibility and Ecosystem Considerations

AI networking architecture must align with the GPU platform, network adapters, switches, drivers, firmware, operating systems, communication libraries and deployment frameworks.

GPU Platform

Validate accelerator and memory interconnect compatibility.

Network Adapter

Confirm supported link speeds, transport modes and RDMA features.

Switching Platform

Verify buffering, routing, congestion and telemetry capabilities.

Software Stack

Check drivers, firmware and communication library support.

Automation

Assess configuration management and deployment repeatability.

Observability

Confirm availability of fabric and endpoint performance counters.

6.7 — Total Cost of Ownership

Fabric selection should consider the full lifecycle cost rather than the purchase price of switches and network adapters alone.

Illustrative AI fabric TCO model Total Cost of Ownership = Network switches + Network adapters + Optics and cabling + Power and cooling + Support and maintenance + Software and management + Engineering operations + Deployment and migration + Downtime and recovery impact Evaluate cost per useful workload outcome, not only cost per nominal Gbit/s.

A lower-cost network may be less economical if communication bottlenecks significantly reduce expensive GPU utilisation. Conversely, a premium fabric may provide limited additional value for workloads that are not network constrained.

6.8 — Workload-Based Technology Selection

Workload scenario Key requirements Design consideration
General AI services Reliable application connectivity and scalability Conventional Ethernet/IP may be sufficient
Distributed GPU training High bandwidth, low communication overhead and predictable congestion Evaluate Ethernet/RoCEv2 and InfiniBand
Large collective workloads Efficient all-reduce and all-to-all communication Benchmark real collective traffic on candidate fabrics
Mixed enterprise infrastructure Integration with existing networking operations Ethernet ecosystem and operational compatibility may matter
Dedicated HPC environment Specialised high-performance interconnect and fabric management InfiniBand may be a strong candidate
Inference clusters Latency, throughput and communication characteristics of serving architecture Choose based on tensor parallelism and model-serving requirements

6.9 — Production Engineering Decision Checklist

AI Network Technology Evaluation [ ] Define the workload communication pattern [ ] Measure required aggregate bandwidth [ ] Identify latency and tail-latency targets [ ] Validate GPU and NIC compatibility [ ] Evaluate fabric topology and scaling [ ] Benchmark representative collectives [ ] Test congestion and incast behaviour [ ] Validate failure recovery [ ] Review management and automation [ ] Compare complete lifecycle costs [ ] Test operational troubleshooting [ ] Document the final design trade-offs

Architecture recommendation: Use a representative workload benchmark to compare candidate fabrics. Measure not only synthetic network throughput, but also distributed training step time, collective latency, GPU utilisation, failure behaviour and operational complexity.

Section 6 — Engineering Takeaway

The choice between conventional Ethernet/IP, Ethernet with RoCEv2 and InfiniBand should be driven by workload communication requirements, RDMA capabilities, congestion behaviour, topology scalability, ecosystem compatibility, operational maturity and total cost of ownership. No single network technology is universally optimal for every AI infrastructure deployment.

Network Insight · Engineering Lab 06 · AI Networking

AI Network Technology Decision Lab

Compare Ethernet/TCP, Ethernet/RoCEv2 and InfiniBand for distributed AI workloads. Change deployment requirements and evaluate architectural suitability, operational complexity, ecosystem compatibility, RDMA requirements and infrastructure reuse.

AI infrastructure requirements

Ethernet / TCP
0
Conventional IP networking
Ethernet / RoCEv2
0
RDMA over converged Ethernet
InfiniBand
0
Native RDMA fabric
Weighted suitability comparison
Reference architecture
GPU Servers Compute / RNICs
Network Fabric Transport
GPU Servers Distributed workload

Architecture decision inspector

Technology evaluation

Configure the workload and infrastructure requirements to evaluate technology suitability.

Engineering check: validate RDMA requirements, software support and operational constraints.
Decision diagnostics console

Engineering recommendation and trade-offs

Decision model: The scores are illustrative suitability heuristics, not benchmark measurements. Ethernet/TCP refers to conventional socket-based transport rather than RoCEv2. RoCEv2 and InfiniBand both support RDMA but differ in fabric architecture and operations. Actual technology selection requires validating GPU/NIC support, communication libraries, switch capabilities, congestion management, cost, availability, software integration and measured application performance. A higher score does not establish universally lower latency or greater throughput.
Network Insight / AI Networking / Section 07

Distributed GPU Communication Explained: NCCL, All-Reduce, All-Gather, Reduce-Scatter and Parallel AI Training

Large AI models are frequently trained across multiple GPUs and compute nodes. These distributed workloads depend on communication operations that exchange gradients, model parameters, activations and other intermediate data between accelerators.

The efficiency of this communication can significantly influence overall training performance. A fast GPU may remain idle while waiting for another participant to complete a collective operation.

Communication libraries such as NVIDIA Collective Communications Library (NCCL) provide optimised collective operations across supported GPU interconnects and network transports.

7.1 — Distributed GPU Communication Architecture

A distributed training job may span multiple GPU servers. Within a server, GPUs communicate through supported scale-up interconnects. Between servers, communication crosses a scale-out network fabric.

DISTRIBUTED GPU COMMUNICATION SCALE-OUT FABRIC Ethernet / RoCEv2 or InfiniBand GPU NODE A GPU NODE B GPU NODE C GPU 0 GPU 1 GPU 0 GPU 1 GPU 0 GPU 1 ALL-REDUCE · ALL-GATHER · REDUCE-SCATTER · ALL-TO-ALL

Architecture interpretation: The diagram represents a logical distributed training topology. Actual communication patterns depend on the collective algorithm, parallelism strategy, GPU topology and network transport. A collective does not necessarily send every participant's data to every other participant using a direct physical connection.

7.2 — Fundamental Collective Operations

01
All-Reduce

Combines values across participants and distributes the reduced result to all.

02
All-Gather

Collects data partitions from all participants and distributes the combined result.

03
Reduce-Scatter

Reduces distributed data and delivers a partition of the result to each participant.

04
Broadcast

Distributes data from one participant to all others.

05
All-to-All

Exchanges different data partitions between participants.

06
Point-to-Point

Transfers data between specific participants or pipeline stages.

7.3 — Ring All-Reduce Architecture

Ring all-reduce is a collective algorithm in which participants are arranged in a logical ring. Data is partitioned and exchanged through repeated communication steps.

A common ring all-reduce implementation uses a reduce-scatter phase followed by an all-gather phase.

RING ALL-REDUCE GPU 0 GPU 1 GPU 2 GPU 3 REDUCE-SCATTER ALL-GATHER

Engineering distinction: The ring is a logical communication structure. Its participants may be connected through NVLink, NVSwitch, Ethernet/RoCEv2 or InfiniBand, depending on the deployment. The physical fabric does not need to form a literal ring topology.

7.4 — Ring All-Reduce Communication Volume

For a ring all-reduce with N participants and a total buffer size of S bytes per participant, the idealised communication volume per participant is:

Ring All-Reduce Reduce-Scatter: (N - 1) / N × S All-Gather: (N - 1) / N × S Total data sent per participant: 2 × (N - 1) / N × S Example: N = 8 GPUs S = 1 GiB 2 × (7 / 8) × 1 GiB = 1.75 GiB Idealised data sent per GPU: 1.75 GiB

This calculation describes the algorithmic communication volume, not the total physical network traffic across every switch link. Actual transfer time also depends on bandwidth, latency, protocol overhead, contention and communication overlap.

7.5 — NCCL Communication Architecture

NCCL provides collective communication operations for supported NVIDIA GPU environments. It can select communication algorithms and transport paths based on hardware topology, message size, software configuration and available network resources.

Layer Role
AI framework Coordinates distributed training and collective calls
NCCL Implements and schedules supported GPU communication operations
GPU interconnect Provides local accelerator communication where supported
Network transport Provides communication between compute nodes
Network fabric Forwards traffic across Ethernet/RoCEv2 or InfiniBand infrastructure
Telemetry Exposes communication timing, transport errors and fabric performance

Important: NCCL does not always use a single collective algorithm. Ring, tree and other algorithmic approaches may be selected depending on the software version, hardware topology and workload characteristics.

7.6 — Data, Tensor and Pipeline Parallelism

Parallelism Communication pattern Network consideration
Data parallelism Gradient synchronisation between replicas All-reduce or related collective communication
Tensor parallelism Exchange intermediate tensor partitions Frequent latency-sensitive communication
Pipeline parallelism Transfer activations and gradients between stages Point-to-point bandwidth and pipeline scheduling
Expert parallelism Exchange tokens or activations among experts Potentially intensive all-to-all communication

Different parallelism strategies produce different network traffic patterns. A fabric designed for large gradient all-reduce operations may experience different congestion behaviour under all-to-all expert routing or frequent tensor-parallel communication.

7.7 — Communication Latency and Training Efficiency

Distributed training step time depends on compute execution, communication, synchronisation and any overlap between these activities.

Illustrative training step GPU compute: 90 ms Communication: 30 ms No overlap: Total step time = 120 ms If 20 ms of communication overlaps with useful compute: Total step time = 100 ms Effective communication overhead outside compute: 30 ms - 20 ms = 10 ms

The example illustrates why overlapping communication with computation can improve training efficiency. Actual overlap depends on framework scheduling, kernel execution, memory resources and communication dependencies.

7.8 — Collective Communication Performance Metrics

Collective Latency

Time required to complete a collective operation.

Algorithm Bandwidth

Useful data size divided by collective completion time.

Bus Bandwidth

A derived metric accounting for the collective communication pattern.

GPU Utilisation

Indicates accelerator activity but requires workload context.

Network Throughput

Measures traffic carried by links and adapters.

Tail Latency

Highlights slow operations that may delay synchronisation.

Benchmarking: NCCL performance tests can help measure collective communication behaviour. Results should be interpreted alongside network counters, GPU telemetry and representative application workloads.

7.9 — Production Collective Communication Troubleshooting

Symptom Investigation area
Slow all-reduce Fabric congestion, algorithm selection and GPU topology
Collective timeout Transport failures, process synchronisation and endpoint availability
Uneven GPU progress Stragglers, workload imbalance and communication dependencies
Low bus bandwidth NIC performance, routing imbalance and communication configuration
Intermittent stalls RDMA retries, PFC pauses, ECN activity and link errors
Scaling efficiency decline Communication overhead, oversubscription and synchronisation cost
Collective investigation workflow 1. Identify the affected collective 2. Record message size and GPU count 3. Inspect NCCL/application logs 4. Validate GPU and NIC topology 5. Check fabric throughput and errors 6. Inspect ECN, PFC and RDMA counters 7. Compare against a healthy baseline 8. Run representative collective tests 9. Correlate network and GPU timelines

Engineering warning: A slow collective does not automatically indicate a network fault. Compute imbalance, GPU memory pressure, software synchronisation and workload scheduling can produce similar symptoms. Troubleshooting must correlate compute and network evidence.

Section 7 — Engineering Takeaway

Distributed GPU training depends on collective communication, parallelism strategy, GPU topology, network transport and fabric performance. Operations such as all-reduce, all-gather, reduce-scatter and all-to-all create different traffic patterns and communication requirements. Efficient AI network engineering therefore requires understanding both the application's communication algorithm and the physical network that carries its data.

Network Insight · Engineering Lab 07 · AI Networking

Distributed GPU Collective Communication Visualiser

Investigate how GPU communication libraries coordinate collective operations across distributed accelerators. Compare ring and tree algorithms, explore All-Reduce, All-Gather, Reduce-Scatter and Broadcast, and estimate communication volume and idealised transfer time.

Collective communication configuration

8 Participating GPUs
0 Communication stages
0 MB Bytes per GPU
0 ms Idealised time
Collective communication ready

Build the model, then advance through the communication stages.

Collective engineering inspector

Distributed GPU Communication

Collective operations coordinate data exchange across multiple GPU ranks.

Engineering check: compare message size, communication steps and network bandwidth.
Collective diagnostics console

Communication engineering findings

Model assumptions: The ring formulas assume a single logical ring, equal-sized chunks and a payload interpreted as the full collective tensor size. Ring All-Reduce requires 2(N−1) communication steps and transfers 2(N−1)/N times the tensor size per GPU. Ring All-Gather and Reduce-Scatter each transfer (N−1)/N times the tensor size per GPU. The tree model uses simplified logarithmic communication stages. Actual NCCL algorithms, hierarchical communication, GPU placement, NVLink, network topology, pipelining and protocol overhead can substantially change performance.
Network Insight / AI Networking / Section 08

AI Network Performance Engineering: Throughput, Latency, RDMA Counters, Congestion Telemetry and GPU Fabric Observability

High-performance AI networking requires continuous measurement of the relationship between network behaviour and GPU workload performance. Link bandwidth alone does not reveal whether a distributed training cluster is operating efficiently.

Engineers must correlate switch telemetry, network adapter counters, RDMA transport behaviour, collective communication performance and GPU utilisation.

The objective is to identify where communication time is being lost, determine whether the network is responsible and establish which infrastructure changes will produce measurable improvements.

8.1 — End-to-End AI Fabric Observability Architecture

A production AI fabric should expose telemetry from compute nodes, network adapters, switches, congestion-control mechanisms and application communication libraries.

AI FABRIC OBSERVABILITY ARCHITECTURE OBSERVABILITY PLATFORM Metrics / Events / Correlation FABRIC SWITCH A FABRIC SWITCH B Queue / ECN / PFC / Drops Port / Buffer / Error Counters GPU NODE A GPU NODE B GPU NODE C GPU / NIC / NCCL Metrics GPU / NIC / NCCL Metrics GPU / NIC / NCCL Metrics Correlate GPU, NIC, transport and fabric measurements

Architecture interpretation: Observability should connect workload-level symptoms with the underlying network path. A training step slowdown may correlate with RDMA retries, PFC pauses, ECN marking, switch queue growth or GPU execution imbalance. Correlation does not automatically establish causation, so targeted validation remains necessary.

8.2 — The Six Core AI Fabric Performance Metrics

01
Throughput

Useful data transferred per unit of time.

02
Latency

Time required for a packet or operation to complete.

03
Tail Latency

High-percentile delay affecting slow communication operations.

04
Packet Loss

Dropped packets that may trigger transport recovery.

05
Congestion

Queue pressure, ECN marking, PFC activity and rate adaptation.

06
GPU Efficiency

Useful accelerator progress relative to compute and communication overhead.

8.3 — Throughput and Effective Bandwidth

Interface line rate represents the nominal transmission capacity of a link. Effective application throughput is lower because of protocol overhead, contention, scheduling, transport behaviour and workload limitations.

Illustrative throughput calculation Data transferred: 40 GiB Elapsed time: 1 second 40 GiB/s × 8 ≈ 343.6 Gbit/s On a nominal 400 Gbit/s link: 343.6 / 400 × 100 ≈ 85.9% of nominal line rate

This calculation assumes the measurement represents a single relevant transfer path. Aggregate multi-link measurements must be interpreted according to the topology and number of active interfaces.

8.4 — Latency, Tail Latency and Queueing Delay

Average latency can conceal performance problems that affect only a small fraction of communication operations.

Tail latency metrics such as p95, p99 and p99.9 help identify slow operations that may delay distributed synchronisation.

Metric Interpretation
Average latency Mean completion time across measured operations
p50 Median observed latency
p95 95th percentile latency
p99 99th percentile latency
p99.9 99.9th percentile latency

Engineering implication: Collective communication may be delayed by the slowest required participant. Consequently, a small number of unusually slow network operations can have a disproportionate effect on training step time.

8.5 — RDMA and NIC Performance Counters

RDMA-capable network adapters expose counters that can help diagnose transport errors, packet loss, retries, congestion feedback and hardware problems.

Counter category Engineering significance
Transmit / receive bytes Measures network traffic volume
Packet errors Indicates malformed or otherwise invalid traffic
Transport retries May indicate loss, timeouts or transport recovery
Completion errors Highlights failed RDMA work requests
CNP counters Shows congestion notification activity where supported
ECN-related counters Indicates congestion-marked traffic where exposed
Physical link errors Supports diagnosis of cabling, optics or link faults

Operational note: Counter names and semantics vary between network adapter vendors, drivers, firmware versions and transport implementations. Always confirm the exact meaning of a counter before using it for automated alerting.

8.6 — ECN, PFC and Congestion Correlation

Congestion telemetry should be analysed as a sequence of related events rather than as isolated counter values.

Illustrative congestion timeline T0 GPU workload begins large collective | T1 Switch egress queue occupancy rises | T2 ECN marking threshold reached | T3 Receiver generates congestion feedback | T4 Sender adjusts transmission rate | T5 Queue pressure begins to decline If congestion continues: T6 PFC may activate if configured | T7 Excessive pauses affect throughput | T8 Application communication slows

The precise ordering and timing depend on the hardware, traffic conditions and congestion control configuration. PFC is not guaranteed to occur after ECN in every deployment.

8.7 — GPU Utilisation and Network Bottlenecks

Low GPU utilisation can be caused by network delays, but it may also result from data loading, CPU bottlenecks, memory pressure, kernel scheduling or workload imbalance.

Network-Constrained Workload

GPU execution is delayed while waiting for remote data or collective communication to complete. Network counters may show congestion, transport retries or high communication latency.

Compute-Constrained Workload

The GPU spends most of its time executing compute kernels. Increasing network bandwidth may produce little improvement if communication is not the limiting factor.

Performance engineering: Measure training step time, collective completion time, GPU activity and network utilisation together. A single GPU utilisation percentage is insufficient to determine whether the fabric is the bottleneck.

8.8 — Fabric Telemetry Collection Architecture

Modern AI fabric monitoring can combine streaming telemetry, device counters, application logs, GPU metrics and event correlation.

Telemetry source Typical measurements
Ethernet switches Port utilisation, queue occupancy, drops, ECN and PFC
InfiniBand switches Port counters, link errors, congestion and fabric state
RDMA adapters Transport errors, throughput and congestion feedback
GPU telemetry Utilisation, memory activity, power and execution behaviour
Communication libraries Collective timing, algorithm behaviour and communication errors
Workload orchestration Job placement, node allocation and workload lifecycle events

A useful monitoring system should support historical comparison, time-aligned events, per-job visibility and topology-aware correlation.

8.9 — Production Monitoring and Alert Design

Alerting thresholds should be derived from hardware capabilities, known-good workloads and measured operational baselines.

Illustrative AI Fabric Monitoring Policy Physical layer: Link state and error trends Network layer: Throughput and queue occupancy Congestion: ECN marks / CNP / PFC duration RDMA transport: Retry and completion error trends Application: Collective latency and training step time GPU: Utilisation and execution stalls Correlation: Match events by timestamp, node, port, flow and workload

Alerting guidance: Avoid treating every ECN mark or PFC pause as an incident. Some congestion signals are expected under load. Alert on sustained, abnormal or workload-impacting patterns using validated thresholds.

8.10 — Performance Investigation Workflow

AI Fabric Performance Investigation 1. Define the workload symptom 2. Record training step or inference latency 3. Identify affected GPU nodes 4. Inspect collective communication timing 5. Check NIC throughput and transport errors 6. Review switch queue occupancy 7. Correlate ECN, CNP and PFC activity 8. Validate fabric path distribution 9. Check physical link health 10. Compare with a healthy baseline 11. Test a controlled workload 12. Document the root cause and impact

The most effective investigations combine application-level evidence with network telemetry and a repeatable test methodology.

Section 8 — Engineering Takeaway

AI network performance engineering requires end-to-end observability across GPUs, RDMA adapters, Ethernet or InfiniBand switches, congestion-control mechanisms and collective communication libraries. The most important measurements are those that explain how network behaviour affects real workload completion time, rather than isolated infrastructure metrics alone.

Network Insight · Engineering Lab 08 · AI Networking

AI Fabric Performance Analyzer

Investigate production-inspired AI fabric telemetry. Correlate GPU utilisation, throughput, tail latency, ECN marking, PFC pauses and RDMA retry counters. Select devices, inspect network paths and identify the likely cause of distributed training slowdowns.

Performance investigation

— GPU utilisation
— Fabric throughput
— p99 latency
Ready Observed state
Traffic Hot path Backpressure Degraded link
GPU utilisation
Percent of observed compute activity
Network throughput
Aggregate modeled throughput · Gb/s
RDMA operation p99 latency
Illustrative tail latency · µs
Congestion signals
ECN marks and PFC pause events per second

Device telemetry inspector

Fabric Observability

Select a GPU host, leaf switch or spine switch to inspect its engineering role.

Compare endpoint telemetry with fabric-wide counters before identifying a bottleneck.
Telemetry event console

Engineering diagnosis

Select the most likely explanation for the observed telemetry.

Capture telemetry and inspect the evidence before validating your diagnosis.

Network interface and fabric counters

0 ECN-marked packets · modeled
0 PFC pause events · modeled
0 RC retry events · modeled
0 Link error events · modeled
Telemetry interpretation: The four counters above are cumulative synthetic event counts. Their increments should be examined over a time interval rather than interpreted as instantaneous rates. ECN marks indicate congestion signalling, but do not by themselves establish packet loss. PFC pause events indicate link-level backpressure for a traffic priority. RDMA retries require investigation of transport conditions and do not independently identify a specific root cause.

Production engineering considerations

Simulation scope: All telemetry is synthetic and scenario-driven. The displayed GPU utilisation, bandwidth, latency and event counters are educational examples, not measurements from physical GPUs, NICs or switches. The p99 latency represents an illustrative RDMA operation distribution, not wire propagation latency. Real investigations require timestamp alignment, counter definitions, device-specific telemetry, workload context, traffic-class configuration and evidence from both endpoints and switches.
Network Insight / AI Networking / Section 09

Production AI Fabric Troubleshooting: Congestion, RoCEv2 Failures, InfiniBand Incidents and GPU Communication Stalls

Troubleshooting a production AI network requires understanding the complete communication path between distributed GPU workloads. A training job may experience slow collective operations, intermittent timeouts, RDMA transport errors or unexpected performance degradation even when physical network links appear healthy.

These incidents can originate from multiple layers, including GPU software, RDMA adapters, network configuration, switch buffering, congestion control, routing, physical connectivity and workload scheduling.

Effective troubleshooting therefore requires a structured methodology that correlates application symptoms with endpoint and fabric evidence.

9.1 — Production AI Network Incident Architecture

The following architecture illustrates a distributed GPU workload experiencing communication degradation across a high-performance network fabric.

AI FABRIC INCIDENT INVESTIGATION INCIDENT DETECTED Collective latency / RDMA retries SPINE 1 SPINE 2 Normal forwarding Congestion pressure LEAF A LEAF B LEAF C GPU access GPU access GPU access ! GPU NODE A GPU NODE B GPU NODE C

Investigation principle: The highlighted spine path is an illustrative suspect, not a confirmed root cause. A real incident requires validation using switch counters, RDMA transport telemetry, workload timing and path analysis before attributing the problem to a specific device.

9.2 — Six Major AI Fabric Failure Domains

01
Physical Infrastructure

Optics, cabling, link errors, port speed and hardware failures.

02
Network Forwarding

Routing imbalance, path failures, incorrect forwarding and MTU problems.

03
Congestion Control

ECN, DCQCN, PFC, buffer pressure and QoS misconfiguration.

04
RDMA Transport

Queue pair errors, timeouts, retries and memory access failures.

05
GPU Communication

NCCL collective stalls, topology problems and communication imbalance.

06
Workload and Software

Framework synchronisation, driver issues, resource contention and application configuration.

9.3 — Incident 1: RoCEv2 Congestion and RDMA Retries

A distributed training workload begins experiencing longer collective completion times. Network adapters report increased transport retry activity while switch telemetry shows elevated queue occupancy.

Illustrative incident evidence Application: All-reduce completion time increasing RDMA adapter: Retry counters increasing Switch: High egress queue occupancy Congestion: ECN marking and CNP activity rising Potential root causes: - Persistent egress congestion - Traffic concentration on limited paths - Incorrect congestion thresholds - Insufficient fabric capacity

The investigation should establish whether congestion is causing packet loss and retries, or whether multiple symptoms share another underlying cause.

Operational caution: Do not immediately increase buffer thresholds or disable congestion controls. These changes may conceal the symptom while worsening fabric stability. Validate the traffic pattern, queue behaviour and rate-control response first.

9.4 — Incident 2: Incorrect QoS and PFC Configuration

RoCEv2 traffic may be classified incorrectly at the network adapter or switching layer. A mismatch between DSCP, PCP, traffic classes and PFC priorities can produce unexpected loss or pause behaviour.

Evidence Investigation
Unexpected packet drops Verify the actual switch queue used by RoCEv2 traffic
No expected PFC activity Check whether traffic is mapped to the intended priority
Excessive PFC pauses Investigate congestion, pause thresholds and priority sharing
Inconsistent behaviour across switches Compare QoS mappings and trust settings along the complete path
Transport retries under load Correlate drops, congestion feedback and endpoint counters

Engineering approach: Trace the actual classification of representative traffic from the sending adapter through each network hop. Do not assume that configured DSCP values automatically produce consistent queue treatment.

9.5 — Incident 3: ECMP Imbalance and Network Hotspots

An Ethernet leaf-spine fabric may have sufficient aggregate bandwidth but still experience congestion on individual links.

ECMP generally distributes flows according to a hashing mechanism. Large flows can collide on the same path, creating uneven utilisation even when alternative paths have available capacity.

Illustrative four-path fabric Path A: 95% utilisation Path B: 88% utilisation Path C: 32% utilisation Path D: 25% utilisation Aggregate capacity exists, but individual paths are hot. Investigation: - Inspect per-link utilisation - Review flow hashing behaviour - Check available ECMP next hops - Validate route consistency - Examine large-flow distribution

Important distinction: ECMP does not guarantee equal byte distribution across all paths. Hash-based forwarding can create uneven utilisation, especially when traffic contains a small number of large flows.

9.6 — Incident 4: InfiniBand Fabric Failure

An InfiniBand incident may involve link state, Subnet Manager operation, LID assignment, forwarding tables, partition membership or transport errors.

Symptom Potential cause Diagnostic direction
Port not active Physical link or fabric initialization issue Check port state, optics and link negotiation
Missing LID Subnet management or discovery failure Inspect Subnet Manager and addressing state
Unexpected forwarding Routing configuration or fabric topology issue Review switch forwarding tables
Communication denied Partition membership mismatch Validate P_Key configuration
Reduced throughput Congestion, link degradation or path imbalance Inspect port counters and fabric routes
Illustrative InfiniBand commands ibstat ibv_devinfo ibnetdiscover iblinkinfo ibroute perfquery Validate: Port state Link width and speed Subnet Manager status LID assignment Routing tables Partition membership Link error counters

Command availability and supported options vary by installed InfiniBand software, drivers and hardware platform.

9.7 — Incident 5: NCCL Collective Communication Stalls

A distributed training job may appear to hang during a collective operation. The cause may be network-related, but collective timeouts can also result from software or workload problems.

Transport Failure

RDMA errors, connection failures or unreachable participants.

GPU Straggler

One participant reaches the collective later than the others.

Communication Mismatch

Participants execute incompatible collective sequences or shapes.

Resource Pressure

GPU memory, CPU scheduling or host resource contention.

Fabric Congestion

Excessive queueing, retries or flow-control pressure.

Software Configuration

Incorrect transport, interface or library configuration.

Diagnostic principle: Identify the first participant that stops making progress. Correlate its GPU execution, collective calls, network adapter state and fabric telemetry with the remaining participants.

9.8 — RDMA Transport and Memory Access Failures

RDMA operations can fail because of incorrect queue pair state, memory registration, access permissions, remote addressing or transport errors.

Failure category Investigation area
Queue pair state error QP transitions and connection setup
Remote access error Remote memory permissions, address and rkey
Local protection error Local memory registration, lkey and access validity
Transport retry exhaustion Packet loss, reachability and timeout behaviour
Completion failure Work completion status and vendor-specific error details
MTU mismatch Endpoint path MTU and network forwarding constraints

Security consideration: Remote memory keys and registered memory regions must be managed carefully. Incorrect access permissions can cause operation failures or create unnecessary memory exposure.

9.9 — Structured Fault Isolation Methodology

Production investigations should progress from the observed workload symptom toward the underlying infrastructure, using evidence at each stage.

AI FABRIC FAULT ISOLATION LAYER 1 — APPLICATION Training step time Collective latency Framework errors | v LAYER 2 — GPU COMMUNICATION NCCL events GPU topology Synchronisation delays | v LAYER 3 — RDMA TRANSPORT Queue pair state Retries / completions Memory access errors | v LAYER 4 — NETWORK FABRIC Routing / forwarding Queue occupancy ECN / CNP / PFC Link utilisation | v LAYER 5 — PHYSICAL Port state Optics / cabling CRC / symbol errors Link speed / width | v ROOT CAUSE VALIDATION

This layered approach helps avoid unnecessary configuration changes and reduces the risk of misdiagnosing application problems as network failures.

9.10 — Production Incident Investigation Workflow

Network Insight — AI Fabric Incident Runbook 01. Record the incident start time 02. Identify affected jobs and GPU nodes 03. Capture application and NCCL symptoms 04. Check GPU execution and resource health 05. Inspect RDMA adapter state and counters 06. Verify physical link status 07. Review switch forwarding and routing 08. Inspect queue occupancy and packet drops 09. Correlate ECN, CNP and PFC activity 10. Compare healthy and affected paths 11. Form a testable root-cause hypothesis 12. Validate using controlled diagnostics 13. Apply an approved corrective change 14. Confirm workload performance recovery 15. Record evidence and prevention actions

Operational recommendation: Preserve telemetry before making changes. Where possible, compare the affected environment against a known-good baseline using the same workload, hardware topology and measurement methodology.

9.11 — Root Cause, Corrective Action and Prevention

Incident finding Possible corrective action Prevention
Persistent egress congestion Review traffic distribution and available capacity Capacity planning and congestion telemetry
QoS mapping mismatch Correct validated traffic classification Automated configuration consistency checks
ECMP path imbalance Review hashing, routing and flow distribution Per-path utilisation monitoring
RDMA configuration failure Correct QP or memory access configuration Endpoint configuration validation
InfiniBand fabric management issue Restore correct subnet management state Management redundancy and monitoring
Physical link degradation Repair or replace faulty components Physical error trend monitoring
Collective software mismatch Correct distributed workload configuration Pre-deployment workload validation

9.12 — Final AI Fabric Engineering Checklist

PRODUCTION AI NETWORK READINESS [ ] Physical fabric topology documented [ ] GPU and NIC connectivity validated [ ] Ethernet or InfiniBand routing verified [ ] RDMA endpoint configuration tested [ ] QoS classification consistent [ ] Congestion controls validated [ ] Switch queue telemetry available [ ] NIC transport counters monitored [ ] GPU communication benchmarks recorded [ ] Failure recovery procedures tested [ ] Known-good performance baseline saved [ ] Incident runbooks documented [ ] Change-control procedures established [ ] Operational ownership assigned

A mature AI fabric should be designed not only for high performance, but also for predictable failure behaviour, rapid fault isolation and repeatable operational recovery.

Section 9 — Final Engineering Takeaway

Production AI network troubleshooting requires a complete understanding of GPU communication, RDMA transport, Ethernet and InfiniBand forwarding, congestion control, physical infrastructure and application performance. The most effective engineering teams use time-correlated telemetry, structured fault isolation, known-good baselines and controlled validation to identify root causes and restore reliable GPU workload performance.

Network Insight · Engineering Lab 09 · AI Networking

AI Network Incident Investigation Lab

Investigate a production-inspired AI training incident. Inspect GPU utilisation, RDMA transport counters, Ethernet fabric paths, congestion signals, QoS configuration and physical link diagnostics. Identify the root cause, apply the correct remediation and verify that distributed GPU communication recovers.

Incident investigation workspace

91% GPU utilisation
720G Fabric throughput
15µs RDMA p99 latency
Healthy Fabric status
Healthy AI fabric The distributed GPU workload is operating normally. Start an incident to investigate a degraded state.

Evidence timeline

Engineering device inspector

AI Fabric Operations

Select a GPU host, leaf switch or spine switch to inspect its engineering role.

Use CLI commands to collect evidence before selecting a remediation.
Troubleshooting CLI

Root-cause analysis and remediation

Collect evidence, select the most likely root cause and choose the appropriate corrective action.

No incident has been investigated yet.
Engineering workflow: Establish a healthy baseline, reproduce the incident, collect evidence from endpoints and switches, correlate the observations, identify the fault domain, select a corrective action and verify the resulting network state. In a real production fabric, changes should be tested, approved and deployed using appropriate change controls.
☕
Support Network Insight
Help support the development of interactive networking labs, BGP simulations, and educational content.
Support the Project
Matt Conran
Latest posts by Matt Conran (see all)
Tags: No tags

Comments are closed.