GPU Cluster Networking
Engineering the AI Fabric.
Explore the network infrastructure that connects modern AI accelerators into distributed computing systems. Engineer InfiniBand and RoCEv2 fabrics, investigate Remote Direct Memory Access, understand GPU collective communication, and troubleshoot the congestion, latency and transport behaviour that determine distributed AI training and inference performance.
Multi-Rack GPU Fabric Explorer
Switch fabric modes, inspect devices, and introduce network faults to investigate how cluster communication is affected.
InfiniBand Fabric Architecture
InfiniBand is a high-performance switched fabric widely used in AI and HPC clusters. RDMA enables data transfer between registered memory regions with reduced CPU involvement in the data path.
- Subnet Manager discovers and configures the fabric
- Switch forwarding uses local identifiers (LIDs)
- Credit-based flow control protects virtual lanes
- GPU communication uses RDMA-capable network adapters
Fabric Engineering Telemetry
Educational architecture model. Topology, path state, fault effects and telemetry are simulated illustrations, not measurements from physical GPU hardware. The three modes represent alternative logical fabric views rather than automatically interchangeable physical networks.
Understand the Network Behind Distributed AI
A GPU cluster is more than a collection of accelerators. Its communication fabric determines how efficiently compute nodes exchange gradients, model parameters, activation tensors, checkpoint data and inference traffic. This workspace connects the physical network architecture to the behaviour of distributed workloads.
Design GPU Networks
Compare scale-up and scale-out communication, multi-rack topology, switch tiers, network rails, oversubscription and fault domains.
Understand RDMA
Explore queue pairs, completion queues, memory registration, RDMA reads and writes, transport operations and adapter behaviour.
Engineer InfiniBand Fabrics
Investigate Subnet Managers, LIDs, forwarding tables, virtual lanes, credit flow control and path selection.
Engineer RoCEv2
Study UDP/IP encapsulation, DSCP and priority mapping, PFC, ECN, DCQCN, congestion control and routing.
Optimize GPU Communication
Explore collective operations, NCCL, all-reduce, all-to-all, bandwidth, communication overlap and stragglers.
Troubleshoot Fabric Failures
Diagnose congestion, packet loss, pause propagation, degraded paths, adapter faults and workload slowdown.
Explore. Trace. Diagnose. Validate.
Progress from fundamental GPU fabric architecture through RDMA transport, InfiniBand and RoCEv2 engineering, into performance analysis and production incident investigation. Fifteen interactive labs across five learning stages culminate in a GPU cluster networking capstone.
GPU Cluster Networking Engineering Dashboard
Explore, simulate and troubleshoot high-performance GPU networking across InfiniBand, RoCEv2, RDMA, Ethernet fabrics, collective communication and production cluster operations. Navigate 15 interactive labs across five engineering stages, and track your progress as you advance.
Complete the labs in any order. Your completion choices are saved locally and synchronized with the five stage trackers.
0 of 15 labs complete
Complete the 15 engineering labs to finish the Network Insight GPU Cluster Networking Learning Workspace.
Completion is self-reported and saved in this browser. It is synchronized with the individual labs and stage trackers. Completing a lab does not require passing its engineering investigation.
Lab 01 — GPU Cluster Network Architecture Explorer
Design a multi-rack GPU cluster and investigate how accelerator count, network rails, switch connectivity, link bandwidth and fabric failures influence scale-out communication. Build the topology, trace inter-rack traffic and validate the resilience of your network architecture.
Fabric Configuration
Modify the GPU cluster topology and rebuild the fabric to investigate architecture trade-offs.
Dual-rail mode represents two logically separate network paths per rack. The simulator assumes one uplink per rack per rail for simplified capacity analysis.
Interactive GPU Fabric Topology
Select a switch or GPU rack to inspect its role.
Architecture Inspection
Select a switch or GPU rack to inspect its networking responsibilities and traffic paths.
- GPU racks connect through leaf switches
- Fabric rails provide independent logical paths
- Spine switches connect racks across the cluster
Capacity & Resilience Analysis
The current design uses dual-rail connectivity across three GPU racks.
Design, Trace and Troubleshoot the GPU Fabric
Use the topology controls to investigate how network design decisions influence connectivity, bandwidth and resilience. Complete the following engineering challenges before marking the lab complete.
Configure six racks with eight GPUs per rack. Determine the total accelerator count and the aggregate modeled rack uplink capacity.
Compare single-rail and dual-rail designs. Inject a link failure and determine whether an alternate rail remains available.
Trace inter-rack traffic through the fabric. Identify the source leaf, spine and destination leaf involved in the selected path.
Configure the cluster, investigate a failure, trace traffic and run validation to review your engineering progress.
Completion is self-reported and synchronized with the GPU Cluster Networking dashboard.
Educational topology model: capacity represents aggregate nominal rack uplink bandwidth, not measured application throughput. It excludes protocol overhead, GPU-to-NIC mapping, PCIe constraints, congestion and workload-specific communication patterns. Failure behaviour is modeled at the logical rail level.
Lab 02 — Scale-Up vs Scale-Out GPU Network Design
Compare high-bandwidth communication within GPU systems against RDMA-based communication between compute nodes. Configure the accelerator topology, select a scale-out fabric, trace traffic through network adapters and switches, and investigate how transport bandwidth and failures affect distributed workloads.
Communication Configuration
Adjust the architecture and inspect the modeled communication paths.
All bandwidth values are configurable educational assumptions. Local fabric capacity is not a claim about a particular GPU generation.
GPU Communication Architecture
Select components to inspect their function.
Architecture Inspection
Select a GPU, local fabric, network adapter or fabric switch to investigate the communication architecture.
- Local GPU traffic remains inside a compute node
- Remote communication traverses network adapters
- Fabric switches provide inter-node connectivity
Communication Performance Analysis
Two Communication Domains. One Distributed Workload.
Scale-Up — Within the GPU System
Scale-up fabrics connect accelerators within a supported GPU system or scale-up domain. NVLink and NVSwitch can provide high-bandwidth GPU communication without requiring the external Ethernet or InfiniBand fabric.
- GPU memory and accelerator interconnects
- NVLink / NVSwitch or other local interconnects
- High-bandwidth local tensor exchange
- Topology-dependent performance and capacity
Scale-Out — Between Compute Nodes
Scale-out fabrics connect separate GPU systems through RDMA-capable network adapters and switches. InfiniBand and RoCEv2 are common transport choices for distributed training and high-performance AI infrastructure.
- Inter-node and inter-rack communication
- InfiniBand or RoCEv2 Ethernet transport
- RDMA memory operations and collective traffic
- Congestion, routing and fabric fault domains
Explore Communication Boundaries
Build the architecture and complete the engineering investigations. Use the controls to compare local and remote communication, calculate transfer lower bounds and investigate network failures.
Trace communication through the local GPU fabric. Identify why the external network adapter is not required for this modeled path.
Compare 200 Gbps and 800 Gbps links with a 16 GB payload. Investigate the theoretical serialization-time difference.
Inject an inter-node network failure. Determine which communication domain is affected and whether local GPU communication remains possible.
Build the topology, trace both communication domains, compare link speeds and inject a network failure.
Self-reported progress is synchronized with the workspace dashboard.
Not marked complete
The transfer-time estimates are idealized serialization lower bounds: payload size divided by assumed usable link capacity. They do not model message latency, protocol overhead, PCIe bottlenecks, GPU memory behaviour, topology contention, collective algorithms or real hardware performance.
Lab 03 — GPU-to-GPU Communication and Traffic Flow
Trace distributed GPU communication from source memory through RDMA-capable network adapters, the scale-out fabric and destination memory. Explore point-to-point transfers, collective communication patterns, network congestion and transport failures that affect distributed AI workloads.
Communication Configuration
Configure a distributed GPU communication operation and investigate the resulting data path.
GPU Memory-to-Memory Traffic Explorer
Select components to inspect the communication path.
Communication Path Inspection
Select a GPU, memory region, RDMA adapter or fabric switch to inspect its role in the data path.
- Source data resides in GPU or host memory
- Registered memory is accessed through RDMA operations
- The network fabric forwards inter-node traffic
- Destination memory receives the transferred data
Distributed Communication Analysis
Trace, Analyze and Troubleshoot Distributed GPU Traffic
Complete the following activities using the interactive communication model. Investigate the difference between point-to-point transfers and collective operations, and identify how congestion and failures affect the GPU-to-GPU communication path.
Trace a point-to-point transfer and identify the source memory, source adapter, fabric switch, destination adapter and destination memory.
Compare All-Reduce and All-to-All operations. Investigate how the communication pattern changes with the number of participating GPUs.
Increase the fabric load to 95%, then inject a link failure. Explain the difference between congestion-related slowdown and an unavailable path.
Build the communication flow, trace the RDMA path, compare collective operations, test congestion and investigate a link failure.
Completion is self-reported and synchronized with the GPU Cluster Networking dashboard.
Not marked complete
This is a conceptual educational model. Collective communication is represented through simplified traffic patterns, not a complete NCCL algorithm implementation. Available-rate and transfer-time estimates are illustrative and do not represent measured GPU performance, actual fabric congestion-control behaviour or application-level throughput.
GPU Fabric Foundations — Completion Tracker
Review your progress across the first three GPU Cluster Networking engineering labs. Mark activities complete as you work through GPU cluster architecture, scale-up versus scale-out design, and distributed GPU communication.
GPU Cluster Network Architecture Explorer
Configure GPU racks, leaf-spine connectivity, network rails, uplink bandwidth and fabric failure scenarios.
Scale-Up vs Scale-Out GPU Network Design
Compare local NVLink/NVSwitch communication against inter-node InfiniBand and RoCEv2 scale-out networking.
GPU-to-GPU Communication and Traffic Flow
Trace RDMA data paths, investigate GPU memory communication, compare collective operations and diagnose fabric congestion.
Complete the three GPU Fabric Foundations labs to finish this learning stage.
Lab 04 — RDMA Operations and Memory Registration
Investigate the fundamental RDMA transport mechanisms used in high-performance GPU cluster networks. Register memory, configure queue pairs, post work requests, trace RDMA Read and Write operations, and inspect completion queues and memory protection failures in an interactive two-host architecture.
RDMA Transport Configuration
Configure the memory and transport state before posting RDMA operations.
Interactive RDMA Transport Architecture
Select a component to inspect its role.
RDMA Component Inspection
Select a memory region, queue pair, network adapter or fabric switch to investigate RDMA transport.
- Memory registration establishes access permissions
- Queue pairs contain send and receive work queues
- Work requests describe transport operations
- Completion queues report operation outcomes
RDMA Engineering Analysis
Initialize the environment to examine the selected RDMA transport operation.
Contains posted send-side work requests, including RDMA Read, RDMA Write and Send.
Contains receive buffers for incoming Send operations. RDMA Read and Write do not consume receive work requests.
Holds completion entries generated by signaled operations and supported receive events.
RDMA Transport Event Console
Investigate RDMA Memory Protection and Queue Processing
Use the simulator to compare one-sided and two-sided RDMA operations, observe queue processing, and diagnose a memory-protection failure.
Execute RDMA Write and RDMA Read with valid memory registration and permissions. Identify why the remote application does not need to post a receive request for these operations.
Execute Send with a posted receive buffer. Remove the receive work request and investigate why a two-sided transfer requires receive resources.
Inject an invalid remote key, execute an RDMA operation, and inspect the resulting protection error and completion status.
Initialize the RDMA engine, execute successful Read, Write and Send operations, poll completions and investigate a protection failure.
Self-reported completion is synchronized with the GPU Cluster Networking dashboard.
Not marked complete
This simulator models selected RDMA verbs concepts. In production, memory registration, queue-pair transitions, completion behaviour, access permissions, transport retries and error handling depend on the RDMA provider, adapter, transport and application implementation. The animation represents logical operations rather than measured packet-level behaviour.
Lab 05 — InfiniBand Fabric, Subnet Manager and LIDs
Explore how an InfiniBand subnet manager discovers a connected fabric, assigns Local Identifiers (LIDs), computes forwarding information and enables communication between GPU hosts. Investigate forwarding-table decisions, path selection, topology changes and subnet-management recovery.
InfiniBand Fabric Controls
Discover the fabric, assign LIDs and investigate the forwarding path between GPU hosts.
Interactive InfiniBand GPU Fabric
Select any host, switch or subnet manager to inspect its engineering role.
InfiniBand Fabric Inspection
Select a fabric component to investigate InfiniBand discovery, routing and subnet management.
- HCAs connect GPU hosts to the fabric
- Switches forward traffic using LFT entries
- The subnet manager discovers and configures the subnet
- LIDs identify ports within the InfiniBand subnet
Path and Subnet Analysis
Discover the fabric and assign LIDs before tracing host communication.
InfiniBand Linear Forwarding Table — LFT
Inspect the destination LID to output-port mapping for the selected switch. Entries are generated from a simplified deterministic shortest-path model.
| Destination | LID | Output Port | Next Hop | State |
|---|---|---|---|---|
| Run subnet discovery and LID assignment. | ||||
Subnet Manager Event Console
Discover, Route and Recover an InfiniBand GPU Fabric
Complete the following activities to understand subnet-manager responsibilities, LID-based forwarding and the operational consequences of fabric failures.
Discover the fabric, assign LIDs, inspect an HCA and identify how the subnet manager establishes forwarding information.
Trace communication between GPU Host 01 and GPU Host 03. Inspect the LFT at a leaf and spine switch to identify the forwarding path.
Fail an uplink, recompute forwarding and verify an alternate path. Stop the subnet manager and investigate the difference between management-plane availability and existing data-plane forwarding.
Discover the subnet, assign LIDs, inspect forwarding entries, trace traffic, investigate a link failure and recover the fabric.
Self-reported completion is synchronized with the GPU Cluster Networking dashboard.
Not marked complete
This lab uses a simplified InfiniBand subnet model. LIDs are assigned deterministically for educational purposes; switch forwarding entries are calculated using shortest paths with fixed tie-breaking. Production subnet managers may use different routing algorithms, partition policies, LMC configurations, switch-port addressing and failover mechanisms. Stopping the simulated subnet manager does not automatically disable previously configured data-plane forwarding.
Lab 06 — InfiniBand Routing, Virtual Lanes and Credits
Investigate how InfiniBand switches forward traffic through a leaf-spine fabric and how virtual lanes provide independent buffering and credit-based flow control. Configure Service Level mapping, generate traffic, inspect receive-buffer credits and explore congestion and head-of-line blocking.
InfiniBand Flow-Control Configuration
Configure routing, virtual lanes and the receive-buffer credit budget.
InfiniBand Routing and Credit Flow
Select a switch or host to inspect the routing and flow-control architecture.
InfiniBand Flow-Control Inspection
Select a host, leaf or spine switch to investigate routing, buffering and virtual lanes.
- Service Levels classify traffic
- SL-to-VL mapping selects a virtual lane
- Virtual lanes maintain separate buffering resources
- Credits regulate link-level transmission
Engineering Analysis
Initialize the simulation to examine the configured routing and credit model.
VL0 maintains its own modeled receive-buffer credit budget and traffic queue.
VL1 provides an independent buffering domain when the selected SL-to-VL mapping separates traffic.
InfiniBand Flow-Control Event Console
Investigate Credit Exhaustion and Virtual-Lane Isolation
Complete the following engineering activities to understand why credits, buffering and traffic separation matter in high-performance InfiniBand fabrics.
Configure two receive credits, generate a burst of eight packets and advance the simulator. Observe how available credits limit transmission and produce stalls.
Inject congestion into VL0, then compare separate and shared SL-to-VL mappings. Determine whether SL1 traffic can make progress independently.
Switch between Spine 01 and Spine 02, inspect the selected path, and drain receiver buffers to restore credits and forwarding progress.
Initialize the fabric, generate traffic, exhaust credits, investigate VL isolation, compare routes and restore forwarding.
Self-reported completion is synchronized with the GPU Cluster Networking dashboard.
Not marked complete
The simulator represents credits as a fixed number of equal-sized packet-buffer slots. Real InfiniBand flow control accounts for buffer capacity and link-level credit advertisement in implementation- specific units. Virtual lanes isolate buffer resources but do not guarantee performance without suitable arbitration, routing and traffic policies.
RDMA & InfiniBand — Completion Tracker
Review your progress across three advanced GPU networking labs. Track your understanding of RDMA memory access, InfiniBand subnet management, LID-based forwarding, virtual lanes and credit-based flow control.
RDMA Operations and Memory Registration
Configure registered memory, local and remote keys, queue pairs, RDMA Read and Write, Send/Receive and completion queues.
InfiniBand Fabric, Subnet Manager and LIDs
Discover a GPU fabric, assign LIDs, inspect switch forwarding tables, trace routes and recover subnet management after failures.
InfiniBand Routing, Virtual Lanes and Credits
Investigate SL-to-VL mapping, receive-buffer credits, credit exhaustion, congestion, head-of-line blocking and virtual-lane isolation.
Complete all three RDMA & InfiniBand labs to finish this learning stage.
Lab 07 — RoCEv2 Packet Encapsulation and Routing
Investigate how RDMA traffic is transported over Ethernet and routed IP networks using RoCEv2. Inspect packet encapsulation, DSCP and ECN fields, UDP destination port 4791, InfiniBand transport headers, IP forwarding and MTU-related failures in a GPU cluster leaf-spine fabric.
RoCEv2 Packet Configuration
Configure the packet and network before tracing traffic between GPU hosts.
Routed RoCEv2 GPU Fabric
Select a device to inspect the packet's forwarding behaviour.
Interactive RoCEv2 Packet Header Explorer
Select an encapsulation layer to inspect its fields and engineering role. The packet-size calculation uses a simplified transport header model.
Simplified example excludes optional transport extension headers, padding, ICRC, VLAN tags, Ethernet FCS and physical-layer overhead.
Packet and Fabric Inspection
Select a packet header or fabric device to investigate RoCEv2 encapsulation.
- RoCEv2 encapsulates RDMA transport over UDP/IP
- UDP destination port 4791 identifies RoCEv2 traffic
- DSCP and ECN occupy the IPv4 traffic-class field
- Routers forward according to destination IP
Packet Engineering Analysis
Build a packet and trace its forwarding path to examine routing and encapsulation.
RoCEv2 Packet Trace Console
Investigate RoCEv2 Encapsulation and Routed Forwarding
Complete the following activities to investigate packet encapsulation, forwarding and common network-layer failures affecting RoCEv2 fabrics.
Build a RoCEv2 packet, inspect Ethernet, IPv4, UDP and BTH fields, and identify how DSCP and ECN markings are represented.
Trace traffic through Spine 01 and Spine 02. Observe TTL processing and identify the destination IP used for routing.
Inject an MTU mismatch and TTL expiry. Explain why these failures can prevent successful GPU-to-GPU communication.
Build and inspect a packet, trace both routes, investigate MTU and TTL failures, and restore successful forwarding.
Self-reported completion is synchronized with the GPU Cluster Networking dashboard.
Not marked complete
This is a conceptual IPv4 RoCEv2 packet model. Real RoCEv2 packets include transport-specific headers and integrity fields not fully represented here. Path MTU handling, packetization, QoS classification and congestion control depend on the NIC, operating system, switches and RDMA stack.
Lab 08 — Priority Flow Control, ECN and DCQCN
Engineer RoCEv2 congestion behaviour in a GPU cluster Ethernet fabric. Configure switch buffer thresholds, generate competing traffic, investigate Priority Flow Control (PFC), observe Explicit Congestion Notification (ECN) and Congestion Notification Packets (CNPs), and examine DCQCN-inspired sender-rate adaptation.
Congestion-Control Configuration
Configure a simplified per-priority switch queue and sender.
RoCEv2 Congestion-Control Fabric
Follow data traffic, ECN marking, CNP feedback and PFC backpressure.
Congestion-Control Mechanisms
- ECN: Marks congestion-capable IP packets rather than relying solely on drops.
- CNP: Feedback from the receiving RNIC toward the sender following congestion notification.
- DCQCN: Adjusts sender transmission rate in response to congestion feedback.
- PFC: Pauses traffic for an Ethernet priority to protect buffer capacity.
- Risk: Excessive PFC can propagate backpressure and create head-of-line blocking.
Live Engineering Analysis
Initialize the simulator and advance traffic to inspect congestion behaviour.
RoCEv2 Congestion Event Console
Investigate Queue Growth, Congestion Feedback and Recovery
Run each scenario and inspect how ECN and PFC interact. This lab uses simplified feedback timing and sender adaptation rather than a hardware-accurate DCQCN implementation.
Enable ECN/DCQCN, configure sender demand above receiver capacity, and advance traffic until ECN and CNP events occur.
Switch to PFC-only mode, inject a bottleneck and observe queue growth, XOFF activation and subsequent XON recovery.
Compare operation without congestion control against combined ECN and PFC. Explain the consequences for drops, pauses and sender rates.
Initialize the fabric, observe ECN/CNP, trigger PFC, compare modes and restore receiver capacity.
Self-reported completion synchronizes with the GPU Cluster Networking dashboard.
Not marked complete
Engineering note: The simulation uses abstract traffic units per step and a simplified deterministic ECN/CNP loop. Real implementations use packet-level marking, feedback timing, rate recovery algorithms, PFC headroom, link propagation delay, priority mapping and vendor-specific hardware behaviour. PFC is not a guarantee of end-to-end losslessness.
Lab 09 — Lossless Ethernet Fabric Design
Design a RoCEv2-ready Ethernet leaf-spine network for GPU clusters. Evaluate uplink oversubscription, priority mapping, PFC buffer headroom, ECN/PFC threshold ordering, link-failure capacity and operational risks before approving a production fabric design.
Production Fabric Configuration
Configure a representative GPU leaf with four 400 Gb/s server-facing ports.
GPU Cluster Ethernet Fabric
Illustrative two-leaf, four-spine design with configurable active uplink capacity.
Engineering Design Principles
- Plan for traffic locality, all-to-all communication and incast, not just average utilization.
- Maintain consistent DSCP-to-queue and priority mappings across the RoCEv2 path.
- Configure ECN to provide early congestion feedback before severe queue pressure.
- Reserve sufficient PFC headroom for in-flight data after pause generation.
- Validate capacity after uplink failures and avoid prolonged PFC pause propagation.
Live Fabric Engineering Analysis
Evaluate the fabric to calculate capacity, thresholds and headroom.
Production Readiness Assessment
Each check is calculated from the current design. Passing these simplified checks does not certify a real fabric.
GPU Ethernet Fabric Event Console
Design and Validate a Production RoCEv2 Fabric
Compare two and four uplinks per leaf. Investigate how offered GPU load affects oversubscription.
Increase the reaction-time budget and observe how required headroom changes. Adjust buffers and XOFF thresholds.
Inject an uplink failure, examine reduced capacity, restore connectivity and apply a resilient baseline.
Evaluate the design, trace traffic, compare capacity, inject failure, restore links and examine headroom.
Self-reported completion synchronizes with the GPU Cluster Networking dashboard.
Not marked complete
The headroom estimate uses 400 Gb/s × reaction time plus 20% illustrative margin. Real buffer engineering requires hardware-specific cell accounting, pause generation and propagation delays, cable lengths, NIC behaviour, scheduling, shared-buffer architecture and measured traffic bursts. Ethernet PFC mitigates selected loss mechanisms; it does not make a network universally lossless.
RoCEv2 & Ethernet — Completion Tracker
Review your progress across three advanced RoCEv2 and Ethernet engineering labs. Track your understanding of packet encapsulation, routed RDMA traffic, PFC, ECN, DCQCN, QoS, buffer headroom and production fabric design.
RoCEv2 Packet Encapsulation and Routing
Explore Ethernet, IPv4, UDP port 4791, InfiniBand transport headers, DSCP/ECN, path MTU, TTL and GPU-to-GPU routing.
Priority Flow Control, ECN and DCQCN
Investigate RoCEv2 congestion behaviour, switch queues, ECN marking, CNP feedback, sender-rate adaptation, PFC XOFF/XON and congestion recovery.
Lossless Ethernet Fabric Design
Engineer a GPU leaf-spine fabric with oversubscription planning, QoS priorities, ECN thresholds, PFC headroom, resilience and production readiness checks.
Complete all three RoCEv2 & Ethernet labs to finish this learning stage.
Lab 10 — Collective Communication and NCCL
Investigate how distributed AI training workloads coordinate communication across GPU accelerators. Compare Ring AllReduce, Tree AllReduce, AllGather and ReduceScatter operations. Examine how GPU interconnect bandwidth, Ethernet or InfiniBand fabric capacity, message size and communication algorithms influence distributed training performance.
Collective Communication Engine
Configure an eight-GPU collective spanning two accelerator nodes.
Eight-GPU Collective Communication Fabric
Two GPU nodes, local accelerator links and a shared inter-node RDMA fabric.
Collective Communication Architecture
- AllReduce: combines values across GPUs and distributes the result.
- AllGather: distributes input partitions to all participants.
- ReduceScatter: reduces data and distributes result partitions.
- Ring: exchanges chunks between neighboring participants.
- Tree: organizes communication into hierarchical reduction or distribution.
Live Performance Analysis
Build a collective communication topology to calculate estimated communication cost.
NCCL-Inspired Communication Trace
Investigate GPU Collective Communication Performance
Explore communication algorithms and determine how inter-node network bandwidth affects collective performance.
Build and run Ring AllReduce. Inspect the reduce-scatter and all-gather communication phases.
Compare ring and tree communication for the same message size and fabric.
Reduce inter-node bandwidth, observe the estimated communication penalty and restore the original fabric.
Build and run a collective, compare algorithms, inject a network bottleneck and restore capacity.
Self-reported completion synchronizes with the GPU Cluster Networking dashboard.
Not marked complete
This simulator uses simplified analytical communication-cost models for educational comparison. Real NCCL implementations select algorithms and protocols based on topology, GPU architecture, message size, network transport, collective type and runtime tuning. The displayed performance estimates are illustrative rather than benchmark results.
Lab 11 — ECMP, Congestion and Adaptive Routing
Investigate how GPU cluster Ethernet and RDMA fabrics distribute traffic across equal-cost spine paths. Compare deterministic ECMP hashing with a simplified adaptive flow-placement policy, inject elephant flows, inspect congestion hotspots and evaluate the effect of link failures on GPU-to-GPU communication.
Fabric Routing Configuration
Configure the routing policy and GPU traffic demand.
GPU Cluster Four-Spine Fabric
Two leaf switches, eight GPU endpoints and four equal-cost routed paths.
ECMP and Adaptive Routing
- ECMP: Hashes each flow to an available equal-cost next hop.
- Hash collision: Multiple large flows may land on the same spine path.
- Adaptive routing: May use congestion or load information to choose a path.
- Flow ordering: Moving established flows can introduce packet reordering.
- Failure recovery: Link failures change the set of available paths and can create new hotspots.
Live Fabric Analysis
Build the fabric and route traffic to inspect path utilization.
GPU Fabric Routing Event Console
Investigate Flow Hashing, Congestion and Path Recovery
Build an ECMP fabric, route eight GPU flows and inspect distribution across the four spine paths.
Inject a large flow, compare ECMP and adaptive placement and investigate any congestion hotspots.
Fail a busy spine path, observe traffic redistribution and restore the fabric.
Route traffic, inject an elephant flow, compare policies, fail a path and restore the fabric.
Self-reported completion synchronizes with the GPU Cluster Networking dashboard.
Not marked complete
This lab uses an educational aggregate-demand model. Adaptive routing is represented by least-loaded placement of new flows, not packet spraying or a vendor-specific implementation. Offered traffic exceeding path capacity is reported as excess demand; actual packet loss depends on queues, buffering, congestion control and traffic timing.
Lab 12 — GPU Fabric Performance and Bottleneck Analysis
Investigate the complete GPU-to-GPU data path from source accelerator memory through PCIe, the RDMA NIC, Ethernet or InfiniBand fabric and the destination GPU. Identify bandwidth bottlenecks, estimate transfer latency and investigate how incast traffic and fabric oversubscription affect distributed AI workloads.
Performance Investigation Controls
Configure the GPU communication path and workload demand.
GPU-to-GPU RDMA Performance Path
Accelerator → PCIe → RNIC → Fabric → RNIC → GPU
Performance Engineering Principles
- End-to-end throughput cannot exceed the slowest effective path component.
- PCIe bandwidth can limit GPU Direct RDMA even when network links are faster.
- NIC line rate and application throughput differ because of protocol and system overhead.
- Incast concentrates multiple senders on shared receiver and fabric resources.
- Network congestion can introduce queueing delay, ECN feedback and PFC activity.
- Use hardware counters and real benchmarks to validate a production diagnosis.
Live Bottleneck Analysis
Evaluate the GPU fabric to inspect throughput and latency estimates.
GPU Fabric Performance Investigation Console
Identify and Resolve GPU Communication Bottlenecks
Evaluate the baseline and trace the GPU-to-GPU RDMA data path.
Inject multiple concurrent senders and inspect receiver contention and effective throughput.
Inject a fabric bandwidth fault, identify the bottleneck and apply an optimized configuration.
Evaluate, trace, inject incast, introduce a fabric fault and optimize the path.
Self-reported completion synchronizes with the GPU Cluster Networking dashboard.
Not marked complete
This simulator estimates steady-state throughput using a minimum-capacity path model and simplified equal-share incast contention. Latency is estimated from a configured base delay plus transfer serialization. It does not simulate PCIe transaction details, actual GPU memory throughput, transport protocol overhead, switch queues or hardware congestion control. All results are illustrative.
GPU Fabric Performance — Completion Tracker
Review your progress across three advanced GPU fabric performance engineering labs. Track your understanding of collective communication, NCCL, ECMP routing, congestion, adaptive path selection, GPU-to-GPU latency, throughput and bottleneck analysis.
Collective Communication and NCCL
Investigate AllReduce, AllGather, ReduceScatter, ring and tree algorithms, GPU topology, inter-node bandwidth and distributed training communication performance.
ECMP, Congestion and Adaptive Routing
Engineer a four-spine GPU cluster fabric, examine flow hashing and elephant flows, compare routing policies, investigate congestion hotspots and recover from failures.
GPU Fabric Performance and Bottleneck Analysis
Analyze GPU Direct RDMA performance across PCIe, RNICs and the leaf-spine fabric. Identify throughput constraints, latency, incast contention and network bottlenecks.
Complete all three GPU Fabric Performance labs to finish this learning stage.
Lab 13 — RDMA Latency and Packet-Loss Investigation
Investigate production GPU cluster communication failures using switch and RNIC telemetry. Analyze congestion, ECN marking, Priority Flow Control, path MTU mismatches, physical link errors and receiver saturation. Correlate counters and latency changes to identify root causes and evaluate remediation strategies.
Production Incident Controls
Select an incident, inspect the live fabric, run traffic and identify the underlying failure.
Production GPU-to-GPU RDMA Fabric
Source RNIC → Leaf → Spine → Leaf → Destination RNIC
Latency Trend — Recent Simulation Steps
RDMA Troubleshooting Method
- Establish baseline latency and throughput.
- Inspect switch egress queues and ECN CE counters.
- Correlate PFC pause activity with congestion.
- Verify consistent path MTU across RDMA interfaces.
- Inspect CRC/FCS errors and physical link health.
- Check receiver RNIC utilization and resource limits.
- Apply remediation and verify counter trends.
Live Incident Analysis
Load an incident and run traffic to generate diagnostic telemetry.
RDMA Production Incident Console
Diagnose and Recover a Production RDMA Fabric
Load the healthy fabric and establish baseline latency and throughput.
Inject a fault, run traffic, inspect counters and submit the correct root cause.
Apply remediation and verify that the fabric returns to healthy operation.
Establish a baseline, diagnose a fault, inspect counters and recover the fabric.
Self-reported completion synchronizes with the GPU Cluster Networking dashboard.
Not marked complete
Educational discrete-time incident model. Counters, latency, congestion and throughput are illustrative rather than hardware-accurate. In production, correlate RNIC statistics, switch queue telemetry, interface error counters, ECN/PFC measurements and application-level benchmarks. PFC is link-local flow control; ECN provides congestion marking, while RoCEv2 congestion-control implementations may use receiver-generated CNP feedback.
Lab 14 — GPU Cluster Fabric Failure and Recovery
Investigate how high-performance GPU clusters maintain connectivity when network components fail. Inject spine and leaf failures, inspect available ECMP paths, observe communication disruption, advance through reconvergence and recovery, and evaluate the resilience of the GPU fabric.
Fabric Failure Controls
Configure the topology, inject failures and advance the recovery process.
Multi-Rack GPU Cluster High-Availability Fabric
Two leaf switches, four spines and eight GPU endpoints.
Traffic Availability — Recovery Timeline
High-Availability Engineering Principles
- ECMP distributes traffic across available equal-cost next hops.
- Spine redundancy preserves alternate paths when a spine fails.
- Reconvergence depends on failure detection and forwarding updates.
- Leaf failures can isolate directly attached GPU endpoints without a redundant attachment design.
- Reduced path capacity can create congestion even after routing recovers.
- Recovery should be evaluated using service impact as well as control-plane convergence.
Live Failure Analysis
Build the fabric and inject a failure to inspect resilience.
GPU Fabric Failure and Recovery Console
Investigate GPU Fabric Resilience and Recovery
Build the healthy fabric and inspect the four available spine paths.
Inject a spine or leaf failure and investigate service disruption and reconvergence.
Complete the recovery process, compare capacity and restore failed components.
Build the fabric, inject a failure, complete recovery, compare capacity and restore components.
Self-reported completion synchronizes with the GPU Cluster Networking dashboard.
Not marked complete
This simulator uses a simplified recovery state machine and aggregate path-capacity model. Actual GPU fabric resilience depends on physical topology, routing protocols, BFD, ECMP implementation, host networking, RNIC behavior and application-level recovery. Recovery-time values are illustrative rather than vendor-specific guarantees.
Lab 15 — Production GPU Cluster Design and Troubleshooting Capstone
Engineer and troubleshoot a production-style 16-GPU cluster. Choose InfiniBand or RoCEv2, evaluate network capacity, inspect collective communication performance, inject incidents, correlate telemetry, identify root causes and restore the fabric to a healthy operating state.
Production Engineering Console
Configure the cluster, inject a fault and complete the investigation.
16-GPU Production Leaf-Spine Architecture
Four GPU racks, four leaves, four spines and an RDMA transport.
Production GPU Fabric Engineering
- Design the GPU cluster around workload communication patterns.
- Understand the differences between InfiniBand and RoCEv2.
- Size leaf-spine paths against expected aggregate demand.
- Evaluate collective communication and network bottlenecks.
- Use transport-appropriate congestion telemetry.
- Diagnose failures using evidence rather than assumptions.
- Verify recovery using service-level performance measurements.
Live Capstone Assessment
Deploy the GPU cluster to begin the engineering assessment.
Production GPU Cluster Incident Console
Complete the Production GPU Cluster Incident
Deploy a 16-GPU cluster, select its RDMA transport and establish a performance baseline.
Inject a production fault, inspect telemetry and identify the correct root cause.
Apply remediation, verify service recovery and review the engineering score.
Complete design, baseline, incident investigation, RCA, remediation and recovery verification.
Self-reported completion synchronizes with the GPU Cluster Networking dashboard.
Not marked complete
Educational capstone simulator. Telemetry, collective latency and capacity calculations are simplified and illustrative. The congestion-control behavior of InfiniBand differs from RoCEv2 Ethernet; ECN/PFC counters are shown only for the Ethernet transport in this model. Assessment score is local to this lab and is not a professional certification.
Production Troubleshooting — Completion Tracker
Complete the final three engineering labs to finish the GPU Cluster Networking Learning Workspace. Review your progress across RDMA incident investigation, fabric resilience and recovery, and the integrated production GPU cluster engineering capstone.
Complete Labs 13, 14 and 15 to finish Production Troubleshooting Engineering.
RDMA Latency and Packet-Loss Investigation
Diagnose congestion, PFC pauses, ECN marking, path MTU mismatches, physical link errors, RNIC saturation and GPU-to-GPU latency problems.
GPU Cluster Fabric Failure and Recovery
Investigate spine failures, leaf-switch outages, redundant ECMP paths, capacity degradation, convergence timing and GPU communication recovery.
Production GPU Cluster Design and Troubleshooting
Engineer and troubleshoot an integrated 16-GPU production cluster using InfiniBand or RoCEv2, collective communication, fabric telemetry, root-cause analysis and recovery verification.
Complete all three Production Troubleshooting labs to finish the final engineering stage.