NETWORK INSIGHT / LEARNING WORKSPACE 12

GPU Cluster Networking
Engineering the AI Fabric.

Explore the network infrastructure that connects modern AI accelerators into distributed computing systems. Engineer InfiniBand and RoCEv2 fabrics, investigate Remote Direct Memory Access, understand GPU collective communication, and troubleshoot the congestion, latency and transport behaviour that determine distributed AI training and inference performance.

INFINIBAND ROCEV2 RDMA GPU FABRICS NCCL PFC / ECN / DCQCN MULTI-RAIL NETWORKING PRODUCTION TROUBLESHOOTING
15 Interactive engineering labs
5 Progressive learning stages
3 Fabric architecture scenarios
1 Production engineering capstone
INTERACTIVE ENGINEERING ARCHITECTURE

Multi-Rack GPU Fabric Explorer

Switch fabric modes, inspect devices, and introduce network faults to investigate how cluster communication is affected.

SPINE / CORE FABRIC
LEAF / TOP-OF-RACK
ACCELERATOR COMPUTE RACKS
GPU RACK A
Compute Pod 01
GPU RACK B
Compute Pod 02
GPU RACK C
Compute Pod 03
Scale-out fabric Intra-node GPU interconnect Storage traffic

InfiniBand Fabric Architecture

InfiniBand is a high-performance switched fabric widely used in AI and HPC clusters. RDMA enables data transfer between registered memory regions with reduced CPU involvement in the data path.

  • Subnet Manager discovers and configures the fabric
  • Switch forwarding uses local identifiers (LIDs)
  • Credit-based flow control protects virtual lanes
  • GPU communication uses RDMA-capable network adapters

Fabric Engineering Telemetry

Selected fabric InfiniBand
Fabric health Healthy
Active uplinks 6 / 6
Fault scenario None
Illustrative path state Redundant

Educational architecture model. Topology, path state, fault effects and telemetry are simulated illustrations, not measurements from physical GPU hardware. The three modes represent alternative logical fabric views rather than automatically interchangeable physical networks.

ENGINEERING LEARNING OBJECTIVES

Understand the Network Behind Distributed AI

A GPU cluster is more than a collection of accelerators. Its communication fabric determines how efficiently compute nodes exchange gradients, model parameters, activation tensors, checkpoint data and inference traffic. This workspace connects the physical network architecture to the behaviour of distributed workloads.

01 / FABRIC ARCHITECTURE

Design GPU Networks

Compare scale-up and scale-out communication, multi-rack topology, switch tiers, network rails, oversubscription and fault domains.

02 / MEMORY TRANSPORT

Understand RDMA

Explore queue pairs, completion queues, memory registration, RDMA reads and writes, transport operations and adapter behaviour.

03 / INFINIBAND

Engineer InfiniBand Fabrics

Investigate Subnet Managers, LIDs, forwarding tables, virtual lanes, credit flow control and path selection.

04 / ETHERNET RDMA

Engineer RoCEv2

Study UDP/IP encapsulation, DSCP and priority mapping, PFC, ECN, DCQCN, congestion control and routing.

05 / DISTRIBUTED COMPUTE

Optimize GPU Communication

Explore collective operations, NCCL, all-reduce, all-to-all, bandwidth, communication overlap and stragglers.

06 / PRODUCTION OPERATIONS

Troubleshoot Fabric Failures

Diagnose congestion, packet loss, pause propagation, degraded paths, adapter faults and workload slowdown.

THE NETWORK INSIGHT METHOD

Explore. Trace. Diagnose. Validate.

Progress from fundamental GPU fabric architecture through RDMA transport, InfiniBand and RoCEv2 engineering, into performance analysis and production incident investigation. Fifteen interactive labs across five learning stages culminate in a GPU cluster networking capstone.

NETWORK INSIGHT / GPU CLUSTER NETWORKING / LEARNING DASHBOARD V2

GPU Cluster Networking Engineering Dashboard

Explore, simulate and troubleshoot high-performance GPU networking across InfiniBand, RoCEv2, RDMA, Ethernet fabrics, collective communication and production cluster operations. Navigate 15 interactive labs across five engineering stages, and track your progress as you advance.

15 INTERACTIVE LABS 5 ENGINEERING STAGES INFINIBAND RoCEv2 RDMA NCCL GPU FABRIC PERFORMANCE PRODUCTION TROUBLESHOOTING
GPU Cluster Networking — Overall Learning Progress

Complete the labs in any order. Your completion choices are saved locally and synchronized with the five stage trackers.

0 of 15 labs complete

0% WORKSPACE COMPLETION
Engineering Labs 15
Completed Labs 0
Remaining Labs 15
Completed Stages 0 / 5
STAGE 01 GPU Fabric Foundations
0 / 3 Labs
STAGE 02 RDMA & InfiniBand
0 / 3 Labs
STAGE 03 RoCEv2 & Ethernet
0 / 3 Labs
STAGE 04 GPU Fabric Performance
0 / 3 Labs
STAGE 05 Production Troubleshooting
0 / 3 Labs
Showing 15 engineering labs Choose a lab to open its interactive engineering environment.
No engineering labs match the current search or filter.
NI
GPU Cluster Networking — Learning in Progress

Complete the 15 engineering labs to finish the Network Insight GPU Cluster Networking Learning Workspace.

Completion is self-reported and saved in this browser. It is synchronized with the individual labs and stage trackers. Completing a lab does not require passing its engineering investigation.

NETWORK INSIGHT / GPU CLUSTER NETWORKING / STAGE 01

Lab 01 — GPU Cluster Network Architecture Explorer

Design a multi-rack GPU cluster and investigate how accelerator count, network rails, switch connectivity, link bandwidth and fabric failures influence scale-out communication. Build the topology, trace inter-rack traffic and validate the resilience of your network architecture.

ARCHITECTURE DESIGN MULTI-RACK GPU CLUSTERS LEAF-SPINE MULTI-RAIL CAPACITY PLANNING FAILURE INVESTIGATION

Fabric Configuration

Modify the GPU cluster topology and rebuild the fabric to investigate architecture trade-offs.

Dual-rail mode represents two logically separate network paths per rack. The simulator assumes one uplink per rack per rail for simplified capacity analysis.

Interactive GPU Fabric Topology

Select a switch or GPU rack to inspect its role.

Fabric Healthy
SPINE / FABRIC CORE
LEAF / RACK NETWORK
GPU COMPUTE RACKS
Rail A Rail B Active trace Failed link
Total GPUs 12
Fabric Rails 2
Aggregate Rack Uplink Capacity 2.4 Tbps
Available Uplinks 6 / 6
Architecture Inspection

Select a switch or GPU rack to inspect its networking responsibilities and traffic paths.

  • GPU racks connect through leaf switches
  • Fabric rails provide independent logical paths
  • Spine switches connect racks across the cluster
Capacity & Resilience Analysis

The current design uses dual-rail connectivity across three GPU racks.

    ENGINEERING INVESTIGATION

    Design, Trace and Troubleshoot the GPU Fabric

    Use the topology controls to investigate how network design decisions influence connectivity, bandwidth and resilience. Complete the following engineering challenges before marking the lab complete.

    CHALLENGE 01 — SCALE

    Configure six racks with eight GPUs per rack. Determine the total accelerator count and the aggregate modeled rack uplink capacity.

    CHALLENGE 02 — REDUNDANCY

    Compare single-rail and dual-rail designs. Inject a link failure and determine whether an alternate rail remains available.

    CHALLENGE 03 — TRAFFIC PATH

    Trace inter-rack traffic through the fabric. Identify the source leaf, spine and destination leaf involved in the selected path.

    Investigation Not Yet Validated

    Configure the cluster, investigate a failure, trace traffic and run validation to review your engineering progress.

    Lab 01 — Learning Completion

    Completion is self-reported and synchronized with the GPU Cluster Networking dashboard.

    Not marked complete

    Educational topology model: capacity represents aggregate nominal rack uplink bandwidth, not measured application throughput. It excludes protocol overhead, GPU-to-NIC mapping, PCIe constraints, congestion and workload-specific communication patterns. Failure behaviour is modeled at the logical rail level.

    NETWORK INSIGHT / GPU CLUSTER NETWORKING / STAGE 01

    Lab 02 — Scale-Up vs Scale-Out GPU Network Design

    Compare high-bandwidth communication within GPU systems against RDMA-based communication between compute nodes. Configure the accelerator topology, select a scale-out fabric, trace traffic through network adapters and switches, and investigate how transport bandwidth and failures affect distributed workloads.

    NVLINK / NVSWITCH SCALE-UP SCALE-OUT INFINIBAND ROCEV2 RDMA COMMUNICATION PERFORMANCE

    Communication Configuration

    Adjust the architecture and inspect the modeled communication paths.

    All bandwidth values are configurable educational assumptions. Local fabric capacity is not a claim about a particular GPU generation.

    GPU Communication Architecture

    Select components to inspect their function.

    Fabric Healthy
    Local GPU fabric RDMA scale-out fabric Failed network path
    Total GPUs 16
    Scale-Out Link 400 Gbps
    Local Transfer Lower Bound 4.44 ms
    Scale-Out Transfer Lower Bound 80 ms
    Architecture Inspection

    Select a GPU, local fabric, network adapter or fabric switch to investigate the communication architecture.

    • Local GPU traffic remains inside a compute node
    • Remote communication traverses network adapters
    • Fabric switches provide inter-node connectivity
    Communication Performance Analysis

      ENGINEERING ARCHITECTURE COMPARISON

      Two Communication Domains. One Distributed Workload.

      Scale-Up — Within the GPU System

      Scale-up fabrics connect accelerators within a supported GPU system or scale-up domain. NVLink and NVSwitch can provide high-bandwidth GPU communication without requiring the external Ethernet or InfiniBand fabric.

      • GPU memory and accelerator interconnects
      • NVLink / NVSwitch or other local interconnects
      • High-bandwidth local tensor exchange
      • Topology-dependent performance and capacity
      Scale-Out — Between Compute Nodes

      Scale-out fabrics connect separate GPU systems through RDMA-capable network adapters and switches. InfiniBand and RoCEv2 are common transport choices for distributed training and high-performance AI infrastructure.

      • Inter-node and inter-rack communication
      • InfiniBand or RoCEv2 Ethernet transport
      • RDMA memory operations and collective traffic
      • Congestion, routing and fabric fault domains
      ENGINEERING INVESTIGATION

      Explore Communication Boundaries

      Build the architecture and complete the engineering investigations. Use the controls to compare local and remote communication, calculate transfer lower bounds and investigate network failures.

      CHALLENGE 01 — LOCAL GPU COMMUNICATION

      Trace communication through the local GPU fabric. Identify why the external network adapter is not required for this modeled path.

      CHALLENGE 02 — SCALE-OUT PERFORMANCE

      Compare 200 Gbps and 800 Gbps links with a 16 GB payload. Investigate the theoretical serialization-time difference.

      CHALLENGE 03 — FAILURE DOMAIN

      Inject an inter-node network failure. Determine which communication domain is affected and whether local GPU communication remains possible.

      Investigation Not Yet Validated

      Build the topology, trace both communication domains, compare link speeds and inject a network failure.

      Lab 02 — Learning Completion

      Self-reported progress is synchronized with the workspace dashboard.

      Not marked complete

      The transfer-time estimates are idealized serialization lower bounds: payload size divided by assumed usable link capacity. They do not model message latency, protocol overhead, PCIe bottlenecks, GPU memory behaviour, topology contention, collective algorithms or real hardware performance.

      NETWORK INSIGHT / GPU CLUSTER NETWORKING / STAGE 01

      Lab 03 — GPU-to-GPU Communication and Traffic Flow

      Trace distributed GPU communication from source memory through RDMA-capable network adapters, the scale-out fabric and destination memory. Explore point-to-point transfers, collective communication patterns, network congestion and transport failures that affect distributed AI workloads.

      GPU MEMORY RDMA TRANSPORT INFINIBAND ROCEV2 NCCL ALL-REDUCE ALL-TO-ALL TRAFFIC TRACING

      Communication Configuration

      Configure a distributed GPU communication operation and investigate the resulting data path.

      GPU Memory-to-Memory Traffic Explorer

      Select components to inspect the communication path.

      Fabric Healthy
      GPU memory path RDMA fabric path Failed link
      Participating GPUs 4
      Nominal Link Speed 400 Gbps
      Illustrative Available Rate 200 Gbps
      Transfer Time Estimate 160 ms
      Communication Path Inspection

      Select a GPU, memory region, RDMA adapter or fabric switch to inspect its role in the data path.

      • Source data resides in GPU or host memory
      • Registered memory is accessed through RDMA operations
      • The network fabric forwards inter-node traffic
      • Destination memory receives the transferred data
      Distributed Communication Analysis

        ENGINEERING INVESTIGATION

        Trace, Analyze and Troubleshoot Distributed GPU Traffic

        Complete the following activities using the interactive communication model. Investigate the difference between point-to-point transfers and collective operations, and identify how congestion and failures affect the GPU-to-GPU communication path.

        CHALLENGE 01 — RDMA PATH

        Trace a point-to-point transfer and identify the source memory, source adapter, fabric switch, destination adapter and destination memory.

        CHALLENGE 02 — COLLECTIVE COMMUNICATION

        Compare All-Reduce and All-to-All operations. Investigate how the communication pattern changes with the number of participating GPUs.

        CHALLENGE 03 — CONGESTION & FAILURE

        Increase the fabric load to 95%, then inject a link failure. Explain the difference between congestion-related slowdown and an unavailable path.

        Investigation Not Yet Validated

        Build the communication flow, trace the RDMA path, compare collective operations, test congestion and investigate a link failure.

        Lab 03 — Learning Completion

        Completion is self-reported and synchronized with the GPU Cluster Networking dashboard.

        Not marked complete

        This is a conceptual educational model. Collective communication is represented through simplified traffic patterns, not a complete NCCL algorithm implementation. Available-rate and transfer-time estimates are illustrative and do not represent measured GPU performance, actual fabric congestion-control behaviour or application-level throughput.

        NETWORK INSIGHT / WORKSPACE 12 / STAGE 01

        GPU Fabric Foundations — Completion Tracker

        Review your progress across the first three GPU Cluster Networking engineering labs. Mark activities complete as you work through GPU cluster architecture, scale-up versus scale-out design, and distributed GPU communication.

        Stage 1 Learning Progress 0 of 3 labs complete
        Complete all three activities to finish Stage 1. You can change your completion status at any time.
        0% Stage Completion
        LAB 01 Not Completed

        GPU Cluster Network Architecture Explorer

        Configure GPU racks, leaf-spine connectivity, network rails, uplink bandwidth and fabric failure scenarios.

        LAB 02 Not Completed

        Scale-Up vs Scale-Out GPU Network Design

        Compare local NVLink/NVSwitch communication against inter-node InfiniBand and RoCEv2 scale-out networking.

        LAB 03 Not Completed

        GPU-to-GPU Communication and Traffic Flow

        Trace RDMA data paths, investigate GPU memory communication, compare collective operations and diagnose fabric congestion.

        01
        Stage 1 — In Progress

        Complete the three GPU Fabric Foundations labs to finish this learning stage.

        NETWORK INSIGHT / GPU CLUSTER NETWORKING / STAGE 02

        Lab 04 — RDMA Operations and Memory Registration

        Investigate the fundamental RDMA transport mechanisms used in high-performance GPU cluster networks. Register memory, configure queue pairs, post work requests, trace RDMA Read and Write operations, and inspect completion queues and memory protection failures in an interactive two-host architecture.

        RDMA VERBS MEMORY REGIONS LKEY / RKEY QUEUE PAIRS SEND / RECEIVE RDMA READ RDMA WRITE COMPLETION QUEUES

        RDMA Transport Configuration

        Configure the memory and transport state before posting RDMA operations.

        Interactive RDMA Transport Architecture

        Select a component to inspect its role.

        QP Ready
        Memory / local path RDMA transport Protection error
        Queue Pair State RTS
        Work Requests Posted 0
        Pending CQ Entries 0
        Last Operation Idle
        RDMA Component Inspection

        Select a memory region, queue pair, network adapter or fabric switch to investigate RDMA transport.

        • Memory registration establishes access permissions
        • Queue pairs contain send and receive work queues
        • Work requests describe transport operations
        • Completion queues report operation outcomes
        RDMA Engineering Analysis

        Initialize the environment to examine the selected RDMA transport operation.

          Send Queue — SQ
          0 pending

          Contains posted send-side work requests, including RDMA Read, RDMA Write and Send.

          Receive Queue — RQ
          1 posted

          Contains receive buffers for incoming Send operations. RDMA Read and Write do not consume receive work requests.

          Completion Queue — CQ
          0 completions

          Holds completion entries generated by signaled operations and supported receive events.

          RDMA Transport Event Console
          [RDMA ENGINE] Ready to initialize.
          ENGINEERING INVESTIGATION

          Investigate RDMA Memory Protection and Queue Processing

          Use the simulator to compare one-sided and two-sided RDMA operations, observe queue processing, and diagnose a memory-protection failure.

          CHALLENGE 01 — ONE-SIDED RDMA

          Execute RDMA Write and RDMA Read with valid memory registration and permissions. Identify why the remote application does not need to post a receive request for these operations.

          CHALLENGE 02 — SEND / RECEIVE

          Execute Send with a posted receive buffer. Remove the receive work request and investigate why a two-sided transfer requires receive resources.

          CHALLENGE 03 — MEMORY PROTECTION

          Inject an invalid remote key, execute an RDMA operation, and inspect the resulting protection error and completion status.

          Investigation Not Yet Validated

          Initialize the RDMA engine, execute successful Read, Write and Send operations, poll completions and investigate a protection failure.

          Lab 04 — Learning Completion

          Self-reported completion is synchronized with the GPU Cluster Networking dashboard.

          Not marked complete

          This simulator models selected RDMA verbs concepts. In production, memory registration, queue-pair transitions, completion behaviour, access permissions, transport retries and error handling depend on the RDMA provider, adapter, transport and application implementation. The animation represents logical operations rather than measured packet-level behaviour.

          NETWORK INSIGHT / GPU CLUSTER NETWORKING / STAGE 02

          Lab 05 — InfiniBand Fabric, Subnet Manager and LIDs

          Explore how an InfiniBand subnet manager discovers a connected fabric, assigns Local Identifiers (LIDs), computes forwarding information and enables communication between GPU hosts. Investigate forwarding-table decisions, path selection, topology changes and subnet-management recovery.

          INFINIBAND SUBNET MANAGER LID ASSIGNMENT LFT HCA FABRIC DISCOVERY PATH TRACING FAILURE RECOVERY

          InfiniBand Fabric Controls

          Discover the fabric, assign LIDs and investigate the forwarding path between GPU hosts.

          Interactive InfiniBand GPU Fabric

          Select any host, switch or subnet manager to inspect its engineering role.

          Awaiting Discovery
          Operational fabric link Selected forwarding path Failed link
          Discovered Nodes 0
          Assigned Host LIDs 0
          Operational Links 8 / 8
          Subnet Manager Active
          InfiniBand Fabric Inspection

          Select a fabric component to investigate InfiniBand discovery, routing and subnet management.

          • HCAs connect GPU hosts to the fabric
          • Switches forward traffic using LFT entries
          • The subnet manager discovers and configures the subnet
          • LIDs identify ports within the InfiniBand subnet
          Path and Subnet Analysis

          Discover the fabric and assign LIDs before tracing host communication.

            InfiniBand Linear Forwarding Table — LFT

            Inspect the destination LID to output-port mapping for the selected switch. Entries are generated from a simplified deterministic shortest-path model.

            Destination LID Output Port Next Hop State
            Run subnet discovery and LID assignment.
            Subnet Manager Event Console
            [IB FABRIC] Awaiting subnet discovery.
            ENGINEERING INVESTIGATION

            Discover, Route and Recover an InfiniBand GPU Fabric

            Complete the following activities to understand subnet-manager responsibilities, LID-based forwarding and the operational consequences of fabric failures.

            CHALLENGE 01 — SUBNET DISCOVERY

            Discover the fabric, assign LIDs, inspect an HCA and identify how the subnet manager establishes forwarding information.

            CHALLENGE 02 — LID FORWARDING

            Trace communication between GPU Host 01 and GPU Host 03. Inspect the LFT at a leaf and spine switch to identify the forwarding path.

            CHALLENGE 03 — FABRIC RECOVERY

            Fail an uplink, recompute forwarding and verify an alternate path. Stop the subnet manager and investigate the difference between management-plane availability and existing data-plane forwarding.

            Investigation Not Yet Validated

            Discover the subnet, assign LIDs, inspect forwarding entries, trace traffic, investigate a link failure and recover the fabric.

            Lab 05 — Learning Completion

            Self-reported completion is synchronized with the GPU Cluster Networking dashboard.

            Not marked complete

            This lab uses a simplified InfiniBand subnet model. LIDs are assigned deterministically for educational purposes; switch forwarding entries are calculated using shortest paths with fixed tie-breaking. Production subnet managers may use different routing algorithms, partition policies, LMC configurations, switch-port addressing and failover mechanisms. Stopping the simulated subnet manager does not automatically disable previously configured data-plane forwarding.

            NETWORK INSIGHT / GPU CLUSTER NETWORKING / STAGE 02

            Lab 06 — InfiniBand Routing, Virtual Lanes and Credits

            Investigate how InfiniBand switches forward traffic through a leaf-spine fabric and how virtual lanes provide independent buffering and credit-based flow control. Configure Service Level mapping, generate traffic, inspect receive-buffer credits and explore congestion and head-of-line blocking.

            INFINIBAND ROUTING VIRTUAL LANES SERVICE LEVELS BUFFER CREDITS FLOW CONTROL HEAD-OF-LINE BLOCKING CONGESTION GPU FABRICS

            InfiniBand Flow-Control Configuration

            Configure routing, virtual lanes and the receive-buffer credit budget.

            InfiniBand Routing and Credit Flow

            Select a switch or host to inspect the routing and flow-control architecture.

            Fabric Ready
            Fabric connectivity Selected route Virtual-lane isolation
            Selected Route Spine 01
            Packets Forwarded 0
            Packets Queued 0
            Credit Stalls 0
            InfiniBand Flow-Control Inspection

            Select a host, leaf or spine switch to investigate routing, buffering and virtual lanes.

            • Service Levels classify traffic
            • SL-to-VL mapping selects a virtual lane
            • Virtual lanes maintain separate buffering resources
            • Credits regulate link-level transmission
            Engineering Analysis

            Initialize the simulation to examine the configured routing and credit model.

              Virtual Lane 0 — VL0 General Traffic
              Queued Packets 0
              Available Credits 4

              VL0 maintains its own modeled receive-buffer credit budget and traffic queue.

              Virtual Lane 1 — VL1 Priority Traffic
              Queued Packets 0
              Available Credits 4

              VL1 provides an independent buffering domain when the selected SL-to-VL mapping separates traffic.

              InfiniBand Flow-Control Event Console
              [IB FLOW CONTROL] Ready to initialize.
              ENGINEERING INVESTIGATION

              Investigate Credit Exhaustion and Virtual-Lane Isolation

              Complete the following engineering activities to understand why credits, buffering and traffic separation matter in high-performance InfiniBand fabrics.

              CHALLENGE 01 — CREDIT EXHAUSTION

              Configure two receive credits, generate a burst of eight packets and advance the simulator. Observe how available credits limit transmission and produce stalls.

              CHALLENGE 02 — VIRTUAL-LANE ISOLATION

              Inject congestion into VL0, then compare separate and shared SL-to-VL mappings. Determine whether SL1 traffic can make progress independently.

              CHALLENGE 03 — ROUTING & RECOVERY

              Switch between Spine 01 and Spine 02, inspect the selected path, and drain receiver buffers to restore credits and forwarding progress.

              Investigation Not Yet Validated

              Initialize the fabric, generate traffic, exhaust credits, investigate VL isolation, compare routes and restore forwarding.

              Lab 06 — Learning Completion

              Self-reported completion is synchronized with the GPU Cluster Networking dashboard.

              Not marked complete

              The simulator represents credits as a fixed number of equal-sized packet-buffer slots. Real InfiniBand flow control accounts for buffer capacity and link-level credit advertisement in implementation- specific units. Virtual lanes isolate buffer resources but do not guarantee performance without suitable arbitration, routing and traffic policies.

              NETWORK INSIGHT / WORKSPACE 12 / STAGE 02

              RDMA & InfiniBand — Completion Tracker

              Review your progress across three advanced GPU networking labs. Track your understanding of RDMA memory access, InfiniBand subnet management, LID-based forwarding, virtual lanes and credit-based flow control.

              Stage 2 Learning Progress 0 of 3 labs complete
              Complete Labs 4, 5 and 6 to finish the RDMA & InfiniBand engineering stage.
              0% Stage Completion
              LAB 04 Not Completed

              RDMA Operations and Memory Registration

              Configure registered memory, local and remote keys, queue pairs, RDMA Read and Write, Send/Receive and completion queues.

              LAB 05 Not Completed

              InfiniBand Fabric, Subnet Manager and LIDs

              Discover a GPU fabric, assign LIDs, inspect switch forwarding tables, trace routes and recover subnet management after failures.

              LAB 06 Not Completed

              InfiniBand Routing, Virtual Lanes and Credits

              Investigate SL-to-VL mapping, receive-buffer credits, credit exhaustion, congestion, head-of-line blocking and virtual-lane isolation.

              02
              Stage 2 — In Progress

              Complete all three RDMA & InfiniBand labs to finish this learning stage.

              NETWORK INSIGHT / GPU CLUSTER NETWORKING / STAGE 03

              Lab 07 — RoCEv2 Packet Encapsulation and Routing

              Investigate how RDMA traffic is transported over Ethernet and routed IP networks using RoCEv2. Inspect packet encapsulation, DSCP and ECN fields, UDP destination port 4791, InfiniBand transport headers, IP forwarding and MTU-related failures in a GPU cluster leaf-spine fabric.

              RoCEv2 ETHERNET IPv4 UDP 4791 BTH DSCP / ECN MTU IP ROUTING

              RoCEv2 Packet Configuration

              Configure the packet and network before tracing traffic between GPU hosts.

              Routed RoCEv2 GPU Fabric

              Select a device to inspect the packet's forwarding behaviour.

              Packet Ready
              IPv4 Packet Size —
              Path MTU 1500
              Remaining TTL 64
              Forwarding Result Idle

              Interactive RoCEv2 Packet Header Explorer

              Select an encapsulation layer to inspect its fields and engineering role. The packet-size calculation uses a simplified transport header model.

              Simplified example excludes optional transport extension headers, padding, ICRC, VLAN tags, Ethernet FCS and physical-layer overhead.

              Packet and Fabric Inspection

              Select a packet header or fabric device to investigate RoCEv2 encapsulation.

              • RoCEv2 encapsulates RDMA transport over UDP/IP
              • UDP destination port 4791 identifies RoCEv2 traffic
              • DSCP and ECN occupy the IPv4 traffic-class field
              • Routers forward according to destination IP
              Packet Engineering Analysis

              Build a packet and trace its forwarding path to examine routing and encapsulation.

                RoCEv2 Packet Trace Console
                [RoCEv2 ENGINE] Ready to build packet.
                ENGINEERING INVESTIGATION

                Investigate RoCEv2 Encapsulation and Routed Forwarding

                Complete the following activities to investigate packet encapsulation, forwarding and common network-layer failures affecting RoCEv2 fabrics.

                CHALLENGE 01 — PACKET ENCAPSULATION

                Build a RoCEv2 packet, inspect Ethernet, IPv4, UDP and BTH fields, and identify how DSCP and ECN markings are represented.

                CHALLENGE 02 — ROUTED GPU FABRIC

                Trace traffic through Spine 01 and Spine 02. Observe TTL processing and identify the destination IP used for routing.

                CHALLENGE 03 — MTU & TTL FAILURES

                Inject an MTU mismatch and TTL expiry. Explain why these failures can prevent successful GPU-to-GPU communication.

                Investigation Not Yet Validated

                Build and inspect a packet, trace both routes, investigate MTU and TTL failures, and restore successful forwarding.

                Lab 07 — Learning Completion

                Self-reported completion is synchronized with the GPU Cluster Networking dashboard.

                Not marked complete

                This is a conceptual IPv4 RoCEv2 packet model. Real RoCEv2 packets include transport-specific headers and integrity fields not fully represented here. Path MTU handling, packetization, QoS classification and congestion control depend on the NIC, operating system, switches and RDMA stack.

                NETWORK INSIGHT / GPU CLUSTER NETWORKING / STAGE 03

                Lab 08 — Priority Flow Control, ECN and DCQCN

                Engineer RoCEv2 congestion behaviour in a GPU cluster Ethernet fabric. Configure switch buffer thresholds, generate competing traffic, investigate Priority Flow Control (PFC), observe Explicit Congestion Notification (ECN) and Congestion Notification Packets (CNPs), and examine DCQCN-inspired sender-rate adaptation.

                RoCEv2 PFC IEEE 802.1Qbb ECN CNP DCQCN QUEUE TELEMETRY BACKPRESSURE CONGESTION RECOVERY

                Congestion-Control Configuration

                Configure a simplified per-priority switch queue and sender.

                RoCEv2 Congestion-Control Fabric

                Follow data traffic, ECN marking, CNP feedback and PFC backpressure.

                Fabric Ready
                LOSSLESS ETHERNET CONGESTION-CONTROL LOOP GPU SENDER RoCEv2 RNIC 70 units/step ETHERNET SWITCH RoCE Priority Queue 0 / 200 units GPU RECEIVER RoCEv2 RNIC 50 units/step PFC XOFF / XON • ECN → CNP → RATE FEEDBACK No congestion feedback Conceptual discrete-time queue and feedback model
                Sender Rate70
                Queue Occupancy0%
                ECN / CNP Events0 / 0
                PFC Pause Events0
                Switch Egress Queue Occupancy 0 / 200 units
                Queue ECN Threshold PFC XOFF
                Congestion-Control Mechanisms
                • ECN: Marks congestion-capable IP packets rather than relying solely on drops.
                • CNP: Feedback from the receiving RNIC toward the sender following congestion notification.
                • DCQCN: Adjusts sender transmission rate in response to congestion feedback.
                • PFC: Pauses traffic for an Ethernet priority to protect buffer capacity.
                • Risk: Excessive PFC can propagate backpressure and create head-of-line blocking.
                Live Engineering Analysis

                Initialize the simulator and advance traffic to inspect congestion behaviour.

                  RoCEv2 Congestion Event Console
                  [RoCEv2 CONGESTION] Ready to initialize.
                  ENGINEERING INVESTIGATION

                  Investigate Queue Growth, Congestion Feedback and Recovery

                  Run each scenario and inspect how ECN and PFC interact. This lab uses simplified feedback timing and sender adaptation rather than a hardware-accurate DCQCN implementation.

                  CHALLENGE 01 — ECN / CNP

                  Enable ECN/DCQCN, configure sender demand above receiver capacity, and advance traffic until ECN and CNP events occur.

                  CHALLENGE 02 — PFC BACKPRESSURE

                  Switch to PFC-only mode, inject a bottleneck and observe queue growth, XOFF activation and subsequent XON recovery.

                  CHALLENGE 03 — CONTROL COMPARISON

                  Compare operation without congestion control against combined ECN and PFC. Explain the consequences for drops, pauses and sender rates.

                  Investigation Not Yet Validated

                  Initialize the fabric, observe ECN/CNP, trigger PFC, compare modes and restore receiver capacity.

                  Lab 08 — Learning Completion

                  Self-reported completion synchronizes with the GPU Cluster Networking dashboard.

                  Not marked complete

                  Engineering note: The simulation uses abstract traffic units per step and a simplified deterministic ECN/CNP loop. Real implementations use packet-level marking, feedback timing, rate recovery algorithms, PFC headroom, link propagation delay, priority mapping and vendor-specific hardware behaviour. PFC is not a guarantee of end-to-end losslessness.

                  NETWORK INSIGHT / GPU CLUSTER NETWORKING / STAGE 03

                  Lab 09 — Lossless Ethernet Fabric Design

                  Design a RoCEv2-ready Ethernet leaf-spine network for GPU clusters. Evaluate uplink oversubscription, priority mapping, PFC buffer headroom, ECN/PFC threshold ordering, link-failure capacity and operational risks before approving a production fabric design.

                  LEAF-SPINERoCEv2 QoS MAPPINGPFC HEADROOM ECN THRESHOLDSOVERSUBSCRIPTION FAILURE CAPACITYFABRIC VALIDATION

                  Production Fabric Configuration

                  Configure a representative GPU leaf with four 400 Gb/s server-facing ports.

                  GPU Cluster Ethernet Fabric

                  Illustrative two-leaf, four-spine design with configurable active uplink capacity.

                  Design Ready
                  RoCEv2 ETHERNET LEAF-SPINE FABRIC Select Trace GPU-to-GPU Traffic
                  Physical Oversubscription2.00:1
                  Offered Load / Uplink Capacity1.00×
                  Headroom Required195 KiB
                  Available Headroom128 KiB
                  Engineering Design Principles
                  • Plan for traffic locality, all-to-all communication and incast, not just average utilization.
                  • Maintain consistent DSCP-to-queue and priority mappings across the RoCEv2 path.
                  • Configure ECN to provide early congestion feedback before severe queue pressure.
                  • Reserve sufficient PFC headroom for in-flight data after pause generation.
                  • Validate capacity after uplink failures and avoid prolonged PFC pause propagation.
                  Live Fabric Engineering Analysis

                  Evaluate the fabric to calculate capacity, thresholds and headroom.

                    Production Readiness Assessment

                    Each check is calculated from the current design. Passing these simplified checks does not certify a real fabric.

                    GPU Ethernet Fabric Event Console
                    [FABRIC DESIGN] Ready to evaluate.
                    ENGINEERING INVESTIGATION

                    Design and Validate a Production RoCEv2 Fabric

                    CHALLENGE 01 — CAPACITY

                    Compare two and four uplinks per leaf. Investigate how offered GPU load affects oversubscription.

                    CHALLENGE 02 — PFC HEADROOM

                    Increase the reaction-time budget and observe how required headroom changes. Adjust buffers and XOFF thresholds.

                    CHALLENGE 03 — FAILURE RECOVERY

                    Inject an uplink failure, examine reduced capacity, restore connectivity and apply a resilient baseline.

                    Investigation Not Yet Validated

                    Evaluate the design, trace traffic, compare capacity, inject failure, restore links and examine headroom.

                    Lab 09 — Learning Completion

                    Self-reported completion synchronizes with the GPU Cluster Networking dashboard.

                    Not marked complete

                    The headroom estimate uses 400 Gb/s × reaction time plus 20% illustrative margin. Real buffer engineering requires hardware-specific cell accounting, pause generation and propagation delays, cable lengths, NIC behaviour, scheduling, shared-buffer architecture and measured traffic bursts. Ethernet PFC mitigates selected loss mechanisms; it does not make a network universally lossless.

                    NETWORK INSIGHT / WORKSPACE 12 / STAGE 03

                    RoCEv2 & Ethernet — Completion Tracker

                    Review your progress across three advanced RoCEv2 and Ethernet engineering labs. Track your understanding of packet encapsulation, routed RDMA traffic, PFC, ECN, DCQCN, QoS, buffer headroom and production fabric design.

                    Stage 3 Learning Progress 0 of 3 labs complete
                    Complete Labs 7, 8 and 9 to finish the RoCEv2 & Ethernet engineering stage.
                    0% Stage Completion
                    LAB 07 Not Completed

                    RoCEv2 Packet Encapsulation and Routing

                    Explore Ethernet, IPv4, UDP port 4791, InfiniBand transport headers, DSCP/ECN, path MTU, TTL and GPU-to-GPU routing.

                    LAB 08 Not Completed

                    Priority Flow Control, ECN and DCQCN

                    Investigate RoCEv2 congestion behaviour, switch queues, ECN marking, CNP feedback, sender-rate adaptation, PFC XOFF/XON and congestion recovery.

                    LAB 09 Not Completed

                    Lossless Ethernet Fabric Design

                    Engineer a GPU leaf-spine fabric with oversubscription planning, QoS priorities, ECN thresholds, PFC headroom, resilience and production readiness checks.

                    03
                    Stage 3 — In Progress

                    Complete all three RoCEv2 & Ethernet labs to finish this learning stage.

                    NETWORK INSIGHT / GPU CLUSTER NETWORKING / STAGE 04

                    Lab 10 — Collective Communication and NCCL

                    Investigate how distributed AI training workloads coordinate communication across GPU accelerators. Compare Ring AllReduce, Tree AllReduce, AllGather and ReduceScatter operations. Examine how GPU interconnect bandwidth, Ethernet or InfiniBand fabric capacity, message size and communication algorithms influence distributed training performance.

                    NCCL ALLREDUCE ALLGATHER REDUCESCATTER RING TREE NVLINK RDMA FABRIC

                    Collective Communication Engine

                    Configure an eight-GPU collective spanning two accelerator nodes.

                    Eight-GPU Collective Communication Fabric

                    Two GPU nodes, local accelerator links and a shared inter-node RDMA fabric.

                    Collective Ready
                    Estimated Collective Time —
                    Estimated Effective Bandwidth —
                    Communication Steps —
                    Inter-Node Bandwidth 50 GB/s
                    Collective Communication Architecture
                    • AllReduce: combines values across GPUs and distributes the result.
                    • AllGather: distributes input partitions to all participants.
                    • ReduceScatter: reduces data and distributes result partitions.
                    • Ring: exchanges chunks between neighboring participants.
                    • Tree: organizes communication into hierarchical reduction or distribution.
                    Live Performance Analysis

                    Build a collective communication topology to calculate estimated communication cost.

                      NCCL-Inspired Communication Trace
                      [COLLECTIVE ENGINE] Ready.
                      ENGINEERING INVESTIGATION

                      Investigate GPU Collective Communication Performance

                      Explore communication algorithms and determine how inter-node network bandwidth affects collective performance.

                      CHALLENGE 01 — ALLREDUCE

                      Build and run Ring AllReduce. Inspect the reduce-scatter and all-gather communication phases.

                      CHALLENGE 02 — ALGORITHM COMPARISON

                      Compare ring and tree communication for the same message size and fabric.

                      CHALLENGE 03 — FABRIC BOTTLENECK

                      Reduce inter-node bandwidth, observe the estimated communication penalty and restore the original fabric.

                      Investigation Not Yet Validated

                      Build and run a collective, compare algorithms, inject a network bottleneck and restore capacity.

                      Lab 10 — Learning Completion

                      Self-reported completion synchronizes with the GPU Cluster Networking dashboard.

                      Not marked complete

                      This simulator uses simplified analytical communication-cost models for educational comparison. Real NCCL implementations select algorithms and protocols based on topology, GPU architecture, message size, network transport, collective type and runtime tuning. The displayed performance estimates are illustrative rather than benchmark results.

                      NETWORK INSIGHT / GPU CLUSTER NETWORKING / STAGE 04

                      Lab 11 — ECMP, Congestion and Adaptive Routing

                      Investigate how GPU cluster Ethernet and RDMA fabrics distribute traffic across equal-cost spine paths. Compare deterministic ECMP hashing with a simplified adaptive flow-placement policy, inject elephant flows, inspect congestion hotspots and evaluate the effect of link failures on GPU-to-GPU communication.

                      ECMPFLOW HASHING ADAPTIVE ROUTINGELEPHANT FLOWS LEAF-SPINEQUEUE OCCUPANCY LINK FAILUREFABRIC TELEMETRY

                      Fabric Routing Configuration

                      Configure the routing policy and GPU traffic demand.

                      GPU Cluster Four-Spine Fabric

                      Two leaf switches, eight GPU endpoints and four equal-cost routed paths.

                      Fabric Ready
                      Peak Path Utilization0%
                      Active Spine Paths4 / 4
                      Offered Traffic0 Gb/s
                      Estimated Excess Demand0 Gb/s
                      ECMP and Adaptive Routing
                      • ECMP: Hashes each flow to an available equal-cost next hop.
                      • Hash collision: Multiple large flows may land on the same spine path.
                      • Adaptive routing: May use congestion or load information to choose a path.
                      • Flow ordering: Moving established flows can introduce packet reordering.
                      • Failure recovery: Link failures change the set of available paths and can create new hotspots.
                      Live Fabric Analysis

                      Build the fabric and route traffic to inspect path utilization.

                        GPU Fabric Routing Event Console
                        [ECMP ENGINE] Ready.
                        ENGINEERING INVESTIGATION

                        Investigate Flow Hashing, Congestion and Path Recovery

                        CHALLENGE 01 — ECMP

                        Build an ECMP fabric, route eight GPU flows and inspect distribution across the four spine paths.

                        CHALLENGE 02 — ELEPHANT FLOWS

                        Inject a large flow, compare ECMP and adaptive placement and investigate any congestion hotspots.

                        CHALLENGE 03 — FAILURE RECOVERY

                        Fail a busy spine path, observe traffic redistribution and restore the fabric.

                        Investigation Not Yet Validated

                        Route traffic, inject an elephant flow, compare policies, fail a path and restore the fabric.

                        Lab 11 — Learning Completion

                        Self-reported completion synchronizes with the GPU Cluster Networking dashboard.

                        Not marked complete

                        This lab uses an educational aggregate-demand model. Adaptive routing is represented by least-loaded placement of new flows, not packet spraying or a vendor-specific implementation. Offered traffic exceeding path capacity is reported as excess demand; actual packet loss depends on queues, buffering, congestion control and traffic timing.

                        NETWORK INSIGHT / GPU CLUSTER NETWORKING / STAGE 04

                        Lab 12 — GPU Fabric Performance and Bottleneck Analysis

                        Investigate the complete GPU-to-GPU data path from source accelerator memory through PCIe, the RDMA NIC, Ethernet or InfiniBand fabric and the destination GPU. Identify bandwidth bottlenecks, estimate transfer latency and investigate how incast traffic and fabric oversubscription affect distributed AI workloads.

                        GPU DIRECT RDMAPCIe RNICLEAF-SPINE THROUGHPUTLATENCY INCASTBOTTLENECK ANALYSIS

                        Performance Investigation Controls

                        Configure the GPU communication path and workload demand.

                        GPU-to-GPU RDMA Performance Path

                        Accelerator → PCIe → RNIC → Fabric → RNIC → GPU

                        Performance Ready
                        Estimated Throughput per Sender—
                        Estimated Transfer Time—
                        Primary Bottleneck—
                        Aggregate Incast Demand Ratio—
                        Performance Engineering Principles
                        • End-to-end throughput cannot exceed the slowest effective path component.
                        • PCIe bandwidth can limit GPU Direct RDMA even when network links are faster.
                        • NIC line rate and application throughput differ because of protocol and system overhead.
                        • Incast concentrates multiple senders on shared receiver and fabric resources.
                        • Network congestion can introduce queueing delay, ECN feedback and PFC activity.
                        • Use hardware counters and real benchmarks to validate a production diagnosis.
                        Live Bottleneck Analysis

                        Evaluate the GPU fabric to inspect throughput and latency estimates.

                          GPU Fabric Performance Investigation Console
                          [GPU PERFORMANCE ENGINE] Ready.
                          ENGINEERING INVESTIGATION

                          Identify and Resolve GPU Communication Bottlenecks

                          CHALLENGE 01 — PATH ANALYSIS

                          Evaluate the baseline and trace the GPU-to-GPU RDMA data path.

                          CHALLENGE 02 — INCAST

                          Inject multiple concurrent senders and inspect receiver contention and effective throughput.

                          CHALLENGE 03 — REMEDIATION

                          Inject a fabric bandwidth fault, identify the bottleneck and apply an optimized configuration.

                          Investigation Not Yet Validated

                          Evaluate, trace, inject incast, introduce a fabric fault and optimize the path.

                          Lab 12 — Learning Completion

                          Self-reported completion synchronizes with the GPU Cluster Networking dashboard.

                          Not marked complete

                          This simulator estimates steady-state throughput using a minimum-capacity path model and simplified equal-share incast contention. Latency is estimated from a configured base delay plus transfer serialization. It does not simulate PCIe transaction details, actual GPU memory throughput, transport protocol overhead, switch queues or hardware congestion control. All results are illustrative.

                          NETWORK INSIGHT / WORKSPACE 12 / STAGE 04

                          GPU Fabric Performance — Completion Tracker

                          Review your progress across three advanced GPU fabric performance engineering labs. Track your understanding of collective communication, NCCL, ECMP routing, congestion, adaptive path selection, GPU-to-GPU latency, throughput and bottleneck analysis.

                          Stage 4 Learning Progress 0 of 3 labs complete
                          Complete Labs 10, 11 and 12 to finish the GPU Fabric Performance Engineering stage.
                          0% Stage Completion
                          LAB 10 Not Completed

                          Collective Communication and NCCL

                          Investigate AllReduce, AllGather, ReduceScatter, ring and tree algorithms, GPU topology, inter-node bandwidth and distributed training communication performance.

                          LAB 11 Not Completed

                          ECMP, Congestion and Adaptive Routing

                          Engineer a four-spine GPU cluster fabric, examine flow hashing and elephant flows, compare routing policies, investigate congestion hotspots and recover from failures.

                          LAB 12 Not Completed

                          GPU Fabric Performance and Bottleneck Analysis

                          Analyze GPU Direct RDMA performance across PCIe, RNICs and the leaf-spine fabric. Identify throughput constraints, latency, incast contention and network bottlenecks.

                          04
                          Stage 4 — In Progress

                          Complete all three GPU Fabric Performance labs to finish this learning stage.

                          NETWORK INSIGHT / GPU CLUSTER NETWORKING / STAGE 05

                          Lab 13 — RDMA Latency and Packet-Loss Investigation

                          Investigate production GPU cluster communication failures using switch and RNIC telemetry. Analyze congestion, ECN marking, Priority Flow Control, path MTU mismatches, physical link errors and receiver saturation. Correlate counters and latency changes to identify root causes and evaluate remediation strategies.

                          RDMA TELEMETRY LATENCY SPIKES PACKET LOSS PFC PAUSE ECN MARKING MTU NIC COUNTERS ROOT-CAUSE ANALYSIS

                          Production Incident Controls

                          Select an incident, inspect the live fabric, run traffic and identify the underlying failure.

                          Production GPU-to-GPU RDMA Fabric

                          Source RNIC → Leaf → Spine → Leaf → Destination RNIC

                          Fabric Healthy
                          Estimated One-Way Latency —
                          Delivered Throughput —
                          Egress Queue Occupancy —
                          Simulation Time 0 ms
                          ECN CE Marks 0
                          PFC Pause Events 0
                          Estimated Packet Drops 0
                          Physical Error Events 0
                          Latency Trend — Recent Simulation Steps
                          RDMA Troubleshooting Method
                          • Establish baseline latency and throughput.
                          • Inspect switch egress queues and ECN CE counters.
                          • Correlate PFC pause activity with congestion.
                          • Verify consistent path MTU across RDMA interfaces.
                          • Inspect CRC/FCS errors and physical link health.
                          • Check receiver RNIC utilization and resource limits.
                          • Apply remediation and verify counter trends.
                          Live Incident Analysis

                          Load an incident and run traffic to generate diagnostic telemetry.

                            RDMA Production Incident Console
                            [RDMA INCIDENT ENGINE] Ready.
                            ENGINEERING INVESTIGATION

                            Diagnose and Recover a Production RDMA Fabric

                            CHALLENGE 01 — BASELINE

                            Load the healthy fabric and establish baseline latency and throughput.

                            CHALLENGE 02 — INCIDENT RCA

                            Inject a fault, run traffic, inspect counters and submit the correct root cause.

                            CHALLENGE 03 — RECOVERY

                            Apply remediation and verify that the fabric returns to healthy operation.

                            Investigation Not Yet Validated

                            Establish a baseline, diagnose a fault, inspect counters and recover the fabric.

                            Lab 13 — Learning Completion

                            Self-reported completion synchronizes with the GPU Cluster Networking dashboard.

                            Not marked complete

                            Educational discrete-time incident model. Counters, latency, congestion and throughput are illustrative rather than hardware-accurate. In production, correlate RNIC statistics, switch queue telemetry, interface error counters, ECN/PFC measurements and application-level benchmarks. PFC is link-local flow control; ECN provides congestion marking, while RoCEv2 congestion-control implementations may use receiver-generated CNP feedback.

                            NETWORK INSIGHT / GPU CLUSTER NETWORKING / STAGE 05

                            Lab 14 — GPU Cluster Fabric Failure and Recovery

                            Investigate how high-performance GPU clusters maintain connectivity when network components fail. Inject spine and leaf failures, inspect available ECMP paths, observe communication disruption, advance through reconvergence and recovery, and evaluate the resilience of the GPU fabric.

                            FABRIC RESILIENCEECMP RECONVERGENCE SPINE FAILURELEAF OUTAGE REDUNDANT PATHSRECOVERY TIME GPU COMMUNICATIONHIGH AVAILABILITY

                            Fabric Failure Controls

                            Configure the topology, inject failures and advance the recovery process.

                            Multi-Rack GPU Cluster High-Availability Fabric

                            Two leaf switches, four spines and eight GPU endpoints.

                            Fabric Healthy
                            Available Spine Paths4 / 4
                            Available Fabric Capacity800 Gb/s
                            Traffic Served400 Gb/s
                            Estimated Recovery Time0 ms
                            Traffic Availability — Recovery Timeline
                            High-Availability Engineering Principles
                            • ECMP distributes traffic across available equal-cost next hops.
                            • Spine redundancy preserves alternate paths when a spine fails.
                            • Reconvergence depends on failure detection and forwarding updates.
                            • Leaf failures can isolate directly attached GPU endpoints without a redundant attachment design.
                            • Reduced path capacity can create congestion even after routing recovers.
                            • Recovery should be evaluated using service impact as well as control-plane convergence.
                            Live Failure Analysis

                            Build the fabric and inject a failure to inspect resilience.

                              GPU Fabric Failure and Recovery Console
                              [FABRIC HA ENGINE] Ready.
                              ENGINEERING INVESTIGATION

                              Investigate GPU Fabric Resilience and Recovery

                              CHALLENGE 01 — BASELINE

                              Build the healthy fabric and inspect the four available spine paths.

                              CHALLENGE 02 — FAILURE

                              Inject a spine or leaf failure and investigate service disruption and reconvergence.

                              CHALLENGE 03 — RECOVERY

                              Complete the recovery process, compare capacity and restore failed components.

                              Investigation Not Yet Validated

                              Build the fabric, inject a failure, complete recovery, compare capacity and restore components.

                              Lab 14 — Learning Completion

                              Self-reported completion synchronizes with the GPU Cluster Networking dashboard.

                              Not marked complete

                              This simulator uses a simplified recovery state machine and aggregate path-capacity model. Actual GPU fabric resilience depends on physical topology, routing protocols, BFD, ECMP implementation, host networking, RNIC behavior and application-level recovery. Recovery-time values are illustrative rather than vendor-specific guarantees.

                              NETWORK INSIGHT / GPU CLUSTER NETWORKING / STAGE 05 / FINAL CAPSTONE

                              Lab 15 — Production GPU Cluster Design and Troubleshooting Capstone

                              Engineer and troubleshoot a production-style 16-GPU cluster. Choose InfiniBand or RoCEv2, evaluate network capacity, inspect collective communication performance, inject incidents, correlate telemetry, identify root causes and restore the fabric to a healthy operating state.

                              16 GPU CLUSTERINFINIBAND RoCEv2RDMA NCCLPFC / ECN FAILURE RECOVERYPRODUCTION RCA

                              Production Engineering Console

                              Configure the cluster, inject a fault and complete the investigation.

                              16-GPU Production Leaf-Spine Architecture

                              Four GPU racks, four leaves, four spines and an RDMA transport.

                              Cluster Ready
                              Available Fabric Capacity—
                              Delivered Workload Traffic—
                              Illustrative Collective Latency—
                              Engineering Score0 / 100
                              ECN Marks0
                              PFC Events0
                              Packet Drops0
                              Failed Spine Paths0
                              Production GPU Fabric Engineering
                              • Design the GPU cluster around workload communication patterns.
                              • Understand the differences between InfiniBand and RoCEv2.
                              • Size leaf-spine paths against expected aggregate demand.
                              • Evaluate collective communication and network bottlenecks.
                              • Use transport-appropriate congestion telemetry.
                              • Diagnose failures using evidence rather than assumptions.
                              • Verify recovery using service-level performance measurements.
                              Live Capstone Assessment

                              Deploy the GPU cluster to begin the engineering assessment.

                                Production GPU Cluster Incident Console
                                [GPU CAPSTONE ENGINE] Ready.
                                FINAL ENGINEERING ASSESSMENT

                                Complete the Production GPU Cluster Incident

                                PHASE 01 — DESIGN

                                Deploy a 16-GPU cluster, select its RDMA transport and establish a performance baseline.

                                PHASE 02 — INVESTIGATE

                                Inject a production fault, inspect telemetry and identify the correct root cause.

                                PHASE 03 — RECOVER

                                Apply remediation, verify service recovery and review the engineering score.

                                Capstone Assessment Pending

                                Complete design, baseline, incident investigation, RCA, remediation and recovery verification.

                                Lab 15 — Final Capstone Completion

                                Self-reported completion synchronizes with the GPU Cluster Networking dashboard.

                                Not marked complete

                                Educational capstone simulator. Telemetry, collective latency and capacity calculations are simplified and illustrative. The congestion-control behavior of InfiniBand differs from RoCEv2 Ethernet; ECN/PFC counters are shown only for the Ethernet transport in this model. Assessment score is local to this lab and is not a professional certification.

                                NETWORK INSIGHT / WORKSPACE 12 / FINAL STAGE

                                Production Troubleshooting — Completion Tracker

                                Complete the final three engineering labs to finish the GPU Cluster Networking Learning Workspace. Review your progress across RDMA incident investigation, fabric resilience and recovery, and the integrated production GPU cluster engineering capstone.

                                Stage 5 Learning Progress 0 of 3 labs complete

                                Complete Labs 13, 14 and 15 to finish Production Troubleshooting Engineering.

                                0% Stage Completion
                                LAB 13 Not Completed

                                RDMA Latency and Packet-Loss Investigation

                                Diagnose congestion, PFC pauses, ECN marking, path MTU mismatches, physical link errors, RNIC saturation and GPU-to-GPU latency problems.

                                LAB 14 Not Completed

                                GPU Cluster Fabric Failure and Recovery

                                Investigate spine failures, leaf-switch outages, redundant ECMP paths, capacity degradation, convergence timing and GPU communication recovery.

                                LAB 15 / FINAL CAPSTONE Not Completed

                                Production GPU Cluster Design and Troubleshooting

                                Engineer and troubleshoot an integrated 16-GPU production cluster using InfiniBand or RoCEv2, collective communication, fabric telemetry, root-cause analysis and recovery verification.

                                GPU Cluster Networking — Entire Workspace 0 / 15 Labs
                                Five engineering stages · Fifteen interactive labs 0%
                                STAGE 01 0 / 3 Labs
                                STAGE 02 0 / 3 Labs
                                STAGE 03 0 / 3 Labs
                                STAGE 04 0 / 3 Labs
                                STAGE 05 0 / 3 Labs
                                05
                                Stage 5 — In Progress

                                Complete all three Production Troubleshooting labs to finish the final engineering stage.

                                ☕
                                Support Network Insight
                                Help support the development of interactive networking labs, BGP simulations, and educational content.
                                Support the Project