Network Insight · Interactive Learning Workspace

AI Infrastructure Learning Workspace

Explore how modern AI infrastructure works in practice. Investigate GPU servers, high-speed cluster fabrics, distributed training, model-serving platforms and production performance through connected engineering activities.

05 Learning stages 15 Interactive labs GPU · RDMA · Networking Training · Inference · Operations

Interactive AI Infrastructure Architecture

Training traffic · Animated
Orchestration Kubernetes · Scheduling GPU Cluster Compute · HBM · NVLink Cluster Fabric Ethernet · RoCEv2 · IB Data & Storage Datasets · Checkpoints Model Serving Inference · vLLM · APIs

Select a component in the diagram to inspect its engineering role. Flow paths are conceptual, not measured network traces.

STAGE 01 Infrastructure Foundations Compute · GPU · Capacity
STAGE 02 GPU Cluster Networking Fabric · RDMA · Congestion
STAGE 03 Distributed Training Parallelism · Collectives
STAGE 04 AI Inference Serving · Scaling · K8s
STAGE 05 Production Operations Incidents · Diagnosis
Learning mode: This workspace uses interactive educational models. Architecture, flow animations, capacity examples and troubleshooting outcomes are illustrative; they are not connected to live infrastructure.
Network Insight · AI Infrastructure Learning Progress

Learning Control Bar

Navigate five engineering stages and track all fifteen interactive activities.

0%
0 of 5 stages 0 of 15 activities
Selected stage · 01

Infrastructure Foundations

Explore GPU compute and infrastructure architecture.

0 / 3 Complete
Ready · Begin with infrastructure foundations.
Learning progress: Select a stage to inspect its activities. Check activities as you complete the corresponding engineering labs. Progress is stored locally in this browser. The completion checkboxes are self-reported; they do not automatically validate lab results.
Network Insight · AI Infrastructure Learning Progress

Unified AI Infrastructure Learning Dashboard

Track your progress across five engineering stages and fifteen interactive activities. Explore GPU architecture, cluster networking, distributed training, inference infrastructure and production troubleshooting.

0%
0/15 Overall Progress
0/15 Activities Completed
0/5 Stages Completed
15 Activities Remaining

AI Infrastructure Learning Pathway

0 of 15 activities completed
01 Architecture
02 Networking
03 Training
04 Inference
05 Operations
Your next engineering activity

Begin with Infrastructure Foundations

Explore the first learning stage.

Progress synchronization: This dashboard reads the same browser storage as the Learning Control Bar. Activity completion updates automatically when progress changes. Activities are self-reported until individual lab validation is connected.
Network Insight · Your AI Infrastructure Learning Pathway

Explore the AI Infrastructure Workspace

Follow five connected engineering stages, from understanding GPU infrastructure and high-speed networking to distributed training, production inference and failure investigation. Select a stage to explore its objectives and three practical labs. Your learning progress is saved automatically.

05Learning stages
15Interactive activities
0% Learning progress
Stage 01 · Architecture

Infrastructure Foundations

Three Engineering Activities
Ready · Begin with AI infrastructure architecture Complete fifteen engineering activities across five stages.
Learning pathway: This navigator shares progress with the Learning Control Bar and Unified Dashboard. The stage buttons select the corresponding learning stage. The actual labs appear below in the workspace.
Stage 01 / 05 · AI Infrastructure Foundations

AI Infrastructure Architecture & Foundations

Artificial intelligence infrastructure is more than a collection of powerful GPUs. Production AI systems depend on tightly coordinated compute, memory, networking, storage and orchestration layers. Understanding how these components interact is essential for designing scalable training clusters, reliable inference platforms and efficient AI data centres.

In this first learning stage, we examine the physical and logical architecture supporting AI workloads. We will follow how infrastructure resources are provisioned, how GPU servers communicate, where datasets and model checkpoints are stored, and why architectural bottlenecks can significantly reduce application performance.

GPU Server Architecture Accelerator Memory High-Speed Networking AI Storage Kubernetes & Orchestration

What Makes AI Infrastructure Different?

Traditional enterprise applications frequently distribute independent requests across CPU-based servers. Large-scale AI training instead requires many accelerators to cooperate on shared computational workloads, exchanging intermediate data and synchronizing results.

This changes infrastructure design priorities. GPU memory capacity, interconnect topology, communication latency, storage throughput and workload placement can all influence whether expensive compute resources remain productive.

Core engineering principle: AI performance depends on the complete infrastructure system—not GPU compute capability alone. A slow network, storage bottleneck or poorly balanced workload can leave powerful accelerators waiting.
Compute GPU accelerators and host processors execute workloads.
Memory HBM and system memory constrain model and batch placement.
Fabric Interconnects carry distributed training traffic.
Storage Data pipelines supply training input and persist checkpoints.
Interactive Architecture Overview

Explore the Five AI Infrastructure Layers

Select a component to inspect its role, dependencies and operational risks.

Orchestration Layer Kubernetes · Scheduling · Deployment GPU Compute Accelerators · HBM · Host CPU High-Speed Network Fabric Leaf-Spine · Ethernet · RoCEv2 · InfiniBand Memory & Data Movement HBM · DRAM · Transfers AI Data Storage Datasets · Checkpoints Model Serving Inference · API
1. GPU Servers and Accelerator Architecture

GPU servers combine host processors, accelerator devices, high-bandwidth memory, PCIe connectivity and often specialized accelerator interconnects. The topology determines which data transfers remain local to a server and which must traverse the external network fabric.

2. High-Speed AI Networking

Distributed workloads generate significant east-west communication. Leaf-spine networks, Ethernet with RoCEv2, and InfiniBand can provide high-throughput interconnects. Their operational characteristics, congestion controls and management requirements differ.

3. Data Pipelines and Storage Performance

Training clusters require reliable access to large datasets and persistent checkpoint storage. Storage throughput, metadata operations, caching and data locality affect how efficiently GPUs receive work and recover after faults.

4. Infrastructure Capacity and Utilization

Capacity planning must consider accelerator memory, compute utilization, network bandwidth, storage performance, power, cooling and workload concurrency. Increasing GPU count alone does not guarantee proportional throughput improvement.

Stage 01 · Practical Learning Objectives

What You Will Investigate

The following three labs move from architectural understanding to GPU server inspection and infrastructure capacity decisions.

LAB 01 AI Infrastructure Architecture Explorer Trace dependencies between compute, network, storage, orchestration and model-serving components.
LAB 02 GPU Server Architecture Lab Inspect accelerator topology, memory hierarchy, local interconnects and host connectivity.
LAB 03 AI Cluster Capacity Planner Estimate accelerator requirements and investigate practical compute, memory and infrastructure constraints.
Stage 01 · Engineering Lab 01 / 15

AI Infrastructure Architecture Explorer

Investigate a production-style AI cluster. Trace workload dependencies, inject infrastructure failures, inspect operational evidence and determine the correct recovery action.

AI Cluster Dependency Topology

● Healthy
AI Orchestration Kubernetes · GPU Scheduling GPU Compute Training · Inference Cluster Fabric Ethernet · RDMA AI Storage Datasets · Checkpoints GPU Memory HBM · Allocation Model Serving Inference API
Healthy Degraded Fault Workload path
Healthy AI Cluster

All infrastructure components are operational. Select an incident and run an investigation.

Workload Dependency Trace

Infrastructure Journey

Follow the selected workload and identify where the simulated incident affects execution.

Operational Evidence

Cluster Telemetry

Illustrative measurements generated by the current scenario model.

82% GPU utilization
68% Fabric utilization
18 GB/s Storage throughput
100% Workload health
Engineering Investigation

Diagnose and Recover the AI Cluster

Select the suspected root cause and the corresponding remediation. Correctly resolve an incident to complete Lab 01.

Simulation scope: This is an educational dependency and incident model. Telemetry values are synthetic, not collected from real GPUs or network devices. A successful diagnosis records completion of Activity 01 in your local learning progress.
Stage 01 · Engineering Lab 02 / 15

GPU Server Architecture Lab

Explore the internal architecture of an AI GPU server. Investigate accelerator topology, CPU connectivity, GPU memory and high-speed interconnects. Configure data-transfer experiments and identify the component limiting effective communication performance.

Interactive GPU Server Cutaway

● Architecture Ready
AI GPU SERVER · LOGICAL CUTAWAY CPU Host NIC Network Host DRAM System Memory Interconnect Fabric NVLink-style / NVSwitch-style GPU High-Bandwidth Memory (HBM)
CPU / PCIe GPU Network Memory
8 GPU Accelerators
640 GB Aggregate HBM (illustrative)
300 GB/s Selected Peer-Link Bandwidth
NVLink Interconnect Mode
Engineering Experiment

GPU Communication Performance Investigator

Configure a data transfer and investigate how interconnect selection and injected bottlenecks affect its estimated completion time. This is a simplified bandwidth model, not a GPU hardware benchmark.

300 GB/s Effective Path Bandwidth
0.213 s Estimated Transfer Time
GPU Link Limiting Resource
Engineering Validation

Identify the Performance Bottleneck

Run a transfer experiment with an injected bottleneck. Use the reported bandwidth and communication trace to identify the constrained resource and an appropriate remediation.

Model assumptions: Illustrative bandwidth ceilings are PCIe 32 GB/s, GPU peer links 300 GB/s, and a network adapter at 50 GB/s. These are example effective rates, not specifications for a particular GPU generation. Transfers use size ÷ bandwidth without startup latency, contention, protocol overhead or multi-hop effects. HBM is modeled as 80 GB per GPU for learning purposes.
Stage 01 · Engineering Lab 03 / 15

AI Cluster Capacity Planner

Design an AI accelerator cluster and investigate whether its available GPU memory and compute capacity can support a selected workload. Compare model sizes, precision formats, GPU counts and workload assumptions to understand practical infrastructure sizing constraints.

175 GB Estimated Workload Memory
512 GB Usable GPU Memory
3 GPUs Minimum Memory-Based GPU Count
337 GB Remaining Usable Memory

GPU Cluster Allocation Visualizer

● Capacity Available
8-GPU Cluster Memory-Based Allocation
Required GPUs Available headroom Unallocated

Workload memory compared with usable cluster memory.

Capacity assessment will appear here.
Infrastructure Planning Scenario

Evaluate the Engineering Trade-Offs

Increasing GPU count increases aggregate memory and potential compute capacity, but real workloads also depend on communication overhead, memory distribution, scheduling, network bandwidth and storage throughput.

0% Estimated memory allocation relative to usable capacity
65% Assumed target compute utilization
Not evaluated Memory capacity assessment
Engineering Validation

Capacity Planning Challenge

Create a configuration that exceeds the cluster's available GPU memory. Run the capacity calculation, identify the constraint and choose an appropriate capacity remediation.

Engineering assumptions: This planner uses decimal GB and a simplified aggregate-memory model. Inference memory includes model weights plus a configurable overhead factor. Full-parameter training uses an illustrative mixed-precision optimizer-state estimate of 16 bytes per parameter, plus overhead. Training calculations assume FP16/BF16 parameters and FP32 optimizer/master states; other precision selections affect inference only. The model does not validate per-device tensor placement, GPU topology, KV-cache growth, activation checkpointing, ZeRO/FSDP sharding, interconnect bandwidth, training batch size, or actual compute throughput.
AI Infrastructure Learning Workspace · Stage 01 / 05

AI Infrastructure Foundations — Stage Progress

Review your progress across the three foundational engineering labs. Completion is recorded automatically when you successfully validate each lab's investigation.

Stage 01 Learning Record

Foundations Completion

● In Progress

Complete all three activities to finish the foundations stage.

0 / 3 Labs Completed
3 Labs Remaining
0 / 15 Workspace Activities
ENGINEERING LAB 01 Not Complete

AI Infrastructure Architecture Explorer

Investigate AI cluster dependencies, inject infrastructure faults and validate recovery actions.

Open Architecture Lab →
ENGINEERING LAB 02 Not Complete

GPU Server Architecture Lab

Examine accelerator hardware, memory placement, interconnect bandwidth and GPU transfer bottlenecks.

Open GPU Server Lab →
ENGINEERING LAB 03 Not Complete

AI Cluster Capacity Planner

Estimate model memory requirements, evaluate cluster sizing and diagnose capacity constraints.

Open Capacity Planner →
Stage 1 Engineering Outcomes

Skills Developed

01
AI Infrastructure Dependencies Understand orchestration, GPU compute, networking, storage and serving dependencies.
02
GPU Server Architecture Compare host-device and peer communication paths and recognize bandwidth constraints.
03
AI Cluster Capacity Planning Evaluate model-memory demand, accelerator count and capacity trade-offs.
Up Next · Stage 02 / 05

GPU Cluster Networking

Move from individual GPU servers into high-performance AI cluster networking. Investigate leaf-spine fabrics, RDMA transports and congestion behavior.

Leaf-Spine Fabrics RoCEv2 InfiniBand Congestion Analysis
Stage 2 is available for exploration. Complete all three Stage 1 labs to earn Stage 1 completion.
Stage 02 / 05 · GPU Cluster Networking

GPU Cluster Networking Architecture

Large AI workloads depend on fast, predictable communication between accelerators distributed across multiple servers. The network connecting these servers is therefore a fundamental part of the AI compute system—not simply a supporting connectivity service.

In this stage, we investigate how GPU clusters use high-performance network fabrics to exchange tensors, synchronize distributed workloads, access remote storage and support large-scale training. We examine the differences between conventional Ethernet networking and RDMA-capable transport, including RoCEv2 and InfiniBand.

Leaf-Spine Architecture GPU East-West Traffic RDMA RoCEv2 InfiniBand ECMP Congestion Control

Why AI Networking Requires a Different Design Approach

In distributed AI training, accelerators repeatedly exchange gradients, activations and other intermediate data. Communication may involve point-to-point transfers or collective operations across many workers.

If the network introduces excessive latency, congestion or insufficient throughput, GPUs can spend more time waiting for synchronization rather than performing useful computation.

Modern AI clusters therefore emphasize predictable east-west bandwidth, high-radix switching, appropriate oversubscription, efficient transport and comprehensive fabric telemetry.

Engineering principle: The effective performance of a distributed AI workload depends on both accelerator compute and the communication system connecting participating workers.
Network Design Priorities

Four Critical Engineering Dimensions

Bandwidth Sufficient link and fabric capacity for distributed communication.
Latency Low communication delay for synchronization-sensitive workloads.
Congestion Queue management, traffic distribution and transport control.
Resilience Multiple paths, fault detection and predictable failure behavior.
Interactive Network Architecture

Explore an AI Leaf-Spine Fabric

Select a traffic pattern, then inspect GPU servers, leaf switches, spine switches and the external network connection.

AI CLUSTER · HIGH-SPEED NETWORK FABRIC Spine 1 Fabric Core Spine 2 Fabric Core Leaf 1 To GPUs Leaf 2 To GPUs Leaf 3 To GPUs Leaf 4 To GPUs External Network Services / Storage
Spine Leaf GPU Server Active Path Congestion
Full Fabric Architecture

Every leaf switch connects to both spine switches, providing multiple fabric paths.

1. Leaf-Spine and Clos Network Design

Leaf-spine networks provide predictable connectivity between server-facing leaf switches through a shared spine layer. Multiple equal-cost paths can improve bandwidth utilization and resilience. Actual performance depends on link capacity, hashing, oversubscription and traffic distribution.

2. RDMA, RoCEv2 and InfiniBand

Remote Direct Memory Access enables data movement with reduced CPU involvement. RoCEv2 transports RDMA over routable Ethernet/IP networks, while InfiniBand uses its own networking architecture and transport mechanisms. Both require appropriate fabric design and operational controls.

3. Distributed GPU Communication

Distributed training frequently uses collective communication operations such as All-Reduce, All-Gather and Reduce-Scatter. Their performance depends on message size, topology, bandwidth, latency and communication algorithms.

4. Congestion and Network Telemetry

AI communication can create synchronized bursts and sustained high-throughput flows. Engineers investigate queue occupancy, ECN marking, packet loss, retransmissions, PFC behavior where deployed, and application-level communication delays.

Stage 02 · Practical Engineering Objectives

Three GPU Networking Labs

These labs progress from fabric construction to transport behavior and production congestion troubleshooting.

LAB 04 AI Leaf-Spine Fabric Builder Design a GPU cluster fabric, configure topology and link capacity, and investigate redundancy and oversubscription.
LAB 05 RDMA Transport Explorer Compare RoCEv2 and InfiniBand concepts, inspect communication paths and investigate transport-level behavior.
LAB 06 Network Congestion Investigation Analyze congestion symptoms, queue behavior, traffic distribution and possible remediation strategies.
Stage 02 · Engineering Lab 04 / 15

AI Leaf-Spine Fabric Builder

Design a high-performance GPU cluster network. Configure spine and leaf switches, GPU server density, link bandwidth and network failures. Evaluate fabric oversubscription, available capacity and path resilience.

16 Total GPU Servers
800 Gb/s Server Capacity Per Leaf
800 Gb/s Healthy Uplink Capacity Per Leaf
1.00:1 Healthy Oversubscription Ratio

Configurable GPU Network Fabric

● Fabric Healthy
GPU CLUSTER · LEAF-SPINE NETWORK DESIGN
Spine Leaf GPU Servers Failed Path

The diagram represents logical server groups and leaf-to-spine connections. Parallel physical uplinks are represented by the configured link count.

Network Design Assessment

Traffic Demand and Failure Impact

Compare aggregate server-side traffic demand with available uplink capacity. This simplified model assumes traffic crosses the leaf-to-spine fabric and is distributed evenly across leaf switches.

2.4 Tb/s Modeled Total Fabric Demand
3.2 Tb/s Available Aggregate Uplink Capacity
2 Available Spine Next Hops for Unaffected Leaves
Engineering Validation

Fabric Failure Investigation

Inject a spine failure and evaluate the resulting uplink capacity and path redundancy. Identify the infrastructure impact and select an appropriate engineering response.

Model scope: Oversubscription is calculated as total server-facing bandwidth divided by total leaf uplink bandwidth. Each leaf is assumed to connect to every spine with the configured number of parallel links. A single spine failure removes all links to that spine. A single uplink failure removes one physical link from one leaf. Real-world designs must also consider switch radix, optics, physical cabling, ECMP hashing, packet spraying policies, traffic locality, congestion control and transport behavior.
Stage 02 · Engineering Lab 05 / 15

RDMA Transport Explorer

Investigate how distributed AI workloads move data between accelerator servers. Compare conventional TCP communication with RoCEv2 and InfiniBand, inspect the transport journey, and explore how network conditions influence communication performance.

— Illustrative Transfer Time
— Modeled Effective Throughput
— CPU Involvement
Healthy Transport Condition

GPU-to-GPU Communication Journey

● Transport Ready
SOURCE GPU SERVER DESTINATION GPU SERVER GPU Memory Source Buffer GPU Memory Target Buffer Network Adapter Transport Engine Network Adapter Transport Engine Network Fabric GPU DATA MOVEMENT · TRANSPORT PATH RoCEv2 · RDMA-capable Ethernet transport

The diagram is a conceptual data path. Actual transfers depend on registered memory, NIC capabilities, GPU-direct support, operation type and transport implementation.

Conventional TCP TCP provides reliable byte-stream transport over IP. Conventional socket-based applications typically involve kernel networking and CPU processing, although offloads and zero-copy techniques can reduce overhead.
RoCEv2 RoCEv2 transports RDMA over UDP/IP on Ethernet. RDMA-capable adapters perform data movement with reduced CPU involvement. Congestion control and Ethernet fabric design are important.
InfiniBand InfiniBand provides an RDMA-capable fabric with its own link, network and transport architecture. Credit-based link flow control and fabric management are important operational elements.
Transport Performance Analysis

Interpreting the Transfer Estimate

This educational model estimates serialization time using configured transfer size and an assumed fraction of available bandwidth. It illustrates how degraded network conditions can reduce effective throughput.

— Assumed Link Efficiency
— Modeled Bandwidth Reduction
— Observed Scenario
Engineering Validation

RoCEv2 Congestion Investigation

Select RoCEv2 and inject fabric congestion. Compare the healthy and degraded transfer estimates, then identify the likely network-level issue and an appropriate investigation or remediation action.

Engineering model: Transfer times are illustrative, not benchmark results. The model uses decimal bandwidth and transfer-size conversions and assigns simplified efficiency factors to demonstrate sensitivity to congestion. It does not model real TCP congestion windows, RDMA queue pairs, protocol headers, RTT, application processing, GPU synchronization, topology-dependent collectives, or hardware-specific throughput. PFC is an Ethernet mechanism used in some RoCE deployments, not a native InfiniBand flow-control mechanism.
Stage 02 · Engineering Lab 06 / 15

Network Congestion Investigation

Investigate a simulated production AI cluster incident. Examine switch queue occupancy, ECN marking, PFC pauses, packet drops and GPU synchronization delays. Use the available telemetry to identify the most likely network bottleneck and select a remediation strategy.

— Peak Queue Occupancy
— ECN-Marked Packets
— PFC Pause Events
— Packet Drops
— GPU Collective Delay

AI Cluster Fabric Incident Map

● Fabric Operational
GPU CLUSTER · EAST-WEST TRAFFIC INVESTIGATION Spine 1 ECMP Path A Spine 2 ECMP Path B
Spine Leaf GPU Workers Traffic Congested Path

Select either spine switch to inspect its modeled traffic distribution and queue condition.

Production Troubleshooting Evidence

Incident Timeline and Engineering Interpretation

[READY] GPU fabric telemetry initialized [INFO] Select an incident scenario [INFO] Run investigation to collect evidence
Investigation Ready

Compare network-level counters with GPU collective communication delay. A high application delay alone does not prove network congestion; supporting fabric evidence is required.

Investigation Checklist

1. Identify the affected fabric path.
2. Examine queue and ECN behavior.
3. Check PFC pauses and packet drops.
4. Compare ECMP traffic distribution.
5. Correlate network symptoms with GPU delays.

Engineering Validation

Diagnose the Network Incident

Run an incident investigation, inspect the telemetry and select the most likely root cause and engineering response. Validation is based on the current scenario.

Simulation scope: All telemetry values are synthetic educational data, not measurements from a real network. ECN percentages, PFC counts, packet drops and collective delays are scenario-specific examples. PFC is relevant to selected Ethernet/RoCE deployments; it is not universally enabled. Real incident diagnosis also requires hardware counters, workload traces, transport telemetry, topology awareness and time correlation.
AI Infrastructure Learning Workspace · Stage 02 / 05

GPU Cluster Networking — Stage Progress

Review your progress across the three GPU networking engineering labs. Completion is recorded automatically when you successfully validate the fabric design, RDMA transport investigation and network congestion troubleshooting exercises.

Stage 02 Learning Record

GPU Networking Completion

○ Not Started

Complete all three activities to finish the GPU Cluster Networking stage.

0 / 3 Labs Completed
3 Labs Remaining
0 / 15 Workspace Activities
ENGINEERING LAB 04 Not Complete

AI Leaf-Spine Fabric Builder

Configure GPU network topology, calculate oversubscription, evaluate ECMP resilience and investigate spine failures.

Open Fabric Builder →
ENGINEERING LAB 05 Not Complete

RDMA Transport Explorer

Compare TCP, RoCEv2 and InfiniBand, investigate GPU data movement and evaluate congestion-related throughput loss.

Open RDMA Lab →
ENGINEERING LAB 06 Not Complete

Network Congestion Investigation

Diagnose ECMP imbalance, microbursts, PFC pause propagation, queue pressure and packet loss from telemetry evidence.

Open Congestion Lab →
Stage 2 Engineering Outcomes

Networking Skills Developed

04
AI Network Fabric Design Evaluate spine and leaf topology, capacity, oversubscription and path resilience.
05
High-Performance RDMA Transport Understand GPU communication paths, RDMA concepts and transport trade-offs.
06
Network Congestion Troubleshooting Correlate switch telemetry with distributed AI workload performance symptoms.
Up Next · Stage 03 / 05

Distributed AI Training

Move from the network fabric into distributed training execution. Investigate data parallelism, tensor and pipeline parallelism, GPU collective communication, synchronization overhead and training performance bottlenecks.

Data Parallelism Tensor Parallelism Pipeline Parallelism All-Reduce Training Performance
Stage 3 is available for exploration. Complete all three Stage 2 labs to earn Stage 2 completion.
Stage 03 / 05 · Distributed AI Training

Distributed AI Training Architecture

Training large AI models often requires many GPUs working together across multiple servers. Distributed training coordinates computation, model parameters, activations and gradients across those accelerators while balancing compute efficiency, memory capacity and communication overhead.

In this stage, we investigate how parallel training strategies divide work between GPUs, how collective communication synchronizes distributed workers, and how engineers diagnose performance bottlenecks that reduce training throughput.

Data Parallelism Tensor Parallelism Pipeline Parallelism All-Reduce All-Gather Reduce-Scatter Training Efficiency

Why Distributed Training Is an Infrastructure Problem

A single GPU has finite compute throughput and high-bandwidth memory. Large models and training datasets may require parallel execution across many accelerators.

Parallelism introduces communication. Depending on the training strategy, workers may exchange gradients, partial tensor results, activations or model-state information.

The training system must coordinate these operations while minimizing idle time, balancing memory usage and maintaining sufficient data delivery.

Engineering principle: Adding GPUs does not guarantee proportional training speedup. Communication overhead, synchronization, pipeline bubbles, workload imbalance and data loading can limit scaling efficiency.
Distributed Training Foundations

Four Core Engineering Concepts

Parallelism Divide training computation and model state across accelerators.
Synchronization Coordinate distributed updates and intermediate results.
Communication Exchange gradients, activations and tensor partitions.
Efficiency Reduce GPU idle time and communication bottlenecks.
Interactive Distributed Training Architecture

Explore Multi-GPU Training Strategies

Select a parallelism strategy and inspect how four GPU workers participate in distributed computation.

DISTRIBUTED AI TRAINING · GPU WORKER GROUP Network Fabric Data Parallelism · Replicated model, different data
Data Parallel Training

Training Execution Journey

Follow a Distributed Training Step

1. Data Parallelism

Multiple workers maintain model replicas while processing different training data. Gradients must be synchronized before equivalent model updates are applied. Distributed Data Parallel and sharded variants use different model-state management strategies.

2. Tensor Parallelism

Model computations and tensors are partitioned across multiple GPUs. Workers cooperate during layer execution, often requiring frequent communication of partial results. Placement and interconnect bandwidth strongly influence performance.

3. Pipeline Parallelism

Different model layers or stages execute on different GPUs. Microbatches flow through the pipeline to improve utilization. Pipeline bubbles, stage imbalance and activation transfers can limit efficiency.

4. Collective Communication

Collective operations such as All-Reduce, All-Gather and Reduce-Scatter coordinate data across workers. Their cost depends on message size, worker count, topology, algorithm and network performance.

Stage 03 · Practical Engineering Objectives

Three Distributed Training Labs

Progress from parallelism architecture into collective communication and then production training performance investigation.

LAB 07 Distributed Training Architecture Configure worker counts, parallelism strategies and model partitioning. Evaluate memory and communication trade-offs.
LAB 08 GPU Collective Communication Lab Investigate All-Reduce, All-Gather and Reduce-Scatter communication patterns, message movement and bandwidth demand.
LAB 09 Training Performance Investigator Diagnose slow training steps, GPU idle time, communication overhead, pipeline imbalance and data-loading bottlenecks.
Stage 03 · Engineering Lab 07 / 15

Distributed Training Architecture Planner

Design a multi-GPU training configuration. Select model size, GPU resources and parallelism strategy, then investigate model-state memory, worker communication, pipeline efficiency and scaling limitations.

— Modeled State Memory / GPU
— GPU Memory Utilization
— Data Parallel Replicas
— Pipeline Utilization Estimate
Ready Architecture Assessment

Distributed GPU Worker Architecture

● Configuration Ready
DISTRIBUTED TRAINING · GPU RESOURCE MAP Training Fabric Hybrid parallel training configuration

The diagram shows up to eight representative GPU workers. Configurations with more GPUs are summarized in the metrics and worker inspector.

Architecture Capacity Analysis

Memory and Communication Trade-offs

Training memory includes model parameters, gradients and optimizer state. The planner uses a simplified mixed-precision Adam-style model-state budget and estimates the effect of partitioning and state sharding.

Per-GPU Memory Budget
Model State —
Activation Allowance —
Remaining HBM —
Architecture Interpretation
Parallelism Layout —
Communication Demand —
Primary Scaling Risk —
Engineering Validation

Resolve a Distributed Training Capacity Incident

Configure a 70B-parameter model on eight 80 GB GPUs using TP 4, PP 2 and sharded model states. Calculate the architecture, inspect the GPU memory estimate and identify why the design requires model partitioning and state sharding.

Model assumptions: Model-state memory is approximated at 16 bytes per parameter for a common mixed-precision Adam-style training configuration, including parameters, gradients, master weights and optimizer states. This is not universal. The planner uses a fixed illustrative activation allowance and a simplified sharding model. It excludes temporary communication buffers, fragmentation, checkpointing, optimizer variants, framework overhead and detailed layer placement. Pipeline utilization is approximated as microbatches divided by microbatches plus pipeline stages minus one. Real training systems require profiling and framework-specific capacity planning.
Stage 03 · Engineering Lab 08 / 15

GPU Collective Communication Lab

Explore how distributed GPU workers exchange gradients, tensor partitions and intermediate results. Compare communication algorithms, investigate collective phases and estimate the effect of bandwidth, message size and network bottlenecks.

— Communication Phases
— Estimated Bytes Sent / GPU
— Effective Link Bandwidth
— Estimated Collective Time
Ready Communication Assessment

GPU Collective Communication Topology

● Fabric Ready
DISTRIBUTED GPU COLLECTIVE COMMUNICATION Collective Operation All-Reduce Ring collective communication

Select a collective operation and run the simulation.

Collective Performance Analysis

Communication Cost and Bandwidth Sensitivity

Collective operations exchange data across multiple GPU workers. Performance depends on algorithm choice, message size, topology, link bandwidth, synchronization overhead and communication contention.

Communication Cost Breakdown
Healthy Baseline Time —
Current Collective Time —
Additional Delay —
Engineering Interpretation
Communication Algorithm —
Bandwidth Limitation —
Recommended Investigation —
Engineering Validation

Diagnose a Slow All-Reduce Operation

Configure eight GPU workers, Ring All-Reduce, a 1 GB tensor, 200 Gbps links, 75% link efficiency and a congested fabric. Run the collective operation and investigate why communication time increases.

Simulation assumptions: This lab uses simplified bandwidth-dominated collective communication models. Ring All-Reduce is approximated using 2(N−1)/N times the full tensor size per GPU. Tree All-Reduce uses a simplified logarithmic phase model. All-Gather and Reduce-Scatter assume the selected message size represents the total assembled tensor. The model excludes protocol overhead, topology-specific routing, hardware collectives, overlapping communication, detailed startup latency and real NCCL scheduling. Congestion and slow-path effects are illustrative performance multipliers, not measured values.
Stage 03 · Engineering Lab 09 / 15

Training Performance Investigator

Investigate a simulated distributed AI training performance incident. Examine GPU utilization, training-step duration, communication overhead, pipeline bubbles and data-loading delays. Correlate infrastructure telemetry with training execution behavior to identify the most likely bottleneck.

— Average GPU Utilization
— Training Step Duration
— Communication Share
— Pipeline Idle Share
— Relative Training Throughput

Distributed Training Execution Timeline

● Training Operational

Each row represents an illustrative GPU worker training step. Segment widths indicate the relative contribution of compute, communication, data waiting and pipeline idle time.

Compute Communication Data Wait Pipeline Idle
Production Troubleshooting Evidence

Training Incident Timeline and Analysis

[READY] Distributed training telemetry initialized [INFO] Select a training incident [INFO] Run investigation to collect evidence
Investigation Ready

A slow training step can originate from data loading, GPU compute, collective communication or pipeline scheduling. Use the execution timeline and supporting telemetry to distinguish these causes.

Investigation Checklist

1. Compare step duration with baseline.
2. Inspect GPU utilization.
3. Examine communication overhead.
4. Identify data-loading stalls.
5. Check pipeline idle time and worker imbalance.

Engineering Validation

Diagnose the Training Performance Incident

Select one of the four degraded training scenarios, run the investigation and examine the telemetry. Identify the root cause and select the appropriate engineering response. Successfully diagnose all four incident types to complete Lab 09.

Validated incident types: 0 / 4

Simulation scope: All training telemetry is synthetic educational data. GPU utilization, step duration, throughput and time breakdowns are illustrative, not measurements or predictions for specific AI models or GPU hardware. Communication, data waiting and pipeline idle time are simplified, non-overlapping categories in this exercise. Real training performance analysis requires framework profilers, GPU kernel traces, collective timing, network counters and data-pipeline telemetry.
AI Infrastructure Learning Workspace · Stage 03 / 05

Distributed AI Training — Learning Progress

Review your progress across the three distributed AI training engineering labs. Confirm your understanding of training architecture, GPU collective communication and performance troubleshooting before advancing to AI inference infrastructure.

0% STAGE COMPLETION
Stage In Progress

0 of 3 engineering labs completed

Engineering Lab Checklist

Stage 3 Activities

07
Distributed Training Architecture Not yet completed
Open Lab
08
GPU Collective Communication Not yet completed
Open Lab
09
Training Performance Investigation Not yet completed
Open Lab
Engineering Competencies

Skills Demonstrated in Stage 3

🧩
Distributed Training Architecture

Configure tensor, pipeline and data parallelism. Interpret GPU model-state memory requirements and distributed training resource allocation.

Awaiting Lab 07
🔗
Collective Communication Engineering

Explain All-Reduce, All-Gather and Reduce-Scatter. Investigate how communication bandwidth influences distributed training.

Awaiting Lab 08
📊
Training Performance Troubleshooting

Correlate GPU utilization, training-step latency, data-loading stalls, collective overhead and pipeline idle time.

Awaiting Lab 09
Workspace Learning Record

Overall AI Infrastructure Progress

Your progress is shared with the workspace dashboard and learning pathway using the existing browser learning record.

0 / 15 Completed Engineering Labs
0% Overall Workspace Completion
0 / 5 Fully Completed Stages
Next Learning Stage · Stage 04

AI Inference Infrastructure

Move from distributed model training to production inference systems. Explore request execution, model serving, scaling behavior and Kubernetes-based AI infrastructure.

10 — AI Inference Request Journey 11 — Inference Scaling Lab 12 — Kubernetes AI Infrastructure Lab
INFERENCE MODEL SERVING
Stage 04 / 05 · AI Inference Infrastructure

AI Inference Infrastructure Architecture

AI inference infrastructure transforms a trained model into a production service. Applications send inference requests through API gateways and routing layers to model-serving workers, where GPUs execute model computations and generate responses.

Production inference introduces engineering challenges that differ from distributed training: request latency, concurrency, token throughput, batching efficiency, KV cache memory, model replica placement, autoscaling and service availability.

API Gateway Request Routing Model Serving GPU Inference Continuous Batching KV Cache Autoscaling Kubernetes

From Training Clusters to Production Inference

Distributed training focuses on efficiently updating model parameters across large compute clusters. Inference focuses on serving predictions or generated outputs to applications under latency, throughput and availability constraints.

An inference request may pass through authentication, admission control, model routing, request scheduling, GPU execution and response streaming. Each stage contributes to service behavior.

Engineers must balance GPU utilization against user-facing latency. Larger batches may improve throughput, but queuing and scheduling decisions can increase time to first token or end-to-end response latency.

Engineering principle: Inference capacity is not determined by GPU count alone. Model size, context length, token generation rate, KV cache allocation, concurrency, batching and workload mix all influence achievable service performance.
Production Inference Fundamentals

Four Key Service Dimensions

Latency Time to first token and time to complete an inference response.
Throughput Requests or tokens processed per second.
Concurrency Number of requests actively processed or queued.
Availability Ability to continue serving requests during failures and scaling.
Interactive Production Inference Architecture

Explore the Model-Serving Request Path

Select an operating mode and inspect individual infrastructure components.

PRODUCTION AI INFERENCE · SERVICE ARCHITECTURE Request Flow · Gateway → Router → Model Replica → GPU

Inference Request Journey

Follow a Production Inference Request

1. Request Routing and Model Serving

Gateways enforce authentication, quotas and admission policies. Routers select suitable model deployments and healthy replicas according to capacity, availability and routing policy.

2. GPU Execution and Continuous Batching

Model servers schedule inference work onto GPUs. Continuous batching allows requests to enter and leave active batches dynamically, improving resource utilization while introducing scheduling and latency trade-offs.

3. KV Cache and Memory Management

Autoregressive transformer inference commonly stores attention keys and values from previously processed tokens. KV cache capacity affects concurrent sequences, long-context workloads and GPU memory pressure.

4. Kubernetes and Autoscaling

Kubernetes can orchestrate model-serving workloads, GPU resource allocation, readiness and service discovery. Autoscaling controllers may use queue depth, request rate, utilization or custom inference metrics to adjust serving capacity.

Stage 04 · Practical Engineering Objectives

Three AI Inference Infrastructure Labs

Progress from request-level inference behavior into scaling architecture and Kubernetes production operations.

LAB 10 AI Inference Request Journey Trace a request through gateway, router, scheduler, GPU prefill, token decoding and response streaming. Investigate time to first token and latency.
LAB 11 Inference Scaling Lab Configure serving replicas, concurrency, batching and capacity thresholds. Investigate queue growth, GPU utilization and autoscaling behavior.
LAB 12 Kubernetes AI Infrastructure Lab Investigate GPU scheduling, model-serving deployments, readiness probes, services, resource requests and production infrastructure failures.
Stage 04 · Engineering Lab 10 / 15

AI Inference Request Journey

Trace an LLM inference request through gateway admission, scheduling, GPU prefill, token decoding and response streaming. Investigate how concurrency, context length, batching and KV cache capacity influence latency and request admission.

— Estimated TTFT
— Estimated TPOT
— Estimated Queue Wait
— Peak KV Cache Demand
Ready Inference Assessment

Inference Request Execution Path

● Service Ready
PRODUCTION INFERENCE REQUEST EXECUTION Request Admission → Scheduling → GPU Execution → Streaming
Run the inference simulation to trace the request journey.
Inference Performance Analysis

Latency and Capacity Breakdown

Time to first token includes admission, queueing, scheduling and prefill. Time per output token describes the average interval between generated tokens after the first token.

Estimated Latency Components
Gateway and Routing —
Queue Waiting —
GPU Prefill —
Scheduling Overhead —
Capacity and Serving Interpretation
Active Requests / Replica —
KV Cache Admission —
Batching Behavior —
Recommended Investigation —
Engineering Validation

Resolve an Inference Latency Incident

Configure 64 concurrent requests, 2,048 input tokens, 512 output tokens, one model-serving replica, continuous batching enabled and a 131,072-token KV cache limit per replica. Run the investigation and identify the admission bottleneck.

Simulation assumptions: This lab uses a simplified illustrative inference model rather than an actual LLM serving engine. KV cache demand is approximated using concurrent requests multiplied by input plus output tokens, evenly distributed across replicas. This represents a conservative peak reservation estimate, not actual instantaneous cache occupancy. Real KV cache allocation depends on model layers, KV heads, head dimensions, precision, paging, prefix sharing and sequence lifecycle. TTFT, TPOT, queueing and batching effects are synthetic estimates.
Stage 04 · Engineering Lab 11 / 15

Inference Scaling Lab

Investigate a production AI inference service under changing request demand. Configure incoming traffic, GPU replica capacity, autoscaling thresholds and model warm-up delays. Observe how request queues grow, how replicas become ready and whether the service can recover from overload.

— Ready Serving Replicas
— Final Queued Requests
— Final Serving Throughput
— Estimated Queue Wait
Ready Service Assessment

GPU Inference Replica Fleet

● Service Ready

Serving replicas move from requested capacity through model warm-up to ready capacity. Only ready replicas contribute throughput.

Inference Gateway Request Admission / Routing Model-Serving Replicas · GPU Execution Capacity
Production Investigation

Autoscaling Event Timeline

[READY] Inference scaling simulator initialized [INFO] Configure load and scaling policy [INFO] Run simulation to inspect events
Investigation Ready

Incoming request demand must be matched by ready serving capacity. Autoscaling decisions do not add immediate throughput: GPU scheduling, model loading and readiness introduce delays.

Engineering Checks

1. Compare incoming RPS with ready capacity.
2. Check queue growth during warm-up.
3. Inspect desired versus ready replicas.
4. Evaluate final queue and recovery.
5. Confirm the scaling policy can meet demand.

Engineering Validation

Resolve the Inference Overload Incident

Load the production incident and run it to observe queue growth. Then select the root cause and engineering response. Adjust the autoscaling configuration and run the simulation again. Completion requires a successful recovery with acceptable final queueing delay.

Simulation assumptions: The model uses deterministic one-second time steps, constant incoming RPS, a fixed per-replica service capacity, ideal request distribution and a simplified autoscaling controller. Warm-up delays represent model loading and readiness. Queue wait is approximated as queued requests divided by ready service capacity. Real inference capacity depends on input/output token lengths, batching, GPU memory, model architecture, request mix, scheduling and infrastructure. This simulator does not model a real Kubernetes autoscaler or GPU scheduler.
Stage 04 · Engineering Lab 12 / 15

Kubernetes AI Infrastructure Lab

Investigate how Kubernetes schedules GPU-backed model-serving pods onto accelerator nodes. Configure node GPU capacity, deployment replicas, GPU requests, node availability and readiness behavior. Diagnose Pending pods, unavailable service endpoints and resource fragmentation.

— Ready Model-Serving Pods
— Pending Pods
— GPU Allocation
— Ready Service Endpoints
Ready Deployment Assessment

Kubernetes GPU Cluster

● Cluster Ready

The scheduler places model-serving pods onto nodes with sufficient allocatable GPU resources. A scheduled pod must also become Ready before it can serve traffic through a Kubernetes Service.

☸ Kubernetes Control Plane API Server · Scheduler · Deployment Controller
Scheduling Active
Model-Serving Pod Placement
Production Kubernetes Troubleshooting

Scheduler Events and Deployment Investigation

[READY] Kubernetes GPU cluster initialized [INFO] Configure nodes and model-serving pods [INFO] Run scheduler to inspect placement
Investigation Ready

Compare GPU requests per pod with per-node allocatable capacity. Inspect Pending pods and distinguish scheduler failures from readiness failures.

Engineering Checks

1. Inspect GPU capacity per node.
2. Compare pod GPU requests.
3. Identify Pending pod events.
4. Inspect node readiness.
5. Check model-serving endpoints.

Engineering Validation

Recover a Failed GPU Model-Serving Deployment

Load the GPU scheduling incident and run the scheduler. The deployment requests two model-serving pods with eight GPUs each, but the cluster contains only four GPUs per node.

Identify why the pods cannot be scheduled, then adjust the GPU requests to a feasible value and rerun the scheduler. Completion requires all requested pods to be scheduled and Ready.

Simulation assumptions: This educational scheduler models whole-GPU requests, fixed per-node GPU capacity, node readiness and model-serving readiness. Pods are placed using a simple first-fit algorithm. Real Kubernetes scheduling also evaluates CPU, memory, taints, tolerations, affinity, topology, extended-resource availability, device plugins and admission constraints. GPU allocations do not guarantee sufficient HBM or model compatibility. Service endpoint counts assume a matching Service selector and standard readiness behavior.
AI Infrastructure Learning Workspace · Stage 04

AI Inference Infrastructure — Stage Review

Consolidate the engineering skills developed across inference request processing, GPU serving capacity and Kubernetes model deployment. Review your validated lab results and prepare for production AI infrastructure troubleshooting.

Stage 4 Engineering Completion

0% STAGE COMPLETE
0 / 3 Labs

Complete the three engineering labs to validate Stage 4.

In Progress

Overall Workspace Progress

Validated Engineering Labs 0 / 15
Stage 1
Stage 2
Stage 3
Stage 4
Stage 5

Progress is shared with the Learning Control Bar, dashboard and lab widgets using the workspace's existing browser storage record.

LAB 10 Not Validated

AI Inference Request Journey

Investigate request admission, scheduling, GPU prefill, decoding, TTFT, TPOT and KV cache capacity constraints.

LAB 11 Not Validated

Inference Scaling Lab

Investigate request throughput, GPU replica capacity, autoscaling thresholds, queue growth and warm-up delays.

LAB 12 Not Validated

Kubernetes AI Infrastructure

Diagnose GPU resource requests, Pending model-serving pods, node capacity, scheduling failures and readiness problems.

Architecture Knowledge Review

Production AI Inference Service

These three labs represent connected layers of a production inference platform: request processing, serving capacity and Kubernetes GPU infrastructure.

Inference Gateway Request Admission · TTFT Model Serving Fleet Replicas · Queue · Scaling GPU Infrastructure Kubernetes · GPU Nodes Production Observability Latency · Throughput · GPU Utilization · Pod Health
Inference Latency Diagnosis

Distinguish queueing, prefill, decoding and KV cache pressure.

Not Validated
GPU Serving Capacity Planning

Investigate scaling thresholds, warm-up and request queue recovery.

Not Validated
Kubernetes GPU Troubleshooting

Diagnose scheduling constraints, Pending pods and service readiness.

Not Validated
Next Learning Stage · 05 / 05

Production AI Infrastructure Operations

Move from individual inference components to full production incident investigations. Diagnose GPU cluster failures, compare healthy and degraded infrastructure states, and complete a multi-layer production AI cluster capstone.

Labs 13–15: GPU Cluster Failure Investigation · AI Infrastructure State Comparison · Production AI Cluster Capstone.

Stage 4 progress will update automatically when lab validations are completed.
AI Infrastructure Learning Workspace · Stage 05 / 05

Production AI Infrastructure Operations

Production AI systems depend on coordinated GPU compute, high-performance networking, distributed storage, Kubernetes orchestration and model-serving infrastructure. A failure in one layer can affect training performance, inference latency or application availability. In this final stage, you will investigate interconnected failures rather than isolated components.

Interactive Production GPU Cluster

● Cluster Healthy

Select a failure domain to inspect its effect on the cluster architecture. Click individual components to examine their operational responsibilities.

PRODUCTION AI INFRASTRUCTURE · FAILURE DOMAIN MAP Select an incident to highlight the affected infrastructure layer
Production Troubleshooting Method

Five-Step Infrastructure Investigation

01 Establish Symptoms

Identify failed training jobs, inference errors, elevated latency or degraded GPU utilization.

02 Inspect Telemetry

Review GPU metrics, fabric counters, storage throughput, pod status and application logs.

03 Compare States

Contrast healthy and incident infrastructure configurations and observed behavior.

04 Identify Root Cause

Correlate symptoms across layers and distinguish primary faults from downstream effects.

05 Validate Recovery

Apply a controlled remediation, rerun the workload and verify restored service health.

Final Engineering Labs

Stage 5 Learning Activities

Three advanced engineering investigations combine the concepts developed throughout the AI Infrastructure Learning Workspace.

LAB 13 · FAILURE INVESTIGATION

GPU Cluster Failure Investigation

Investigate realistic multi-layer production failures. Inspect GPU, RDMA, storage and Kubernetes telemetry, identify root causes and validate engineering remediation.

LAB 14 · STATE COMPARISON

AI Infrastructure State Comparison

Compare healthy baseline and degraded production states. Inspect before-and-after configuration, GPU capacity, network throughput, storage and deployment health.

LAB 15 · FINAL CAPSTONE

Production AI Cluster Capstone

Complete a comprehensive production incident. Diagnose interacting infrastructure failures, apply a recovery plan and verify that the end-to-end AI workload is healthy.

Select a failure domain or begin the GPU Cluster Failure Investigation.
Stage 05 · Engineering Lab 13 / 15

GPU Cluster Failure Investigation

Investigate a production AI cluster containing GPU compute, high-performance networking, distributed storage, Kubernetes orchestration and inference serving. Inject an incident, inspect operational evidence, identify the root cause and validate service recovery.

— GPU Utilization
— RDMA Fabric Throughput
— Storage Read Throughput
— Ready Serving Pods
Healthy Cluster Assessment

Production AI Cluster Topology

● Healthy Baseline

Select an infrastructure component to inspect its operational responsibility. Incident injection highlights the affected fault domain and associated infrastructure links.

Production Cluster · Select a Component to Inspect
Production Cluster

All infrastructure domains are available in the healthy baseline.

Inspect component telemetry and correlate symptoms across layers.
Incident Investigation

Production Event Timeline and Evidence

[READY] GPU cluster incident lab initialized [INFO] Select a production incident [INFO] Inject the incident to inspect telemetry
Healthy Production Baseline

GPU compute, networking, storage, Kubernetes and inference serving are operating within the simulated healthy baseline.

Engineering Interpretation

Baseline metrics provide a reference for investigating future degraded states.

Engineering Validation

Diagnose and Recover the Production Incident

Inject an incident and inspect the telemetry and event timeline. Select the primary root cause and an appropriate engineering response. Apply the remediation and validate that the cluster has returned to its healthy baseline.

Simulation assumptions: Telemetry values and incident events are illustrative engineering scenarios rather than measured production data. Each scenario represents one primary fault domain with simplified downstream effects. Remediation is modeled as a deterministic recovery action. Real production diagnosis requires validating device, network, storage, orchestration and application evidence before implementing changes.
Stage 05 · Engineering Lab 14 / 15

AI Infrastructure State Comparison

Compare a healthy AI infrastructure baseline with a degraded production state. Identify configuration and telemetry changes across GPU compute, RDMA networking, distributed storage, Kubernetes and inference serving. Correlate the differences to determine the primary root cause and validate recovery.

— Detected State Changes
— Primary Fault Domain
— GPU Utilization Impact
Ready Comparison Assessment

Infrastructure Change Map

● Awaiting Comparison

Compare baseline and incident states. The primary changed infrastructure component is highlighted in red. Select a component to inspect its state and operational significance.

Baseline → Incident → Recovery
Infrastructure State Explorer

Run a comparison to inspect changes between healthy and degraded states.

State Comparison Engine

Healthy Baseline vs Incident Snapshot

Healthy Baseline REFERENCE STATE
Incident Snapshot AWAITING COMPARISON
Detected Metric Deltas
Engineering Investigation

State Difference Report and Incident Evidence

[READY] Infrastructure comparison engine initialized [INFO] Select an incident snapshot [INFO] Compare against healthy baseline
Awaiting Infrastructure Comparison

Run a state comparison to identify infrastructure changes.

Root-Cause Correlation

Compare primary fault evidence with downstream workload symptoms.

Engineering Validation

Diagnose and Verify Infrastructure Recovery

Compare a degraded snapshot with the healthy baseline. Identify the primary fault domain, select an appropriate engineering remediation, then verify that the recovered state matches the reference baseline.

Simulation assumptions: Baseline and incident values are deterministic educational snapshots. Differences represent correlated infrastructure symptoms, not proof of causation. Real production state comparison requires timestamps, configuration revisions, telemetry correlation, workload context and recovery verification.
Stage 05 · Engineering Lab 15 / 15 · Final Capstone

Production AI Cluster Capstone

Investigate a multi-layer production AI infrastructure incident. Three independent failures affect GPU compute, RDMA networking, distributed storage, Kubernetes or inference serving. Collect evidence using the investigation console, identify the fault domains, apply the correct remediations and validate complete workload recovery.

— GPU Utilization
— RDMA Throughput
— Storage Throughput
— Ready Serving Pods
Ready Production Cluster State

Production AI Cluster Topology

● Awaiting Incident

Multiple independent faults may be active. Red components indicate unresolved infrastructure failures. Select any component to inspect its role.

Multi-Layer Production Infrastructure
Production Infrastructure

Start an incident and inspect the affected infrastructure domains.

Production Investigation Console

Inspect Infrastructure Evidence

Use the built-in investigation commands to inspect cluster health, GPU devices, RDMA networking, storage, Kubernetes and inference serving. This is an educational command interface rather than a real operating-system shell.

[READY] Production AI Cluster Capstone [INFO] Select and start an incident [HELP] Available commands: help show cluster show gpu show rdma show storage show kubernetes show inference show incidents
Incident Remediation Workbench

Identify and Recover Three Independent Failures

Each incident contains three independent faults. Collect evidence from the console, then choose a root cause and remediation for each investigation slot. Recovery actions are applied to the simulated infrastructure state.

Final Engineering Validation

Production Cluster Recovery Checklist

○ Three Root Causes Identified
○ Three Remediations Applied
○ Infrastructure Baseline Restored
○ End-to-End Workload Healthy
Simulation assumptions: The capstone uses deterministic fault scenarios and simplified infrastructure metrics. Each scenario contains three independent root causes. Recovery restores the corresponding modeled infrastructure domain. Real production troubleshooting requires deeper telemetry, change control, device-level diagnostics, workload testing and operational approval.
Stage 05 · Final Progress & Achievement

AI Infrastructure Engineering Achievement

Review your progress across the complete AI Infrastructure Learning Workspace. Five learning stages and fifteen engineering labs cover GPU architecture, high-performance networking, distributed training, inference infrastructure and production operations.

0% Workspace Complete
Engineering Learning Record

Your Learning Journey

Your completed labs will appear here.

0 / 15 Engineering Labs Completed
0 / 5 Learning Stages Completed
15 Labs Remaining
Five-Stage Learning Journey

Engineering Lab Completion

Engineering Competencies

Skills Demonstrated Through the Labs

Each stage represents a different area of AI infrastructure engineering. Completion reflects the corresponding lab validation recorded in this browser.

Network Insight · Learning Achievement

Final Achievement Locked

Complete all fifteen engineering labs to unlock the AI Infrastructure Engineering Learning Achievement.

Learning record: Progress is stored in your browser using the workspace's existing localStorage system. It is not synchronized across devices. This achievement is an educational completion record, not an accredited professional certification.
☕
Support Network Insight
Help support the development of interactive networking labs, BGP simulations, and educational content.
Support the Project