AI Infrastructure Learning Workspace
Explore how modern AI infrastructure works in practice. Investigate GPU servers, high-speed cluster fabrics, distributed training, model-serving platforms and production performance through connected engineering activities.
Interactive AI Infrastructure Architecture
Training traffic · AnimatedSelect a component in the diagram to inspect its engineering role. Flow paths are conceptual, not measured network traces.
Learning Control Bar
Navigate five engineering stages and track all fifteen interactive activities.
Infrastructure Foundations
Explore GPU compute and infrastructure architecture.
Unified AI Infrastructure Learning Dashboard
Track your progress across five engineering stages and fifteen interactive activities. Explore GPU architecture, cluster networking, distributed training, inference infrastructure and production troubleshooting.
AI Infrastructure Learning Pathway
0 of 15 activities completedBegin with Infrastructure Foundations
Explore the first learning stage.
Explore the AI Infrastructure Workspace
Follow five connected engineering stages, from understanding GPU infrastructure and high-speed networking to distributed training, production inference and failure investigation. Select a stage to explore its objectives and three practical labs. Your learning progress is saved automatically.
Infrastructure Foundations
AI Infrastructure Architecture & Foundations
Artificial intelligence infrastructure is more than a collection of powerful GPUs. Production AI systems depend on tightly coordinated compute, memory, networking, storage and orchestration layers. Understanding how these components interact is essential for designing scalable training clusters, reliable inference platforms and efficient AI data centres.
In this first learning stage, we examine the physical and logical architecture supporting AI workloads. We will follow how infrastructure resources are provisioned, how GPU servers communicate, where datasets and model checkpoints are stored, and why architectural bottlenecks can significantly reduce application performance.
What Makes AI Infrastructure Different?
Traditional enterprise applications frequently distribute independent requests across CPU-based servers. Large-scale AI training instead requires many accelerators to cooperate on shared computational workloads, exchanging intermediate data and synchronizing results.
This changes infrastructure design priorities. GPU memory capacity, interconnect topology, communication latency, storage throughput and workload placement can all influence whether expensive compute resources remain productive.
Explore the Five AI Infrastructure Layers
Select a component to inspect its role, dependencies and operational risks.
1. GPU Servers and Accelerator Architecture
GPU servers combine host processors, accelerator devices, high-bandwidth memory, PCIe connectivity and often specialized accelerator interconnects. The topology determines which data transfers remain local to a server and which must traverse the external network fabric.
2. High-Speed AI Networking
Distributed workloads generate significant east-west communication. Leaf-spine networks, Ethernet with RoCEv2, and InfiniBand can provide high-throughput interconnects. Their operational characteristics, congestion controls and management requirements differ.
3. Data Pipelines and Storage Performance
Training clusters require reliable access to large datasets and persistent checkpoint storage. Storage throughput, metadata operations, caching and data locality affect how efficiently GPUs receive work and recover after faults.
4. Infrastructure Capacity and Utilization
Capacity planning must consider accelerator memory, compute utilization, network bandwidth, storage performance, power, cooling and workload concurrency. Increasing GPU count alone does not guarantee proportional throughput improvement.
What You Will Investigate
The following three labs move from architectural understanding to GPU server inspection and infrastructure capacity decisions.
AI Infrastructure Architecture Explorer
Investigate a production-style AI cluster. Trace workload dependencies, inject infrastructure failures, inspect operational evidence and determine the correct recovery action.
AI Cluster Dependency Topology
● HealthyAll infrastructure components are operational. Select an incident and run an investigation.
Infrastructure Journey
Follow the selected workload and identify where the simulated incident affects execution.
Cluster Telemetry
Illustrative measurements generated by the current scenario model.
Diagnose and Recover the AI Cluster
Select the suspected root cause and the corresponding remediation. Correctly resolve an incident to complete Lab 01.
GPU Server Architecture Lab
Explore the internal architecture of an AI GPU server. Investigate accelerator topology, CPU connectivity, GPU memory and high-speed interconnects. Configure data-transfer experiments and identify the component limiting effective communication performance.
Interactive GPU Server Cutaway
● Architecture ReadyGPU Communication Performance Investigator
Configure a data transfer and investigate how interconnect selection and injected bottlenecks affect its estimated completion time. This is a simplified bandwidth model, not a GPU hardware benchmark.
Identify the Performance Bottleneck
Run a transfer experiment with an injected bottleneck. Use the reported bandwidth and communication trace to identify the constrained resource and an appropriate remediation.
AI Cluster Capacity Planner
Design an AI accelerator cluster and investigate whether its available GPU memory and compute capacity can support a selected workload. Compare model sizes, precision formats, GPU counts and workload assumptions to understand practical infrastructure sizing constraints.
GPU Cluster Allocation Visualizer
● Capacity AvailableWorkload memory compared with usable cluster memory.
Evaluate the Engineering Trade-Offs
Increasing GPU count increases aggregate memory and potential compute capacity, but real workloads also depend on communication overhead, memory distribution, scheduling, network bandwidth and storage throughput.
Capacity Planning Challenge
Create a configuration that exceeds the cluster's available GPU memory. Run the capacity calculation, identify the constraint and choose an appropriate capacity remediation.
AI Infrastructure Foundations — Stage Progress
Review your progress across the three foundational engineering labs. Completion is recorded automatically when you successfully validate each lab's investigation.
Foundations Completion
Complete all three activities to finish the foundations stage.
AI Infrastructure Architecture Explorer
Investigate AI cluster dependencies, inject infrastructure faults and validate recovery actions.
Open Architecture Lab →GPU Server Architecture Lab
Examine accelerator hardware, memory placement, interconnect bandwidth and GPU transfer bottlenecks.
Open GPU Server Lab →AI Cluster Capacity Planner
Estimate model memory requirements, evaluate cluster sizing and diagnose capacity constraints.
Open Capacity Planner →Skills Developed
GPU Cluster Networking
Move from individual GPU servers into high-performance AI cluster networking. Investigate leaf-spine fabrics, RDMA transports and congestion behavior.
GPU Cluster Networking Architecture
Large AI workloads depend on fast, predictable communication between accelerators distributed across multiple servers. The network connecting these servers is therefore a fundamental part of the AI compute system—not simply a supporting connectivity service.
In this stage, we investigate how GPU clusters use high-performance network fabrics to exchange tensors, synchronize distributed workloads, access remote storage and support large-scale training. We examine the differences between conventional Ethernet networking and RDMA-capable transport, including RoCEv2 and InfiniBand.
Why AI Networking Requires a Different Design Approach
In distributed AI training, accelerators repeatedly exchange gradients, activations and other intermediate data. Communication may involve point-to-point transfers or collective operations across many workers.
If the network introduces excessive latency, congestion or insufficient throughput, GPUs can spend more time waiting for synchronization rather than performing useful computation.
Modern AI clusters therefore emphasize predictable east-west bandwidth, high-radix switching, appropriate oversubscription, efficient transport and comprehensive fabric telemetry.
Four Critical Engineering Dimensions
Explore an AI Leaf-Spine Fabric
Select a traffic pattern, then inspect GPU servers, leaf switches, spine switches and the external network connection.
Every leaf switch connects to both spine switches, providing multiple fabric paths.
1. Leaf-Spine and Clos Network Design
Leaf-spine networks provide predictable connectivity between server-facing leaf switches through a shared spine layer. Multiple equal-cost paths can improve bandwidth utilization and resilience. Actual performance depends on link capacity, hashing, oversubscription and traffic distribution.
2. RDMA, RoCEv2 and InfiniBand
Remote Direct Memory Access enables data movement with reduced CPU involvement. RoCEv2 transports RDMA over routable Ethernet/IP networks, while InfiniBand uses its own networking architecture and transport mechanisms. Both require appropriate fabric design and operational controls.
3. Distributed GPU Communication
Distributed training frequently uses collective communication operations such as All-Reduce, All-Gather and Reduce-Scatter. Their performance depends on message size, topology, bandwidth, latency and communication algorithms.
4. Congestion and Network Telemetry
AI communication can create synchronized bursts and sustained high-throughput flows. Engineers investigate queue occupancy, ECN marking, packet loss, retransmissions, PFC behavior where deployed, and application-level communication delays.
Three GPU Networking Labs
These labs progress from fabric construction to transport behavior and production congestion troubleshooting.
AI Leaf-Spine Fabric Builder
Design a high-performance GPU cluster network. Configure spine and leaf switches, GPU server density, link bandwidth and network failures. Evaluate fabric oversubscription, available capacity and path resilience.
Configurable GPU Network Fabric
● Fabric HealthyThe diagram represents logical server groups and leaf-to-spine connections. Parallel physical uplinks are represented by the configured link count.
Traffic Demand and Failure Impact
Compare aggregate server-side traffic demand with available uplink capacity. This simplified model assumes traffic crosses the leaf-to-spine fabric and is distributed evenly across leaf switches.
Fabric Failure Investigation
Inject a spine failure and evaluate the resulting uplink capacity and path redundancy. Identify the infrastructure impact and select an appropriate engineering response.
RDMA Transport Explorer
Investigate how distributed AI workloads move data between accelerator servers. Compare conventional TCP communication with RoCEv2 and InfiniBand, inspect the transport journey, and explore how network conditions influence communication performance.
GPU-to-GPU Communication Journey
● Transport ReadyThe diagram is a conceptual data path. Actual transfers depend on registered memory, NIC capabilities, GPU-direct support, operation type and transport implementation.
Interpreting the Transfer Estimate
This educational model estimates serialization time using configured transfer size and an assumed fraction of available bandwidth. It illustrates how degraded network conditions can reduce effective throughput.
RoCEv2 Congestion Investigation
Select RoCEv2 and inject fabric congestion. Compare the healthy and degraded transfer estimates, then identify the likely network-level issue and an appropriate investigation or remediation action.
Network Congestion Investigation
Investigate a simulated production AI cluster incident. Examine switch queue occupancy, ECN marking, PFC pauses, packet drops and GPU synchronization delays. Use the available telemetry to identify the most likely network bottleneck and select a remediation strategy.
AI Cluster Fabric Incident Map
● Fabric OperationalSelect either spine switch to inspect its modeled traffic distribution and queue condition.
Incident Timeline and Engineering Interpretation
Compare network-level counters with GPU collective communication delay. A high application delay alone does not prove network congestion; supporting fabric evidence is required.
Investigation Checklist
1. Identify the affected fabric path.
2. Examine queue and ECN behavior.
3. Check PFC pauses and packet drops.
4. Compare ECMP traffic distribution.
5. Correlate network symptoms with GPU delays.
Diagnose the Network Incident
Run an incident investigation, inspect the telemetry and select the most likely root cause and engineering response. Validation is based on the current scenario.
GPU Cluster Networking — Stage Progress
Review your progress across the three GPU networking engineering labs. Completion is recorded automatically when you successfully validate the fabric design, RDMA transport investigation and network congestion troubleshooting exercises.
GPU Networking Completion
Complete all three activities to finish the GPU Cluster Networking stage.
AI Leaf-Spine Fabric Builder
Configure GPU network topology, calculate oversubscription, evaluate ECMP resilience and investigate spine failures.
Open Fabric Builder →RDMA Transport Explorer
Compare TCP, RoCEv2 and InfiniBand, investigate GPU data movement and evaluate congestion-related throughput loss.
Open RDMA Lab →Network Congestion Investigation
Diagnose ECMP imbalance, microbursts, PFC pause propagation, queue pressure and packet loss from telemetry evidence.
Open Congestion Lab →Networking Skills Developed
Distributed AI Training
Move from the network fabric into distributed training execution. Investigate data parallelism, tensor and pipeline parallelism, GPU collective communication, synchronization overhead and training performance bottlenecks.
Distributed AI Training Architecture
Training large AI models often requires many GPUs working together across multiple servers. Distributed training coordinates computation, model parameters, activations and gradients across those accelerators while balancing compute efficiency, memory capacity and communication overhead.
In this stage, we investigate how parallel training strategies divide work between GPUs, how collective communication synchronizes distributed workers, and how engineers diagnose performance bottlenecks that reduce training throughput.
Why Distributed Training Is an Infrastructure Problem
A single GPU has finite compute throughput and high-bandwidth memory. Large models and training datasets may require parallel execution across many accelerators.
Parallelism introduces communication. Depending on the training strategy, workers may exchange gradients, partial tensor results, activations or model-state information.
The training system must coordinate these operations while minimizing idle time, balancing memory usage and maintaining sufficient data delivery.
Four Core Engineering Concepts
Explore Multi-GPU Training Strategies
Select a parallelism strategy and inspect how four GPU workers participate in distributed computation.
Follow a Distributed Training Step
1. Data Parallelism
Multiple workers maintain model replicas while processing different training data. Gradients must be synchronized before equivalent model updates are applied. Distributed Data Parallel and sharded variants use different model-state management strategies.
2. Tensor Parallelism
Model computations and tensors are partitioned across multiple GPUs. Workers cooperate during layer execution, often requiring frequent communication of partial results. Placement and interconnect bandwidth strongly influence performance.
3. Pipeline Parallelism
Different model layers or stages execute on different GPUs. Microbatches flow through the pipeline to improve utilization. Pipeline bubbles, stage imbalance and activation transfers can limit efficiency.
4. Collective Communication
Collective operations such as All-Reduce, All-Gather and Reduce-Scatter coordinate data across workers. Their cost depends on message size, worker count, topology, algorithm and network performance.
Three Distributed Training Labs
Progress from parallelism architecture into collective communication and then production training performance investigation.
Distributed Training Architecture Planner
Design a multi-GPU training configuration. Select model size, GPU resources and parallelism strategy, then investigate model-state memory, worker communication, pipeline efficiency and scaling limitations.
Distributed GPU Worker Architecture
● Configuration ReadyThe diagram shows up to eight representative GPU workers. Configurations with more GPUs are summarized in the metrics and worker inspector.
Memory and Communication Trade-offs
Training memory includes model parameters, gradients and optimizer state. The planner uses a simplified mixed-precision Adam-style model-state budget and estimates the effect of partitioning and state sharding.
Per-GPU Memory Budget
Architecture Interpretation
Resolve a Distributed Training Capacity Incident
Configure a 70B-parameter model on eight 80 GB GPUs using TP 4, PP 2 and sharded model states. Calculate the architecture, inspect the GPU memory estimate and identify why the design requires model partitioning and state sharding.
GPU Collective Communication Lab
Explore how distributed GPU workers exchange gradients, tensor partitions and intermediate results. Compare communication algorithms, investigate collective phases and estimate the effect of bandwidth, message size and network bottlenecks.
GPU Collective Communication Topology
● Fabric ReadySelect a collective operation and run the simulation.
Communication Cost and Bandwidth Sensitivity
Collective operations exchange data across multiple GPU workers. Performance depends on algorithm choice, message size, topology, link bandwidth, synchronization overhead and communication contention.
Communication Cost Breakdown
Engineering Interpretation
Diagnose a Slow All-Reduce Operation
Configure eight GPU workers, Ring All-Reduce, a 1 GB tensor, 200 Gbps links, 75% link efficiency and a congested fabric. Run the collective operation and investigate why communication time increases.
Training Performance Investigator
Investigate a simulated distributed AI training performance incident. Examine GPU utilization, training-step duration, communication overhead, pipeline bubbles and data-loading delays. Correlate infrastructure telemetry with training execution behavior to identify the most likely bottleneck.
Distributed Training Execution Timeline
● Training OperationalEach row represents an illustrative GPU worker training step. Segment widths indicate the relative contribution of compute, communication, data waiting and pipeline idle time.
Training Incident Timeline and Analysis
A slow training step can originate from data loading, GPU compute, collective communication or pipeline scheduling. Use the execution timeline and supporting telemetry to distinguish these causes.
Investigation Checklist
1. Compare step duration with baseline.
2. Inspect GPU utilization.
3. Examine communication overhead.
4. Identify data-loading stalls.
5. Check pipeline idle time and worker imbalance.
Diagnose the Training Performance Incident
Select one of the four degraded training scenarios, run the investigation and examine the telemetry. Identify the root cause and select the appropriate engineering response. Successfully diagnose all four incident types to complete Lab 09.
Validated incident types: 0 / 4
Distributed AI Training — Learning Progress
Review your progress across the three distributed AI training engineering labs. Confirm your understanding of training architecture, GPU collective communication and performance troubleshooting before advancing to AI inference infrastructure.
0 of 3 engineering labs completed
Skills Demonstrated in Stage 3
Configure tensor, pipeline and data parallelism. Interpret GPU model-state memory requirements and distributed training resource allocation.
Awaiting Lab 07Explain All-Reduce, All-Gather and Reduce-Scatter. Investigate how communication bandwidth influences distributed training.
Awaiting Lab 08Correlate GPU utilization, training-step latency, data-loading stalls, collective overhead and pipeline idle time.
Awaiting Lab 09Overall AI Infrastructure Progress
Your progress is shared with the workspace dashboard and learning pathway using the existing browser learning record.
AI Inference Infrastructure
Move from distributed model training to production inference systems. Explore request execution, model serving, scaling behavior and Kubernetes-based AI infrastructure.
AI Inference Infrastructure Architecture
AI inference infrastructure transforms a trained model into a production service. Applications send inference requests through API gateways and routing layers to model-serving workers, where GPUs execute model computations and generate responses.
Production inference introduces engineering challenges that differ from distributed training: request latency, concurrency, token throughput, batching efficiency, KV cache memory, model replica placement, autoscaling and service availability.
From Training Clusters to Production Inference
Distributed training focuses on efficiently updating model parameters across large compute clusters. Inference focuses on serving predictions or generated outputs to applications under latency, throughput and availability constraints.
An inference request may pass through authentication, admission control, model routing, request scheduling, GPU execution and response streaming. Each stage contributes to service behavior.
Engineers must balance GPU utilization against user-facing latency. Larger batches may improve throughput, but queuing and scheduling decisions can increase time to first token or end-to-end response latency.
Four Key Service Dimensions
Explore the Model-Serving Request Path
Select an operating mode and inspect individual infrastructure components.
Follow a Production Inference Request
1. Request Routing and Model Serving
Gateways enforce authentication, quotas and admission policies. Routers select suitable model deployments and healthy replicas according to capacity, availability and routing policy.
2. GPU Execution and Continuous Batching
Model servers schedule inference work onto GPUs. Continuous batching allows requests to enter and leave active batches dynamically, improving resource utilization while introducing scheduling and latency trade-offs.
3. KV Cache and Memory Management
Autoregressive transformer inference commonly stores attention keys and values from previously processed tokens. KV cache capacity affects concurrent sequences, long-context workloads and GPU memory pressure.
4. Kubernetes and Autoscaling
Kubernetes can orchestrate model-serving workloads, GPU resource allocation, readiness and service discovery. Autoscaling controllers may use queue depth, request rate, utilization or custom inference metrics to adjust serving capacity.
Three AI Inference Infrastructure Labs
Progress from request-level inference behavior into scaling architecture and Kubernetes production operations.
AI Inference Request Journey
Trace an LLM inference request through gateway admission, scheduling, GPU prefill, token decoding and response streaming. Investigate how concurrency, context length, batching and KV cache capacity influence latency and request admission.
Inference Request Execution Path
● Service ReadyLatency and Capacity Breakdown
Time to first token includes admission, queueing, scheduling and prefill. Time per output token describes the average interval between generated tokens after the first token.
Estimated Latency Components
Capacity and Serving Interpretation
Resolve an Inference Latency Incident
Configure 64 concurrent requests, 2,048 input tokens, 512 output tokens, one model-serving replica, continuous batching enabled and a 131,072-token KV cache limit per replica. Run the investigation and identify the admission bottleneck.
Inference Scaling Lab
Investigate a production AI inference service under changing request demand. Configure incoming traffic, GPU replica capacity, autoscaling thresholds and model warm-up delays. Observe how request queues grow, how replicas become ready and whether the service can recover from overload.
GPU Inference Replica Fleet
● Service ReadyServing replicas move from requested capacity through model warm-up to ready capacity. Only ready replicas contribute throughput.
Autoscaling Event Timeline
Incoming request demand must be matched by ready serving capacity. Autoscaling decisions do not add immediate throughput: GPU scheduling, model loading and readiness introduce delays.
Engineering Checks
1. Compare incoming RPS with ready capacity.
2. Check queue growth during warm-up.
3. Inspect desired versus ready replicas.
4. Evaluate final queue and recovery.
5. Confirm the scaling policy can meet demand.
Resolve the Inference Overload Incident
Load the production incident and run it to observe queue growth. Then select the root cause and engineering response. Adjust the autoscaling configuration and run the simulation again. Completion requires a successful recovery with acceptable final queueing delay.
Kubernetes AI Infrastructure Lab
Investigate how Kubernetes schedules GPU-backed model-serving pods onto accelerator nodes. Configure node GPU capacity, deployment replicas, GPU requests, node availability and readiness behavior. Diagnose Pending pods, unavailable service endpoints and resource fragmentation.
Kubernetes GPU Cluster
● Cluster ReadyThe scheduler places model-serving pods onto nodes with sufficient allocatable GPU resources. A scheduled pod must also become Ready before it can serve traffic through a Kubernetes Service.
Model-Serving Pod Placement
Scheduler Events and Deployment Investigation
Compare GPU requests per pod with per-node allocatable capacity. Inspect Pending pods and distinguish scheduler failures from readiness failures.
Engineering Checks
1. Inspect GPU capacity per node.
2. Compare pod GPU requests.
3. Identify Pending pod events.
4. Inspect node readiness.
5. Check model-serving endpoints.
Recover a Failed GPU Model-Serving Deployment
Load the GPU scheduling incident and run the scheduler. The deployment requests two model-serving pods with eight GPUs each, but the cluster contains only four GPUs per node.
Identify why the pods cannot be scheduled, then adjust the GPU requests to a feasible value and rerun the scheduler. Completion requires all requested pods to be scheduled and Ready.
AI Inference Infrastructure — Stage Review
Consolidate the engineering skills developed across inference request processing, GPU serving capacity and Kubernetes model deployment. Review your validated lab results and prepare for production AI infrastructure troubleshooting.
Stage 4 Engineering Completion
Complete the three engineering labs to validate Stage 4.
In ProgressOverall Workspace Progress
Progress is shared with the Learning Control Bar, dashboard and lab widgets using the workspace's existing browser storage record.
AI Inference Request Journey
Investigate request admission, scheduling, GPU prefill, decoding, TTFT, TPOT and KV cache capacity constraints.
Inference Scaling Lab
Investigate request throughput, GPU replica capacity, autoscaling thresholds, queue growth and warm-up delays.
Kubernetes AI Infrastructure
Diagnose GPU resource requests, Pending model-serving pods, node capacity, scheduling failures and readiness problems.
Production AI Inference Service
These three labs represent connected layers of a production inference platform: request processing, serving capacity and Kubernetes GPU infrastructure.
Distinguish queueing, prefill, decoding and KV cache pressure.
Not ValidatedInvestigate scaling thresholds, warm-up and request queue recovery.
Not ValidatedDiagnose scheduling constraints, Pending pods and service readiness.
Not ValidatedProduction AI Infrastructure Operations
Move from individual inference components to full production incident investigations. Diagnose GPU cluster failures, compare healthy and degraded infrastructure states, and complete a multi-layer production AI cluster capstone.
Labs 13–15: GPU Cluster Failure Investigation · AI Infrastructure State Comparison · Production AI Cluster Capstone.
Production AI Infrastructure Operations
Production AI systems depend on coordinated GPU compute, high-performance networking, distributed storage, Kubernetes orchestration and model-serving infrastructure. A failure in one layer can affect training performance, inference latency or application availability. In this final stage, you will investigate interconnected failures rather than isolated components.
Interactive Production GPU Cluster
● Cluster HealthySelect a failure domain to inspect its effect on the cluster architecture. Click individual components to examine their operational responsibilities.
Five-Step Infrastructure Investigation
Identify failed training jobs, inference errors, elevated latency or degraded GPU utilization.
Review GPU metrics, fabric counters, storage throughput, pod status and application logs.
Contrast healthy and incident infrastructure configurations and observed behavior.
Correlate symptoms across layers and distinguish primary faults from downstream effects.
Apply a controlled remediation, rerun the workload and verify restored service health.
Stage 5 Learning Activities
Three advanced engineering investigations combine the concepts developed throughout the AI Infrastructure Learning Workspace.
GPU Cluster Failure Investigation
Investigate realistic multi-layer production failures. Inspect GPU, RDMA, storage and Kubernetes telemetry, identify root causes and validate engineering remediation.
AI Infrastructure State Comparison
Compare healthy baseline and degraded production states. Inspect before-and-after configuration, GPU capacity, network throughput, storage and deployment health.
Production AI Cluster Capstone
Complete a comprehensive production incident. Diagnose interacting infrastructure failures, apply a recovery plan and verify that the end-to-end AI workload is healthy.
From Architecture to Incident Resolution
Begin the final set of infrastructure troubleshooting labs.
GPU Cluster Failure Investigation
Investigate a production AI cluster containing GPU compute, high-performance networking, distributed storage, Kubernetes orchestration and inference serving. Inject an incident, inspect operational evidence, identify the root cause and validate service recovery.
Production AI Cluster Topology
● Healthy BaselineSelect an infrastructure component to inspect its operational responsibility. Incident injection highlights the affected fault domain and associated infrastructure links.
All infrastructure domains are available in the healthy baseline.
Inspect component telemetry and correlate symptoms across layers.Production Event Timeline and Evidence
GPU compute, networking, storage, Kubernetes and inference serving are operating within the simulated healthy baseline.
Engineering Interpretation
Baseline metrics provide a reference for investigating future degraded states.
Diagnose and Recover the Production Incident
Inject an incident and inspect the telemetry and event timeline. Select the primary root cause and an appropriate engineering response. Apply the remediation and validate that the cluster has returned to its healthy baseline.
AI Infrastructure State Comparison
Compare a healthy AI infrastructure baseline with a degraded production state. Identify configuration and telemetry changes across GPU compute, RDMA networking, distributed storage, Kubernetes and inference serving. Correlate the differences to determine the primary root cause and validate recovery.
Infrastructure Change Map
● Awaiting ComparisonCompare baseline and incident states. The primary changed infrastructure component is highlighted in red. Select a component to inspect its state and operational significance.
Run a comparison to inspect changes between healthy and degraded states.
Healthy Baseline vs Incident Snapshot
Detected Metric Deltas
State Difference Report and Incident Evidence
Run a state comparison to identify infrastructure changes.
Root-Cause Correlation
Compare primary fault evidence with downstream workload symptoms.
Diagnose and Verify Infrastructure Recovery
Compare a degraded snapshot with the healthy baseline. Identify the primary fault domain, select an appropriate engineering remediation, then verify that the recovered state matches the reference baseline.
Production AI Cluster Capstone
Investigate a multi-layer production AI infrastructure incident. Three independent failures affect GPU compute, RDMA networking, distributed storage, Kubernetes or inference serving. Collect evidence using the investigation console, identify the fault domains, apply the correct remediations and validate complete workload recovery.
Production AI Cluster Topology
● Awaiting IncidentMultiple independent faults may be active. Red components indicate unresolved infrastructure failures. Select any component to inspect its role.
Start an incident and inspect the affected infrastructure domains.
Inspect Infrastructure Evidence
Use the built-in investigation commands to inspect cluster health, GPU devices, RDMA networking, storage, Kubernetes and inference serving. This is an educational command interface rather than a real operating-system shell.
Identify and Recover Three Independent Failures
Each incident contains three independent faults. Collect evidence from the console, then choose a root cause and remediation for each investigation slot. Recovery actions are applied to the simulated infrastructure state.
Production Cluster Recovery Checklist
AI Infrastructure Engineering Achievement
Review your progress across the complete AI Infrastructure Learning Workspace. Five learning stages and fifteen engineering labs cover GPU architecture, high-performance networking, distributed training, inference infrastructure and production operations.
Your Learning Journey
Your completed labs will appear here.
Engineering Lab Completion
Skills Demonstrated Through the Labs
Each stage represents a different area of AI infrastructure engineering. Completion reflects the corresponding lab validation recorded in this browser.
Final Achievement Locked
Complete all fifteen engineering labs to unlock the AI Infrastructure Engineering Learning Achievement.