NVIDIA NVLink and NVSwitch Architecture Explained
Modern AI systems depend on moving enormous volumes of data between GPUs. As model sizes increase and training becomes distributed across many accelerators, GPU interconnect architecture becomes a critical determinant of performance.
NVIDIA NVLink provides high-bandwidth GPU interconnect connectivity, while NVSwitch extends communication across larger groups of accelerators through a switched fabric. Together, these technologies enable tightly coupled scale-up architectures such as the GB200 NVL72 system.
This guide examines how GPU communication works, how NVSwitch fabrics are organized, how collective operations use the interconnect and how engineers design and troubleshoot production GPU systems.
NVLink / NVSwitch Fabric
A switched scale-up fabric enables communication between participating GPUs. NVSwitch provides the switching infrastructure, while NVLink supplies the high-bandwidth interconnect.
Why GPU Interconnect Architecture Matters
Large AI models frequently distribute parameters, activations and computations across multiple GPUs. During training, accelerators exchange gradients and intermediate results. During inference, model parallelism can require repeated communication between participating GPUs.
If data movement is slow relative to computation, GPUs may spend significant time waiting for communication rather than performing useful model execution.
From Direct Links to Switched Fabrics
Direct GPU interconnects provide high-bandwidth communication between connected accelerators. As the number of GPUs increases, building efficient connectivity becomes more complex.
NVSwitch enables switched communication across a larger scale-up domain, supporting architectures in which multiple GPUs participate in tightly coordinated execution.
Core architecture distinction: NVLink and NVSwitch are scale-up technologies connecting tightly coupled accelerators. Ethernet and InfiniBand commonly provide scale-out connectivity between systems and racks. Their roles are complementary, not interchangeable.
Three Layers of GPU Communication Architecture
What This Engineering Guide Will Cover
The following nine sections investigate NVLink architecture fundamentals, the evolution of NVLink generations, NVSwitch internals, GPU peer-to-peer transfers, GB200 NVL72 rack-scale design, NCCL collectives, scale-up versus scale-out networking, performance engineering and production troubleshooting.
Each section combines technical explanation with an architecture visualization, followed by a dedicated interactive engineering lab for deeper exploration.
The architecture visual is educational and simplified. Physical GPU connectivity, switch implementation and supported features vary by NVIDIA platform and generation.
NVIDIA NVLink & NVSwitch — Architecture Navigator
Explore the engineering layers of NVIDIA GPU interconnect architecture, from NVLink fundamentals and NVSwitch internals to GB200 NVL72, NCCL collectives, scale-out networking, performance analysis and production troubleshooting.
NVLink Architecture Fundamentals
Understand how NVIDIA NVLink enables high-bandwidth communication between supported processors and GPUs, and how its role differs from PCIe and switched fabrics.
Explore All Nine Engineering Sections
Understanding GPU Interconnect Architecture
NVIDIA NVLink is a high-bandwidth interconnect technology designed to enable efficient communication between supported processors and GPUs. It plays a central role in multi-GPU systems where accelerators must exchange model parameters, activations, gradients and intermediate results.
Traditional PCIe connectivity provides a general-purpose I/O interconnect, while NVLink offers a specialized communication path for tightly coupled accelerator workloads. NVSwitch extends NVLink connectivity through switching infrastructure, allowing larger groups of GPUs to communicate within a scale-up domain.
PCIe Communication Path
Two GPUs communicate using a PCIe topology. Peer-to-peer capability and the exact route depend on the system's PCIe hierarchy and platform support.
1. What Is NVIDIA NVLink?
NVLink is NVIDIA's high-speed interconnect architecture for supported processors and accelerators. It enables data movement between connected devices and supports the communication patterns required by tightly coupled GPU workloads.
In multi-GPU AI systems, accelerators may need to exchange intermediate tensors, synchronize gradients or access data associated with other devices. The efficiency of these operations can significantly influence total workload execution time.
NVLink provides an alternative to relying exclusively on conventional PCIe pathways for supported GPU communication.
2. NVLink Versus PCIe
PCIe is a general-purpose interconnect used for a wide range of devices, including GPUs, network adapters and storage controllers. Its capabilities depend on PCIe generation, lane width, switching topology and platform configuration.
NVLink is designed for high-bandwidth communication between supported processors. Depending on the NVIDIA platform, it may provide substantially greater aggregate GPU interconnect bandwidth than the available PCIe connection.
However, bandwidth figures must be compared carefully. Aggregate bidirectional NVLink bandwidth is not directly equivalent to one-direction PCIe bandwidth, and practical transfer performance depends on workload and topology.
Engineering distinction: PCIe is a general-purpose I/O interconnect. NVLink is a specialized high-bandwidth processor interconnect. NVSwitch uses NVLink connectivity to create larger switched GPU communication domains.
3. GPU Peer-to-Peer Communication
Peer-to-peer GPU communication allows data to move between supported GPU memory spaces without requiring the application to stage every transfer through host system memory.
Whether a particular transfer can use direct peer access depends on GPU capabilities, platform topology, driver configuration and software support.
In distributed AI applications, efficient peer communication is important for exchanging tensor partitions, intermediate activations and collective communication data.
4. NVLink Bandwidth and Latency
Interconnect bandwidth determines how quickly data can be transferred under suitable conditions. Latency measures the delay associated with initiating and completing communication operations.
Both matter for AI workloads. Large tensor transfers may be bandwidth-sensitive, while repeated small synchronization operations can be sensitive to latency and software overhead.
Real application performance also depends on communication libraries, memory access behavior, transfer sizes, contention and the ability to overlap communication with computation.
| Architecture | Connectivity model | Primary purpose | Design consideration |
|---|---|---|---|
| PCIe | General-purpose switched I/O hierarchy | Device connectivity and data movement | Lane bandwidth, topology and peer support |
| Direct NVLink | Supported point-to-point processor links | High-bandwidth peer communication | Available direct connections and link capacity |
| NVSwitch | Switched NVLink fabric | Multi-GPU scale-up communication | Fabric topology, collective traffic and system integration |
| Ethernet / InfiniBand | Network fabric between systems | Scale-out communication | RDMA, congestion and network capacity |
5. Scale-Up GPU Communication Domains
Scale-up architecture connects multiple accelerators within a tightly coupled compute environment. The objective is to provide efficient communication for workloads that partition computation or model state across participating GPUs.
NVLink and NVSwitch support these architectures by enabling high-bandwidth communication inside the supported scale-up domain.
Beyond that domain, larger AI clusters commonly use Ethernet or InfiniBand networks to connect systems. The boundary between scale-up and scale-out affects model placement, collective communication and distributed training efficiency.
6. Why NVLink Matters for AI Workloads
Tensor parallelism, pipeline parallelism and distributed model execution can generate substantial GPU-to-GPU communication. The interconnect must transfer data quickly enough to prevent accelerators from spending excessive time waiting for results from other devices.
NVLink is particularly important when workloads require frequent exchange of intermediate tensors or collective communication within a tightly coupled GPU group.
However, the best communication architecture depends on the model, framework, partitioning strategy and physical system topology.
Design takeaway: The objective is not simply to maximize nominal link bandwidth. Engineers must match the GPU communication topology to the application's transfer patterns, synchronization requirements and memory architecture.
7. Engineering Takeaway
NVLink forms an important part of NVIDIA's multi-GPU architecture by providing high-bandwidth connectivity between supported devices. NVSwitch expands that capability through switched fabrics, enabling larger scale-up GPU systems.
Understanding PCIe, direct NVLink and NVSwitch communication paths provides the foundation for studying NVLink generations, NVSwitch internals and rack-scale GPU architectures.
The architecture diagram is simplified for educational use. Actual device paths, peer-memory capabilities and interconnect performance vary by GPU generation, platform and system configuration.
NVLink GPU Interconnect Architecture Explorer
Explore how GPUs communicate over PCIe, direct NVLink connections and switched NVLink fabrics. Inspect peer reachability, interconnect boundaries, transfer paths and bandwidth limitations.
Interactive GPU communication topology
Select a GPU, PCIe root complex or NVSwitch component to inspect its architectural role.
Select a topology and trace a GPU-to-GPU transfer.
Direct NVLink and switched NVLink fabrics provide high-bandwidth GPU communication paths. PCIe remains important for host connectivity and systems without dedicated GPU peer fabrics.
How NVIDIA NVLink Evolved from Pascal to Blackwell
NVIDIA NVLink has evolved across multiple GPU generations to provide increasing interconnect bandwidth and support increasingly demanding multi-GPU workloads.
Each generation introduces improvements in communication capability, while the overall system architecture determines how the available bandwidth can be used. Comparing NVLink generations requires distinguishing per-link specifications, aggregate GPU bandwidth and complete switched-fabric capacity.
NVLink 5 — Blackwell
Fifth-generation NVLink significantly increases aggregate GPU interconnect bandwidth and supports Blackwell's rack-scale switched GPU architectures.
1. Why NVLink Generations Matter
As GPU compute capability increases, the communication infrastructure must support the movement of larger tensors, gradients and model state between accelerators.
Interconnect bandwidth influences the efficiency of workloads that partition computation across multiple GPUs. If communication cannot keep pace with computation, accelerators may spend increasing time waiting for remote data.
2. The Evolution of NVLink
Early NVLink generations focused on providing high-bandwidth connections between supported processors. Later generations increased bandwidth and expanded support for larger multi-GPU communication architectures.
With NVSwitch, NVLink became the foundation for switched GPU fabrics rather than only direct processor connections. Blackwell extends this approach into rack-scale systems.
3. NVLink Generation Comparison
| Generation | Representative GPU | Peak aggregate bidirectional GPU bandwidth | Architecture context |
|---|---|---|---|
| NVLink 1 | Tesla P100 SXM2 | 160 GB/s | Early high-bandwidth GPU interconnect |
| NVLink 2 | Tesla V100 SXM2 | 300 GB/s | Expanded multi-GPU and CPU–GPU connectivity |
| NVLink 3 | NVIDIA A100 SXM | 600 GB/s | Higher-bandwidth Ampere GPU communication |
| NVLink 4 | NVIDIA H100 SXM | 900 GB/s | Hopper GPU communication and NVSwitch systems |
| NVLink 5 | NVIDIA B200 | 1,800 GB/s | Blackwell scale-up and NVL72 architectures |
Bandwidth interpretation: These values describe representative GPUs' theoretical aggregate bidirectional NVLink bandwidth. They are not single-link bandwidth figures, one-way transfer rates or guaranteed application throughput. Product variants can have different interconnect specifications.
4. What Changed with Fifth-Generation NVLink?
Blackwell's fifth-generation NVLink increases aggregate GPU interconnect bandwidth and supports large switched communication domains. The architecture is particularly relevant to large AI models requiring frequent communication between GPUs.
NVLink 5 is associated with the GB200 NVL72 architecture, which integrates 72 Blackwell GPUs into a rack-scale NVLink domain.
This changes the engineering perspective from designing communication inside a relatively small GPU server to managing a larger integrated accelerator fabric.
5. Link Bandwidth Versus Fabric Bandwidth
A single NVLink connection, a GPU's total NVLink bandwidth and the aggregate capability of a complete NVSwitch fabric describe different aspects of the architecture.
GPU aggregate bandwidth represents the combined capability of the available links on that GPU. Fabric capacity depends on switching resources, topology and the number of participating devices.
Application performance is further influenced by collective communication algorithms, memory bandwidth, transfer size, contention and synchronization overhead.
6. Engineering Implications for AI Workloads
Higher interconnect bandwidth can reduce communication overhead for workloads that exchange large quantities of data between GPUs. Tensor parallelism and collective operations are particularly sensitive to interconnect capability.
However, adding bandwidth does not automatically improve every workload. Applications limited by computation, memory capacity or software scheduling may see less benefit from a faster GPU interconnect.
Engineers should therefore evaluate the communication-to-computation ratio and the physical GPU topology before making performance assumptions.
Architecture takeaway: NVLink's evolution is not only a bandwidth story. It also reflects the transition toward increasingly integrated, switched GPU systems designed for large-scale AI execution.
7. Final Engineering Takeaway
From Pascal through Blackwell, NVLink has evolved into a major component of NVIDIA's multi-GPU architecture. Its increasing bandwidth supports the communication demands of modern AI training and inference systems.
Understanding the differences between NVLink generations prepares engineers to examine NVSwitch switching architecture, rack-scale GPU connectivity and collective communication performance.
Figures are representative published peak specifications for the listed GPU products. Actual performance and supported connectivity depend on platform configuration and software.
NVLink Generation & Bandwidth Comparison Lab
Compare five NVLink generations and investigate how aggregate GPU interconnect bandwidth evolved to support larger AI workloads and scale-up domains.
NVLink aggregate bandwidth evolution
Fifth-generation NVLink supports up to 1.8 TB/s aggregate bidirectional bandwidth per Blackwell GPU in supported configurations.
Choose a workload and run the bandwidth analysis.
Increasing NVLink bandwidth helps reduce communication bottlenecks for distributed workloads, but realised performance depends on topology, link allocation, memory access and communication patterns.
Inside the NVIDIA NVSwitch Fabric
NVSwitch extends NVIDIA NVLink connectivity by providing switching infrastructure for communication between multiple GPUs. Rather than requiring every accelerator to have a direct physical connection to every other accelerator, a switched fabric enables communication across a larger group of devices.
The NVSwitch architecture must support demanding traffic patterns generated by distributed AI applications, including simultaneous GPU transfers, tensor exchanges and collective communication operations.
GPU Peer Transfer
A source GPU communicates with a destination GPU through the switched NVLink domain. The diagram illustrates the logical forwarding relationship rather than a physical chip-level path.
1. What Is NVSwitch?
NVSwitch is NVIDIA's switching technology for NVLink-connected GPU systems. It extends the communication capabilities of NVLink by allowing larger groups of GPUs to exchange data through a switched interconnect fabric.
Instead of relying only on direct GPU-to-GPU connections, a system can use NVSwitch devices and NVLink connections to establish communication across a larger scale-up domain.
The exact switch topology, connectivity and bandwidth characteristics depend on the NVIDIA platform and generation.
2. The NVSwitch Data Plane
The data plane is responsible for transporting communication traffic across the available fabric connectivity. GPU-generated transfers enter the interconnect through supported NVLink interfaces and traverse the fabric toward their destinations.
Switching hardware must support concurrent traffic while maintaining the performance and reliability expected by distributed AI workloads.
The visual architecture above simplifies the fabric into one logical switching element. Real systems may contain multiple NVSwitch ASICs, links and switching resources.
Architecture distinction: NVLink provides the processor interconnect, while NVSwitch supplies switched connectivity. NVSwitch is not simply a conventional Ethernet switch with GPU-facing ports; it belongs to NVIDIA's specialized NVLink fabric architecture.
3. GPU-Facing Ports and Fabric Connectivity
NVSwitch devices connect to participating GPUs through NVLink interfaces. The distribution of links and switching resources determines the supported connectivity and aggregate communication capability.
At the system level, engineers must consider GPU connectivity, switch resources, physical cabling or backplane integration, supported topology and platform-specific redundancy characteristics.
These details become especially important in rack-scale systems where many accelerators participate in a shared scale-up communication domain.
4. Concurrent GPU Communication
AI workloads rarely consist of a single isolated GPU transfer. Multiple accelerators may exchange tensors or participate in communication operations at the same time.
Concurrent transfers place demands on fabric bandwidth, internal switching resources and endpoint communication behavior.
Performance depends not only on the nominal interconnect capacity but also on traffic patterns, communication scheduling, transfer sizes and contention.
5. Collective Communication and NVSwitch
Collective operations coordinate data exchange among multiple GPUs. Common examples include All-Reduce, All-Gather, Reduce-Scatter and broadcast.
These operations are central to distributed training and model parallelism. Their efficiency depends on the communication algorithm, available topology and the characteristics of the underlying interconnect.
Supported NVSwitch generations also introduce capabilities that can accelerate certain collective communication operations. The exact behavior depends on the hardware generation and software stack.
6. NVSwitch Fabric Management
The fabric management plane handles responsibilities associated with initializing and managing supported NVLink/NVSwitch configurations. NVIDIA provides platform-specific software components and management interfaces for these tasks.
Engineers must distinguish fabric management from application communication. A workload may use NCCL or another supported communication interface, while underlying system software manages the connectivity required by the hardware platform.
Operational troubleshooting should examine both application communication behavior and the underlying fabric state.
7. NVSwitch Fault Domains and Reliability
Fabric problems can arise from unhealthy links, hardware faults, initialization failures, platform configuration issues or communication software errors.
A degraded communication path can affect more than one workload, especially when multiple GPUs share the affected switching infrastructure.
Effective troubleshooting requires correlating supported fabric health telemetry with GPU communication traces and workload performance.
| Fabric component | Engineering responsibility | Potential symptom | Investigation focus |
|---|---|---|---|
| GPU NVLink interface | Endpoint connectivity | Peer communication failures | Link state and supported error counters |
| NVSwitch device | Switched fabric connectivity | Communication degradation | Fabric health and device status |
| Fabric management | Initialization and management | Unavailable or incomplete connectivity | Management services and initialization events |
| Collective library | Multi-GPU communication orchestration | Slow or stalled collectives | Communication traces and topology awareness |
Troubleshooting principle: A slow collective operation does not automatically indicate a defective NVSwitch. First distinguish application synchronization, communication library behavior, endpoint performance and actual fabric health.
8. Engineering Takeaway
NVSwitch transforms NVLink from a direct processor interconnect into the foundation of larger switched GPU communication domains.
Understanding the separation between GPU endpoints, NVLink transport, switching resources and fabric management is essential when designing and troubleshooting high-performance AI infrastructure.
The interactive architecture is a simplified logical model. Actual NVSwitch ASIC internals, port assignments, routing behavior and redundancy mechanisms vary by platform and generation.
NVSwitch Internal Fabric Path Explorer
Investigate GPU-to-GPU communication through a switched NVLink fabric. Trace ingress, switching and egress stages, observe contention and inject logical switch-plane faults.
Logical switched GPU fabric
Select a GPU or logical NVSwitch plane to inspect its role in fabric connectivity.
Choose a source and destination GPU, then trace the switched communication path.
A switched fabric enables communication between GPUs without requiring every GPU to have a dedicated direct physical link to every peer.
How GPUs Exchange Data Across NVLink and NVSwitch
GPU-to-GPU communication is a fundamental requirement for distributed AI execution. When a model is partitioned across multiple accelerators, GPUs must exchange intermediate tensors, gradients and other data to coordinate computation.
NVLink and NVSwitch provide high-bandwidth communication pathways for supported multi-GPU systems. However, efficient data movement also depends on memory access capabilities, software configuration, transfer size and communication scheduling.
Direct NVLink Transfer
Two supported GPUs exchange data through a direct NVLink connection. A peer transfer can move data between GPU memory spaces without staging the complete transfer through host memory.
1. GPU Memory Spaces and Data Movement
Each discrete GPU typically has its own local memory subsystem. In Blackwell systems, HBM provides high-bandwidth storage for model parameters, activations, intermediate tensors and other execution data.
When a workload spans multiple GPUs, data may need to move between these memory spaces. The interconnect architecture determines the available communication pathways, while software determines how transfers are initiated and synchronized.
2. Peer-to-Peer GPU Transfers
Peer-to-peer communication allows supported GPUs to exchange data without requiring every transfer to be staged through host system memory.
Depending on the platform, software may use peer-memory access capabilities, GPU communication libraries or dedicated data movement engines to perform transfers.
Peer access must be supported by the hardware topology and enabled through the appropriate software interfaces.
Important distinction: Peer-to-peer communication and remote memory access are related but not identical. A bulk memory copy transfers data between memory locations, while supported remote memory access allows a GPU to address memory associated with another device.
3. Direct NVLink Communication
Direct NVLink connectivity provides a high-bandwidth communication path between supported processors without requiring an intervening NVSwitch device.
Direct connections are useful in systems where the physical GPU topology provides the required communication relationships.
However, the number of available direct links and supported peer connections depend on the GPU generation and system design.
4. NVSwitch-Based GPU Transfers
NVSwitch extends GPU communication through a switched NVLink fabric. A transfer between GPUs can traverse the fabric rather than relying on a dedicated direct connection between those endpoints.
This enables larger scale-up GPU domains and supports communication-intensive AI workloads that require frequent data exchange among many accelerators.
The actual route and performance depend on the system's switching topology, fabric resources and software configuration.
5. Host-Staged Communication
When a direct peer path is unavailable or unsuitable, software may move data through host memory. In a simplified staged transfer, data moves from the source GPU into system memory and then from system memory to the destination GPU.
This can introduce additional memory traffic, transfer overhead and synchronization requirements compared with an efficient supported peer path.
Host staging should not be confused with all PCIe communication: supported PCIe topologies can also provide GPU peer-to-peer transfers without staging the complete data through host memory.
6. Remote GPU Memory Access
Some supported GPU architectures and software configurations allow a GPU to access memory associated with another GPU through peer-memory mappings.
This can enable communication patterns beyond explicit bulk copies, but remote access behavior depends on hardware support, memory mappings, access granularity and synchronization semantics.
Remote memory should not automatically be treated as equivalent to local HBM. Access costs and effective bandwidth can differ substantially.
7. Communication Overhead and Performance
The performance of a GPU transfer depends on more than the interconnect's peak bandwidth. Transfer size, startup overhead, memory access patterns, contention and synchronization can all influence effective throughput.
Large transfers may approach the bandwidth capabilities of the available communication path under suitable conditions, while small transfers can be dominated by latency and software overhead.
Applications can sometimes improve execution efficiency by overlapping communication with useful GPU computation.
| Communication method | Data path | Typical consideration | Potential limitation |
|---|---|---|---|
| Direct NVLink | GPU → direct NVLink → GPU | High-bandwidth peer transfers | Physical connectivity and support |
| NVSwitch | GPU → NVLink fabric → GPU | Larger scale-up domains | Fabric resources and traffic contention |
| PCIe peer transfer | GPU → PCIe hierarchy → GPU | Supported peer communication | PCIe topology and bandwidth |
| Host staging | GPU → host memory → GPU | Fallback or application-specific transfers | Additional copies and overhead |
| Remote peer access | GPU accesses mapped peer memory | Fine-grained remote access patterns | Remote access cost and synchronization |
Performance engineering principle: The fastest interconnect is not always the most important optimization. Data placement, transfer frequency, message size and communication overlap can be equally important to application performance.
8. Engineering Takeaway
GPU-to-GPU communication connects the compute and memory resources of a multi-accelerator system. NVLink and NVSwitch provide high-bandwidth paths for supported peer transfers, while PCIe and host memory remain important parts of the broader system architecture.
Understanding these data movement paths prepares engineers to examine the GB200 NVL72 rack-scale architecture, where many GPUs participate in a shared switched NVLink domain.
The transfer visualization is conceptual. Actual peer access, memory semantics and interconnect performance vary by platform, GPU generation and software configuration.
GPU Peer-to-Peer Transfer Simulator
Investigate GPU memory-to-memory transfers through direct NVLink, switched NVLink fabrics and host-staged PCIe paths. Compare transfer overhead, bandwidth and synchronization.
GPU memory movement architecture
Each GPU has its own HBM address space and memory controllers. Peer transfers move data between memory domains through a supported interconnect path.
Configure a GPU transfer and run the analysis.
Direct GPU peer transfers can avoid host-memory staging when the hardware, driver and software stack support peer access.
Inside the NVIDIA GB200 NVL72 Rack
The NVIDIA GB200 NVL72 represents a shift from conventional multi-GPU server architecture toward rack-scale accelerated computing. It combines 72 Blackwell GPUs and 36 Grace CPUs within an integrated system connected by a fifth-generation NVLink switching fabric.
Rather than treating each GPU server as an independent compute island, the NVL72 architecture creates a larger tightly coupled scale-up domain. This requires coordinated design across GPU compute, NVSwitch connectivity, power distribution, liquid cooling, management and scale-out networking.
Complete NVL72 Architecture
The GB200 NVL72 combines Grace CPU and Blackwell GPU compute resources with NVLink switching, liquid cooling, power infrastructure and external networking.
1. From GPU Servers to Rack-Scale Computing
Traditional accelerator clusters commonly organize GPUs into individual servers, with each server connected to a larger network fabric. GPU communication inside a server may use NVLink or NVSwitch, while communication between servers uses Ethernet or InfiniBand.
The GB200 NVL72 extends the tightly coupled GPU domain to rack scale. Its integrated NVLink fabric connects a much larger group of accelerators within a single system architecture.
2. Grace CPUs and Blackwell GPUs
The NVL72 integrates 36 NVIDIA Grace CPUs and 72 Blackwell GPUs. These resources are organized into Grace Blackwell superchip compute assemblies.
The Grace CPUs support host-side processing, orchestration and memory functions, while the Blackwell GPUs provide accelerated computation for AI training and inference.
NVLink-C2C connectivity within Grace Blackwell superchips is distinct from the NVLink 5 switched GPU fabric connecting the larger GPU domain.
Architecture distinction: NVLink-C2C connects the Grace CPU and Blackwell GPU components within a superchip, while the rack-scale NVLink/NVSwitch fabric connects GPUs across the broader NVL72 system. These should not be treated as the same interconnect.
3. Compute Tray Architecture
The GB200 NVL72 uses compute trays containing Grace Blackwell superchip assemblies. Each compute tray contributes CPU and GPU resources to the integrated rack architecture.
The compute layer provides the processing and memory resources needed for AI workloads. The switching fabric allows participating GPUs to exchange data within the supported scale-up domain.
Rack integration must coordinate compute placement, thermal management, power delivery and communication connectivity.
4. NVSwitch Tray Architecture
The NVL72 incorporates 18 NVLink switch trays to provide the switched GPU communication infrastructure. These switching resources form the foundation of the rack-scale NVLink domain.
The NVSwitch layer supports communication between GPUs located across the integrated rack system. Its role is separate from external Ethernet or InfiniBand networking used for communication beyond the rack.
Fabric initialization, connectivity health and switch status become important operational concerns at this scale.
5. Liquid Cooling Architecture
Rack-scale accelerator systems generate substantial heat. The GB200 NVL72 uses liquid cooling to support the thermal requirements of its densely integrated compute and switching infrastructure.
Cooling design must account for heat removal, coolant distribution, facility integration and operational reliability.
Thermal conditions can affect accelerator operating behavior and system performance. Cooling telemetry therefore belongs alongside GPU and fabric telemetry in production monitoring.
6. Power and System Integration
Power delivery is a critical part of rack-scale AI architecture. The compute, switching and supporting infrastructure must receive stable power within the operating limits of the platform.
Power architecture must be coordinated with facility capacity, cooling infrastructure and operational procedures.
Engineers should distinguish the electrical requirements of the complete rack from the power characteristics of an individual GPU or compute tray.
7. Scale-Up and Scale-Out Connectivity
Within the NVL72, NVLink and NVSwitch provide the tightly coupled GPU communication domain. Beyond that domain, Ethernet or InfiniBand networks connect systems to the wider AI cluster.
This creates a hierarchy of communication resources. Workloads can benefit from placing communication-intensive GPU groups within an efficient scale-up domain while using the scale-out fabric for communication between larger systems.
Model partitioning, collective communication strategy and network topology should be designed with these boundaries in mind.
| NVL72 component | Role | Engineering consideration | Operational focus |
|---|---|---|---|
| Blackwell GPUs | AI computation and HBM memory | GPU utilization and data placement | GPU health and performance |
| Grace CPUs | Host processing and system functions | CPU–GPU integration | Host and memory telemetry |
| NVSwitch trays | Rack-scale GPU switching | Fabric connectivity and bandwidth | Link and switch health |
| Liquid cooling | Thermal management | Heat removal and facility integration | Cooling system health |
| Power infrastructure | Rack electrical supply | Capacity and reliability | Power monitoring |
| Scale-out network | Connectivity beyond the NVLink domain | RDMA and network topology | Network congestion and errors |
Design takeaway: GB200 NVL72 should be evaluated as a complete rack-scale computing platform. GPU performance, NVSwitch connectivity, liquid cooling, power and scale-out networking are interdependent architectural considerations.
8. Engineering Takeaway
The GB200 NVL72 demonstrates how NVLink and NVSwitch enable tightly coupled GPU communication beyond the boundaries of a conventional multi-GPU server.
Understanding its compute, switching and infrastructure layers prepares engineers to investigate NCCL collective communication, scale-up versus scale-out network design and production GPU fabric troubleshooting.
The rack visualization is a simplified educational model. Physical tray placement, cabling, coolant distribution and power implementation depend on the NVIDIA reference architecture and system vendor.
GB200 NVL72 Rack Architecture Explorer
Explore the physical and logical architecture of a rack-scale Grace Blackwell system. Inspect compute trays, NVSwitch trays, GPU density, scale-up connectivity and liquid-cooling infrastructure.
Interactive rack-scale architecture
Select a compute tray or NVSwitch tray to inspect its architectural role.
All logical rack components are available.
GB200 NVL72 integrates 72 Blackwell GPUs and 36 Grace CPUs into a rack-scale system interconnected through an NVLink switch fabric.
How NCCL Coordinates Communication Across GPUs
NVIDIA Collective Communications Library (NCCL) provides communication primitives used by distributed GPU applications. It supports collective operations that coordinate data exchange across multiple accelerators, including All-Reduce, All-Gather, Reduce-Scatter and broadcast.
These operations are essential to distributed AI training and model parallelism. Their performance depends on the communication algorithm, GPU topology, interconnect bandwidth, message size and synchronization behavior.
All-Reduce
Each GPU contributes a value. The values are reduced using an operation such as summation, and the final result becomes available to all participating GPUs.
1. What Is NCCL?
NCCL is NVIDIA's communication library for multi-GPU and multi-node collective operations. It provides optimized communication primitives that distributed applications and machine learning frameworks can use to exchange data between accelerators.
NCCL can use supported GPU interconnects and network transports, including NVLink, NVSwitch and scale-out networking technologies.
The library selects communication approaches based on available hardware, topology, software configuration and operation characteristics.
2. Why Collectives Matter for AI
Distributed training frequently requires GPUs to synchronize gradients or exchange model data. Tensor-parallel workloads may need to gather intermediate results or reduce partial computations across participating accelerators.
Collective operations provide standardized ways to coordinate these exchanges without requiring every application to implement its own low-level communication algorithm.
However, collective operations can become a major performance bottleneck when communication overhead is large relative to useful computation.
Engineering distinction: NCCL is a communication library, not a physical network protocol. NVLink and NVSwitch provide hardware communication capabilities that NCCL can use, while the actual collective algorithm is selected by the software stack.
3. Four Essential Collective Operations
4. All-Reduce in Distributed Training
All-Reduce is commonly used to combine gradients across data-parallel training workers. Each participating GPU computes local gradient contributions, and the collective operation produces a reduced result that is available to every participant.
The reduction operation may be summation, maximum or another supported operation, depending on the application.
Efficient All-Reduce execution can be critical because gradient synchronization may occur repeatedly during training.
5. All-Gather and Tensor Parallelism
All-Gather collects data from each participating GPU and makes the combined result available across the group.
This operation is useful when a model or tensor is partitioned across multiple accelerators and the next computation stage requires the combined data.
All-Gather performance depends on data volume, communication topology and the algorithm used to exchange the distributed segments.
6. Reduce-Scatter
Reduce-Scatter combines values across participating GPUs and distributes separate portions of the reduced result to different participants.
It can be used as part of distributed optimization strategies and communication-efficient collective algorithms.
In many implementations, Reduce-Scatter and All-Gather can form useful building blocks for larger communication operations.
7. Broadcast
Broadcast distributes data from one designated root GPU to other participants in the communication group.
It is useful for distributing parameters, configuration data or other information that must be made available consistently across multiple GPUs.
The implementation may use different communication structures depending on message size, topology and available transports.
8. NCCL Algorithms and Topology Awareness
NCCL can use different communication algorithms, including ring- and tree-based approaches, depending on the operation and environment.
Algorithm selection involves trade-offs between communication startup overhead, total data transferred, available bandwidth and topology characteristics.
NVSwitch-based systems can provide high-bandwidth communication across tightly coupled GPU groups, while multi-node communication may traverse Ethernet or InfiniBand networks.
| Operation | Input pattern | Output pattern | Example AI use case |
|---|---|---|---|
| All-Reduce | Values from all GPUs | Reduced result on all GPUs | Gradient synchronization |
| All-Gather | Distributed data segments | Combined data on all GPUs | Tensor reconstruction |
| Reduce-Scatter | Distributed arrays | Reduced partitions | Distributed optimization |
| Broadcast | Root GPU data | Root data on all GPUs | Parameter distribution |
Performance principle: Collective communication performance is influenced by algorithm selection, message size, GPU topology, available bandwidth, network contention and the ability to overlap communication with computation.
9. Engineering Takeaway
NCCL provides the collective communication operations that allow distributed GPU applications to coordinate computation and data movement efficiently.
Understanding All-Reduce, All-Gather, Reduce-Scatter and broadcast is essential when analyzing multi-GPU performance, NVLink fabric utilization and distributed AI training behavior.
The interactive visualization demonstrates collective operation semantics. Actual NCCL algorithms, communication stages and transport selection vary by hardware topology, operation size and software configuration.
NCCL Collective Communication Simulator
Execute collective communication operations across a multi-GPU topology. Investigate how distributed data is reduced, gathered, scattered and broadcast through different communication patterns.
Animated collective communication topology
Select a GPU node to inspect its local contribution and current collective data state.
Choose an operation and execute the communication sequence.
Collective communication algorithms coordinate data movement across GPUs. NCCL selects algorithms and transport mechanisms according to the operation, topology and environment.
Understanding the AI Fabric Communication Hierarchy
Large AI infrastructure uses multiple communication technologies to connect accelerators. NVLink and NVSwitch provide high-bandwidth scale-up communication within supported GPU domains, while Ethernet and InfiniBand commonly provide scale-out connectivity between systems and racks.
These technologies serve complementary purposes. Efficient AI architecture depends on understanding where communication occurs, which fabric carries the traffic and how model placement influences communication overhead.
Intra-Domain GPU Communication
GPUs exchange data within one scale-up domain through the available NVLink/NVSwitch connectivity. The external network is not part of this conceptual communication path.
1. What Is Scale-Up Networking?
Scale-up architecture increases the compute resources available within a tightly coupled system or accelerator domain. In NVIDIA GPU systems, NVLink and NVSwitch provide high-bandwidth communication between supported GPUs.
Scale-up connectivity is especially important when applications require frequent exchange of intermediate tensors, gradients or model state across participating accelerators.
2. What Is Scale-Out Networking?
Scale-out architecture connects multiple compute systems to create a larger distributed cluster. Ethernet and InfiniBand are widely used to provide network connectivity between GPU systems.
Scale-out networks support communication beyond the local GPU interconnect domain and must handle distributed training traffic, collective operations and other cluster communication.
Architecture distinction: NVLink/NVSwitch and Ethernet/InfiniBand should not be treated as competing replacements. They operate at different levels of the GPU communication hierarchy and are frequently used together in large AI systems.
3. The Boundary Between Scale-Up and Scale-Out
The boundary between scale-up and scale-out communication is important because it determines which interconnect carries a given data exchange.
Communication within a supported NVLink domain can use the local GPU interconnect fabric. Communication between separate systems may require network adapters and an Ethernet or InfiniBand fabric.
The effective cost of communication depends on the complete path, including GPU memory, software transport, network interfaces and switching infrastructure.
4. RDMA in Scale-Out AI Networks
Remote Direct Memory Access (RDMA) allows supported systems to transfer data between memory regions with reduced involvement from host CPUs compared with conventional socket-based data movement.
InfiniBand provides native RDMA capabilities, while Ethernet networks can support RDMA through technologies such as RoCEv2.
RDMA performance depends on adapter capabilities, software support, network design and congestion behavior.
5. Ethernet Versus InfiniBand
Ethernet and InfiniBand can both provide high-performance networking for distributed AI infrastructure, but they differ in ecosystem, management, transport capabilities and network behavior.
Ethernet offers broad interoperability and extensive operational tooling. InfiniBand provides an integrated high-performance interconnect ecosystem with native RDMA support.
The appropriate choice depends on workload requirements, platform support, operational expertise and the architecture of the broader data centre.
6. Distributed Collective Communication
Collective operations may involve GPUs within one scale-up domain or participants distributed across multiple systems.
When collectives span separate systems, communication can involve both local GPU interconnects and the scale-out network.
Effective communication scheduling and topology-aware placement are important for minimizing unnecessary network traffic and synchronization overhead.
7. Congestion and Network Performance
Scale-out network congestion can increase communication delay and reduce effective throughput. This may affect collective operations that require multiple participants to synchronize.
Engineers should examine link utilization, congestion indicators, packet loss or transport-specific recovery behavior, and communication traces.
Slow GPU collectives should not automatically be attributed to the NVLink fabric when the workload spans a larger network.
| Technology | Primary domain | Communication role | Engineering focus |
|---|---|---|---|
| NVLink | Supported processor/GPU connections | High-bandwidth scale-up transport | Peer bandwidth and connectivity |
| NVSwitch | Switched GPU scale-up domain | Multi-GPU fabric connectivity | Fabric health and topology |
| Ethernet | Inter-system networking | Scale-out connectivity | Routing, congestion and RoCEv2 where used |
| InfiniBand | Inter-system networking | Scale-out connectivity and RDMA | Fabric management and transport performance |
Design takeaway: Communication-intensive GPU groups should be placed with awareness of the available scale-up and scale-out boundaries. Poor placement can increase network traffic and reduce distributed workload efficiency.
8. Engineering Takeaway
Modern AI infrastructure uses a hierarchy of communication fabrics. NVLink and NVSwitch provide tightly coupled GPU connectivity, while Ethernet and InfiniBand connect larger distributed systems.
Understanding this hierarchy is essential for designing GPU clusters, diagnosing collective communication bottlenecks and optimizing distributed AI workloads.
The diagram is a conceptual representation. Actual GPU, network adapter and switching topologies depend on the platform and deployment architecture.
Scale-Up vs Scale-Out Traffic Path Lab
Trace GPU communication inside an NVLink domain and across external Ethernet or InfiniBand fabrics. Examine RDMA transport, communication boundaries and the impact of scale-out bandwidth limitations.
Hybrid GPU communication architecture
NVLink and NVSwitch provide high-bandwidth communication within a supported GPU scale-up domain. Communication between independent domains generally requires network adapters and a scale-out fabric.
Select a communication path and run the analysis.
Scale-up and scale-out interconnects serve different purposes. GPU placement determines whether traffic stays inside the NVLink domain or traverses a network fabric.
Investigating GPU Fabric Performance
High theoretical interconnect bandwidth does not guarantee high application throughput. Real GPU communication performance depends on message size, effective link capacity, contention, synchronization overhead and the placement of communicating GPUs.
Performance engineering requires separating the time spent transferring data from the time spent initiating, coordinating and waiting for communication. These effects become especially important in distributed AI workloads with frequent collective operations.
Transfer Investigation
Adjust the workload parameters to examine how transfer size, assumed available bandwidth, contention and startup overhead affect a simplified communication model.
1. Theoretical Versus Effective Bandwidth
NVLink specifications describe the theoretical communication capability of supported GPU interfaces. However, real application throughput is affected by protocol overhead, communication patterns, memory behavior and resource contention.
Engineers should distinguish aggregate bidirectional bandwidth figures from the bandwidth available to a particular unidirectional transfer or application operation.
2. Transfer Size and Startup Overhead
Small GPU transfers can be dominated by launch, synchronization and communication startup costs. Large transfers may be more sensitive to available data-path bandwidth.
This distinction helps explain why an interconnect with very high theoretical bandwidth may still deliver limited effective throughput for workloads consisting of many small messages.
Model definition: The explorer estimates transfer time as startup overhead plus payload size divided by assumed effective bandwidth. Contention is represented as a simple reduction in available bandwidth. This is an educational model, not a physical model of NVLink arbitration or an NCCL performance predictor.
3. Fabric Contention
Multiple GPUs may generate communication traffic simultaneously. Depending on topology and resource availability, competing transfers can contend for shared communication resources.
Contention can reduce effective throughput and increase communication completion time. Its impact depends on the actual traffic distribution and hardware implementation.
Investigations should compare workload traffic patterns with supported fabric utilization and error telemetry.
4. GPU Placement and Communication Locality
GPU placement determines which communication resources a workload uses. Communication within a tightly coupled NVLink domain may have different characteristics from communication crossing a scale-out network.
Model parallelism and collective communication strategies should account for these boundaries when assigning work to GPUs.
Topology-aware scheduling can reduce unnecessary communication across slower or more heavily shared paths.
5. Compute–Communication Overlap
Distributed AI applications can sometimes overlap data communication with GPU computation. This can reduce the portion of communication time exposed on the critical execution path.
Overlap depends on workload dependencies, stream scheduling, available compute resources and communication implementation.
Engineers should distinguish total communication activity from communication time that actually delays useful computation.
6. NCCL Performance Analysis
NCCL collective operations introduce additional considerations beyond a single point-to-point transfer. Performance depends on collective type, algorithm selection, message size and participating GPU topology.
A slow collective may reflect network congestion, synchronization imbalance, communication algorithm behavior or a slow participant.
Collective performance should be investigated using appropriate workload profiling and communication traces.
7. Telemetry and Benchmarking
Useful performance investigations combine application-level profiling with GPU and fabric health information. NVIDIA profiling and diagnostic tools can provide insight into kernel execution, communication behavior and supported hardware telemetry.
Microbenchmarks can help establish baseline communication performance, but they should be interpreted alongside realistic workload measurements.
Benchmark conditions must be controlled to avoid confusing unrelated CPU, GPU, memory or network effects with fabric limitations.
| Performance symptom | Possible explanation | Investigation method | Optimization direction |
|---|---|---|---|
| Slow small transfers | Startup or synchronization overhead | Profile message sizes and launch timing | Batch transfers where appropriate |
| Low large-transfer throughput | Bandwidth limitation or contention | Compare measured throughput with topology and load | Review placement and concurrent traffic |
| Slow collectives | Algorithm, topology or participant imbalance | Inspect NCCL and application traces | Review collective strategy |
| Variable transfer times | Contention or workload interference | Correlate utilization with communication timing | Reduce competing demand |
| Low GPU utilization | Communication exposed on critical path | Profile compute and communication overlap | Improve scheduling and overlap |
Engineering principle: A performance bottleneck should be identified through evidence rather than inferred from nominal bandwidth specifications. Measure the application, establish a baseline and isolate the limiting communication stage.
8. Engineering Takeaway
NVLink performance engineering requires understanding the interaction between transfer size, bandwidth, startup overhead, contention and workload topology.
Effective optimization focuses on reducing communication overhead exposed to the application, improving data locality and ensuring that GPU communication resources are used efficiently.
All calculated values in the interactive explorer are synthetic and illustrative. Actual NVIDIA hardware behavior depends on GPU generation, interconnect topology, transfer direction, software and workload characteristics.
GPU Fabric Performance Investigation Lab
Investigate communication bottlenecks in a switched GPU fabric. Model utilization, workload placement, contention and compute/communication overlap to identify why GPU workloads slow down.
GPU fabric utilization and traffic distribution
High GPU utilization does not necessarily imply high application throughput. Communication imbalance, insufficient fabric bandwidth and synchronization can leave accelerators waiting for data.
Select a workload and analyse the GPU fabric.
Investigate topology placement, collective communication patterns and fabric utilization before assuming the GPU compute architecture is the bottleneck.
Diagnosing GPU Fabric Failures and Communication Bottlenecks
Production AI infrastructure depends on reliable GPU interconnect communication. NVLink connectivity problems, NVSwitch faults, GPU errors, software configuration issues and scale-out network failures can disrupt distributed workloads or reduce communication performance.
Effective troubleshooting requires engineers to correlate application symptoms with GPU topology, fabric health, system logs, communication traces and platform telemetry. A slow NCCL collective does not automatically indicate a physical NVLink fault.
Normal GPU Fabric Operation
GPU endpoints and NVSwitch resources are operational in this conceptual baseline. Communication paths are available and no simulated fabric alarms are present.
GPU FABRIC STATUS
Links: Available
Switches: Operational
Collectives: Completing
Incident: None
1. Understanding GPU Fabric Failure Domains
A GPU communication failure may originate from the GPU itself, the interconnect, a switching component, a network adapter, the operating system or the application communication stack.
Production troubleshooting begins by identifying the failure domain. A physical NVLink issue should be distinguished from a software configuration problem or an external network bottleneck.
2. Establishing a Healthy Baseline
Before investigating a production incident, engineers should understand the expected GPU topology, available interconnects, supported peer relationships and normal communication performance.
Baseline information helps distinguish genuine degradation from normal differences between GPU communication paths.
Diagnostic principle: Start with the observed symptom, identify the affected communication boundary and collect evidence before assigning root cause. NCCL timeouts, low throughput and GPU errors can have multiple explanations.
3. NVLink Link Degradation
NVLink connectivity problems can affect communication between supported GPU endpoints and fabric components. Depending on the hardware and failure characteristics, the issue may manifest as reduced connectivity, link errors, transfer failures or degraded application behavior.
Engineers should inspect supported link status and error counters, correlate the incident with system logs and determine whether the problem is isolated to a particular endpoint or communication path.
Available counters and diagnostics depend on GPU generation, driver version and system platform.
4. NVSwitch Fabric Faults
NVSwitch provides switching resources for multi-GPU communication domains. Faults affecting the switching fabric may disrupt communication between participating GPUs or prevent the system from reaching the expected operational state.
Investigation should include supported switch health information, fabric management status, initialization events and hardware diagnostic results.
Engineers should not assume that a particular switch fault always produces a specific rerouting or failover behavior. The actual outcome is platform-dependent.
5. NCCL Collective Timeouts
A collective communication timeout occurs when the application or communication stack does not complete an operation within the expected time.
Possible causes include stalled participants, software errors, synchronization mismatches, transport failures, resource exhaustion or network problems.
Engineers should correlate NCCL diagnostics with application logs, GPU utilization, topology information and the health of any external network involved in the operation.
6. GPU Topology and Peer Access Problems
GPU applications may perform differently when device placement or peer connectivity differs from the expected topology.
Topology discovery can reveal whether GPUs are connected through NVLink, PCIe or other communication paths supported by the system.
Peer access capabilities and transport selection should be verified using appropriate CUDA, NCCL and platform diagnostics rather than assumed from the number of installed GPUs.
7. Production Diagnostic Tools
NVIDIA provides several tools and interfaces for examining GPU systems. The appropriate choice depends on hardware generation, platform support and the level of access available to the operator.
| Tool or interface | Primary purpose | Typical investigation |
|---|---|---|
| nvidia-smi | GPU management and supported telemetry | Device health, topology and available link information |
| DCGM | GPU monitoring and diagnostics | Health metrics, diagnostics and fleet monitoring |
| NCCL diagnostics | Collective communication troubleshooting | Transport selection, collective errors and communication behavior |
| CUDA diagnostics | Application and peer-access investigation | Device capabilities and runtime errors |
| Fabric management logs | Supported NVSwitch fabric operational information | Fabric initialization and management faults |
| System event logs | Host and driver events | Driver errors, device resets and correlated failures |
8. Structured Incident Investigation Workflow
9. Recovery and Operational Validation
Recovery procedures depend on the fault type and the operational policies of the platform. Some problems may require application changes or configuration corrections, while hardware-related incidents may require vendor-supported maintenance procedures.
After remediation, engineers should verify GPU visibility, supported fabric health indicators, communication functionality and representative workload performance.
Production changes should be performed through approved maintenance and incident-management procedures.
Operational takeaway: The strongest troubleshooting workflow combines a known-good topology baseline, synchronized telemetry, application-level evidence and controlled validation. Avoid treating one error message as proof of a specific hardware failure.
10. Engineering Takeaway
NVLink and NVSwitch troubleshooting requires a layered understanding of GPU hardware, switched fabric connectivity, collective communication software and the broader AI infrastructure.
By correlating physical connectivity, fabric health and application behavior, engineers can identify the likely failure domain and avoid unnecessary changes to healthy components.
This completes the nine-section editorial architecture of the NVIDIA NVLink and NVSwitch guide.
All incident scenarios and telemetry in the explorer are illustrative. Available diagnostics and recovery procedures depend on NVIDIA hardware generation, drivers, firmware and system vendor.
NVLink / NVSwitch Production Troubleshooting Simulator
Investigate GPU fabric incidents using simulated telemetry, diagnostic commands, topology inspection and corrective actions. Identify the root cause before restoring the communication fabric.
GPU fabric incident topology
Select an incident and load the fault into the GPU fabric.
No incident is currently active.
Complete diagnostics, identify the root cause and apply remediation.
- AI Inference Serving Architecture Explained - November 17, 2026
- Distributed AI Training Architecture Explained - November 17, 2026
- AI GPU Cluster Architecture Explained - November 17, 2026
