NVIDIA NCA-AIIO: NVLink and GPU Interconnects

NVLink is the high-bandwidth GPU interconnect NVIDIA uses to create scale-up domains in which multiple accelerators can exchange data more directly than through ordinary host networking. NVSwitch extends that connectivity into switched topologies. The practical design question is not simply whether a server or rack “has NVLink.” It is which GPUs can communicate over which topology, at what effective bandwidth and latency, and how that local domain connects to the scale-out network. NVIDIA AI infrastructure depends on preserving those distinctions between the local scale-up topology and the scale-out network.

For NVIDIA NCA-AIIO candidates, current NVIDIA systems span several scales. DGX B200 integrates eight B200 GPUs with fifth-generation NVLink switches inside the system, while GB200 NVL72 designs extend an NVLink fabric across a rack-scale domain. Those systems still depend on external networking for communication beyond the scale-up domain. Operators should therefore keep NVLink/NVSwitch and InfiniBand or Ethernet roles conceptually separate.

The difference mirrors bandwidth versus throughput: a large aggregate interconnect number describes potential data movement, but useful application performance depends on topology, access pattern, message size, synchronization, locality, and software behavior. A model can underperform on a fast NVLink system if placement forces unexpected paths or communication is poorly balanced.

Scale-up and scale-out solve different communication distances

DGX B200 documentation describes eight B200 GPUs connected through fifth-generation NVLink switches with 14.4 TB/s aggregate NVSwitch bandwidth in the system. That figure is useful for understanding the scale-up capability, but the application still needs a communication pattern capable of exploiting it. Pairwise tests and collective tests answer different questions and should both be part of qualification.

Rack-scale GB200 NVL72 systems extend NVLink connectivity across many compute trays and dedicated NVLink switch trays. The scheduler can therefore treat the rack as a larger locality domain, but doing so means maintenance on rack-level switch or power/cooling components can affect a much larger group of cooperating GPUs.

Scale-up interconnects optimize communication among GPUs inside a server or rack-scale NVLink domain. Scale-out fabrics connect those domains across nodes and racks. Distributed frameworks and NCCL may use both in the same collective. A performance investigation therefore needs to know where a transfer crosses the boundary. Local GPU-to-GPU traffic and inter-node traffic have different latency, bandwidth, and failure characteristics.

Application partitioning should respect that hierarchy. Tensor-parallel groups or other tightly coupled ranks may benefit from being kept within a high-bandwidth scale-up domain, while data-parallel groups can span the external fabric. The correct decomposition depends on model architecture and framework, but the infrastructure must expose enough topology information for the scheduler and launcher to make an informed placement.

NVSwitch creates connectivity, not infinite nonblocking capacity

Application traffic can be asymmetric. Pipeline-parallel stages, expert routing, KV-cache movement, or collective phases may concentrate transfers among subsets of GPUs. Aggregate bandwidth does not reveal whether those hot pairs share constrained paths. Workload-specific peer matrices can expose patterns that a uniform synthetic test misses.

Firmware and fabric-management state matter as well. A system may enumerate every GPU while one link or switch path is degraded. Health monitoring should detect link state and error conditions, and acceptance after maintenance should exercise communication across the whole domain rather than only checking device discovery.

A switched NVLink topology simplifies GPU connectivity compared with relying on only direct point-to-point links, but contention and traffic pattern still matter. Many GPUs exchanging data simultaneously can stress shared paths or switch resources. Teams should validate collective patterns under realistic concurrency instead of inferring performance from a single pairwise bandwidth test.

Topology diagrams are operational tools, not marketing illustrations. They should map GPUs, switches, CPUs, NICs, and failure domains. During an incident, operators need to know whether two affected GPUs share a switch or whether a degraded path is isolated to one portion of the system. The physical relationship often narrows a problem faster than generic GPU utilization metrics.

GPU placement should preserve the topology qualified by the application

Qualification should include partial-topology tests. If one GPU or link is unavailable, determine whether the platform removes the affected resource, degrades to a smaller supported group, or exposes a topology that should be considered unusable for tightly coupled jobs. The scheduler needs an explicit answer; otherwise it may keep launching work into a configuration the application has never been tested to tolerate.

Schedulers often treat accelerators as a countable resource, but large models may depend on a particular number of GPUs inside one NVLink domain. Placement policy should prevent a tightly coupled workload from being split across a slower boundary merely because sufficient GPU count exists globally. Dedicated node classes, gang scheduling, topology-aware plugins, or higher-level platform admission can enforce those constraints.

This is another form of AI latency tuning: locality removes avoidable transfer time and synchronization. The benefit is workload-dependent. Small independent inference replicas may not care about peer bandwidth at all, while tensor-parallel inference or model training can be dominated by it. The platform should expose topology as a capability, not require every application team to reverse-engineer hardware layout.

NCCL translates topology into collective paths

Collective tests should cover both warm steady state and communicator creation. Large jobs can spend meaningful time establishing peer relationships and network state before the first training step. A topology or control-service problem may therefore appear as slow startup even when steady-state bandwidth is excellent after the job finally begins.

NCCL detects available GPU, NVLink, PCIe, and network paths and builds rings, trees, or other communication structures. Manual topology overrides should be exceptional because library behavior evolves with new hardware. The more durable platform responsibility is to present a correct hardware topology, compatible software stack, and stable process placement so the library can make good choices.

If collective performance is lower than expected, compare intra-node and inter-node tests, inspect which GPUs participate, and confirm that communication is actually using the intended NVLink paths. A regression isolated to one chassis or rack can indicate hardware, firmware, or topology state. A regression across every system after a software change points elsewhere.

Observability should include peer and link behavior, not only GPU load

Peer-to-peer status and NVLink counters should be evaluated against the expected topology. An unsupported peer relationship can be normal across a boundary, while the same status inside a required NVLink domain indicates a problem. Monitoring rules need topology awareness to avoid both false alarms and missed failures.

Performance dashboards should also show locality. If a job was placed across a slower boundary, the collective slowdown may be expected rather than a hardware fault. Exposing the placement decision next to communication metrics makes this distinction visible during incidents.

Network metrics for GPU interconnects should capture transfer activity, errors, health state, and peer relationships where the platform exposes them. DCGM can report NVLink-related health and profiling information, helping operators distinguish an idle interconnect from a degraded one. Correlate those signals with collective time rather than alerting on raw traffic volume.

A healthy link can also be underused because the workload never places communicating ranks near each other. This is why telemetry and scheduler metadata need to be combined. The question is not just “is NVLink healthy?” but “did the workload use the intended topology, and did that topology deliver the expected communication time?”

Rack-scale NVLink changes the failure and service boundary

IMEX and related multi-node NVLink services introduce additional control-plane components for systems that share GPU memory capabilities across nodes. Those services should be included in startup ordering, health checks, and upgrade procedures. A process can fail to use the intended memory path even when all GPUs themselves appear healthy.

The larger the scale-up domain, the more important maintenance domains become. Operators should know whether a switch tray, compute tray, or control-service failure requires draining the whole rack or only a subset. Capacity models need degraded-mode assumptions so schedulers do not place jobs that cannot survive the remaining topology.

GB200 NVL72-class systems extend the scale-up fabric across a rack with compute trays and NVLink switch trays. That makes the rack a more integrated computing unit. Maintenance, power, cooling, cable health, and switch availability can influence a larger set of GPUs than in a conventional independent-server rack. Runbooks should define what workload capacity remains after a component failure and whether the platform can isolate a partial fault.

Multi-node NVLink capabilities also introduce services that coordinate access beyond a single OS instance. Platform teams should treat those control components as part of the infrastructure release, monitor them, and include them in upgrade and incident procedures. Expanding the scale-up domain can improve model placement while also expanding the operational blast radius.

Design the interconnect hierarchy as one performance system

The NVIDIA platform should document the complete hierarchy: GPU local memory, intra-domain NVLink/NVSwitch, host PCIe, NIC attachment, and scale-out network. Performance engineering then asks which layer each communication phase crosses and whether that path is appropriate. This prevents teams from blaming the external network for traffic that never left the chassis or blaming NVLink for a slow cross-rack transfer.

Interconnect design is successful when application teams can request a topology class and the platform can deliver it predictably. Hardware bandwidth is only the foundation. Scheduling, collectives, telemetry, maintenance, and failure handling turn the local scale-up fabric into a usable production capability.

Leave a Reply

How It Works

img
Step 1. Choose Exam
on ExamLabs
Download IT Exams Questions & Answers
img
Step 2. Open Exam with
Avanset Exam Simulator
Press here to download VCE Exam Simulator that simulates real exam environment
img
Step 3. Study
& Pass
IT Exams Anywhere, Anytime!