NVIDIA AI infrastructure is the system that keeps expensive accelerators useful. GPUs are only one layer. Training and inference performance also depend on storage throughput, CPU and NUMA placement, network topology, RDMA, host-memory movement, GPU memory behavior, cluster scheduling, health diagnostics, BlueField offload, and the latency budgets that connect server-side optimization to user experience. A GPU can report high utilization and still be part of a badly designed system if data arrives late, memory is fragmented, topology is wrong, or one slow node poisons a distributed job.<\/p>\n
This hub organizes that operating model for the NVIDIA<\/a> ecosystem. The first support cluster covers AI Storage Throughput, BlueField DPU Offload, GPU Cluster Burn-In Testing, GPU ECC Error Monitoring, GPU Memory Bottlenecks, NUMA for GPU Workloads, GPU Scheduling Basics, GPU vs CPU Workloads, NVIDIA GPUDirect RDMA, and Inference Latency Budgets.<\/p>\n The existing storage networking fundamentals<\/a> article provides useful I\/O context, while latency tuning for AI applications<\/a> provides the application-level perspective. This hub focuses on the physical and systems layer that makes those higher-level architectures perform predictably.<\/p>\n Training clusters can consume data faster than conventional shared storage was designed to serve it. The storage question is not only headline GB\/s: it includes file-count behavior, metadata latency, checkpoint write bursts, dataset sharding, read amplification, cache effectiveness, and whether many workers hit the same files simultaneously.<\/p>\n AI Storage Throughput<\/a> explains how to calculate per-node and aggregate demand, separate sequential bandwidth from random metadata pressure, and measure whether GPUs are waiting for input. GPUDirect Storage can create a direct DMA path between storage and GPU memory for supported designs, avoiding a bounce buffer through CPU memory and reducing CPU utilization.<\/p>\n High-performance storage should be evaluated with the actual training or inference access pattern. A benchmark that streams one large file at line rate can hide the small-file and checkpoint behavior that dominates real jobs.<\/p>\n NVIDIA BlueField DPUs combine high-performance networking with programmable Arm cores, hardware accelerators, and the DOCA software framework. Current DOCA documentation covers offload and acceleration for switching, crypto, storage emulation, GPUNetIO, remote GPU offload, DPA programming, SR-IOV, and infrastructure services.<\/p>\n BlueField DPU Offload<\/a> focuses on the architectural boundary: which network, security, storage, or virtualization tasks should run on the DPU so x86 host CPU cycles remain available to application work. Offload is useful only when the DPU path is measurable and recoverable; moving logic away from the host can also move troubleshooting away from familiar tools.<\/p>\n In AI clusters, BlueField can also sit close to GPU\/RDMA traffic, making topology and firmware\/software compatibility part of the end-to-end performance model.<\/p>\n New GPU nodes can pass basic enumeration and still fail under power, thermal, memory, PCIe, NVLink, or sustained compute pressure. NVIDIA DCGM diagnostics include software checks, memory tests, PCIe tests, sustained diagnostic workloads, memory-bandwidth tests, targeted stress\/power tests, NVLink bandwidth tests, and other health-focused plugins.<\/p>\n GPU Cluster Burn-In Testing<\/a> explains how to combine DCGM diagnostics with network, storage, power, and collective tests so one cluster acceptance run exercises the same resources production workloads will depend on.<\/p>\n Burn-in is not a single \u201cGPU passed\u201d badge. The objective is to catch infant mortality, cabling\/topology mistakes, marginal memory, thermal problems, and underperforming nodes before a 256-GPU training job spends hours discovering them expensively.<\/p>\n Datacenter GPUs expose memory-health information through NVIDIA management interfaces and DCGM. Current DCGM health monitoring includes volatile double-bit ECC events, retired-page limits, pending page retirements, row-remap failures, contained\/uncontained errors, and related XID conditions.<\/p>\n GPU ECC Error Monitoring<\/a> explains the difference between correctable and uncorrectable memory signals, why retired pages and row-remapping state matter, when a reset can clear volatile state, and when a node should be isolated for deeper diagnostics or hardware service.<\/p>\n Operations should track trends per GPU UUID, not just the current counter. A slowly growing memory-health problem is easier to address during maintenance than during a distributed training failure.<\/p>\n High HBM bandwidth does not guarantee an application reaches it. NVIDIA\u2019s current compute-triage guidance emphasizes that kernels can remain below peak DRAM throughput simply because they do not keep enough bytes in flight, even when accesses are coalesced and caches are behaving correctly.<\/p>\n GPU Memory Bottlenecks<\/a> covers memory-bandwidth saturation, latency hiding, occupancy, coalescing, host-to-device transfer, cache behavior, HBM capacity, and DCGM\/Nsight signals that distinguish a memory-bound workload from an underfed or low-occupancy one.<\/p>\n The operational lesson is to correlate application throughput with SM activity, DRAM activity, PCIe\/NVLink traffic, and memory allocation rather than treating \u201cGPU utilization\u201d as a single sufficient metric.<\/p>\n In dual-socket and large servers, GPUs and NICs are attached to specific PCIe root complexes and CPU\/memory NUMA nodes. NVIDIA NUMA for GPU Workloads<\/a> focuses on CPU affinity, pinned memory placement, local NIC selection, PCIe topology, GPUDirect RDMA locality, and how Kubernetes\/Slurm scheduling decisions can accidentally separate a workload from its fastest host resources.<\/p>\n Topology-aware placement becomes more important as accelerators and network interfaces get faster because cross-socket detours consume bandwidth and add latency that older systems could sometimes hide.<\/p>\n Kubernetes normally schedules whole GPUs as extended resources. NVIDIA GPU Operator can add MIG management and GPU time-slicing. Current Operator documentation notes that MIG provides memory and fault isolation through hardware partitions, while time-slicing multiplexes access without memory or fault isolation between replicas.<\/p>\n GPU Scheduling Basics<\/a> compares whole-GPU requests, MIG, time-slicing, quotas, topology, scheduling labels, oversubscription, and the observability trade-offs that appear once several workloads share one physical GPU.<\/p>\n The correct sharing model depends on the workload. Interactive notebooks may benefit from high sharing density, while tightly coupled training or latency-sensitive inference usually needs stronger performance isolation.<\/p>\n CUDA guidance continues to emphasize moving highly parallel work to the GPU while minimizing unnecessary transfers between CPU and device memory. CPU\/GPU transfer bandwidth is much lower than GPU-local memory bandwidth, so repeatedly moving small intermediate results back to the CPU can erase kernel speedups.<\/p>\n GPU vs CPU Workloads<\/a> covers arithmetic intensity, parallelism, branching, batch size, preprocessing, memory movement, overlap of transfer\/compute, and hybrid pipelines where the CPU orchestrates while the GPU owns dense parallel work.<\/p>\n The practical rule is not \u201cGPU faster than CPU.\u201d It is \u201cwhich processor can execute this workload with the least total system cost, including transfer, launch, synchronization, memory, and latency?\u201d<\/p>\n GPUDirect RDMA enables a PCIe peer such as a network interface to access GPU memory directly rather than staging traffic through host memory. Current NVIDIA documentation supports modern DMA-BUF-based integration and the older Storage throughput must be sized to keep accelerators fed<\/h3>\n
BlueField moves infrastructure work away from host CPUs<\/h3>\n
Burn-in testing should prove a cluster under sustained stress<\/h3>\n
ECC error monitoring should drive action before corruption or outage<\/h3>\n
GPU memory bottlenecks are about data movement and in-flight work<\/h3>\n
NUMA topology can make or break multi-GPU throughput<\/h3>\n
nvidia-smi topo -m<\/code> reports GPU\/GPU and GPU\/NIC topology plus CPU and memory affinity. A process running on the wrong CPU socket can move data across the inter-socket fabric before it ever reaches the local GPU.<\/p>\nGPU scheduling should distinguish exclusive, partitioned, and shared access<\/h3>\n
GPUs and CPUs should each own the work they execute well<\/h3>\n
GPUDirect RDMA shortens the network-to-GPU data path<\/h3>\n
nvidia-peermem<\/code> path in appropriate Linux environments, with NVIDIA recommending DMA-BUF in current GPU Operator guidance where supported.<\/p>\n