NVIDIA Spectrum-X is an AI-optimized Ethernet platform combining Spectrum switches with NVIDIA SuperNICs and software for lossless RoCE, congestion control, adaptive routing and GPU-to-GPU scale-out networking. Current Network Operator 26.7.x validates Spectrum-X Reference Architecture 2.3 as GA, including single-plane designs with ConnectX-7 or BlueField-3 SuperNICs and dual/quad-plane designs with ConnectX-8 SuperNICs. NVIDIA’s current guidance recommends hardware multiplane where supported, with software multiplane also available.
Within NVIDIA AI Infrastructure, Spectrum-X is the Ethernet alternative for the scale-out compute fabric. It complements NVLink/NVSwitch scale-up inside a rack or server and enables Kubernetes workloads to consume high-performance Ethernet paths as schedulable resources.
RoCE performance depends on the complete Ethernet fabric
RDMA over Converged Ethernet requires consistent NIC, switch, queue, congestion and MTU configuration.
Spectrum-X combines hardware and software reference settings rather than leaving each team to tune ECN/PFC/adaptive routing independently.
Apply the profile matching the validated reference architecture instead of copying values from another GPU generation.
Reference Architecture 2.3 is the current Network Operator path
Current NVIDIA Network Operator 26.7.x marks Spectrum-X RA 2.3 GA.
The operator ships/apply versioned Spectrum-X profiles that include NIC nonvolatile settings and runtime RoCE, adaptive-routing and congestion-control parameters.
Pin the RA profile with the operator release so cluster upgrades do not silently change network tuning.
Single-plane is simplest but provides less path diversity
Single-plane Spectrum-X can use supported ConnectX-7, BlueField-3 or ConnectX-8 designs depending on the RA/hardware.
This reduces cabling and operational complexity.
Use it where workload scale, failure-domain and bandwidth requirements do not justify multiple independent planes.
Dual- and quad-plane designs increase bandwidth and resiliency
Current RA 2.3 supports dual- and quad-plane configurations with ConnectX-8 SuperNIC.
Each plane provides an independent fabric path, allowing workload traffic to be spread and preserving connectivity during a plane failure.
Keep rail/plane wiring symmetric across nodes so software does not route around inconsistent physical topology.
Hardware Multiplane reduces host software load
Current Network Operator supports hardware multiplane load balancing on relevant ConnectX-8 configurations and identifies it as recommended in RA 2.3 platform support.
Software multiplane remains another supported approach.
Benchmark both only when architecture requires a choice; do not replace a validated reference setting based on a single synthetic throughput test.
Network Operator manages the Kubernetes layer
Current Spectrum-X Kubernetes architecture separates host-layer prerequisites from Kubernetes-layer operators/CNI/IPAM/discovery.
The Network Operator handles Kubernetes-native configuration after host firmware, drivers, and base networking are ready.
Document Day-0 host ownership separately so one team does not assume the operator repairs firmware or BIOS prerequisites it does not own.
SR-IOV and resource allocation expose network devices to pods
High-performance GPU workloads often receive SR-IOV or related network resources so pods can access SuperNIC paths directly.
Resource names, NUMA/GPU affinity and plane selection should be aligned with scheduler policy.
NVIDIA GPUDirect RDMA explains why GPU-to-NIC placement affects end-to-end data movement.
Congestion control should be measured under collective traffic
AI traffic has synchronized bursts and incast patterns unlike ordinary web workloads.
Spectrum-X profiles include congestion-control and adaptive-routing settings designed for these patterns.
Use NCCL Collective Performance plus switch/NIC telemetry to validate application-level benefit.
NetQ and telemetry should validate fabric intent
NVIDIA NetQ and switch/NIC telemetry can provide topology, interface, loss/congestion and configuration visibility.
Compare discovered fabric state with the reference design after cable moves, switch replacements or node expansions.
A single node wired to the wrong plane can reduce performance without producing an obvious link-down.
Spectrum-X and NVLink solve different scales
NVLink and GPU Interconnects covers high-bandwidth scale-up inside an NVLink domain.
Spectrum-X carries scale-out GPU communication across nodes/racks and also supports converged AI factory networking patterns.
Troubleshoot local NVLink, host PCIe/SuperNIC and Ethernet fabric as separate layers before changing NCCL settings.
Spectrum-X succeeds when the Ethernet fabric is operated as part of the GPU system
The mature deployment pins a validated RA/operator profile, preserves rail/plane symmetry, aligns GPU/NIC placement, monitors congestion and topology, and changes switch/NIC tuning only with application evidence.
Ethernet can serve large AI clusters effectively when the fabric is engineered for RDMA/collective behavior rather than treated like an ordinary campus or server-access network.
Host-layer firmware and NIC settings should be qualified before Network Operator applies Kubernetes resources. Current NVIDIA architecture explicitly separates Day-0 host provisioning from Day-1/Day-2 Kubernetes reconciliation. Automate BIOS/firmware/driver validation and reject nodes that do not match the Spectrum-X reference profile before they join production workloads.
MTU should be consistent across host vSwitch, SR-IOV VF/PF, leaf, spine and peer nodes for the selected RoCE design. One mismatched link can create application-specific drops that appear only for large messages. Include end-to-end jumbo-frame testing in node qualification and after switch replacement.
Congestion-control telemetry should show ECN/PFC or the current Spectrum-X profile’s equivalent counters by port/queue. Packet loss is only one symptom; excessive pause or congestion marking can raise collective tail latency long before hard drops become obvious. Baseline the expected behavior of a healthy large job.
Plane-aware scheduling can improve both performance and failure isolation. When dual/quad-plane resources are exposed through DRA/SR-IOV resource classes, the scheduler should allocate the intended NIC/rail set consistently across all ranks. A job that gets different plane combinations on one node can defeat the reference architecture’s symmetry.
Switch topology should be validated from the workload perspective. NetQ may show every link up, yet one node can be cabled to the wrong rail/leaf. Compare physical port mapping against node/GPU position and run pairwise bandwidth tests across each rail so configuration and cabling errors are detected before full-scale NCCL jobs.
RoCE security and tenant isolation should be designed separately from performance. SR-IOV/VF assignment, Kubernetes network policy, VLAN/VXLAN/VRF design and cluster tenancy determine which workloads can reach each other. High-performance direct device access should not automatically imply a flat trusted compute network.
Reference profiles should be changed only with a benchmark and rollback plan. NIC runtime settings, adaptive routing, congestion control and inter-packet gap are coupled. Tuning one parameter outside the validated profile can improve one synthetic test while destabilizing mixed workloads.
Firmware/software upgrades should be staged across one fabric slice or node group. Validate link state, RoCE traffic, multipath behavior, congestion counters and NCCL jobs before updating the rest of the cluster. Preserve compatibility across switch and SuperNIC generations during the rollout.
Capacity planning should use oversubscription and rail bandwidth per GPU. Spectrum-X reference architectures are designed around specific switch/NIC ratios. Incremental node additions that exceed those ratios can make an originally non-blocking fabric oversubscribed even though all links run at rated speed.
Spectrum-X operations should share metrics with the GPU scheduler. When one rail or plane degrades, the platform should avoid scheduling the largest communication-heavy jobs onto affected nodes until remediation. Network health then becomes part of accelerator capacity rather than a separate dashboard nobody checks before placement.
Reference architecture selection should match GPU platform generation. RA 2.3 supports specific single-, dual- and quad-plane hardware combinations. Do not assume a ConnectX-7 design can adopt every ConnectX-8 multiplane feature through software alone. Hardware capability and cabling determine which profile is valid.
IPAM and network-resource lifecycle should align with pod scheduling. Leaked SR-IOV VFs, IPs or DRA claims after failed pods can reduce available network capacity even when GPUs are free. Monitor allocatable/allocated NIC resources and clean stale claims as part of cluster health.
Routing and failure convergence should be tested under active collective traffic. Pull one link, plane or switch and measure packet loss, NCCL stall/recovery and job impact. Redundant topology only creates resilience when software and routing react quickly enough for the application timeout.
Fabric telemetry should be normalized by rail/plane and job. A total switch utilization graph can look healthy while one rail used by one large job is congested. Map pod/rank to NIC/VF/rail and switch path so network operators can attribute hotspots to the workload generating them.
Storage and north-south traffic should not interfere with the compute fabric unless the reference architecture intentionally converges them. Separate or prioritize checkpoint/storage/service traffic so large data movement does not create tail latency for east-west collectives.
Network expansion should preserve the reference oversubscription target. Adding nodes to spare switch ports can be safe only if uplink/spine capacity and plane symmetry stay within the design. Update the topology model before physical expansion, not after benchmarks show the fabric slowed.
Switch/NIC counters should be retained long enough to correlate intermittent job regressions. Short-lived congestion or pause storms may be gone when the network team checks hours later. Historical rail/plane telemetry tied to job IDs is essential for proving whether the fabric caused the slowdown.
Operational ownership should span server, Kubernetes and network teams because Spectrum-X crosses all three domains. Define who owns firmware, NIC profile, SR-IOV resources, switch configuration and scheduler placement before a multi-plane incident forces ad hoc coordination.
Review fabric state continuously.
Operate Spectrum-X as an end-to-end fabric rather than a collection of fast ports. Baseline loss, congestion, ECN behavior, queue depth, link health, routing, and GPU communication patterns together, then compare those signals during training or inference slowdowns. Capacity changes should be tested with the actual collective and east-west traffic profile, because a network that looks healthy under ordinary server traffic can still create synchronized stalls across a GPU job. That correlation is what turns Ethernet telemetry into useful AI infrastructure evidence.
