“cross-rack NCCL bus bandwidth meets the approved topology baseline” addresses end-to-end collective communication path has an unverified performance bottleneck, but the current hardware or infrastructure issue is local NVLink measurements do not establish multi-node fabric throughput. The option “multi-node benchmark results verify the intended fabric data path” is more directly tied to the required outcome.

“results show matched parameters and explain expected scaling behavior” addresses collective-size and participant-count effects are ignored in baseline comparisons, but the current hardware or infrastructure issue is local NVLink measurements do not establish multi-node fabric throughput. The option “multi-node benchmark results verify the intended fabric data path” is more directly tied to the required outcome.

“tuned network path meets accepted latency and NCCL bandwidth without persistent congestion” addresses microburst congestion and flow control effects are hidden by averages, but the current hardware or infrastructure issue is local NVLink measurements do not establish multi-node fabric throughput. The option “multi-node benchmark results verify the intended fabric data path” is more directly tied to the required outcome.

“map ranks to the intended GPU pod and fabric topology before benchmarking” addresses process placement and communication topology are misaligned, but the current hardware or infrastructure issue is communication ranks are bound to nonfunctional or slow fabric adapters. The option “choose and verify intended network interfaces for every NCCL process” is more directly tied to the required outcome.

Useful for specific nodes or network paths degrade the larger collective, but it does not remove the specific condition: communication ranks are bound to nonfunctional or slow fabric adapters.

“execute NCCL collective tests that span physical nodes and switches” addresses local NVLink measurements do not establish multi-node fabric throughput, but the current hardware or infrastructure issue is communication ranks are bound to nonfunctional or slow fabric adapters. The option “choose and verify intended network interfaces for every NCCL process” is more directly tied to the required outcome.

“results show matched parameters and explain expected scaling behavior” addresses collective-size and participant-count effects are ignored in baseline comparisons, but the current hardware or infrastructure issue is communication ranks are bound to nonfunctional or slow fabric adapters. The option “NCCL logs show correct HCA selection and full-speed all-reduce” is more directly tied to the required outcome.

“alternative validated rank placements show balanced cross-pod throughput” addresses process placement and communication topology are misaligned, but the current hardware or infrastructure issue is communication ranks are bound to nonfunctional or slow fabric adapters. The option “NCCL logs show correct HCA selection and full-speed all-reduce” is more directly tied to the required outcome.

“tuned network path meets accepted latency and NCCL bandwidth without persistent congestion” addresses microburst congestion and flow control effects are hidden by averages, but the current hardware or infrastructure issue is communication ranks are bound to nonfunctional or slow fabric adapters. The option “NCCL logs show correct HCA selection and full-speed all-reduce” is more directly tied to the required outcome.

Useful for microburst congestion and flow control effects are hidden by averages, but it does not remove the specific condition: collective-size and participant-count effects are ignored in baseline comparisons.

“map ranks to the intended GPU pod and fabric topology before benchmarking” addresses process placement and communication topology are misaligned, but the current hardware or infrastructure issue is collective-size and participant-count effects are ignored in baseline comparisons. The option “standardize rank count message sizes and throughput interpretation across runs” is more directly tied to the required outcome.

“choose and verify intended network interfaces for every NCCL process” addresses communication ranks are bound to nonfunctional or slow fabric adapters, but the current hardware or infrastructure issue is collective-size and participant-count effects are ignored in baseline comparisons. The option “standardize rank count message sizes and throughput interpretation across runs” is more directly tied to the required outcome.

“tuned network path meets accepted latency and NCCL bandwidth without persistent congestion” addresses microburst congestion and flow control effects are hidden by averages, but the current hardware or infrastructure issue is collective-size and participant-count effects are ignored in baseline comparisons. The option “results show matched parameters and explain expected scaling behavior” is more directly tied to the required outcome.

“NCCL logs show correct HCA selection and full-speed all-reduce” addresses communication ranks are bound to nonfunctional or slow fabric adapters, but the current hardware or infrastructure issue is collective-size and participant-count effects are ignored in baseline comparisons. The option “results show matched parameters and explain expected scaling behavior” is more directly tied to the required outcome.

“port counters show both authorized rails carry expected traffic” addresses intended fabric rail balancing is not realized in collective traffic, but the current hardware or infrastructure issue is collective-size and participant-count effects are ignored in baseline comparisons. The option “results show matched parameters and explain expected scaling behavior” is more directly tied to the required outcome.

“bisect NCCL participants and correlate per-node NIC and fabric counters” addresses specific nodes or network paths degrade the larger collective, but the current hardware or infrastructure issue is intended fabric rail balancing is not realized in collective traffic. The option “validate NCCL rail and HCA mappings and supported multirail policy” is more directly tied to the required outcome.

“execute NCCL collective tests that span physical nodes and switches” addresses local NVLink measurements do not establish multi-node fabric throughput, but the current hardware or infrastructure issue is intended fabric rail balancing is not realized in collective traffic. The option “validate NCCL rail and HCA mappings and supported multirail policy” is more directly tied to the required outcome.

“run NCCL tests across the involved racks with per-link congestion monitoring” addresses end-to-end collective communication path has an unverified performance bottleneck, but the current hardware or infrastructure issue is intended fabric rail balancing is not realized in collective traffic. The option “validate NCCL rail and HCA mappings and supported multirail policy” is more directly tied to the required outcome.

“results show matched parameters and explain expected scaling behavior” addresses collective-size and participant-count effects are ignored in baseline comparisons, but the current hardware or infrastructure issue is intended fabric rail balancing is not realized in collective traffic. The option “port counters show both authorized rails carry expected traffic” is more directly tied to the required outcome.

“alternative validated rank placements show balanced cross-pod throughput” addresses process placement and communication topology are misaligned, but the current hardware or infrastructure issue is intended fabric rail balancing is not realized in collective traffic. The option “port counters show both authorized rails carry expected traffic” is more directly tied to the required outcome.

“NCCL logs show correct HCA selection and full-speed all-reduce” addresses communication ranks are bound to nonfunctional or slow fabric adapters, but the current hardware or infrastructure issue is intended fabric rail balancing is not realized in collective traffic. The option “port counters show both authorized rails carry expected traffic” is more directly tied to the required outcome.

“standardize rank count message sizes and throughput interpretation across runs” addresses collective-size and participant-count effects are ignored in baseline comparisons, but the current hardware or infrastructure issue is microburst congestion and flow control effects are hidden by averages. The option “inspect ECN PFC and queue telemetry during actual collective messages” is more directly tied to the required outcome.

“map ranks to the intended GPU pod and fabric topology before benchmarking” addresses process placement and communication topology are misaligned, but the current hardware or infrastructure issue is microburst congestion and flow control effects are hidden by averages. The option “inspect ECN PFC and queue telemetry during actual collective messages” is more directly tied to the required outcome.

“execute NCCL collective tests that span physical nodes and switches” addresses local NVLink measurements do not establish multi-node fabric throughput, but the current hardware or infrastructure issue is microburst congestion and flow control effects are hidden by averages. The option “inspect ECN PFC and queue telemetry during actual collective messages” is more directly tied to the required outcome.

“cross-rack NCCL bus bandwidth meets the approved topology baseline” addresses end-to-end collective communication path has an unverified performance bottleneck, but the current hardware or infrastructure issue is microburst congestion and flow control effects are hidden by averages. The option “tuned network path meets accepted latency and NCCL bandwidth without persistent congestion” is more directly tied to the required outcome.

“results show matched parameters and explain expected scaling behavior” addresses collective-size and participant-count effects are ignored in baseline comparisons, but the current hardware or infrastructure issue is microburst congestion and flow control effects are hidden by averages. The option “tuned network path meets accepted latency and NCCL bandwidth without persistent congestion” is more directly tied to the required outcome.

“NCCL logs show correct HCA selection and full-speed all-reduce” addresses communication ranks are bound to nonfunctional or slow fabric adapters, but the current hardware or infrastructure issue is microburst congestion and flow control effects are hidden by averages. The option “tuned network path meets accepted latency and NCCL bandwidth without persistent congestion” is more directly tied to the required outcome.

“validate NCCL rail and HCA mappings and supported multirail policy” addresses intended fabric rail balancing is not realized in collective traffic, but the current hardware or infrastructure issue is specific nodes or network paths degrade the larger collective. The option “bisect NCCL participants and correlate per-node NIC and fabric counters” is more directly tied to the required outcome.

“choose and verify intended network interfaces for every NCCL process” addresses communication ranks are bound to nonfunctional or slow fabric adapters, but the current hardware or infrastructure issue is specific nodes or network paths degrade the larger collective. The option “bisect NCCL participants and correlate per-node NIC and fabric counters” is more directly tied to the required outcome.

“run NCCL tests across the involved racks with per-link congestion monitoring” addresses end-to-end collective communication path has an unverified performance bottleneck, but the current hardware or infrastructure issue is specific nodes or network paths degrade the larger collective. The option “bisect NCCL participants and correlate per-node NIC and fabric counters” is more directly tied to the required outcome.

“cross-rack NCCL bus bandwidth meets the approved topology baseline” addresses end-to-end collective communication path has an unverified performance bottleneck, but the current hardware or infrastructure issue is specific nodes or network paths degrade the larger collective. The option “repaired node set achieves comparable bandwidth across all test subsets” is more directly tied to the required outcome.

“tuned network path meets accepted latency and NCCL bandwidth without persistent congestion” addresses microburst congestion and flow control effects are hidden by averages, but the current hardware or infrastructure issue is specific nodes or network paths degrade the larger collective. The option “repaired node set achieves comparable bandwidth across all test subsets” is more directly tied to the required outcome.

“port counters show both authorized rails carry expected traffic” addresses intended fabric rail balancing is not realized in collective traffic, but the current hardware or infrastructure issue is specific nodes or network paths degrade the larger collective. The option “repaired node set achieves comparable bandwidth across all test subsets” is more directly tied to the required outcome.