“system remains stable under sustained approved load with both PSUs healthy” addresses power-delivery faults coincide with load rather than application software bugs, but the current hardware or infrastructure issue is chassis cooling fault is causing workload-driven thermal protection. The option “all fans report healthy speed and nodes survive sustained GPU load” is more directly tied to the required outcome.

“acceptable inlet and device temperatures return after airflow correction” addresses inadequate intake cooling is external to the GPU fan modules, but the current hardware or infrastructure issue is chassis cooling fault is causing workload-driven thermal protection. The option “all fans report healthy speed and nodes survive sustained GPU load” is more directly tied to the required outcome.

“NIC enumerates and ports complete full-rate diagnostics after targeted repair” addresses nIC firmware power or PCIe enumeration path may be faulty, but the current hardware or infrastructure issue is chassis cooling fault is causing workload-driven thermal protection. The option “all fans report healthy speed and nodes survive sustained GPU load” is more directly tied to the required outcome.

“investigate driver queue interrupts and host CPU processing alongside link counters” addresses nIC host-side receive queue processing may be saturated or misconfigured, but the current hardware or infrastructure issue is nIC firmware power or PCIe enumeration path may be faulty. The option “isolate HCA PCIe status firmware and host driver initialization” is more directly tied to the required outcome.

Useful for aggregate system power condition is the likely fault domain, but it does not remove the specific condition: NIC firmware power or PCIe enumeration path may be faulty.

“correlate AER errors with the device slot and verify hardware seating” addresses pCIe slot or electrical signaling fault is blocking reliable device transfers, but the current hardware or infrastructure issue is nIC firmware power or PCIe enumeration path may be faulty. The option “isolate HCA PCIe status firmware and host driver initialization” is more directly tied to the required outcome.

“acceptable inlet and device temperatures return after airflow correction” addresses inadequate intake cooling is external to the GPU fan modules, but the current hardware or infrastructure issue is nIC firmware power or PCIe enumeration path may be faulty. The option “NIC enumerates and ports complete full-rate diagnostics after targeted repair” is more directly tied to the required outcome.

“an all-GPU stress run has no current limit or PSU fault after correction” addresses aggregate system power condition is the likely fault domain, but the current hardware or infrastructure issue is nIC firmware power or PCIe enumeration path may be faulty. The option “NIC enumerates and ports complete full-rate diagnostics after targeted repair” is more directly tied to the required outcome.

“corrected queue settings remove host drops while physical link remains error free” addresses nIC host-side receive queue processing may be saturated or misconfigured, but the current hardware or infrastructure issue is nIC firmware power or PCIe enumeration path may be faulty. The option “NIC enumerates and ports complete full-rate diagnostics after targeted repair” is more directly tied to the required outcome.

Useful for aggregate system power condition is the likely fault domain, but it does not remove the specific condition: power-delivery faults coincide with load rather than application software bugs.

“correlate AER errors with the device slot and verify hardware seating” addresses pCIe slot or electrical signaling fault is blocking reliable device transfers, but the current hardware or infrastructure issue is power-delivery faults coincide with load rather than application software bugs. The option “correlate PSU telemetry and GPU load and test the suspect power component” is more directly tied to the required outcome.

“inspect nvidia-smi DCGM and GPU Xid logs before changing hardware” addresses gPU diagnostic events were not correlated with the reported workload failures, but the current hardware or infrastructure issue is power-delivery faults coincide with load rather than application software bugs. The option “correlate PSU telemetry and GPU load and test the suspect power component” is more directly tied to the required outcome.

“all fans report healthy speed and nodes survive sustained GPU load” addresses chassis cooling fault is causing workload-driven thermal protection, but the current hardware or infrastructure issue is power-delivery faults coincide with load rather than application software bugs. The option “system remains stable under sustained approved load with both PSUs healthy” is more directly tied to the required outcome.

“corrected queue settings remove host drops while physical link remains error free” addresses nIC host-side receive queue processing may be saturated or misconfigured, but the current hardware or infrastructure issue is power-delivery faults coincide with load rather than application software bugs. The option “system remains stable under sustained approved load with both PSUs healthy” is more directly tied to the required outcome.

“fault timestamps identify the affected GPU and post-fix workload runs without new Xids” addresses gPU diagnostic events were not correlated with the reported workload failures, but the current hardware or infrastructure issue is power-delivery faults coincide with load rather than application software bugs. The option “system remains stable under sustained approved load with both PSUs healthy” is more directly tied to the required outcome.

“inspect nvidia-smi DCGM and GPU Xid logs before changing hardware” addresses gPU diagnostic events were not correlated with the reported workload failures, but the current hardware or infrastructure issue is inadequate intake cooling is external to the GPU fan modules. The option “inspect facility supply temperatures and rack airflow before replacing GPU” is more directly tied to the required outcome.

Useful for chassis cooling fault is causing workload-driven thermal protection, but it does not remove the specific condition: inadequate intake cooling is external to the GPU fan modules.

“correlate GPU launch power draw with PSU and feed alert logs” addresses aggregate system power condition is the likely fault domain, but the current hardware or infrastructure issue is inadequate intake cooling is external to the GPU fan modules. The option “inspect facility supply temperatures and rack airflow before replacing GPU” is more directly tied to the required outcome.

“an all-GPU stress run has no current limit or PSU fault after correction” addresses aggregate system power condition is the likely fault domain, but the current hardware or infrastructure issue is inadequate intake cooling is external to the GPU fan modules. The option “acceptable inlet and device temperatures return after airflow correction” is more directly tied to the required outcome.

“NIC enumerates and ports complete full-rate diagnostics after targeted repair” addresses nIC firmware power or PCIe enumeration path may be faulty, but the current hardware or infrastructure issue is inadequate intake cooling is external to the GPU fan modules. The option “acceptable inlet and device temperatures return after airflow correction” is more directly tied to the required outcome.

“all fans report healthy speed and nodes survive sustained GPU load” addresses chassis cooling fault is causing workload-driven thermal protection, but the current hardware or infrastructure issue is inadequate intake cooling is external to the GPU fan modules. The option “acceptable inlet and device temperatures return after airflow correction” is more directly tied to the required outcome.

Useful for aggregate system power condition is the likely fault domain, but it does not remove the specific condition: PCIe slot or electrical signaling fault is blocking reliable device transfers.

“correlate PSU telemetry and GPU load and test the suspect power component” addresses power-delivery faults coincide with load rather than application software bugs, but the current hardware or infrastructure issue is pCIe slot or electrical signaling fault is blocking reliable device transfers. The option “correlate AER errors with the device slot and verify hardware seating” is more directly tied to the required outcome.

“verify fan RPM BMC events and inlet temperatures then restore approved cooling” addresses chassis cooling fault is causing workload-driven thermal protection, but the current hardware or infrastructure issue is pCIe slot or electrical signaling fault is blocking reliable device transfers. The option “correlate AER errors with the device slot and verify hardware seating” is more directly tied to the required outcome.

“fault timestamps identify the affected GPU and post-fix workload runs without new Xids” addresses gPU diagnostic events were not correlated with the reported workload failures, but the current hardware or infrastructure issue is pCIe slot or electrical signaling fault is blocking reliable device transfers. The option “repeated DMA tests finish without new uncorrectable PCIe errors” is more directly tied to the required outcome.

“all fans report healthy speed and nodes survive sustained GPU load” addresses chassis cooling fault is causing workload-driven thermal protection, but the current hardware or infrastructure issue is pCIe slot or electrical signaling fault is blocking reliable device transfers. The option “repeated DMA tests finish without new uncorrectable PCIe errors” is more directly tied to the required outcome.

“an all-GPU stress run has no current limit or PSU fault after correction” addresses aggregate system power condition is the likely fault domain, but the current hardware or infrastructure issue is pCIe slot or electrical signaling fault is blocking reliable device transfers. The option “repeated DMA tests finish without new uncorrectable PCIe errors” is more directly tied to the required outcome.

“isolate HCA PCIe status firmware and host driver initialization” addresses nIC firmware power or PCIe enumeration path may be faulty, but the current hardware or infrastructure issue is nIC host-side receive queue processing may be saturated or misconfigured. The option “investigate driver queue interrupts and host CPU processing alongside link counters” is more directly tied to the required outcome.

“inspect nvidia-smi DCGM and GPU Xid logs before changing hardware” addresses gPU diagnostic events were not correlated with the reported workload failures, but the current hardware or infrastructure issue is nIC host-side receive queue processing may be saturated or misconfigured. The option “investigate driver queue interrupts and host CPU processing alongside link counters” is more directly tied to the required outcome.

“correlate PSU telemetry and GPU load and test the suspect power component” addresses power-delivery faults coincide with load rather than application software bugs, but the current hardware or infrastructure issue is nIC host-side receive queue processing may be saturated or misconfigured. The option “investigate driver queue interrupts and host CPU processing alongside link counters” is more directly tied to the required outcome.

“NIC enumerates and ports complete full-rate diagnostics after targeted repair” addresses nIC firmware power or PCIe enumeration path may be faulty, but the current hardware or infrastructure issue is nIC host-side receive queue processing may be saturated or misconfigured. The option “corrected queue settings remove host drops while physical link remains error free” is more directly tied to the required outcome.

“system remains stable under sustained approved load with both PSUs healthy” addresses power-delivery faults coincide with load rather than application software bugs, but the current hardware or infrastructure issue is nIC host-side receive queue processing may be saturated or misconfigured. The option “corrected queue settings remove host drops while physical link remains error free” is more directly tied to the required outcome.

“fault timestamps identify the affected GPU and post-fix workload runs without new Xids” addresses gPU diagnostic events were not correlated with the reported workload failures, but the current hardware or infrastructure issue is nIC host-side receive queue processing may be saturated or misconfigured. The option “corrected queue settings remove host drops while physical link remains error free” is more directly tied to the required outcome.