“the full planned interval completes with valid residual and no reset” addresses short HPL runs did not test sustained power thermal and memory stability, but the current hardware or infrastructure issue is the validation run did not maintain uninterrupted stress for the required duration. The option “result logs show the full planned active duration with no interruption” is more directly tied to the required outcome.

“repeat extended HPL returns correct numerical solution at every required stage” addresses numerical validity is a required condition throughout performance acceptance, but the current hardware or infrastructure issue is the validation run did not maintain uninterrupted stress for the required duration. The option “result logs show the full planned active duration with no interruption” is more directly tied to the required outcome.

“sustained multi-node HPL load stays below site power limits” addresses combined electrical headroom was not exercised under synchronized compute duty, but the current hardware or infrastructure issue is the validation run did not maintain uninterrupted stress for the required duration. The option “result logs show the full planned active duration with no interruption” is more directly tied to the required outcome.

“rerun the entire approved continuous HPL burn-in interval” addresses the validation run did not maintain uninterrupted stress for the required duration, but the current hardware or infrastructure issue is latent memory faults are surfacing during repeated compute and memory stress. The option “check GPU ECC events and isolate the suspected memory device” is more directly tied to the required outcome.

Useful for combined electrical headroom was not exercised under synchronized compute duty, but it does not remove the specific condition: latent memory faults are surfacing during repeated compute and memory stress.

“compare HPL.dat versions driver runtime and NUMA affinity across cohorts” addresses node software or affinity settings produce inconsistent compute efficiency, but the current hardware or infrastructure issue is latent memory faults are surfacing during repeated compute and memory stress. The option “check GPU ECC events and isolate the suspected memory device” is more directly tied to the required outcome.

“sustained multi-node HPL load stays below site power limits” addresses combined electrical headroom was not exercised under synchronized compute duty, but the current hardware or infrastructure issue is latent memory faults are surfacing during repeated compute and memory stress. The option “completed replacement or remediation yields error-free full-duration HPL” is more directly tied to the required outcome.

“repeat extended HPL returns correct numerical solution at every required stage” addresses numerical validity is a required condition throughout performance acceptance, but the current hardware or infrastructure issue is latent memory faults are surfacing during repeated compute and memory stress. The option “completed replacement or remediation yields error-free full-duration HPL” is more directly tied to the required outcome.

“the full planned interval completes with valid residual and no reset” addresses short HPL runs did not test sustained power thermal and memory stability, but the current hardware or infrastructure issue is latent memory faults are surfacing during repeated compute and memory stress. The option “completed replacement or remediation yields error-free full-duration HPL” is more directly tied to the required outcome.

“investigate node cooling and log steady-state GPU clock and temperature” addresses sustained matrix performance is limited by thermal throttling, but the current hardware or infrastructure issue is power-policy differences prevent reliable comparison of sustained performance. The option “capture clock power and thermal settings alongside each burn-in record” is more directly tied to the required outcome.

“correlate full-cluster HPL burn-in with rack and feed measurements” addresses combined electrical headroom was not exercised under synchronized compute duty, but the current hardware or infrastructure issue is power-policy differences prevent reliable comparison of sustained performance. The option “capture clock power and thermal settings alongside each burn-in record” is more directly tied to the required outcome.

“perform extended qualified HPL burn-in under hardware monitoring” addresses short HPL runs did not test sustained power thermal and memory stability, but the current hardware or infrastructure issue is power-policy differences prevent reliable comparison of sustained performance. The option “capture clock power and thermal settings alongside each burn-in record” is more directly tied to the required outcome.

“clocks and FLOPS remain consistent after thermal equilibrium” addresses sustained matrix performance is limited by thermal throttling, but the current hardware or infrastructure issue is power-policy differences prevent reliable comparison of sustained performance. The option “report contains compatible settings and reproduced performance within tolerance” is more directly tied to the required outcome.

“normalized per-node results converge within approved performance variance” addresses node software or affinity settings produce inconsistent compute efficiency, but the current hardware or infrastructure issue is power-policy differences prevent reliable comparison of sustained performance. The option “report contains compatible settings and reproduced performance within tolerance” is more directly tied to the required outcome.

“the full planned interval completes with valid residual and no reset” addresses short HPL runs did not test sustained power thermal and memory stability, but the current hardware or infrastructure issue is power-policy differences prevent reliable comparison of sustained performance. The option “report contains compatible settings and reproduced performance within tolerance” is more directly tied to the required outcome.

“aggregate logs and health traces from every rank during NeMo burn-in” addresses multi-rank failure investigation lacks complete framework and communication records, but the current hardware or infrastructure issue is short framework smoke tests missed cumulative training and storage faults. The option “run an approved duration NeMo workload including periodic checkpoints” is more directly tied to the required outcome.

Useful for framework collective configuration or interconnect stability is unverified, but it does not remove the specific condition: short framework smoke tests missed cumulative training and storage faults.

“pin dataset model tokenizer and NeMo configuration versions for the burn-in” addresses workload artifacts and configuration differ from the approved reference, but the current hardware or infrastructure issue is short framework smoke tests missed cumulative training and storage faults. The option “run an approved duration NeMo workload including periodic checkpoints” is more directly tied to the required outcome.

“expected steps complete with stable loss and valid output checkpoints” addresses framework start-up is not evidence that the training workload computes correctly, but the current hardware or infrastructure issue is short framework smoke tests missed cumulative training and storage faults. The option “sustained run creates and restores valid checkpoints without failure” is more directly tied to the required outcome.

“isolated failing rank and recovery evidence support a full passing rerun” addresses multi-rank failure investigation lacks complete framework and communication records, but the current hardware or infrastructure issue is short framework smoke tests missed cumulative training and storage faults. The option “sustained run creates and restores valid checkpoints without failure” is more directly tied to the required outcome.

“full framework run maintains accepted training speed without unexpected throttling” addresses application-specific workload characteristics expose thermal or power behavior absent in HPL, but the current hardware or infrastructure issue is short framework smoke tests missed cumulative training and storage faults. The option “sustained run creates and restores valid checkpoints without failure” is more directly tied to the required outcome.

Useful for workload artifacts and configuration differ from the approved reference, but it does not remove the specific condition: end-to-end training pipeline throughput is constrained by data loading.

“profile CPU RAM consumption and correct leaking workers or buffering” addresses host-side input or optimizer memory growth was not monitored, but the current hardware or infrastructure issue is end-to-end training pipeline throughput is constrained by data loading. The option “profile dataloader queue memory and storage performance over the burn-in” is more directly tied to the required outcome.

“aggregate logs and health traces from every rank during NeMo burn-in” addresses multi-rank failure investigation lacks complete framework and communication records, but the current hardware or infrastructure issue is end-to-end training pipeline throughput is constrained by data loading. The option “profile dataloader queue memory and storage performance over the burn-in” is more directly tied to the required outcome.

“comparable runs reproduce expected loss and throughput within tolerance” addresses workload artifacts and configuration differ from the approved reference, but the current hardware or infrastructure issue is end-to-end training pipeline throughput is constrained by data loading. The option “sustained training throughput remains within the accepted rate band” is more directly tied to the required outcome.

“full framework run maintains accepted training speed without unexpected throttling” addresses application-specific workload characteristics expose thermal or power behavior absent in HPL, but the current hardware or infrastructure issue is end-to-end training pipeline throughput is constrained by data loading. The option “sustained training throughput remains within the accepted rate band” is more directly tied to the required outcome.

“sustained run creates and restores valid checkpoints without failure” addresses short framework smoke tests missed cumulative training and storage faults, but the current hardware or infrastructure issue is end-to-end training pipeline throughput is constrained by data loading. The option “sustained training throughput remains within the accepted rate band” is more directly tied to the required outcome.

“pin dataset model tokenizer and NeMo configuration versions for the burn-in” addresses workload artifacts and configuration differ from the approved reference, but the current hardware or infrastructure issue is framework collective configuration or interconnect stability is unverified. The option “correlate NeMo parallelism settings NCCL traces and rank logs” is more directly tied to the required outcome.

Useful for application-specific workload characteristics expose thermal or power behavior absent in HPL, but it does not remove the specific condition: framework collective configuration or interconnect stability is unverified.

“run an approved duration NeMo workload including periodic checkpoints” addresses short framework smoke tests missed cumulative training and storage faults, but the current hardware or infrastructure issue is framework collective configuration or interconnect stability is unverified. The option “correlate NeMo parallelism settings NCCL traces and rank logs” is more directly tied to the required outcome.

“expected steps complete with stable loss and valid output checkpoints” addresses framework start-up is not evidence that the training workload computes correctly, but the current hardware or infrastructure issue is framework collective configuration or interconnect stability is unverified. The option “parallel training proceeds through expected steps without collective deadlock” is more directly tied to the required outcome.

“full framework run maintains accepted training speed without unexpected throttling” addresses application-specific workload characteristics expose thermal or power behavior absent in HPL, but the current hardware or infrastructure issue is framework collective configuration or interconnect stability is unverified. The option “parallel training proceeds through expected steps without collective deadlock” is more directly tied to the required outcome.

“comparable runs reproduce expected loss and throughput within tolerance” addresses workload artifacts and configuration differ from the approved reference, but the current hardware or infrastructure issue is framework collective configuration or interconnect stability is unverified. The option “parallel training proceeds through expected steps without collective deadlock” is more directly tied to the required outcome.