NVIDIA Data Center GPU Manager provides a common operational layer for observing, validating, and diagnosing data-center GPUs. The useful distinction is between passive health monitoring, low-overhead profiling telemetry, active diagnostics, and exported metrics. Treating all four as “GPU monitoring” makes runbooks vague. In a production NVIDIA AI infrastructure platform, each mechanism answers a different question and should trigger a different operator action.
When preparing for NVIDIA NCA-AIIO, passive health monitoring interprets watched fields without applying a test workload. Diagnostics actively exercise components to determine whether a GPU or node can pass defined tests. Profiling fields expose utilization and transfer behavior at low overhead but are not kernel traces. DCGM Exporter exposes selected fields in Prometheus format so cluster monitoring can correlate GPU state with workloads and infrastructure.
AI observability is useful only when each signal maps to a decision it supports. A temperature metric, an NVLink error, a health warning, and a failed diagnostic are not interchangeable alerts. The monitoring design should identify when to observe, when to drain a node, when to run deeper tests, and when to escalate to hardware or platform support.
Passive health watches are the first line of evidence
DCGM health results include structured categories and severity that can support automated routing. A memory-related failure may justify immediate isolation, while a monitored performance condition may call for observation or workload migration. Preserve the raw code and category in incident records instead of reducing every non-pass state to a generic “GPU unhealthy” alert.
Health watches also need a retention window appropriate to the workload. If field history is discarded before a long training job completes, post-job health checks cannot explain transient conditions that occurred hours earlier. Monitoring retention should be sized for the longest investigation window the operations team expects to support.
DCGM health monitoring uses retained field samples to assess subsystems such as PCIe, memory, InfoROM, driver state, thermals, power, NVLink, NVSwitch, and related supported devices. Because it is passive, it can run while applications remain active. That makes it suitable for continuous watches, post-job checks, and rapid incident triage when a workload reports a problem.
A health result should not be interpreted without its category and recent workload context. A warning associated with thermal throttling has a different remediation from a driver or memory condition. The alert pipeline should preserve the subsystem, severity, error category, affected GPU, node, and time window so the operator can compare it with job placement and other host telemetry.
Diagnostics answer whether the node can pass an active test
Active DCGM diagnostics are appropriate when the operator needs stronger evidence than passive telemetry can provide. They can test hardware and software paths under controlled conditions, but that work may conflict with production jobs. A node should normally be drained or isolated before disruptive diagnostics. Running tests blindly on a busy node can create new workload symptoms and contaminate the investigation.
Runbooks should specify which diagnostic level is appropriate for a symptom and what constitutes a pass, fail, or retry. Repeated diagnostic failures should lead to quarantine rather than endless job rescheduling. The scheduler, monitoring system, and incident process need a shared node-health state so unhealthy hardware does not return automatically to the pool after a transient reboot.
Profiling metrics show utilization patterns, not source-code causality
The DCGM profiling module provides low-overhead hardware-counter metrics such as SM activity, tensor activity, memory behavior, and PCIe or NVLink traffic. These are useful for identifying whether a workload is compute-bound, memory-active, or moving substantial data. They are interval averages. They do not identify the CUDA kernel, source line, or instruction responsible for the pattern.
That boundary matters for GenAI production observability. Platform teams can use DCGM to show that a GPU is underutilized or communication-heavy, while application profilers or model-level traces explain why. Combining the layers is more useful than expecting one telemetry system to answer every performance question.
DCGM Exporter turns GPU fields into cluster telemetry
Prometheus labels can create cardinality problems when arbitrary job, model, request, or user values are attached to every GPU metric. Use a controlled labeling scheme and join high-cardinality application metadata at a higher layer when possible. The monitoring system should remain reliable during the same large-scale event it is expected to diagnose.
Metric scraping intervals should match the question. Fast sampling is useful for short power or utilization changes but increases monitoring volume; slower sampling can be adequate for capacity trends. Use separate high-resolution diagnostic views when needed rather than making every long-term metric expensive by default.
DCGM Exporter exposes selected DCGM fields as Prometheus metrics, typically one exporter per GPU node. In Kubernetes it can be deployed through Helm or managed as part of NVIDIA GPU Operator. The exporter can initialize an embedded DCGM host engine or connect to a separately managed engine, so operators should know which ownership model their deployment uses before troubleshooting startup or version problems.
Metric selection deserves governance. Exporting every available field at high cardinality can create monitoring cost without improving decisions. Start from operational questions: temperature and power headroom, ECC and memory health, GPU utilization, NVLink behavior, PCIe activity, clocks, and job attribution. Add fields when a runbook or capacity question needs them. Monitoring architecture should be intentional, not an indiscriminate dump of counters.
Alerts should distinguish performance pressure from hardware risk
Node-health automation should include hysteresis and recurrence. A single warning that immediately drains a node can create unnecessary churn, while repeated warnings across jobs may justify quarantine even if each event clears. The policy should state which conditions are immediately fatal, which require confirmation, and which are advisory.
The platform should also test that alerting still works when the affected node loses normal control-plane connectivity. Out-of-band or cluster-level evidence may be necessary to distinguish a dead exporter from a healthy GPU. “No metrics” is its own condition and should not be interpreted as healthy.
High utilization is normally a success condition, while thermal slowdown, persistent power capping, link errors, memory faults, or driver instability may indicate risk. Alert rules should therefore avoid treating “busy” as “unhealthy.” Pair saturation metrics with health status and expected workload behavior. A training cluster may be intentionally near full utilization for hours, whereas a single unexpected reset or rising correctable-error pattern can be far more actionable.
Thresholds also need time context. Brief thermal or power events during workload transitions may be harmless, while sustained throttling can reduce job efficiency and indicate facility or cooling constraints. Alerting should include duration, recurrence, and affected population. A single node behaving differently from otherwise identical nodes is often more informative than a fleet-wide absolute threshold.
Workload attribution makes incidents operationally useful
MIG deployments add another identity level because several logical GPU instances can share one physical device. Monitoring should preserve both the MIG entity and the parent GPU so an instance-level saturation problem is not confused with a physical-device health event that affects every tenant on the card.
Job-completion automation can use DCGM health evidence as part of node scoring. Repeated warnings across independent workloads are stronger evidence of a platform problem than one application failure. Feeding that history into maintenance decisions helps identify degrading nodes before they accumulate enough failures to become obvious.
The monitoring system should be able to map a GPU signal to the node, GPU or MIG instance, pod or job, tenant, model, and deployment version that was active at the time. Without that context, an alert becomes a scavenger hunt across scheduler logs. Kubernetes labels and exporter metadata can help, but teams should verify the mapping end to end rather than assuming dashboards will infer it correctly.
When a GPU-backed pod fails repeatedly, workload failures and DCGM evidence should be read together. Container restart loops, scheduling events, driver errors, device-plugin state, and GPU health can point to different layers. Correlation prevents an application team from endlessly restarting a workload on a node that should have been removed from service.
Monitoring is complete only when it drives node lifecycle decisions
Node history should remain queryable across reprovisioning so recurring hardware identity, slot, or rack-level faults are not erased by a fresh Kubernetes node name.
Return-to-service criteria should be as explicit as quarantine criteria. A repaired node should pass the required diagnostics, export expected telemetry, advertise the correct GPU resources, and complete a representative workload before it rejoins the production pool. A reboot alone is not evidence that the fault is gone.
A mature DCGM deployment has explicit transitions: healthy, suspect, drained, diagnostic, repaired, and returned to service. Automation can mark nodes unschedulable or open incidents, but the criteria should be documented and reversible. Overly aggressive automation wastes capacity; weak automation recycles bad nodes into new jobs. The decision policy should be tested with known failure scenarios.
The NVIDIA telemetry stack is most valuable when operators can move from a signal to a controlled action with preserved evidence. DCGM provides the GPU-specific facts, but reliability comes from integrating those facts with scheduling, incident management, capacity planning, and broader data-center telemetry. Monitoring is not the dashboard; it is the operating loop built around it.