NVIDIA DCGM Monitoring<\/a> explains health, diagnostics, XID, ECC, NVLink, and workload attribution.<\/p>\nAlert when exporter or expected GPU metrics disappear; loss of telemetry is itself a node-health problem.<\/p>\n
Node Feature Discovery and labels need governance<\/h3>\n Operator components depend on Kubernetes labels and feature discovery to know where GPU software belongs.<\/p>\n
Automated node pools should create and remove labels consistently during provisioning, repair, and retirement.<\/p>\n
Manual label drift can cause an operand to disappear from one node or deploy to hardware that cannot support it.<\/p>\n
Upgrades should use a canary node pool<\/h3>\n GPU Operator upgrades can restart operands or interact with the driver lifecycle.<\/p>\n
Drain a representative canary node, upgrade the operator\/operands, run GPU diagnostics and a real workload, then expand gradually.<\/p>\n
Keep the previous chart values and node image available for rollback until the production fleet passes qualification.<\/p>\n
Do not expose every host device to every workload<\/h3>\n Current release notes have fixed cases where feature flags accidentally exposed InfiniBand device nodes broadly to GPU containers.<\/p>\n
This is a useful design lesson: validate which GPU, RDMA, GDS, or networking devices are injected into workloads.<\/p>\n
Least-privilege device exposure matters both for security and for preventing one pod from selecting unintended rails.<\/p>\n
GPU Operator succeeds when it becomes reproducible infrastructure state<\/h3>\n The mature platform pins a supported operator release, controls driver ownership, validates component versions, reconciles MIG and monitoring, canaries upgrades, and compares physical inventory with advertised Kubernetes resources.<\/p>\n
The operator should make every GPU node look predictably ready\u2014not hide differences between node image, driver, runtime, network, and accelerator health.<\/p>\n
GPU Operator installation values should be treated as infrastructure code. Store the Helm values, chart version, driver policy, MIG strategy, toolkit\/runtime settings, monitoring options, and optional operands in version control. The console state should be reproducible from that record so a replacement cluster or disaster recovery event does not depend on remembering which flags an administrator clicked months earlier.<\/p>\n
Platform support must be checked before every Kubernetes or OS upgrade. GPU Operator support matrices tie operator versions to Kubernetes releases, Linux distributions, drivers, and component versions. Upgrade the cluster only after confirming the intended operator line supports the target control-plane\/node OS. A Kubernetes upgrade that leaves the GPU stack unsupported can strand accelerator nodes even though CPU workloads continue normally.<\/p>\n
Air-gapped clusters need an image and repository strategy. Mirror the operator chart and all operand container images into approved internal registries, rewrite image repositories consistently, and include driver packages if nodes cannot reach NVIDIA repositories. Test the full install from a disconnected environment; partial mirroring often fails only when a node replacement triggers one rarely used operand.<\/p>\n
Node drains should precede disruptive driver or MIG changes. Operators can reconcile desired state automatically, but the scheduler still needs to protect active jobs. Integrate maintenance with cordon\/drain, checkpointing where supported, job rescheduling, and node readiness gates so an operator rollout does not interrupt long-running distributed training unexpectedly.<\/p>\n
Security reviews should cover the privileged DaemonSets and host mounts used by GPU infrastructure components. Driver installation, container runtime integration, device plugins, and diagnostics require significant host access. Restrict who can modify the operator namespace and Helm release, enforce image provenance\/admission policy, and monitor changes to service accounts and privileged workload definitions.<\/p>\n
GPU Operator should integrate with cluster autoscaling carefully. New GPU nodes can take meaningful time to boot, install drivers, initialize devices, join monitoring, configure MIG, and become schedulable. Autoscaler readiness checks should wait for the operator’s node state and advertised GPU resources rather than marking a node ready immediately after kubelet joins.<\/p>\n
Node repair should verify all operator conditions, not only pod status. After hardware or driver maintenance, compare GPU count, product labels, MIG state, DCGM metrics, device-plugin resources, Container Toolkit runtime, and one CUDA\/NCCL workload with a known-good node. A green operator pod can coexist with one inaccessible GPU or degraded interconnect.<\/p>\n
Multi-tenant clusters should combine operator-managed resources with namespace quota and scheduler policy. The operator exposes accelerators; it does not decide which team may consume them or whether one namespace can monopolize the fleet. Use ResourceQuota, priority, Run:ai or other scheduler controls so standardized GPU provisioning also produces predictable allocation.<\/p>\n
Observability should include operator reconciliation failures. Alert on operand crash loops, driver validation failures, missing node labels, daemonsets not scheduled, MIG configuration errors, and exporter gaps. Treat these as capacity incidents because a single misconfigured node can repeatedly fail jobs or reduce available GPUs without a hardware failure.<\/p>\n
GPU Operator upgrades should have an explicit rollback boundary. Know whether the previous chart\/operands remain compatible with the new driver or node image, and avoid combining Kubernetes, OS, driver and operator upgrades in one change unless the reference matrix requires it. Smaller upgrade steps make root cause and recovery much easier.<\/p>\n
Helm rollback alone may not reverse a driver change already applied to nodes. Separate operator-control-plane rollback from node-driver rollback and test both. A downgrade can require draining nodes, restoring a prior driver package or image, and verifying CUDA\/NCCL compatibility before workloads return.<\/p>\n
Government-ready or regulated deployments should verify which operands in the current operator release are covered by the relevant supported build. NVIDIA notes that some optional components can have different government-ready status. Compliance teams should validate the exact enabled component set rather than treating the chart version as one monolithic certification.<\/p>\n
GPU Operator status should be surfaced to users through node readiness or cluster dashboards. Data scientists should not waste time debugging CUDA inside a pod when the node is still reconciling its driver or MIG profile. Expose a simple ‘GPU infrastructure ready’ state backed by operator conditions.<\/p>\n
Cluster restore procedures should include operator CRDs, namespace configuration and desired-state values, but not rely solely on backed-up running pods. Operators are designed to recreate operands from declarative state. Disaster recovery should restore that state first, then let reconciliation rebuild the GPU software stack cleanly.<\/p>\n
Operational ownership should be clear between Kubernetes platform and accelerator teams. One group should own the operator release\/Helm state, another may own hardware\/firmware and scheduler policy, but escalation boundaries should be documented. Ambiguous ownership turns every GPU incident into a cross-team handoff.<\/p>\n","protected":false},"excerpt":{"rendered":"
NVIDIA GPU Operator automates the Kubernetes software stack needed to make NVIDIA GPUs usable and observable on cluster nodes. Current GPU Operator 26.7.x is the supported release line, with 26.3.x deprecated and 25.10.x or earlier out of support. The operator manages components such as the NVIDIA driver or Driver Manager, Container Toolkit, Kubernetes device plugin, […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-20000","post","type-post","status-publish","format-standard","hentry","category-general"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t