NVIDIA GPU Operator packages the infrastructure components Kubernetes needs to discover, expose, monitor, and manage NVIDIA GPUs. Instead of treating the driver, container runtime integration, device plugin, feature discovery, monitoring, and MIG support as independent snowflakes on every node, the Operator provides a coordinated lifecycle. That makes it a platform component inside NVIDIA AI infrastructure, not simply a convenient Helm installation.
For readers studying NVIDIA NCA-AIIO, the current release lifecycle matters operationally. NVIDIA’s platform-support documentation lists the 26.7 line as supported, 26.3 as deprecated, and 25.10 and older as end of support. The exact operand versions change with releases, so the safer design is to manage the Operator and its component matrix as a tested platform bundle rather than pinning arbitrary combinations from different dates.
Kubernetes still owns scheduling and workload lifecycle. The Operator does not decide which model should run on which node or whether a rollout is safe for the application. It supplies GPU-aware capabilities that higher layers can use. Clear ownership between the Operator and Kubernetes scheduling prevents platform teams from expecting the device stack to solve placement policy.
Preflight the node and ownership model before installation
Node replacement should use the same automated bootstrap path as new capacity. If operators repair failed nodes manually until they differ from autoscaled nodes, the fleet accumulates hidden configuration classes. Reprovisioning from the qualified image and Operator configuration is often safer than preserving a snowflake host.
Cloud images and enterprise base images may already include an NVIDIA driver or container components. The installation plan should inventory that state rather than layering Operator-managed packages on top and hoping version resolution succeeds. If the host owns the driver, document how that driver is patched and how the Operator is configured not to replace it.
Kernel updates are another coordination point. A node can become unschedulable for GPU work if the running kernel and driver module no longer align after maintenance. Platform patching should sequence OS, kernel, driver, and validation together, with a node pool held back as a known-good recovery option.
GPU nodes differ in operating system, kernel, preinstalled driver state, secure-boot configuration, container runtime, networking, and cloud-provider image. The Operator can manage drivers in many environments, but some platforms intentionally provide a host driver outside the cluster. Teams should decide which layer owns the driver and disable conflicting management paths. Two controllers trying to own the same kernel component create brittle upgrades.
Cluster prerequisites should be validated before production rollout: supported Kubernetes platform, correct node labels, container runtime integration, available storage if needed, network access to registries, and maintenance procedures for node reboots. An Operator install that reports “ready” does not prove every GPU node is usable. Acceptance should include an actual GPU workload and health/telemetry verification.
The Operator coordinates several components with different failure modes
Custom resources and DaemonSets expose useful readiness information, but operators should map those Kubernetes objects to user-visible symptoms. A broken device plugin often manifests as zero allocatable GPUs; a toolkit problem may allow scheduling but prevent the container from starting correctly; monitoring can fail while workloads continue. Symptom-to-component mapping makes incident triage much faster than reading every Operator pod log.
Component logs should be retained long enough to diagnose node bootstrap and upgrade failures. Many issues occur only during reconciliation and disappear after a successful retry. Without logs, a platform can accumulate nodes that “eventually work” but require repeated intervention after every recycle.
A typical deployment includes driver management, NVIDIA Container Toolkit, Kubernetes device plugin, GPU Feature Discovery, DCGM and DCGM Exporter, and optional components such as MIG Manager. Each component has a different responsibility. The device plugin advertises allocatable GPU resources; feature discovery labels node capabilities; the toolkit connects containers to the driver; monitoring exposes health and utilization. Troubleshooting should identify which responsibility is failing before reinstalling the entire stack.
This separation also helps with upgrades. A new Operator release may update multiple operands together. Platform validation should check not only pod readiness but GPU allocation, container start, metrics, device health, and any special capabilities such as MIG. A green DaemonSet count is necessary but insufficient evidence that the full data path works.
Scheduling policy belongs above raw GPU discovery
Once devices are advertised to Kubernetes, placement still depends on requests, limits, labels, taints, affinities, topology, and workload policy. A generic request for one GPU can land on hardware that is technically compatible but wrong for memory size, interconnect topology, tenancy, or model profile. Feature labels and dedicated node pools allow the scheduler to express those distinctions.
Operators should avoid uncontrolled label proliferation. A stable platform taxonomy might encode GPU family, MIG capability, interconnect class, lifecycle ring, and tenancy tier, while model-specific preferences remain in workload configuration. The scheduler becomes easier to reason about when infrastructure labels describe durable hardware facts rather than every temporary application preference.
Upgrades should follow node pools and rollback boundaries
NVIDIA supports GPU Operator upgrades within the same major release or to the next major release, not arbitrary jumps across several major releases. Before updating a fleet, verify the supported source-to-target upgrade path and component compatibility matrix; an intermediate Operator release may be required. Record that tested path with the node image, driver ownership model, hardware class, and rollback evidence so future replacement nodes can reproduce the accepted state.
A small conformance workload should be runnable on demand by operations so a suspect node can be validated quickly without waiting for an application team to reproduce its full model stack.
Cluster autoscaling can interact badly with GPU upgrades if new nodes are created from an image or bootstrap path that has not yet been qualified. During a platform transition, pin autoscaling groups to a tested node image and Operator version combination. Otherwise the fleet can oscillate between old and new states depending on when capacity is added.
A rollback should be tested on an actual node, including any reboot required to restore a prior driver. Documentation that says “helm rollback” is incomplete if the host kernel, packages, or firmware changed as part of the release. Recovery time should be measured so maintenance windows and capacity buffers are realistic.
GPU infrastructure upgrades can affect kernel modules, runtime components, device discovery, monitoring, and application compatibility at once. Use Kubernetes rollouts to canary a small node pool, drain workloads, apply the new stack, run acceptance jobs, and expand only after evidence is clean. A fleet-wide Operator upgrade without a canary turns every unexpected dependency into a cluster-wide event.
The rollback plan should include more than reinstalling the previous Helm chart. Driver changes may require node reboots or package cleanup; CRDs and configuration can evolve; workload images may have changed during the same window. Record the tested downgrade path and preserve a known-good node pool during major transitions when capacity allows.
Monitoring should be installed as part of the platform contract
GPU Feature Discovery and related labels should be monitored for drift as well. If labels disappear after a node recycle, the scheduler may stop placing intended workloads even though the GPU itself is healthy. Platform tests should verify resource advertisement and critical labels after every autoscaled node joins the cluster.
Capacity dashboards should distinguish physical GPUs, allocatable GPUs, requested GPUs, and workloads waiting for a qualified node class. A cluster can report free accelerators while a production job remains pending because the available devices have the wrong generation, MIG layout, or topology. That distinction makes capacity conversations actionable.
DCGM Exporter is a standard companion to GPU Operator because it exposes GPU telemetry to Prometheus-style monitoring. The monitoring layer should be validated at install time, not added only after the first incident. Node and GPU identity, health status, utilization, power and thermal behavior, memory, and interconnect signals should be visible alongside Kubernetes workload metadata.
When a workload fails, Kubernetes failing-workload evidence and GPU telemetry should be examined together. Pending pods can indicate scheduling or resource-advertisement problems; crash loops can indicate image or runtime problems; Xid or hardware health errors can indicate the device layer. A shared dashboard shortens the path from symptom to responsible layer.
Isolation and tenancy require more than resource requests
Kubernetes resource allocation does not automatically create a complete security boundary. The isolation boundary still depends on host configuration, device access, runtime security, kernel trust, network controls, secrets, and administrative permissions. Multi-tenant GPU platforms should decide whether full GPUs, MIG instances, dedicated nodes, or stronger VM boundaries are appropriate for each workload class.
The Operator itself is privileged infrastructure. RBAC around its namespace, Helm releases, custom resources, driver configuration, and image sources should be tightly controlled. Supply-chain validation matters because compromise of the GPU software layer affects high-value compute nodes and every workload scheduled on them.
Platform teams should version the complete GPU service
The NVIDIA GPU service is a bundle of hardware generation, firmware, driver, Operator, operands, Kubernetes version, runtime, monitoring, and policy. Record those versions together and qualify them as a unit. A compatibility matrix maintained by the platform team is more useful than tribal knowledge about which combinations “usually work.”
The result should be a repeatable node lifecycle: provision, label, validate, schedule, observe, drain, upgrade, diagnose, and retire. GPU Operator reduces manual configuration drift, but operational reliability still comes from disciplined ownership and verification. Automation is valuable when it makes the known-good state reproducible rather than merely making changes faster.