Inference Latency Budgets<\/a> explains why queue time is part of the user-facing SLO.<\/p>\nGPU memory is not a Kubernetes schedulable resource by default<\/h3>\n A whole GPU request does not specify how much HBM the application will actually consume.<\/p>\n
Time-sliced workloads can therefore collide in memory and fail even when the scheduler considered the logical GPU request valid.<\/p>\n
MIG solves this more directly with fixed memory slices; otherwise operators need application limits, profiling, admission policy, or specialized schedulers.<\/p>\n
Observability changes when GPUs are shared<\/h3>\n Current GPU Operator documentation notes limitations in associating DCGM exporter metrics with individual containers when time-slicing is enabled through the device plugin.<\/p>\n
Shared GPU scheduling can therefore reduce per-workload attribution compared with exclusive GPU or MIG isolation.<\/p>\n
Decide whether utilization gains justify the loss of precise accountability for cost, capacity, and noisy-neighbor troubleshooting.<\/p>\n
Quotas and priorities should reflect business value<\/h3>\n GPU clusters are scarce shared infrastructure. Namespace\/team quotas, priority classes, queue limits, and reservation policies prevent one experiment from consuming the entire fleet.<\/p>\n
Interactive work may need short queue latency but limited duration; production inference may need reserved capacity; batch research can consume spare capacity opportunistically.<\/p>\n
Make those policies explicit so scheduling conflicts become governed trade-offs rather than whoever submitted first.<\/p>\n
GPU scheduling is successful when utilization rises without making performance unpredictable<\/h3>\n The mature platform offers clear resource classes\u2014whole GPU, MIG, shared GPU\u2014plus topology-aware placement, quotas, health integration, queueing policy, and workload-level observability appropriate to each class.<\/p>\n
High utilization is useful only if users know what performance and isolation they are buying from the scheduler.<\/p>\n
Health should participate in scheduling. DCGM or node-health automation can label, cordon, or drain nodes with ECC, XID, NVLink, thermal, or diagnostic issues so new jobs are not assigned to questionable GPUs. A scheduler that knows capacity but not health can repeatedly place workloads onto hardware that already demonstrated a fault.<\/p>\n
GPU model heterogeneity should be exposed explicitly. Different generations have different HBM capacity, tensor formats, MIG profiles, interconnects, and performance. Queues or node pools should let users request a capability class rather than silently receiving whichever GPU happens to be free and then blaming application variability on the framework.<\/p>\n
Preemption policy matters for long-running training. Killing a job without a recent checkpoint can waste hours of GPU time, while never preempting batch work can block urgent inference or production jobs. Coordinate scheduler priority with checkpoint frequency and graceful termination so capacity can be reclaimed without disproportionate lost work.<\/p>\n
Reservations and quotas solve different problems. A quota limits how much a team can consume; a reservation ensures capacity will be available for a critical service or scheduled training run. Production inference often needs reserved baseline GPUs, while research queues can borrow unused capacity when it is truly spare.<\/p>\n
Topology-aware scheduling should include fabric location for multi-node jobs. Placing ranks across distant or oversubscribed leaf\/spine domains can reduce collective performance even when every node has identical GPUs. Rack, rail, switch, or network-domain labels can help a batch scheduler prefer compact allocations for communication-heavy training.<\/p>\n
Autoscaling GPU nodes has longer lead times than scaling stateless CPU pods. Driver initialization, GPU Operator components, model download, container image pull, MIG configuration, and model warmup can dominate readiness. Capacity policy should account for this delay instead of relying on reactive scale-from-zero for latency-critical workloads.<\/p>\n
Fairness metrics should consider GPU-hours and accelerator value, not only pod count. One user running eight H100s for ten hours consumes very different capacity from ten users each running a time-sliced T4 notebook. Chargeback or allocation policy should normalize usage by GPU class and exclusivity where business governance requires it.<\/p>\n
Scheduling success should be measured as completed-work efficiency: queue wait, GPU duty cycle, job failure\/preemption waste, inference SLO compliance, and fragmentation. A cluster at 95% allocated can still be inefficient if many assigned GPUs are idle because jobs wait on data, network, or each other.<\/p>\n
Fragmentation is an important scheduler metric. A cluster can have many free GPU-hours but still be unable to place an eight-GPU job because free accelerators are scattered across nodes or MIG profiles. Queue policy, defragmentation, backfill, and reservation windows should be designed around the job sizes the organization actually submits.<\/p>\n
Shared GPUs need noisy-neighbor expectations. Time-sliced workloads can interfere through memory pressure, kernel scheduling, caches, and power\/thermal behavior even when they use separate containers. Establish workload classes allowed to share and move latency-sensitive or untrusted jobs to exclusive\/MIG resources.<\/p>\n
Scheduling policy should expose why a pod\/job is pending. Missing GPU model, unavailable MIG profile, quota, topology, taint, health label, or gang requirement should be visible to users. Transparent queue reasons reduce support load and discourage users from weakening placement constraints just to get a job to start.<\/p>\n
Scheduler integration should expose accelerator maintenance state. Firmware upgrades, burn-in, diagnostics, or ECC remediation temporarily remove GPUs from service even when the node remains online. Use taints, drain states, maintenance queues, or dedicated labels so operational work does not race with user scheduling and so capacity dashboards distinguish unavailable-for-maintenance from genuinely free resources.<\/p>\n
Workload requests should express the minimum capability required without overspecifying one SKU. Requesting a particular GPU product everywhere can fragment the cluster and leave compatible newer accelerators idle. Capability classes based on memory, precision support, MIG requirement, RDMA, or performance tier can improve utilization while still giving applications predictable hardware behavior.<\/p>\n
Queue policy should be visible to users before they submit jobs. Expected wait time, maximum runtime, preemption class, allowed GPU types, sharing mode, and checkpoint requirements should be documented per queue or namespace. Clear service classes reduce accidental misuse and make scheduler behavior feel predictable instead of arbitrary when scarce accelerators are contested.<\/p>\n","protected":false},"excerpt":{"rendered":"
GPU scheduling is the process of deciding which workload gets which accelerator, for how long, and with what degree of sharing or isolation. In Kubernetes, GPUs are usually exposed as extended resources. NVIDIA GPU Operator can add drivers, device plugin, feature discovery, DCGM components, MIG management, and sharing configuration so the scheduler has enough information […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-19876","post","type-post","status-publish","format-standard","hentry","category-general"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t