An inference latency budget breaks the user-facing SLO into components that engineering teams can measure and optimize separately. A request might spend time in the client/network, ingress/load balancer, inference queue, input preprocessing, model execution, output processing, token streaming, and downstream application logic. If the product promises p95 under 500 ms, the infrastructure team cannot spend 450 ms on queueing and expect the model kernel to fix the rest.
Within NVIDIA AI Infrastructure, latency budgeting connects accelerator configuration to product behavior. Triton exposes server-side request, queue, compute-input, compute-infer, and compute-output durations, while current NVIDIA model-analysis tools support percentile latency and, for LLM profiling, metrics such as time to first token, inter-token latency, and output token throughput.
The existing latency tuning for AI applications article provides broader application context; this page focuses on GPU-serving infrastructure.
Start with a percentile SLO, not an average
Average latency can look healthy while a minority of users experience extreme queueing or cold-start delays.
Define p50 for typical experience and p95/p99 for tail requirements, plus an error/timeout objective.
Triton Performance Analyzer and Model Analyzer can report percentile latencies so tuning can optimize within the actual product constraint.
Separate network/client time from server time
Client-observed latency includes request serialization, network transit, TLS, proxy/load-balancer behavior, and response transfer.
Server metrics begin later and should be compared with client metrics to estimate non-server overhead.
Benchmark from realistic client locations; localhost testing can hide network and gateway costs that dominate a distributed production service.
Queue time is the first sign of saturation
Triton tracks time spent waiting in the model scheduling queue.
As concurrency rises, throughput may continue increasing while queue latency grows rapidly once model instances approach saturation.
Autoscaling or capacity planning should react before the queue consumes most of the user latency budget.
Dynamic batching trades waiting time for GPU efficiency
Triton dynamic batching combines requests into larger batches that can execute more efficiently.
Current Triton guidance recommends first enabling default dynamic batching, measuring latency/throughput, then increasing batch size or queue delay only while the latency budget remains satisfied.
A larger queue delay can improve throughput but is literally time added to every request that waits for the batch.
Model instances can hide transfer time but can also add contention
Running multiple model instances on one GPU can overlap input/output transfers and inference compute for some models.
For other models, one instance with batching already saturates the GPU, and extra instances increase memory use and queueing without improving throughput.
Use Model Analyzer to sweep instance count and batching under p95/p99 constraints rather than assuming “more replicas per GPU” is faster.
LLM latency has distinct prefill and decode behavior
Time to first token includes queue, prompt processing/prefill, and initial response overhead.
Inter-token latency reflects decode performance once generation begins, while total completion also depends on output token count.
A workload with long prompts and short outputs has a different bottleneck from a chat workload with small prompts and hundreds of generated tokens.
KV cache capacity creates a concurrency ceiling
Each active LLM sequence consumes KV cache memory based on model architecture and sequence length.
Increasing concurrency improves throughput until memory capacity, bandwidth, or scheduling pressure degrades TTFT/ITL or rejects requests.
GPU Memory Bottlenecks explains how decode workloads can become memory-capacity/bandwidth constrained.
Disaggregated prefill/decode introduces transfer overhead and isolation benefits
Current NVIDIA Dynamo documentation describes separating prefill and decode into different worker pools and transferring KV cache between them using high-speed transports.
This can isolate long prompt processing from active decode and improve throughput/ITL under suitable load, but adds KV-transfer overhead and extra GPUs/pools.
Use disaggregation when measured workload characteristics justify it, not because it is architecturally fashionable.
Cold starts and model loading need a separate budget
Loading weights, initializing runtimes, compiling kernels, warming caches, or scaling a new replica can take seconds or minutes—far outside ordinary per-request latency.
Keep enough warm capacity for expected traffic, pre-load critical models, and treat scale-from-zero or model swap as a deployment/capacity event rather than ordinary request handling.
User SLOs should specify whether cold-start requests are excluded or protected by prewarming.
Benchmark with production sequence lengths and request rates
Inference performance depends on batch size, input size, output size, model precision, concurrency, cache behavior, and model instance configuration.
Triton Perf Analyzer supports concurrency and request-rate load modes, while Model Analyzer can search configurations under a latency budget.
Use real or representative input-length distributions; a 128-token benchmark cannot predict an 8k-token production prompt service.
Latency budgets are successful when every millisecond has an owner
The mature service can attribute p95/p99 to client/network, queue, input processing, model compute, output processing, prefill/decode, and downstream application work.
Optimization then becomes targeted: add GPU capacity for queue pressure, change batching for throughput, optimize kernels for compute, fix network for transit, or reduce prompt/output length at the product layer. A single end-to-end number becomes an actionable operating model.
Budget allocation should leave headroom for variance. If a 500-ms p95 SLO allocates exactly 500 ms across components at their average values, normal jitter guarantees tail violations. Give each major stage an operating target below its theoretical maximum and reserve error budget for network variance, GC, kernel scheduling, and bursty queue behavior.
Admission control can be better than infinite queueing. When incoming traffic exceeds serving capacity, letting requests wait unboundedly produces high p99 and wasted work for clients that will time out anyway. Queue-size/time limits, load shedding, backpressure, or request rejection can preserve latency for accepted traffic and trigger autoscaling sooner.
Batching policy should reflect traffic shape. At high steady QPS, batches form quickly with little added delay; at low or bursty QPS, waiting for a preferred batch size can consume most of the latency budget. Benchmark several request-rate distributions rather than one fixed concurrency test before selecting queue delay.
Token streaming changes perceived latency. TTFT drives how quickly the user sees the first response, while inter-token latency drives the smoothness of generation. A service with mediocre total completion time can still feel responsive if TTFT and ITL are strong, whereas excellent average tokens/sec can feel slow if the first token waits in a long queue.
Prompt length and output length should be part of capacity planning. Long prompts increase prefill compute; long outputs occupy decode capacity and KV cache longer. Track the production distribution and consider separate routing/limits for extreme requests so one very long conversation does not consume disproportionate latency budget for ordinary users.
Network placement matters for inference clusters too. A client may be milliseconds from the load balancer but model workers may fetch data, communicate across GPUs, or transfer KV state across the fabric. Measure client-to-edge, edge-to-worker, inter-GPU/network, and worker-to-client paths separately when tail latency is unexplained.
Autoscaling signals should include queue time and SLO pressure, not only GPU utilization. A GPU at 70% can still have poor TTFT if requests serialize behind long prefills, while a GPU at 95% may meet latency when batches are efficient. Use serving metrics that reflect the stage causing user delay.
Release validation should compare throughput-latency curves before and after model, precision, TensorRT/TensorRT-LLM, scheduler, driver, or hardware changes. A new configuration may raise maximum throughput but worsen p99 at the production operating point. The winning design is the one that serves the required request rate inside the latency budget with enough headroom for burst and failure.
Capacity models should include failure mode. If the service must survive one GPU/node/AZ failure, the normal operating point cannot consume 100% of healthy capacity. Reserve enough headroom that queue time and p99 remain acceptable when a replica disappears and the remaining workers absorb its traffic.
Per-tenant fairness can also affect latency. One client sending very long prompts or high concurrency can dominate batching and KV cache. Queue priorities, token/request limits, admission control, or separate model instances may be necessary so one tenant’s throughput optimization does not consume the latency budget of others.
Instrumentation should attach model version, precision, engine/config, GPU type, batch size, sequence lengths, and route/worker identity to latency metrics. Without this context, p99 regressions after a rollout are difficult to attribute because the dashboard blends several serving configurations into one line.
Latency SLOs should be reviewed alongside cost and utilization. Running enough replicas to guarantee extremely low queue time at all hours may leave expensive GPUs mostly idle; pushing utilization too high can make p99 unstable. The correct operating point is an explicit business trade-off between response time, traffic burst tolerance, redundancy headroom, and cost per request.
Performance tests should include overload behavior. Increase request rate beyond sustainable capacity and confirm queue limits, rejection, autoscaling, timeout, and recovery work as designed. A service that meets p95 only below saturation but collapses into minutes of backlog during a burst has not really defined or enforced a latency budget.
Latency governance should include rollback thresholds for releases. If a new engine, quantization method, batching policy, or model version raises p99 beyond the agreed budget under representative load, the deployment should revert or reduce traffic automatically rather than waiting for user complaints to prove the regression.
