Vertex AI endpoint autoscaling changes the number of inference nodes behind a deployed custom or AutoML model as request demand changes. A DeployedModel defines dedicated resources such as machine type, optional accelerator, minimum replicas, and maximum replicas. Standard autoscaling keeps at least one inference node; current Google documentation also offers a preview Scale To Zero mode for selected endpoint types and single-model deployments.
Within AI on Google Cloud, autoscaling should be designed around latency, cold-start time, capacity availability, and cost. More replicas are useful only when the model server and downstream application can use them effectively.
Endpoint scaling is different from Gemini managed API scaling. This article applies to models you deploy to Vertex AI inference endpoints with dedicated resources.
Standard autoscaling starts with minReplicaCount and maxReplicaCount
Set maxReplicaCount higher than minReplicaCount to let Vertex AI increase/decrease inference nodes under load.
In the standard stable pattern, minReplicaCount is at least one, which keeps baseline capacity warm.
Choose the minimum from idle/typical traffic and startup time; choose the maximum from peak load, quotas, and downstream capacity.
Autoscaling reacts to inference demand, not business importance
Vertex AI scales based on serving signals such as concurrent request pressure rather than knowing that one customer or endpoint is more important.
Use separate endpoints/deployed models or traffic policy when two workloads require different SLOs or cost envelopes.
Do not share one deployment merely to simplify model registry if a low-priority batch-like client can consume the same online capacity needed by latency-sensitive production traffic.
Machine type affects both capacity and scale granularity
A larger VM or GPU can serve more concurrent inference per replica but increases cost and makes each scaling step larger.
A smaller replica pool can scale more granularly but may require more nodes and networking overhead.
Benchmark request throughput, p95 latency, memory, and accelerator utilization per machine type before deciding whether to scale up vertically or horizontally.
Scale To Zero is a preview cost optimization
Current Google documentation allows selected deployments to set minimum replicas to zero through the preview Scale To Zero feature.
When no traffic arrives for the configured idle period, the deployment can remove all replicas and stop compute billing.
This is designed for endpoints with long predictable idle periods, not latency-critical services that must answer the first request immediately.
The first request to a zero-scaled endpoint is dropped with 429
Google explicitly documents that traffic arriving while a deployment is at zero receives a 429 indicating the model is not ready; that request triggers scale-up but is not queued automatically for later execution.
Clients must retry after backoff or use a warming mechanism before sending user traffic.
This behavior is the core trade-off: zero idle cost in exchange for cold-start request failure/latency.
initialReplicaCount defines the scale-up target from zero
When Scale To Zero wakes, the deployment scales to the configured initial replica count rather than necessarily only one replica.
Set this from the expected burst when an idle service wakes.
If the first traffic wave needs five nodes, waking to one and then waiting for normal autoscaling can produce prolonged throttling even after the initial 429.
Idle and minimum scale-up periods prevent rapid churn
Scale To Zero exposes timing controls for how long a deployment must stay up before it is eligible to evaluate for scale-down and how long it must be idle before reaching zero.
Current docs set bounded minimum/maximum values for these periods.
Choose values that avoid repeatedly tearing down/starting expensive GPU model servers between short traffic gaps.
Scale To Zero has important limitations
Google currently limits the feature to single-model deployments and excludes multi-host GPU/TPU deployments and shared public endpoints, among other constraints.
A deployment left at zero for more than 30 days without traffic can be automatically undeployed under current documented behavior.
Treat preview limitations as release-sensitive and verify them before using the feature in production architecture.
Capacity reservations can reduce scale-up stockout risk
Scaling back up—especially to scarce GPU shapes—can fail or take longer if capacity is unavailable.
Google documents Compute Engine reservations as a way to improve capacity assurance for Vertex AI inference.
For a critical endpoint, compare the cost of keeping minimum replicas warm versus holding reservation capacity versus accepting cold-start stockout risk.
Traffic splitting supports model canaries independently of autoscaling
Vertex AI endpoints can route percentages of traffic among deployed models.
Each deployed model has its own resource/scaling configuration, so a canary needs enough minimum/maximum capacity for its assigned traffic.
During rollout, monitor whether the candidate scales differently because of slower inference or larger memory use even when request rate is identical.
Endpoint autoscaling succeeds when scaling policy reflects startup physics and SLOs
The mature deployment benchmarks per-replica capacity, sets practical min/max limits, reserves scarce hardware where needed, tests canary scaling, and uses Scale To Zero only when clients can tolerate the 429/retry wake-up path.
Autoscaling should turn varying demand into predictable service—not move capacity surprises from deployment time into the first customer request.
Autoscaling should be load-tested with the production request shape. One large image, long sequence, or high-memory request can consume far more serving capacity than many tiny requests. Benchmark throughput and concurrency across representative p50/p95 inputs so replica targets reflect actual compute demand.
Minimum replicas are an SLO choice. Keeping two or more warm replicas can protect against one-node disruption and sudden traffic while increasing idle cost. For high-availability services, minimum one may be too fragile even though the platform technically supports it.
Maximum replicas should reflect quota and downstream systems. If preprocessing, feature lookup, database access, or response sinks can handle only 100 concurrent calls, scaling the model endpoint to 500 replicas can make the overall system less stable. Capacity limits should be coordinated across the request path.
Scale-up latency matters for large models and accelerators. Loading weights on a GPU can take minutes, so reactive autoscaling may lag behind sudden bursts. Use higher minimum capacity, initial replicas, predictable traffic scheduling, or reservations when the SLO cannot tolerate the warm-up period.
Scale To Zero clients should implement bounded retry with jitter and a user-visible warming state. The documented initial 429 is expected behavior, not necessarily an outage. Infinite retries or synchronized retries from many clients can create a thundering herd as the endpoint wakes.
Health checks should distinguish a zero-scaled endpoint from an unhealthy one. Monitoring should know that no replicas is intentional during idle periods and should alert on failure to scale up, excessive wake time, or repeated 429s after the expected readiness window rather than on replica count alone.
Reservations can conflict with the economic goal of Scale To Zero. Holding scarce GPU capacity guarantees wake-up availability but still has reservation economics; keeping a minimum replica warm may be simpler for some workloads. Compare total cost and SLO risk among warm capacity, reserved cold capacity, and best-effort zero scaling.
Canary deployments need independent scaling limits. If 5% of traffic goes to a new model with min=1 and high per-request latency, that one replica can saturate while the stable model looks healthy. Monitor the candidate’s request queue/latency/utilization separately and increase its capacity before interpreting quality metrics from a throttled canary.
Autoscaling policies should be revisited after model optimization. Quantization, TensorRT/optimized containers, batching, model-size changes, or newer accelerators can alter requests-per-replica significantly. Leaving old min/max settings after a large speedup wastes cost; after a regression it causes throttling.
Endpoint metrics should include request queueing, replica count, utilization, model-server latency, 429/5xx, and autoscaler decisions. A high CPU/GPU utilization number is not necessarily bad if p95 stays within SLO; conversely low average utilization can hide burst-driven tail latency. Scale policy should be tuned against user-visible outcomes.
Traffic splits during rollback deserve capacity planning. If a canary is rolled back from 20% to 0%, the stable deployment immediately receives that extra traffic. Keep enough stable-model max replicas and quota to absorb the shift without creating a second incident. Canary safety requires spare capacity on both directions of the rollback.
Idle cost should be evaluated at the endpoint portfolio level. Many low-traffic models each with minReplicaCount=1 can create substantial stranded compute. Consolidate compatible models where operationally safe, use Scale To Zero for truly intermittent workloads, or move suitable use cases to managed model APIs that do not require dedicated serving nodes.
Autoscaling should be tested with model-server errors and slow requests, not only healthy traffic. If requests hang because of a dependency or model bug, concurrency can rise and trigger scale-out that multiplies the failing load. Circuit breakers, timeouts, and health-based rollout controls are needed so autoscaling does not amplify a bad release.
Scale To Zero should be paired with a clear user experience for scheduled inactivity. For internal tools, a ‘warming model, retrying’ status may be acceptable; for customer APIs, the first-request 429 may violate contracts. Decide at product design time whether zero-scale economics are compatible with the endpoint’s external SLA.
Finally, document the expected scaling envelope per deployed model: machine type, accelerator, min, max, initial scale-from-zero count, target concurrency, expected warm-up time, reservation dependency, and peak request rate. This turns autoscaling from a hidden platform behavior into an SLO contract that application and infrastructure teams can review together.
