agent evaluation<\/a> or model-quality measures so a fast endpoint does not hide declining answer quality.<\/p>\nRedact or restrict logged prompts when they contain sensitive customer content.<\/p>\n
Model Serving succeeds when the endpoint is a governed release boundary<\/h3>\n The mature system uses the right endpoint type, stable URL, controlled model versions, route optimization\/capacity settings, no inappropriate zero-scaling, central governance, canary traffic, and quality\/latency\/cost monitoring.<\/p>\n
Model Serving should turn a model or agent artifact into a production service without turning infrastructure abstraction into operational opacity.<\/p>\n
Endpoint type should match workload lifecycle. Pay-per-token foundation models are easiest for variable demand; provisioned throughput fits steady, latency-sensitive foundation-model traffic; custom CPU\/GPU endpoints fit proprietary ML\/LLMs; external-model endpoints fit centralized governance over third-party providers. Standardize decision criteria so teams do not deploy a GPU endpoint when an existing managed model would be cheaper and easier.<\/p>\n
Model terms and acceptable-use obligations belong in the deployment checklist. Databricks publishes separate terms for some hosted models, and customers remain responsible for compliance. Record which model\/provider terms apply to each production endpoint and review them when changing model versions.<\/p>\n
Route-optimized endpoints change authentication and URL semantics. Migration should update every client, gateway, health check, test script, and service integration that assumes the older endpoint path. Run both paths during a transition where possible and validate OAuth token handling under production load.<\/p>\n
Custom GPU endpoints need capacity planning beyond autoscaling. Large custom LLMs can take significant time to build\/start, GPU capacity may not be available on demand, and some custom LLM serving modes use fixed workload sizes rather than rapid dynamic scale. Keep a warm production footprint where SLOs require it and test regional capacity before a major launch.<\/p>\n
Endpoint-level secrets and environment variables should be governed as deployment configuration. Use Databricks secret references, rotate external-provider credentials, and avoid exposing secrets in endpoint JSON exports or logs. A model artifact should not contain production API keys.<\/p>\n
Fallbacks and rate limits should be applied with semantics in mind. A fallback to another external model may preserve availability but change quality, safety, cost, or data-processing geography. Define which models are safe substitutes and expose fallback usage in telemetry rather than silently treating every successful response as equivalent.<\/p>\n
Inference tables and traces should have explicit retention and PII rules. Payload logging is valuable for debugging and evaluation, but customer prompts and model outputs can contain sensitive data. Sample or redact aggressively and make it possible to delete user-associated records when required.<\/p>\n
Serving rollbacks should restore both model and dependent feature\/prompt versions. A custom ML model with automatic feature lookup, or an agent that loads a Prompt Registry alias, can change behavior even if the endpoint’s model version stays constant. Release manifests should include those dependencies so rollback returns to a tested system, not only a previous binary.<\/p>\n
Autoscaling should be benchmarked with production payload sizes. The same endpoint can have very different model execution time for short versus long inputs, which changes concurrency needs. Load tests should reproduce request-size distribution and streaming behavior rather than send one tiny synthetic payload repeatedly.<\/p>\n
Batch inference and real-time serving should be separated operationally. Databricks recommends AI Functions\/other batch patterns for large offline jobs; pushing offline enrichment through a latency-oriented serving endpoint can distort autoscaling and consume capacity needed by interactive users.<\/p>\n
Endpoint ACLs should be group\/service-principal based and reviewed regularly. CAN QUERY is sufficient for most callers; CAN MANAGE should be restricted to platform\/release identities. Applications should not carry management permission merely because the SDK client can use it.<\/p>\n
Model-serving incidents should distinguish endpoint health, served-entity health, provider\/model errors, AI Gateway\/Unity Gateway policy blocks, and client authentication errors. Alert messages and traces should carry enough context to route the incident correctly instead of labeling every non-200 response ‘model down.’<\/p>\n
Capacity and cost policy should be reconsidered after model optimization. Quantization, a faster model version, different GPU, or route optimization can change QPS per replica dramatically. Keep old workload-size\/provisioned-concurrency settings only if fresh load tests show they are still appropriate.<\/p>\n
Endpoint migrations should include client timeout and retry tuning. A faster route-optimized endpoint or different foundation model can change typical response time and streaming cadence. Keep retries idempotent and avoid overly short timeouts that convert normal long-generation requests into duplicate calls and extra cost.<\/p>\n
Serving governance should include an endpoint inventory with owner, environment, model\/provider, data residency, scale settings, route optimization, fallbacks, and last review. Large workspaces accumulate endpoints quickly; unused or orphaned endpoints create cost, secret, and attack surface.<\/p>\n
Keep serving configuration under release control so capacity, routing, model, fallback, and governance changes remain auditable.<\/p>\n
Production endpoint changes should be rehearsed in staging with the same client authentication, route-optimized URL behavior, secrets, model permissions, and network restrictions so release failures are found before real traffic depends on the new serving configuration.<\/p>\n
Treat the serving endpoint as a release artifact with version, permissions, dependencies, traffic policy, and rollback state. That makes a model deployment auditable and keeps endpoint changes from becoming invisible infrastructure edits disconnected from evaluation evidence.<\/p>\n","protected":false},"excerpt":{"rendered":"
Mosaic AI Model Serving is now generally documented simply as Databricks Model Serving. It provides serverless real-time and batch inference endpoints for custom MLflow models, agents, foundation models, and externally hosted models behind a common REST and MLflow Deployments interface. Current Model Serving automatically manages infrastructure for standard custom endpoints and offers route optimization, autoscaling\/provisioned […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-19944","post","type-post","status-publish","format-standard","hentry","category-general"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t