Generative AI on Databricks

Generative AI on Databricks now spans far more than model serving. Those serving, retrieval, evaluation, governance, and application-identity decisions are central to GenAI Engineer Associate preparation. Production applications can combine Foundation Model APIs, Unity Gateway model services, Databricks Apps, AI Search and Delta Sync indexes, online features, MLflow Prompt Registry, inference tables, tracing, scorers, and service-policy guardrails inside one governed platform. The engineering task is deciding which layer owns model access, application identity, retrieval, prompt versioning, quality monitoring, routing, and safety. A production release should preserve the trace from user identity through retrieval, model routing, tool execution, scorer results, and inference logging so failures can be assigned to the correct control boundary.

A production Databricks GenAI stack spans retrieval, application identity, model access, prompt management, evaluation, logging, and runtime governance. GenAI feature stores add governed online context, while the broader Databricks platform provides the catalog, workload, and security boundary around those components. A Databricks RAG stack is reliable only when retrieval state is observable, and MLflow for GenAI adds tracing and evaluation evidence that can connect a bad answer to the model, prompt, retrieved context, or application behavior that produced it.

Model access should be exposed through a governed service contract

Foundation Model APIs give Databricks users access to hosted foundation models without running their own deployment. Pay-per-token endpoints are the easiest entry point, while current Databricks guidance recommends priority pay-per-token for latency-sensitive production workloads that need more consistent performance, and provisioned throughput when the workload requires dedicated performance characteristics.

Foundation Model APIs expose pay-per-token and provisioned model access, while Unity Gateway can govern external providers and model services behind a stable application-facing contract. The design choice should keep provider-specific details from spreading through the application.

That model choice should remain swappable behind a stable service contract where possible. Hard-coding provider-specific details throughout application code makes migration, routing, cost control, and fallback harder.

Unity Gateway centralizes routing, access, cost, and runtime policy

Current Databricks documentation positions Unity Gateway as the central AI gateway for models, providers, MCP services, tools, skills, and guardrails. It can route inference traffic, apply access controls, attach service policies, log requests and responses, manage rate limits and budgets, and configure traffic splitting or fallbacks across model services.

Routing should be governed through model services rather than hard-coded client endpoints. Databricks model-serving routes can be evaluated in that context: traffic splitting, fallbacks, and provider changes belong behind a stable service identity that clients can keep using.

The architecture should therefore distinguish the stable application-facing service name from the model destinations behind it. That creates room for A/B testing, provider migration, capacity failover, and session affinity without forcing every client to change its model identifier.

Current Databricks governance uses Unity Gateway service policies to constrain AI interactions. AI governance therefore needs two distinct controls: Unity Catalog privileges determine who can invoke a service, while service policies can evaluate request and response content for risks such as sensitive data or prompt injection.

GenAI guardrails now belong in the Unity Gateway service-policy model rather than the older endpoint-only mental model. Access grants and content guardrails should be reviewed separately because passing one control does not satisfy the other.

Generative AI becomes easier to govern when model access, routing, observability, evaluation, and content policy are managed as one platform rather than rebuilt independently inside every application.

That architecture also needs a clean distinction between data governance and AI runtime governance. Unity Catalog governs tables, functions, models, prompts, and other securable objects; Unity Gateway governs how AI requests move through models and tools at runtime. A team can have perfect table permissions and still operate an ungoverned model path, or can have strong gateway policies while exposing the wrong source data upstream.

Application identity should stay explicit on every governed call

Databricks Apps has two complementary authorization modes. Every app has its own dedicated service principal for app-level operations. User authorization can also pass the signed-in user’s Databricks identity through so the application can enforce existing Unity Catalog permissions, row filters, and column masks.

A Databricks App can act with its dedicated service principal for shared operations or use user authorization when downstream access must preserve the signed-in user’s Unity Catalog permissions. The identity choice should be explicit on each governed call.

Identity should remain explicit through the full request path, including model calls, SQL, feature lookups, and downstream tools.

Retrieval quality starts with chunking and index freshness

Retrieval quality depends heavily on the units created before indexing. RAG chunking should preserve structure, stable identifiers, metadata, and source lineage so later search and citation stages can explain why a passage was retrieved. Databricks AI Search cannot reconstruct source relationships that ingestion discarded, so query-time tuning should follow—not replace—sound chunking, metadata, and index-freshness design. Query rewriting and reranking extend that retrieval pipeline, but chunking remains the first major quality decision because every later stage operates on the units created at ingestion time.

Databricks AI Search supports Delta Sync indexes that incrementally follow source Delta tables and Direct Vector Access indexes where the application manages updates explicitly. Delta Sync can run in continuous mode for low-latency freshness or triggered mode when the application wants controlled refresh windows. Vector Search sync determines how source-table changes become searchable evidence; continuous versus triggered sync, endpoint type, selected columns, and embedding management are operational freshness decisions.

Freshness should be treated as a retrieval SLO. A healthy endpoint with a stale index can still produce wrong answers.

Retrieval infrastructure should also be treated as a product. Chunking, Delta Sync freshness, embedding version, metadata filters, hybrid retrieval, query rewriting, and reranking all influence the evidence the model sees. A weak retrieval layer can make a strong foundation model appear unreliable.

Inference logging is a governed data product, not just debug output

Current Unity Gateway inference tables log model-service requests and responses into Unity Catalog Delta tables. They are billed Unity Gateway features and can support debugging, monitoring, optimization, and compliance use cases.

Modern deployments should use Unity Gateway-enabled inference logging. The Databricks inference-table path changed materially in 2026: the legacy experience stopped accepting new enablement in February and lost support after April 30, while current inference tables are governed through Unity Gateway.

Inference logs can contain sensitive prompts and outputs, so access, sampling, retention, and redaction should be designed with the same care as application data.

Prompts should be versioned release artifacts

MLflow Prompt Registry treats prompts as versioned assets in Unity Catalog. Versions are immutable, aliases are mutable pointers, tags and commit messages support change tracking, and deployed applications can resolve aliases such as production without hard-coding one version. The Beta MLflow Prompt Registry makes prompts versioned Unity Catalog assets with immutable versions, mutable aliases, lineage, and rollback, preventing production behavior from depending on silent prompt edits inside application code. The same release discipline applies beyond Databricks: prompt management requires ownership, version promotion, evaluation evidence, and rollback because a prompt change can alter production behavior as materially as a code release.

Evaluation should use consistent criteria before and after deployment

MLflow 3 scorers provide a common interface for evaluation. Built-in LLM judges cover common dimensions such as relevance, safety, retrieval relevance, groundedness, and correctness; custom LLM judges and code-based scorers provide domain-specific or deterministic checks. MLflow scorers can use consistent quality criteria during offline evaluation and production monitoring, making regressions actionable against the same acceptance dimensions used before release.

Evaluation results should feed product decisions. A scorer dashboard is useful only when poor results become curated examples, prompt/model changes, retrieval improvements, or policy changes.

Prompt management and evaluation complete the release loop. Prompt Registry gives the team a versioned artifact, MLflow tracing captures application behavior, scorers evaluate the same quality criteria in development and production, and inference tables add gateway-level request evidence. Together those systems make it possible to explain why a release improved or regressed instead of relying on anecdotal examples.

Applications, model services, and cost controls should remain independently deployable

Model serving and application serving should remain independently deployable. A Databricks App can be updated without replacing the model service, while a model route can change without redeploying the app. That separation is valuable when quality evaluation and capacity management move at different cadences.

Cost governance should follow the same layered design. Foundation Model APIs charge by model usage; AI Search and online features add serving cost; inference tables add logging cost; production scorers consume evaluator compute or model calls; Databricks Apps and serverless workloads add application cost. The unit economics of one user task should roll those layers together.

Release lineage should connect every layer of the GenAI stack

The platform is mature when the organization can change any one layer—model, prompt, retriever, route, guardrail, app, or scorer—without losing lineage or weakening the controls around the rest of the system. That is the real advantage of treating Databricks GenAI as a governed architecture rather than a collection of notebooks and endpoints.

Keep every release attributable to a model service, prompt version, retrieval index, application version, and scorer set so rollback and incident review can reconstruct the full GenAI stack.

Leave a Reply

How It Works

img
Step 1. Choose Exam
on ExamLabs
Download IT Exams Questions & Answers
img
Step 2. Open Exam with
Avanset Exam Simulator
Press here to download VCE Exam Simulator that simulates real exam environment
img
Step 3. Study
& Pass
IT Exams Anywhere, Anytime!