{"id":19945,"date":"2026-10-06T15:14:25","date_gmt":"2026-10-06T15:14:25","guid":{"rendered":"https:\/\/www.exam-labs.com\/blog\/?p=19945"},"modified":"2026-10-06T15:14:25","modified_gmt":"2026-10-06T15:14:25","slug":"databricks-genai-engineer-associate-mosaic-ai-vector-search","status":"publish","type":"post","link":"https:\/\/www.exam-labs.com\/blog\/databricks-genai-engineer-associate-mosaic-ai-vector-search","title":{"rendered":"Databricks GenAI Engineer Associate: Mosaic AI Vector Search"},"content":{"rendered":"

Mosaic AI Vector Search is now documented as Databricks AI Search<\/strong>. The product still provides the same core retrieval role\u2014creating indexes over Delta\/Unity Catalog data and querying them with approximate-nearest-neighbor, hybrid, or full-text search\u2014but current 2026 documentation uses the AI Search name and a new databricks-ai-search<\/code> SDK. Databricks also made reranking generally available and continues to expand query types, filtering, ACLs, and retrieval-quality tooling.<\/p>\n

Within Generative AI on Databricks<\/a>, AI Search is the managed retrieval layer for RAG and agent applications. It can sync from Delta tables, manage embeddings for supported index types, and expose a governed query API without requiring teams to operate a separate vector database.<\/p>\n

The existing Databricks AI Search<\/a> article provides the broader retrieval-quality context. This page focuses on current product capabilities and operating choices.<\/p>\n

AI Search is the current product name<\/h3>\n

Current Databricks query documentation explicitly states that AI Search was formerly known as Databricks Vector Search.<\/p>\n

Teams should update new code, package names, internal docs, and dashboards to the AI Search terminology while preserving older names where legacy APIs or existing resources still expose them.<\/p>\n

This matters during migration because users can otherwise assume Vector Search and AI Search are separate products.<\/p>\n

Delta Sync indexes keep retrieval close to governed source tables<\/h3>\n

A Delta Sync index can track an underlying Delta table and automatically update the search index as source rows change.<\/p>\n

Depending on index configuration, Databricks can compute embeddings from a model endpoint or sync self-managed vector columns.<\/p>\n

Keep the source table narrow and retrieval-oriented where possible; indexing a massive operational table with dozens of unused columns increases sync and governance complexity.<\/p>\n

Direct-access indexes fit externally managed ingestion<\/h3>\n

Direct Access indexes let applications write vectors and metadata directly through the API rather than syncing from one Delta source table.<\/p>\n

This is useful when embeddings are produced in another system or when the retrieval corpus does not map cleanly to a managed Delta table.<\/p>\n

The application then owns consistency and update\/delete semantics that Delta Sync would otherwise manage automatically.<\/p>\n

ANN is the default semantic retrieval path<\/h3>\n

AI Search uses HNSW approximate nearest neighbor search with L2 distance for vector similarity.<\/p>\n

If cosine similarity semantics are required, Databricks documents normalizing embeddings so L2 ordering becomes equivalent to cosine ordering.<\/p>\n

Evaluate the embedding model, normalization, chunking, and k\/result count together rather than treating index algorithm choice as the only relevance variable.<\/p>\n

Hybrid search combines vectors and keyword signals<\/h3>\n

Hybrid queries run semantic\/vector and full-text style retrieval together and merge results using reciprocal rank fusion.<\/p>\n

This works well when user questions mix conceptual language with exact product names, identifiers, or error codes.<\/p>\n

Hybrid is often a stronger general-purpose baseline than pure ANN for enterprise RAG, but it should be measured against the actual corpus and query set.<\/p>\n

Full-text search is available as a separate query type<\/h3>\n

Current AI Search offers full-text search as a Beta capability and supports full-text-oriented indexes on appropriate endpoints.<\/p>\n

Keyword retrieval can outperform vector search for identifiers, exact terminology, SKUs, and code\/error strings.<\/p>\n

RAG systems should choose retrieval method by query behavior rather than assume vector search is always the modern or superior answer.<\/p>\n

Filters should be part of the security and relevance design<\/h3>\n

AI Search supports query filters over indexed metadata.<\/p>\n

Use filters for tenant, product, document type, date, language, access group, or other constraints that narrow the candidate universe before\/around relevance ranking.<\/p>\n

Permission metadata must be present in the indexed corpus and kept fresh; retrieval cannot enforce a security attribute that was never indexed.<\/p>\n

The built-in reranker adds a second semantic pass<\/h3>\n

Current AI Search can rerank the top candidate set using the Databricks reranker or a supported fine-tuned reranker endpoint.<\/p>\n

The reranker can consider selected text\/metadata columns and typically improves result ordering at the cost of additional latency.<\/p>\n

Reranking RAG Results<\/a> covers the retrieval-quality and latency trade-off in detail.<\/p>\n

Retrieval quality should be evaluated with DCG@10 and query-level evidence<\/h3>\n

Databricks provides AI Search retrieval-quality evaluation that compares query types with and without reranking.<\/p>\n

Current dashboards surface metrics such as DCG@10, average relevance, failed queries, and per-query performance.<\/p>\n

Use that workflow before changing production query type or reranker so the improvement is measured across representative questions, not inferred from a handful of demos.<\/p>\n

ACLs and Unity Catalog privileges govern the search resource<\/h3>\n

Users querying an index need appropriate Unity Catalog privileges on its catalog\/schema\/index plus endpoint access.<\/p>\n

AI Search endpoint ACLs and Unity Catalog ownership should be managed through groups\/service principals rather than personal accounts.<\/p>\n

Retrieval infrastructure belongs in the same data-governance model as the source tables and applications that consume it.<\/p>\n

AI Search succeeds when retrieval is operated as a quality-and-governance product<\/h3>\n

The mature design uses current AI Search terminology and SDKs, picks Delta Sync versus Direct Access deliberately, compares ANN\/hybrid\/full-text, applies security filters, evaluates reranking, tracks retrieval quality over time, and keeps Unity Catalog ownership clear.<\/p>\n

A vector database only becomes production RAG infrastructure when the team can explain why a result was indexed, allowed, retrieved, and ranked.<\/p>\n

Embedding lifecycle should be versioned. If the source index uses managed embeddings, record the embedding model\/version and vector dimension; if self-managed, record the pipeline and normalization. Changing embeddings requires coordinated re-indexing and evaluation because old and new vectors should not be mixed in one semantic space.<\/p>\n

Metadata design influences retrieval as much as embeddings. Store parent document ID, title, section, date, language, tenant\/access group, source URL, and other fields used for filtering, citation, or reranking. The search index should return enough evidence for the application to cite and authorize results without another expensive lookup.<\/p>\n

Delta Sync freshness should be monitored from source commit to searchable index. A RAG system can have an up-to-date source table but stale search results if sync is delayed or failing. Track last successful sync, rows pending, index status, and retrieval tests so freshness is part of the application SLO.<\/p>\n

Storage-optimized versus standard endpoint\/index choices should be made from corpus size and query requirements. Newer AI Search features include full-text and large-scale options that may not behave identically on every endpoint class. Check current limits and benchmark with expected index size before standardizing one type.<\/p>\n

Score thresholds can remove weak ANN results, but threshold tuning is embedding\/corpus specific. A threshold that works for one model or normalized vector set can be meaningless after migration. Tune against relevance labels and monitor how many queries return zero results; abstention may be correct, but systematic empty results usually indicate retrieval design problems.<\/p>\n

Pagination and result-count choices affect downstream context. Returning the top 50 chunks can improve reranker recall but increases retrieval latency and potentially token spend if too many are forwarded to generation. Keep first-stage candidate count, reranker candidate count, and final LLM context count as separate knobs.<\/p>\n

Index security should cover both endpoint and data source. A user might have SELECT on an index but not on the source table, depending on architecture; conversely, table access should not automatically imply search endpoint query permission. Define which principals can build, sync, query, and administer indexes through groups\/service identities.<\/p>\n

Retrieval incidents should preserve the complete query configuration: query text, query type, filters, k\/num_results, score threshold, reranker, columns, index version, and source row IDs. Without this trace, teams cannot reproduce why a particular chunk was returned or missed after the index evolves.<\/p>\n

Query routing can reduce unnecessary semantic search. Exact ID or product-code lookups may use full-text; conceptual questions can use hybrid\/ANN; broad ambiguous questions can use query rewriting plus reranking. A small router based on query characteristics can improve both quality and latency compared with forcing every request through the most expensive pipeline.<\/p>\n

Index rebuilds should use blue-green deployment for critical RAG. Build a new index from the candidate embedding\/chunking schema, run retrieval evaluation, then switch the application reference only after quality\/freshness matches expectations. Deleting or mutating the current index in place removes the easiest rollback path.<\/p>\n

Source deletions and ACL changes should be tested for propagation. Remove a sensitive document or revoke a tenant’s access and verify it disappears from AI Search within the expected window. Search freshness includes permission freshness, not only content updates.<\/p>\n

Large-index costs should include endpoint type, sync compute, embedding generation, reranking calls, and serving queries. A more sophisticated retrieval stack may reduce LLM tokens and improve task success, but economics should be evaluated end to end rather than by vector-query price alone.<\/p>\n

Search telemetry should include failed retrievals where top-1 relevance is zero or no results pass thresholds. These queries are excellent candidates for vocabulary improvements, query rewriting, new metadata, or corpus expansion and should feed the RAG evaluation backlog.<\/p>\n

Index schema evolution should use explicit migration when metadata columns used by filters\/reranking change. Add and populate the new column, rebuild or resync the candidate index, validate retrieval, then switch callers. Removing a filter column first can create a security or relevance regression even if the index remains queryable.<\/p>\n

AI Search quality should be reviewed after major corpus changes. Adding a new documentation set or tenant can alter nearest-neighbor neighborhoods and keyword statistics even without changing model\/index configuration. Re-run the benchmark when the corpus distribution changes materially.<\/p>\n","protected":false},"excerpt":{"rendered":"

Mosaic AI Vector Search is now documented as Databricks AI Search. The product still provides the same core retrieval role\u2014creating indexes over Delta\/Unity Catalog data and querying them with approximate-nearest-neighbor, hybrid, or full-text search\u2014but current 2026 documentation uses the AI Search name and a new databricks-ai-search SDK. Databricks also made reranking generally available and continues […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-19945","post","type-post","status-publish","format-standard","hentry","category-general"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t