Category Archives: Data & Analytics
When preparing for Databricks GenAI Engineer Associate, retrieval-augmented generation often fails before the vector index has a chance to help. A user asks a conversational, ambiguous, multi-part, or context-dependent question, while the retrieval system expects a concise search query that exposes the entities and concepts most likely to appear in source documents. Query rewriting is […]
For readers studying Databricks GenAI Engineer Associate, retrieval systems usually produce more candidates than an application should place into a model prompt. The first-stage search is optimized to find plausible matches quickly; reranking applies a more expensive relevance judgment to a smaller candidate set and changes their order before context assembly. Databricks AI Search currently […]
For readers studying Databricks GenAI Engineer Associate, the phrase “Mosaic AI Agent Framework” still appears in older Databricks tutorials and architecture discussions, but the current agent platform is broader and more flexible. Databricks now emphasizes building agents with standard frameworks such as OpenAI Agents SDK, LangGraph, LangChain, LlamaIndex, or custom Python, then integrating those agents […]
For Databricks GenAI Engineer Associate candidates, Mosaic AI Model Serving remains an important Databricks inference surface, but the surrounding platform has evolved. Current Databricks guidance distinguishes custom model serving, Foundation Model APIs, external models, and newer governed model-access patterns such as Unity Gateway. Teams should therefore start with the workload they need to serve rather […]
When preparing for Databricks GenAI Engineer Associate, Databricks renamed Vector Search to Databricks AI Search in 2026. The queued term “Mosaic AI Vector Search” still describes the technology many teams built against, but current documentation uses AI Search for the managed retrieval service. The rename also reflects a broader product direction: the service is not […]
For readers studying Databricks GenAI Engineer Associate, real-time AI systems often need information that changes faster than a model can be retrained. A fraud model needs the customer’s latest transaction count, a recommendation service needs recent engagement, and a personalization layer may need the user’s current account state. An online feature store exists to make […]
For Databricks Data Engineer Associate candidates, streaming tables in Databricks provide a declarative way to maintain incrementally updated datasets from continuously arriving data. In the current Lakeflow Spark Declarative Pipelines model, a developer defines the target table and its transformation while the pipeline manages execution, dependencies, checkpointing, retries, and incremental progress. The result is different […]
When preparing for Databricks Data Engineer Associate, Unity Catalog column masks let Databricks return a transformed value instead of the stored value when a protected column is queried. The transformation is implemented through a user-defined function that can evaluate the current user, group membership, tags, or other policy logic and then decide whether to reveal, […]
For readers studying Databricks Data Engineer Associate, Unity Catalog row filters let Databricks apply a policy function to each row and return only the rows a user is authorized to see. The stored table can remain shared while the visible row set changes according to identity, group membership, tags, or other policy inputs. This is […]
Databricks Clean Rooms are designed for organizations that need to analyze or combine data across organizational boundaries without giving each participant unrestricted access to the other participants’ raw datasets. The clean room creates a controlled collaboration environment in which approved data, notebooks, and computations can be made available under rules that all parties understand. The […]
Databricks administrators have long used cluster policies to put guardrails around how users create Spark clusters. In the current product interface and documentation, the preferred name is compute policies. The older “cluster policy” label still appears in API and CLI surfaces, so both terms matter operationally: the concept has evolved, but existing automation does not […]
A Databricks compute policy becomes valuable when it expresses a workload decision that the organization has already made. The policy should answer questions such as: which runtime families are supportable, which access modes are allowed, how large may a cluster become, which tags are mandatory, which libraries are trusted, and how much hourly cost can […]
For Databricks Data Engineer Professional candidates, deletion vectors change the cost model of row-level modification in Delta Lake. Without them, changing or deleting even a small number of rows can require rewriting the Parquet files that contain those rows. With deletion vectors enabled, Databricks can record which rows are logically removed or replaced in metadata […]
When preparing for Databricks Data Engineer Professional, optimized writes address a simple but expensive problem in distributed data engineering: a parallel job can produce far more files than downstream readers want. Every task may write into multiple partitions, and a large shuffle can leave a table with thousands of small objects. The data is correct, […]
VACUUM is one of the most consequential maintenance operations in Delta Lake because it turns logical history into physical deletion. A Delta table can stop referencing old data files long before those files disappear from object storage. That delay is intentional: it supports time travel, rollback, long-running readers, and recovery from recent mistakes. VACUUM removes […]
