Chunking is the step that turns source documents into the retrieval units an AI Search index can actually return. A document may contain sections, paragraphs, tables, code blocks, headings, and metadata, but a RAG system ultimately retrieves a finite set of chunks. If those chunks sever important relationships, no embedding model or reranker can fully reconstruct the meaning that was lost during ingestion.
Within Generative AI on Databricks, chunking sits before AI Search and before model prompting. The existing RAG chunking article provides the cross-platform principles; this page focuses on how those principles map into a Databricks ingestion and Delta-table design.
The goal is not one “perfect” chunk size. The goal is a chunk representation that gives retrieval enough semantic resolution without destroying the source structure the final answer needs.
Chunk boundaries should follow source meaning before token count
Fixed-size token or character chunks are easy to implement, but they often split a heading from its paragraph, a table row from its headers, or a policy exception from the rule it modifies.
Structure-aware chunking should use document hierarchy where available: title, section, subsection, paragraph, list, table, code block, or page. Token size can then be used to split unusually long structures after semantic boundaries are identified.
This approach produces chunks that are easier to retrieve and easier to cite because each unit has a coherent role inside the source.
Parent-child metadata preserves context without bloating every chunk
A small chunk can retrieve precisely but may not contain enough context for generation. One solution is to store parent identifiers such as section ID, document ID, page, heading path, or parent-chunk ID alongside each small chunk.
The retriever can identify the relevant child and then fetch a larger parent or neighboring context when the application needs more evidence.
This preserves retrieval precision while avoiding one giant chunk that mixes multiple topics.
Overlap should repair boundary risk, not duplicate the corpus
Overlap is useful when meaning crosses chunk boundaries, but excessive overlap increases index size, embedding cost, duplicate retrieval, and context redundancy.
Use overlap where sentence or paragraph boundaries alone are insufficient, especially for prose that carries definitions or conditions across adjacent passages.
Evaluate duplicate-hit rate as well as recall. If top-k results repeatedly contain several near-identical overlapping chunks from the same section, the overlap policy is probably too aggressive.
Chunk identifiers should remain stable across reprocessing
A production RAG system should be able to update one changed section without treating every chunk in the document as a completely new object. Stable document and chunk identifiers help AI Search Delta Sync indexes understand updates and help evaluation systems compare retrieval over time.
One pattern is to derive chunk IDs from source document ID + structural path + deterministic sequence rather than a random UUID on every pipeline run.
When content changes materially, the chunk can update in place while lineage remains traceable.
Metadata should support filtering and governance
Useful chunk metadata can include document owner, tenant, product, language, source timestamp, security classification, section path, page, effective date, or document type.
These fields can drive pre-filters during AI Search queries and can keep retrieval within the user’s permitted business scope before semantic ranking begins.
Authorization-sensitive filters should come from trusted application identity and Unity Catalog policy, not from a model deciding which tenant it thinks the user belongs to.
Tables and code should not be treated like ordinary prose
Tables carry meaning through row/column relationships. Code carries meaning through syntax and surrounding function or class scope. Generic prose splitting can make both nearly unusable.
For tables, consider row-level or logical-section chunks that repeat relevant headers and preserve the table/document source. For code, chunk by function, class, file, or syntactic block with enough parent context to identify dependencies.
Retrieval evaluation should include the non-prose document types users actually query.
Embedding-model limits should influence maximum chunk size
Embeddings models have context limits and may lose retrieval discrimination when one chunk contains many unrelated concepts even before that limit is reached.
Long chunks are not automatically better because the embedding compresses all included semantics into one vector. A focused chunk often produces a more useful neighborhood in vector space.
Chunk-size experiments should measure retrieval relevance and answer quality rather than only whether the embedding API accepts the text.
Delta tables should separate raw source, parsed structure, and indexed chunks
A maintainable Databricks RAG pipeline often uses separate layers: raw source files or parsed documents, normalized structural elements, and final chunk rows with IDs, text, metadata, and embedding inputs.
This separation makes re-chunking possible without re-ingesting the original corpus and allows the team to compare chunking strategies on the same source data.
It also supports audit because the application can trace one returned chunk back through its parsed structure to the original document.
Evaluation should include retrieval metrics before answer metrics
If the final answer is wrong, determine whether the retriever returned weak context before changing the generation prompt. Track recall or relevance of top-k chunks, duplicate-result rate, filter correctness, and source coverage.
The existing Databricks AI Search article reinforces this decomposition.
Later cluster topics on query rewriting and reranking can improve retrieval after chunking, but they should not be used to hide consistently poor source segmentation.
Chunking is complete when updates, retrieval, and citations all remain stable
A production chunking strategy should survive source edits, schema changes, multilingual content, new document types, and index synchronization without breaking IDs or citations.
Version the chunker, store chunk metadata, preserve source lineage, and keep evaluation sets that represent real user questions. That makes chunking an engineering artifact that can improve over time rather than an unrecorded preprocessing choice.
Chunking pipelines should be deterministic enough that the same source version produces the same chunk IDs and boundaries unless the chunker version changes. This makes index updates, evaluation comparisons, and citation stability much easier to reason about.
When the chunker does change, treat it as a retrieval-model release. Reprocess a representative corpus, rebuild or synchronize the index, compare retrieval metrics, and keep the old chunker version available long enough to understand whether quality actually improved.
Large documents should often use hierarchical retrieval. A first-stage index can find the relevant section or page, then a second step can fetch neighboring chunks or a larger parent block. This avoids sending the entire source while preserving enough local structure for grounded generation.
Metadata quality deserves its own validation. Missing tenant IDs, wrong document dates, inconsistent language tags, or incorrect section labels can cause filters and citations to fail even when the embeddings are perfect. Validate metadata before indexing rather than treating it as passive decoration.
Chunk deduplication can reduce noisy retrieval for repeated boilerplate such as legal footers, navigation text, headers, or duplicated template sections. If identical chunks appear in hundreds of documents, they can dominate nearest-neighbor results and waste context.
Evaluation should include source updates and deletions as well as static queries. A production RAG system must prove that corrected source text replaces old chunks, deleted documents disappear from search, and citations still point to the current source after reprocessing.
The strongest Databricks implementation keeps raw source, parsed structure, chunk table, embedding metadata, and AI Search index as separately inspectable stages. That separation gives engineers enough evidence to diagnose whether a retrieval problem started in parsing, chunking, metadata, embedding, sync, or query-time ranking.
Chunk size should also be evaluated against reranking and context-window behavior. A reranker may work well on concise passages but become less discriminative when each candidate contains several topics. Likewise, very small chunks may require too many candidates to provide enough evidence for generation.
Multilingual corpora need representative testing. Sentence length, tokenization, headings, and punctuation vary across languages, so a chunker tuned on English technical prose may create awkward boundaries elsewhere. Store language metadata and evaluate each important language independently.
Chunk-level access control should be inherited from source governance. If a document contains sections with different sensitivity, one document-level permission may be too coarse. In that case, the ingestion pipeline should either split the source along governance boundaries or prevent mixed-sensitivity content from entering one retrievable chunk.
Tables should retain enough provenance to reconstruct the original row and headers. A chunk that contains only values without column names can look semantically similar to many unrelated rows and can produce confident but meaningless answers.
When using managed embedding generation in AI Search, chunk text is the input to a model whose version and dimensions matter. A change to chunking or embedding model should therefore be evaluated together rather than independently, because both reshape the vector neighborhood.
Evaluation should also include latency and index-size effects. Smaller chunks create more rows, more embeddings, more index storage, and potentially more candidates to score. A chunking strategy that improves recall slightly but doubles retrieval cost may not be the best production trade-off.
Source-level deduplication and chunk-level deduplication should be kept separate. Two documents can legitimately share some language while still being distinct sources. The pipeline should remove boilerplate without erasing provenance needed for citations and compliance.
Chunk text should remain human-inspectable. If preprocessing aggressively strips headings, punctuation, code fences, or table formatting, developers may struggle to understand why one chunk retrieved. Preserve enough readable structure that retrieval debugging can be done directly from the chunk table.
