Chunking is often described as dividing text into pieces that fit an embedding model or retrieval pipeline. That description is technically true and operationally incomplete. Documents carry structure: headings define scope, tables bind values to columns, lists express grouping, captions qualify figures, and paragraphs inherit meaning from the sections around them. A retrieval system that ignores those relationships can produce chunks that are small enough to embed but too damaged to answer questions accurately.
For AWS implementations, AIP-C01 includes choices about how source material becomes useful grounding data. In Amazon Bedrock Knowledge Bases, those choices include fixed-size, hierarchical, or semantic chunking, depending on the data source and configuration. The parallel Azure work within AI-103 focuses on retrieval and grounding pipelines, including extraction, indexing, layout, and vector search. Neither path is well served by a universal chunk length. A troubleshooting runbook with ordered steps, a pricing table with row and column context, and a contract with section-level qualifications require different retrieval units. Engineers must preserve those relationships while testing whether users actually retrieve the right evidence.
Document structure is retrieval metadata even when it is not embedded
A section title such as “Cancellation,” “Network Exceptions,” or “Key Rotation” changes how the paragraphs beneath it should be interpreted. If the heading is discarded during extraction, two otherwise similar paragraphs can become indistinguishable. Preserving a hierarchy such as document title, chapter, section, and subsection gives the retrieval layer additional evidence about meaning and gives the generation layer context for citing or qualifying an answer.
The structure does not always have to be copied verbatim into every chunk. Some pipelines prepend a compact hierarchy to the text before embedding; others store hierarchy as filterable metadata while keeping the chunk body clean. RAG chunking succeeds when those semantic boundaries improve retrieval precision without stripping away qualifiers, section identity, or parent context that the answer depends on.
Headings create natural boundaries, but paragraphs still need judgment
Splitting on headings is a strong starting point because headings usually indicate topic changes. It is not sufficient by itself. A section can contain one sentence or twenty pages. A long section may need additional paragraph- or sentence-aware splitting, while several tiny sibling sections may be better represented together if users normally ask about them as a group.
Chunking logic can use a hierarchy of preferred boundaries: document or page landmarks first, then heading boundaries, then paragraphs, then sentences when a segment remains too large. Hard token cuts should be the fallback rather than the first choice. This reduces cases where a definition lands in one chunk and its exception or qualifier lands in another.
Overlap also becomes more useful when it follows semantics. Repeating the final sentence of one arbitrary token window may add little. Carrying a section heading, a short parent summary, or the immediately preceding paragraph can preserve the relationship that the next chunk depends on.
Tables should not be flattened into unrelated lines
Tables are one of the easiest document structures to corrupt. A row value is meaningful because it is associated with a header and often with a row key. Converting a table into a stream of cell text can destroy those associations. A query for a limit, price, model, region, or entitlement may retrieve the number but lose the column that explains what the number represents.
Useful strategies include serializing each row with its column names, retaining the table title and nearby explanatory paragraph, keeping small tables intact, or generating a normalized text representation specifically for retrieval. Large tables may require row groups rather than one enormous chunk. The choice should be driven by the questions users ask and by whether a retrieved unit can stand on its own without reconstructing the original page layout.
Lists, procedures, and code blocks need coherent units
Numbered procedures encode order. Bullet lists encode membership. Code blocks depend on surrounding explanation and sometimes on preceding imports, variable definitions, or configuration. Splitting these structures blindly can return step 4 without steps 1–3, a list item without its category, or a command without the warning that limits where it is safe to run.
A document-aware parser can keep a short procedure together or store the procedure title and step range with each chunk. For very long procedures, parent-child retrieval is useful: retrieve a concise child segment for matching, then supply the larger parent procedure to the model. The same pattern works for policy clauses and technical manuals where precision in matching and completeness in generation require different unit sizes.
Parent-child retrieval separates matching granularity from answer context
Small chunks often improve recall for specific facts because unrelated text does not dilute the embedding. Small chunks can also lack the context needed to interpret the match. Parent-child designs address that tension by embedding or indexing smaller children while retaining an identifier for a larger parent section. Search finds the child; generation receives the parent, selected neighboring material, or both.
Enterprise RAG chunking often needs child chunks for precise ranking and larger parent units for coherent reading because source documents are inconsistent and business answers may depend on qualifiers several paragraphs away. That pattern adds operational requirements: parent boundaries must be stable, permissions must apply consistently to child and parent records, and the retrieval trace should identify which child caused a parent to enter context.
In Amazon Bedrock Knowledge Bases, hierarchical chunking can first retrieve smaller child units and then return the associated parent units for more complete context. The configuration exposes parent size, child size, and overlap; returned parent replacement can produce fewer results than the originally requested number. This makes hierarchical retrieval useful for detailed policy clauses with broader conditions, but it is not a cure for tables that were corrupted before ingestion. A data source that cannot retain row labels or section references should be repaired or preprocessed before embeddings are created, regardless of the chunking option chosen.
Azure AI Search offers different extraction paths. The Document Layout skill analyzes page and heading structure with Document Intelligence, while the Text Split skill works from extracted text and configured boundaries. Newer Azure Content Understanding-based pipelines can preserve richer cross-page structure and semantic chunks for document-heavy retrieval. Those approaches are not interchangeable: choose according to document formats, page context, image and table handling, cost, and the fields that the downstream index can retain. In either case, verify that a chunk can be traced back to a stable document version and the exact section or page on which the answer depends.
Metadata should carry identity, scope, and update semantics
Every chunk needs more than text and a vector. Useful metadata commonly includes source identifier, document version, section path, page or location, content type, language, access-control attributes, effective date, and extraction timestamp. Those fields support filtering, citation, lifecycle management, and debugging.
Versioning is particularly important. If a policy is replaced, stale chunks should not remain retrievable simply because their embeddings are still close to a query. Stable source IDs and explicit version metadata make it possible to invalidate an old set and re-index the replacement without losing traceability. In regulated or frequently updated corpora, retrieval correctness depends as much on this lifecycle discipline as on embedding quality.
OCR and layout extraction errors can dominate chunking quality
Scanned PDFs and complex layouts can create text in the wrong reading order, detach captions, merge columns, or omit characters before the chunker sees anything. No chunk-size tuning can recover structure that extraction already destroyed. Teams should therefore inspect extraction output from representative documents before optimizing embeddings or search parameters.
The agentic AI engineering pipeline should preserve provenance across parsing, chunking, embedding, retrieval, reranking, and generation. When an answer is wrong, that trace lets engineers distinguish a malformed source from a weak chunk boundary, bad embedding, retrieval miss, ranking error, or model misuse. Treating the index as a black box removes the evidence needed to isolate those failure layers.
Chunk size should be measured against retrieval tasks
A single global size is attractive because it is easy to configure. It can also hide poor performance across document families. API references, contracts, runbooks, research papers, FAQs, and product catalogs have different structural units. The correct question is whether the chosen segmentation preserves the facts and relationships needed by real queries while keeping retrieval selective enough to avoid flooding the context window.
Evaluation should include queries that target section-specific facts, cross-paragraph qualifications, table values, procedures, and ambiguous terminology. Measure whether the relevant evidence appears in the retrieved set, where it ranks, whether its structural context survives, and whether irrelevant neighboring text displaces better evidence. A chunking change is successful only when those retrieval and answer outcomes improve.
Mixed-media documents may need more than one retrieval representation
A slide deck can contain short text labels whose real meaning lives in a diagram. A manual can combine prose, tables, screenshots, callouts, and code. A financial report may repeat the same metric in narrative text and in a table with different precision. Flattening every element into one textual stream can either lose meaning or create duplicate evidence that dominates retrieval.
Teams can preserve element type and create separate representations where necessary: text for semantic search, table rows with headers, captions tied to figures, or concise descriptions of visual elements. The generation layer can then receive the representation appropriate to the question while retaining a reference to the original page. The objective is not to embed every artifact in every possible form; it is to avoid pretending that a layout-rich source is ordinary prose when its structure carries information.
Document-aware chunking is an information-model decision
The most reliable pipeline starts with the document’s information model rather than with an arbitrary number of tokens. It identifies meaningful boundaries, preserves hierarchy, treats tables and procedures deliberately, attaches metadata, and keeps a path back to the source. Token limits still matter, but they become constraints applied after the content structure is understood.
Structure-aware splitting also improves citation behavior. When a chunk retains its section path and source location, an answer can point readers to the part of the document that supports the claim instead of citing a whole file ambiguously. This is especially valuable when several sections reuse the same terminology but apply it to different products, jurisdictions, or versions.
That approach also makes later optimization safer. Embedding models can change, rerankers can be added, and context budgets can move without forcing the organization to rediscover what each source document meant. Good chunking converts a document into retrievable units while preserving the relationships that made the original useful in the first place.
