Current Claude frontier models such as Sonnet 5.5 expose a 1-million-token context window, but a large window does not remove the need for retrieval architecture. Long context is most effective when the application supplies a focused set of authoritative documents, metadata, conversation state and tools rather than dumping every available record into one request. Cost, latency, attention competition, freshness and authorization all still matter at million-token scale.
Within Claude Engineering, long-context retrieval should answer a precise question: which evidence should be in the active request, which context can be cached, which history can be summarized, and which data should be retrieved just in time?
Use the long window for coherence, not as a database
A 1M-token window can support large repositories, long reports, extensive tool context or many documents.
But persistent source data still belongs in a database, search index, object store or file system.
The model context should be a task-specific working set built from those durable stores.
Retrieve before you overflow
Even when everything technically fits, irrelevant documents increase input cost and can make the model search a larger evidence surface.
Use keyword/vector/hybrid retrieval, metadata filters and application logic to narrow the corpus.
Long context is most valuable after retrieval has already removed obviously unrelated material.
Keep authorization ahead of retrieval
The search layer should filter by tenant, user permissions and document entitlements before content reaches Claude.
Do not retrieve broadly and rely on the model to avoid mentioning unauthorized passages.
Security boundaries should be deterministic and enforced outside the model.
Chunking should preserve meaning and provenance
Retrieve sections that are large enough to preserve the local argument, table, code function or policy rule, while retaining stable source IDs and offsets.
Claude should be able to cite or trace an answer back to exact source material.
Claude Citations for Enterprise Search provides the evidence-linking pattern.
Context ordering still matters
Place system instructions and task framing predictably, then the most relevant evidence and required metadata.
Separate documents clearly with titles, IDs and delimiters.
Do not intermix retrieved source text with application instructions in ways that make prompt injection harder to identify.
Prompt caching changes long-context economics
If large reference content remains stable across many requests, prompt caching can make repeated reads far cheaper than reprocessing the same input at full price.
Cache stable system/tool/document prefixes and keep volatile user-specific data outside the cached region.
Measure actual cache-hit rate before relying on the savings.
Context editing helps long agent sessions
Current Claude context-management guidance supports context editing to remove stale tool-result blocks once they are no longer useful.
This is different from retrieval: retrieval brings relevant information in; context editing removes dead intermediate state.
Claude Context Editing should be part of any agent that accumulates hundreds of tool results.
Tool search can reduce schema context
Large agent toolsets can consume significant tokens even before documents are added.
Anthropic recommends tool search for large tool inventories so only relevant tool definitions are loaded on demand.
This leaves more of the context window for the actual evidence and conversation.
Token counting should be an admission check
Use the token-counting endpoint to estimate the final request, including messages, tools and documents.
Set application-level context budgets below the model maximum so the system still has output headroom and predictable cost.
If retrieval returns too much, rerank, summarize or ask a clarifying question before sending.
Long-context evals should test distractors
Evaluation should include cases where the correct evidence is buried among similar-but-wrong documents, stale versions, contradictory sources or malicious instructions inside retrieved text.
Grade whether Claude selects the current authoritative source and cites it correctly.
Long-window capacity is not useful if relevance collapses as the corpus grows.
Long-context retrieval succeeds when context size follows evidence quality
The mature application uses deterministic authorization, metadata-aware retrieval, provenance-rich chunks, caching, token budgets, tool search, context editing and long-context regression tests.
A million-token window is valuable because it gives the system room for complex evidence—not because it makes retrieval, data governance or context engineering optional.
Retrieval freshness should be visible to Claude and the application. Include document version, effective date, last-updated timestamp, and authoritative-source priority where relevant. If two retrieved policies conflict, the model needs enough metadata to prefer the current approved version instead of whichever chunk appears first.
Query rewriting can improve recall but should preserve user intent. A retrieval layer may expand acronyms, add product names, or split a compound question into subqueries. Log the rewritten query and compare it with the original during evaluation so relevance improvements do not come from silently changing the question.
Reranking should be separate from first-stage retrieval. Retrieve a broader candidate set cheaply, then use semantic or learned ranking to choose the small set of passages sent to Claude. This reduces context size while maintaining recall. Measure answer quality and retrieval sufficiency rather than assuming top-k similarity scores equal useful evidence.
Long documents need hierarchical retrieval. First identify the relevant document or chapter, then retrieve the local sections around the match. This preserves global context such as definitions and exceptions without injecting hundreds of irrelevant pages. Parent-child metadata is useful for reconstructing the surrounding section after a fine-grained chunk match.
Contradictory evidence should be surfaced, not hidden. If two authoritative documents disagree, include both with provenance and instruct the workflow to escalate or explain the discrepancy. Retrieval systems that select only one convenient answer can create false certainty even when the corpus itself is ambiguous.
Prompt injection defense should treat retrieved text as untrusted data. Separate system/developer instructions from source passages clearly, and avoid letting document text redefine tool permissions or output policy. Security-sensitive agents should scan or classify retrieved instructions and keep deterministic authorization outside Claude.
Cache strategy should follow document volatility. A stable technical manual can be cached for many requests; a price list or incident dashboard may be stale minutes later. Attach version/ETag or last-modified information to cached context and invalidate it when the source changes rather than relying only on TTL guesses.
Context compression should preserve citations. If the system summarizes multiple chunks before sending them to Claude, retain source IDs and map summary statements back to originals. Otherwise a concise context can improve token cost while making audit and citation impossible.
Retrieval observability should include candidate count, filters, reranker scores, selected chunks, document versions, token contribution, cache hits, and final citations. These traces let engineers tell whether a wrong answer came from bad search, bad ranking, stale data, or model reasoning.
Long-context design should include a no-answer path. When retrieval finds insufficient authoritative evidence, Claude should ask for clarification or state the limitation rather than filling the 1M-token window with loosely related content. More context is not a substitute for relevant context.
Retrieval should include a version-selection policy. If a repository contains v1, v2 and archived copies of the same policy, metadata filters should prefer the active version before semantic ranking. Otherwise a highly similar obsolete document can outrank the current authoritative one simply because its wording is closer to the user’s query.
Document-level access control should survive chunking. Every chunk should inherit tenant, ACL and sensitivity metadata from its parent document so filters cannot accidentally expose one paragraph detached from the source’s permissions. Reindexing pipelines should test that inherited metadata remains complete after schema changes.
Long context should be cost-tested at realistic scale. One impressive demo with a 700k-token prompt may be acceptable, while thousands of such requests can dominate spend and throughput. Measure quality gain versus a smaller retrieved context and use the long window only where the additional evidence materially improves success.
Conversation history and document retrieval should have separate budgets. A long-running assistant can consume most of the context with past turns before retrieval adds source material. Reserve token headroom for current evidence and output, and summarize old conversation state before it competes with the documents needed to answer today’s question.
Retrieval incident reviews should ask whether the source existed, whether it was indexed, whether filters allowed it, whether it ranked high enough, whether the model used it, and whether the answer cited it. This sequence isolates search-pipeline failures from reasoning failures much faster than reviewing the final response alone.
Long-context pipelines should keep a separate retrieval benchmark from end-answer evaluation. Measure recall@k, relevance, freshness and authorization before the model sees the passages. If retrieval fails to return the right evidence, changing the Claude prompt cannot repair the missing source reliably.
Document ingestion should preserve structural elements such as headings, tables, code blocks and page/section numbers. Flattening everything into one plain-text stream can destroy relationships that matter for interpretation. Retrieval chunks should carry enough structure that Claude can understand whether a value came from a table row, note or normative policy section.
Source authority should be encoded as metadata. A signed policy, official runbook and informal chat message may all mention the same topic but should not have equal rank. Retrieval can use authority tiers so a lower-quality note does not outrank current controlled documentation because its wording is more similar.
Use Prompt Management at Application Scale to keep retrieval instructions versioned with the rest of the application. Query construction, reranking prompts and answer-grounding rules are production prompts and should move through the same review/change process.
Storage/index changes should trigger regression tests. Rechunking documents, changing embeddings, rebuilding metadata filters or switching a search backend can change the evidence Claude sees without any model or prompt change. Treat retrieval infrastructure as part of the released AI system.
