Retrieval-augmented generation is often summarized as “search first, then ask the model,” but that shortcut hides the decisions that determine whether the system is useful. A production RAG application has to ingest source material, segment it, represent it for retrieval, enforce access rules, interpret a user query, retrieve candidate evidence, rank that evidence, fit it into a context budget, and ask a model to produce an answer that remains faithful to what was found. A failure at any stage can appear to the user as a model problem.
The current AI-103 blueprint explicitly includes choosing retrieval and indexing methods and implementing retrieval-augmented generation in applications built with Microsoft Foundry. Microsoft now describes both classic RAG and newer agentic retrieval patterns, with Azure AI Search serving as a common retrieval layer. The durable skill is not memorizing a wizard; it is understanding which stage is responsible for which kind of error.
A practical mental model is a pipeline with feedback. Content moves forward from source to index to retrieved context to generated answer, while evaluation moves backward from bad answers to the stage that most likely introduced the defect. That model keeps troubleshooting causal and helps architects compare design options without treating every RAG system as the same.
The source corpus sets the ceiling on answer quality
RAG cannot ground an answer in information that is missing, outdated, contradictory, or inaccessible. Before thinking about embeddings, identify authoritative sources, freshness expectations, document ownership, and the audience allowed to see each source. A polished index built from weak source material simply makes weak information easier to retrieve.
Data lifecycle also matters. Documents may be replaced, revoked, reclassified, or corrected. The ingestion process needs a way to remove stale chunks and preserve metadata that controls filtering and access. The broader secure data lifecycle concept is relevant because retrieval systems create another copy or representation of enterprise content that must follow retention and governance rules.
Chunking is an information-design decision, not a magic number
Chunks must be small enough to match focused queries but large enough to preserve the relationships needed for an answer. A policy paragraph may be self-contained; a troubleshooting procedure may depend on ordered steps; a table may lose meaning when split from its headings. Fixed-size chunking is convenient, but document structure often deserves more influence than character count.
Good chunk design preserves useful boundaries and metadata. Section title, document type, product, region, effective date, and security label can all improve filtering or ranking. Teams should test chunking against real questions and inspect the actual passages returned. If retrieval repeatedly returns half an answer, the index structure is telling you something about how the source should be represented.
Retrieval has several levers before the model sees anything
Keyword search is strong when users name exact terms. Vector search is useful when wording differs but meaning is similar. Hybrid search can combine both signals, and semantic ranking can reorder candidates. Newer agentic retrieval patterns can decompose a complex request into multiple focused searches before returning grounding data. Each option adds capability and, potentially, latency, cost, and operational complexity.
The decision should follow query shape. A narrow support corpus with stable terminology may work well with a simple hybrid pattern. A research-style assistant handling conversational questions may benefit from richer query planning. The goal is not maximum sophistication; it is reliable evidence retrieval for the kinds of questions users actually ask.
Access control must survive the journey into the index
A user who cannot open a document in the source system should not gain its contents because the document was indexed for an agent. Security trimming therefore belongs in the retrieval design. That may involve identity-aware filters, metadata, separate indexes, permission-aware knowledge layers, or architectural isolation depending on the platform and risk.
The most dangerous failure is silent over-retrieval: the answer looks accurate, but it was grounded in content the user was never entitled to see. The Microsoft Entra identity model provides the right question: whose identity is being evaluated at retrieval time, and which authorization decision is actually enforced?
Generation should be constrained by retrieved evidence
Once passages are retrieved, the prompt should make their role explicit. The model needs instructions about how to use sources, how to handle conflicts, when to cite, and what to do when evidence is insufficient. A RAG system should be allowed to say that the available material does not support an answer. Forcing completion turns missing evidence into confident fabrication.
The underlying foundation-model behavior still matters: the model predicts plausible output from its context and learned parameters. Grounding changes the context; it does not convert the model into a database. Evaluation should therefore distinguish retrieval failure from generation that ignores or distorts good evidence.
Classic RAG and agentic retrieval solve different complexity levels
Classic RAG gives the application direct control over query construction, search calls, ranking, and prompt composition. That can be an advantage when latency must be tightly controlled or the team already has a mature retrieval pipeline. Agentic retrieval adds model-assisted query planning that can break complex questions into multiple searches and produce structured grounding results.
Agentic retrieval is appealing for ambiguous, multi-part questions, but it introduces more moving pieces and potentially more model work per request. Architects should ask whether the extra reasoning improves answer quality enough to justify the additional latency, cost, and observability requirements. A complex retrieval path should earn its place through measurable gains.
Cost and latency accumulate across the whole RAG transaction
A RAG request may involve query rewriting, embeddings, several search operations, semantic ranking, reranking, and a generation call with a larger prompt because retrieved passages are included. Each stage consumes time and resources. Index size, search tier, number of passages, context length, and orchestration strategy all influence the final user experience.
Measure the latency distribution, not just the average. A system that usually responds in two seconds but occasionally takes twenty can be difficult to use in an interactive workflow. Cost should likewise be calculated per useful completed answer. Removing low-value passages or reducing unnecessary retrieval steps can improve both economics and accuracy by giving the model a cleaner context.
Evaluate retrieval and generation separately before evaluating the whole system
Create a set of representative questions with known evidence. First measure whether the correct passages appear in the retrieved set. Then evaluate whether the model uses those passages faithfully. Finally evaluate the end-to-end answer for usefulness, citations, safety, and task completion. This layered method turns “the RAG is bad” into an actionable diagnosis.
The operational relationship with AI-300 becomes important here because evaluation and monitoring continue after deployment. Query patterns drift, source content changes, and models evolve. Production telemetry should reveal falling retrieval quality, rising no-answer rates, citation failures, latency changes, and unusual access patterns before users have to report them.
The best RAG design keeps evidence inspectable
Imagine an internal engineering assistant answering a question about a newly changed deployment policy. The source system contains the current policy and several obsolete versions. A sound RAG design marks the current document, filters by effective date and user access, retrieves the relevant section, and returns an answer with evidence. A weak design indexes everything equally and lets semantic similarity choose among conflicting versions.
That scenario captures the durable mental model: source quality, representation, retrieval, authorization, context, generation, and evaluation are separate responsibilities. When an answer is wrong, inspect the chain in that order. When designing a new system, make each stage observable enough that the team can prove what happened. That is more useful than memorizing any single RAG implementation recipe.
Freshness is another retrieval variable that deserves an explicit objective. Some corpora change monthly; others change minute by minute. If users expect near-real-time answers, a nightly indexing job creates a correctness problem even when retrieval relevance is excellent. Track source-to-index lag and make it visible to the application when necessary. In some workflows, telling the user that the index is several hours old is better than presenting an outdated answer with false confidence.
Citation design should be tested as part of usefulness rather than added cosmetically. A citation should lead to the passage that actually supports the claim, not merely to the correct document. If the source is long, deep links, page references, section names, or quoted evidence can reduce verification effort. The goal is to let a reader audit important answers without reconstructing the retrieval pipeline themselves.
RAG systems also need a policy for conflicting sources. Search may return two authoritative-looking documents with different effective dates or business owners. The application can rank by recency or metadata, but some conflicts require explicit presentation to the user or escalation to a domain owner. Hiding disagreement behind one synthesized answer can create false certainty. Preserving source metadata and conflict signals makes the system more honest and easier to govern.
The application should also distinguish retrieval confidence from answer confidence. A model may sound certain even when the search results are weak. Exposing retrieval signals to orchestration can trigger a broader search, a clarifying question, or a safe no-answer path before generation. This creates a control point between evidence gathering and language generation instead of letting fluent prose hide poor evidence.
