Microsoft AI-103: Foundry Evaluation Datasets
Microsoft Foundry evaluation datasets provide a reusable test asset for measuring model or agent quality across prompt, model, and agent versions. Current Foundry guidance describes JSONL as the common reusable format and supports turn-level and conversation-level evaluation, uploaded/versioned datasets, inline data, synthetic generation, trace-derived datasets, and evaluation directly from stored responses or traces. For Microsoft AI agents, the important design choice is deciding when a stable dataset is the right evidence rather than collecting examples without a release purpose.
A dataset is most valuable when the team intends to rerun the same cases. That makes it ideal for regression gates and comparisons. LLM evaluation release gates should therefore treat the dataset as versioned source material: every case has a reason for existing, expected evidence or outcome, slice labels, and provenance. Random production logs copied into a JSONL file are not automatically a good evaluation set.
Not every Foundry evaluation requires a standalone dataset. Existing agent responses, Application Insights traces, or response identifiers can sometimes provide relevant evaluation inputs where they already live. A curated dataset becomes valuable when a team needs repeatability, human review, controlled model comparisons, simulation, or regression checks across releases.
Define the decision before collecting cases
Start with the release question: Are we deciding whether a prompt can ship, whether a new model is safe to adopt, whether retrieval improved, whether an agent’s tool plan regressed, or whether a safety control works? The cases, fields, and evaluators should be chosen to answer that question. Without a decision, datasets tend to become large archives that are expensive to run and difficult to interpret.
Write acceptance criteria before the first run. For example: no regression in task completion on the stable core, less than a specified unsafe-tool rate, improved groundedness on retrieval failures, or latency below a product target. This prevents teams from changing thresholds after seeing results.
Each evaluation case should map to the system behavior it exercises so prompt orchestration and evaluation can distinguish prompt failures from retrieval, permission, tool, and orchestration failures. A prompt-only test cannot prove a tool workflow is correct when the production failure occurs in another layer.
Assign severity to failures before scoring. A minor style regression and an unauthorized action should not have equal weight simply because both are one failed row. Release policy can require zero critical failures while tolerating a small number of low-severity misses, making the gate reflect product risk rather than raw accuracy alone.
Build a stable core and targeted slices
The stable core contains representative cases that should remain comparable across releases. Slices capture known risk areas: long conversations, ambiguous intents, missing evidence, tool failures, policy-sensitive requests, multilingual inputs, or difficult domain categories. Slice labels let a team see where a score changed instead of hiding a severe regression inside a strong average.
Balance frequent traffic with high-consequence edge cases. Production frequency alone may underrepresent rare but expensive failures. A financial approval error or data-leak path deserves explicit coverage even if it is uncommon. Conversely, a suite made only of pathological edge cases can optimize the system away from its normal workload.
Keep near-duplicates under control. Ten superficial paraphrases of one case can overweight a behavior and create false confidence. Add variations when they test a distinct linguistic or contextual failure, not just to increase row count.
Track slice coverage as the product evolves. When a new tool, language, customer segment, or policy boundary is introduced, decide whether the stable suite needs a new slice before launch. Coverage should follow new risk, not wait for the first incident. A lightweight coverage matrix keeps evaluation work connected to product scope. Review slice balance periodically so new high-volume cases do not crowd rare critical failures out of the suite.
Use the dataset schema to preserve conversation meaning
Foundry evaluation datasets can represent interactions through a `messages` field and support separate `query`/`response` forms depending on scenario. Choose the shape that preserves the behavior being evaluated. Multi-turn agents need enough conversation history to reproduce the decision point; isolated model extraction may need only one query and a reference field.
Turn-level and conversation-level evaluation answer different questions. A correct final answer can hide a poor intermediate response, while a single imperfect turn may be acceptable if the conversation recovers safely. Select the evaluation level that matches the product experience and ensure the chosen evaluator supports it.
Reflection and self-critique loops should be tested as bounded software workflows, not accepted simply because a model declares that its own answer is correct. For a grounded response, an application can generate a draft, compare its claims with the retrieved evidence, run explicit checks for missing conditions or unsafe tool requests, and request one controlled revision. A self-reviewing model may repeat the same mistake, so independent reference checks, deterministic policy rules, evaluation rubrics, and human oversight still determine whether the result passes. Set a maximum number of revisions and define a safe abstention or escalation path when the checks keep failing.
The AI-103 objective also names chain-of-thought evaluations. For operational agent evaluation, focus on observable process evidence—tool selection, argument validation, approval decisions, intermediate outcomes, citations, and final task success—rather than assuming access to a model’s private internal reasoning. Microsoft Foundry distinguishes system-level evaluation of an outcome from process-level evaluation of the steps used to reach it. A practical test can compare the original answer, its supporting sources, a proposed tool call, a revised answer, and independently scored results against a known test case. Do not interpret a fluent rationale or the model’s self-rating as proof that its hidden reasoning was correct.
Document custom columns. Ground truth, expected tool, policy label, severity, source ID, or slice name can make the dataset far more useful, but only if their meaning is stable. A data dictionary prevents different evaluators or analysts from interpreting the same column differently.
Avoid leaking evaluator hints into the model input. Reference answers, expected tool names, severity labels, and judge notes are useful dataset fields, but they should be mapped only to evaluators unless the application would genuinely see them. An evaluation can look excellent if the target is accidentally given the answer key.
Choose curated, synthetic, and production-derived cases deliberately
Curated cases are best for known requirements and high-risk boundaries. Synthetic generation can expand coverage before launch or explore combinations that are tedious to author. Production traces reveal real user distributions and surprising failures. A strong suite uses all three without pretending they have equal authority.
Review synthetic cases before promoting them to the stable suite. Generated test data can encode unrealistic phrasing, incorrect facts, or model-specific assumptions. Production-derived cases should be sanitized, privacy-reviewed, and deduplicated before reuse. Preserve provenance so the team knows why a test exists.
When traces or user content become evaluation material, private-data governance determines where the dataset can live, who may access it, and how long it should be retained. The evaluation environment and retention policy may differ from the production path. Do not create a long-lived sensitive dataset simply because the evaluation SDK makes upload convenient.
For trace-derived data, sample intentionally. Uniform random sampling can miss rare failure modes; selecting only incidents can overstate difficulty. Maintain separate representative and risk-focused sets so the team can evaluate both everyday quality and high-consequence boundaries without confusing their distributions.
Map evaluator inputs explicitly and prefer deterministic checks where possible
A trustworthy evaluation dataset combines LLM judges for qualities such as groundedness with deterministic checks for schema and policy invariants. Use model-based judges where interpretation is genuinely needed.
Foundry evaluators can require specific columns. Make field mappings explicit so a renamed dataset column does not silently feed the wrong content to an evaluator. Validate a small sample and inspect individual results before trusting aggregate scores.
Store evaluator version/configuration with the run. If the dataset stays fixed but the scoring model, prompt, or evaluator version changes, scores are not directly comparable without that context.
Calibrate model-based judges against human review on a sample. If judges disagree frequently with domain experts, tune the rubric or use a different measure before building a release threshold on top of it. Record judge agreement as part of evaluator validation; an unstable judge should not decide whether another model ships.
Turn evaluation results into a release gate and a debugging tool
A release report should show overall scores, slice scores, failed cases, severity, latency/cost if measured, and comparison with the current production baseline. Engineers need the individual failures to fix the system; approvers need a concise statement of whether agreed thresholds passed. Both should come from the same run record.
Generative AI evaluation pipelines become actionable only when a failed case can be traced to prompt interpretation, retrieval, tool selection, tool result, model behavior, or evaluator noise. Add or refine tests only after understanding the failure; otherwise the suite grows without becoming more diagnostic.
Production incidents should feed the dataset when they represent a reusable failure class. A real failure that cannot be reproduced offline remains a future regression risk.
Set policies for flaky cases. If a stochastic task passes only intermittently, record repeated-run behavior rather than deleting the case or accepting a single favorable result. Some scenarios need a pass-rate threshold across several runs; others should be redesigned to use a deterministic assertion. The gate should make variability visible.
Version datasets as durable engineering assets
Microsoft Foundry evaluation datasets need version control, clear owners, reproducible splits, and release gates just like application code. Record added/removed cases, rationale, expected outcomes, privacy review where needed, and the baseline that changed. Avoid editing historical versions in place, because that destroys the evidence behind an earlier release decision.
Keep the stable suite small enough to run frequently and maintain, then add broader scheduled suites for expensive or low-frequency scenarios. Fast feedback is part of quality: a gate that takes so long everyone bypasses it is not effective governance.
A good Foundry evaluation dataset is therefore not a benchmark trophy. It is a versioned contract between product expectations and model behavior, designed to be rerun, inspected, and connected directly to release decisions.
Archive the mapping from dataset version to releases that used it. When an incident occurs months later, the team should be able to reconstruct which cases and evaluator configuration approved the affected model. That historical linkage is essential for improving the gate instead of merely fixing the immediate prompt.