Evaluation datasets give Microsoft Foundry teams a stable set of cases that can be rerun when a model, prompt, agent, tool, or orchestration policy changes. The important word is stable. Without a reusable test set, teams can demonstrate that a system works on today’s hand-picked examples but cannot tell whether tomorrow’s version quietly breaks a scenario that mattered yesterday. A dataset turns evaluation from an occasional demo into an engineering control.<\/p>\n
Within a broader Microsoft AI agent program<\/a>, evaluation data should represent the work the system is expected to perform, the risks it must avoid, and the edge cases that are likely to expose weak behavior. Microsoft Foundry supports reusable datasets, direct evaluation of existing responses, trace-based evaluation, and generated test cases, so a dataset is not mandatory for every evaluation. For Microsoft AI-103<\/a>, the architectural skill is knowing when a versioned test set creates repeatability and when live interaction or trace data is the better source of evidence.<\/p>\n Microsoft describes an evaluation dataset as a reusable collection of test cases, commonly represented as JSONL with one JSON object per line. That format matters less than the engineering property it enables: the same cases can be run against different versions. A prompt revision, new model deployment, agent instruction change, or tool update can be evaluated against the same baseline rather than judged from a fresh set of examples each time.<\/p>\n This is the foundation of LLM regression testing<\/a>. A release candidate should not merely look better on new examples; it should preserve behavior on known requirements unless a change is intentional. The evaluation set becomes an executable record of those requirements. When a failure is discovered in production, add a representative case after the root cause is understood so future versions have to prove they do not reintroduce the same weakness.<\/p>\n A dataset is especially useful for CI\/CD gates because it produces comparable runs. The gate does not need to require every metric to improve. It can enforce minimum thresholds, block large regressions, or require review when sensitive scenarios change. This keeps evaluation aligned with release risk instead of treating one aggregate score as an absolute quality measure.<\/p>\n Foundry can evaluate individual turns or whole conversations, and that distinction should be chosen before the dataset is built. A turn-level case asks whether a particular response is good given preceding context. A conversation-level case asks whether the interaction as a whole achieved the intended outcome. These are different questions. A support agent can produce individually reasonable turns yet fail the overall task by looping, forgetting a constraint, or escalating too late.<\/p>\n Microsoft’s dataset schema uses fields such as `messages` for conversational interactions and can also represent separate query-and-response structures. The selected evaluators may require additional columns. If the system under test generates new responses, the dataset can supply the input context while Foundry invokes the target. If the dataset already contains completed responses and the goal is to score them directly, those stored responses become the evaluation object instead.<\/p>\n The test unit should match the failure mode. Retrieval quality can often be evaluated at a turn level. A tool-using agent that must gather several facts, ask for missing information, and complete a workflow may need conversation-level evaluation. Mixing those goals into one undifferentiated score can hide whether the failure came from one answer or from the orchestration across several turns.<\/p>\n A strong evaluation set is not a random sample of prompts. Start with the system’s intended capabilities and create representative cases for each. Then add boundary cases: ambiguous requests, incomplete information, conflicting instructions, uncommon but valid inputs, and cases that should be refused or escalated. Finally, add cases derived from observed failures. This produces a portfolio of evidence rather than a collection of easy demonstrations.<\/p>\n Production traces can be valuable because they reveal how people actually use the system. Foundry can evaluate Application Insights traces directly, and traces can also be converted into reusable datasets. That creates a useful workflow: monitor production, identify meaningful patterns, curate representative cases, and add them to the regression set. The dataset then evolves with the application rather than becoming a frozen artifact from the pilot phase.<\/p>\n Synthetic data is another source, but it should expand coverage rather than replace domain review. Foundry can generate synthetic evaluation data from agent definitions, prompts, reference files, and simulation scenarios. Synthetic cases are useful for exploring combinations and scaling test volume. Human review is still needed to confirm that the generated scenarios reflect real requirements and that expected behavior is meaningful.<\/p>\n Evaluation data should be versioned for the same reason application interfaces are versioned: the meaning of success changes over time. A new policy may introduce a required disclosure. A business process may remove an obsolete path. An agent may gain a new tool that changes the expected answer for a previously static question. If the dataset is edited in place without a record, trend comparisons can become misleading because the test itself changed.<\/p>\n A practical approach keeps a core regression set stable and adds versioned scenario groups around new capabilities. Record why a case was added, what requirement it represents, and whether the expected outcome changed intentionally. If a case is retired, preserve the reason. This makes evaluation results auditable and helps reviewers distinguish a product regression from a test-set revision.<\/p>\n The same discipline applies to labels and reference answers. A \u201cgolden\u201d response should not be treated as sacred prose if multiple responses could be correct. When possible, encode the requirement being evaluated\u2014facts that must be present, actions that must or must not occur, citation expectations, policy boundaries, or task completion\u2014rather than forcing every model to imitate one wording.<\/p>\n One metric rarely captures the behavior of a generative system. Relevance, groundedness, task completion, safety, tool correctness, format compliance, latency, and cost can move independently. A change that improves style may hurt retrieval fidelity. A larger model may improve reasoning while increasing response time. A stricter instruction may reduce unsafe behavior but create more escalations. Evaluation therefore needs a small set of metrics tied to real requirements.<\/p>\n The existing discussion of LLM judges and metrics<\/a> is useful because automated evaluators also have limitations. A model-based judge can scale subjective assessment, but its output is still evidence to interpret, not ground truth. Pair automated scores with deterministic checks where possible\u2014for example, required JSON fields, successful tool calls, prohibited actions, citation presence, or exact business rules.<\/p>\nA reusable dataset is a regression asset<\/h3>\n
Choose the evaluation unit before collecting examples<\/h3>\n
Build the dataset from requirements, failures, and real traffic<\/h3>\n
Version datasets when the contract changes<\/h3>\n
Use several evaluators because quality is multidimensional<\/h3>\n