Microsoft Foundry provides risk and safety evaluators for harmful content and agent-specific risks, including prohibited actions and sensitive-data leakage. Agent evaluation extends beyond a final text response because a system can fail through tool selection, data access, or an unsafe action even when the visible answer appears harmless.<\/p>\n
AI-103<\/a> frames production safety as an evidence problem: teams need repeatable tests that show whether an agent respects instructions and policy across representative conversations, tools, and failure paths. That evidence should be versioned with the agent so later model or tool changes can be compared against a known baseline.<\/p>\n Use AI guardrails<\/a> during runtime, but evaluate them offline and in controlled preproduction runs. Guardrails enforce policy on live traffic; evaluation tells you whether the complete system still behaves acceptably after prompts, models, tools, or policies change.<\/p>\n Define the safety claim before choosing evaluators. \u201cThis agent is safe\u201d is too broad to test. \u201cThe agent does not disclose secrets, execute prohibited actions, or generate severe self-harm guidance in these supported workflows\u201d can be tied to datasets, thresholds, and release decisions.<\/p>\n Safety requirements should name the protected subject and unacceptable effect. \u201cNo sensitive data leakage\u201d becomes more testable when the team identifies which data classes, tool outputs, memory fields, and retrieved sources count as sensitive for that particular agent.<\/p>\n Foundry risk evaluators cover categories such as violence, sexual content, self-harm, hateful or unfair content, indirect attack jailbreaks, and code vulnerability, while agent evaluators can assess behavior such as tool-call accuracy, task adherence, prohibited actions, or data leakage. Use the smallest set that maps to the application\u2019s real risks.<\/p>\n A customer-service agent may need strong privacy, harassment, and escalation tests. A coding agent may need code-vulnerability and prohibited-action checks. A research agent that reads external documents needs indirect prompt-injection scenarios. Reusing the same evaluator list for every agent can create impressive dashboards without meaningful coverage.<\/p>\n LLM evaluation<\/a> should also include human or deterministic checks where model-based judges are weak. Safety scores are useful signals, but high-consequence decisions should not depend on one evaluator interpreting ambiguous policy text.<\/p>\n Document what each evaluator cannot prove. A content-safety score does not prove authorization, a tool-call score does not prove business correctness, and a leakage evaluator cannot find secrets that never appear in the test data.<\/p>\n Evaluator selection should also consider language coverage and domain vocabulary. A risk classifier that performs well on ordinary English may need additional human review for specialized medical, legal, or multilingual content where benign terminology can resemble harmful language.<\/p>\n Safety evaluation needs adversarial examples, but it also needs normal user requests. An agent that blocks every difficult conversation can score well on some risk metrics while failing the product. Include benign, borderline, and clearly disallowed cases so false positives are visible.<\/p>\n Represent the ways users actually interact with the system: short prompts, multi-turn clarification, copied documents, multilingual input, tool results, long context, ambiguous requests, and role transitions. Attackers exploit boundary conditions, and ordinary users trigger them accidentally.<\/p>\n Keep provenance for each test case. Record why the example exists, what risk it targets, expected behavior, source incident if any, and whether sensitive data has been sanitized. A large unlabeled prompt list becomes hard to maintain as policy changes.<\/p>\n Version the dataset with the application. If a release changes tools or data sources, add cases for the new attack surface before comparing scores so the new version is not evaluated on an obsolete threat model.<\/p>\n Adversarial datasets should include chained attacks that unfold over several turns. An agent may resist a direct prohibited request but gradually accept a dangerous premise after repeated benign-seeming setup messages, especially when conversation memory persists across turns.<\/p>\n Agent systems can reach the right final answer through an unsafe path. An agent might retrieve data it was not entitled to see, call a privileged tool unnecessarily, or expose sensitive intermediate results before eventually returning a compliant response.<\/p>\n Process evaluation examines the steps taken to reach the outcome. Tool-selection accuracy, parameter correctness, prohibited actions, task navigation, and trace analysis can show whether the agent behaved safely throughout the workflow.<\/p>\n Agent tools<\/a> belong inside the safety test plan because effectful capabilities create risks that final-response scoring cannot see. Negative cases should prove the agent refuses or requests approval when identity, resource scope, or business preconditions are not satisfied.<\/p>\n Preserve traces for failed tests so engineers can see the decision sequence. A pass\/fail score without the tool call, model response, guardrail event, and policy decision can be too thin to support remediation.<\/p>\n Process evaluation is particularly important for actions that are later reversed. A final state can look correct even if the agent briefly disclosed information or executed an unnecessary write before compensating, and that intermediate behavior still matters for safety.<\/p>\n Foundry evaluation can support thresholds and repeated reports, but a release gate should reflect business tolerance rather than an arbitrary target. A threshold that is too low normalizes known failures; a threshold that is too high can block releases because of evaluator noise rather than real risk.<\/p>\n Use regression testing<\/a> to compare versions on the same cases and slices. Relative degradation can be more actionable than a single absolute score, especially when the evaluator itself is probabilistic.<\/p>\n Gate high-severity risks separately. One severe data-leakage example should not disappear inside an average of hundreds of harmless cases. Report counts and worst-case examples alongside aggregate rates.<\/p>\n Require investigation for new failure clusters even if the overall threshold passes. A model update can introduce a narrow failure mode affecting one language, tool, or customer segment while improving the overall score.<\/p>\n Release gates should store the exact failing examples, not only the score. When a threshold regresses, engineers need representative cases to understand whether the issue is broad policy drift, one evaluator change, or a narrow workflow defect.<\/p>\n Model-based evaluators scale coverage but cannot replace policy owners for ambiguous or high-impact judgments. Human reviewers should examine a sample of passes and failures, especially around new policy, edge cases, and severe outcomes.<\/p>\n Calibrate reviewers with shared examples and decision rules. If two reviewers consistently disagree, the policy or expected behavior may be underspecified rather than the agent being uniquely difficult to judge.<\/p>\n Use disagreement as data. It can reveal language that should be clarified in agent instructions, tool approval policy, or evaluator rubric, and it can identify where a binary safety label is too coarse.<\/p>\n Keep reviewers focused on risk evidence instead of writing style. A response can be awkward but safe, or polished but unsafe; evaluation should not let general quality preferences obscure the specific safety claim.<\/p>\n Human calibration sessions should periodically revisit old examples after policy changes. An answer that was acceptable six months ago may no longer match current product rules, and stale labels can make evaluation look stable while governance has changed.<\/p>\n Offline evaluation cannot enumerate every real conversation. Production incidents, near misses, blocked actions, user reports, and unusual tool paths should feed new sanitized test cases so the evaluation set evolves with the system.<\/p>\nChoose evaluators that match the actual failure modes<\/h3>\n
Build test datasets around realistic attacks and ordinary traffic<\/h3>\n
Evaluate process behavior as well as outcomes<\/h3>\n
Turn evaluator scores into release gates carefully<\/h3>\n
Human review remains necessary for policy nuance<\/h3>\n
Production telemetry should feed the next evaluation cycle<\/h3>\n