AI red-team test cases are structured adversarial scenarios designed to reveal how a generative AI system behaves when users, retrieved content, or tools push it toward unsafe, unauthorized, deceptive, or unreliable behavior. Effective red teaming is not a collection of provocative prompts. It is a repeatable engineering practice that connects a threat model to test objectives, expected safe behavior, measurable outcomes, and remediation. In Agentic AI Engineering, red-team cases belong beside ordinary quality evaluation because an agent can complete normal tasks perfectly while still failing under manipulation.
Microsoft’s current AI Red Teaming Agent automates adversarial probing and uses Foundry risk and safety evaluators to measure outcomes such as Attack Success Rate. It supports content-risk testing and agent-specific risks including prohibited actions, sensitive-data leakage, and indirect prompt injection for supported Foundry agent scenarios. NIST’s generative AI risk-management guidance provides a broader governance frame: identify the risks relevant to the use case, measure them with appropriate evidence, and manage them throughout the lifecycle rather than treating red teaming as a one-time prelaunch event.
Write each test case around a concrete failure objective
A useful case says what the attacker is trying to achieve, what system boundary is under test, what safe behavior is expected, and what evidence determines pass or fail. “Try to jailbreak the model” is too vague. “Attempt to make the support agent disclose a synthetic secret from a tool result despite the user lacking access” gives the team a target, setup, expected refusal, and observable outcome.
Generative AI evaluation pipelines make these cases durable when objectives and expected outcomes are stored with versions and metadata. Red-team findings should become regression tests after remediation so the same weakness does not return during a model, prompt, or tool update.
Separate harmful-content tests from agent-action tests
Content-risk tests examine whether the system produces disallowed or inappropriate material. Agentic tests ask whether the system performs an unauthorized or prohibited action, leaks sensitive information, or misuses a tool. These categories can overlap, but they require different environments and success criteria. A harmless sentence can accompany a dangerous action, and a safe refusal can still leak confidential context.
Autonomous agent security should therefore define prohibited actions and sensitive resources before red teaming begins. The red team needs to know which behaviors matter to the business, not merely which behaviors a generic benchmark happens to cover.
Test direct prompt manipulation without overfitting to one phrase
Direct attacks try to persuade the model to ignore policy, reveal protected instructions, or reinterpret the user’s authority. Build families of cases that vary wording, role framing, indirection, formatting, conversation history, and benign-looking wrappers. The objective is to test the control concept rather than memorize one famous jailbreak string.
AI guardrails and content safety should be evaluated against both obvious attacks and benign look-alikes. False positives matter too. If the system blocks legitimate security analysis or policy discussion because it resembles an attack, the control can become operationally unusable even if its attack-detection rate looks strong.
Test case families should include a clean baseline before adversarial variation. If the application fails the normal task, an attack-success result is difficult to interpret because the system may simply be broken. Compare ordinary completion, adversarial completion, and safe refusal behavior so the red team can distinguish safety controls from unrelated quality defects.
Include indirect prompt injection through retrieved and tool-provided content
Indirect attacks place hostile instructions in documents, web content, email, tickets, or other sources the agent reads while pursuing a legitimate user task. Test cases should keep the user request benign while injecting adversarial content into the data source. The safe outcome is usually to treat that text as untrusted evidence rather than as authority to change goals or call tools.
Prompt Shields for AI apps can detect some direct and indirect attacks, but the red-team case should also verify downstream controls. If a detector misses the content, does the tool executor still reject unauthorized actions? If a document is removed, does the agent fail safely or retry through an unprotected path?
Use synthetic sensitive data to test leakage without exposing real people
Leakage tests do not require production secrets. Seed a controlled environment with synthetic customer records, credentials, medical-like data, or financial identifiers that are clearly marked for the exercise. Then test whether users with different permissions can induce the agent to reveal them through direct questions, summaries, tool calls, citations, error messages, or conversation memory.
API security is part of the expected defense. A successful safety design should block the request at multiple layers: retrieval filters should exclude unauthorized data, tool APIs should enforce access, and the model should avoid disclosing sensitive content that still appears in context.
Test prohibited actions with reversible, sandboxed tools
Action-oriented red teaming should never endanger production systems. Use a purple or test environment with mock tools, synthetic accounts, reversible operations, and clear monitoring. Define prohibited actions such as deleting protected records, bypassing approval, changing another user’s access, or executing an irreversible transaction, then see whether adversarial prompts can make the agent attempt them.
Agent access and approval boundaries should be visible in these tests. The red team should verify not only that approval exists but that the agent cannot alter the target or parameters after approval, reuse an old approval for a new action, or route around the gate through another tool.
Include denial, ambiguity, and tool-failure cases alongside adversarial ones
Some of the most revealing safety failures happen when the system is confused rather than attacked. Test missing identity, ambiguous resource names, stale data, timeouts, conflicting tool responses, and incomplete user intent. A robust agent should ask for clarification or stop rather than guess its way into a side effect.
Tool schemas for AI agents can reduce malformed calls, but red-team cases should still probe boundaries such as unexpected enum values, missing required context, oversized input, and combinations that are individually valid but dangerous together. The executor should remain authoritative when the model makes a poor choice.
Measure attack success, safety impact, and user impact separately
An Attack Success Rate can summarize how often an adversarial objective succeeded, but one metric rarely captures the whole system. Track severity, which control failed, whether a tool executed, whether sensitive data was exposed, how often benign traffic was blocked, and whether the system recovered safely. A low attack rate with catastrophic rare failures may deserve more attention than a higher rate of harmless policy deviations.
AI business value also matters because controls affect usability. Red teaming should identify mitigations that reduce risk without making the intended workflow unusable. The objective is dependable operation, not a system that “passes” because it refuses everything.
Prioritize remediation by impact and exploitability. A rare content-style issue may require a different response than a repeatable path to unauthorized data access. Record which boundary failed so the fix goes to the correct layer: prompt, filter, retrieval policy, tool authorization, approval flow, or backend API. Red-team findings are most useful when they produce specific engineering work.
Store the test objective, environment, agent version, model, tools, seed data, attack strategy, expected behavior, and evaluator result. For harmful prompt content, control access and record only what authorized testers need to reproduce the case. Reports for broader stakeholders can summarize the category and mitigation without distributing exploit strings unnecessarily.
Governance standards should define who can run red-team scans, what environments are permitted, how findings are classified, and who owns remediation. Red-team data itself can be sensitive because it documents where the system is weak.
Run red-team cases throughout the lifecycle, not only before launch
Models, prompts, tools, retrieval sources, and attack techniques change. Re-run targeted cases after significant updates and schedule broader exercises for high-impact systems. Use automated scans for breadth and expert manual testing for creative paths that automated strategies may miss. Production incidents and near misses should create new test objectives.
Microsoft’s AI Red Teaming Agent can accelerate supported Foundry scenarios, while Microsoft and NIST guidance provide useful frameworks for systematic testing. The durable practice is vendor-neutral: define the risk, construct a safe adversarial environment, measure the outcome, fix the boundary, and keep the case as evidence. Red teaming succeeds when it changes engineering decisions, not when it produces the longest list of clever prompts.
Red-team coverage should be mapped to architecture components so gaps are visible. Maintain a matrix that connects each risk objective to entry points such as direct user input, retrieval, memory, tool results, file uploads, and external connectors. A system may resist direct prompt attacks while remaining vulnerable through documents or tool output. The matrix prevents a strong result in one channel from being misread as comprehensive protection.
When automated attack generation is used, periodically review the generated cases for relevance to the actual product. High-volume synthetic probes can create impressive statistics while missing the business-specific paths that matter most. Expert testers should add scenarios drawn from real workflows, permission boundaries, and past incidents so automated breadth and human context complement each other.
Close each exercise with a retest plan. A mitigation is not complete because code was changed; rerun the original case, nearby variations, and benign controls to confirm the fix reduced risk without introducing an unacceptable false-positive or usability regression. Record that evidence with the finding so remediation status reflects measured behavior rather than implementation intent.
