Microsoft AI-103: Azure OpenAI Structured Outputs
Structured Outputs in Azure OpenAI through Microsoft Foundry move application integration beyond “please return JSON.” Current Microsoft documentation distinguishes JSON mode, which guarantees syntactically valid JSON, from structured outputs, which constrain supported models to a supplied JSON Schema. For Microsoft AI agents, that difference is an architectural boundary: a parser can trust a validated contract far more than it can trust natural-language instructions about field names and types.
Structured outputs are available through both Chat Completions and the Responses API, but the JSON Schema is supplied in different request locations. Chat Completions uses response_format with json_schema; Responses uses text.format. In either case, strict schema handling affects downstream validation, refusals, fallback behavior, and versioning. A correctly shaped response remains a component of a typed software interface, not proof that its facts are correct.
That does not eliminate application validation. Schema adherence proves shape, not business correctness. A well-formed amount can still be semantically wrong, an enum can be valid but inappropriate, and a tool argument can be authorized incorrectly. The contract should therefore combine tool and function-call design with ordinary domain validation and permission checks.
Use structured outputs when the consumer needs a contract, not merely JSON
JSON mode is still useful when an application only needs parseable JSON and can validate the structure itself. Structured outputs are stronger when code generation, extraction, routing, or workflow steps depend on specific required properties, nested objects, arrays, enums, and `additionalProperties` behavior. The schema should reflect what the consumer truly requires instead of mirroring every possible field the model might mention.
Keep the schema intentionally small. Each optional branch increases the number of states the model and downstream code must handle. If a workflow has materially different outcomes, consider a discriminated result type with a clear `status` or `kind` field rather than a large object where most fields are nullable. This makes failure handling explicit and simplifies tests.
Schema validity does not make an output safe at the API security boundary. A schema-valid URL, command, identifier, or tool argument can still target the wrong resource or exceed the caller’s authority. Validate authorization, resource ownership, allowed hosts, numerical ranges, and other domain constraints before using model output in a privileged operation.
Design the JSON Schema for model reliability as well as software typing
A schema that is technically legal can still be difficult for a model to satisfy consistently. Prefer descriptive property names, short field descriptions, bounded enums, explicit required fields, and predictable nesting. Avoid deeply recursive structures and ambiguous unions when a flatter representation can express the same decision. The schema is part of the prompt surface, so clarity improves both model behavior and maintainability.
Use `additionalProperties: false` when the consuming code should reject unplanned fields. This prevents the model from “helpfully” inventing keys that look plausible but are not recognized by the application. If extension data is genuinely needed, define an explicit metadata structure or versioned extension field so the flexibility is deliberate.
Separate display text from machine state. A model can return a stable `decision_code` or enum for workflow logic and a human-readable explanation for the user. Downstream code should branch on the stable field, not on phrases in the explanation. That pattern makes localization and wording changes much less risky.
Descriptions inside the schema should clarify semantics that a type alone cannot express. A field typed as string might represent an ISO date, an internal resource key, or a user-facing label; those meanings change what a correct value looks like. Keep descriptions concise and test them with ambiguous cases. If the model repeatedly confuses two fields, redesigning the contract is often better than adding more prompt prose around a poor schema.
Version contracts independently from prompts and models
Treat the schema as an API contract with a version. Prompt wording can change without changing the contract; a model can be upgraded while the same contract remains; and a new product requirement can introduce a contract version even if the model stays fixed. Store the schema version with traces and evaluation results so a malformed or incorrect production result can be tied to the exact interface that produced it.
In Microsoft Foundry project design, schemas need an independent version lifecycle so prompt assets, model/deployment references, application code, and interface contracts can move through environments without accidental coupling. A schema change that is incompatible with an existing consumer should fail CI before it reaches the deployed agent or service.
For rolling upgrades, support old and new versions intentionally. A producer may emit v2 while a downstream consumer still expects v1. Either coordinate an atomic release or provide an adapter. Do not rely on the model to infer which historical structure a caller can handle.
Schema evolution also needs consumer telemetry. Record which contract version each client or workflow is using so old versions can be retired deliberately. Without usage evidence, teams either keep legacy schemas forever or remove them based on guesswork. A short deprecation window is safe only when the organization can prove that dependent consumers have migrated.
Handle refusals, truncation, and exceptional outcomes explicitly
Structured output does not mean every request necessarily produces a business-success object. Safety refusals, request errors, model limits, or upstream failures still need separate handling. Inspect response status and finish conditions before deserializing the expected schema. A parser error should not be the first signal that the model could not complete the task.
Define application-level outcomes for “cannot determine,” “needs review,” or “insufficient evidence” where those states are legitimate. Forcing the model to populate a confident-looking object when the source data is incomplete converts uncertainty into false precision. A typed failure or review state is safer than fabricated fields.
If a response can be long, design the task so the structured object remains bounded. Large free-text arrays or unconstrained evidence blocks increase latency and make truncation more likely. Store detailed evidence separately when possible and let the schema carry identifiers, summaries, scores, or selected excerpts.
Plan for version skew between SDKs and API surfaces. A schema feature that works in one request type or model can be unsupported elsewhere, and model support changes over time. Deployment validation should send a representative structured-output request to the exact model/deployment before promotion. If the endpoint rejects the feature, fail the release or switch to a documented fallback rather than silently dropping strictness.
Test schemas with adversarial and boundary cases
A strong LLM evaluation gate should validate both shape and meaning. Include missing information, contradictory inputs, extra irrelevant text, prompt injection attempts, values at numerical boundaries, malformed source data, and cases that should produce a review state. The output can pass JSON Schema validation while still failing the business objective, so semantic assertions remain necessary.
Property-level tests should be deterministic where possible. Verify that required identifiers come from the allowed source set, totals reconcile, dates are valid for the domain, and enum choices match known scenarios. Reserve model-based judges for qualities that genuinely require interpretation rather than using another model to check facts that ordinary code can validate exactly.
Run the same suite when the model deployment changes. Schema adherence may stay perfect while extraction quality, field selection, refusal rates, or latency shift. The contract protects the parser; the evaluation suite protects the product behavior.
Property-based test generation can widen deterministic coverage without inventing new production behavior. Generate strings at length limits, empty arrays, maximum list sizes, enum boundaries, Unicode text, and optional-field combinations, then verify both the API response and downstream deserializer. This is ordinary contract testing applied to a probabilistic producer; it catches brittle assumptions that example-only tests miss.
Integrate structured outputs with tools without collapsing the boundaries
In production Agentic orchestration, structured final responses and tool/function calls represent separate contracts that require validation. Keep their roles distinct. Tool schemas describe callable operations and arguments. A final structured response describes the application result returned after the plan completes. Reusing one giant schema for both tends to blur permissions, intermediate state, and user-facing output.
Validate tool arguments at execution time even if the model created them through strict schema enforcement. The model does not own access control. Likewise, validate tool results before feeding them back into later model steps when an external system can return unexpected or hostile content.
A useful pattern is: model selects or fills a tool contract, application authorizes and executes it, application normalizes the result, then the model produces a final structured result. Each boundary has a clear owner and can be logged independently.
Measure reliability as contract success plus task success
Structured outputs can improve AI cost and performance by reducing retries, parser failures, defensive prompt tokens, and downstream exceptions even when raw inference pricing does not change. Track schema-valid completion rate, semantic pass rate, retries, latency, token usage, and downstream exceptions together. The best contract is the one that lowers total application failure cost.
A Foundry workload should record its model deployment and output-schema version together so responses remain reproducible across releases. When a production regression appears, operators need to know whether the contract changed, the prompt changed, the model changed, or the input distribution shifted. That traceability is more valuable than a generic “JSON parse succeeded” metric.
Structured outputs are most useful when they are treated as software architecture rather than a formatting trick. Small schemas, explicit versions, strong business validation, disciplined tool boundaries, and regression testing turn generative output into an interface that ordinary systems can safely consume.