Retries are one of the easiest ways to make an agent system look more reliable while quietly making it less safe. A transient 429 from a model deployment may deserve another attempt. A malformed tool request does not. A timeout after a payment API call is especially dangerous because the call may have completed even though the agent never received the response. A useful retry policy therefore begins with failure classification, not with a loop that sleeps and tries again.<\/p>\n
Microsoft Foundry documentation reflects this distinction. Current service guidance recommends exponential backoff with jitter for rate-limit and certain capacity failures. Foundry routines treat 408, 429, and 5xx downstream responses as retryable while attempts remain, while other 4xx responses are treated as terminal. Those defaults are useful signals, but an application still has to understand whether retrying the business operation itself is safe.<\/p>\n
Inside the broader Microsoft AI agents<\/a> architecture, retries are a control-plane decision. The model should not decide on its own that \u201ctrying again\u201d is harmless simply because the previous tool result looked unsuccessful.<\/p>\n A transport failure means the system could not complete or confirm the technical exchange: a timeout, temporary network error, 429, or service-side 5xx. A semantic failure means the request reached the service but the operation itself was invalid: bad input, missing permission, unsupported tool arguments, or a business rule violation. Retrying semantic failures without changing the request wastes capacity and can hide bugs.<\/p>\n The distinction should be encoded in software. Tool adapters can return structured error categories instead of a prose string such as \u201csomething went wrong.\u201d The orchestrator can then decide whether to retry, ask the model to repair the request, escalate to a human, or stop. A model can help interpret ambiguous failures, but deterministic status codes and typed error contracts should drive the default policy.<\/p>\n The same principle appears in Azure AI service failure patterns<\/a>: resilience depends on recognizing the failure mode rather than applying one generic recovery path.<\/p>\n When a service is throttling because capacity is exhausted, immediate retries create more pressure. Exponential backoff spreads repeated attempts over a longer period, and jitter prevents many clients from waking up at exactly the same time. Microsoft recommends this pattern for Foundry rate-limit and regional session-capacity conditions.<\/p>\n Retry-After headers should be respected when the upstream service provides them. A gateway or client that ignores explicit recovery guidance and uses a shorter hard-coded delay can turn a transient throttle into sustained overload. The planned API Management for AI gateways<\/a> article covers how gateway policies and circuit-breaking behavior can reduce the amount of custom retry logic every application needs to own.<\/p>\n Backoff is not a substitute for capacity planning. If a workload is continuously above its model or session quota, a retry loop merely converts insufficient capacity into latency. Teams should examine sustained throughput, provisioned capacity options, regional design, and workload shaping rather than congratulating themselves on having a sophisticated retry algorithm.<\/p>\n The hardest retry problem occurs after an operation may have changed external state. Suppose an agent submits a support ticket, creates a deployment, or transfers a case and then loses the response. A second attempt can create a duplicate even though the first request returned no success message to the agent.<\/p>\n High-impact tools should therefore support idempotency keys, operation identifiers, or a read-after-write recovery path. Before retrying, the application can ask the downstream system whether the intended operation already exists. If the remote API cannot support idempotency, the orchestrator may need to stop and escalate rather than risk duplicate side effects.<\/p>\n This is where agent design differs from ordinary text generation. Repeating a model call may only cost tokens. Repeating a tool call can affect money, infrastructure, permissions, or customer records. Tool schemas and execution policy should make that difference visible.<\/p>\n One user request can cross several dependencies: Foundry, a search service, an A2A agent, a business API, and a database. If each layer retries three times, the combined request can explode into dozens of calls. A top-level \u201cthree retries\u201d setting does not control this multiplication.<\/p>\n Each dependency needs a bounded retry budget that reflects its latency and failure behavior, while the workflow needs an overall deadline. A fast metadata API may tolerate two quick retries. A long-running remote agent may be better handled asynchronously with task polling. A model deployment under load may deserve backoff, but not if the remaining user-facing latency budget is already exhausted.<\/p>\n Observability should record both the per-dependency attempts and the end-to-end attempt count. AI observability<\/a> is incomplete if a trace shows only the successful final model call and hides the six failed attempts that came before it.<\/p>\n A mature system does not wait for the backend to throttle every caller. AI gateway token quotas<\/a> can give applications their own token budgets before they compete for a shared deployment. Request limits and queueing can smooth bursts so fewer calls reach the model in a state that is likely to return 429.<\/p>\n This also creates clearer ownership. If one application is consuming its allocated budget, the platform can throttle that application without forcing every other application into retry mode. The gateway becomes a fairness boundary rather than just a proxy.<\/p>\n Semantic caching can also reduce load for workloads with repeated or near-duplicate prompts, but cached responses must be safe for reuse. AI gateway semantic caching<\/a> should never be treated as an invisible reliability hack when a reused answer could be stale or tenant-specific.<\/p>\n When a tool rejects an input schema, the right recovery may be to let the model repair the arguments and submit a new logical request. That is not the same as replaying the failed HTTP request. The distinction matters for telemetry and for side-effect safety.<\/p>\n A retry should preserve the same intent and usually the same idempotency key. A repair changes the request because the original was invalid. Logging both as \u201cretry #2\u201d hides the engineering difference. Better traces record whether the system retried transport, regenerated tool arguments, switched a model deployment, or escalated to a fallback path.<\/p>\nSeparate transport failures from semantic failures<\/h3>\n
Backoff protects recovering systems from synchronized load<\/h3>\n
Side effects require idempotency or explicit recovery<\/h3>\n
Attempt budgets should be local to the dependency<\/h3>\n
Rate limits and quotas should reduce retries before they happen<\/h3>\n
Model repair and request retry are different actions<\/h3>\n