{"id":20124,"date":"2026-10-06T15:15:25","date_gmt":"2026-10-06T15:15:25","guid":{"rendered":"https:\/\/www.exam-labs.com\/blog\/?p=20124"},"modified":"2026-10-06T15:15:25","modified_gmt":"2026-10-06T15:15:25","slug":"microsoft-ai-103-telemetry-for-production-agents","status":"publish","type":"post","link":"https:\/\/www.exam-labs.com\/blog\/microsoft-ai-103-telemetry-for-production-agents","title":{"rendered":"Microsoft AI-103: Telemetry for Production Agents"},"content":{"rendered":"

Production agents need more than application logs because one user request can become a chain of model calls, retrieval operations, tool invocations, approval steps, retries, and external side effects. Telemetry has to reconstruct that chain well enough for operators to answer what happened, why it happened, how long it took, what it cost, and whether the result matched policy. In Microsoft AI Agents<\/a>, Microsoft Foundry integrates agent tracing with Azure Monitor Application Insights and OpenTelemetry conventions, giving teams a foundation for distributed traces while still requiring thoughtful choices about what to capture and how to protect it.<\/p>\n

Microsoft’s current Foundry guidance stores traces in Application Insights and uses OpenTelemetry semantic conventions for agent activity. Foundry projects can be connected to an Application Insights resource so traces are visible from the agent experience and queryable through the monitoring stack. That is a strong starting point, but a usable production design also needs metrics, structured events, business outcomes, privacy controls, sampling, retention, and a consistent correlation strategy across services outside Foundry.<\/p>\n

Give every user request a correlation identity that survives tool boundaries<\/h3>\n

A trace becomes useful when a request can be followed across the client, agent runtime, model, search service, tool gateway, and downstream system. Generate or propagate correlation identifiers at the session and turn level, then carry them through tool calls and business APIs. If an external service emits its own request ID, record the mapping so support teams can move between systems without guessing which events belong together.<\/p>\n

Agent analytics and monitoring<\/a> are strongest when evidence can be assembled from one trace rather than manually correlating timestamps. A useful trace shows the user’s intent, the agent version, relevant model deployment, retrieval call, selected tool, execution result, and final response path without requiring raw private content to be stored everywhere.<\/p>\n

Separate traces, metrics, logs, and evaluations by purpose<\/h3>\n

Traces explain a single execution path. Metrics summarize behavior across many executions. Logs record discrete operational events. Evaluations judge quality or policy against defined criteria. Trying to force all four into one data structure produces expensive telemetry that is difficult to query. Design each signal for its job and connect them through shared identifiers and version metadata.<\/p>\n

AI observability<\/a> becomes much more actionable when these signals are complementary. A spike in tool failures should be visible as a metric; representative failed traces should explain the cause; logs should show the downstream error; and an evaluation can determine whether the agent handled the failure acceptably.<\/p>\n

Define a small canonical event model so every agent team records the same core fields even if their workflows differ. Agent identifier, version, model deployment, session ID, turn ID, tool name, outcome, duration, and environment are common dimensions. Standardization makes cross-agent dashboards and fleet-level incident response possible without forcing every application to log its private domain data.<\/p>\n

Measure model activity without reducing everything to token counts<\/h3>\n

Tokens, latency, model name, retries, and finish status are essential operational fields, but they do not tell whether the model did useful work. Add context such as the stage of the workflow, whether a tool was available, how many candidate documents were retrieved, and whether the turn completed the user task. This allows cost and latency to be compared with value rather than viewed in isolation.<\/p>\n

AI cost and performance<\/a> should be analyzed per workflow and outcome. An expensive call that resolves a complex case may be efficient, while several cheap calls that loop without progress are waste. Production telemetry should make repeated reasoning, unnecessary retrieval, and avoidable retries visible.<\/p>\n

Instrument tool calls as first-class spans with structured outcomes<\/h3>\n

For each tool call, record the tool identity, version, duration, success or failure, safe parameter metadata, response status, retry count, and whether the call produced a side effect. Avoid logging secrets, full document payloads, or sensitive tool arguments simply because they are available. A structured outcome such as \u201cauthorization denied\u201d is usually more useful and safer than dumping the entire request body.<\/p>\n

API security<\/a> should guide telemetry design. Logs must never become a bypass around application access controls. Redact credentials, tokenize sensitive identifiers when possible, and apply monitoring-data permissions that reflect the sensitivity of the underlying workflows.<\/p>\n

Trace retrieval so grounded answers can be investigated<\/h3>\n

RAG failures are difficult to diagnose from the final response alone. Record the search service, query or subquery, filters, candidate count, latency, source identifiers, and the passages selected for context where policy allows. You do not need to copy entire documents into traces; stable source references and ranking metadata are often sufficient for investigation.<\/p>\n

Enterprise RAG chunking<\/a> and retrieval metadata should appear in observability together. When an answer cites the wrong section, operators need to know whether the chunk boundaries were poor, the query was wrong, the ranker selected weak evidence, or the model ignored strong evidence.<\/p>\n

Capture approvals and policy decisions as part of the execution story<\/h3>\n

An approval gate is not an invisible pause. Record when approval was requested, what action was proposed, who or what policy decided, how long the decision took, and whether the final action matched the approved intent. This makes it possible to distinguish agent errors from governance delays and to audit high-impact workflows without relying on chat transcripts.<\/p>\n

Agent access and approval boundaries<\/a> become operational only when their decisions are observable. If a workflow repeatedly requests unnecessary approvals, that is a design problem. If an agent bypasses an expected approval path, that is a control failure. Telemetry should reveal both.<\/p>\n

Protect monitoring data with retention, sampling, and field-level discipline<\/h3>\n

Agent traces can contain prompts, retrieved text, tool parameters, personal data, and business secrets. Capture only what is required for operations and evaluation, and apply redaction before data leaves the application when possible. Define different retention periods for raw traces, aggregate metrics, and audit records rather than storing everything indefinitely.<\/p>\n

Governance standards<\/a> should identify who can query trace data and under what circumstances. Application Insights access and Log Analytics roles are real security controls. Monitoring teams need enough data to diagnose incidents, but broad access to raw AI traces can create a secondary data-exposure risk.<\/p>\n

Traditional availability remains important, but an agent can be \u201cup\u201d while failing users. Define SLOs for task completion, error rate, time to useful response, tool success, retrieval latency, approval latency, and policy incidents. Segment by agent version, model, tool, user journey, and region so aggregate averages do not hide a failing path.<\/p>\n

GenAI observability<\/a> should combine technical and quality signals. A change that improves latency but reduces groundedness may not be an improvement. Likewise, a safer policy that doubles abandonment may need a better user experience rather than weaker enforcement.<\/p>\n

Alerting should focus on symptoms that demand action. A single high-latency trace may be normal for a complex request, while a sustained rise in p95 tool latency or authorization failures can justify an alert. Quality metrics often need trend detection and sampled review rather than paging an operator for every low-scoring response.<\/p>\n

Turn production traces into evaluation data without copying production blindly<\/h3>\n

Representative traces are valuable for regression testing, especially when they capture edge cases the original test set missed. Sanitize sensitive content, preserve only the context needed to reproduce the behavior, and label the expected result. Then run the case in a controlled environment against candidate versions before deployment.<\/p>\n

Generative AI evaluation pipelines<\/a> close the loop between observability and quality. A production incident should create a future test; a failing test should block the same defect from reappearing. Telemetry becomes much more valuable when it improves the next release rather than only explaining the previous one.<\/p>\n

Use telemetry to operate the agent as a system, not a black box<\/h3>\n

Foundry tracing and Application Insights give teams a practical way to see agent execution, but the default trace is only the beginning. Extend correlation into downstream tools, preserve version metadata, define metrics that reflect outcomes, and protect the resulting data. Operators should be able to answer whether a failure came from the model, retrieval, a tool, policy, or infrastructure within minutes rather than reconstructing the story from scattered logs.<\/p>\n

Microsoft<\/a> provides the monitoring foundation, while the application team defines what operational truth looks like. Good telemetry makes reliability, safety, cost, and quality measurable at the same workflow boundary. The aim is not maximum data collection. It is enough trustworthy evidence to diagnose, improve, and govern the agent without creating a new privacy or security problem.<\/p>\n

Deployment metadata should be queryable without joining several private systems. Record environment, release identifier, agent definition version, model deployment, and critical tool versions on root spans or equivalent events. During an incident, operators can then compare a failing cohort with the previous release and determine whether the issue began at a rollout boundary. This is especially useful for gradual deployments where only a percentage of traffic sees a new model or prompt.<\/p>\n

Finally, verify the telemetry path itself. Missing traces, delayed ingestion, broken sampling, or an expired Application Insights connection can create the illusion that the agent is healthy because failures are simply invisible. Include monitoring-pipeline health in operational checks and maintain a known test transaction that confirms end-to-end trace collection after infrastructure changes.<\/p>\n","protected":false},"excerpt":{"rendered":"

Production agents need more than application logs because one user request can become a chain of model calls, retrieval operations, tool invocations, approval steps, retries, and external side effects. Telemetry has to reconstruct that chain well enough for operators to answer what happened, why it happened, how long it took, what it cost, and whether […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-20124","post","type-post","status-publish","format-standard","hentry","category-general"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t