Key Vault is for secrets that still need to exist
Azure Key Vault gives AI applications a controlled place to store secrets, keys, and certificates instead of embedding them in source code or configuration files. Microsoft Foundry documentation also recommends Microsoft Entra authentication and managed identities where possible, because the strongest secret is one the application does not need to store. For Microsoft AI-103, the design decision is therefore not ‘put every credential in Key Vault’ but ‘eliminate static credentials first, then protect the unavoidable ones.’
This is a recurring theme in Microsoft AI Agents: model endpoints, search services, tool APIs, and data stores all have identities. Authentication architecture should be explicit before prompt or agent logic is built.
Inventory credentials before designing the vault layout. Separate model API keys, third-party SaaS tokens, database passwords, certificates, signing keys, and customer-managed encryption keys because they have different owners and rotation processes. Then challenge each entry: can Entra ID, workload identity, or a federated identity remove it? This exercise often shrinks the secret estate more effectively than building a sophisticated vault hierarchy around credentials that never needed to exist.
Prefer managed identity for Azure-to-Azure authentication
When an Azure-hosted workload can authenticate to Foundry or another Azure service through Microsoft Entra and managed identity, there is no client secret to rotate or accidentally leak. Assign the identity only the roles required by the workload and let the platform acquire and refresh tokens.
Use Key Vault for credentials to systems that cannot use that identity model, third-party API keys, signing certificates, or customer-managed encryption keys. This division is described well by Azure Key Vault: a vault centralizes sensitive material, but applications should still minimize how much material exists.
Choose system-assigned identity when the permission should disappear with one workload, and user-assigned identity when multiple resources intentionally share the same principal or the identity must exist before the compute resource. The choice affects lifecycle and blast radius. Whichever model is used, scope RBAC at the narrowest practical resource and avoid broad subscription roles simply because they make initial testing easier.
Separate secret read permission from vault administration
A runtime identity that needs to read one secret should not also have permission to create keys, change vault policy, or delete secrets. Use Azure RBAC or appropriate Key Vault access policy boundaries so management and data access are distinct.
The deployment pipeline may need to create or rotate a secret while the running application only needs to read it. Separate identities make compromise less damaging and simplify audit trails because each operation has a clear owner.
Administrative separation should extend to deployment automation. Infrastructure pipelines may create the vault and role assignments, a security workflow may rotate a secret, and the application identity may only retrieve a specific value. Distinct principals make it possible to revoke one function without disabling the others. Review role assignments periodically because temporary troubleshooting permissions have a habit of becoming permanent if no owner is responsible for removing them.
For AI applications with multiple components, use separate runtime identities when the trust boundaries differ. A web front end, retrieval worker, evaluation job, and background agent may not need access to the same secrets. Splitting those identities makes a compromised component less valuable and produces clearer audit trails than a single shared principal with broad vault access.
Do not pull secrets into prompts or model context
A secret retrieved from Key Vault belongs in the application layer that calls the protected service. It should not be inserted into the system prompt, tool description, conversation history, or error message. Language-model context is not a secure transport for credentials.
The broader issue is captured by secrets and privilege: authentication material and AI reasoning have different security lifecycles. Keep the credential boundary outside model-visible text.
Apply redaction before logs and traces as well. Agent frameworks can capture prompts, tool arguments, errors, and intermediate messages for observability; a credential accidentally included in a tool argument can therefore be copied into several telemetry systems. Treat secret values as tainted data that must stay in the client or connector layer. Prefer references or connection identifiers that the tool runtime resolves without exposing the underlying credential to the model.
Rotation needs application behavior as well as vault support
A vault can hold a new secret version, but the application must know how quickly to pick it up and what happens to in-flight work. Prefer clients that retrieve secrets at startup with controlled refresh, or use a cache with a documented lifetime. Avoid fetching Key Vault on every tiny request if it creates unnecessary latency and dependency pressure. Rotation tests should verify cache invalidation, retry behavior, rollback, and stale-secret detection across every consumer. A vault-side rotation job is incomplete if a worker keeps the old value in memory until restart or if a failed refresh silently falls back to expired credentials.
Test rotation in a staging environment. Verify the old credential can be revoked without restarting unrelated services and that rollback is possible when an external API has a staged credential change.
Design for overlap when the downstream system permits two active credentials. Publish the new version, allow applications to refresh, verify successful use, then revoke the old version. When only one credential can be active, coordinate maintenance and rollback explicitly. Track secret version or credential generation in telemetry without logging the value itself so operators can determine which version a failing instance is using during a rotation incident.
Rotation testing should include failure behavior as well as the happy path. Verify what happens when an old version expires, a new version is not yet readable, the vault is temporarily unavailable, or a cached credential remains in memory. Applications should fail predictably and recover without requiring engineers to copy emergency secrets into configuration files.
Customer-managed keys solve a different problem from API secrets
Foundry and related Azure services can use customer-managed encryption keys stored in Key Vault for data-at-rest controls. That is not the same as storing an API key used by application code. CMK designs require soft delete, purge protection, regional considerations, and managed-identity permissions such as key operations.
Keep these use cases separate in architecture diagrams and runbooks. One is application authentication; the other governs encryption keys used by platform services. Conflating them creates confusing role assignments and incident procedures.
CMK availability can become a control-plane dependency for the platform service. Protect the key from accidental deletion, enable required recovery controls, and understand who can rotate or revoke it. A key-per-environment or key-per-data-boundary strategy may simplify auditability, but excessive fragmentation increases operational overhead. Choose boundaries from regulatory and ownership requirements rather than assuming every application needs a unique encryption key.
Private networking changes how the vault is reached
Organizations using private endpoints need to ensure DNS, subnet routing, firewall policy, and service dependencies allow the application to reach Key Vault and Foundry without unintentionally falling back to public paths. Network isolation should be tested from the real runtime environment, not inferred from portal settings.
A secret architecture is incomplete if the runtime cannot reach the vault during an incident or scales into a subnet without the required name resolution. Include connectivity in deployment validation.
Private endpoint designs also need name-resolution discipline. A workload can have the right identity and still fail because the vault hostname resolves publicly, a private DNS zone is missing from the VNet, or egress policy blocks the path. Test token acquisition and secret retrieval from every runtime subnet used in production. Include failure behavior so a temporary vault outage does not cause the application to expose secrets or silently fall back to insecure configuration.
Private endpoints also create a name-resolution dependency that is easy to overlook during deployment. A workload can have correct RBAC and still fail because the vault name resolves to the wrong address or because a build agent sits outside the permitted network path. Test resolution and connectivity from the actual runtime environment, not from an administrator workstation.
Key Vault should reduce operational surprise
The healthy pattern is clear: use Microsoft Entra identity and managed identity for first-party service access, keep unavoidable secrets in Key Vault, grant least privilege, rotate deliberately, and prevent secret values from entering prompts or logs. AI applications do not need a special exemption from ordinary credential engineering.
That discipline is especially valuable for agents because tools can touch many systems. Central secret management plus identity-based access keeps the agent’s expanding capability surface from turning into a collection of hard-coded credentials.
Measure whether the design actually reduces credential incidents: fewer static secrets, shorter secret lifetimes, successful rotation drills, narrowly scoped identities, and no credentials in repositories or traces. A vault is valuable because it supports disciplined lifecycle management, not because it moves the same unmanaged secrets into a different service. Mature teams can explain who owns each remaining secret, why it exists, and how it is revoked.
Mature secret management also includes inventory and retirement. Periodically confirm which application identity still reads each secret, which dependency requires it, and whether a managed-identity or federation path now removes the need for the credential entirely. Centralizing unused secrets is not the same as eliminating secret sprawl; stale credentials should be revoked, not merely stored safely.
