Microsoft AI-103: CI/CD for Foundry Projects
CI/CD for Microsoft Foundry projects involves more than uploading agent code. A reliable pipeline coordinates application source, agent definitions, model and deployment references, infrastructure, permissions, environment settings, evaluation results, and post-deployment checks. Microsoft documents GitHub-based CI/CD templates for hosted agents that can deploy updated code and verify a response, offering one implementation pattern for broader Microsoft AI agent delivery.
A reproducible Microsoft Foundry project keeps declarative intent and tested code in source control, environment-specific resource identities and secrets in controlled configuration, and deployment tooling responsible for turning those inputs into a known runtime state. Manual portal changes after deployment create configuration drift that makes failures difficult to reproduce.
A mature pipeline therefore combines ordinary CI/CD automation practices with model-specific release gates. Passing unit tests is not enough if the agent can no longer choose the right tool, meet an evaluation threshold, or stay within latency/cost limits after a model change.
Define what belongs in source control and what belongs in the environment
Keep agent source, prompt/instruction files, schemas, tool interface definitions, evaluation code, deployment manifests, and infrastructure definitions versioned. Do not commit secrets, access tokens, private endpoint credentials, or environment-specific IDs that should be resolved during deployment. The repository should make the desired state reviewable without making sensitive runtime material portable by accident.
Use environment variables or a secret store for endpoints and credentials, but avoid turning every setting into an untracked variable. Model deployment name, schema version, feature flags, and policy mode should be represented in a typed configuration contract whose values are visible in the deployment record. Operators need to know what was deployed, not merely that a workflow succeeded.
In Copilot Studio ALM, environment promotion is reliable only when the agent artifact and its dependencies move under control; Foundry pipelines need the same dependency discipline. Undocumented portal-only dependencies turn promotion into a manual reconstruction exercise and make rollback harder to trust.
Separate infrastructure provisioning from application deployment
Infrastructure changes and application-code changes often have different risk profiles. Provision Foundry resources, identity, networking, storage, monitoring, and supporting services through infrastructure as code. Then deploy agent or application code onto the verified infrastructure. This split lets a small source change avoid recreating stable infrastructure while still allowing infrastructure evolution through its own reviewed path.
For new environments, an Azure Developer CLI or Terraform workflow can establish the resource baseline before the hosted-agent pipeline runs. For existing environments, validate that expected resources and role assignments exist before deploying code. A pipeline should fail early on missing permissions or wrong endpoints rather than partially updating the agent and failing during smoke tests.
Use least-privilege workload identity for the CI runner. The pipeline should have only the permissions needed to provision or deploy its target scope. Human administrator credentials in CI are difficult to rotate, easy to overgrant, and unnecessary for repeatable deployment.
Keep environment topology explicit. Development may share a low-cost model deployment while production uses a separate capacity pool, private networking, or stricter identity controls. A pipeline should resolve these differences from environment configuration without changing application source. When production needs an emergency quota or model change, the new value should still flow back into the declared configuration after the incident.
Build quality gates before the deployment job
After static checks, unit tests, and dependency scans pass, the LLM evaluation release gate should test the agent’s important intents, tool routes, refusal boundaries, and known regressions before deployment. If the candidate cannot clear that offline evidence threshold, there is no reason to spend time or risk deploying it.
Make thresholds versioned and explainable. A quality score by itself is weak if nobody knows which slice failed. Report pass rate by scenario, tool-selection accuracy, structured-output validity, unsafe-action attempts, latency, and estimated or measured cost. A deployment approval should show what changed and why the evidence is acceptable.
When a model deployment changes independently of code, run the same gates. Model lifecycle changes are release events because they can change reasoning, tool choice, token usage, and latency. Pin or explicitly configure deployment names rather than silently inheriting an untested new model.
Gate ordering matters. Run fast deterministic checks before expensive model evaluations, and run representative evaluation slices before broad or costly suites. This gives developers rapid feedback for obvious failures while preserving deeper evidence near release. Cache immutable dependencies where safe, but never cache evaluation results across a prompt, model, dataset, or tool-contract change that can alter behavior.
Deploy with immutable identifiers and traceable metadata
Every deployment should emit a release identifier that ties runtime behavior to a source commit, pipeline run, configuration version, schema version, evaluation result, and model/deployment identity. Put that identifier in application telemetry and, where practical, in the agent deployment metadata. Incident responders should be able to answer “what is running?” without opening the CI logs and guessing.
Avoid editing production artifacts in place through the portal after a successful pipeline run. If an emergency change is necessary, capture it immediately back into source control and redeploy so the declared state becomes authoritative again. Otherwise the next normal deployment can erase the fix or reintroduce the original problem.
For progressive delivery, deploy a new revision or environment alongside the current one when the platform architecture allows it. Shift a controlled workload, compare behavior, then promote. This creates a safer rollback than attempting to reverse a sequence of mutable portal changes.
Artifact integrity should be verifiable as well. Build the deployable package once, attach a checksum or immutable artifact identifier, and promote that same artifact through environments where practical. Rebuilding from the same commit for each stage can introduce dependency drift, mutable package versions, or toolchain differences that weaken the claim that production is the version tested earlier.
Smoke tests must exercise the real runtime path
Microsoft’s hosted-agent CI/CD quickstart includes deployment followed by an invocation that verifies a response. Extend that idea beyond a generic “hello.” Test identity, model access, a representative tool call, knowledge or data dependency, structured output if used, and the monitoring path. A response from the model proves far less than a successful end-to-end agent task.
Tool use and function calling introduces external dependencies that can be misconfigured even when the agent deployment itself succeeds. Smoke tests should detect missing permissions, wrong environment URLs, schema drift, and connector failures. Keep them fast enough to run on every deployment but realistic enough to catch integration breakage.
Tag smoke traffic so it can be filtered from business analytics. Synthetic test calls should not inflate adoption, task completion, or cost-per-user metrics. They should still appear in technical telemetry so the deployment record contains objective evidence that the production path worked.
Smoke-test data should be purpose-built and reversible. Do not use a real customer record merely because it already exercises the workflow. Create known synthetic identities, documents, or tickets that the target environment can access, and clean up any side effects. This makes repeat deployments deterministic and avoids turning deployment automation into a source of privacy or data-quality incidents.
Roll back based on measured failure conditions
Define rollback criteria before the release: failed smoke test, error-rate increase, latency threshold breach, evaluation canary regression, or tool-side fault above a known baseline. The rollback action should restore the last known-good application and configuration, not merely redeploy the previous source commit while leaving model or environment changes behind.
In Foundry agent architecture, rollback decisions depend on distinguishing control-plane failures from runtime and tool failures. If the model is healthy but an external tool is failing, rollback may not help. Conversely, if a new prompt or code route causes unsafe actions, the application revision should be reverted even when infrastructure metrics look normal.
Practice rollback. A process that exists only in a runbook is not a control until it has been executed against a non-production environment and its recovery time is known.
Treat deployment evidence as part of the product
A Microsoft Foundry delivery pipeline needs an auditable record of its source commit, artifacts, infrastructure versions and deployment approvals. That record supports audit, incident response, and future migration without relying on tribal knowledge.
The strongest pipeline is boring in the best way. A developer changes source; automation builds and tests; evaluators enforce behavioral gates; infrastructure and permissions are checked; deployment is traceable; smoke tests exercise the actual runtime; and rollback is clear. The generative parts of the application do not remove the need for deterministic delivery discipline.
CI/CD for Foundry is therefore less about one Microsoft template than about joining software delivery and model governance into a single release mechanism. Once that mechanism is reliable, teams can iterate faster precisely because each change is easier to prove and reverse.
For multi-team environments, publish a small release contract: which team owns infrastructure, which owns agent logic, who owns model capacity, who approves production, and who responds to tool-side incidents. CI/CD can automate the handoff, but it cannot resolve unclear accountability. The release record should make ownership as visible as the technical version.