I can stand up a useful Azure AI Foundry agent in an afternoon. Most people reading this can too. Connect some data, pick a model, write a few instructions, hit deploy. Demo works. Stakeholders nod. Then someone asks when it can go live.
That is usually where it gets messy.
Prompts and model choice are not the hard bit. Azure AI Foundry production readiness is mostly engineering: identity, change control, ownership and a way to roll back when Thursday’s “quick tweak” lands badly.
Foundry is Microsoft’s managed workspace for wiring models, data, tools and agents on Azure. Great for exploration. I would not run production change control out of the portal click path.
Where proofs of concept start to drift
A proof of concept tests an idea. A production system supports a business process. Mix those up and you get the middle ground I see constantly: the agent “works”, the slides look good and nobody wants to own the operational risk.
Uncontrolled change is the usual culprit. Tuesday the playground answers look fine. Thursday someone “just tweaks” the system prompt before a stakeholder session. A tool gets nudged. The model deployment moves. Suddenly the answers are different and nobody can say which version users hit. Rollback is whoever still has the old text in a draft email.
That is change management failing. The model is rarely the villain.
Once real users, real data or real decisions are in play, the agent picks up the same baggage as any other business-critical Azure app. Rebuildable infra. Security that was designed in. A release path. Someone to support it. A route back at 2am.
The teams getting value are almost never the ones with the cleverest prompt. They are the ones using the same standards they already apply to App Service, AKS or a data platform.
Azure AI Foundry is more than a model endpoint
I still see Foundry treated like a side sandbox. Fun for demos. Someone else’s problem when security turns up. That story ends the moment a second team depends on it, or the agent can reach something it should not.
In the platforms I build, a production-shaped deployment is a normal Azure footprint with a chatty front end:
- Foundry project and model deployments
- Agent instructions and tools
- Identity and access
- Search indexes and knowledge sources
- Storage, Key Vault, monitoring
- Private networking and, when the boundary needs it, an API gateway
Treat it like a workload, because that is what it is.
If your DR plan is “rebuild it in the portal”, you are not production-ready. Same if you cannot recreate the environment from source control.
I put the shared platform in Terraform and keep agent behaviour in Git. Foundry does not get a free pass because the blade looks modern.
A practical Azure AI Foundry production readiness baseline
Skip the six-month transformation theatre. You need a minimum bar before real users and anything business critical show up.
| Area | Minimum bar |
|---|---|
| Environments | At least dev and prod. Add pre-prod if the workload is sensitive. |
| Identity | Managed identities at runtime. No shared keys in notebooks, chat logs or Teams. |
| Data boundary | Written list of what the agent can read and call. |
| Change control | Instructions, model choice, tools and safety config in Git. Portal save is not a prod release. |
| Promotion | A small set of golden questions before prod. Regressions do not get promoted. |
| Observability | Logs for failures, latency, tool errors and token spend. |
| Ownership | Named owners for platform and agent behaviour. |
| Recovery | You can redeploy last known good without archaeology. |
Minimum bar
- Environments
- At least dev and prod. Add pre-prod if the workload is sensitive.
- Identity
- Managed identities at runtime. No shared keys in notebooks, chat logs or Teams.
- Data boundary
- Written list of what the agent can read and call.
- Change control
- Instructions, model choice, tools and safety config in Git. Portal save is not a prod release.
- Promotion
- A small set of golden questions before prod. Regressions do not get promoted.
- Observability
- Logs for failures, latency, tool errors and token spend.
- Ownership
- Named owners for platform and agent behaviour.
- Recovery
- You can redeploy last known good without archaeology.
Not every agent needs APIM on day one. Every agent that matters needs identity, a data boundary, change control and a named owner.
If most of that table is red, you still have a demo with ambitions.
Security first, in the right order.
Most teams start with “can we make the agent work?” I start with “can we make it work without leaking the estate?”
I push the work in this order:
- Identity and data boundaries. Which identity does the agent run as? What can it reach? Who can call it? This is where the damage happens.
- Network posture. Public with tight auth can be fine for an internal pilot. Private endpoints earn their keep on sensitive data or locked-down networks. Choose that deliberately. Do not keep the playground default forever because nobody revisited it.
- Change control for prompts and models. A portal edit in prod is an unreviewed release. Say it out loud in the design authority if you have to.
- Monitoring for failures, odd traffic, misuse and cost spikes.
I will not take “we trust the model” into a design authority. Trust is not a control. Architecture is.
The further you go, the more familiar it looks: managed identities, RBAC, Key Vault, logging, Defender and a proper access path. Boring on purpose.
Prompt changes deserve the same governance as code
Your app teams already know the path. PR, review, pipeline, approval, rollback. Then the same organisation rewrites live agent behaviour in a Foundry text box on a Thursday afternoon.
That should make you uncomfortable.
A system prompt is not a harmless setting. It defines runtime behaviour. Change the instructions and you can change outcomes, tone, tool use and what data gets surfaced. That is code. It just happens to be English.
So put prompts, tools, model choice and safety config in Git. Review them. Test them. Ship them through a pipeline.
A split that has worked well for me...
| Change Type | Where it lives | Who merges |
|---|---|---|
| Network, Foundry account, shared monitoring | Platform IaC | Platform |
| Agent instructions, tools, guardrails | Agent config repo | AI / product |
| Model SKU and quotas | Decide up front | Joint |
| Gateway policy and auth | APIM / gateway config | Platform + AI |
Where it lives
- Network, Foundry account, shared monitoring
- Platform IaC
- Agent instructions, tools, guardrails
- Agent config repo
- Model SKU and quotas
- Decide up front
- Gateway policy and auth
- APIM / gateway config
Who merges
- Network, Foundry account, shared monitoring
- Platform
- Agent instructions, tools, guardrails
- AI / product
- Model SKU and quotas
- Joint
- Gateway policy and auth
- Platform + AI
I care less about perfect repo taxonomy than the rule underneath. If a change can alter production behaviour, it uses the same gate as application code.
The risks that catch Azure teams out
Even teams who are solid on App Service or AKS walk into the same traps with agents.
No validation gate is the big one. If you cannot re-run a handful of known questions before prod, you are promoting on vibes.
Data and permission drift sneaks in next. Dev starts on a softer dataset, someone retargets a connection, and the agent can suddenly see more than the design allowed.
Cost is the line item people undercook. Tool loops, chatty retrieval and fat context windows burn money faster than the compute you remembered to watch.
Playground versus real path is another classic. Works in Foundry. Falls over behind app auth, APIM or the network route users actually take.
Ownership is the quiet one. Platform owns the subscription. Nobody owns “the answers got worse after last week’s tweak.”
None of that needs a transformation programme. It needs a written decision and a name against it.
What to do next
Still evaluating Foundry? Keep the first slice thin: managed identity, non-prod data, instructions in Git. Prove value before you build the private network cathedral.
PoC stuck in the portal? Export instructions, tools, model choices and connections into source control this week. Stand up a second environment so prod is not your laboratory.
Already serving users? Lock down who can edit prod agents, add a minimal eval gate on promote, and make sure logs can answer what changed and what users hit.
Four moves for this week:
- Write down scope: data, tools, users, environments.
- Move prod instructions and model config into Git with PR review.
- Check runtime identity and portal RBAC. Kill shared keys.
- Define last known good and how you restore it.
Then answer the three questions that separate a demo from a platform:
- What is live right now?
- Who changed it?
- How do we roll back?
If those answers are fuzzy, fix that before you chase the next model upgrade.
How are you handling Foundry changes today: portal edits, or the same path you use for everything else in production?
