Every AI agent demo ends the same way: a confident answer, a slick tool call, applause. What the demo never shows you is the part that decides whether the thing is safe to put in front of a patient, a policyholder, or anyone else whose problem is not hypothetical.
I work with mid-sized companies in Europe on exactly this transition, and a fair number of them sit in regulated corners — healthcare, legal, finance — where a confident wrong answer stops being an embarrassment and becomes a breach. When a leader in one of those firms has watched the vendor demo and asks me what “in production” actually costs, I don’t reach for a benchmark. I reach for a short list of four disciplines, because every serious deployment I have studied or worked near converges on the same four, and every failed one skipped at least one of them.
This is that list, written as a playbook. If you are building — or commissioning — an AI agent for a healthcare or otherwise regulated business, these are the points you need to take care of, and what a responsible first deployment looks like.
None of it is theory. The clearest public proof that this shape works is a US digital-healthcare deployment whose engineering team spoke about it with unusual candour on a podcast — I’ll come back to it once, by name, at the end, because it is worth your listening time. Everything until then is the abstracted discipline, because the point is not their story. The point is what you should demand from yours.
Discipline one: a router, not a genius
The intuitive way to build a capable agent is to make one very capable agent and hand it every tool it might ever need. Production teams learn to do close to the opposite.
The shape that survives contact with real users is a small front-door agent that answers nothing itself. It reads what the user wants and hands the request to exactly one narrower specialist — one for appointments, one for search, one for account questions, one for the domain’s knowledge. Each specialist has a bounded remit and a handful of tools. Across the whole system you want a tool count in the low tens, not the hundreds; tool overload is one of the better-documented ways to make an agent worse, because the more near-identical choices a model can see at once, the more often it picks the wrong one.
Two boundary rules matter more than any cleverness inside the boxes. First, your existing APIs become the agent’s tools — you do not invent a parallel surface for the model. Second, identity and application state are injected by code, never left for the model to decide. The agent must never get to choose whose data it is looking at. In a healthcare deployment that single rule is carrying a remarkable amount of your compliance story.
The request path to copy: a user’s message hits deterministic guardrails first; only then does a lead agent route it to one of a few bounded specialists. Identity and account state are supplied by code, not chosen by the model.
If you have ever watched a demo where a single agent confidently juggles everything, this is the quiet correction. The production shape is smaller, more boring, and far easier to reason about. You can point at where any given request goes and why — which is the first thing an auditor or a nervous board member will ask you to do.
Discipline two: the gates run before the agent decides anything
Look again at that diagram and notice the order. The safety checks do not sit inside the clever part of the system. They sit in front of it.
A serious deployment runs hard, deterministic guardrails before any routing happens. The gravest example sets the pattern: in a healthcare context, language that suggests self-harm should route straight to human support and never reach the agent at all. That is not the model showing good judgement. It is the system refusing to let the model near the decision in the first place. The same logic extends to your data boundary — a closed environment that sensitive records never leave — and to a standing human review of conversation samples for bias and drift.
For a European regulated firm, this is the part to copy exactly, because it is the part that is easiest to skip. A great deal of AI-safety energy goes into making the model behave: better prompts, better instructions, better fine-tuning. The production posture assumes the model will misbehave anyway, and puts the non-negotiable rules where the model cannot reach them. The GDPR-sensitive lookup, the professional-secrecy boundary, the category of question that always goes to a human: those are deterministic gates that run before the agent thinks, not hopeful sentences buried in a prompt.
If your compliance officer can only see safety expressed as instructions to a language model, you do not yet have a control. You have a wish.
Discipline three: every failure becomes a test that cannot break the same way twice
The least glamorous work in an agent deployment is reading production conversations. It is also where reliability actually comes from.
The method is simple to state. When the agent gets something wrong in production, that specific failure does not get quietly patched and forgotten. It becomes a permanent regression case — a test the system has to pass forever. Some of these are almost embarrassingly plain: an agent that misstates who built it becomes a string check on the output. Tool calls are verified the same deterministic way, because whether the agent called the right function with the right arguments is an exact, checkable fact.
Not everything reduces to a string. Whether a health answer is clinically sound, or the tone right for someone who is anxious, cannot be checked with a plain rule. For those you use language models as judges — and then do the step most teams skip: calibrate those judges against human labels, so the automated grader tracks what a clinician would say rather than drifting off on its own. The suite that grows out of this is also what you still own when the model underneath it changes.
The reliability loop: a production failure becomes a permanent regression case, checked by code where it can be and by a calibrated LLM judge where it cannot. The release bar is then set by the consequence of getting it wrong.
There is a hard-nosed economic decision hiding inside this loop, and it belongs to you, not the engineers. For low-stakes capabilities, nine passes out of ten may be genuinely good enough to ship — chasing the tenth is effort your team could spend elsewhere. For high-stakes capabilities, nine out of ten is a scandal. A mature team does not chase perfection everywhere; it decides, deliberately and in writing, where “good enough” sits for each thing the agent is allowed to do.
Discipline four: ship a capability, not “the agent”
The first three disciplines are engineering. The fourth is the one a leader personally owns, because it is a governance posture disguised as a rollout plan.
Never make a single yes-or-no decision about whether “the agent” is ready. Make a separate decision for each capability, with the bar set by the consequence of getting that capability wrong. What that looks like in practice:
The launch goes to a small cohort first — a fifth of your users, not all of them — with someone actually reading the conversations that come back. The capability with the most riding on it stays switched off at launch, deliberately, until it clears a much higher bar. In healthcare that is typically the money question: anything touching a member’s benefits or coverage, where a wrong answer can push someone toward a decision worth tens of thousands of euros. And some capabilities are never granted at all — crisis language is not a feature waiting on a higher test score, it is a category the agent is permanently not allowed to attempt.
Done right, the visible milestone a few months in is not “the AI is live.” It is a wider cohort plus one more capability that has finally earned its place. “Is the agent ready to launch” is the wrong question, and it is the question every demo is quietly designed to make you ask. The question that earns its keep is which specific things the agent is allowed to do so far, and how each of those calls was made. You can hold a vendor or an internal team to that without writing a line of code.
The bill nobody shows you: latency
There is an honest objection to everything above, and I would rather raise it than have you discover it in week one: every discipline on this list costs time. A routing hop is a model call. A specialist is another. The gates run before any of it, the tools run between, and your patient is standing there watching a spinner. The demo answered instantly precisely because it skipped all of this. So where does the time actually go — and what do serious teams do about it?
Less goes where you would fear, and more where you would not. The deterministic gates are effectively free: a hard pattern check costs a fraction of a millisecond — one independent benchmark puts fast safety classifiers at a few hundredths of a second, while a regex check measures in microseconds, against roughly half a second for a managed guardrail service doing similar work. And the reliability loop costs the user nothing at all, because the expensive part never sits in the request path: the calibrated judges grade logged conversations afterwards, offline, on samples. What genuinely costs is the model-call chain itself. Every hop from router to specialist to synthesis is a full round trip, and generating the answer dominates the clock.
The production toolkit for clawing that time back is well established, and you can ask any vendor which parts of it they use. Guardrails run in parallel with generation, not before it — the checking and the answering start at the same moment, and the check interrupts the answer only on a violation; the industry’s name for it is optimistic execution. The stable part of every request, the system prompt and the tool definitions, gets cached, which the providers report cuts latency on long prompts by well over half. Simple intents route to small, fast models, with the heavyweight reserved for the turns that deserve it. And everything streams, because the metric a user actually feels is time-to-first-word, not time-to-last.
Then there is the part of the latency story that is not engineering at all. People do not hate waiting; they hate unexplained waiting. In a classic Harvard study, people preferred the slower of two websites when it showed the work it was doing while they waited. Chatbots that take visibly longer on harder questions get rated more human and more satisfying than instant ones, and in a recent study identical answers were judged more thoughtful when they arrived more slowly. Do not read that as a licence for theatre — the researchers who found it warn against faking slowness, and so do I. But it means “checking your coverage details…” written honestly across a two-second pause is not a failure of the product. In a healthcare context it may be the most trust-building sentence your agent produces.
Which brings latency back to being a governance question, owned by the same person who owns discipline four. Set a latency budget per capability, in writing, next to its release bar: the simple path should answer in about a second; the consequential path is allowed its gated seconds and should say what it is doing with them; and the highest-stakes tier’s honest latency is a human, measured in minutes and framed as exactly that. Then hold the budget with the same discipline as the safety bar — because the documented failure mode runs in the opposite direction to the one you fear. Teams rarely lose users to a two-second answer. They lose the safety architecture: latency creeps, someone bypasses a gate “temporarily” to rescue the experience, and the postmortem writes itself. A slow, safe answer is a product decision. A fast, wrong one is a breach.
The public proof, and where to hear it
I promised one named example, and it has earned the mention. Maven Clinic — a US women’s-and-family digital-healthcare platform — took a member-facing assistant through precisely this shape: a routing agent over a handful of bounded specialists, deterministic guardrails ahead of the model, failures converted into permanent regression tests with calibrated LLM judges, a twenty-per-cent launch cohort, and the benefits-question capability held back until it had earned its way in. Their staff AI engineer, William Horton, told the whole story on the Vanishing Gradients podcast, including the four-month follow-up — and it is brilliant listening, precisely because it is the rare public account that shows the discipline rather than the demo. I’d rather send you to the source than retell it: what you have just read is the generalised playbook; that episode is what it sounds like when a team lives it.
What to ask before you believe the next demo
You do not need to understand the engineering to run this playbook. You need to ask four questions and listen for whether the answer is a design decision or a hopeful gesture.
Ask where requests actually go. A team in control can draw you the router and the handful of specialists behind it on a whiteboard, in about a minute. A team that cannot will describe one clever agent that does everything, which is the demo talking.
Ask what runs before the model gets involved. You want names here: the categories that go straight to a human, the data boundaries enforced in code, the checks nobody can prompt their way around. A very well-behaved prompt is not an answer to that question.
Ask what happens to a failure. The answer you want is that a production mistake becomes a permanent test the system re-runs forever, with cheap checks in code and calibrated judges for everything else. “We fixed it and moved on” means the same failure is free to come back next week.
Then ask the question that outranks the other three: which capabilities have shipped, which are deliberately still held back, and what has to be true before each one goes live. A team that can answer that is running a governance posture. A team that cannot is running a demo, and hoping you will not notice which of the two you bought.
The capability in these systems is real, improving fast, and genuinely useful in a regulated business. The part that decides whether it is safe to put in front of your customers is the governance around it, and it is the part most teams — and most demos — quietly leave out. Take the four questions into your next vendor meeting and watch which of them gets a straight answer. That conversation, once the demo is over and the real deployment begins, is the work I do with clients.
The one named case in this piece, Maven Clinic’s member-facing assistant, is drawn entirely from the team’s own public account: staff AI engineer William Horton on the Vanishing Gradients podcast and its four-month production follow-up, plus the company’s launch announcement in the healthcare trade press. Go listen — it is the best public reference deployment I know of.
