How to build a production-scale AI agent

What a production AI agent actually is

A chatbot answers; an agent finishes. That one distinction drives everything else in this article. A chatbot returns text and leaves the work to a human. An agent takes a goal, reasons over your business rules, calls tools, writes the result into your systems of record, and involves a human only at the approval points you define.

The demo version of an agent needs a model, a prompt and a few tools. It will work convincingly on the happy path, in front of an audience. The production version also needs rails, integrations and an improvement loop, because production is where the malformed PDF and the surprise provider API change live. The demo is the easy 20%. This article is about the other 80.

To keep things concrete, we will build one agent end to end: a hypothetical invoice-processing agent for a mid-market finance team. Every figure in the worked examples below is illustrative, not a measured result. We picked this workflow because every B2B company runs it and it exercises all five layers. The same frame carries an insurance claims agent, a healthcare billing agent or an underwriting triage agent without modification. Only the knowledge, rules and tools change.

The AI agent architecture, in five layers

Read Fig. 1 top to bottom. That is also the path a single request takes:

• 01 · Channels. How work arrives: email, voice, chat, documents. The agent meets work where it already lands instead of forcing a new inbox on anyone.
• 02 · Agent core. An orchestration layer routing between LLMs, tools, short and long-term memory, and an agent wiki: a versioned knowledge base holding your business rules, retrieved at runtime.
• 03 · Production rails. Checks that run on every request, not in a test suite: guardrails for scope and policy, runtime checks on quality, and a human-in-the-loop gate wherever an action is consequential.
• 04 · Your systems. Where actions land: CRM, ERP, claims or policy admin, the warehouse. Results have to reach these systems, or nothing was automated.
• 05 · Improvement loop. Every model call, tool call, token and cent is traced. Every run is scored after it completes, against targets set before launch. The fixes flow back into the core as better rules, prompts and wiki pages.

The seven build steps below walk these layers in order. Steps 1 to 3 build what your demo probably already has. Steps 4 to 7 are the production layers, and if you already run an agent, that is where your gaps are.

Building it, step by step

Step 1 · Layer 01 · ChannelsMeet the work where it lands

Our invoice agent's work arrives three ways: supplier emails with PDF attachments, a supplier portal upload, and a monthly machine-to-machine feed (EDI) from one large vendor. The design rule is to normalize at the edge. Whatever the channel, everything becomes the same internal envelope before the core sees it: source, timestamp, raw payload, extracted attachments. Channel quirks handled at the boundary never leak into agent logic.

Step 2 · Layer 02 · OrchestrationStructure before intelligence

Resist the single mega-prompt. Our invoice agent is really four small agents behind a coordinator: a router (what kind of document is this?), an extractor (pull the header, line items and totals), a matcher (find the purchase order this invoice bills against, and check the two agree within tolerance), and an exception handler (decide what a human needs to see). Each has its own instructions, its own tools, and its own failure modes you can test in isolation.

The orchestration layer's real job is state. If the ERP is down for two minutes, a production run checkpoints and resumes rather than starting over. One caveat that bites almost everyone: resuming replays the step that failed, so the write inside it must be idempotent (Step 5 covers how). Checkpointing stops the rework; idempotency stops the double post. You need both, for different reasons. Route models per task while you are here. Extraction does not need your most expensive model, and exception reasoning probably does.

Step 3 · Layer 02 · KnowledgeMemory and the agent wiki

Two different things get called "memory." Short-term memory is run state: what has been extracted so far, which tools were called. Long-term memory is cross-run knowledge: this vendor always sends the PDF a day after the goods arrive, that vendor's line descriptions never match the PO wording.

Separate from both is the agent wiki: a human-readable, versioned knowledge base of your business rules. Ours holds pages like "matching tolerance: price within 2% or ₹500, quantity exact" and "shipping charges: match against the standing PO that covers freight" (a standing PO is an ongoing blanket order for recurring charges). Because it is plain text under version control, the finance manager can read it, correct it, and see exactly what changed when the agent's behavior changes. That readability is what lets the finance team audit the rules, and audit is where their trust comes from.

The wiki is indexed by an LLM, and the timing matters. At design time, a model reads every page and writes the index: summaries, cross-links, and retrieval hints that tell a future run where to look. At runtime, the agent retrieves through that index instead of searching raw text. Embeddings can be part of the index, but they are not the index. When a rule changes, the index is rebuilt from the wiki, so the wiki stays the single source of truth and retrieval stays cheap and predictable.

Step 4 · Layer 03 · Production railsChecks on every request

This is the first layer your demo almost certainly lacks, and it is the layer that decides whether the agent survives real traffic. Three rails, all inline:

• Guardrails bound what the agent may do at all: which vendors, which amounts, what is out of scope entirely. An invoice from an unknown vendor never auto-posts, full stop.
• Runtime checks measure quality during the run: extraction confidence, whether every extracted value actually appears in the source document, latency and cost budgets. A run that scores below target does not proceed. It degrades to the human queue instead.
• The human-in-the-loop gate is a design decision, placed where consequence lives. Our invoice agent auto-posts within-tolerance invoices under a threshold (the auto-post ceiling) and queues everything else with its reasoning attached: the match it found, the rule it applied, the confidence it computed. The reviewer approves in seconds because the agent already did the reading.

One honest nuance on inline checks: they cost latency and tokens. Back-office work like invoice processing is asynchronous, so it can afford heavyweight checks on every run. A latency-sensitive agent gates on cheap deterministic checks inline and pushes the expensive scoring to the post-run loop instead.

Step 5 · Layer 04 · Your systemsWhere actions land

The agent's output is a write, not a message: a posted invoice in the ERP (the company's system of record for money and operations), an updated vendor record, an acknowledgment email. Integration is where agent projects quietly die, so it deserves classical engineering discipline. Make every write idempotent: use provider idempotency keys where the API supports them, and external-ID dedup or query-before-post where it does not, because most ERP APIs do not take a key. Surface every failure loudly; a tool that fails silently is the most dangerous component in the whole system. Scope credentials per tool, least privilege.

Step 6 · Layer 05 · ObservabilityIf you can't replay it, you can't run it

Every model call, tool call, retrieval, token count and cent, per run, queryable. The payoff comes the first time the agent does something surprising: the difference between a five-minute diagnosis and a lost week is whether you can replay the exact run: what it retrieved, what the model saw, what the tool returned, what it cost.

Cost observability doubles as an early-warning system. A prompt regression that doubles token use shows up in traces on day one, not on the invoice at month end.

Step 7 · Layer 05 · Improvement loopThe part that compounds

After every run, three things happen. The run is scored against targets you committed to before launch: extraction accuracy, auto-post precision, cost per invoice, human-touch rate. Failures are diagnosed from traces rather than anecdotes. And the fix is fed back as a concrete artifact: a wiki rule tightened, a prompt versioned up, a check target raised, a failing case pinned as a regression test so it cannot return unnoticed.

To be precise about the name: this is an improvement loop driven by scored experience, not reinforcement learning in the machine-learning sense. No weights change. What changes is the agent's knowledge, rules and prompts, weekly, with evidence. Run this loop and the agent your team uses in week four should be measurably better than the one that shipped in week one. Skip it and the agent quietly gets worse as its inputs drift.

Multi-agent systems: when one agent isn't enough

The honest default is one agent. If a single agent with three tools does the job reliably, ship that. Split when one prompt starts doing several jobs, and you will recognize the moment: the instructions grow contradictory ("be thorough" for extraction, "be decisive" for routing), tools multiply past what one context handles well, and a failure in one concern takes down the others.

The pattern that works in production is a coordinator plus specialists, which is the shape our invoice agent already has. The coordinator owns state and sequencing; each specialist owns one competence. Three rules keep it sane. Every hand-off is typed and traced, so nothing passes between agents as vibes. State lives in the orchestration framework's state object, not in any agent's context window. And each specialist can be evaluated and improved on its own. Treat multi-agent design as an ordinary engineering decomposition.

What actually breaks in production

These are the four boundaries we pressure-test when we assess an agent, because they account for most post-launch failures, and none of them show up in a demo. A demo runs one request.

• Tool invocation. Malformed calls under unusual inputs; integrations and MCP servers that fail silently, so the agent proceeds on a phantom success. The invoice version: the ERP times out, the retry has no dedup, the invoice posts twice and a supplier gets paid double.
• Model-API compatibility. Provider models and APIs change underneath you, and behavior shifts without a code change on your side. Pin versions, run compatibility checks before upgrading, and never auto-track "latest" in production. Budget for forced migrations too, since providers retire pinned snapshots on their own schedule.
• Orchestration state. Works for one request, drifts under fifty concurrent ones: race conditions, lost checkpoints, runs that resume from the wrong step. This is the layer where pilots fail to become products.
• Retrieval and tracing. The agent answers confidently from a stale wiki page, and without traces nobody can reconstruct which knowledge it saw. Retrieval bugs look identical to model errors until you read the trace.

The order of this article is the defense. Rails catch the first, pinning and checks the second, real orchestration the third, and the improvement loop the fourth. We run structured production-readiness assessments against exactly these four boundaries.

Score your agent against this checklist

FAQ