Production Agent Architecture for Real Applications
How to design AI agents with tool calling, tracing, guardrails, retries, and human approval instead of shipping an unreliable prompt loop.
The Problem
An agent demo is easy: ask a model to call a tool and return an answer. Production is harder. The agent needs permissions, timeouts, traceability, retries, evaluation, and a way to stop before it performs expensive or risky work.
Why It Matters
Agentic systems now sit between users and real business operations. A broken tool call can create tickets, update records, send emails, or leak data. Treat the agent as a service boundary, not a chat box.
Core Concepts
A production agent has a model, tools, state, policy, and observability. Tools should be typed, permissioned, and narrow. State should be explicit: short-term conversation context, durable task state, and auditable events are different things. Guardrails validate inputs and outputs. Tracing records each model turn and tool call so failures can be debugged.
Implementation
Start with a small orchestration loop:
type ToolCall = {
name: string;
args: unknown;
risk: "low" | "high";
};
async function runAgent(taskId: string, input: string) {
const trace = await traces.start({ taskId, input });
const plan = await model.plan(input, { tools: toolSchemas });
for (const call of plan.toolCalls as ToolCall[]) {
if (call.risk === "high") await approvals.require(taskId, call);
const result = await tools.execute(call.name, call.args, { timeoutMs: 8000 });
await trace.recordToolCall(call, result);
}
return model.finalize({ taskId, traceId: trace.id });
}
The important choice is not the syntax. The important choice is that every tool call passes through a runtime you control.
Common Mistakes
- Giving the model broad tools like
runSql(query)orsendEmail(to, body). - Storing hidden state only in the prompt.
- Retrying non-idempotent tool calls without an idempotency key.
- Shipping without trace logs for model decisions and tool results.
Production Considerations
Add budgets per task: maximum model turns, maximum tool calls, maximum spend, and maximum wall-clock time. For long jobs, persist a task record with status, intermediate outputs, and cancellation support.
Security
Authorize the user before each tool call. The model can propose an action, but your service decides whether the caller is allowed to perform it.
Performance
Cache stable retrieval results, keep tool outputs compact, and prefer deterministic tools for calculations. Long context is slower and more expensive than good state design.
Summary
Production agents need service engineering: typed tools, approvals, idempotency, state, budgets, tracing, and evaluation. The prompt is only one component.
The weekly engineering digest
Production-grade engineering writing in your inbox. No spam, unsubscribe anytime.