Building AI Agents for Production: Tools, Guardrails and Evaluation

To build an AI agent for production, treat the model as an untrusted planner inside a loop you control. Give it a small set of narrowly scoped tools with strict schemas, validate every output, enforce limits and approvals in code, log every step, and measure behaviour with an evaluation suite before and after every change. Most agent failures are ordinary software failures — missing validation, unbounded loops, vague interfaces — not model failures.

Do you actually need an agent?

An agent is an LLM that chooses its own next step. That flexibility costs you predictability, latency and money. Before building one, ask whether the task is really open-ended. Extracting fields from an invoice, classifying a ticket, or answering a question from documents usually fits a workflow: a fixed sequence of LLM calls and code. A retrieval pipeline like the one in my guide to building production RAG systems with LangChain is a workflow, and that is a feature, not a limitation.

Agents earn their place when the number and order of steps depends on what the model discovers along the way: investigating a support case across several systems, exploring a database schema to answer an analytics question, or triaging an incident. A useful rule: if you can draw the flowchart, build the flowchart.

The agent loop, without the magic

Every agent framework wraps the same loop. Writing it yourself once is the fastest way to understand where to put guardrails and logging.

type Step = { tool: string; args: unknown; result: unknown; ms: number };

async function runAgent(task: string, ctx: RunContext): Promise<AgentResult> {
  const messages: Message[] = [system(ctx), user(task)];
  const steps: Step[] = [];

  for (let i = 0; i < ctx.limits.maxIterations; i++) {
    const res = await llm.chat({ messages, tools: toolSchemas, timeoutMs: 30_000 });
    ctx.budget.charge(res.usage);                  // throws if over spend cap
    messages.push(res.message);

    if (res.stopReason !== "tool_use") {
      return finalize(res.message, steps);         // validate final answer
    }

    for (const call of res.toolCalls) {
      const started = Date.now();
      const result = await executeTool(call, ctx); // validation + permissions inside
      steps.push({ tool: call.name, args: call.args, result, ms: Date.now() - started });
      messages.push(toolResult(call.id, truncate(result, 4_000)));

      if (result?.status === "pending_approval") { // stop; resume when a human approves
        await runs.save(ctx.runId, { messages, steps, ticketId: result.ticketId });
        return { status: "pending_approval", ticketId: result.ticketId, steps };
      }
    }
  }
  return { status: "gave_up", steps };             // never loop forever
}

Notice what lives outside the model: the iteration cap, the budget, timeouts, truncation of tool output, and the explicit "gave up" state. Those are the parts that keep an agent from becoming an incident.

Tool design is API design

The model only knows your system through tool names, descriptions and schemas. Treat them like a public API with a very literal-minded consumer. The principles from designing REST APIs properly apply almost directly.

• Prefer task-shaped tools over generic ones. get_order_status(order_id) is safer and easier for the model than run_sql(query).
• Write descriptions for the model. Say when to use the tool, when not to, and what the output looks like.
• Constrain arguments. Enums, formats, min/max — every constraint is one less thing the model can get wrong.
• Return compact, structured results. Strip fields the model doesn't need; large payloads burn tokens and distract.
• Return errors as data. "Order not found; ask the user to confirm the ID" lets the model recover. A stack trace doesn't.
• Keep the toolset small. Past roughly 10–20 tools, selection accuracy tends to drop; split into sub-agents or route first.

import { z } from "zod";

export const refundTool = {
  name: "create_refund_request",
  description:
    "Create a refund REQUEST for a delivered order. Does not move money; a human approves it. " +
    "Use only after get_order_status confirms status=delivered.",
  input: z.object({
    orderId: z.string().regex(/^ORD-\d{8}$/),
    amountCents: z.number().int().positive().max(50_000),
    reason: z.enum(["damaged", "not_received", "wrong_item", "other"]),
  }),
  sideEffect: "write" as const,   // used by the approval guardrail
};

Structured outputs everywhere

Free text is fine for the user-facing reply, but anything your code consumes should be structured. Most major model APIs now support JSON-schema-constrained output or strict tool calling; use it. Then validate anyway — schema compliance doesn't guarantee semantic correctness.

const FinalAnswer = z.object({
  answer: z.string().max(2_000),
  citations: z.array(z.object({ source: z.string(), quote: z.string() })).min(1),
  confidence: z.enum(["high", "medium", "low"]),
  needsHuman: z.boolean(),
});

function finalize(msg: Message, steps: Step[]) {
  const parsed = FinalAnswer.safeParse(extractJson(msg));
  if (!parsed.success) return { status: "invalid_output", steps, error: parsed.error.issues };
  if (parsed.data.needsHuman || parsed.data.confidence === "low") return escalate(parsed.data, steps);
  return { status: "ok", output: parsed.data, steps };
}

This pattern — the model returns a typed object, code decides what happens next — is the same one behind text-to-SQL systems, where the generated query is parsed and checked before it ever touches a database. The Audience Craft text-to-SQL case study walks through a real example of putting validation between an LLM and production data.

Guardrails that actually hold

A system prompt saying "never delete data" is a suggestion. Real guardrails are enforced by the runtime, so they hold even when the model is confused or the input contains a prompt injection.

Input guardrails

• Treat retrieved documents, emails and web pages as untrusted data. Mark them clearly and never let them change tool permissions.
• Classify or filter obviously out-of-scope or abusive requests before the expensive loop starts.
• Scope data access to the requesting user — pass tenant and user IDs from the session, never from model arguments.

Action guardrails

• Least privilege: read-only database roles for read tools, separate credentials per tool.
• Allow-lists for destinations (domains, email recipients, tables).
• Idempotency keys on write tools so a retried step can't double-charge or double-send.
• Hard caps: iterations, total tokens, wall-clock time, and cost per run and per user per day.

Output guardrails

• Schema validation and business-rule checks on the final answer.
• Grounding checks: does every claim have a citation from the retrieved context?
• PII and secret scanning before anything is shown or sent.

Human-in-the-loop by design

The simplest, most effective guardrail is a human approving consequential actions. Design for it from day one rather than bolting it on: the agent proposes, a person approves, code executes.

async function executeTool(call: ToolCall, ctx: RunContext) {
  const tool = registry.get(call.name);
  if (!tool) return { error: `Unknown tool ${call.name}` };

  const args = tool.input.safeParse(call.args);
  if (!args.success) return { error: "Invalid arguments", issues: args.error.issues };

  if (tool.sideEffect === "write" && !ctx.policy.autoApprove(tool.name, args.data)) {
    const ticket = await approvals.create({ runId: ctx.runId, tool: tool.name, args: args.data });
    return { status: "pending_approval", ticketId: ticket.id }; // runAgent saves state and stops here
  }
  return tool.run(args.data, ctx);
}

To pause and resume, persist the message history and step log with the run ID (PostgreSQL is fine), and continue the loop when the approval arrives. Start with approval on every write, then relax specific, low-risk actions once your logs show they are consistently correct.

Controlling cost and latency

Agents multiply calls: one user request can become five to fifteen model round-trips. Treat the numbers below as rough rules of thumb, not benchmarks — they vary widely by model and provider.

Technique	Effect	Trade-off
Model routing (small model for simple steps)	Often the largest cost reduction	Needs evals to prove quality holds
Prompt caching of stable prefixes	Lower input cost and time-to-first-token	Keep system prompt and tools stable and first
Truncating / summarising tool output	Smaller context, better focus	Risk of dropping a needed detail
Parallel tool calls	Lower wall-clock time	Only for independent, read-only calls
Streaming + progress events	Better perceived latency	More frontend work
Iteration and spend caps	Bounded worst case	Some hard tasks end as "gave up"

Long-running agents should not hold an HTTP request open. Put runs on a queue, process them in workers, and push progress to the client over SSE or WebSockets — the same patterns covered in backend architecture best practices.

Observability: trace every run

You cannot debug an agent from a final answer. For every run, record a trace with: the prompt version, model and parameters, each model call (tokens, latency, stop reason), each tool call (arguments, result size, duration, error), guardrail decisions, approvals, and the final status. OpenTelemetry spans with one span per model call and per tool call work well, and most LLM observability tools ingest them.

Dashboards worth having from week one:

• Runs by final status (ok, escalated, invalid_output, gave_up, error).
• Cost and latency per run, at p50 and p95, broken down by prompt version.
• Tool error rate and the most common invalid-argument errors.
• Average steps per successful run — a rising number usually means a regression.

Evaluation: the part teams skip

Every change — a new model, a reworded tool description, a prompt tweak — can silently change behaviour. An eval suite is your regression test.

1. Build a dataset of 50–200 realistic tasks from real (anonymised) traffic, including edge cases and adversarial inputs.
2. Score outcomes with deterministic checks where possible: correct record updated, correct value returned, valid schema.
3. Score trajectories: required tools called, forbidden tools not called, step count within budget.
4. Use an LLM judge for open-ended answers, with a written rubric, and spot-check it against human ratings.
5. Run in CI on every prompt, tool or model change, and track pass rate over time.

for (const c of evalCases) {
  const run = await runAgent(c.input, testContext({ fixtures: c.fixtures }));
  report.add(c.id, {
    passed: c.check(run.output),
    usedRequiredTools: c.requiredTools.every(t => run.steps.some(s => s.tool === t)),
    avoidedForbidden: !run.steps.some(s => c.forbiddenTools.includes(s.tool)),
    steps: run.steps.length,
  });
}

Common failure modes and fixes

• Looping on the same tool: detect repeated identical calls and return a "you already tried this" result.
• Hallucinated tool arguments: stricter schemas, enums, and a lookup tool so the model can find real IDs.
• Context overflow: truncate tool results and summarise older turns.
• Prompt injection via retrieved content: runtime permissions plus approval on writes.
• Confident wrong answers: require citations and route low-confidence results to a human.
• Silent regressions after a model upgrade: pin model versions and gate upgrades on the eval suite.

FAQ

What is the difference between an AI agent and a regular LLM call?

A regular LLM call maps one input to one output. An agent runs a loop: the model decides which tool to call, your code executes it, the result goes back to the model, and the loop repeats until the model produces a final answer or a limit is hit. The model controls the control flow, which is what makes agents powerful and what makes them harder to make reliable.

Should I use an agent framework or write the loop myself?

For a first production agent, a hand-written loop of roughly 50 to 100 lines is often easier to debug, test and observe than a large framework. Frameworks such as LangChain or LangGraph become worthwhile when you need durable state, branching graphs, or many integrations you would otherwise rebuild yourself.

How do I stop an AI agent from taking dangerous actions?

Enforce safety in code, not in the prompt. Give tools the narrowest permissions possible, validate every argument with a schema, run write actions through an approval step, cap iterations and spend, and keep an audit log of every tool call. The prompt can describe the rules, but the runtime must enforce them.

How do you evaluate an AI agent?

Build a dataset of realistic tasks with expected outcomes, then score both the final answer and the trajectory: which tools were called, with which arguments, and how many steps it took. Combine deterministic checks with an LLM-as-judge for open-ended answers, and run the suite in CI whenever prompts, tools or models change.

How can I reduce the cost and latency of an agent?

Use smaller models for routing and simple steps, cache stable prompt prefixes, trim tool outputs before returning them to the model, run independent tool calls in parallel, and cap the number of iterations. Often the biggest win is replacing an agent with a fixed workflow for paths that are predictable.

Next steps

If you're taking an agent from prototype to production and want a second pair of eyes on tool design, guardrails or evaluation, see my AI engineering services or get in touch.