How to Evaluate LLM and RAG Applications in Production

Short answer: evaluate LLM and RAG applications in layers. Build an offline evaluation set from real questions, measure retrieval separately from generation (recall@k, MRR), score answers for correctness and groundedness using deterministic checks plus carefully validated LLM judges, run the suite as a regression gate in CI, and monitor live traffic for drift, failures and user feedback. Treat it like testing any other software, with the twist that outputs are probabilistic.

Why "it looked good in testing" isn't enough

LLM features fail quietly. A prompt tweak that fixes one case breaks three others. A model upgrade changes output formatting. A new batch of documents shifts retrieval results. Without a repeatable evaluation, every change is a guess, and teams end up afraid to touch prompts that "work". Evaluation is what makes iteration safe.

Step 1: Build an offline evaluation set

Each case should contain the input, any context the system needs (user, tenant, conversation history), and the expected outcome. Expected outcomes don't have to be exact strings. Useful forms:

• Reference answer for factual questions.
• Must-include / must-not-include facts (for example, the correct refund window; never a competitor's name).
• Relevant document IDs for retrieval scoring.
• Expected behaviour: refuse, ask a clarifying question, call a specific tool.

Source cases from real usage logs, support tickets, domain experts and, most importantly, every production failure you find. Tag cases by category (billing, onboarding, edge case, adversarial) so you can see where quality changes, not just whether the average moved.

{
  "id": "refund-007",
  "tags": ["billing", "policy"],
  "question": "Can I get a refund after 20 days on the annual plan?",
  "relevant_doc_ids": ["policy/refunds#annual"],
  "must_include": ["30 days"],
  "must_not_include": ["no refunds"],
  "expected_behaviour": "answer_with_citation"
}

Step 2: Measure retrieval on its own

In a RAG system, if the right passage isn't retrieved, the generator can't produce a grounded answer. Score retrieval before you look at answers:

def recall_at_k(retrieved: list[str], relevant: set[str], k: int) -> float:
    hits = relevant.intersection(retrieved[:k])
    return len(hits) / len(relevant) if relevant else 0.0

def reciprocal_rank(retrieved: list[str], relevant: set[str]) -> float:
    for rank, doc_id in enumerate(retrieved, start=1):
        if doc_id in relevant:
            return 1.0 / rank
    return 0.0

# MRR = mean of reciprocal_rank over all eval cases

Low recall points to chunking, embedding or query-rewriting problems; good recall with low MRR suggests adding a reranker. These are the levers discussed in building production RAG systems with LangChain.

Step 3: Score the answers

Deterministic checks first

Anything you can check with code, check with code: valid JSON against a schema, required fields present, must-include strings, forbidden phrases, citation IDs that actually exist in the retrieved set, length limits, correct tool called. These checks are cheap, fast and never flaky. For structured tasks like text-to-SQL, compare execution results directly, as described in the text-to-SQL architecture guide.

Groundedness and correctness

For free-text answers, you need two separate judgements:

• Groundedness (faithfulness): is every claim supported by the retrieved context? This is your hallucination metric.
• Correctness / answer relevance: does the answer actually address the question and match the reference?

An answer can be correct but ungrounded (the model used prior knowledge, which may be stale) or grounded but unhelpful. Track both.

LLM-as-judge, and its pitfalls

Using a model to grade outputs scales well, but judges have well-known failure modes:

• Position bias in pairwise comparisons: the first (or second) option wins more often. Randomise order or evaluate both orders.
• Verbosity bias: longer answers get higher scores even when they're worse.
• Self-preference: a model may favour outputs that resemble its own style.
• Vague rubrics: "rate helpfulness 1–10" produces noisy numbers. Ask narrow, binary questions instead.
• Inconsistency: the same input can get different grades across runs.

JUDGE_PROMPT = """You are grading whether an answer is supported by context.
Context:
{context}

Answer:
{answer}

List each factual claim in the answer. For each, say SUPPORTED or
UNSUPPORTED based only on the context. Then output JSON:
{"unsupported_claims": [...], "grounded": true|false}"""

Before trusting a judge, have a human label 50 or so outputs and check how often the judge agrees. If agreement is poor, fix the rubric before scaling it up. Re-check whenever you change the judge model.

Step 4: Regression testing in CI

Treat prompts, retrieval settings and model versions as code. Every change runs the evaluation suite and is compared against the last accepted baseline:

• Pin model versions and use temperature zero for evaluation runs where possible.
• Run a small smoke set on every commit and the full suite on changes to prompts, models, chunking or indexes.
• Fail the build on deterministic-check failures and on aggregate score drops beyond a tolerance, not on single noisy cases.
• Report results per tag so a drop in one category isn't hidden by gains elsewhere.
• Store every run's outputs so you can diff answers between versions.

This fits naturally into an existing pipeline; see DevOps for full-stack developers for the CI fundamentals.

Step 5: Monitor in production

Offline evaluation tells you a change is safe to ship. Monitoring tells you whether reality agrees.

• Trace every request: input, retrieved document IDs and scores, prompt version, model, output, latency and token usage.
• Operational metrics: latency percentiles, error and timeout rates, cost per request, refusal rate.
• Quality signals: thumbs up/down, follow-up rephrasing, escalations to a human, sampled groundedness scores from an async judge.
• Drift: new topics the eval set doesn't cover, falling retrieval scores, shifts in the question mix.
• Feedback loop: triage low-scoring traces weekly and add the real failures to the evaluation set.

Be deliberate about privacy: redact personal data in logs and set retention limits, especially when traces contain user documents.

Evaluating agents and multi-step flows

When the application calls tools or takes several steps, the final answer isn't the only thing that matters. Evaluate the trajectory as well:

• Tool selection: did it call the right tool, with valid arguments?
• Step count and cost: did it reach the answer without loops or unnecessary calls?
• Side effects: for tools that write data or send messages, assert in a sandbox that exactly the expected actions happened, and nothing else.
• Recovery: when a tool returns an error, does the agent retry sensibly or give up gracefully?

Most of these checks are deterministic once you record the trace, which makes them ideal CI assertions.

Common evaluation mistakes

• Tuning on the test set. If you rewrite prompts until every eval case passes, you've overfit. Keep a held-out slice you look at less often.
• Only averages. A stable average can hide a collapse in one important category.
• Only happy paths. Include out-of-scope questions, prompt-injection attempts and questions whose answer isn't in the documents; the correct behaviour there is to say so.
• Stale eval sets. If the set never grows, it stops reflecting what users actually ask.

A minimal starting stack

You don't need a platform on day one. A JSON or CSV eval set in the repository, a script that runs the pipeline over it and writes scores, a few deterministic assertions, one calibrated groundedness judge, and structured request logging will cover most teams for a long time. Adopt a dedicated evaluation or tracing tool once the volume of runs and traces makes spreadsheets painful. If you're still deciding whether retrieval or fine-tuning fits your use case, the same eval set is how you settle it; see RAG vs fine-tuning.

Frequently asked questions

How big should an LLM evaluation set be?

Start with 50–100 carefully chosen cases that cover your main use cases, known edge cases and past failures. That's enough to catch regressions. Grow it continuously from production failures and user feedback rather than trying to build a perfect set up front.

Can I trust LLM-as-a-judge scores?

Partly. Judges are useful for scaling subjective checks, but they have known biases: position bias, preference for longer answers, self-preference and inconsistency across runs. Use specific rubrics, binary or narrow scales, a strong judge model, and validate the judge against human labels before trusting it.

What is the difference between recall@k and MRR?

Recall@k measures whether the relevant documents appear anywhere in the top k retrieved results. MRR (mean reciprocal rank) measures how high the first relevant result is ranked, averaged over queries. Recall@k tells you whether the answer is available to the model; MRR tells you how well your ranking puts it first.

What is groundedness in RAG evaluation?

Groundedness, sometimes called faithfulness, measures whether every claim in the answer is supported by the retrieved context. An answer can be correct but ungrounded (the model used prior knowledge) or grounded but incomplete. Low groundedness is the main signal of hallucination in a RAG system.

How do I run LLM evaluations in CI without them being flaky?

Use a fixed eval set, temperature zero where possible, deterministic assertions for structure and must-include facts, and compare aggregate scores against a baseline with a tolerance rather than requiring every case to pass. Run the full suite on prompt, model or retrieval changes and a smaller smoke set on every commit.

Want help setting up evaluation?

If you're shipping an LLM or RAG feature and need an evaluation and monitoring setup you can trust, see my AI engineering services or get in touch.