RAG Evaluation in Production

The Problem#

RAG can look correct in a demo and fail in production. The answer may sound confident while using the wrong document, an old policy, or no source at all.

Why It Matters#

Users trust answers that cite internal knowledge. If retrieval is poor, the model will still produce fluent text. You need evaluation that tests the retrieval layer and the generation layer separately.

Core Concepts#

Measure retrieval with recall at k, precision at k, mean reciprocal rank, and source freshness. Measure generation with faithfulness, citation coverage, and answer completeness. Keep a fixed regression set of real user questions and known good documents.

Implementation#

Store test cases as data:

{
  "question": "How do we rotate production API keys?",
  "expectedDocumentIds": ["runbook-api-key-rotation"],
  "mustMention": ["create new key", "deploy", "revoke old key"]
}

Run the same suite after chunking changes, embedding model changes, rerankers, prompt edits, and index rebuilds. Fail the build if recall or faithfulness drops below the threshold.

Real Project Scenario#

A customer-support assistant answers refund policy questions. One chunking change improves general answers but stops retrieving the regional exception for EU customers. Without a regression set, the team notices only after support agents report wrong answers.

Production Setup#

Keep an evaluation dataset with real questions, expected source documents, expected facts, and known forbidden claims. Track retrieval recall separately from answer quality. When an answer is wrong, first inspect which chunks were retrieved before changing the prompt.

Common Mistakes#

• Judging RAG quality by reading ten hand-picked examples.
• Evaluating only the final answer and ignoring whether the right chunks were retrieved.
• Mixing stale documents with current documents without freshness rules.
• Letting generated citations point to documents that were not actually used.

Production Considerations#

Log the query, retrieved document ids, chunk scores, model version, prompt version, and final citations. Sample real traffic into an offline review queue, but remove private data before it reaches human reviewers.

Security#

Evaluation datasets often contain internal policy, customer data, or incidents. Apply the same access controls and retention policy as production logs.

Performance#

Reranking improves quality but adds latency. Measure whether the extra recall changes user outcomes before adding it to every request.

Summary#

RAG quality is measurable. Test retrieval, generation, citations, freshness, and regressions separately so the system improves without relying on vibes.