Run FRAMES for multi-hop reasoning, MIRAGE for a compact retrieval-and-generation check, and MTRAG for conversational grounding, then widen the sweep with CRAG and RAGEval or LiveRAG once you know where the system struggles. The posture that makes this work: evaluate

retrieval and generation separately, and weight faithfulness and hard negatives over surface-level answer quality. We’ll walk through the benchmarks, the metrics, and a CI pipeline you can build this quarter.

TL;DR:

  • Build a corpus specific test set with 300 to 330 token chunks, labeled positives, and topic matched hard negatives to expose retrieval errors.
  • For single turn tests, track recall among the top five results, groundedness, and answer relevance; use an oracle retrieval baseline to isolate reasoning failures across documents.
  • Add no retrieval and keyword search baselines because little improvement over keyword search points to retrieval rather than generation as the main failure.
  • Gate pull requests with 50 to 100 gold queries, fail builds when faithfulness or context relevance falls below set thresholds, and schedule full evaluations.
  • Review test sets quarterly, update gold answers after corpus changes, and replace questions the system always passes with harder cases drawn from production failures.

Table of Contents

Which public RAG benchmarks should you run first?

Each benchmark tests a different failure mode, so picking the right one first saves you weeks of chasing the wrong problem.

FRAMES, MIRAGE, and MTRAG ship with ready-made retrieval pools, so you can start scoring the same day. CRAG and RAGEval expect you to build or adapt a pool to your own corpus, which takes longer but gets you closer to your actual production conditions.

What metrics actually tell you something about RAG quality?

A single accuracy number hides which half of your pipeline is broken. Separate your metrics by component so a bad score points you somewhere specific.

  • Faithfulness (groundedness) checks each claim in the generated answer against the retrieved context, flagging statements the model invented rather than retrieved.
  • Answer relevance asks a narrower question: does the response actually address what the user asked, independent of whether it’s grounded?
  • Context relevance and precision measure whether the retrieved chunks are focused on the query or padded with noise that dilutes the generator’s attention.
  • Retriever recall at top-k tells you whether the right chunk was even in the candidate set, and testing against hard negatives shows whether your retriever can tell a near-miss from a real answer.
  • Hallucination rate, checked against human judgments on a sample, catches cases where automated scoring and human intuition diverge.

The source of truth here is RAGAS, which validated its faithfulness, answer relevance, and context relevance metrics against a human-annotated dataset called WikiEval and found strong correlation with human judgment, which is why reference-free scoring has become a practical substitute for manual review at scale.

How do you build an evaluation dataset from your own corpus?

Public benchmarks tell you how your system handles general-purpose questions. They say nothing about your actual documents, so building a dataset tied to your corpus is the step most teams skip and later regret.

  1. Extract contexts and chunk them. Chunks of roughly 300 to 330 tokens tend to preserve sentence integrity while keeping the retrieval pool small enough to search efficiently, a balance MIRAGE’s own construction reflects.
  2. Generate candidate questions. Use persona and question-type controls so you’re not just producing variations of the same easy lookup question.
  3. Create positives and several hard negatives per query. A single correct chunk plus a handful of near-miss distractors forces the retriever to discriminate instead of guessing.
  4. Map labels explicitly. Every query needs a documented link to its correct chunk and its negatives, stored in a format your evaluation script can consume directly.
  5. Validate in layers. Run automated checks first, sample a subset for human annotators, and keep a small meta-evaluation set to confirm your automated scores still track human judgment as the dataset grows.
  6. Package it with a clear contract. Fix your train and test splits, document the JSON schema, and write down exactly what fields your evaluation script expects before anyone starts using the set.

Pro Tip: Keep your hard negatives close to the real answer in topic and vocabulary. Easy negatives inflate recall scores and hide retrieval weaknesses you’ll discover later in production.

How do synthetic generation tools compare for building test sets?

Writing test questions by hand doesn’t scale past a few hundred examples, which is why most teams lean on synthetic generation. The tools differ mainly in how much control they give you over diversity.

  • DataMorgana-style pipelines let you set fine-grained quotas for user personas and question categories, which produces lexically and semantically varied sets instead of the repetitive phrasing a single prompt template tends to generate.
  • RAGEval and other schema-based frameworks generate documents, questions, answers, and references together in a structured format, and propose their own metrics like completeness, hallucination, and irrelevance for scoring the output.
  • RAGAS-style iterative generation can evolve questions over multiple passes, pushing a flat distribution of easy lookups toward a mix that includes comparison and multi-step reasoning questions.
  • The knobs that matter most are persona variety, explicit question-type quotas, and a defined rule for how distractors get selected, since leaving any of the three to default settings tends to produce a dataset that looks diverse but tests the same narrow skill repeatedly.

What should a first-pass test setup look like for each scenario?

Start small and specific rather than trying to cover every scenario at once.

  • Single-turn: a few queries against your production corpus, measuring top-5 recall, groundedness, and answer relevance as your first three numbers.
  • Multi-hop: build cross-document positives deliberately, with an oracle baseline that retrieves the correct chunks by hand so you can isolate whether failures come from retrieval or reasoning.
  • Multi-turn: represent conversation state explicitly in your test cases, and score both turn-level grounding and whether the system stays consistent with earlier answers.
  • Initial baselines: run a no-retrieval baseline and a keyword-search baseline alongside your actual system. If your RAG pipeline barely beats keyword search, the problem is almost always in retrieval, not generation.

How do you move from evaluation to production with confidence?

Evaluation numbers only matter if they gate releases. A CI pipeline that checks faithfulness and recall on every change catches regressions before they reach users.

  • Sample readiness checks: run 50 to 100 spot checks against your gold set on every pipeline change, a pattern we’ve detailed in our guide to CI-gated LLM evaluation.
  • Automated grounding thresholds: fail the build when faithfulness drops below a set bar rather than relying on someone noticing later.
  • Vector database pilots: test retrieval latency under your actual workload, watching p95 numbers, before committing to a vector store at scale, which we cover in our vector database selection guide.

Pro Tip: Budget 60 to 80 percent of your project timeline for data preparation, not model tuning. Our data prep guide lays out the SLAs worth setting before the clock starts.

How do you pick the right evaluation dataset for your domain?

A benchmark built for open-domain trivia tells you little about a RAG system answering questions from internal engineering docs or medical records. Start by matching the benchmark’s query style to your actual users: short factual lookups call for a single-turn set like MIRAGE, while a support chatbot that holds context across messages needs something closer to MTRAG’s multi-turn structure.

Domain also changes what counts as a hard negative. In a legal or medical corpus, two documents can share almost identical language while describing different situations entirely, so your distractor selection needs to pull from genuinely similar passages rather than random unrelated chunks. Generic benchmarks rarely capture this kind of near-duplicate confusion, which is one reason a corpus-specific test set earns its cost even after you’ve run the public ones.

Match your question-type mix to real usage patterns too. If your logs show users asking comparison questions or multi-step queries, your evaluation set needs enough examples of each type to catch a regression in that specific skill, not just an overall grounding score. Pulling a sample of actual user queries, anonymized, and using it to seed your synthetic generation step keeps the distribution honest instead of reflecting whatever a generic prompt template happens to produce.

Finally, revisit the chosen benchmark mix whenever your corpus changes meaningfully. A dataset built for last year’s document set won’t catch failures introduced by a new content type or a schema change in your knowledge base.

Should you rely on public benchmarks or build your own?

Public benchmarks give you something proprietary datasets can’t: a comparison point. When FRAMES or MIRAGE scores are published for other systems, you know roughly where your own numbers stand, and you skip the cost of building a retrieval pool and labeling questions from scratch.

The tradeoff is relevance. None of the public sets reflect your document structure, your users’ phrasing, or the specific failure modes your corpus invites. A proprietary set built from your own content catches issues a generic benchmark never will, like a retriever that confuses two product manuals with near-identical section headers.

The practical answer is to run both, treating them as different instruments. Public benchmarks are useful early, before you’ve built anything custom, and they stay useful afterward as a sanity check that your system still handles general-purpose reasoning correctly. Modern evaluation practice treats this as a closed loop: decouple retriever and generator scoring, then feed results back into iterative data synthesis rather than treating evaluation as a one-time gate.

Proprietary sets need more upkeep, since every corpus update risks making old gold answers stale, but they’re the only way to know how your system performs on the questions your actual users ask. Teams that skip this step tend to discover retrieval gaps only after a customer complaint, which is a far more expensive way to find them.

How should evaluation datasets fit into your CI pipeline?

An evaluation dataset that only gets run manually before a big release catches regressions too late. The goal is to make faithfulness and recall checks part of the same pipeline that runs your unit tests.

Wire a sample of your gold set, 50 to 100 queries is usually enough for a fast signal, into every pull request that touches retrieval, prompting, or the underlying model. Score faithfulness and context relevance automatically, and fail the build when either drops below a threshold you set from your current baseline. Our CI-gated LLM evaluation guide walks through setting that threshold without producing so many false failures that the team starts ignoring the gate.

Keep a second, larger evaluation run on a schedule rather than every commit, since full-dataset scoring against hundreds or thousands of queries is often too slow to run on every push. A nightly or weekly full pass catches slower drift that the fast sample check might miss, like a retriever whose recall degrades gradually as your document corpus grows.

Fast pull request gate and scheduled full evaluation

Log every run’s scores over time rather than just pass or fail. A recall number that’s technically above threshold but trending down for three weeks straight is a warning sign a binary gate alone won’t surface. Pair this with infrastructure assessment practices that treat model performance monitoring the same way you’d treat uptime or latency monitoring, as a continuous signal instead of a one-time check.

How often should you update your evaluation dataset?

An evaluation set built once and never touched again quietly loses relevance as your corpus grows and your model stack changes. Treat your gold answers as versioned, not fixed: when the underlying documents change, the correct answer to a question may change with them, and an automated check against current retrieved evidence can flag when a gold label needs a second look.

Set a cadence rather than waiting for a failure to prompt a review. A quarterly pass through your test set, checking whether questions still make sense against the current corpus and whether new document types need representation, catches drift before it shows up as a confusing production bug. Add new questions whenever you ship a feature that changes what users ask, since a static set stops reflecting real usage the moment your product does.

Retire questions that no longer test anything useful. A question every version of your system answers correctly isn’t adding signal, and keeping dozens of them around slows down every evaluation run without improving what it tells you. Replace them with harder cases pulled from recent production failures, which keeps the dataset honest about where your current weaknesses actually are.

Finally, revisit your metric thresholds alongside the dataset itself. A faithfulness bar that made sense for last year’s model may be too lax or too strict for a newer one, so recalibrate it against a fresh baseline run rather than carrying the old number forward indefinitely.

How often should you update your evaluation dataset? — overview diagram

Evaluation is a discipline, not a one-time project

The most common mistake we see is treating evaluation as a checkbox before launch rather than a habit that runs alongside development. Decoupling retriever and generator scoring tells you which half of the system actually needs the next sprint of work, instead of guessing. Iterating synthetic generation with real human feedback, even a small amount, catches diversity gaps that a single prompt template never surfaces on its own. And claim-level diagnostics, checking faithfulness one assertion at a time rather than scoring an answer as a whole, cut release risk more than any aggregate accuracy number ever will.

— Chad

Get your RAG evaluation pipeline production-ready with Bowtie

We build the CI-gated evaluation pipelines, retrieval pool construction, and faithfulness scoring that turn a one-off benchmark run into a repeatable release gate. Our AI Engineering Assistance work covers exactly this: wiring your evaluation dataset into CI, tuning retrieval thresholds, and auditing existing RAG code for the gaps a benchmark alone won’t catch.

Bowtie

If you’re not sure where your current pipeline stands, our Vibe Check service, a flat $449 one-off review, gives you a fast read on what’s solid and what needs attention before you commit to a larger engagement. Start there, or head to our pricing page to see the full range of engagements, from a Senior Developer Review to ongoing Crew Service support.

FAQ

How do you evaluate a RAG system?

You evaluate retrieval and generation as separate stages: measure retriever recall and precision against a labeled pool, then measure the generator’s faithfulness and answer relevance against the retrieved context. RAGAS is a widely used framework for scoring the generation side without needing hand-written reference answers for every query.

What metrics are used to evaluate LLMs as judges?

LLM-as-judge setups typically score faithfulness, answer relevance, and context relevance, then get validated against a sample of human annotations to confirm the automated scores track human judgment. The RAGAS framework validated these metrics against a human-annotated dataset called WikiEval and found they correlated well with human scoring.

What are the different evaluation methods used in AI?

Evaluation methods generally split into automated metric scoring, human annotation, and hybrid approaches that use automated scores at scale with periodic human spot checks for calibration. For RAG specifically, this usually means separate scoring for retrieval quality and generation quality rather than one blended score.

What is RAG analytics?

RAG analytics refers to tracking retrieval and generation performance metrics over time, such as recall, faithfulness, and hallucination rate, rather than evaluating a system once before launch. NVIDIA’s RAG evaluation documentation outlines metrics like answer accuracy, context relevancy, and response groundedness as part of this ongoing monitoring approach.

What size should a RAG evaluation dataset be to start?

A few hundred queries is enough for an initial single-turn evaluation set, as long as it includes both positives and hard negatives for each question. For comparison, MIRAGE includes thousands of instances with a large retrieval pool, showing the scale a mature benchmark eventually reaches, though you don’t need that scale to get a useful first signal.

Sources