Retrieval-Augmented Generation (RAG) Questions

Direct answer

Hold everything fixed except the embedding model: same documents, same chunking, same index type and settings, same queries, same relevance labels. Score each model per query on retrieval metrics (recall@k, MRR, NDCG), compare the two models query by query on the same test queries (a paired comparison: each query gives one result per model, and the analysis looks at the per-query differences), and report the difference with a confidence interval rather than two bare averages. Split the data so no document or near-duplicate appears on both sides, tune on the dev side, and open the test side once.

Design, step by step

1. Define the decision. "Which model retrieves the right passage more often for our users' questions?" The metric should match the use. For a RAG system that passes the top $k$ chunks to a generator, recall@$k$ (the fraction of queries with a relevant chunk among the top $k$) is the primary metric; MRR (mean reciprocal rank, using the first relevant item) and NDCG (normalised discounted cumulative gain: add up each retrieved item's relevance grade, divide by log2(rank + 1) so lower ranks count less, and divide by the best possible total so the score lies between 0 and 1) are secondary. State which gain convention NDCG uses (the grade itself, or 2 to the power of the grade minus 1) if you report it.
2. Control the pipeline. One chunker, one corpus, one vector index with exact (brute-force) search so index approximation does not mix into the comparison. Use each model's documented query and passage input format, and normalise vectors the way each model expects; a wrong prefix or missing normalisation is the most common self-inflicted loss. Record dimension and index size, because a 4x larger vector costs memory.
3. Dataset and queries. Prefer real user questions with labels from annotators or click logs. If queries must be generated (for example a generator model writes questions from passages), say so, check a sample by hand, and note the bias: generated questions share wording with their passage, which favours lexical matching and can hide the model's real strength. Mix generated and real queries and report them separately. Include hard cases (acronyms, numbers, multi-hop, no-answer).
4. Splits. Split by document (or by source), never by query, so the same passage is not both in tuning and in the test. Use the dev split for choices such as $k$, text prefixes and thresholds, and score the test split once. If either model was trained on data that overlaps your test corpus, the comparison is contaminated; say what you know.
5. Statistics. Compute the per-query metric for both models, take the paired differences, and use a paired bootstrap confidence interval or an exact sign test (McNemar's test for binary hits) on the discordant queries, those where exactly one model succeeds. A paired bootstrap confidence interval is built by redrawing the test queries with replacement (so some repeat and some are left out), recomputing the average paired difference on each redraw, doing this thousands of times, and reading the middle 95% of the results (the 2.5th to the 97.5th percentile) as the interval. An exact sign test ignores the queries where both models agree and asks how surprising the split of the rest would be if each model were equally likely to win a discordant query, a fair coin; McNemar's test is the same idea for yes/no outcomes. Resample documents when several queries come from one document, because queries from the same document are not independent. Report the interval, and decide ahead of time what difference matters to the product.
6. Power. Power is the probability that the experiment detects a real gap of a given size; 80% power means a 4 in 5 chance. Estimate the number of test queries from the spread of paired differences: roughly $n \approx ((z_{\alpha/2}+z_{\beta})\,\sigma_d/\delta)^2$, where $\sigma_d$ is the standard deviation of per-query differences, $\delta$ the smallest gap you care about, and $z_{\alpha/2}=1.96$, $z_\beta=0.84$ for a 5% two-sided test with 80% power. These are points on the standard normal curve: 1.96 leaves 2.5% of the probability in each tail, which gives the 5% two-sided test, and 0.84 leaves 20% in the upper tail, which gives 80% power. The formula says to collect enough queries that the gap $\delta$ is about 2.8 standard errors wide.
7. Slice and then decide. Break results down by query type, document length and language. Add cost and latency (embedding throughput, vector size) before choosing.

Worked example

The two "embedding models" below are stand-ins, not neural embedding models: scikit-learn 1.5.2 TF-IDF vectors (each text becomes a sparse list of weighted word counts, compared by cosine similarity) with word features (A) and with character 3-to-5-letter-sequence features (B). They are a fair stand-in for the purpose of this example because the experiment is about the evaluation procedure, and any two systems that map queries and documents to comparable scores can be put through the same paired test; what differs here is a known property (B tolerates word endings such as s and ing, A does not). The corpus is synthetic: 150 documents of 16 made-up words, three queries per document built from five words of that document (half of them with an added suffix s or ing) plus two noise words. A real study would use real data; this shows the mechanics and the statistics.

import os
os.environ["OMP_NUM_THREADS"] = "1"
import numpy as np
from scipy.stats import binomtest
from sklearn.feature_extraction.text import TfidfVectorizer

rng = np.random.default_rng(0)
# ---- synthetic labelled QA set (stand-in for a real one) ----
syll = ["ba", "ko", "ri", "mu", "te", "sa", "lo", "ni", "pe", "du"]
stems = sorted({"".join(rng.choice(syll, 3)) for _ in range(900)})[:400]
p = 1 / np.arange(1, len(stems) + 1); p /= p.sum()          # Zipf-like word frequency
form = lambda w, k: [w, w + "s", w + "ing"][k]
N_DOC, Q_PER = 150, 3
docs, queries, qdoc = [], [], []
for d in range(N_DOC):
    ws = list(rng.choice(stems, 16, p=p))
    docs.append(" ".join(ws))
    for _ in range(Q_PER):
        pick = rng.choice(ws, 5, replace=False)
        q = [form(w, rng.integers(0, 3) if rng.random() < 0.5 else 0) for w in pick]
        q += list(rng.choice(stems, 2, p=p))                  # two noise words
        queries.append(" ".join(q)); qdoc.append(d)
qdoc = np.array(qdoc)

# ---- two stand-in "embedding models": word TF-IDF vs character n-gram TF-IDF ----
def ranks_for(vec):
    D = vec.fit_transform(docs); Q = vec.transform(queries)
    S = (Q @ D.T).toarray()
    gold = S[np.arange(len(queries)), qdoc]
    return 1 + (S > gold[:, None]).sum(axis=1)               # rank of the gold doc (1 = best)
rank_a = ranks_for(TfidfVectorizer(analyzer="word"))
rank_b = ranks_for(TfidfVectorizer(analyzer="char_wb", ngram_range=(3, 5)))

# ---- split by DOCUMENT so no passage appears in both dev and test ----
test_docs = rng.permutation(N_DOC)[N_DOC // 2:]
test = np.isin(qdoc, test_docs)
print("dev queries", int((~test).sum()), "test queries", int(test.sum()),
      "docs shared across splits:", len(set(qdoc[test]) & set(qdoc[~test])))

k = 3
hit_a, hit_b = (rank_a <= k)[test], (rank_b <= k)[test]
rr_a, rr_b = (1 / rank_a)[test], (1 / rank_b)[test]
print(f"recall@{k}: A={hit_a.mean():.3f} B={hit_b.mean():.3f}   MRR: A={rr_a.mean():.3f} B={rr_b.mean():.3f}")

dev = ~test
print(f"dev split (where choices such as k would be made): recall@{k} A={(rank_a <= k)[dev].mean():.3f} B={(rank_b <= k)[dev].mean():.3f}")

# ---- paired comparison on the SAME test queries ----
diff = hit_b.astype(float) - hit_a.astype(float)
print("first 10 test queries (1 = gold document in top 3):")
print("  A hit :", hit_a[:10].astype(int).tolist())
print("  B hit :", hit_b[:10].astype(int).tolist())
print("  B - A :", diff[:10].astype(int).tolist(), "-> discordant where B - A is not 0")
b_only, a_only = int(((hit_b) & (~hit_a)).sum()), int(((~hit_b) & (hit_a)).sum())
pv = binomtest(b_only, b_only + a_only, 0.5).pvalue
print(f"discordant: B-only={b_only} A-only={a_only}  exact sign-test p={pv:.1e}")

bt = np.random.default_rng(1); qd = qdoc[test]; ud = np.unique(qd)
def ci(resample):
    vals = [resample() for _ in range(2000)]
    return np.percentile(vals, [2.5, 97.5])
by_query = ci(lambda: diff[bt.integers(0, len(diff), len(diff))].mean())
groups = {u: diff[qd == u] for u in ud}
by_doc = ci(lambda: np.concatenate([groups[u] for u in bt.choice(ud, len(ud))]).mean())
print(f"diff in recall@{k} = {diff.mean():+.3f}; 95% CI resampling queries {np.round(by_query, 3)}, resampling documents {np.round(by_doc, 3)}")

# ---- how many test queries to detect a given gap (normal approximation, paired) ----
sd = diff.std(ddof=1)
print(f"paired sd by hand: mean of diff = {diff.mean():.3f}; sd of the per-query differences = {sd:.3f}")
for delta in (0.10, 0.05, 0.02):
    n = int(np.ceil(((1.96 + 0.84) * sd / delta) ** 2))
    print(f"to detect a paired gap of {delta:.2f} at 5% level, 80% power: about {n} queries (paired sd={sd:.3f})")

Output:

dev queries 225 test queries 225 docs shared across splits: 0
recall@3: A=0.800 B=0.924   MRR: A=0.727 B=0.888
dev split (where choices such as k would be made): recall@3 A=0.804 B=0.924
first 10 test queries (1 = gold document in top 3):
  A hit : [1, 1, 1, 1, 1, 1, 1, 0, 0, 1]
  B hit : [1, 1, 1, 1, 1, 1, 1, 1, 0, 1]
  B - A : [0, 0, 0, 0, 0, 0, 0, 1, 0, 0] -> discordant where B - A is not 0
discordant: B-only=31 A-only=3  exact sign-test p=7.7e-07
diff in recall@3 = +0.124; 95% CI resampling queries [0.08  0.173], resampling documents [0.076 0.173]
paired sd by hand: mean of diff = 0.124; sd of the per-query differences = 0.369
to detect a paired gap of 0.10 at 5% level, 80% power: about 107 queries (paired sd=0.369)
to detect a paired gap of 0.05 at 5% level, 80% power: about 428 queries (paired sd=0.369)
to detect a paired gap of 0.02 at 5% level, 80% power: about 2671 queries (paired sd=0.369)

How to read it:

• The split shares no documents between dev and test (the printed count is 0). The dev half is where you would make choices such as k, input prefixes and thresholds, and the program prints its recall@3 for reference; the test half is the one the comparison is reported on, and it is scored once.
• A hand-sized paired example: in the 10 queries printed, both models hit on 8 of them (1 and 1), both missed on 1 (0 and 0), and B alone hit on 1 (B - A = 1). Only that one query is discordant. Over all 225 test queries, B alone hits on 31 and A alone on 3, so the mean paired difference is (31 - 3) / 225 = 0.124, the printed gap.
• On the same 225 test queries, B has the higher recall@3. B hits where A misses on 31 queries and the reverse on 3 queries, and the exact sign test on those discordant queries gives p = 7.7e-07 (0.00000077), meaning that if the two models were equally likely to win a discordant query, a split as lopsided as 31 to 3 would almost never occur.
• The 95% interval for the recall@3 gap is about 0.08 to 0.17 whether queries or documents are resampled; here the two agree because each document contributes few queries, but with many queries per document the document-level interval is the honest one.
• The last lines give the number of test queries needed for a given gap at the paired spread observed here. By hand for a gap of 0.10: $\sigma_d = 0.369$, so $n \approx (2.8 \times 0.369 / 0.10)^2 = 10.33^2 \approx 107$ queries; the 0.05 line gives about 428 with the unrounded spread. The count grows with the inverse square of the gap (halving the gap quadruples the queries), so a 0.02 difference needs thousands of queries.

Pitfalls

• Comparing two averages without a paired test, then reading a 1-point gap as real.
• Tuning on the test queries, or letting a document appear on both sides of the split.
• Comparing at different index settings, different chunk sizes or different input prefixes.
• Reporting only one metric: a model can win recall@10 and lose MRR.
• Generated queries as the only evaluation, which rewards word overlap.
• Ignoring that the final system also has a reranker and a generator; confirm the winner with an end-to-end check on a small set.

Direct answer

Retrieval-augmented generation (RAG) is a pattern where, before the language model answers, the system looks up relevant passages in an external knowledge source and puts them into the prompt. The model then writes its answer from that supplied evidence instead of from memory alone. This reduces hallucination (a fluent statement that nothing supports) because the answer is tied to text that can be shown, cited and checked, and because the system can refuse to answer when nothing relevant is found.

A plain-language picture: a standalone LLM takes a closed-book exam from whatever it absorbed during training. RAG turns it into an open-book exam where the book is your own, current, searchable document set.

The five core components

1. Ingestion and chunking. Source documents are cleaned and split into chunks (passages of a few hundred tokens, where a token is the sub-word unit a model reads).
2. Index. Each chunk is stored so it can be searched: as term statistics for keyword search (BM25 is a keyword scoring formula that rewards rare matching words), as an embedding (a list of numbers that represents meaning) in a vector index, or both.
3. Retriever. At question time it turns the question into the same representation and returns the top-k most similar chunks (k is how many to return, a setting such as 5). An optional reranker (a slower model that re-scores those k candidates) can sit after it.
4. Context assembly. The chosen chunks are placed in the prompt with an instruction such as "answer only from the sources below, cite the source id, say you do not know if the sources do not cover it". Chunk order, de-duplication and a token budget are decided here.
5. Generator. The LLM writes the answer conditioned on the question plus the sources, and the system returns the answer together with its citations.

Why grounding reduces hallucination

• Evidence is in the context window (the text the model reads for one request: instructions, question and retrieved chunks, up to a fixed maximum length). The model copies and paraphrases supplied facts, which is more reliable than recalling a rare fact from its parameters (the numbers learned in training).
• Verifiability. Each claim can point to a chunk id, so a human or an automatic check can compare claim and source.
• Abstention path. If retrieval returns nothing above a relevance threshold (a minimum similarity score, chosen by checking on labelled questions which cut-off separates useful passages from noise), the system can answer "no supporting passage" instead of guessing.
• Knowledge is not frozen. Facts that changed after training, or that were never public (internal policies), are available at inference time.

RAG lowers the rate; it does not remove it. Remaining failure modes: the right chunk was never retrieved (recall failure: recall at k asks whether a passage containing the answer is among the k returned; averaged over many questions it is the share of questions that retrieval got right), irrelevant chunks distract the model, two chunks conflict, or the model ignores the context and answers from memory. Measuring those separately is what RAG evaluation is for.

Worked example

The program below uses BM25 (the rank_bm25 library, version 0.2.2) as the retriever. The "standalone LLM" and the "generator" are deterministic stand-ins, not real models: the first is a frozen lookup of an older policy, the second copies the best retrieved sentence and cites its id. A real system would put an LLM in both places. The latency lines use assumed rates, so only the arithmetic is computed.

import os
os.environ["OMP_NUM_THREADS"] = "1"
import re
from rank_bm25 import BM25Okapi

STOP = {"the", "a", "an", "of", "is", "what", "who", "to", "in", "can", "be", "are", "or", "for", "and", "until", "after", "within", "from", "per", "not", "by"}

def tok(text):
    return [w for w in re.findall(r"[a-z0-9]+", text.lower()) if w not in STOP]

# A small "knowledge base". Passage p6 is the current policy; the closed-book
# stand-in below only knows the older 30-day rule.
passages = {
    "p1": "Standard shipping takes five business days within the country.",
    "p2": "Gift cards never expire and cannot be exchanged for cash.",
    "p3": "Support is available by chat from nine to five on weekdays.",
    "p4": "Orders can be cancelled until they leave the warehouse.",
    "p5": "Damaged items are replaced free of charge after a photo is sent.",
    "p6": "Refund window: customers can request a refund within 45 days of delivery.",
    "p7": "Loyalty points are credited after the return window closes.",
    "p8": "Express shipping is charged per kilogram and is not refundable.",
}

# Stand-in for a standalone LLM: a frozen lookup of what it "learned" before the policy changed.
frozen_knowledge = {"refund": "Refunds are accepted within 30 days of delivery."}

def closed_book(question):
    for key, answer in frozen_knowledge.items():
        if key in tok(question):
            return answer
    return "I am not sure."

def build_index(store):
    ids = list(store)
    return ids, BM25Okapi([tok(store[i]) for i in ids])

def retrieve(question, store, index, k=2):
    ids, bm25 = index
    scores = bm25.get_scores(tok(question))
    order = sorted(range(len(ids)), key=lambda j: (-scores[j], ids[j]))[:k]
    return [(ids[j], float(scores[j])) for j in order]

def rag_answer(question, store, index, min_score=1.0):
    # Stand-in for the generator: copy the retrieved sentence that shares most words with the question.
    hits = retrieve(question, store, index)
    best_id, best_score = hits[0]
    if best_score < min_score:
        return "No supporting passage found, so I will not answer.", hits
    return f"{store[best_id]} [source: {best_id}]", hits

store = dict(passages)
index = build_index(store)

q = "What is the refund window?"
print("closed-book :", closed_book(q))
ans, hits = rag_answer(q, store, index)
print("RAG         :", ans)
print("top hits    :", [(i, round(s, 3)) for i, s in hits])

q2 = "Who is the CEO of the company?"
ans2, hits2 = rag_answer(q2, store, index)
print("off-topic   :", ans2, [(i, round(s, 3)) for i, s in hits2])

# Freshness: the policy changes again; only the index is updated, no retraining.
store["p6"] = "Refund window: customers can request a refund within 60 days of delivery."
index = build_index(store)
print("after update:", rag_answer(q, store, index)[0])
print("closed-book after update:", closed_book(q))

# Latency budget: ASSUMED figures, only the arithmetic is computed.
retrieval_ms, chunks, tokens_per_chunk, prefill_ms_per_token = 30, 4, 250, 0.4
extra_tokens = chunks * tokens_per_chunk
added_ms = retrieval_ms + extra_tokens * prefill_ms_per_token
print(f"extra prompt tokens = {extra_tokens}; added latency = {added_ms:.0f} ms (assumed rates)")

Output:

closed-book : Refunds are accepted within 30 days of delivery.
RAG         : Refund window: customers can request a refund within 45 days of delivery. [source: p6]
top hits    : [('p6', 3.006), ('p7', 0.99)]
off-topic   : No supporting passage found, so I will not answer. [('p1', 0.0), ('p2', 0.0)]
after update: Refund window: customers can request a refund within 60 days of delivery. [source: p6]
closed-book after update: Refunds are accepted within 30 days of delivery.
extra prompt tokens = 1000; added latency = 430 ms (assumed rates)

Reading it: the closed-book stand-in still says 30 days because that is what it "learned". The RAG path finds passage p6 (BM25 score 3.006, well above the runner-up p7 at 0.99) and answers 45 days with a citation. The off-topic question has no matching terms, both scores are 0.0, and the pipeline abstains rather than invents an answer. After the policy passage is edited, the next answer says 60 days with no retraining, while the closed-book stand-in is unchanged.

Operational challenges: latency and freshness

• Latency. RAG adds a retrieval call and a longer prompt. With the assumed figures above (30 ms retrieval, four chunks of 250 tokens, 0.4 ms of prompt processing per token) the added time is 30 + 1000 x 0.4 = 430 ms. Tuning levers: fewer or shorter chunks, a cheaper first-stage retriever, caching frequent queries, streaming the answer.
• Freshness. Updating knowledge means re-ingesting changed documents, not retraining. The new risks are stale chunks (an old version still in the index), partial re-indexing, and embedding-model upgrades that force re-embedding the whole corpus. Track a document version or timestamp per chunk and delete superseded chunks.
• Cost. Each query pays for the extra prompt tokens; the index costs storage and rebuild time.

Common mistakes

• Saying RAG "eliminates" hallucination. It reduces one cause and adds retrieval-quality as a new failure point.
• Evaluating only the final answer. If the answer is wrong you cannot tell whether retrieval or generation failed unless you also measure the retriever's recall at k (the share of questions for which a passage containing the answer is among the k returned).
• Treating the vector index as the only option: keyword search is often better for exact identifiers.
• No abstention rule, so the model answers confidently from irrelevant chunks.