LLM Evaluation and Observability Questions

Direct answer

Build the benchmark from real dashboards and metrics paired with reference facts, not from reference text: for each item, store the data, the facts a good narrative must state (key movements, comparisons, caveats) and the claims it must not make. Score offline with checks that compare the output to those facts, and with a rubric-based human review on a sample. Prevent overfitting with a frozen decision set, a separate tuning set, a limit on how often the frozen set is looked at, and a scheduled refresh from fresh production cases. The core risk is that every time the team picks the best variant on the same items, the benchmark score stops being an honest estimate.

1. Build the labeled benchmark

• Items: a dashboard snapshot (tables, charts' underlying numbers, period, definitions) plus an intended audience. Cover varied shapes: flat, seasonal, a single outlier, a missing period, conflicting metrics, a tiny denominator, a data-quality problem. Stratify by those shapes (sample each shape on purpose, in planned numbers, instead of leaving it to chance) so rare, risky cases are not drowned out.
• Labels, per item, are facts rather than gold text: (a) must-mention findings (for example "revenue down 8% month on month, driven by one region"), (b) must-not-claim items (causes not in the data), (c) required caveats (partial month, definition change). Facts allow many valid narratives; a single reference paragraph does not.
  A concrete item makes the three label types visible. The numbers here are invented for illustration (revenue in thousands):

{
  "data": {"period": "Feb vs Jan", "total": [1000, 920], "West": [700, 620], "North": [300, 300]},
  "audience": "regional sales director",
  "must_mention": [
    "total revenue fell 8.0% (1,000 to 920)",
    "the whole decline is in West (700 to 620, down 11.4%)",
    "North was flat at 300"
  ],
  "must_not_claim": [
    "any cause for the West decline (the data holds no cause field)",
    "that the decline will continue"
  ],
  "required_caveats": ["February has fewer days than January"]
}

Checks: 80 / 1000 = 8.0% for the total, 80 / 700 = 11.4% for West. Two quite different narratives can both pass this item, one terse and one long, as long as each states the three findings, claims no cause and carries the caveat. A narrative that says "the decline was caused by a pricing change" fails the second label even if it is well written.

• Who labels: two domain reviewers per item independently, a third resolves disagreements. Agreement is measured with Cohen's kappa, which is the agreement between the two reviewers after subtracting the agreement that chance alone would produce: 0 means no better than chance, 1 means perfect agreement, and by a common rule of thumb 0.6 to 0.8 is read as substantial and above 0.8 as near-perfect. If it is low the rubric is the problem, not the reviewers.
• Size: choose from the precision you need (computed below), not from habit.

2. Score offline and with reviewers

Use the cheap layers on all items and every change, and humans on a stratified sample and on every item where the automatic layers disagree.

3. Why benchmarks get overfit, computed

When a team tries several prompt or model variants and keeps the one with the best benchmark score, the winner's score is inflated by selection even if all variants are equally good. The program simulates this under stated assumptions: every variant has the same true pass rate of 0.70, each is scored on a benchmark of n items, the best-scoring variant is kept, and that winner is then scored again on fresh items. Averages are over 2000 simulated rounds with a fixed seed. It then computes how many items a paired comparison needs and the interval width on a 300-item benchmark, and the kappa from a small reviewer pilot (the pilot labels are invented).

import os
os.environ["OMP_NUM_THREADS"] = "1"; os.environ["OPENBLAS_NUM_THREADS"] = "1"
import numpy as np
from sklearn.metrics import cohen_kappa_score
from math import sqrt
# Winner's curse: K prompt variants all have the SAME true pass rate 0.70.
# The team keeps the one with the best score on a benchmark of n items, then we re-score it on fresh items.
true_p, trials = 0.70, 2000
rng = np.random.default_rng(0)
print("n_items K_variants  benchmark_score_of_winner  fresh_score_of_winner")
for n in (100, 400):
    for K in (1, 5, 20):
        bench, fresh = [], []
        for _ in range(trials):
            scores = rng.binomial(n, true_p, size=K) / n
            winner = scores.argmax()
            bench.append(scores[winner]); fresh.append(rng.binomial(n, true_p) / n)
        print(f"{n:7d} {K:10d}  {np.mean(bench):.3f}                      {np.mean(fresh):.3f}")

# Benchmark size for a paired comparison: detect a 5-point difference in pass rate
# when the two systems disagree on 20% of items (paired design: the difference is determined by discordant items)
disc = 0.20; delta = 0.05; z_a, z_b = 1.96, 0.84
n_paired = ((z_a*sqrt(disc) + z_b*sqrt(disc - delta**2))**2) / delta**2
print("paired items needed (alpha 0.05 two-sided, power 0.80): %d" % np.ceil(n_paired))
se = sqrt(0.7*0.3/300)
print("95%% interval half-width for a pass rate of 0.70 on 300 items: %.3f" % (1.96*se))

# Reviewer agreement on a 12-item rubric pilot (1 = acceptable narrative)
a = [1,1,1,0,1,0,1,1,0,1,1,0]; b = [1,1,0,0,1,0,1,1,1,1,1,0]
print("pilot kappa:", round(cohen_kappa_score(a, b), 3))

Output:

n_items K_variants  benchmark_score_of_winner  fresh_score_of_winner
    100          1  0.701                      0.702
    100          5  0.752                      0.700
    100         20  0.783                      0.700
    400          1  0.700                      0.699
    400          5  0.726                      0.700
    400         20  0.742                      0.700
paired items needed (alpha 0.05 two-sided, power 0.80): 625
95% interval half-width for a pass rate of 0.70 on 300 items: 0.052
pilot kappa: 0.625

Reading it:

• With one variant, the benchmark score is honest (0.701 versus 0.702 fresh).
• With 20 variants on 100 items, the winner scores 0.783 on the benchmark and 0.700 on fresh items: about 8 points of the apparent gain are selection noise. The bigger the benchmark, the smaller the inflation (0.742 at n = 400 with 20 variants), but it never reaches zero while selection continues.
• A paired comparison of two systems (same items, both scored) needs 625 items to detect a 5-point difference at the stated assumptions, when the systems disagree on 20% of items. Fewer items than that cannot reliably separate two similar narratives. The 625 is a count of benchmark items, each scored by both systems. Where it comes from: alpha 0.05 (two-sided) is the accepted 5% risk of declaring a winner when the systems are equal, and power 0.80 is the chance of detecting a real 5-point gap. The constants 1.96 and 0.84 are the standard normal-curve values for those two settings (a two-sided 5% risk leaves 2.5% in each tail; 80% power leaves 20% in the far tail). Discordant items are the items where exactly one of the two systems passes; items both pass or both fail cancel out and say nothing about which system is better. With 20% discordant items and a 5-point gap, 12.5% of items favour system A and 7.5% favour B. The calculation is:

n = (1.96 * sqrt(0.20) + 0.84 * sqrt(0.20 - 0.05^2))^2 / 0.05^2
  = (0.8765 + 0.3733)^2 / 0.0025
  = 624.8, rounded up to 625

• On 300 items a 70% pass rate carries a 95% interval of roughly plus or minus 5.2 points, so a 2-point gain is invisible.
• Pilot kappa of 0.625 can be redone by hand. The two reviewers agree on 10 of 12 items, so observed agreement is 0.833. Each says "acceptable" on 8 of 12 and "not acceptable" on 4 of 12, so agreement by chance alone would be (8/12)^2 + (4/12)^2 = 0.556. Kappa is (0.833 - 0.556) / (1 - 0.556) = 0.625. On the rule of thumb above that is substantial but short of near-perfect, and with only 12 items it is a rough estimate, so tighten the rubric and re-run the pilot on more items before scaling the labeling.

4. Keep the team honest over time

1. Three splits: dev (tune freely), validation (choose between candidates, limited looks), frozen holdout (items held back from all tuning, consulted only for release decisions, and logged each time).
2. Count the looks. Every selection step against the same items spends some of their validity. Cap the number of comparisons per set; report how many were made.
3. Report intervals, not points, and require improvements to exceed the interval.
4. Refresh on a schedule: replace a slice (for example a quarter) of the holdout each cycle with newly labeled production cases, retire contaminated items to dev, and always re-score the previous system on the new set so scores stay comparable.
5. Prevent leakage: keep holdout items out of prompts, few-shot examples (worked examples placed inside the prompt) and fine-tuning data (examples used to train the model further); deduplicate against them.
6. Cross-check with fresh signals: weekly human review of random production outputs and user feedback, which cannot be gamed by tuning.
7. Watch for the proxy trap: if a rubric score rises while production complaints do not fall, the benchmark has drifted from the goal.

Pitfalls

• Scoring against a single reference narrative, which punishes valid alternatives.
• Letting the model generating narratives also write the benchmark items; its blind spots get copied into the test.
• Reporting a headline gain without the number of variants tried to get it.

Direct answer

The claim bundles two measurable effects: fewer hallucinations and slower replies. Design the test so that hallucination rate is the primary metric (the one that decides), and latency, error rate, refusal rate and cost are guardrails with limits set before the test. Roll out in stages: offline check on a fixed set, shadow traffic (the new model runs on real requests but users never see it), a small canary (a new version served to a small share of traffic first, to catch crashes and big regressions cheaply), then a 50/50 A/B test over at least a full weekly cycle. Randomise by user so one person does not see both versions. Ship only if the primary effect is real and every guardrail is inside its limit.

Sizing the test, with computed numbers

Assumptions: baseline hallucination rate 5% (replace with your measured value), so "20% fewer" means 4%. Two-sided test at 5% significance with 80% power. Significance of 5% means that if the versions were truly equal, the test would wrongly call a winner 5% of the time, split as 2.5% in each direction (that is what two-sided means). Power is the chance the test detects the effect if it is real. The sample per arm is

where $p_0$ and $p_1$ are the two rates, $\bar p$ is their mean, and the two $z$ values are standard normal quantiles for the significance level and the power. By hand: $z_{0.975} = 1.960$ and $z_{0.80} = 0.842$; with $p_0 = 0.05$, $p_1 = 0.04$, $\bar p = 0.045$, the first term is $1.960\sqrt{2 \times 0.045 \times 0.955} = 0.5746$ and the second is $0.842\sqrt{0.05 \times 0.95 + 0.04 \times 0.96} = 0.2467$. The numerator is $(0.5746 + 0.2467)^2 = 0.6745$, the denominator is $0.01^2 = 0.0001$, and $0.6745 / 0.0001 \approx 6745$ requests per arm. The labels come from a judge (a model or a person that marks each reply as hallucinated or not). An imperfect judge, with assumed sensitivity 0.85 (it flags 85% of truly hallucinated replies) and specificity 0.97 (it correctly passes 97% of clean replies, so it wrongly flags 3%), shrinks the visible gap, so the sample must grow. The rate the judge reports is sensitivity times the true rate plus the false-flag rate times the clean share: for a true 5%, $0.85 \times 0.05 + 0.03 \times 0.95 = 0.0425 + 0.0285 = 7.1\%$, and for a true 4%, $0.85 \times 0.04 + 0.03 \times 0.96 = 6.28\%$. The visible gap falls from 1.00 to 0.82 percentage points. The sample grows with the inverse square of the gap (about 1.5 times) and with the extra variance at the higher rates, so together it more than doubles. Latency is checked with a bootstrap interval (resample the requests with replacement many times, recompute the ratio each time, and take the middle 95%) on the ratio of p95 values (the time within which 95% of requests finish), using synthetic lognormal latencies (a right-skewed shape: most requests are quick and a few are slow, as real latencies are). The canary table assumes 400,000 requests a day.

import os
os.environ["OMP_NUM_THREADS"] = "1"
os.environ["OPENBLAS_NUM_THREADS"] = "1"
import numpy as np
from scipy.stats import norm

def n_per_arm(p0, p1, alpha=0.05, power=0.8):
    za, zb = norm.ppf(1 - alpha / 2), norm.ppf(power)
    pbar = (p0 + p1) / 2
    num = (za * np.sqrt(2 * pbar * (1 - pbar)) + zb * np.sqrt(p0 * (1 - p0) + p1 * (1 - p1))) ** 2
    return int(np.ceil(num / (p0 - p1) ** 2))

p0 = 0.05; p1 = p0 * 0.8      # baseline 5% is an assumption; the claim is 20% fewer
n = n_per_arm(p0, p1)
print(f"hallucination {p0:.1%} -> {p1:.1%}: n per arm with perfect labels = {n}")

# label noise: a judge with assumed sensitivity/specificity shrinks the visible difference
sens, spec = 0.85, 0.97
obs = lambda p: sens * p + (1 - spec) * (1 - p)
print(f"observed through judge: {obs(p0):.3%} -> {obs(p1):.3%}; n per arm = {n_per_arm(obs(p0), obs(p1))}")

def power_sim(n, a, b, sims=4000, seed=0):
    r = np.random.default_rng(seed)
    ka, kb = r.binomial(n, a, sims), r.binomial(n, b, sims)
    pa, pb = ka / n, kb / n
    pp = (ka + kb) / (2 * n)
    se = np.sqrt(2 * pp * (1 - pp) / n)
    z = (pa - pb) / se
    return float(np.mean(z > norm.ppf(0.975)))
print(f"simulated power at n={n}: {power_sim(n, p0, p1):.3f}; at n/2: {power_sim(n//2, p0, p1):.3f}; "
      f"false positive rate (no real change) at n: {power_sim(n, p0, p0):.3f} (one-sided 2.5% expected)")

# latency guardrail: bootstrap CI for the ratio of p95
r = np.random.default_rng(1)
old = r.lognormal(np.log(1.8), 0.45, 5000)
new = r.lognormal(np.log(1.8 * 1.10), 0.45, 5000)
ratios = []
for _ in range(300):
    a = r.choice(old, old.size); b = r.choice(new, new.size)
    ratios.append(np.percentile(b, 95) / np.percentile(a, 95))
lo, hi = np.percentile(ratios, [2.5, 97.5])
print(f"p95 old {np.percentile(old,95):.2f}s new {np.percentile(new,95):.2f}s ratio CI [{lo:.3f}, {hi:.3f}] with 5000 requests per arm")

# canary ramp at an assumed 400,000 requests/day, error-rate guardrail 0.5% -> 1.0%
daily = 400_000
def n_new_arm(p0, p1, share, alpha=0.05, power=0.8):
    """Requests needed in the new arm when it gets `share` of traffic and the old version gets the rest."""
    za, zb = norm.ppf(1 - alpha / 2), norm.ppf(power)
    k = (1 - share) / share                     # old-arm requests per new-arm request
    pbar = (p1 + k * p0) / (1 + k)
    num = (za * np.sqrt(pbar * (1 - pbar) * (1 + 1 / k)) + zb * np.sqrt(p1 * (1 - p1) + p0 * (1 - p0) / k)) ** 2
    return int(np.ceil(num / (p1 - p0) ** 2))
print("\nguardrail: error rate 0.5% -> 1.0%")
print("stage  new-arm requests needed  new-arm requests/hour  hours to reach it")
for share in [0.01, 0.05, 0.25, 0.50]:
    need = n_new_arm(0.005, 0.010, share)
    per_hour = daily * share / 24
    print(f"{share:5.0%}  {need:23d}  {per_hour:21.0f}  {need/per_hour:17.1f}")

# cost check (assumed token counts, unit prices of 1 per 1000 tokens)
old_tok, new_tok = 900, 1100
print(f"\ncost units per 1000 requests: old {old_tok} new {new_tok} ({new_tok/old_tok-1:+.0%})")

Output:

hallucination 5.0% -> 4.0%: n per arm with perfect labels = 6745
observed through judge: 7.100% -> 6.280%; n per arm = 14573
simulated power at n=6745: 0.806; at n/2: 0.506; false positive rate (no real change) at n: 0.026 (one-sided 2.5% expected)
p95 old 3.78s new 4.07s ratio CI [1.046, 1.120] with 5000 requests per arm

guardrail: error rate 0.5% -> 1.0%
stage  new-arm requests needed  new-arm requests/hour  hours to reach it
   1%                     2000                    167               12.0
   5%                     2119                    833                2.5
  25%                     2892                   4167                0.7
  50%                     4673                   8333                0.6

cost units per 1000 requests: old 900 new 1100 (+22%)

Reading the results

1. Sample size. 6,745 requests per arm (13,490 in total) for the 5% to 4% change when labels are perfect. A simulation of 4,000 trials gives power 0.806 at that size, 0.506 at half of it, and a 0.026 chance of a false win when nothing changed. The two-sided 5% test gives each direction 2.5%, and the simulation counts only the new-version-wins direction, so about 0.025 is expected there (0.05 across both directions); there is no contradiction between the design and the printed line. A test at half size would miss a real effect about half the time.
2. Label noise raises the cost. Through the assumed judge the rates become 7.100% and 6.280%, and 14,573 requests per arm are needed, more than double. Spend on label quality (audit a human-labelled sample to measure the judge's sensitivity and specificity) rather than only on traffic. These sizes count requests per arm, but assignment is by user, and one user's requests resemble each other, so they carry less independent information than the same number from different users. Analyse at the user level (for example, the share of each user's requests that were labelled hallucinated) or treat the request count as a floor.
3. Latency. The bootstrap interval on the p95 ratio is [1.046, 1.120] with 5,000 requests per arm, so "10% higher" is an estimate with a range. Decide before the test what increase is acceptable, for example an upper bound at or under 1.15, and judge against that bound.
4. Canary stages. Seeing an error rate rise from 0.5% to 1.0% takes a different number of new-version requests at each stage, because the old version serves the rest of the traffic and a large old group adds little noise to the comparison: 2,000 new-version requests at a 1% canary, 2,119 at 5%, 2,892 at 25% and 4,673 at 50/50. At 1% of traffic those 2,000 requests take 12.0 hours, at 5% about 2.5 hours, and at 25% or 50% under an hour. So start at 1% to catch crashes and large regressions, move to 5% for the first rate-based guardrail checks, then 25% and 50% for the real A/B read.
5. Cost. With assumed 900 versus 1,100 tokens per request, cost per 1,000 requests rises 22%. Compare cost per resolved task, not per request: a model that needs fewer retries or escalations can cost more per call and less per outcome.

Decision rule and traffic design

• Ship if the hallucination reduction is significant and its interval excludes zero, the p95 ratio's upper bound is inside the limit, the error and refusal rates have not worsened beyond their limits, and cost per resolved task is inside budget.
• Do not ship if refusals jumped (the model may "hallucinate less" by answering less) or a segment such as long documents regressed.
• Traffic split: 50/50 gives the most power for a fixed total because both arms are as well measured as possible. At 400,000 requests a day, 14,573 per arm (29,146 in total) arrives in under two hours of full traffic ($29{,}146 / 400{,}000 \approx 0.07$ days), so the sample is not what sets the run length; the 7-day minimum is, because behaviour differs on weekdays and weekends.
• Run at least 7 days to cover weekday and weekend behaviour and novelty effects (users reacting to anything new, which fades); fix the sample size in advance, or use a sequential method (a test designed so that you may check results repeatedly while it runs without inflating false wins), because stopping at the first significant result inflates false wins.
• For comparing output quality directly, show both versions' replies to raters in random order and ask which is better (a paired design, where each rater sees both replies to the same prompt, needs fewer items than rating each alone because prompt difficulty cancels).

Common mistakes

• Taking "20% fewer" without the baseline and the way it was measured.
• Using traffic split by request so users see both versions.
• Peeking daily and stopping on the first significant result.
• Ignoring refusal rate.
• Judging latency by the mean.

Direct answer

When an LLM summarizes data, answers questions over it, or drafts code, the common failures are: fabricated facts or numbers, wrong arithmetic or aggregation, wrong direction or comparison (up versus down, largest versus smallest), wrong entity, omission of what matters, unsupported cause-and-effect stories, misreading the question (wrong filter, period or definition), and, for code, code that does not run or runs and answers a different question. You tell them apart with checks that compare the output to the source data: each failure type leaves a different fingerprint. Before launching to business users, put guardrails on the input, the output and the experience, and require every figure shown to be traceable to a computed value.

Two terms used below: a guardrail is a check or limit placed around the model so a bad output is stopped or flagged before a user relies on it, and a sandbox is a locked-down place to run code you do not trust, with no access to real credentials or the network, so a bug or a malicious line can do far less damage (a sandbox limits the harm and is only as strong as its configuration; the exec call in the program below is not a sandbox).

Telling the failure types apart

The numeric lookup cannot separate wrong aggregation from fabrication: both show up as a figure that is not in the table. Recomputing the aggregate does. A narrative that says "average monthly sales were 77.5" quotes a number that exists nowhere in the table (it is the mean of the six rows, 465 / 6), while the question was about monthly totals, whose mean is 232.5. Omission and causal stories need different checks, shown in the program below: a list of what a good summary must mention (here, the one region that fell) and a list of causal words that send a narrative to a human, who checks whether any data field backs the cause.

Order the checks from cheapest to most specific: parse and execute, then verify numbers, then direction and ranking, then compare to a reference, and only then spend human time on omission and causal claims.

Worked example on a six-row table

The program builds a small sales table, runs deterministic checks on seven hand-written narratives, and executes three hand-written pieces of "generated" pandas code against a reference. The narratives and code are invented examples of model output, and the checks are plain Python stand-ins for what a production checker or judge would do.

How to read the pandas and exec lines: df.groupby("month").sales.sum() adds up the sales within each month. df.pivot(index="region", columns="month", values="sales") reshapes the six rows into a table with one row per region and one column per month, so pivot["Feb"] - pivot["Jan"] is each region's change, and idxmax() returns the row label (the region name) with the largest value, not the value itself. exec(src, env) is Python's built-in for running a string as code: src is the generated code and env is the dictionary of names that code can see (the table, the pivot, the region totals). The generated code stores its result in ans, which the harness reads back from env. Running model-written code like this is what the sandbox from the guardrail list is for; here the code is hand-written and safe.

import pandas as pd
df = pd.DataFrame({"region": ["North","North","South","South","East","East"], "month": ["Jan","Feb"]*3,
                   "sales": [100, 120, 80, 60, 50, 55]})
tot = df.groupby("month").sales.sum()
by_region = df.groupby("region").sales.sum()
print("month totals:", tot.to_dict(), "region totals:", by_region.to_dict())
pivot = df.pivot(index="region", columns="month", values="sales")
change = pivot["Feb"] - pivot["Jan"]
print("change Jan->Feb by region:", change.to_dict())
fell = [r for r in pivot.index if change[r] < 0]   # regions a good summary must not skip

# (label, narrative, numbers it quotes, direction it claims for Jan->Feb, region it calls largest)
narratives = [
 ("A", "Total sales rose from 230 in Jan to 235 in Feb.",       [230, 235], "rose", None),
 ("B", "Total sales rose from 230 in Jan to 250 in Feb.",       [230, 250], "rose", None),
 ("C", "Total sales fell from 230 in Jan to 235 in Feb.",       [230, 235], "fell", None),
 ("D", "South was the largest region with 140 in total.",       [140],      None,   "South"),
 ("E", "Sales were strong.",                                    [],         None,   None),
 ("F", "Total sales rose from 230 in Jan to 220 in Feb.",       [230, 220], "rose", None),
 ("G", "Total sales rose because of the spring promotion.",     [],         "rose", None),
]
CAUSAL = ("because", "due to", "driven by", "thanks to")
def classify(text, nums, direction, largest):
    """Deterministic stand-in checks against the source table."""
    valid = set(df.sales) | set(tot) | set(by_region)
    bad = [n for n in nums if n not in valid]
    if bad: return f"fabricated number {bad}"
    actual = "rose" if tot["Feb"] > tot["Jan"] else "fell"
    if direction and direction != actual: return f"wrong direction (actual: {actual})"
    if largest and largest != by_region.idxmax(): return f"wrong entity (largest is {by_region.idxmax()})"
    if any(w in text.lower() for w in CAUSAL): return "causal claim: needs a human check against the data"
    if not nums: return "no checkable claim"
    return "passes numeric checks"
for label, text, nums, d, big in narratives:
    omitted = [r for r in fell if r not in text]
    print(label, "->", classify(text, nums, d, big), "| omits:", omitted)

# Generated pandas code for "which region grew most from Jan to Feb": run it, compare with the reference answer
reference = change.idxmax()
generated = {
 "code_ok":    "ans = (pivot['Feb'] - pivot['Jan']).idxmax()",
 "code_wrong": "ans = by_region.idxmin()",
 "code_crash": "ans = df.growth.idxmax()",
}
for k, src in generated.items():
    env = {"df": df, "pivot": pivot, "by_region": by_region}
    try:
        exec(src, env)
        r = env["ans"]
        out = "matches reference" if r == reference else f"runs, wrong answer ({r} vs {reference})"
    except Exception as e:
        out = "does not execute: " + type(e).__name__
    print(k, "->", out)

Output:

month totals: {'Feb': 235, 'Jan': 230} region totals: {'East': 105, 'North': 220, 'South': 140}
change Jan->Feb by region: {'East': 5, 'North': 20, 'South': -20}
A -> passes numeric checks | omits: ['South']
B -> fabricated number [250] | omits: ['South']
C -> wrong direction (actual: rose) | omits: ['South']
D -> wrong entity (largest is North) | omits: []
E -> no checkable claim | omits: ['South']
F -> passes numeric checks | omits: ['South']
G -> causal claim: needs a human check against the data | omits: ['South']
code_ok -> matches reference
code_wrong -> runs, wrong answer (East vs North)
code_crash -> does not execute: AttributeError

Reading it:

• A passes the numeric checks: 230 and 235 are the month totals and the direction is right. Passing the checks does not prove the narrative is good, only that it contains no checkable error. It still omits South, the only region that fell, which only the omission list catches.
• B quotes 250 for February, a number that exists nowhere in the table: a fabrication.
• C quotes real numbers with the wrong word. The numbers check passes; only the direction check catches it, which is why numeric lookup alone is not enough.
• D uses a real figure (140 is South's total) attached to the wrong entity, because North is largest at 220.
• E contains nothing checkable, which is itself a finding for a data summary.
• F shows a limit of the number check: 220 is a real figure in the table (North's total), so it is accepted as a February total, and the direction word "rose" also matches, so the narrative passes although February's total is 235. A lookup that accepts any number found anywhere in the table can be fooled by a real-but-wrong figure; the stronger check ties each quoted number to the thing it is claimed to be, for example by recomputing the exact aggregate the sentence names.
• G contains no number and a causal word ("because"), so it is flagged for human review rather than passed. The stand-in flags any causal word; it does not look at the table. The table has no promotion column, which is why the human check will find the story unsupported, and a production checker would first look for a data field that could back the claim.
• Code: the first snippet matches the reference; the second runs but answers a different question (the region with the lowest total, East, instead of the one that grew most, North); the third fails to execute because the table has no growth column. The three need different fixes: the second is a question-interpretation error (the model chose the wrong aggregate and the wrong direction, which only comparing against the reference, or reading the code's plan, reveals), the third is a plain schema error that the interpreter itself reports, and the first needs nothing.

Guardrails before launching to business users

1. Constrain the source: the model sees only the rows it needs for the question, with column definitions, so it does not guess what "revenue" means.
2. Compute numbers outside the model: let the model write the query or choose the metric; run the query in the database and insert the result. Narrative text refers to computed values.
3. Verify before display: number lookup, direction and ranking checks as above; failures route to a fallback (show the table, not the story).
4. Show the work: display the query or code and the underlying rows, so a user can check.
5. Restrict code execution: read-only credentials, a sandbox, timeouts, no network.
6. Say when it cannot answer: out-of-scope or ambiguous questions get a clarifying question or a refusal, not a guess.
7. Narrative-generation risk: stories about "why" a metric moved invite plausible but unsupported causes. Restrict narrative to what the data shows (what moved, by how much, compared to what), label any hypothesis as one, and keep a human in the loop for executive-facing commentary.
8. Monitor after launch: sample outputs for human review, log failed verifications by type, and track user corrections.

Pitfalls

• Trusting a fluent summary because the first three numbers were right.
• Evaluating code only by whether it runs.
• Letting the model do arithmetic on large tables in its head.
• Treating a verification pass as proof of quality: it only rules out the error types it tests.

Direct answer

Randomize by user (so one person always sees one arm), measure the share of answers containing at least one unsupported claim as the primary metric using a fixed grading procedure that is blind to arm, and watch resolution, escalation, latency and refusal rate as guardrails. Size the test for the measured difference, not the true one: judge error shrinks the visible effect, and several answers per user are correlated, so both inflate the sample you need. For an 8 percent to 5 percent reduction the computed requirement is about 2,570 graded answers per arm (roughly 640 users per arm at 4 answers each), and the guardrail on resolution rate, not the hallucination metric, is the largest requirement.

Design

Unit of randomization. The user (or the account), assigned by a hash of the id and a salt, held fixed for the whole experiment. Randomizing per message lets the same person see both versions, which contaminates their later behaviour and makes the two arms share users. Analysis unit is the answer, but standard errors must respect the user clustering.

Arms. Control: the current assistant. Treatment: the same assistant with retrieval grounding (it is given passages from the support knowledge base and told to answer from them). Everything else, including model, prompt skeleton and temperature, is held identical, so any difference is attributable to grounding.

Primary metric. Hallucination rate: share of answers with at least one claim unsupported or contradicted by the knowledge base or order data. Define "supported" in a written rubric, grade a random sample of answers from both arms with the same procedure, and hide the arm from graders. A model judge scales, but it is checked against human labels on a calibration subset.

Safety and guardrail metrics. Resolution rate (the user did not need to reopen or escalate), escalation to a human agent, refusal or "I don't know" rate, latency at the 95th percentile, cost per conversation, and a harmful-content rate. Grounding can lower hallucinations simply by refusing more, so a drop in hallucination is only a win if resolution holds.

Regression guardrails and stopping. Pre-register a non-inferiority margin for each guardrail (for example, resolution may not fall by more than 2 points), a sample-ratio check on assignment counts, and a hard stop for severe incidents (a harmful or policy-violating answer found in review ends the treatment arm immediately, without waiting for the test to finish). Do not peek and stop on the first significant hallucination result: fix the sample, or use a sequential method that was chosen in advance.

Sample size, computed

The per-arm size for a two-sided two-proportion test comparing rates $p_1$ and $p_2$ is

where $z$ values are standard normal quantiles, $\alpha = 0.05$ is the false-positive rate and $1-\beta = 0.80$ is the power (the chance of detecting a real effect). The program applies it four times and then checks the formula by simulation. The 8 percent baseline, 5 percent target, judge sensitivity 0.85, judge specificity 0.97 (sensitivity is the share of real hallucinations the judge flags, specificity the share of clean answers it passes), 4 answers per user and within-user correlation 0.10 are all assumed design inputs, not measurements.

Terms in plain words. A type I error is declaring an effect when there is none; its rate is the 0.05 above. A one-sided test asks only whether the treatment is worse, not whether it differs in either direction. A non-inferiority margin is the largest worsening of a guardrail metric you would tolerate (here 2 points of resolution rate), fixed before the test starts. Within-user correlation (0.10) says how much two answers from the same person resemble each other beyond chance; because they carry overlapping information, four answers from one user are worth fewer than four independent answers, which is called clustering. The design effect, 1 + (m - 1) x rho = 1 + 3 x 0.10 = 1.30, is the factor by which the sample must grow; equivalently, each user's 4 answers count as 4 / 1.30 = 3.1 independent answers. The sample-ratio check (SRM, for sample ratio mismatch) tests whether the numbers of users in the two arms match the planned split. Monte Carlo means repeating a random experiment many times on a computer to see how often it succeeds. Intention to treat means analysing each user in the arm they were assigned to, whatever happened to them. Pre-registering is writing down the metrics, margins and stopping rule before seeing results. A sequential method is a testing procedure built to allow planned early looks without inflating false positives.

import os
os.environ["OMP_NUM_THREADS"] = "1"
import numpy as np
from scipy.stats import norm, chi2

def n_two_prop(p1, p2, alpha=0.05, power=0.80):
    """Per-arm n for a two-sided two-proportion z-test (unpooled variance, normal approximation)."""
    za, zb = norm.ppf(1 - alpha / 2), norm.ppf(power)
    return (za + zb) ** 2 * (p1 * (1 - p1) + p2 * (1 - p2)) / (p1 - p2) ** 2

def n_noninferior(p, margin, alpha=0.05, power=0.80):
    """Per-arm n to show treatment is not worse than control by more than margin (one-sided), true difference 0."""
    za, zb = norm.ppf(1 - alpha), norm.ppf(power)
    return (za + zb) ** 2 * 2 * p * (1 - p) / margin ** 2

p_ctrl, p_trt = 0.08, 0.05            # true hallucination rates (assumed design targets)
n_true = n_two_prop(p_ctrl, p_trt)
print(f"1) answers per arm, true rates 8% vs 5%: {n_true:.0f}")

se, sp = 0.85, 0.97                   # assumed judge sensitivity and specificity
obs = lambda p: p * se + (1 - p) * (1 - sp)
o1, o2 = obs(p_ctrl), obs(p_trt)
n_obs = n_two_prop(o1, o2)
print(f"2) judge se={se} sp={sp}: observed rates {o1:.4f} vs {o2:.4f}, answers per arm {n_obs:.0f}")

m, icc = 4, 0.10                      # answers per user, within-user correlation (assumed)
deff = 1 + (m - 1) * icc
print(f"3) design effect {deff:.2f}: answers per arm {n_obs * deff:.0f}, users per arm {n_obs * deff / m:.0f}")

n_guard = n_noninferior(0.70, 0.02)
print(f"4) guardrail (resolution 70%, margin 2 points): answers per arm {n_guard:.0f}")

# Monte Carlo power check of formula 1 (independent answers)
rng = np.random.default_rng(0)
n = int(np.ceil(n_true)); reps = 4000
a = rng.binomial(n, p_ctrl, reps); b = rng.binomial(n, p_trt, reps)
pa, pb = a / n, b / n
pool = (a + b) / (2 * n)
z = (pa - pb) / np.sqrt(2 * pool * (1 - pool) / n)
print(f"5) simulated power at n={n}: {np.mean(np.abs(z) > norm.ppf(0.975)):.3f}  type I error at equal rates: ", end="")
a = rng.binomial(n, p_ctrl, reps); b = rng.binomial(n, p_ctrl, reps)
pool = (a + b) / (2 * n); z = (a / n - b / n) / np.sqrt(2 * pool * (1 - pool) / n)
print(f"{np.mean(np.abs(z) > norm.ppf(0.975)):.3f}")

# Sample ratio mismatch check: planned 50/50
def srm_p(c, t):
    e = (c + t) / 2
    return chi2.sf((c - e) ** 2 / e + (t - e) ** 2 / e, 1)
print(f"6) SRM p-value for 5000/5100: {srm_p(5000, 5100):.3f}; for 5000/5300: {srm_p(5000, 5300):.5f}")

Output:

1) answers per arm, true rates 8% vs 5%: 1056
2) judge se=0.85 sp=0.97: observed rates 0.0956 vs 0.0710, answers per arm 1977
3) design effect 1.30: answers per arm 2570, users per arm 642
4) guardrail (resolution 70%, margin 2 points): answers per arm 6492
5) simulated power at n=1057: 0.802  type I error at equal rates: 0.043
6) SRM p-value for 5000/5100: 0.320; for 5000/5300: 0.00312

Reading the numbers:

1. With perfect labels, 1,056 answers per arm detect a drop from 8 percent to 5 percent. By hand, with the rounded constants: (1.96 + 0.84)^2 x [0.08 x 0.92 + 0.05 x 0.95] / 0.03^2 = 7.84 x 0.1211 / 0.0009 = 1,055. The program uses the unrounded normal quantiles 1.95996 and 0.84162 and gets 1,056.
2. An imperfect judge changes what is observed: the visible rates become 9.56 percent and 7.10 percent, a 2.46 point gap instead of 3, and the requirement rises to 1,977 per arm. The visible rate is the true rate times the judge's sensitivity plus the clean share times its false-alarm rate: control 0.08 x 0.85 + 0.92 x 0.03 = 0.0680 + 0.0276 = 0.0956; treatment 0.05 x 0.85 + 0.95 x 0.03 = 0.0425 + 0.0285 = 0.0710. Missed hallucinations shrink the real part of each rate, and the 3 percent false alarms add a floor that both arms share, so the gap becomes the true gap times (sensitivity + specificity - 1): 0.03 x 0.82 = 0.0246.
3. Four answers per user with correlation 0.10 gives a design effect of $1 + (m-1)\rho = 1.30$, so 2,570 answers, or 642 users, per arm.
4. The non-inferiority check on resolution rate (baseline 70 percent, margin 2 points, one-sided) needs 6,492 independent answers per arm, more than the primary metric. By hand: (1.6449 + 0.8416)^2 x 2 x 0.70 x 0.30 / 0.02^2 = 6.183 x 0.42 / 0.0004 = 6,492 (the one-sided 5 percent constant is 1.6449 and the 80 percent power constant is 0.8416). Plan the run length from the largest requirement, and note that clustering would raise this one too.
5. A simulation of 4,000 experiments at n = 1,057 gives power 0.802 under the true rates and a false-positive rate of 0.043 when both arms are equal; these are the values from the run shown, close to the nominal 0.80 and 0.05 for independent answers. The code draws 4,000 fake experiments, each giving every arm n answers with the assumed hallucination rate; for each it computes the z statistic, the observed gap divided by its standard error under the pooled rate, and counts how often the absolute z exceeds 1.96. Run with the true 8 and 5 percent rates, that share is the power; run again with both arms at 8 percent, it is the type I error.
6. With a flagging threshold of p < 0.01, the assignment-count check flags 5000 versus 5300 (p = 0.00312) but not 5000 versus 5100 (p = 0.320). The threshold is a choice made before the test: a stricter 0.001, which some teams use because the check runs on every experiment, would not flag 5000 versus 5300. The chi-square statistic is the sum over the two arms of (observed - expected)^2 / expected, where a 50/50 plan expects half the users in each arm. For 5000 versus 5100 the expected count is 5,050 per arm, so the statistic is 50^2 / 5050 + 50^2 / 5050 = 0.99 and the p-value (chance of a split at least this uneven under a fair assignment) is 0.320. For 5000 versus 5300 the expected count is 5,150 and the statistic is 2 x 150^2 / 5150 = 8.74, p = 0.00312. A flagged mismatch means the assignment or logging is broken and the result should not be read.

Trade-offs and pitfalls

• Human grading is the expensive step. Grade the sampled 2,570 answers per arm with the model judge and have humans label a subset to estimate judge sensitivity and specificity; correct or at least report the bias.
• Measurement error attenuates. A judge with imperfect sensitivity hides real improvements; an unblinded or arm-aware judge can invent them.
• Novelty and learning effects. Users may adapt over days; run for whole weeks and look at the time trend.
• Exposure counts. Analyse by assignment (intention to treat), not by whether retrieval actually fired, otherwise the arms are no longer comparable.
• The program uses the normal approximation; very small rates or small samples call for an exact or simulation-based power calculation.

Direct answer

Group the labels by model_version, compute the mean of the 0/1 column (that mean is the hallucination rate), build a bootstrap confidence interval for each rate by resampling the rows of that group with replacement many times, and compare every non-baseline version with the baseline by bootstrapping the difference of rates. Because you test several versions against one baseline, tighten the threshold for "significant" (here a Bonferroni correction) or you will flag a version by luck. The code below does all of this, and cross-checks the flags with Fisher's exact test (scipy 1.14.1), which is explained below.

What the code does, step by step

1. Data. A DataFrame with one row per labelled request. The program builds a synthetic one with a fixed seed: 400 labelled requests for each of v1, v2, v3 and only 60 for v4. v1 is the baseline.
2. Rate. groupby("model_version") then the mean of hallucinated. A mean of 0/1 values is the proportion of 1s.
3. Bootstrap. For a group with $n$ rows, draw $B = 5000$ index arrays of length $n$ with replacement (rng.integers(0, n, size=(B, n))), take the mean of each row of the resampled matrix, and read the 2.5th and 97.5th percentiles. That is a percentile bootstrap 95% interval. Each of the 5000 means is "what the rate would have been on a different draw of the same size from the same population", so the spread of those means approximates the sampling noise. Resampling with replacement means every pick is made from all the original rows, so a row can appear several times in one resample and others not at all. A tiny case with five labelled requests (labels 1, 0, 0, 1, 0, rate 0.4), resampled twice by the program further down, gives row picks [4, 3, 2, 1, 1] (rate 0.2) the first time and [0, 0, 0, 0, 4] (rate 0.8) the second: the rate wobbles from draw to draw, and the bootstrap interval is the middle 95% of thousands of such wobbles.
4. Difference against baseline. Subtract the baseline's bootstrap rates from the candidate's, element by element, giving 5000 draws of the difference. The two groups are resampled independently, which is right only when the two versions were labelled on different requests, as here.
5. Significance flag. A version is flagged when the interval of the difference excludes zero. Zero is the difference "no change", so if the plausible range of differences includes 0, the data are compatible with the two versions having the same rate, and there is no evidence of a change; if the whole range sits on one side of 0, the sign of the change is clear. With 3 comparisons, the interval uses the adjusted level $\alpha / 3 = 0.0167$ (the 0.83rd and 99.17th percentiles). This is the Bonferroni correction: each single test is allowed a false-flag chance of $\alpha / 3$, so across three tests the chance of at least one false flag is at most about $3 \times \alpha / 3 = \alpha$, which is 5%. The program below computes it exactly: three tests at 0.05 each give 14.3% (the figure in the pitfalls), and three at 0.05 / 3 each give 4.9%.
6. Cross-check. Fisher's exact test works from the 2x2 table of counts: one row per version, one column for hallucinated and one for not (for v3 against v1 the program prints [[79, 321], [42, 358]]). It returns a p-value: the probability, assuming both versions truly hallucinate at the same rate, of seeing a split at least as lopsided as this one. A small p-value means the data would be surprising if nothing had changed; it is not the probability that the versions are the same. Flag when the p-value is below the same adjusted level. It should agree with the bootstrap flag, and in this run it does for all three versions.

import numpy as np
import pandas as pd
from scipy.stats import fisher_exact

rng = np.random.default_rng(0)

# Synthetic labels table: different requests per version (independent samples).
spec = {"v1": (400, 0.10), "v2": (400, 0.12), "v3": (400, 0.19), "v4": (60, 0.20)}
rows = []
for v, (n, p) in spec.items():
    for i in range(n):
        rows.append((f"{v}-{i}", v, int(rng.random() < p)))
df = pd.DataFrame(rows, columns=["request_id", "model_version", "hallucinated"])

B = 5000
BASELINE = "v1"
ALPHA = 0.05
m = df["model_version"].nunique() - 1          # number of comparisons against baseline
alpha_adj = ALPHA / m                           # Bonferroni

def boot_rates(x, B, rng):
    # resample rows with replacement, B times; return B bootstrap rates
    idx = rng.integers(0, len(x), size=(B, len(x)))
    return x[idx].mean(axis=1)

groups = {v: g["hallucinated"].to_numpy() for v, g in df.groupby("model_version")}
boots = {v: boot_rates(x, B, rng) for v, x in groups.items()}

out = []
for v, x in groups.items():
    lo, hi = np.percentile(boots[v], [2.5, 97.5])
    row = {"version": v, "n": len(x), "rate": x.mean(), "ci_lo": lo, "ci_hi": hi}
    if v != BASELINE:
        d = boots[v] - boots[BASELINE]
        dlo, dhi = np.percentile(d, [100 * alpha_adj / 2, 100 * (1 - alpha_adj / 2)])
        row.update(diff=x.mean() - groups[BASELINE].mean(), d_lo=dlo, d_hi=dhi,
                   flag_boot=(dlo > 0) or (dhi < 0))
        tbl = [[x.sum(), len(x) - x.sum()],
               [groups[BASELINE].sum(), len(groups[BASELINE]) - groups[BASELINE].sum()]]
        row["fisher_p"] = fisher_exact(tbl)[1]
        row["flag_fisher"] = row["fisher_p"] < alpha_adj
    out.append(row)

res = pd.DataFrame(out).round(3)
pd.set_option("display.width", 200)
print(res.to_string(index=False))
print("comparisons:", m, "adjusted alpha:", round(alpha_adj, 4))
b, c3 = groups["v1"], groups["v3"]
print("v3 vs v1 2x2 table [[v3 hallucinated, v3 not], [v1 hallucinated, v1 not]]:",
      [[int(c3.sum()), int(len(c3) - c3.sum())], [int(b.sum()), int(len(b) - b.sum())]])

Output:

version   n  rate  ci_lo  ci_hi   diff   d_lo  d_hi flag_boot  fisher_p flag_fisher
     v1 400 0.105  0.075  0.135    NaN    NaN   NaN       NaN       NaN         NaN
     v2 400 0.085  0.058  0.112 -0.020 -0.070 0.030     False     0.399       False
     v3 400 0.198  0.160  0.238  0.093  0.032 0.152      True     0.000        True
     v4  60 0.217  0.117  0.317  0.112 -0.015 0.249     False     0.019       False
comparisons: 3 adjusted alpha: 0.0167
v3 vs v1 2x2 table [[v3 hallucinated, v3 not], [v1 hallucinated, v1 not]]: [[79, 321], [42, 358]]

Reading the result

• v2 has a rate 2.0 points below baseline, but the difference interval (-0.070 to 0.030) contains zero, and Fisher's p is 0.399, meaning that a gap this large would be common (about 40% of the time) if the two versions were truly the same: no evidence of a change.
• v3 is 9.3 points above baseline, the adjusted interval (0.032 to 0.152) excludes zero, and Fisher's p is far below 0.0167: flagged by both methods.
• v4 has the highest observed rate (0.217) but only 60 labels. Its own interval is wide (0.117 to 0.317) and its difference interval (-0.015 to 0.249) contains zero, so it is not flagged. Fisher's p is 0.019, just above the adjusted 0.0167, so both methods agree in this run (a different draw of 60 labels could land on the other side of the line). The honest reading is "not enough labels", not "safe". A 60-label sample can only detect large differences.

Pitfalls a reviewer should look for

• Duplicate request_id values. If the same request was labelled twice, rows are not independent and the interval is too narrow. De-duplicate or resample by request_id.
• Paired data. If every version answered the same requests, resampling the groups independently throws away the pairing and widens the interval. Resample request ids once and use the same ids for both versions, or use McNemar's test, a test for paired yes/no outcomes that looks only at the requests where the two versions disagree.
• Clustering. Several requests from one user are correlated; resample users, not rows.
• Small groups and rare rates. A percentile bootstrap is rough when a group has few labels or a rate near 0 or 1; with 60 labels treat the interval as approximate, and prefer an exact or Wilson interval for a single rate (both are closed-form or exact intervals for a proportion that behave better than the percentile bootstrap when counts are small).
• Multiple comparisons. Without the correction, three comparisons at 0.05 each give about a 14% chance of at least one false flag when nothing changed (1 - 0.95^3 = 0.143 if the tests were independent). Bonferroni is simple and conservative. Holm is a step-by-step variant that sorts the p-values, compares the smallest with $\alpha / 3$, the next with $\alpha / 2$ and the last with $\alpha$, and stops at the first one that fails; it never flags fewer versions than Bonferroni while keeping the same overall false-flag limit. The simulation below shows the effect of the correction: with three versions that truly have the same rate, flagging at 0.05 per test raised some false flag in about 10% of experiments, and dividing by 3 cut that to about 3% (the tests share one baseline and counts are discrete, so both figures sit below the independent-test arithmetic).

import numpy as np

labels = np.array([1, 0, 0, 1, 0])          # five labelled requests, rate = 2/5 = 0.4
rng = np.random.default_rng(0)
for k in range(2):
    idx = rng.integers(0, len(labels), size=len(labels))   # pick 5 row positions, repeats allowed
    print(f"resample {k + 1}: rows {idx.tolist()} -> labels {labels[idx].tolist()} -> rate {labels[idx].mean():.1f}")

# Why dividing alpha by the number of comparisons keeps false flags near 5%
alpha, m = 0.05, 3
print("chance of at least one false flag, 3 tests at 0.05 each:", round(1 - (1 - alpha) ** m, 3))
print("chance of at least one false flag, 3 tests at 0.05/3 each:", round(1 - (1 - alpha / m) ** m, 3))

# Simulation: versions with identical true rates, 3 tests per experiment, 2000 experiments
from scipy.stats import fisher_exact
r = np.random.default_rng(1)
def false_flag_rate(threshold, reps=2000):
    hits = 0
    for _ in range(reps):
        base = r.binomial(400, 0.1)
        ps = [fisher_exact([[x, 400 - x], [base, 400 - base]])[1] for x in r.binomial(400, 0.1, size=3)]
        hits += any(p < threshold for p in ps)
    return hits / reps
print("simulated, no real change, flag at p < 0.05      :", false_flag_rate(0.05))
print("simulated, no real change, flag at p < 0.05 / 3  :", false_flag_rate(0.05 / 3))

Output:

resample 1: rows [4, 3, 2, 1, 1] -> labels [0, 1, 0, 0, 0] -> rate 0.2
resample 2: rows [0, 0, 0, 0, 4] -> labels [1, 1, 1, 1, 0] -> rate 0.8
chance of at least one false flag, 3 tests at 0.05 each: 0.143
chance of at least one false flag, 3 tests at 0.05/3 each: 0.049
simulated, no real change, flag at p < 0.05      : 0.1015
simulated, no real change, flag at p < 0.05 / 3  : 0.0325

• Label quality. The interval covers sampling noise only. If the labellers disagree or the label definition changed between versions, no interval fixes that.
• Flagging is not a launch decision. A significant difference of one point may not matter; decide the smallest difference you care about before looking, and report the interval of the difference, not only the flag.