LLM Evaluation and Observability Questions

Direct answer

In the first hour, contain first, diagnose second, decide on rollback third. Containment means stopping further harm with the cheapest reversible lever (turn off the affected feature path, route to a fallback or return to the previous model version), not finding the root cause. Then measure how much worse the new version really is compared with the old one, find where the problem concentrates, and tell the privacy or legal owner early if personal data is involved (PII, personally identifiable information: names, emails, phone numbers, account numbers), since notification clocks, the legal deadlines for telling regulators and affected customers, often start when the company becomes aware of the breach.

Timeline

What to inspect

Pull these fields for both versions over the same window, so that only the version differs:

• Version fields on every request: model version and prompt template version (they separate "the new model is worse" from "the new prompt is worse").
• Concentration fields: tenant, language, tool called, retrieval source. A problem that lives in one value of one field is contained by switching off that value.
• Output-safety metrics: toxicity flag rate from the output classifier, per version and per template; the number of requests, so every rate has its denominator; the filter's own pass rate, to see whether the filter itself changed.
• Personal-data fields: hits from a personal-data scan of replies, with a note on whether each hit string also appears in the retrieved context.
• Human signals: user reports and support tickets, which catch what the classifier misses.

The program builds a synthetic one-hour request log: a flag from a stub toxicity classifier for 30,000 requests on each of two versions, spread over five prompt templates. The new version's rate is raised on one template by construction; the point is the method. Then it runs a deliberately minimal personal-data scan on five hand-written replies. The scan covers plain email addresses and 10-digit phone numbers written with dashes, dots or spaces; it is an illustration, not a production detector. The email pattern reads: letters, digits or the characters . _ % + - , then @, then a domain name, a dot and at least two letters. The phone pattern reads: three digits, a dash, dot or space, three digits, another separator, then four digits, with no further digit touching either end. A flag rate from 30,000 requests is an estimate, so the program also computes a 95% bound: upper() is a rate the true value is unlikely to exceed and lower() a rate it is unlikely to fall below (both are one-sided Clopper-Pearson bounds, read from a Beta distribution). Comparing the new version's lower bound with the old version's upper bound asks whether the gap survives sampling noise. The factor 3 in the rollback rule is not derived from anything: it is a margin chosen in advance so that only a large, clearly meaningful increase triggers a rollback, and your team would set its own. Libraries: numpy 2.1.3, pandas 2.2.3, scipy 1.14.1.

import os
os.environ["OMP_NUM_THREADS"] = "1"
import re
import numpy as np
import pandas as pd
from scipy.stats import beta

def upper(k, n, conf=0.95):
    return float(beta.ppf(conf, k + 1, n - k)) if k < n else 1.0
def lower(k, n, conf=0.95):
    return float(beta.ppf(1 - conf, k, n - k + 1)) if k > 0 else 0.0

# Synthetic one-hour request log (stand-in for real telemetry); the flag is a stub classifier's output.
rng = np.random.default_rng(0)
templates = ["faq", "billing", "returns", "roleplay", "summarize"]
def make(version, n, bump):
    t = rng.choice(templates, n, p=[0.35, 0.25, 0.2, 0.1, 0.1])
    p = np.where(t == "roleplay", bump, 0.001)
    return pd.DataFrame({"version": version, "template": t, "flagged": rng.random(n) < p})
log = pd.concat([make("v1", 30000, 0.001), make("v2", 30000, 0.06)], ignore_index=True)

g = log.groupby("version").flagged.agg(["sum", "count"])
for v, row in g.iterrows():
    print(f"{v}: {int(row['sum'])}/{int(row['count'])} flagged = {row['sum']/row['count']:.3%}")
k1, n1 = g.loc["v1"]; k2, n2 = g.loc["v2"]
print(f"v1 upper 95% bound {upper(int(k1),int(n1)):.3%}; v2 lower 95% bound {lower(int(k2),int(n2)):.3%}")
print("rollback rule 'v2 lower bound > 3 x v1 upper bound':",
      lower(int(k2), int(n2)) > 3 * upper(int(k1), int(n1)))

by = log[log.version == "v2"].groupby("template").flagged.agg(["sum", "count"])
by["rate"] = by["sum"] / by["count"]
by["share_of_flags"] = by["sum"] / by["sum"].sum()
print("\nv2 by template")
print(by.round(4).to_string())

# Minimal illustration of a PII scan: covers plain email addresses and 10-digit US-style phone numbers only.
EMAIL = re.compile(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}")
PHONE = re.compile(r"(?<![0-9])[0-9]{3}[-. ][0-9]{3}[-. ][0-9]{4}(?![0-9])")
def pii(text):
    return EMAIL.findall(text) + PHONE.findall(text)

context = "Order 5521 for jo.park@example.com ships Friday."
outputs = {
  "echoes the customer from context": "Hi, I see jo.park@example.com has order 5521.",
  "unrelated address not in context": "You can also reach lee.cho@example.org about this.",
  "phone number not in context": "Call 555-014-2297 for help.",
  "obfuscated address": "Write to jo dot park at example dot com.",
  "clean": "Your order ships Friday.",
}
print("\nPII scan: hit, hit string(s) found in the retrieved context?")
for name, text in outputs.items():
    hits = pii(text)
    in_ctx = [h in context for h in hits]
    print(f"{name:<34} hits={hits} in_context={in_ctx}")

Output:

v1: 25/30000 flagged = 0.083%
v2: 223/30000 flagged = 0.743%
v1 upper 95% bound 0.116%; v2 lower 95% bound 0.664%
rollback rule 'v2 lower bound > 3 x v1 upper bound': True

v2 by template
           sum  count    rate  share_of_flags
template                                     
billing      5   7581  0.0007          0.0224
faq         12  10496  0.0011          0.0538
returns     10   6095  0.0016          0.0448
roleplay   192   2860  0.0671          0.8610
summarize    4   2968  0.0013          0.0179

PII scan: hit, hit string(s) found in the retrieved context?
echoes the customer from context   hits=['jo.park@example.com'] in_context=[True]
unrelated address not in context   hits=['lee.cho@example.org'] in_context=[False]
phone number not in context        hits=['555-014-2297'] in_context=[False]
obfuscated address                 hits=[] in_context=[]
clean                              hits=[] in_context=[]

Reading the results

1. Is it real? The old version flagged 25 of 30,000 (0.083%, upper bound 0.116%); the new one 223 of 30,000 (0.743%, lower bound 0.664%). The rule used here, "new lower bound above three times the old upper bound", fires. A rule of this kind is a policy choice; write yours down before the incident.
2. Where is it? One template, roleplay, has a 6.71% flag rate and accounts for 192 of the 223 flags (86.1%) from about a tenth of the traffic. Other templates sit between 0.07% and 0.16%. This points to an interaction between the new model and that prompt, and gives a surgical containment: disable or tighten that template while the rest keeps serving.
3. Personal data: where did the string come from? A hit whose string is in the retrieved context (the first reply) is a retrieval or authorisation question: was that customer's data supposed to be in this conversation? A hit that is not in the retrieved context (the second and third replies) still needs tracing before it is called invented or memorised: check the system prompt (the third reply's number could be the company's own support line), the earlier turns of the conversation, tool results, and other customers' records that a retrieval or caching bug could have mixed in. Only when none of those holds is the string invented or reproduced from text the model saw during training (memorised). The program checks the retrieved context only; the other sources need the full request log. The scan missed the obfuscated address in the fourth reply, which shows why detection must not rely on one pattern. A production system also needs names, postal addresses, account and card numbers with checksum validation (a built-in check digit, such as the Luhn check on card numbers, that rejects most random digit strings but still lets about one in ten through, so it narrows candidates rather than proving a match), other languages and formats, and a model-based detector, and still sends a sample to human review.

Rollback decision

Roll back when the harm is clearly attributable to the new version, containment of one path is not enough or not possible, and the old version is still deployable (model and prompt versions pinned, meaning fixed to exact identifiers so nothing changed silently; data formats the old version can still read; warm capacity, meaning old-version servers already running that can take the full traffic). Prefer rollback over a live patch for personal-data leakage, since each hour adds exposure. Prefer path-level containment when the data localises the problem and rollback would remove a fix the old version needs. After rollback, confirm the rates return to the old baseline.

Failure modes and common mistakes

• Debugging first while the leak continues.
• Searching raw logs for personal data and copying it into chat tools.
• Comparing against a hand-picked "normal" instead of the previous version on the same traffic.
• Trusting the filter that was supposed to catch the problem, since it may have shared the same blind spot.
• Rolling back the model but not the prompt or retrieval change that shipped with it.

Direct answer

Perplexity measures how surprised a language model is by a piece of text. It is the exponential of the average negative log-probability the model assigned to each token that actually occurred:

Here $N$ is the number of predicted tokens (a token is the unit the model reads: a word or word-piece), $x_i$ is the $i$-th token and $x_{<i}$ is everything before it. The average inside the exponent is the model's cross-entropy on that text: the average surprise per token, where the surprise of a token is its negative log-probability (the natural logarithm, written $\ln$, of a probability below 1 is negative, so its negative is a positive number that grows as the probability shrinks). That surprise is measured in nats when the logarithm is natural, and in bits when it is base 2. So perplexity is the exponential of the cross-entropy. A perplexity of $k$ means the model is, on average, as uncertain as if it were choosing uniformly among $k$ equally likely tokens. Lower is better at predicting that text; higher means the model found the text surprising. On its own it says how well a model fits text, not whether the text it generates is correct, helpful or safe.

Perplexity by hand

Suppose a model gave probability 0.25 to each of four tokens that actually occurred. The surprise of each token is $-\ln 0.25 = 1.386$, the average is 1.386, and $\exp(1.386) = 4$: the model behaved as if it were choosing among 4 equally likely tokens at each step. If it had given 0.90 to each token, the surprise per token is 0.105 and the perplexity is 1.11, almost no uncertainty. The two cases, computed:

import math
# Perplexity by hand: ln is the natural logarithm, exp undoes it.
for name, probs in [("uniform 0.25 on every token", [0.25, 0.25, 0.25, 0.25]),
                    ("0.90 on every token", [0.90, 0.90, 0.90, 0.90])]:
    surprise = [-math.log(p) for p in probs]       # negative log-probability of each token
    cross_entropy = sum(surprise) / len(surprise)  # average surprise per token, in nats
    print(f"{name}: surprise per token {surprise[0]:.3f}, average {cross_entropy:.3f}, perplexity {math.exp(cross_entropy):.2f}")

Output:

uniform 0.25 on every token: surprise per token 1.386, average 1.386, perplexity 4.00
0.90 on every token: surprise per token 0.105, average 0.105, perplexity 1.11

What higher and lower values tell you

• Low perplexity on a test set: the model assigns high probability to what really came next. This usually goes with fluent, in-distribution text (text that resembles what the model was trained on).
• High perplexity: the text is unlike the training data (another domain, a typo-laden source, a new language) or the model is weak.
• It is only comparable between models when they are evaluated on the same text with the same tokenization, meaning the same rule for splitting text into tokens (a tokenizer is the program that applies that rule).

A worked example on a tiny model

The program trains word-level unigram and bigram models (a unigram model ignores context; a bigram model conditions on the previous word) with add-0.1 smoothing on a 42-token invented corpus about capitals and rivers (the full stops count as tokens). Smoothing gives unseen words a small non-zero probability. These tiny models are stand-ins for a neural language model; the computation of perplexity is the same. It scores four test sentences, then re-expresses one sentence's loss per character.

import math
from collections import Counter
train = ("the capital of france is paris . the capital of germany is berlin . "
         "the capital of italy is rome . the capital of spain is madrid . "
         "the river seine is in paris . the river spree is in berlin .").split()
V = sorted(set(train)); K = 0.1   # add-K smoothing
uni = Counter(train); bi = Counter(zip(train, train[1:]))
def p_uni(w): return (uni[w] + K) / (len(train) + K * len(V))
def p_bi(w, prev): return (bi[(prev, w)] + K) / (uni[prev] + K * len(V))
def ppl(text, model):
    toks = text.split(); nll = 0.0
    for i in range(1, len(toks)):
        p = p_uni(toks[i]) if model == "uni" else p_bi(toks[i], toks[i-1])
        nll -= math.log(p)
    return math.exp(nll / (len(toks) - 1)), nll
tests = {
 "true      ": "the capital of germany is berlin .",
 "false     ": "the capital of germany is paris .",
 "scrambled ": "is of the germany capital berlin .",
 "repetitive": "the the the the the the .",
}
for name, t in tests.items():
    print(name, "unigram ppl %.2f   bigram ppl %.2f" % (ppl(t, "uni")[0], ppl(t, "bi")[0]))

# Same text, same model, two tokenisations: perplexity not comparable, total NLL is the invariant.
text = "the capital of germany is berlin ."
_, nll_words = ppl(text, "bi"); n_words = len(text.split()) - 1
n_chars = len(text) - len(text.split()[0])
print("words:", n_words, "perplexity per word %.2f" % math.exp(nll_words / n_words))
print("same total NLL spread over %d characters: perplexity per char %.2f" % (n_chars, math.exp(nll_words / n_chars)))
print("bits per character %.3f" % (nll_words / math.log(2) / n_chars))

Output:

true       unigram ppl 13.00   bigram ppl 2.73
false      unigram ppl 13.00   bigram ppl 2.73
scrambled  unigram ppl 13.00   bigram ppl 31.17
repetitive unigram ppl 7.16   bigram ppl 77.00
words: 6 perplexity per word 2.73
same total NLL spread over 31 characters: perplexity per char 1.21
bits per character 0.280

Reading it:

• A false sentence scores exactly like a true one. "the capital of germany is paris" and "the capital of germany is berlin" both get bigram perplexity 2.73, because in the training text "is" is followed by "paris" once and by "berlin" once. The unigram column ties because it ignores context entirely and "paris" and "berlin" each occur twice in the training text. The scrambled sentence also gets 13.00, but only because the word it scores in place of "is" is "the" (the first word of each sentence is context only and is not scored), and both of those words occur 6 times in the training text. The model measures word-pattern fit, not truth. A real model sees far more context than this toy and will often prefer the true sentence, but nothing in the perplexity formula checks facts: it rewards text that matches patterns in the training data, so a fluent falsehood that fits those patterns can still earn a low perplexity.
• Word order matters to a context model, not to the unigram. Apart from which word comes first, the scrambled sentence is the same bag of words, so its unigram perplexity is the same 13.00, but its bigram perplexity is 31.17.
• Low perplexity is not good generation. The repetitive string scores a low 7.16 under the unigram model (the word "the" is common), while the bigram model rates it 77.00 because "the the" never occurred. Which model you ask changes the verdict, and a looping real model can also drive its own perplexity down by repeating.
• Tokenization changes the number. The same total loss gives 2.73 per word but 1.21 per character, because dividing by more tokens gives a smaller per-token value. Perplexities from models with different vocabularies are not comparable. Bits per character puts models on one basis: it takes the total loss (6.02 nats for this sentence), converts nats to bits by dividing by $\ln 2 \approx 0.693$ (8.68 bits), and divides by the 31 characters, giving 0.280. The count of characters does not depend on the tokenizer, so the number can be compared across models.

How far perplexity can carry evaluation

For open-ended generation add task metrics, reference-based overlap scores where a reference exists, human ratings against a rubric, and model-graded evaluation checked against human labels.

Pitfalls

• Reporting perplexity on text the model trained on, which flatters it.
• Comparing across tokenizers or languages.
• Ignoring the first-token convention: the first token has no context, so implementations differ on whether it is scored. State yours (here the first token is context only and is not scored).

Direct answer

Use a cascade: cheap deterministic checks on all traffic, a model judge (a second model that grades answers against a rubric) on a stratified daily sample (drawn separately within each group of requests), and humans on a smaller periodic sample whose main job is to keep the judge honest. Money is spent where it buys information: rules cover what can be checked exactly, the judge covers what needs reading, and human labels calibrate the judge and settle disputed cases. Daily regression detection compares a rolling window against a stored baseline with a threshold derived from the sampling noise. The computed example shows that a sample of 600 judged conversations per day can see a large shift only after several days of pooling, which is the central budget trade-off.

The layers

The golden set (a frozen collection of questions with reference answers or rubric labels) catches regressions on known behaviour; the sampled live traffic catches new behaviour. A change ships only when both are clean.

Sampling

Split traffic by intent, the kind of request the customer is making (billing, shipping or account questions in the example), and sample each group, called a stratum, separately. This is stratifying: failure rates differ between strata, and a plain random sample can under-represent a small one. Neyman allocation gives stratum $h$ the judged sample

where $N$ is the total judged per day, $w_h$ is the stratum's share of traffic and $\sigma_h = \sqrt{p_h(1-p_h)}$ is the standard deviation of its pass-or-fail outcome ($p_h$ is its failure rate). Strata whose outcomes vary more get more sample. The standard error is the typical size of the chance error in an estimate made from a sample: the smaller it is, the less a day's number wobbles. Always oversample rare high-severity categories separately, and reweight when reporting: weight each stratum's rate by its true share of traffic (not its share of the sample), so oversampling does not distort the overall rate.

Worked example

The program uses assumed failure rates per stratum, a budget of 600 judged conversations per day, and relative cost units (arbitrary: only the ratios matter, here a judge call costs 100 times a rule check and a human label 150 times a judge call). It computes the allocation, simulates windows of 1, 3 and 7 days to see how long it takes to detect a billing regression of 4 points (2 points overall), using this alarm rule: alert when the average of the last k daily estimates exceeds the baseline rate by 3 standard errors of a k-day average (3 standard errors keeps false alarms near 1 in 1,000 checks), tabulates the daily cost per layer, and estimates the agreement between judge and human on a periodic sample, with a bootstrap (re-drawing the 120 labelled items with replacement 2,000 times, recomputing kappa each time and reading off the middle 95 percent, which shows how far kappa would move with a different 120 items). The human counts in part 4 are illustrative, hand-set values.

import os
os.environ["OMP_NUM_THREADS"] = "1"
import numpy as np

# Strata: share of traffic and failure rate (assumed for a hypothetical support assistant)
names = ["billing", "shipping", "account"]
w = np.array([0.50, 0.30, 0.20])
p = np.array([0.10, 0.04, 0.20])
sd = np.sqrt(p * (1 - p))
N_JUDGE = 600                                      # judge-scored conversations per day

n_prop = N_JUDGE * w
n_ney = N_JUDGE * w * sd / np.sum(w * sd)
var = lambda n: np.sum(w ** 2 * p * (1 - p) / n)
print("1) allocation of", N_JUDGE, "judged conversations per day")
print("   strata:       ", names)
print("   proportional: ", np.round(n_prop).astype(int), f"std error of overall rate {np.sqrt(var(n_prop)):.4f}")
print("   Neyman:       ", np.round(n_ney).astype(int), f"std error of overall rate {np.sqrt(var(n_ney)):.4f}")

rng = np.random.default_rng(0)
def day_estimate(n, pp):
    n = np.round(n).astype(int)
    return np.sum(w * rng.binomial(n, pp) / n)
base = np.sum(w * p)
def alarm_rate(n, pp, k, runs=3000, z=3.0):
    """Alarm if the mean of k daily estimates exceeds baseline by z standard errors of a k-day mean."""
    se = np.sqrt(var(n) / k)
    return np.mean([np.mean([day_estimate(n, pp) for _ in range(k)]) > base + z * se for _ in range(runs)])
regress = p + np.array([0.04, 0.0, 0.0])    # billing gets 4 points worse (overall +2 points)
print(f"2) alarm at 3 standard errors above baseline {base:.3f} (3000 simulated windows each, Neyman allocation)")
for k in (1, 3, 7):
    print(f"   {k}-day window: false alarm rate {alarm_rate(n_ney, p, k):.4f}   detection of +4 points in billing {alarm_rate(n_ney, regress, k):.3f}")

# Cost per day, relative units (assumed): rules on all, judge on N_JUDGE, humans on 80
TRAFFIC = 50000
c_rule, c_judge, c_human = 0.0001, 0.01, 1.5
n_human = 80                        # a weekly batch of 560 items averages 80 per day
cost = [("rules on all traffic", TRAFFIC * c_rule), ("judge on sample", N_JUDGE * c_judge), ("human review", n_human * c_human)]
print("3) daily cost, relative units:")
for k, v in cost: print(f"   {k:<22}{v:8.2f}")
print(f"   total {sum(v for _, v in cost):.2f}; judging everything would cost {TRAFFIC * c_judge:.0f}; humans on everything {TRAFFIC * c_human:.0f}")

# Judge vs human agreement on a periodic human sample (illustrative counts)
a, b, c, d = 82, 6, 10, 22           # both ok, judge ok/human bad, judge bad/human ok, both bad
n = a + b + c + d
po = (a + d) / n
pe = ((a + b) * (a + c) + (c + d) * (b + d)) / n ** 2
kappa = (po - pe) / (1 - pe)
labels = np.array([0] * a + [1] * b + [2] * c + [3] * d)
ks = []
for _ in range(2000):
    s = rng.choice(labels, n)
    aa, bb, cc, dd = [(s == i).sum() for i in range(4)]
    po_ = (aa + dd) / n; pe_ = ((aa + bb) * (aa + cc) + (cc + dd) * (bb + dd)) / n ** 2
    ks.append((po_ - pe_) / (1 - pe_))
lo, hi = np.percentile(ks, [2.5, 97.5])
print(f"4) judge vs human on {n} items: agreement {po:.4f}, chance agreement {pe:.4f}, Cohen kappa {kappa:.3f}, bootstrap 95% interval ({lo:.3f}, {hi:.3f})")
m = 0.05
print(f"   items needed to estimate agreement within +-{m} at agreement 0.87: {int(np.ceil(1.96**2 * 0.87 * 0.13 / m**2))}")

Output:

1) allocation of 600 judged conversations per day
   strata:        ['billing', 'shipping', 'account']
   proportional:  [300 180 120] std error of overall rate 0.0121
   Neyman:        [312 122 166] std error of overall rate 0.0118
2) alarm at 3 standard errors above baseline 0.102 (3000 simulated windows each, Neyman allocation)
   1-day window: false alarm rate 0.0017   detection of +4 points in billing 0.112
   3-day window: false alarm rate 0.0010   detection of +4 points in billing 0.478
   7-day window: false alarm rate 0.0020   detection of +4 points in billing 0.917
3) daily cost, relative units:
   rules on all traffic      5.00
   judge on sample           6.00
   human review            120.00
   total 131.00; judging everything would cost 500; humans on everything 75000
4) judge vs human on 120 items: agreement 0.8667, chance agreement 0.6244, Cohen kappa 0.645, bootstrap 95% interval (0.479, 0.795)
   items needed to estimate agreement within +-0.05 at agreement 0.87: 174

Reading it:

• Proportional allocation gives 300, 180 and 120 conversations to the three strata, and Neyman allocation gives 312, 122 and 166. The standard error of the overall rate moves only from 0.0121 to 0.0118. Stratified allocation is a free small gain here, not a large one; the main saving comes from not judging everything.
• A one-day window detects the billing regression only 0.112 of the time at a false-alarm rate of 0.0017. A 3-day pooled window detects it 0.478 of the time and a 7-day window 0.917, with false alarms staying near 0.001 to 0.002. At this sample size, the design choice is therefore "alert within about a week for a 2 point overall regression", and faster alerts need a larger sample or a golden-set check that runs on every deploy.
• Daily cost in the assumed units is 5.00 for rules, 6.00 for the judge and 120.00 for human review, total 131.00. Judging everything would cost 500, and human review of everything 75,000. Human time dominates, so the human sample is the lever to size carefully.
• On 120 doubly-labelled items the judge and human agree on 0.8667 of cases (the 82 + 22 = 104 items where both say ok or both say bad, over 120). Kappa corrects that for luck. The judge says ok on 88 items and the human on 92, so if each guessed independently with its own rate they would agree by chance with probability pe = (88/120)(92/120) + (32/120)(28/120) = 0.6244. Then kappa = (0.8667 - 0.6244) / (1 - 0.6244) = 0.645, with a bootstrap interval of 0.479 to 0.795: with only 120 items the true kappa could plausibly sit anywhere in that range.
• To estimate agreement within plus or minus 5 points at about 0.87 takes 174 items (1.96 squared x 0.87 x 0.13 / 0.05 squared). A weekly batch of 560 items split evenly over three intents is about 187 per intent, just above 174, so it supports per-intent agreement tracking if each intent's agreement is near 0.87; lower agreement needs more items. The 80 human items per day in the cost table are that weekly batch spread over seven days.

Review cost balance

• Spend human effort on disagreements and on high severity: send to a person the cases where the judge is unsure, the judge and rules disagree, or severity is high.
• Use human labels twice: to score the system and to measure the judge's error, which can then be used to correct the judged rate.
• Cache: judge only changed or new conversations, reuse judgments for identical prompt and answer pairs.
• Schedule: golden set on each candidate and nightly, judge sample nightly, humans weekly or when drift alerts fire.

Trade-offs and pitfalls

• A judge from the same model family as the assistant tends to favour its style; check against human labels.
• A stale golden set stops predicting live quality; refresh it from sampled traffic with human labels.
• Stratum weights drift with traffic; recompute them.
• The sampling and detection logic here is a minimal illustration on assumed rates; production systems also need seasonality handling, a baseline that updates, and an incident path when an alert fires.

Direct answer

Treat it as four suspects and look for the signature each one leaves. "Model drift" here means the model's behaviour changed over time while the product looked the same, either because you switched versions or because a provider updated one behind a fixed name. A model change hits every user, on every client, from the instant the model is served, and leaves a model-version field that can be compared on the same day. A prompt change shows up as a step aligned with a prompt-version deploy and affects the conversations that use that prompt. A backend failure appears in error, timeout, empty-answer and tool-failure rates even when CPU and latency look fine. A UX change hits only the clients or surfaces that received it, and can change how feedback is collected as well as how users feel. The fastest separator is a segment cut at the change date: who dropped, who did not, and whether the suspect's rollout lines up with that segment. When a model update shipped just before the drop, timing alone does not convict it; compare the new and old model on the same day and same client.

Order of checks

1. Confirm the drop is real and find its start. Plot the daily rate with intervals, find the change point (the day where the level shifts), and check that the feedback response rate itself did not change (a button that moved can change who votes, not how people feel).
2. Build a change calendar: model versions, prompt versions, client releases, feature flags (switches that turn a feature on for chosen users without a new deploy), backend deploys and dependency changes, each with its rollout percentage (the share of traffic that had it) by day.
3. Cut by segment. Client (web, mobile, API), language, intent category (what the user was trying to do, such as billing or password reset), user tenure (how long the account has existed), model version, prompt version. A cause that applies everywhere cannot produce a drop confined to one client.
4. Look at the failure signals the infrastructure dashboards omit: tool-call error rate, empty or truncated responses, retrieval returning nothing, refusal rate, response length. Healthy CPU and latency say the servers are fine, not that answers are correct.
5. Run a same-day contrast. If a rollout was staggered, meaning a canary (a new version served to a small share of traffic first while the rest stays on the old one), compare treated (got the change) versus untreated (did not) within the same day and segment. If it was an instant 100 percent switch, there is no concurrent control: replay (an offline replay) a fixed set of logged prompts through the old and new configuration and compare graded quality.
6. Read examples. Twenty low-rated conversations from the affected segment, with the config version attached, often name the cause in minutes.

Worked example

The program simulates 28 days of 3,000 requests per day with a fixed seed. By construction: a new model is rolled out to 20 percent of traffic on day 14 and 100 percent from day 18; a new mobile interface reaches all mobile users on day 16; the thumbs-up probability falls by 0.07 only for the new interface; tool errors stay at 1.5 percent. That is the simulated truth. The analysis then runs the checks above as an investigator would, without using the truth. The data is synthetic and the rates are illustrative, not from any real product. The "+-" values are 95 percent half-widths: each is how far the interval reaches on either side of the estimate, computed as $1.96\sqrt{p(1-p)/n}$ for a rate $p$ over $n$ requests. The data is one simulated run, so the statements below describe this run; they are not a claim about every possible draw.

How the code reads: np.column_stack builds one table with a row per request and six columns, and R[:, i] pulls out column i, which is how the line after np.vstack splits the table back into named columns. The change-point line tries every split day k, takes the gap between the average daily rate before and after, and multiplies it by $\sqrt{k(\text{DAYS}-k)/\text{DAYS}}$. The scaling is there because a mean over only a few days is noisy: the noise in the gap between two means is proportional to $\sqrt{\text{DAYS}/(k(\text{DAYS}-k))}$, so multiplying by the inverse stops a split near either end from looking large by luck.

import os
os.environ["OMP_NUM_THREADS"] = "1"
import numpy as np

rng = np.random.default_rng(0)
DAYS, N = 28, 3000
rows = []   # day, client_mobile, model_v2, ui_v2, thumbs_up, tool_error
for d in range(DAYS):
    mobile = rng.random(N) < 0.40
    share_v2 = 0.0 if d < 14 else (0.2 if d < 18 else 1.0)      # model update: canary then full rollout
    v2 = rng.random(N) < share_v2
    ui2 = mobile & (d >= 16)                                      # new mobile UI from day 16
    p_up = 0.70 - 0.07 * ui2                                      # simulated truth: only the UI change matters
    up = rng.random(N) < p_up
    err = rng.random(N) < 0.015                                   # backend tool error, unchanged
    rows.append(np.column_stack([np.full(N, d), mobile, v2, ui2, up, err]))
R = np.vstack(rows)
day, mob, v2, ui2, up, err = [R[:, i] for i in range(6)]

def rate(mask):
    n = mask.sum(); p = up[mask].mean()
    return p, 1.96 * np.sqrt(p * (1 - p) / n), int(n)

daily = np.array([up[day == d].mean() for d in range(DAYS)])
# best single change point: maximise the gap between means before and after
gaps = [(abs(daily[:k].mean() - daily[k:].mean()) * np.sqrt(k * (DAYS - k) / DAYS), k) for k in range(3, DAYS - 3)]
print("1) estimated change day (largest scaled mean gap):", max(gaps)[1])
print("   daily thumbs-up, days 12..19:", np.round(daily[12:20], 3))

for name, m in [("web", mob == 0), ("mobile", mob == 1)]:
    a = rate(m & (day < 16)); b = rate(m & (day >= 16))
    print(f"2) {name:<6} before day 16: {a[0]:.3f}+-{a[1]:.3f}   from day 16: {b[0]:.3f}+-{b[1]:.3f}")

# Same-day contrast during the canary (days 14-17), web only (no UI change there)
c = (day >= 14) & (day < 18) & (mob == 0)
a = rate(c & (v2 == 1)); b = rate(c & (v2 == 0))
diff = a[0] - b[0]; se = np.sqrt(a[1] ** 2 + b[1] ** 2)
print(f"3) canary days 14-17, web: new model {a[0]:.3f} (n={a[2]}) vs old {b[0]:.3f} (n={b[2]}); diff {diff:+.3f} +-{se:.3f}")

# Mobile on the canary days after the UI change (day 16-17): model effect holding UI fixed
c = (day >= 16) & (day < 18) & (mob == 1)
a = rate(c & (v2 == 1)); b = rate(c & (v2 == 0))
print(f"4) mobile days 16-17 (new UI both arms): new model {a[0]:.3f} vs old {b[0]:.3f}; diff {a[0]-b[0]:+.3f} +-{np.sqrt(a[1]**2+b[1]**2):.3f}")

e_pre = err[day < 16].mean(); e_post = err[day >= 16].mean()
print(f"5) tool error rate before/after day 16: {e_pre:.4f} / {e_post:.4f}")

Output:

1) estimated change day (largest scaled mean gap): 16
   daily thumbs-up, days 12..19: [0.702 0.705 0.698 0.704 0.663 0.67  0.671 0.674]
2) web    before day 16: 0.704+-0.005   from day 16: 0.696+-0.006
2) mobile before day 16: 0.702+-0.006   from day 16: 0.630+-0.008
3) canary days 14-17, web: new model 0.694 (n=1437) vs old 0.695 (n=5728); diff -0.001 +-0.027
4) mobile days 16-17 (new UI both arms): new model 0.627 vs old 0.630; diff -0.003 +-0.050
5) tool error rate before/after day 16: 0.0154 / 0.0151

Reading it:

• Check 1: the change point is day 16, two days after the model update began its canary on day 14, and the daily rate steps from about 0.70 to about 0.67. Proximity to the model update was a coincidence of calendar.
• Check 2: web is flat (0.704 before, 0.696 after, a gap of 0.008 that is about the size of the interval), while mobile drops from 0.702 to 0.630. A change to the model or to a prompt shared by all clients would be expected to move both.
• Check 3: on the canary days, web users on the new model rate 0.694 against 0.695 on the old model, a difference of -0.001 with half-width 0.027. The two groups' half-widths combine as $\sqrt{0.024^2 + 0.012^2} \approx 0.027$ (the smaller new-model group, n=1437, has the wider interval). The model shows no measurable harm in this run, though a drop smaller than about 0.027 could hide inside that interval.
• Check 4: on mobile days 16 and 17, where both arms had the new interface, the new-versus-old model difference is -0.003 +-0.050, again no sign of a model effect (the wide interval says only that a small model effect cannot be excluded at this sample).
• Check 5: tool error rate is 0.0154 before and 0.0151 after: no backend failure signal.

What a model-caused drop would have looked like in the same five checks: the change point would sit on the days the model reached users, not two days later; web and mobile would fall together; check 3 and check 4 would show the new model below the old one by more than their half-widths; and tool errors would stay flat unless the new model produced malformed tool calls, in which case they would rise from the days it was served (a model signature in its own right, which is why the tool error rate is read next to the calendar). Every check here pointed the other way.

The conclusion that the mobile interface is the cause is supported by the segment cut and the same-day contrasts, then confirmed by the intervention: roll the interface back for a random half of mobile users and see the rate recover in that half.

Trade-offs and pitfalls

• Instant full rollouts remove the control. Insist on canaries for model and prompt changes; with none, an offline replay on a fixed prompt set is the only clean comparison, and it covers quality but not the live user experience.
• Multiple simultaneous changes need an interaction check: a model that is fine with the old prompt can be worse with the new one.
• Feedback is a biased sample. Compare thumbs rates only with their response rates, and prefer a sampled human grading of answers as a second signal.
• Data drift on the input side (new topics, a marketing campaign bringing a different user mix) also lowers satisfaction; segment by intent and tenure to separate it.
• A model update with truly slow effects (caching, retrieval index rebuilt) can show up days later, so keep the calendar view for at least two weeks.

Direct answer

Capture signals in four families: system (latency, errors, tokens, cost), input and output content (prompts, retrieved context, replies), quality (groundedness, meaning every claim in the reply is supported by the retrieved text; safety flags; format validity; drift in output statistics) and user feedback (thumbs, edits, retries, abandonment). Each one detects a different failure; no single signal covers them all. Store cheap numeric aggregates for everything and full content only for a sample plus every flagged request, because raw content is large, private and mostly uninteresting.

Signal by signal

Aggregation pitfalls, computed

Three terms first. A percentile is the value that a given share of requests beat: p50 is the median, p95 is the time 95 percent of requests beat, and p99 = 2163 ms would mean 99 requests in 100 finish faster than 2.2 seconds, so 1 user in 100 waits longer. Percentiles describe the slow tail that an average hides. A mergeable histogram counts how many requests fell into each of a fixed set of latency ranges (buckets); counts from different servers or minutes can simply be added bucket by bucket, whereas two p99 numbers cannot be combined into one. PSI, the population stability index, compares two distributions bucket by bucket: for each bucket take (new share minus old share) times the natural log of (new share divided by old share), and add them up. On two buckets by hand, shares 0.7 and 0.3 that become 0.5 and 0.5 give (0.5 - 0.7) x ln(0.5/0.7) + (0.5 - 0.3) x ln(0.5/0.3) = 0.0673 + 0.1022 = 0.1695 (the program prints it). A widely used rule of thumb, which comes from credit-risk scorecard monitoring and is not a standard, reads PSI below 0.1 as little change, 0.1 to 0.25 as moderate and above 0.25 as a large shift; the cut-offs shift with sample size and with how the buckets are drawn, so treat them as guidelines.

The program uses a seeded synthetic stream (invented numbers, not data from a real system): ten one-minute windows of 1000 request latencies, where minute 7 has a burst of slow retries. It shows why you store mergeable histograms rather than per-window percentiles, and how an output-length shift appears as a population stability index (PSI, defined above).

import os
os.environ["OMP_NUM_THREADS"] = "1"; os.environ["OPENBLAS_NUM_THREADS"] = "1"
import numpy as np
rng = np.random.default_rng(0)
# 10 one-minute windows x 1000 requests. Latency in ms (lognormal body); minute 7 has 8% slow retries.
mins = []
for m in range(10):
    x = rng.lognormal(mean=np.log(800), sigma=0.35, size=1000)
    if m == 7:
        slow = rng.random(1000) < 0.08
        x[slow] *= 6
    mins.append(x)
allx = np.concatenate(mins)
print("pooled mean %.0f  p50 %.0f  p95 %.0f  p99 %.0f" % (allx.mean(), *np.percentile(allx, [50, 95, 99])))
per_min_p99 = np.array([np.percentile(x, 99) for x in mins])
print("per-minute p99:", np.round(per_min_p99).astype(int).tolist())
print("mean of per-minute p99: %.0f  (pooled p99 above)" % per_min_p99.mean())

# Fixed log-spaced histogram (the mergeable summary a metrics store keeps)
edges = np.geomspace(100, 20000, 41)
def hist(x): return np.histogram(x, bins=edges)[0]
merged = sum(hist(x) for x in mins)
def hist_pct(h, q):
    c = np.cumsum(h); i = int(np.searchsorted(c, q * c[-1]))
    return edges[i + 1]   # upper edge of the bucket holding the quantile
print("histogram p99 from merged buckets: %.0f" % hist_pct(merged, 0.99))
print("buckets stored per window:", len(merged))

# Output-length drift: share of replies per length bucket, week 1 vs week 2, PSI.
w1 = rng.normal(180, 40, 5000).clip(5); w2 = rng.normal(150, 55, 5000).clip(5)
bins = [0, 100, 150, 200, 250, 1000]
p = np.histogram(w1, bins)[0] / 5000; q = np.histogram(w2, bins)[0] / 5000
psi = float(np.sum((q - p) * np.log(q / p)))
print("length bucket shares week1:", np.round(p, 3).tolist())
print("length bucket shares week2:", np.round(q, 3).tolist())
print("PSI = %.3f" % psi)

# Storage arithmetic with stated assumptions
req_per_day = 1_000_000; raw_kb = 6; kept_fraction = 0.02
print("raw traces for every request: %.1f GB/day" % (req_per_day * raw_kb / 1e6))
print("traces for 2%% sample plus all flagged (assume 1%%): %.2f GB/day" % (req_per_day * (kept_fraction + 0.01) * raw_kb / 1e6))

# PSI by hand on two buckets: shares [0.7, 0.3] become [0.5, 0.5]
p2, q2 = np.array([0.7, 0.3]), np.array([0.5, 0.5])
terms = (q2 - p2) * np.log(q2 / p2)
print("two-bucket PSI terms", np.round(terms, 4).tolist(), "total %.4f" % terms.sum())

Output:

pooled mean 885  p50 798  p95 1467  p99 2163
per-minute p99: [1745, 1762, 1715, 1916, 1740, 1863, 1772, 7395, 1850, 1829]
mean of per-minute p99: 2359  (pooled p99 above)
histogram p99 from merged buckets: 2402
buckets stored per window: 40
length bucket shares week1: [0.021, 0.2, 0.462, 0.278, 0.038]
length bucket shares week2: [0.178, 0.324, 0.318, 0.148, 0.032]
PSI = 0.534
raw traces for every request: 6.0 GB/day
traces for 2% sample plus all flagged (assume 1%): 0.18 GB/day
two-bucket PSI terms [0.0673, 0.1022] total 0.1695

How the code works, step by step:

• rng.lognormal(...) draws synthetic latencies. Real latencies are skewed: most requests cluster near a typical value and a few are very slow. A lognormal shape (the logarithm of the value is bell-shaped) imitates that. Minute 7 multiplies 8 percent of its requests by 6 to mimic a burst of slow retries.
• np.percentile computes the exact percentile from raw values; that needs every raw latency kept, so it cannot be done after the fact from aggregates.
• np.geomspace(100, 20000, 41) makes 41 bucket edges (so 40 buckets) from 100 ms to 20,000 ms, each edge about 14 percent above the last, so buckets are narrow where latencies live and wide in the tail. np.histogram counts requests per bucket, and sum(hist(x) for x in mins) adds the ten windows' counts together: that addition is the merge.
• hist_pct finds the percentile from counts alone: np.cumsum gives running totals, np.searchsorted returns the first bucket whose running total reaches 99 percent of all requests, and the function reports that bucket's upper edge. That is why it can only be accurate to within one bucket width.
• The last block computes PSI on five output-length buckets, and then on the two-bucket example above.

Reading it:

• The mean (885 ms) is far below the p99 (2163 ms): the tail is where users feel pain, so alert on percentiles, not means.
• Averaging per-minute p99 values gives 2359, but the pooled p99 is 2163. Percentiles do not average. Minute 7's p99 of 7395 ms dominates the average, yet it affected one minute of ten. Store bucket counts and merge them, then read the percentile from the merged histogram.
• The merged histogram reports 2402 for p99: an upper bucket edge, so it can overshoot the exact value by up to one bucket width (here 239 ms above 2163, in a bucket about 300 ms wide). Narrower buckets shrink this error at the cost of more series; 40 buckets per window is already small.
• Output length: week 2 shifted mass into the shortest bucket (0.021 to 0.178), PSI 0.534, far above the 0.25 rule-of-thumb line for a large shift. That is a drift alarm. It does not say whether replies got better or worse; a quality sample must answer that.
• Storage: with the assumed 1,000,000 requests a day and 6 KB per trace (an assumption), keeping every trace is 6.0 GB per day, while a 2% random sample plus an assumed 1% of flagged requests is 0.18 GB per day. Aggregates are kept for all requests either way.

Retention and privacy

Aggregates can live for months at low cost. Raw prompts and replies need redaction of personal data, access control, a short retention window, and a documented rule for what gets sampled. Keep all flagged traces and a random baseline sample; the baseline is what lets you measure the flag rate honestly.

Pitfalls

• Alerting on a single metric. Latency can improve exactly because a prompt change made replies shorter and worse.
• Dashboards without slices (topic, user segment, language): a regression confined to one slice vanishes in the average.
• Judge-based quality metrics with no calibration. Treat their trend as a signal and audit them against human labels.