LLM Evaluation and Observability Questions

Direct answer

Readiness is a set of gates, one per risk, each with a metric, a test set and a pass line fixed before anyone sees results. A gate passes on the upper confidence bound of a failure rate (how bad the true rate could plausibly be given the sample), not on the observed rate, because a few hundred reviewed conversations can look clean by luck. Runtime guardrails (a threshold below which the assistant hands the conversation to a person, an output filter, caps on tokens and spend, a kill switch) then enforce the same limits after launch. Every guardrail setting moves two things at once, which is the trade-off: tighter on hallucination or harm costs automation rate and money.

The gates

The limits below are example starting points to agree with the business, not universal values.

Why the bound, with numbers

Two kinds of bound appear. For failure gates (where the risk is that the true rate is higher than observed) the program uses the Clopper-Pearson upper bound: the largest true failure rate that would still produce so few failures at least 5% of the time. In code it is beta.ppf(0.95, k + 1, n - k), the 95th percentile of a Beta distribution (a curve over rates between 0 and 1) whose shape is set by the counts of failures and clean items; with zero failures in $n$ items it simplifies to $1-0.05^{1/n}$, close to $3/n$ (the "rule of three"). For example 6 failures in 500 is 1.2% observed, but the true rate could plausibly be as high as 2.355%. For the satisfaction gate the risk runs the other way (the true share could be lower than observed), so the program uses the Wilson lower bound, a normal-approximation interval that stays sensible for proportions, and checks that it clears the floor. Both are one-sided 95% limits in the direction that would hurt. The program below applies it to invented review counts (inputs, not measurements of any product), then sweeps two thresholds on synthetic scores. The groundedness score (how well a reply is supported by the sources, from 0 to 1) and the harm score are stand-ins: random numbers drawn from Beta distributions that mimic a scorer. Replies that are truly bad get scores centred near 0.43 and good ones near 0.75, with overlap, and 8% of replies are bad; the harm scorer is built the same way with 2% harmful. The percentages show the shape of the trade-off and not what your scorer will measure. Unit costs (1 unit per automated conversation, 25 units per human-handled one) are assumptions. In part 2, a threshold $t$ means a reply scoring below $t$ is handed to a person; in part 3, a threshold $t$ means a reply scoring at or above $t$ is blocked, so the two sweeps run in opposite directions.

import os
os.environ["OMP_NUM_THREADS"] = "1"
os.environ["OPENBLAS_NUM_THREADS"] = "1"
import numpy as np
from scipy.stats import beta, norm

def upper(k, n, conf=0.95):
    """One-sided Clopper-Pearson upper bound on a failure rate."""
    return 1.0 if k == n else float(beta.ppf(conf, k + 1, n - k))

def wilson_lower(k, n, conf=0.95):
    z = norm.ppf(conf)
    p = k / n
    centre = p + z * z / (2 * n)
    half = z * np.sqrt(p * (1 - p) / n + z * z / (4 * n * n))
    return float((centre - half) / (1 + z * z / n))

# Part 1: gates on invented review counts (inputs, not measurements of a real product)
print("gate                     k/n       observed  95%bound  limit   verdict")
for name, k, n, limit in [("unsupported claims", 6, 500, 0.02),
                          ("unsafe replies", 0, 600, 0.005),
                          ("leaked personal data", 1, 800, 0.002)]:
    ub = upper(k, n)
    print(f"{name:<24} {k}/{n:<6} {k/n:8.3%} {ub:8.3%} {limit:6.1%}   {'PASS' if ub <= limit else 'FAIL'}")
k, n = 1170, 1500
lb = wilson_lower(k, n)
print(f"{'satisfied responders':<24} {k}/{n:<5} {k/n:8.3%} lower {lb:.3%} floor 75.0%   {'PASS' if lb >= 0.75 else 'FAIL'}")

print("\nclean items needed for a 95% bound at or under the limit (zero failures observed)")
for limit in [0.02, 0.005, 0.002]:
    n = 1
    while upper(0, n) > limit:
        n += 1
    print(f"limit {limit:.1%}: {n} items")

# Part 2: threshold sweep on a STAND-IN groundedness score (synthetic Beta draws, not a real scorer)
rng = np.random.default_rng(0)
N = 20000
bad = rng.random(N) < 0.08
score = np.where(bad, rng.beta(3, 4, N), rng.beta(6, 2, N))
LLM_COST, HUMAN_COST = 1.0, 25.0   # assumed cost units per conversation
print("\nthreshold  automated  halluc_among_answered  cost_units_per_1000")
for t in [0.0, 0.4, 0.5, 0.6, 0.7]:
    ans = score >= t
    h = bad[ans].mean()
    cost = 1000 * (LLM_COST + HUMAN_COST * (1 - ans.mean()))
    print(f"{t:9.1f}  {ans.mean():9.1%}  {h:21.2%}  {cost:19.0f}")

# Part 3: harmful-content filter threshold on a second stand-in score
harm = rng.random(N) < 0.02
hs = np.where(harm, rng.beta(6, 2, N), rng.beta(2, 7, N))
print("\nblock_at  harmful_passed  benign_blocked")
for t in [0.5, 0.6, 0.7, 0.8]:
    blocked = hs >= t
    print(f"{t:8.1f}  {(~blocked[harm]).mean():14.1%}  {blocked[~harm].mean():14.1%}")

# Part 4: latency against a budget (synthetic lognormal, seconds to full reply)
lat = rng.lognormal(mean=np.log(2.0), sigma=0.45, size=N)
print(f"\nlatency p50 {np.percentile(lat,50):.2f}s  p95 {np.percentile(lat,95):.2f}s  share over 5s {(lat>5).mean():.2%}")

Output:

gate                     k/n       observed  95%bound  limit   verdict
unsupported claims       6/500      1.200%   2.355%   2.0%   FAIL
unsafe replies           0/600      0.000%   0.498%   0.5%   PASS
leaked personal data     1/800      0.125%   0.592%   0.2%   FAIL
satisfied responders     1170/1500   78.000% lower 76.191% floor 75.0%   PASS

clean items needed for a 95% bound at or under the limit (zero failures observed)
limit 2.0%: 149 items
limit 0.5%: 598 items
limit 0.2%: 1497 items

threshold  automated  halluc_among_answered  cost_units_per_1000
      0.0     100.0%                  8.05%                 1000
      0.4      94.7%                  4.59%                 2317
      0.5      89.2%                  3.10%                 3690
      0.6      79.1%                  1.83%                 6226
      0.7      62.8%                  0.93%                10292

block_at  harmful_passed  benign_blocked
     0.5            4.5%            3.7%
     0.6           14.7%            0.8%
     0.7           30.1%            0.1%
     0.8           56.3%            0.0%

latency p50 2.00s  p95 4.22s  share over 5s 2.00%

Reading the results

1. Observed 1.2% unsupported claims fails a 2% gate. 6 of 500 gives a bound of 2.355%. Zero unsafe replies in 600 gives 0.498%, which only just clears 0.5%: 598 clean items is the minimum for that limit, and a 0.2% limit needs 1,497 clean items. The personal-data gate shows the cost of a strict limit: 1 failure in 800 already fails it (0.592% bound), so the response is to fix and retest, not to relax the limit. Satisfaction passes because the lower bound (76.191%) clears the 75% floor, but survey responders are self-selected, so treat it as a soft gate.
2. Hallucination versus automation versus cost. Raising the hand-off threshold from 0.5 to 0.6 (more replies scoring below it go to a person) cuts hallucination among answered conversations from 3.10% to 1.83% but automation falls from 89.2% to 79.1% and cost per 1,000 conversations rises from 3,690 to 6,226 units. At 0.7 hallucination reaches 0.93% but only 62.8% is automated and cost is 10,292 units. For a 2% limit the sweep points to a threshold near 0.6 (the first setting in the sweep where hallucination among answered conversations, 1.83%, is under 2%; that is one draw of 20,000 synthetic replies, and another draw can land a little above or below the 2% line, so 0.6 sits close to the limit rather than safely under it); the numbers belong to the stand-in, so you would repeat the sweep with your real scorer on labelled data. For the cost gate: a human-only baseline costs 25 units per conversation, 25,000 per 1,000, so the 6,226 at threshold 0.6 is about a quarter of it. The cost gate passes with a large margin here, so quality, not cost, sets the threshold.
3. Safety filters trade harm against over-blocking, and the costs are not symmetric. Blocking at 0.5 lets 4.5% of harmful items through and blocks 3.7% of benign ones; at 0.7, 30.1% of harmful pass and 0.1% of benign are blocked. Here a lower block threshold is the stricter one. In support, a harmful reply reaching a customer is worse than a wrongly blocked helpful reply, which just routes to an agent, so the asymmetry favours the strict end (a block threshold of 0.5 or lower in this sweep), with human review of replies near the line and the over-block rate watched as its own metric.
4. Latency is a tail metric. The median is 2.00 s but p95 is 4.22 s and 2.00% of replies exceed 5 s. A mean would hide this. Also measure time to first token if the reply streams, since users judge responsiveness by when text starts.

Operational guardrails to ship with it

• Hand-off path: low groundedness, a refusal, an angry customer or a request for money moves the chat to a person with the transcript.
• Output filter and personal-data scan before the reply is shown.
• Caps: tokens per conversation, tool calls per conversation, spend per hour.
• Kill switch that routes to the human-only flow without a deploy.
• Staged rollout to a small share of traffic with the gates computed on live traffic before widening.
• Monitoring: the same metrics on sampled live conversations, with alerts when a bound crosses its limit.

Recommendation and what would flip it

Launch on a limited share of traffic only if the two hard gates (hallucination, safety) pass on their bounds and the hand-off threshold sits near 0.6 (on the stand-in). Flip toward a stricter threshold when errors carry money or legal consequences (refund promises, medical or financial content). Flip toward a looser one when the human queue is the bottleneck and wrong answers are cheap and visible.

Common mistakes

• Passing a gate on the observed rate from a small sample.
• Tuning thresholds on the same data used for the go decision.
• Averaging latency instead of reading p95.
• One combined "quality score" that lets a good tone hide a safety failure.
• Counting deflection (the customer left) as resolution.

Direct answer

Observability means you can answer "why did the system do that?" from the data it emits, without shipping new code. For a conventional service, the questions are about health: did the request succeed, how long did it take, what resource was used. For an LLM feature there is a second question that status codes cannot answer: was the output any good? The model can return HTTP 200 with a fluent, confident, wrong answer, so observability for an LLM feature adds the content of each step (prompt, retrieved context, output) and quality signals computed on it, linked to the usual latency and error data.

How it differs from a conventional service

The signals to capture

A trace is the full record of one request; each step inside it is a span (one timed operation with attributes). Capture these per request:

1. Latency per span and end to end, including time to first token (how long until the first word of the reply appears) for streamed replies (replies shown word by word as the model writes them).
2. Errors and retries, including provider rate limits and timeouts.
3. Token counts in and out, and derived cost.
4. Inputs and outputs: the final prompt after template filling, retrieved passages with their IDs and scores, tool calls and results, the reply.
5. Versions: model identifier, prompt template version, index version, sampling settings.
6. Quality signals: groundedness (does the reply stay within the retrieved context), safety flags, format validity, plus user feedback (thumbs, edits, retries, abandonment).
7. User and session identifiers, to find all requests affected by a problem.

A worked example

The program below uses one hand-written trace (timings and token counts are invented inputs) for a retrieval-augmented feature (one that first looks up relevant documents and puts them in the prompt), and six hand-written replies. The groundedness check is a deterministic stand-in: a real system would put a model-based judge or an entailment model (a model that decides whether a passage supports a claim) there. The stand-in only tests that every number in the reply appears in the retrieved context.

How to read the code. Part 1: each span is a tuple (name, start_ms, end_ms, tokens_in, tokens_out), so ("retrieve", 12, 95, 0, 0) means retrieval ran from millisecond 12 to 95 and used no model tokens. The guardrail spans are the automatic input and output checks, and rerank re-orders the retrieved passages with a more careful score. The program prints each span's duration, its share of the 940 ms total and its tokens, then a cost from the token counts. Part 2: each request is a (question, reply, retrieved context) triple, and stub_check pulls every number out of the reply and the context and compares the two sets.

import re
# One request through a retrieval-augmented chat feature, hand-written spans.
# Each tuple is (name, start_ms, end_ms, tokens_in, tokens_out).
spans = [
    ("guardrail_in",   0,  12, 0,   0),
    ("retrieve",      12,  95, 0,   0),
    ("rerank",        95, 160, 0,   0),
    ("llm_call",     160, 910, 1450, 180),
    ("guardrail_out",910, 940, 0,   0),
]
total_ms = spans[-1][2] - spans[0][1]
tin = sum(s[3] for s in spans); tout = sum(s[4] for s in spans)
print("end-to-end ms:", total_ms)
for name, a, b, ti, to in spans:
    print(f"{name:14s} {b-a:4d} ms  {100*(b-a)/total_ms:5.1f}%  tokens in/out {ti}/{to}")
UNIT_IN, UNIT_OUT = 1.0, 3.0   # assumed cost units per 1000 tokens, for illustration only
print("cost units:", round(tin/1000*UNIT_IN + tout/1000*UNIT_OUT, 3))

# Six requests: every one returns HTTP 200. A quality check on the reply tells them apart.
reqs = [
 ("What was Q2 revenue?", "Revenue in Q2 was 4.2 million.", "Q2 revenue: 4.2 million. Q1 revenue: 3.9 million."),
 ("What was Q1 revenue?", "Revenue in Q1 was 3.9 million.", "Q2 revenue: 4.2 million. Q1 revenue: 3.9 million."),
 ("Q2 growth?",           "Growth was 12 percent.",         "Q2 revenue: 4.2 million. Q1 revenue: 3.9 million."),
 ("Q2 revenue?",          "I could not find that.",         "No relevant documents."),
 ("Q2 revenue?",          "Revenue in Q2 was 4.2 million.", "Q1 headcount: 40."),
 ("Q1 revenue?",          "Revenue in Q1 was 3.9 million.", "Q2 revenue: 4.2 million. Q1 revenue: 3.9 million."),
]
def numbers(t): return set(re.findall(r"\d+(?:\.\d+)?", t))
def stub_check(answer, context):
    """Stand-in for a real groundedness judge: every number in the reply must appear in the context."""
    nums = numbers(answer)
    if not nums: return "no_claim"
    return "grounded" if nums <= numbers(context) else "ungrounded"
print()
for q, a, c in reqs:
    print(f"http=200  {stub_check(a, c):10s} {q}")

Output:

end-to-end ms: 940
guardrail_in     12 ms    1.3%  tokens in/out 0/0
retrieve         83 ms    8.8%  tokens in/out 0/0
rerank           65 ms    6.9%  tokens in/out 0/0
llm_call        750 ms   79.8%  tokens in/out 1450/180
guardrail_out    30 ms    3.2%  tokens in/out 0/0
cost units: 1.99

http=200  grounded   What was Q2 revenue?
http=200  grounded   What was Q1 revenue?
http=200  ungrounded Q2 growth?
http=200  no_claim   Q2 revenue?
http=200  ungrounded Q2 revenue?
http=200  grounded   Q1 revenue?

What this shows:

• The trace pins the time: the model call is 750 of 940 ms, so tuning retrieval would barely move the total. A conventional service dashboard would show this too.
• The cost line is in invented relative units (output tokens assumed three times the price of input tokens). The point is that cost comes from the token columns, which a conventional dashboard has no column for.
• All six requests return status 200. The quality check separates them: the "Q2 growth" reply quotes 12 percent that appears nowhere in the context (ungrounded), the fifth reply states Q2 revenue from a context holding only headcount (also ungrounded), and the fourth declines honestly. The stand-in is crude: it counts the digit in a label such as "Q2" as a number, so reply 5's numbers are 2 and 4.2 while its context holds 1 and 40, and reply 1 passes only because the context happens to contain "Q2" and "Q1" too.
• The program does not join anything; the join is done by the person reading the logged trace. Because the trace also stores the retrieved passages, an ungrounded reply can be sorted into two causes: the context held the facts but the reply invented others (a model fault, as with the growth figure), or the context never held the facts (a retrieval fault, as in reply 5, where the passage was about headcount). Without the retrieved text in the trace you could not tell these apart.

Limits and pitfalls

• Quality signals are themselves measurements with error; a stand-in or judge needs calibration against human labels.
• Logging full prompts is a privacy risk; redact, restrict access, and set retention.
• Without version tags on every trace, you cannot tell a regression caused by a prompt edit from one caused by a data change.
• Dashboards of averages hide tails and slices; keep percentiles and group by user segment and topic.

Direct answer

Perplexity measures how surprised a language model is by a piece of text. It is the exponential of the average negative log-probability the model assigned to each token that actually occurred:

Here $N$ is the number of predicted tokens (a token is the unit the model reads: a word or word-piece), $x_i$ is the $i$-th token and $x_{<i}$ is everything before it. The average inside the exponent is the model's cross-entropy on that text: the average surprise per token, where the surprise of a token is its negative log-probability (the natural logarithm, written $\ln$, of a probability below 1 is negative, so its negative is a positive number that grows as the probability shrinks). That surprise is measured in nats when the logarithm is natural, and in bits when it is base 2. So perplexity is the exponential of the cross-entropy. A perplexity of $k$ means the model is, on average, as uncertain as if it were choosing uniformly among $k$ equally likely tokens. Lower is better at predicting that text; higher means the model found the text surprising. On its own it says how well a model fits text, not whether the text it generates is correct, helpful or safe.

Perplexity by hand

Suppose a model gave probability 0.25 to each of four tokens that actually occurred. The surprise of each token is $-\ln 0.25 = 1.386$, the average is 1.386, and $\exp(1.386) = 4$: the model behaved as if it were choosing among 4 equally likely tokens at each step. If it had given 0.90 to each token, the surprise per token is 0.105 and the perplexity is 1.11, almost no uncertainty. The two cases, computed:

import math
# Perplexity by hand: ln is the natural logarithm, exp undoes it.
for name, probs in [("uniform 0.25 on every token", [0.25, 0.25, 0.25, 0.25]),
                    ("0.90 on every token", [0.90, 0.90, 0.90, 0.90])]:
    surprise = [-math.log(p) for p in probs]       # negative log-probability of each token
    cross_entropy = sum(surprise) / len(surprise)  # average surprise per token, in nats
    print(f"{name}: surprise per token {surprise[0]:.3f}, average {cross_entropy:.3f}, perplexity {math.exp(cross_entropy):.2f}")

Output:

uniform 0.25 on every token: surprise per token 1.386, average 1.386, perplexity 4.00
0.90 on every token: surprise per token 0.105, average 0.105, perplexity 1.11

What higher and lower values tell you

• Low perplexity on a test set: the model assigns high probability to what really came next. This usually goes with fluent, in-distribution text (text that resembles what the model was trained on).
• High perplexity: the text is unlike the training data (another domain, a typo-laden source, a new language) or the model is weak.
• It is only comparable between models when they are evaluated on the same text with the same tokenization, meaning the same rule for splitting text into tokens (a tokenizer is the program that applies that rule).

A worked example on a tiny model

The program trains word-level unigram and bigram models (a unigram model ignores context; a bigram model conditions on the previous word) with add-0.1 smoothing on a 42-token invented corpus about capitals and rivers (the full stops count as tokens). Smoothing gives unseen words a small non-zero probability. These tiny models are stand-ins for a neural language model; the computation of perplexity is the same. It scores four test sentences, then re-expresses one sentence's loss per character.

import math
from collections import Counter
train = ("the capital of france is paris . the capital of germany is berlin . "
         "the capital of italy is rome . the capital of spain is madrid . "
         "the river seine is in paris . the river spree is in berlin .").split()
V = sorted(set(train)); K = 0.1   # add-K smoothing
uni = Counter(train); bi = Counter(zip(train, train[1:]))
def p_uni(w): return (uni[w] + K) / (len(train) + K * len(V))
def p_bi(w, prev): return (bi[(prev, w)] + K) / (uni[prev] + K * len(V))
def ppl(text, model):
    toks = text.split(); nll = 0.0
    for i in range(1, len(toks)):
        p = p_uni(toks[i]) if model == "uni" else p_bi(toks[i], toks[i-1])
        nll -= math.log(p)
    return math.exp(nll / (len(toks) - 1)), nll
tests = {
 "true      ": "the capital of germany is berlin .",
 "false     ": "the capital of germany is paris .",
 "scrambled ": "is of the germany capital berlin .",
 "repetitive": "the the the the the the .",
}
for name, t in tests.items():
    print(name, "unigram ppl %.2f   bigram ppl %.2f" % (ppl(t, "uni")[0], ppl(t, "bi")[0]))

# Same text, same model, two tokenisations: perplexity not comparable, total NLL is the invariant.
text = "the capital of germany is berlin ."
_, nll_words = ppl(text, "bi"); n_words = len(text.split()) - 1
n_chars = len(text) - len(text.split()[0])
print("words:", n_words, "perplexity per word %.2f" % math.exp(nll_words / n_words))
print("same total NLL spread over %d characters: perplexity per char %.2f" % (n_chars, math.exp(nll_words / n_chars)))
print("bits per character %.3f" % (nll_words / math.log(2) / n_chars))

Output:

true       unigram ppl 13.00   bigram ppl 2.73
false      unigram ppl 13.00   bigram ppl 2.73
scrambled  unigram ppl 13.00   bigram ppl 31.17
repetitive unigram ppl 7.16   bigram ppl 77.00
words: 6 perplexity per word 2.73
same total NLL spread over 31 characters: perplexity per char 1.21
bits per character 0.280

Reading it:

• A false sentence scores exactly like a true one. "the capital of germany is paris" and "the capital of germany is berlin" both get bigram perplexity 2.73, because in the training text "is" is followed by "paris" once and by "berlin" once. The unigram column ties because it ignores context entirely and "paris" and "berlin" each occur twice in the training text. The scrambled sentence also gets 13.00, but only because the word it scores in place of "is" is "the" (the first word of each sentence is context only and is not scored), and both of those words occur 6 times in the training text. The model measures word-pattern fit, not truth. A real model sees far more context than this toy and will often prefer the true sentence, but nothing in the perplexity formula checks facts: it rewards text that matches patterns in the training data, so a fluent falsehood that fits those patterns can still earn a low perplexity.
• Word order matters to a context model, not to the unigram. Apart from which word comes first, the scrambled sentence is the same bag of words, so its unigram perplexity is the same 13.00, but its bigram perplexity is 31.17.
• Low perplexity is not good generation. The repetitive string scores a low 7.16 under the unigram model (the word "the" is common), while the bigram model rates it 77.00 because "the the" never occurred. Which model you ask changes the verdict, and a looping real model can also drive its own perplexity down by repeating.
• Tokenization changes the number. The same total loss gives 2.73 per word but 1.21 per character, because dividing by more tokens gives a smaller per-token value. Perplexities from models with different vocabularies are not comparable. Bits per character puts models on one basis: it takes the total loss (6.02 nats for this sentence), converts nats to bits by dividing by $\ln 2 \approx 0.693$ (8.68 bits), and divides by the 31 characters, giving 0.280. The count of characters does not depend on the tokenizer, so the number can be compared across models.

How far perplexity can carry evaluation

For open-ended generation add task metrics, reference-based overlap scores where a reference exists, human ratings against a rubric, and model-graded evaluation checked against human labels.

Pitfalls

• Reporting perplexity on text the model trained on, which flatters it.
• Comparing across tokenizers or languages.
• Ignoring the first-token convention: the first token has no context, so implementations differ on whether it is scored. State yours (here the first token is context only and is not scored).

Direct answer

Capture signals in four families: system (latency, errors, tokens, cost), input and output content (prompts, retrieved context, replies), quality (groundedness, meaning every claim in the reply is supported by the retrieved text; safety flags; format validity; drift in output statistics) and user feedback (thumbs, edits, retries, abandonment). Each one detects a different failure; no single signal covers them all. Store cheap numeric aggregates for everything and full content only for a sample plus every flagged request, because raw content is large, private and mostly uninteresting.

Signal by signal

Aggregation pitfalls, computed

Three terms first. A percentile is the value that a given share of requests beat: p50 is the median, p95 is the time 95 percent of requests beat, and p99 = 2163 ms would mean 99 requests in 100 finish faster than 2.2 seconds, so 1 user in 100 waits longer. Percentiles describe the slow tail that an average hides. A mergeable histogram counts how many requests fell into each of a fixed set of latency ranges (buckets); counts from different servers or minutes can simply be added bucket by bucket, whereas two p99 numbers cannot be combined into one. PSI, the population stability index, compares two distributions bucket by bucket: for each bucket take (new share minus old share) times the natural log of (new share divided by old share), and add them up. On two buckets by hand, shares 0.7 and 0.3 that become 0.5 and 0.5 give (0.5 - 0.7) x ln(0.5/0.7) + (0.5 - 0.3) x ln(0.5/0.3) = 0.0673 + 0.1022 = 0.1695 (the program prints it). A widely used rule of thumb, which comes from credit-risk scorecard monitoring and is not a standard, reads PSI below 0.1 as little change, 0.1 to 0.25 as moderate and above 0.25 as a large shift; the cut-offs shift with sample size and with how the buckets are drawn, so treat them as guidelines.

The program uses a seeded synthetic stream (invented numbers, not data from a real system): ten one-minute windows of 1000 request latencies, where minute 7 has a burst of slow retries. It shows why you store mergeable histograms rather than per-window percentiles, and how an output-length shift appears as a population stability index (PSI, defined above).

import os
os.environ["OMP_NUM_THREADS"] = "1"; os.environ["OPENBLAS_NUM_THREADS"] = "1"
import numpy as np
rng = np.random.default_rng(0)
# 10 one-minute windows x 1000 requests. Latency in ms (lognormal body); minute 7 has 8% slow retries.
mins = []
for m in range(10):
    x = rng.lognormal(mean=np.log(800), sigma=0.35, size=1000)
    if m == 7:
        slow = rng.random(1000) < 0.08
        x[slow] *= 6
    mins.append(x)
allx = np.concatenate(mins)
print("pooled mean %.0f  p50 %.0f  p95 %.0f  p99 %.0f" % (allx.mean(), *np.percentile(allx, [50, 95, 99])))
per_min_p99 = np.array([np.percentile(x, 99) for x in mins])
print("per-minute p99:", np.round(per_min_p99).astype(int).tolist())
print("mean of per-minute p99: %.0f  (pooled p99 above)" % per_min_p99.mean())

# Fixed log-spaced histogram (the mergeable summary a metrics store keeps)
edges = np.geomspace(100, 20000, 41)
def hist(x): return np.histogram(x, bins=edges)[0]
merged = sum(hist(x) for x in mins)
def hist_pct(h, q):
    c = np.cumsum(h); i = int(np.searchsorted(c, q * c[-1]))
    return edges[i + 1]   # upper edge of the bucket holding the quantile
print("histogram p99 from merged buckets: %.0f" % hist_pct(merged, 0.99))
print("buckets stored per window:", len(merged))

# Output-length drift: share of replies per length bucket, week 1 vs week 2, PSI.
w1 = rng.normal(180, 40, 5000).clip(5); w2 = rng.normal(150, 55, 5000).clip(5)
bins = [0, 100, 150, 200, 250, 1000]
p = np.histogram(w1, bins)[0] / 5000; q = np.histogram(w2, bins)[0] / 5000
psi = float(np.sum((q - p) * np.log(q / p)))
print("length bucket shares week1:", np.round(p, 3).tolist())
print("length bucket shares week2:", np.round(q, 3).tolist())
print("PSI = %.3f" % psi)

# Storage arithmetic with stated assumptions
req_per_day = 1_000_000; raw_kb = 6; kept_fraction = 0.02
print("raw traces for every request: %.1f GB/day" % (req_per_day * raw_kb / 1e6))
print("traces for 2%% sample plus all flagged (assume 1%%): %.2f GB/day" % (req_per_day * (kept_fraction + 0.01) * raw_kb / 1e6))

# PSI by hand on two buckets: shares [0.7, 0.3] become [0.5, 0.5]
p2, q2 = np.array([0.7, 0.3]), np.array([0.5, 0.5])
terms = (q2 - p2) * np.log(q2 / p2)
print("two-bucket PSI terms", np.round(terms, 4).tolist(), "total %.4f" % terms.sum())

Output:

pooled mean 885  p50 798  p95 1467  p99 2163
per-minute p99: [1745, 1762, 1715, 1916, 1740, 1863, 1772, 7395, 1850, 1829]
mean of per-minute p99: 2359  (pooled p99 above)
histogram p99 from merged buckets: 2402
buckets stored per window: 40
length bucket shares week1: [0.021, 0.2, 0.462, 0.278, 0.038]
length bucket shares week2: [0.178, 0.324, 0.318, 0.148, 0.032]
PSI = 0.534
raw traces for every request: 6.0 GB/day
traces for 2% sample plus all flagged (assume 1%): 0.18 GB/day
two-bucket PSI terms [0.0673, 0.1022] total 0.1695

How the code works, step by step:

• rng.lognormal(...) draws synthetic latencies. Real latencies are skewed: most requests cluster near a typical value and a few are very slow. A lognormal shape (the logarithm of the value is bell-shaped) imitates that. Minute 7 multiplies 8 percent of its requests by 6 to mimic a burst of slow retries.
• np.percentile computes the exact percentile from raw values; that needs every raw latency kept, so it cannot be done after the fact from aggregates.
• np.geomspace(100, 20000, 41) makes 41 bucket edges (so 40 buckets) from 100 ms to 20,000 ms, each edge about 14 percent above the last, so buckets are narrow where latencies live and wide in the tail. np.histogram counts requests per bucket, and sum(hist(x) for x in mins) adds the ten windows' counts together: that addition is the merge.
• hist_pct finds the percentile from counts alone: np.cumsum gives running totals, np.searchsorted returns the first bucket whose running total reaches 99 percent of all requests, and the function reports that bucket's upper edge. That is why it can only be accurate to within one bucket width.
• The last block computes PSI on five output-length buckets, and then on the two-bucket example above.

Reading it:

• The mean (885 ms) is far below the p99 (2163 ms): the tail is where users feel pain, so alert on percentiles, not means.
• Averaging per-minute p99 values gives 2359, but the pooled p99 is 2163. Percentiles do not average. Minute 7's p99 of 7395 ms dominates the average, yet it affected one minute of ten. Store bucket counts and merge them, then read the percentile from the merged histogram.
• The merged histogram reports 2402 for p99: an upper bucket edge, so it can overshoot the exact value by up to one bucket width (here 239 ms above 2163, in a bucket about 300 ms wide). Narrower buckets shrink this error at the cost of more series; 40 buckets per window is already small.
• Output length: week 2 shifted mass into the shortest bucket (0.021 to 0.178), PSI 0.534, far above the 0.25 rule-of-thumb line for a large shift. That is a drift alarm. It does not say whether replies got better or worse; a quality sample must answer that.
• Storage: with the assumed 1,000,000 requests a day and 6 KB per trace (an assumption), keeping every trace is 6.0 GB per day, while a 2% random sample plus an assumed 1% of flagged requests is 0.18 GB per day. Aggregates are kept for all requests either way.

Retention and privacy

Aggregates can live for months at low cost. Raw prompts and replies need redaction of personal data, access control, a short retention window, and a documented rule for what gets sampled. Keep all flagged traces and a random baseline sample; the baseline is what lets you measure the flag rate honestly.

Pitfalls

• Alerting on a single metric. Latency can improve exactly because a prompt change made replies shorter and worse.
• Dashboards without slices (topic, user segment, language): a regression confined to one slice vanishes in the average.
• Judge-based quality metrics with no calibration. Treat their trend as a signal and audit them against human labels.

Direct answer

Use several cheap, independent signals, each scored automatically on every output (or a large sample), then combine them in two steps. Step one turns the signals into a single probability that the output violates policy, using a small model (logistic regression) fitted on audited examples. Step two multiplies that probability by the harm weight of the policy category and by an exposure term, giving a severity score that orders the review queue. Add an override so the most harmful categories reach a human quickly whatever the ranking says.

Signals that are lightweight

1. Content classifier score per policy category (a small model, run on the output and sometimes the input).
2. Pattern and keyword hits: blocked terms, personal-data patterns (phone, email, ID formats), links to disallowed domains, system-prompt text appearing in the reply.
3. Judge-model score: a rubric-prompted model asked "does this reply break rule X?", run on a sample because it costs more.
4. Behavioural signals: the model complied with a request that matches a known jailbreak template (a prompt pattern written to talk a model out of its rules); the refusal that is normally given did not occur; the reply is far longer or in a different language than usual for the prompt.
5. User signals: reports, thumbs-down, immediate re-asks.
6. Moderation-hold or rule outcomes from guardrails already in the serving path (outputs an existing guardrail held back or flagged).

Each has a different weakness (keywords miss paraphrases, classifiers drift, reports are sparse and slow), which is why combining them helps.

Step one: probability of violation

Fit a logistic regression: $P(\text{violation}) = \sigma(w_0 + w_1 x_1 + \dots + w_m x_m)$, where $x_j$ are the signal values, $w_j$ their learned weights and $\sigma$ the logistic function that maps a sum to a probability between 0 and 1. Fit on a labelled sample (human auditors judged the outputs), including a random slice of clean traffic so that the base rate (the overall share of outputs that really violate policy) is right. Check calibration afterwards: a calibrated model is one whose predicted probabilities match observed frequencies, so "0.7" means about 70% of such items are real violations; otherwise the severity arithmetic is meaningless. The last part of the program shows how to check it.

The program simulates 1500 sampled outputs per run with an 8% violation rate and three noisy signals (a stand-in for audited logs), fits the combiner on one draw, and measures AUC (area under the ROC curve, which plots the share of violations caught against the share of clean items wrongly flagged as the threshold is swept; equivalently, the probability that a random violating output gets a higher score than a random clean one; 0.5 is chance, 1.0 is perfect) on a fresh draw, averaged over 20 seeds.

Step two: severity for triage

Harm weights come from the written policy (self-harm and violent threats highest, spam lowest) and are agreed with the policy owner, not learned. One way to agree them: ask how many spam items the owner would read to avoid missing one self-harm item; if the answer is "about ten", the weights are 1 and 10, and the other categories are placed between by the same question. The values 10, 9, 6, 6 and 1 below are illustrative. The P(violation) values in the triage table are typed in by hand for illustration; in production they come from the fitted combiner above. The logarithm of exposure stops a single viral item from swamping everything while still ranking widely seen content higher. The weights and the log are design choices, shown here as an example.

import os
os.environ["OMP_NUM_THREADS"] = "1"
os.environ["OPENBLAS_NUM_THREADS"] = "1"
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score

# ---- Part 1: learn how to combine signals (synthetic labelled sample; stand-in for audited logs) ----
def make(seed, n=1500):
    r = np.random.default_rng(seed)
    y = (r.random(n) < 0.08).astype(int)                       # 8% of sampled outputs violate policy
    clf = np.clip(0.25 + 0.35 * y + 0.18 * r.standard_normal(n), 0, 1)   # content classifier score
    kw = np.clip(0.10 + 0.30 * y + 0.22 * r.standard_normal(n), 0, 1)    # keyword / pattern hit strength
    rep = (r.random(n) < (0.02 + 0.30 * y)).astype(float)                # a user reported the reply
    return np.column_stack([clf, kw, rep]), y

names = ["classifier", "keyword", "user_report"]
auc_single = np.zeros(3); auc_comb = 0.0; SEEDS = 20
for s in range(SEEDS):
    Xtr, ytr = make(s); Xte, yte = make(1000 + s)
    lr = LogisticRegression(max_iter=1000).fit(Xtr, ytr)
    auc_comb += roc_auc_score(yte, lr.predict_proba(Xte)[:, 1]) / SEEDS
    for j in range(3):
        auc_single[j] += roc_auc_score(yte, Xte[:, j]) / SEEDS
print("held-out AUC averaged over", SEEDS, "seeds")
for n_, a in zip(names, auc_single):
    print(f"  {n_:12s} {a:.3f}")
print(f"  {'combined':12s} {auc_comb:.3f}")

# ---- Part 2: severity = P(violation) x harm weight of the category x exposure ----
HARM = {"self_harm": 10, "violent_threat": 9, "personal_data": 6, "hate": 6, "spam": 1}
items = [  # (id, category, P(violation) from the combined score, users who saw it)
    ("a", "spam", 0.95, 40),
    ("b", "self_harm", 0.40, 3),
    ("c", "personal_data", 0.70, 12),
    ("d", "hate", 0.30, 200),
    ("e", "violent_threat", 0.85, 1),
]
rows = [(i, c, p, n, p * HARM[c] * np.log1p(n)) for i, c, p, n in items]
print("\nid category        P(viol) seen  severity  rank_by_P  rank_by_severity")
byP = {r[0]: k + 1 for k, r in enumerate(sorted(rows, key=lambda r: -r[2]))}
bySev = {r[0]: k + 1 for k, r in enumerate(sorted(rows, key=lambda r: -r[4]))}
for i, c, p, n, sev in rows:
    print(f"{i}  {c:14s} {p:5.2f} {n:5d} {sev:8.2f}  {byP[i]:8d}  {bySev[i]:8d}")

# Override rule: the most harmful categories skip the ranking when the probability is moderate or higher.
urgent = [i for i, c, p, n in items if HARM[c] >= 9 and p >= 0.5]
print("\nurgent queue (harm weight >= 9 and P >= 0.5):", urgent)

# ---- Part 3: read the fitted combiner and check it ----
Xtr, ytr = make(0)
lr = LogisticRegression(max_iter=1000).fit(Xtr, ytr)
print("\nfitted on seed 0: intercept", round(float(lr.intercept_[0]), 2),
      "| weights", dict(zip(names, [round(float(w), 2) for w in lr.coef_[0]])))
x = np.array([0.62, 0.45, 1.0])                       # one output: classifier 0.62, keyword 0.45, user reported
z = float(lr.intercept_[0] + lr.coef_[0] @ x)
print(f"one output: z = {z:.2f} -> sigma(z) = 1/(1+exp(-z)) = {1 / (1 + np.exp(-z)):.3f}")
x0 = np.array([0.25, 0.10, 0.0])                      # a typical clean output
z0 = float(lr.intercept_[0] + lr.coef_[0] @ x0)
print(f"typical clean output: z = {z0:.2f} -> probability {1 / (1 + np.exp(-z0)):.3f}")

# Calibration: compare predicted probability with the observed violation share per bin, on fresh draws.
# Repeat for 10 fits (seeds 1 to 10, each scored on its own fresh 30000 outputs) with the default
# regularisation (C=1) and a weaker penalty (C=100); report the average per bin and the min / max gap.
bins = [(0.0, 0.05), (0.05, 0.2), (0.2, 0.5), (0.5, 1.01)]
for C in (1.0, 100.0):
    pred = np.zeros((10, 4)); obs = np.zeros((10, 4)); cnt = np.zeros((10, 4))
    for s in range(1, 11):
        Xa, ya = make(s)
        model = LogisticRegression(C=C, max_iter=2000).fit(Xa, ya)
        Xc, yc = make(5000 + s, n=30000)
        pc = model.predict_proba(Xc)[:, 1]
        for b, (lo, hi) in enumerate(bins):
            m = (pc >= lo) & (pc < hi)
            cnt[s - 1, b] = m.sum()
            pred[s - 1, b] = pc[m].mean() if m.any() else np.nan
            obs[s - 1, b] = yc[m].mean() if m.any() else np.nan
    print(f"\ncalibration, C={C:g}, mean over 10 fits")
    print("predicted bin   items  mean_predicted  observed_share  observed-minus-predicted min / max")
    for b, (lo, hi) in enumerate(bins):
        gap = obs[:, b] - pred[:, b]
        print(f"{lo:.2f} to {min(hi, 1.0):.2f}  {cnt[:, b].mean():6.0f}  {np.nanmean(pred[:, b]):14.3f}  {np.nanmean(obs[:, b]):14.3f}  {np.nanmin(gap):+.3f} / {np.nanmax(gap):+.3f}")

Output:

held-out AUC averaged over 20 seeds
  classifier   0.919
  keyword      0.831
  user_report  0.661
  combined     0.967

id category        P(viol) seen  severity  rank_by_P  rank_by_severity
a  spam            0.95    40     3.53         1         5
b  self_harm       0.40     3     5.55         4         3
c  personal_data   0.70    12    10.77         3         1
d  hate            0.30   200     9.55         5         2
e  violent_threat  0.85     1     5.30         2         4

urgent queue (harm weight >= 9 and P >= 0.5): ['e']

fitted on seed 0: intercept -6.24 | weights {'classifier': 6.31, 'keyword': 4.23, 'user_report': 2.63}
one output: z = 2.21 -> sigma(z) = 1/(1+exp(-z)) = 0.901
typical clean output: z = -4.24 -> probability 0.014

calibration, C=1, mean over 10 fits
predicted bin   items  mean_predicted  observed_share  observed-minus-predicted min / max
0.00 to 0.05   21789           0.016           0.005  -0.013 / -0.009
0.05 to 0.20    5430           0.096           0.087  -0.026 / +0.012
0.20 to 0.50    1525           0.316           0.450  +0.071 / +0.210
0.50 to 1.00    1257           0.757           0.912  +0.121 / +0.182

calibration, C=100, mean over 10 fits
predicted bin   items  mean_predicted  observed_share  observed-minus-predicted min / max
0.00 to 0.05   24038           0.008           0.008  -0.003 / +0.003
0.05 to 0.20    2987           0.101           0.111  -0.012 / +0.043
0.20 to 0.50    1238           0.322           0.345  -0.030 / +0.092
0.50 to 1.00    1737           0.820           0.838  -0.025 / +0.042

• Reading the fitted combiner: the weights (6.31, 4.23, 2.63 on classifier, keyword and report, intercept -6.24) say each signal pushes the sum $z$ up, with the classifier counting most. For one output with classifier 0.62, keyword 0.45 and a user report, $z = -6.24 + 6.31 \times 0.62 + 4.23 \times 0.45 + 2.63 \times 1 = 2.21$, and $\sigma(z) = 1/(1+e^{-z}) = 0.901$. A typical clean output (0.25, 0.10, no report) gives $z = -4.24$ and probability 0.014.
• Calibration, from the last two tables (10 fits, each scored on fresh simulated outputs): with scikit-learn's default penalty the model is under-confident at the top. Outputs it scores in the 0.50 to 1.00 bin were real violations 0.912 of the time on average against a mean prediction of 0.757, and the gap was positive in all 10 fits (+0.121 to +0.182). With a weaker penalty (C=100) the gaps shrink to -0.025 to +0.042 in that bin. This is a property of this simulation, not a universal rule, but it is why the check is not optional: the default fit ranks well (AUC above) and still reports probabilities that are too low, which would understate severity. If a check like this fails, refit with a weaker penalty or recalibrate the scores on held-out audited data.
• Averaged over 20 simulated repeats, the classifier alone reaches AUC 0.919, keywords 0.831, user reports 0.661, and the fitted combination 0.967. The combination beats every single signal here because the signals' errors are independent in the simulation; with correlated signals the gain is smaller, so measure it on real audited data.
• In the triage table, spam has the highest violation probability (0.95) but ranks last on severity (5) because its harm weight is 1. The personal-data item (probability 0.70, seen by 12 users) ranks first with severity 10.77, and the hate item (probability 0.30, seen by 200) ranks second at 9.55.
• The violent-threat item (probability 0.85, seen by 1 user) ranks only fourth, because exposure is 1. That is the weakness of a pure product score, which is why the override exists: any item in a category with harm weight at least 9 and probability at least 0.5 goes to the urgent queue, the last line of the output shows that only e qualifies. Item b (self-harm, probability 0.40) is below the probability bar, so a team that wants it urgent lowers the bar for the top-weight categories.

Operating it

• Thresholds: set the severity cut-offs from reviewer capacity: if reviewers can read 200 items a day, the queue takes the top 200, and the cut-off follows from that.
• Feedback loop: reviewer verdicts become new labels; refit weekly and re-check calibration per category.
• Sampling below the line: audit a random sample of items scored low, or you will never learn about misses.
• Metrics: precision of the urgent queue, share of confirmed violations found by each signal, time from output to review.
• Failure modes: signals that all rely on the same keyword list fail together; a drifting classifier changes the scale of the score, so recalibrate; a harm weight set once will not reflect a new policy, so version the weights.