LLM Evaluation and Observability Questions
Measuring, monitoring and releasing generative and LLM-powered systems. Covers evaluation of open-ended outputs (human review, calibrated model-graded judges, reference-based and embedding scores, perplexity), hallucination, toxicity, privacy-leak and red-team checks, offline benchmarks versus online monitoring, and production observability (logs, traces, metrics, cost and token attribution, privacy-safe telemetry, drift and quality alerts). Also release decisions: A/B tests, canary rollouts, SLOs and go or no-go gates. Building the grounding architecture itself, calibration of predictive models and general software testing are covered elsewhere.
You are launching a generative text assistant for customer support. Define the success criteria and operational guardrails: which metrics and thresholds decide readiness for hallucination, satisfaction, latency, cost and safety, and what trade-offs are involved?
Sample Answer
Direct answer
Readiness is a set of gates, one per risk, each with a metric, a test set and a pass line fixed before anyone sees results. A gate passes on the upper confidence bound of a failure rate (how bad the true rate could plausibly be given the sample), not on the observed rate, because a few hundred reviewed conversations can look clean by luck. Runtime guardrails (a threshold below which the assistant hands the conversation to a person, an output filter, caps on tokens and spend, a kill switch) then enforce the same limits after launch. Every guardrail setting moves two things at once, which is the trade-off: tighter on hallucination or harm costs automation rate and money.
The gates
The limits below are example starting points to agree with the business, not universal values.
| Dimension | Metric | Data it needs | Example launch limit |
|---|---|---|---|
| Hallucination | Share of replies with a claim the policy or order data does not support | A few hundred real-style conversations labelled by trained reviewers | 95% upper bound at or under 2% |
| Safety | Share of unsafe replies; share of replies exposing personal data | Red-team prompts plus sampled normal traffic | Unsafe at or under 0.5%; personal-data exposure at or under 0.2% (both on the bound) |
| Satisfaction | Share of survey responders who rate the answer satisfied | Post-chat survey | 95% lower bound at or above 75% |
| Latency | p95 time to the full reply (the time within which 95% of replies finish) | Load test with realistic prompt lengths | At or under 5 s |
| Cost | Cost per conversation including the human handoffs | Token counts, handoff rate, staff cost | At or under the human-only baseline minus an agreed margin |
Why the bound, with numbers
Two kinds of bound appear. For failure gates (where the risk is that the true rate is higher than observed) the program uses the Clopper-Pearson upper bound: the largest true failure rate that would still produce so few failures at least 5% of the time. In code it is beta.ppf(0.95, k + 1, n - k), the 95th percentile of a Beta distribution (a curve over rates between 0 and 1) whose shape is set by the counts of failures and clean items; with zero failures in n items it simplifies to 1−0.051/n, close to 3/n (the "rule of three"). For example 6 failures in 500 is 1.2% observed, but the true rate could plausibly be as high as 2.355%. For the satisfaction gate the risk runs the other way (the true share could be lower than observed), so the program uses the Wilson lower bound, a normal-approximation interval that stays sensible for proportions, and checks that it clears the floor. Both are one-sided 95% limits in the direction that would hurt. The program below applies it to invented review counts (inputs, not measurements of any product), then sweeps two thresholds on synthetic scores. The groundedness score (how well a reply is supported by the sources, from 0 to 1) and the harm score are stand-ins: random numbers drawn from Beta distributions that mimic a scorer. Replies that are truly bad get scores centred near 0.43 and good ones near 0.75, with overlap, and 8% of replies are bad; the harm scorer is built the same way with 2% harmful. The percentages show the shape of the trade-off and not what your scorer will measure. Unit costs (1 unit per automated conversation, 25 units per human-handled one) are assumptions. In part 2, a threshold t means a reply scoring below t is handed to a person; in part 3, a threshold t means a reply scoring at or above t is blocked, so the two sweeps run in opposite directions.
import os
os.environ["OMP_NUM_THREADS"] = "1"
os.environ["OPENBLAS_NUM_THREADS"] = "1"
import numpy as np
from scipy.stats import beta, norm
def upper(k, n, conf=0.95):
"""One-sided Clopper-Pearson upper bound on a failure rate."""
return 1.0 if k == n else float(beta.ppf(conf, k + 1, n - k))
def wilson_lower(k, n, conf=0.95):
z = norm.ppf(conf)
p = k / n
centre = p + z * z / (2 * n)
half = z * np.sqrt(p * (1 - p) / n + z * z / (4 * n * n))
return float((centre - half) / (1 + z * z / n))
# Part 1: gates on invented review counts (inputs, not measurements of a real product)
print("gate k/n observed 95%bound limit verdict")
for name, k, n, limit in [("unsupported claims", 6, 500, 0.02),
("unsafe replies", 0, 600, 0.005),
("leaked personal data", 1, 800, 0.002)]:
ub = upper(k, n)
print(f"{name:<24} {k}/{n:<6} {k/n:8.3%} {ub:8.3%} {limit:6.1%} {'PASS' if ub <= limit else 'FAIL'}")
k, n = 1170, 1500
lb = wilson_lower(k, n)
print(f"{'satisfied responders':<24} {k}/{n:<5} {k/n:8.3%} lower {lb:.3%} floor 75.0% {'PASS' if lb >= 0.75 else 'FAIL'}")
print("\nclean items needed for a 95% bound at or under the limit (zero failures observed)")
for limit in [0.02, 0.005, 0.002]:
n = 1
while upper(0, n) > limit:
n += 1
print(f"limit {limit:.1%}: {n} items")
# Part 2: threshold sweep on a STAND-IN groundedness score (synthetic Beta draws, not a real scorer)
rng = np.random.default_rng(0)
N = 20000
bad = rng.random(N) < 0.08
score = np.where(bad, rng.beta(3, 4, N), rng.beta(6, 2, N))
LLM_COST, HUMAN_COST = 1.0, 25.0 # assumed cost units per conversation
print("\nthreshold automated halluc_among_answered cost_units_per_1000")
for t in [0.0, 0.4, 0.5, 0.6, 0.7]:
ans = score >= t
h = bad[ans].mean()
cost = 1000 * (LLM_COST + HUMAN_COST * (1 - ans.mean()))
print(f"{t:9.1f} {ans.mean():9.1%} {h:21.2%} {cost:19.0f}")
# Part 3: harmful-content filter threshold on a second stand-in score
harm = rng.random(N) < 0.02
hs = np.where(harm, rng.beta(6, 2, N), rng.beta(2, 7, N))
print("\nblock_at harmful_passed benign_blocked")
for t in [0.5, 0.6, 0.7, 0.8]:
blocked = hs >= t
print(f"{t:8.1f} {(~blocked[harm]).mean():14.1%} {blocked[~harm].mean():14.1%}")
# Part 4: latency against a budget (synthetic lognormal, seconds to full reply)
lat = rng.lognormal(mean=np.log(2.0), sigma=0.45, size=N)
print(f"\nlatency p50 {np.percentile(lat,50):.2f}s p95 {np.percentile(lat,95):.2f}s share over 5s {(lat>5).mean():.2%}")
Output:
gate k/n observed 95%bound limit verdict
unsupported claims 6/500 1.200% 2.355% 2.0% FAIL
unsafe replies 0/600 0.000% 0.498% 0.5% PASS
leaked personal data 1/800 0.125% 0.592% 0.2% FAIL
satisfied responders 1170/1500 78.000% lower 76.191% floor 75.0% PASS
clean items needed for a 95% bound at or under the limit (zero failures observed)
limit 2.0%: 149 items
limit 0.5%: 598 items
limit 0.2%: 1497 items
threshold automated halluc_among_answered cost_units_per_1000
0.0 100.0% 8.05% 1000
0.4 94.7% 4.59% 2317
0.5 89.2% 3.10% 3690
0.6 79.1% 1.83% 6226
0.7 62.8% 0.93% 10292
block_at harmful_passed benign_blocked
0.5 4.5% 3.7%
0.6 14.7% 0.8%
0.7 30.1% 0.1%
0.8 56.3% 0.0%
latency p50 2.00s p95 4.22s share over 5s 2.00%
Reading the results
- Observed 1.2% unsupported claims fails a 2% gate. 6 of 500 gives a bound of 2.355%. Zero unsafe replies in 600 gives 0.498%, which only just clears 0.5%: 598 clean items is the minimum for that limit, and a 0.2% limit needs 1,497 clean items. The personal-data gate shows the cost of a strict limit: 1 failure in 800 already fails it (0.592% bound), so the response is to fix and retest, not to relax the limit. Satisfaction passes because the lower bound (76.191%) clears the 75% floor, but survey responders are self-selected, so treat it as a soft gate.
- Hallucination versus automation versus cost. Raising the hand-off threshold from 0.5 to 0.6 (more replies scoring below it go to a person) cuts hallucination among answered conversations from 3.10% to 1.83% but automation falls from 89.2% to 79.1% and cost per 1,000 conversations rises from 3,690 to 6,226 units. At 0.7 hallucination reaches 0.93% but only 62.8% is automated and cost is 10,292 units. For a 2% limit the sweep points to a threshold near 0.6 (the first setting in the sweep where hallucination among answered conversations, 1.83%, is under 2%; that is one draw of 20,000 synthetic replies, and another draw can land a little above or below the 2% line, so 0.6 sits close to the limit rather than safely under it); the numbers belong to the stand-in, so you would repeat the sweep with your real scorer on labelled data. For the cost gate: a human-only baseline costs 25 units per conversation, 25,000 per 1,000, so the 6,226 at threshold 0.6 is about a quarter of it. The cost gate passes with a large margin here, so quality, not cost, sets the threshold.
- Safety filters trade harm against over-blocking, and the costs are not symmetric. Blocking at 0.5 lets 4.5% of harmful items through and blocks 3.7% of benign ones; at 0.7, 30.1% of harmful pass and 0.1% of benign are blocked. Here a lower block threshold is the stricter one. In support, a harmful reply reaching a customer is worse than a wrongly blocked helpful reply, which just routes to an agent, so the asymmetry favours the strict end (a block threshold of 0.5 or lower in this sweep), with human review of replies near the line and the over-block rate watched as its own metric.
- Latency is a tail metric. The median is 2.00 s but p95 is 4.22 s and 2.00% of replies exceed 5 s. A mean would hide this. Also measure time to first token if the reply streams, since users judge responsiveness by when text starts.
Operational guardrails to ship with it
- Hand-off path: low groundedness, a refusal, an angry customer or a request for money moves the chat to a person with the transcript.
- Output filter and personal-data scan before the reply is shown.
- Caps: tokens per conversation, tool calls per conversation, spend per hour.
- Kill switch that routes to the human-only flow without a deploy.
- Staged rollout to a small share of traffic with the gates computed on live traffic before widening.
- Monitoring: the same metrics on sampled live conversations, with alerts when a bound crosses its limit.
Recommendation and what would flip it
Launch on a limited share of traffic only if the two hard gates (hallucination, safety) pass on their bounds and the hand-off threshold sits near 0.6 (on the stand-in). Flip toward a stricter threshold when errors carry money or legal consequences (refund promises, medical or financial content). Flip toward a looser one when the human queue is the bottleneck and wrong answers are cheap and visible.
Common mistakes
- Passing a gate on the observed rate from a small sample.
- Tuning thresholds on the same data used for the go decision.
- Averaging latency instead of reading p95.
- One combined "quality score" that lets a good tone hide a safety failure.
- Counting deflection (the customer left) as resolution.
For a generative AI product, which business and quality metrics would you track after launch, and how does each one inform a product decision? Use a content-writing or developer-assistant product as the example.
Sample Answer
Direct answer
Track three layers and tie each to a decision. Usage and value (do people come back, and do they finish what they came to do), quality as users experience it (is the output kept, edited or thrown away), and safety and cost guardrails (harmful output, variety, spend per active user). A metric earns a place only if you can name the action a movement would trigger.
Developer assistant (code suggestions)
The worked table is hand-written: 12 suggestions shown, with whether each was accepted, whether the accepted code was still in the file 30 minutes later, the seconds a developer spent reading it, and the seconds saved if it was kept. The saved-seconds values are invented inputs. Time saved is never observed directly in real data; it needs an estimate or a holdout group (a randomly chosen set of users who do not get the feature, kept for comparison). The per-task table shows counts so each rate can be redone by hand: acceptance is accepted divided by shown, and kept is divided by accepted, the same denominator as the headline 71.4%.
import os
os.environ["OMP_NUM_THREADS"] = "1"
import difflib
import math
import pandas as pd
# Developer assistant: one row per suggestion shown (hand-written event log).
rows = [
# id, task, accepted, kept_30min, review_s, saved_s_if_kept
(1, "boilerplate", 1, 1, 4, 60),
(2, "boilerplate", 1, 1, 3, 45),
(3, "boilerplate", 0, 0, 5, 0),
(4, "tests", 1, 1, 8, 90),
(5, "tests", 1, 0, 8, 0),
(6, "tests", 0, 0, 6, 0),
(7, "debugging", 0, 0, 10, 0),
(8, "debugging", 1, 0, 12, 0),
(9, "debugging", 0, 0, 9, 0),
(10, "refactor", 1, 1, 7, 70),
(11, "refactor", 0, 0, 6, 0),
(12, "refactor", 1, 1, 6, 50),
]
df = pd.DataFrame(rows, columns=["id", "task", "accepted", "kept", "review_s", "saved_s"])
acc = df.accepted.mean()
kept_of_acc = df.kept.sum() / df.accepted.sum()
net = (df.saved_s.sum() - df.review_s.sum()) / len(df)
print(f"acceptance rate {acc:.1%}")
print(f"kept after 30 min {df.kept.sum()}/{df.accepted.sum()} accepted = {kept_of_acc:.1%}")
print(f"gross saved per shown {df.saved_s.sum()/len(df):.1f} s")
print(f"net saved per shown {net:.1f} s (review time of every shown suggestion subtracted)")
# Per task: counts, so every rate can be redone by hand (accepted/shown, kept/accepted).
by_task = df.groupby("task").agg(shown=("id", "count"), accepted=("accepted", "sum"), kept=("kept", "sum"))
by_task["kept_of_accepted"] = (by_task.kept / by_task.accepted).round(2)
print(by_task.to_string())
# Content writing: draft vs published text, and variety across drafts.
drafts = [
"Our new plan saves your team hours every week.",
"Our new plan saves your team hours every month.",
"Our new plan saves your team time every week.",
"Meet the plan built for busy support teams.",
]
published = [
"Our new plan saves your team hours every week.",
"The new plan gives your team hours back each month.",
"Our plan saves support teams time every week.",
"Meet the plan built for busy support teams.",
]
edit = [1 - difflib.SequenceMatcher(None, d, p).ratio() for d, p in zip(drafts, published)]
print("\nedit ratio per draft ", [round(e, 2) for e in edit])
print(f"mean edit ratio draft->published {sum(edit)/len(edit):.2f} (0 = published untouched)")
print(f"drafts published unchanged {sum(e==0 for e in edit)}/{len(edit)}")
def distinct2(texts):
bigrams = []
for t in texts:
w = t.lower().rstrip(".").split()
bigrams += list(zip(w, w[1:]))
return len(set(bigrams)) / len(bigrams)
print(f"distinct-2 across drafts {distinct2(drafts):.2f} (unique bigrams / all bigrams)")
print(f"distinct-2 if every draft were the first one {distinct2([drafts[0]]*4):.2f}")
# Business metrics from hand-written counts.
cohort, active_wk4 = 200, 84
print(f"\n4-week retention {active_wk4}/{cohort} = {active_wk4/cohort:.1%}")
n_use = n_hold = 400
done_use, done_hold = 252, 220
p1, p0 = done_use / n_use, done_hold / n_hold
half = 1.96 * math.sqrt(p1 * (1 - p1) / n_use + p0 * (1 - p0) / n_hold)
print(f"completion with feature {p1:.1%}, holdout {p0:.1%}, gap {100*(p1-p0):.1f} points +-{100*half:.1f}")
Output:
acceptance rate 58.3%
kept after 30 min 5/7 accepted = 71.4%
gross saved per shown 26.2 s
net saved per shown 19.2 s (review time of every shown suggestion subtracted)
shown accepted kept kept_of_accepted
task
boilerplate 3 2 2 1.0
debugging 3 1 0 0.0
refactor 3 2 2 1.0
tests 3 2 1 0.5
edit ratio per draft [0.0, 0.22, 0.13, 0.0]
mean edit ratio draft->published 0.09 (0 = published untouched)
drafts published unchanged 2/4
distinct-2 across drafts 0.58 (unique bigrams / all bigrams)
distinct-2 if every draft were the first one 0.25
4-week retention 84/200 = 42.0%
completion with feature 63.0%, holdout 55.0%, gap 8.0 points +-6.8
What each number tells the product team:
| Metric | Value here | Decision it informs |
|---|---|---|
| Acceptance rate (accepted / shown) | 58.3% | Whether suggestions look useful at first glance. It rises when suggestions are easy to accept, not only when they are good. |
| Kept after 30 minutes (kept / accepted) | 5 of 7, 71.4% | Whether accepted code survived. This is the quality check on acceptance. |
| Net seconds saved per suggestion shown | 19.2 against 26.2 gross | Whether the feature pays for the reading time it costs. Gross value ignores the review time of the suggestions nobody used. |
| Breakdown by task | debugging: 3 shown, 1 accepted, 0 kept | Where to look first. Here the one accepted debugging suggestion did not survive. That is a single case, so it is a reason to read those suggestions, not a finding; stop or change the model only once a task shows the pattern across many accepted events. |
For the table: boilerplate was accepted 2 of 3 times and both accepted ones were kept, so kept of accepted is 2/2 = 1.0; tests were accepted 2 of 3 and kept 1 of 2 = 0.5. With 12 rows these figures are only an illustration of the arithmetic; a real read needs thousands of events and a confidence interval.
Content-writing assistant (marketing drafts)
The same logic applies with different signals. Edit ratio is how much of a draft the writer changed before publishing (here one minus the character-level similarity from Python's difflib, so 0 means published untouched). Variety is distinct-2: unique word pairs divided by all word pairs across a set of drafts; low variety means near-identical drafts. The four drafts and published versions are hand-written.
The run shows per-draft edit ratios of 0.0, 0.22, 0.13 and 0.0, whose mean is (0+0.22+0.13+0)/4≈0.09, with two of four drafts published unchanged. A value near 0 means writers accept drafts as they are; a value near 1 means they rewrite them. Distinct-2 is 0.58 for the drafts: the four drafts contain 8 + 8 + 8 + 7 = 31 word pairs, of which 18 are different. The baseline 0.25 is four identical 9-word drafts: 4 x 8 = 32 pairs, only 8 different, and 8/32 = 0.25. Decisions: a rising edit ratio means the drafts are drifting from what writers want; a falling distinct-2 means the tool is converging on one template, which for email campaigns shows up as recipients seeing the same line repeatedly.
Business and guardrail metrics for both products
- Weekly active users and 4-week retention of people who used the feature at least once: the share of a starting group still using it four weeks later, which shows whether it became a habit. In the program's hand-written cohort, 84 of 200 first-week users were active in week 4, a retention of 42.0%. Decides investment.
- Task completion (a draft published, a pull request merged) among users who used the feature versus a comparison group who did not. A matched group is a set of non-users chosen to look like the users (same plan, team size and tenure); a randomly held-out group is better because it removes the choice. In the hand-written counts, 252 of 400 users with the feature completed the task (63.0%) against 220 of 400 in the holdout (55.0%): a gap of 8.0 points, plus or minus 6.8 at 95%, so the lower end of the range is barely above zero and more data is needed before crediting the feature. Decides whether the feature changes outcomes.
- Cost per active user (model calls plus review): decides pricing and whether to route easy requests to a cheaper model.
- Harmful or off-brand output rate from a classifier plus human spot checks (toxicity in drafts, insecure code patterns in suggestions): a guardrail with a limit, not a goal.
- Complaint and thumbs-down rate by task: tells you where to look, not how good the product is, since few users give feedback.
Common mistakes
- Treating acceptance rate as the quality metric.
- Reporting time saved from the suggestion's length instead of from a holdout comparison.
- Looking only at averages and missing a task type where the product fails.
- Counting usage of the feature as value without checking whether outcomes improved.
Design the dashboards you would build to monitor an LLM product: one for infrastructure and reliability owners, one for product owners. What would each show, which alerts, and how would a viewer drill down? Include how bias, privacy-leak risk and user satisfaction surface.
Sample Answer
Direct answer
Build two dashboards on the same trace data (a trace is the record of one request's path through the system, made of spans, one per step such as retrieval or a model call). The reliability dashboard answers "is it up, fast, and within budget?" for SRE and infrastructure owners, and pages people. The product dashboard answers "is it good, fair, safe and liked?" for product owners, and feeds weekly decisions rather than pages. Both drill from a headline number to a slice, to a single trace. Bias, privacy-leak risk and satisfaction live on the product side, behind access controls, because the drill-down ends in user conversations.
Reliability dashboard (SRE, infrastructure)
| Panel | Shows | Alert |
|---|---|---|
| Availability and error rate | Successful responses over total, by endpoint, model version, region | SLO burn rate (below) |
| Latency | p50, p95, p99 of time to first token and total time | p95 above the limit for several windows |
| Throughput and saturation | Requests per second, queue depth, rate-limit rejections | Queue depth growing |
| Tokens and cost | Input and output tokens per request, spend per feature | Spend per request outside a band |
| Dependencies | Retrieval, tool and provider call latency and failure | Dependency failure share |
| Deploys | Version markers on every chart | None, context only |
Terms in the table: a percentile such as p95 is the value that 95 percent of requests beat (p50 is the median, p99 means only 1 request in 100 is slower), which shows the slow tail that an average hides. Time to first token is how long a user waits until the first word of a streamed reply appears. Saturation is how full the system is relative to what it can handle, and queue depth is how many requests are waiting for a free worker (a growing queue means saturation). Rate-limit rejections are requests refused because a quota was exceeded, either ours or the model provider's.
Drill-down: headline chart, then split by version, region and endpoint, then the slowest or failed traces, then the span waterfall, a chart that lays a trace's spans out along a time axis so you can see which step took the time.
Alerting on an SLO (service level objective: a target such as 99.5% of requests succeed). The error budget is 1 minus the target, here 0.005. The burn rate is the observed bad fraction divided by the budget: 1.0 means spending the budget exactly as fast as allowed, and below 1.0 is healthy; there is no universal normal for a latency percentile, so judge p95 against your own latency limit and its history. A robust page requires a short window and a long window to both exceed the threshold, so a one-hour blip does not page and a real incident is not missed. The program below computes it on hand-set hourly data (20,000 requests per hour, an incident starting at hour 7).
Product dashboard (product owners)
| Panel | Shows | Notes |
|---|---|---|
| Quality | Share of sampled answers passing a rubric (human or judge model), faithfulness for grounded products | Judge scores are proxies validated on human labels |
| Safety | Policy-violation rate, refusal rate, over-refusal on benign prompts | Both directions matter |
| Satisfaction | Thumbs up/down, task resolved, regenerate and abandon rates, by feature | Self-selected feedback, so show response counts |
| Bias | The same quality, refusal and satisfaction metrics split by language or user group, with intervals and the gap to the best group | Show uncertainty; small groups produce wild rates |
| Privacy-leak risk | Count of responses where a detector finds personal data patterns or memorised text, and of prompts that contained personal data and were retained | A detector count is a lower bound, not a guarantee |
| Topic and intent mix | What users ask about and where failures cluster | Finds the slice to fix |
Alerts: week-over-week drop in the quality pass rate beyond its interval, a rise in policy violations, a gap between groups that widens, and any leak-detector hit above zero for a restricted category (this one goes to the security owner, not the product channel).
Drill-down: metric, then slice, then a sample of examples with the rubric result, then the full conversation only if the viewer's role allows it.
Access controls on the sensitive panels
- Roles: SREs see operational traces with content redacted; product owners see aggregates and redacted examples; a small named reviewer group (privacy, trust and safety) can open raw conversations, with each open logged and a stated purpose.
- Redaction at ingestion: strip or hash identifiers and personal data before traces reach the dashboard store, and set retention limits.
- Small-cell suppression: do not show a group metric below a minimum count, because a rate over 5 users can identify them. The program uses a floor of 100 for illustration; choose the floor with your privacy owner.
- Leak panel is itself sensitive: store only counts and categories, not the leaked strings, in the shared view.
Worked example
The program has two parts. Part 1 computes the SLO burn rate over one-hour and six-hour windows and pages when both exceed 3 (a threshold chosen here as policy, not a standard). Part 2 builds a small synthetic conversation log, then shows the refusal rate with a Wilson interval (a 95 percent range for the true rate that stays sensible on small counts) per language group, suppressing any group below the floor. The last lines count personal-data pattern hits in four hand-written outputs; the regular expression is a minimal illustration of the kind of detector that feeds the leak panel, covering only a US-style phone fragment, a social-security-like number and a simple email shape, and production detectors must also handle other formats, obfuscation and memorised text.
import numpy as np
import pandas as pd
# ---- Part 1: SLO burn-rate alert for the reliability dashboard (hand-set hourly data) ----
SLO = 0.995 # target: 99.5% of requests succeed within the latency limit
budget = 1 - SLO # allowed bad fraction = 0.005
THRESH = 3.0 # page when the budget is spent 3x too fast (a policy choice)
req_per_hour = 20_000
bad_per_hour = np.array([60, 80, 70, 90, 75, 85, 70, 400, 450, 420, 430, 410]) # incident starts at hour 7
def burn(bad, window):
frac = bad[-window:].sum() / (req_per_hour * window)
return frac / budget # 1.0 = spending the budget exactly as fast as allowed
for hour in (6, 7, 8, 9, 10, 11):
b = bad_per_hour[:hour + 1]
short, long_ = burn(b, 1), burn(b, min(6, len(b)))
page = short > THRESH and long_ > THRESH # both windows must exceed the threshold
print(f"hour {hour:2d}: burn(1h)={short:5.2f} burn(6h)={long_:5.2f} page={page}")
# ---- Part 2: product dashboard slice with a privacy floor (synthetic conversation log) ----
rng = np.random.default_rng(0)
groups = {"lang_a": (900, 0.040, 0.78), "lang_b": (700, 0.045, 0.77), "lang_c": (40, 0.12, 0.60)}
rows = []
for g, (n, refuse_p, sat_p) in groups.items():
for _ in range(n):
rows.append((g, int(rng.random() < refuse_p), int(rng.random() < sat_p)))
d = pd.DataFrame(rows, columns=["group", "refused", "thumbs_up_or_resolved"])
K_MIN = 100 # privacy floor: cells smaller than this are not shown to product viewers
def wilson(k, n, z=1.96):
p = k / n
c = (p + z * z / (2 * n)) / (1 + z * z / n)
h = z * np.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / (1 + z * z / n)
return c - h, c + h
lo2, hi2 = wilson(2, 10)
print(f"Wilson interval for 2 refusals in 10: [{lo2:.3f}, {hi2:.3f}]; the plain p +/- 1.96*sqrt(p(1-p)/n) lower end would be {0.2 - 1.96 * np.sqrt(0.2 * 0.8 / 10):.3f}")
out = []
for g, x in d.groupby("group"):
n = len(x)
if n < K_MIN:
out.append((g, n, "suppressed (n < %d)" % K_MIN, "", ""))
continue
lo, hi = wilson(x.refused.sum(), n)
out.append((g, n, f"{x.refused.mean():.3f} [{lo:.3f}, {hi:.3f}]", f"{x.thumbs_up_or_resolved.mean():.3f}", ""))
print(pd.DataFrame(out, columns=["group", "n", "refusal rate [95% Wilson]", "satisfaction", ""]).iloc[:, :4].to_string(index=False))
# PII-leak counter: simple pattern count on logged outputs (stub detector, illustrates a count not a guarantee)
import re
outputs = ["Call me at 555-0142 or mail a@b.co", "No personal data here", "ID 123-45-6789 appeared", "Plain answer"]
pat = re.compile(r"\b[0-9]{3}-[0-9]{4}\b|\b[0-9]{3}-[0-9]{2}-[0-9]{4}\b|[A-Za-z0-9._-]+@[A-Za-z0-9.-]+\.[a-z]{2,}")
hits = [bool(pat.search(o)) for o in outputs]
print("outputs with a pattern hit:", sum(hits), "of", len(outputs))
Output:
hour 6: burn(1h)= 0.70 burn(6h)= 0.78 page=False
hour 7: burn(1h)= 4.00 burn(6h)= 1.32 page=False
hour 8: burn(1h)= 4.50 burn(6h)= 1.95 page=False
hour 9: burn(1h)= 4.20 burn(6h)= 2.50 page=False
hour 10: burn(1h)= 4.30 burn(6h)= 3.09 page=True
hour 11: burn(1h)= 4.10 burn(6h)= 3.63 page=True
Wilson interval for 2 refusals in 10: [0.057, 0.510]; the plain p +/- 1.96*sqrt(p(1-p)/n) lower end would be -0.048
group n refusal rate [95% Wilson] satisfaction
lang_a 900 0.039 [0.028, 0.054] 0.774
lang_b 700 0.041 [0.029, 0.059] 0.780
lang_c 40 suppressed (n < 100)
outputs with a pattern hit: 2 of 4
- At hour 7 an incident raised the bad rate to about 2% (400 of 20,000, four times the budget rate). The one-hour burn is already 4.00 at hour 7 and 4.50 at hour 8, but the six-hour window still holds quiet hours, so its burn is only 1.32 and then 1.95, and the page waits. By hand at hour 10: the last six hours (5 to 10) hold 85 + 70 + 400 + 450 + 420 + 430 = 1,855 bad requests out of 120,000, which is 1.55% and a burn of 1.55 / 0.5 = 3.09, above 3 together with the one-hour burn of 4.30. So the first page is at hour 10, the fourth bad hour (7, 8, 9, 10), and it stays on at hour 11. This delay is the price of avoiding false pages; a lower long-window threshold pages earlier and more often.
- The Wilson line shows why that interval is used: for 2 refusals in 10 the plain margin would put the lower end below zero, while the Wilson interval stays between 0 and 1 and is centred slightly toward the middle. It widens as n shrinks, which is why lang_c would be too uncertain to show anyway.
lang_aandlang_bhave refusal rates of 0.039 and 0.041 with overlapping intervals: no evidence of a gap.lang_c, with 40 conversations, is suppressed. Suppression protects individuals but can hide the group at highest risk, so give the restricted reviewer role the pooled figure over a longer period for that group.- Two of the four sample outputs matched a personal-data pattern, which is a number of the sample, not a rate.
Mistakes to avoid
- One dashboard for everyone: product owners get paged on latency, SREs get raw conversations.
- Averages only. Always show a percentile or a distribution for latency and an interval for rates.
- Bias metrics without group sizes or intervals.
- Satisfaction shown without the number of responses behind it.
What privacy and compliance issues arise when collecting user prompts and model responses for observability, and what minimally invasive collection strategies would you use? How would you document the decisions?
Sample Answer
Direct answer
Prompts and responses are often personal data: users paste names, health or financial details, and confidential business text. Collecting them raises issues of lawful basis and consent, purpose limitation, data minimization, retention, access, cross-border transfer, deletion and access rights, security, and third-party sharing (including whether a provider or vendor uses the data to train models). The minimally invasive approach is to collect metadata by default, content only when needed and in a redacted, sampled, short-lived form, and to write each decision in a short, reviewable record. Which laws and contracts apply depends on the users, the data and the region, so the final reading belongs to legal counsel; the product and engineering job is to make the compliant option technically available.
The issues, one line each
- Lawful basis and notice. A lawful basis is the legal ground that lets you process personal data at all (for example the user's consent, a contract, or a legitimate business interest; which ones exist depends on the law). Users need to be told what is collected and why; some uses (analytics, model improvement) may need separate consent or an opt-out.
- Purpose limitation. Data may be used only for the purpose you told the user about when you collected it. Data collected to debug errors should not quietly become training data.
- Data minimization. Collect the least that answers the question you actually have.
- Sensitive categories. Health, children's data, credentials and payment details raise the bar; many teams exclude them from collection entirely.
- Retention. Keep content only as long as its purpose lasts.
- Access and deletion rights. A data subject (the person the data is about) may ask to see or delete what you hold on them, so you must be able to find and delete one user's records on request.
- Third parties and transfers. Observability vendors and model providers are processors (companies that handle the data on your behalf and under your instructions); contracts matter, and so does data residency (the country or region where the data is physically stored), because moving data across borders can carry its own legal requirements.
- Security. Logs are a favourite place for leaks; encryption, access control and audit apply.
- Enterprise customers. Many contracts forbid logging their content at all, so the pipeline needs a per-tenant "no content" switch.
Minimally invasive collection strategies
- Metadata only by default: token counts, latency, model version, template id, outcome flags, coarse topic labels produced by an on-path classifier that stores the label and not the text.
- Flag-triggered capture: store the content only when a safety classifier, error or user thumbs-down fires.
- Small random sample of redacted content for quality review.
- Opt-in or contractual capture for accounts that agreed to it.
- Redaction before storage, short retention for content, and a per-user pseudonym (a stable code that stands in for the user id, computed with a secret key) kept as an index on each stored record, so a deletion request can look up that one code instead of scanning everything.
A computed comparison
The program counts, for 100,000 requests a day, how much content is held under four options. Exposure is measured as record-days: records stored per day times the days each is kept. In steady state, once the system has run longer than the retention period, each day adds that many new records and drops the oldest day's records, so the content on disk at any moment equals records per day times retention days. It is a simple proxy for how much personal text sits in storage. All four options also keep metadata; the table counts only content. The 0.5% flagged slice is synthetic and the retention periods (400 days for option A, 30 days for the others) are assumptions. Sampling is done by hashing the trace id, so every run picks the same requests.
import hashlib
N_PER_DAY = 100_000
def want_content(trace_id, rate, flagged):
if flagged: return True
return int(hashlib.sha256(trace_id.encode()).hexdigest()[:8], 16) / 0xFFFFFFFF < rate
ids = [f"r{i}" for i in range(N_PER_DAY)]
flag = {i for i in range(0, N_PER_DAY, 200)} # 0.5% flagged by a safety classifier (synthetic)
print("Option content records/day record-days of content held at steady state")
for name, rate, ret in [("A: log everything, 400 d", 1.0, 400), ("B: log everything, 30 d", 1.0, 30),
("C: metadata + 2% sampled", 0.02, 30), ("D: flagged cases only", 0.0, 30)]:
n = sum(want_content(t, rate, i in flag) for i, t in enumerate(ids))
print(f"{name:34}{n:>9,} {n*ret:>14,}")
expected_c = len(flag) + 0.02 * (N_PER_DAY - len(flag))
print(f"Option C by arithmetic: {len(flag)} flagged + 2% of the other {N_PER_DAY - len(flag):,} = {expected_c:,.0f}")
# data subject deletion: cost of locating one user's records
print("Deletion lookup: with per-user pseudonym index, records scanned = that user's only; without index = all retained")
print(f" without index at 30 d retention, option B: scan {N_PER_DAY*30:,} records per request")
Output:
Option content records/day record-days of content held at steady state
A: log everything, 400 d 100,000 40,000,000
B: log everything, 30 d 100,000 3,000,000
C: metadata + 2% sampled 2,480 74,400
D: flagged cases only 500 15,000
Option C by arithmetic: 500 flagged + 2% of the other 99,500 = 2,490
Deletion lookup: with per-user pseudonym index, records scanned = that user's only; without index = all retained
without index at 30 d retention, option B: scan 3,000,000 records per request
Reading the output:
- Logging everything for 400 days holds 40,000,000 record-days of content; simply cutting retention to 30 days brings that to 3,000,000.
- Metadata plus a 2% sample plus flagged cases (option C) holds 2,480 content records a day and 74,400 record-days: about 40 times less than option B, while still giving reviewers examples and every flagged failure. The 2,480 comes from 500 flagged records plus 2% of the other 99,500 requests, which is 2,490 by arithmetic; the hash-based sample lands on 2,480 because it picks about 2%, not exactly 2%.
- Option D (content kept only for flagged cases, with metadata for everything else) holds 15,000 record-days but gives no unbiased quality sample, so you cannot estimate the rate of problems from it. Minimization has a measurement cost; choose the lowest-exposure option that still answers your questions.
- The deletion note at the bottom is the reason for a pseudonym index: if each stored record carries the user's pseudonym and that field is indexed (like a book's index, it jumps straight to the matching records), a deletion request touches only that user's records. Without one, answering it means scanning all 3,000,000 retained records under option B.
How to document the decisions
Write a short decision record per data stream: purpose, fields collected, why each is needed, lawful basis and notice text, retention, who can access it and how access is audited, where it is stored, which vendors process it, how deletion requests are handled, who approved it, and the review date. A filled-in record, with illustrative values, looks like this:
Decision record: prompt and response capture for the support assistant (illustrative values)
Stream: prompt and response text, support assistant, production
Purpose: debug failed answers and review answer quality; not model training
Fields collected: redacted prompt, redacted response, template id, model version, latency, tool outcome
Why each field: text is needed to reproduce a failure; the rest locate it
Sampling: flagged cases always; 2% random sample otherwise
Lawful basis and notice: basis chosen with counsel (consent or legitimate interest); notice text in the product's privacy page, section on assistant logs
Retention: content 30 days; metadata 400 days
Access: redacted text for engineers; raw text only by break-glass request, audited
Storage and vendors: one region; observability vendor acts as a processor under a signed agreement; vendor retention set to 30 days
Deletion: lookup by user pseudonym; completed within the period counsel sets
Opt-out: per-tenant "no content" switch
Approved by: product owner, security, privacy counsel
Review date: six months from approval, or sooner if the product or the law changes
Around the records, keep a data inventory (a list of every place personal data is stored, with its owner) and run a privacy impact assessment (a structured review of the risks to users, written before a new kind of collection starts). Revisit each record when the product or the law changes.
Mistakes
Collecting everything "in case we need it", logging prompts in general-purpose application logs, using production content to train without a stated purpose, forgetting the vendor's own retention, and having no way to honour a deletion request.
How would you quantify the business cost of hallucinations from a generative assistant, including direct, indirect and detection costs? What would you instrument and how would you estimate the numbers?
Sample Answer
Direct answer
Cost of hallucinations = (how many wrong answers) x (how often a wrong answer causes harm) x (what that harm costs), summed over three buckets: direct (refunds, support contacts, rework, liability), indirect (lost trust, churn, lost sales, reputational damage) and detection (reviewers, automated checking, engineering). You estimate each factor separately, give every input a range, and report the total as a range with the input that drives most of the spread. Then you instrument that input first.
The model, with every term named
For a month, with V answers served, hallucination rate h, probability a that a hallucinated answer is acted on in a way that causes a cost, direct cost cd per such event, probability r that such an event makes the customer leave, and lifetime value lost L per departing customer:
cost=directVhacd+indirectVharL+detectionHw+Vgwhere H is reviewer hours, w the hourly rate, and g the automated-check cost per answer. All money amounts are in dollars and all counts are per month: cd is dollars per event, and L is the future revenue one departing customer would have brought (their lifetime value), in dollars. Each factor is observable in a different way:
| Input | How you instrument or estimate it |
|---|---|
| h | Random sample of production answers, labelled by trained reviewers against a written definition of "hallucinated". The sample size sets the precision. |
| a | Link answers to later actions: a support ticket within a day that cites the answer, a refund, a "this was wrong" button, a repeat question, an edit of generated text. Tag high-stakes intents separately. |
| cd | Finance data: average handling cost of the ticket type, refund amounts, rework time, and legal cost for the few severe cases. Use the distribution; one rare severe case can dominate the mean. |
| r, L | Cohort comparison: users who had a confirmed bad answer versus matched users who did not, then their retention difference. Matching means pairing each affected user with an unaffected user of similar tenure, plan and usage, then comparing the share of each group still active after, say, 90 days; r is that gap, and L comes from finance. The comparison is observational and other causes of leaving are mixed in, so report r as an upper estimate and say what you matched on. |
| H, w, g | Review hours logged, loaded hourly cost, and the metered cost of automated checking. |
Worked example with an explicit uncertainty range
Every input in the program is an illustrative assumption for a made-up assistant, not a measurement; the one data-like input is the labelled sample (21 hallucinated of 600 labelled). The rate is drawn from a Beta(22, 580) distribution, which is the plausible range for the true rate after seeing 21 hallucinations in 600 answers when you start with no opinion about it (a uniform prior); the program prints its 5th and 95th percentiles. The first block is the point estimate; the second draws the uncertain inputs from stated ranges 20000 times (Monte Carlo: repeat the calculation many times, each time picking every uncertain input at random from its range, so the spread of totals shows the uncertainty) and reports percentiles (the 5th percentile is the value that 5% of the totals fall below); the next block ranks which input drives the spread, by Spearman rank correlation (replace each value by its rank, then correlate; +1 means the input and the total always rise together, 0 means no relation); the last block repeats that ranking on 10 other seeds, with the p_act range wide and then narrow.
import os
os.environ["OMP_NUM_THREADS"] = "1"
os.environ["OPENBLAS_NUM_THREADS"] = "1"
import numpy as np
rng = np.random.default_rng(0)
# ALL inputs below are illustrative assumptions for a made-up assistant, not measurements.
monthly_answers = 1_000_000
labelled = 600 # random sample labelled by reviewers
halluc_found = 21 # hallucinated answers in that sample
p_hat = halluc_found / labelled
# Cost model per month. Each line is (rate of occurrence among hallucinations, cost per occurrence).
# Direct: user acts on a wrong answer and support/refund/rework follows.
# Indirect: churn / lost trust, estimated from a cohort comparison (assumed here).
# Detection: reviewer time + automated judge calls + engineering to maintain.
def monthly_cost(h, p_act, direct_cost, p_churn, ltv_loss, review_hours, hour_rate, judge_cost_per_1k):
n_bad = monthly_answers * h
direct = n_bad * p_act * direct_cost
indirect = n_bad * p_act * p_churn * ltv_loss
detection = review_hours * hour_rate + monthly_answers / 1000 * judge_cost_per_1k
return direct, indirect, detection
point = monthly_cost(p_hat, p_act=0.15, direct_cost=18.0, p_churn=0.02, ltv_loss=240.0,
review_hours=160, hour_rate=45.0, judge_cost_per_1k=0.8)
print("rate estimate:", round(p_hat, 4))
for name, v in zip(["direct", "indirect", "detection"], point):
print(f"{name:9s} {v:12,.0f}")
print(f"{'total':9s} {sum(point):12,.0f}")
# Uncertainty: sample every uncertain input from a stated range (Monte Carlo, 20000 draws).
D = 20000
h = rng.beta(halluc_found + 1, labelled - halluc_found + 1, D) # plausible values of the true rate given 21 of 600: Beta(22, 580), a uniform-prior posterior
p_act = rng.uniform(0.05, 0.30, D)
direct_cost = rng.uniform(8, 35, D)
p_churn = rng.uniform(0.005, 0.05, D)
tot = []
for i in range(D):
d, ind, det = monthly_cost(h[i], p_act[i], direct_cost[i], p_churn[i], 240.0, 160, 45.0, 0.8)
tot.append(d + ind + det)
tot = np.array(tot)
q = np.percentile(tot, [5, 50, 95])
print("monthly total, 5th / 50th / 95th percentile:", [f"{x:,.0f}" for x in q])
# Which input drives the spread? rank correlation with the total.
from scipy.stats import spearmanr
for name, x in [("halluc rate", h), ("p_act", p_act), ("direct_cost", direct_cost), ("p_churn", p_churn)]:
print(f"{name:12s} spearman {spearmanr(x, tot)[0]:.2f}")
# The hallucination rate: 5th and 95th percentile of the Beta draws (the plausible range for the true rate).
print("rate, 5th / 95th percentile:", [round(float(x), 4) for x in np.percentile(h, [5, 95])])
# Does the ranking depend on the ranges chosen, and on the random seed? Repeat on 10 other seeds,
# once with the wide p_act range used above and once with a narrow one (0.13 to 0.17).
def sensitivity(seed, act_lo, act_hi, n=5000):
r = np.random.default_rng(seed)
hh = r.beta(halluc_found + 1, labelled - halluc_found + 1, n)
pa = r.uniform(act_lo, act_hi, n)
dc = r.uniform(8, 35, n)
pc = r.uniform(0.005, 0.05, n)
d, ind, det = monthly_cost(hh, pa, dc, pc, 240.0, 160, 45.0, 0.8)
t = d + ind + det
return [spearmanr(x, t)[0] for x in (hh, pa, dc, pc)]
labels = ["halluc rate", "p_act", "direct_cost", "p_churn"]
for lo, hi in [(0.05, 0.30), (0.13, 0.17)]:
res = np.array([sensitivity(s, lo, hi) for s in range(1, 11)])
top = [labels[i] for i in res.argmax(axis=1)]
print(f"\np_act range {lo} to {hi}, 10 seeds: spearman min / max")
for j, nm in enumerate(labels):
print(f" {nm:12s} {res[:, j].min():.2f} / {res[:, j].max():.2f}")
print(" largest input on every seed:", sorted(set(top)), "| seeds where it was p_act:", top.count("p_act"), "of 10")
Output:
rate estimate: 0.035
direct 94,500
indirect 25,200
detection 8,000
total 127,700
monthly total, 5th / 50th / 95th percentile: ['57,402', '167,084', '383,456']
halluc rate spearman 0.32
p_act spearman 0.76
direct_cost spearman 0.47
p_churn spearman 0.20
rate, 5th / 95th percentile: [0.0249, 0.0497]
p_act range 0.05 to 0.3, 10 seeds: spearman min / max
halluc rate 0.31 / 0.35
p_act 0.75 / 0.77
direct_cost 0.46 / 0.49
p_churn 0.17 / 0.21
largest input on every seed: ['p_act'] | seeds where it was p_act: 10 of 10
p_act range 0.13 to 0.17, 10 seeds: spearman min / max
halluc rate 0.50 / 0.52
p_act 0.17 / 0.22
direct_cost 0.75 / 0.76
p_churn 0.28 / 0.30
largest input on every seed: ['direct_cost'] | seeds where it was p_act: 0 of 10
Check by hand against the formula: 106×0.035×0.15×18=94,500 for direct; indirect multiplies the same event count (35,000 hallucinations times 0.15 acted on = 5,250 events) by 0.02 x 240 = 4.8, giving 25,200; detection is 160 x 45 + 1,000 x 0.8 = 8,000. Total 127,700.
What the numbers say:
- The point estimate (127,700) sits below the median of the simulation (167,084) because several ranges are skewed above the point values used. The honest statement is the interval: with these assumptions, the 5th to 95th percentile is about 57,000 to 383,000 per month, a factor of roughly 7 between ends.
- In this run the spread is driven mainly by a (Spearman 0.76), then the direct cost per event (0.47), then the hallucination rate itself (0.32). That ranking is a consequence of the ranges chosen: a was allowed to vary sixfold (0.05 to 0.30), while the rate's 5th to 95th percentile spans only 0.0249 to 0.0497. The last block of the output repeats the simulation on 10 other seeds. With the same ranges a comes first on all 10 (0.75 to 0.77). With a narrowed to 0.13 to 0.17 it drops to 0.17 to 0.22 and the direct cost per event comes first on all 10. So the ranking is stable across random seeds but not across assumptions, and the honest conclusion is conditional: if you really are that unsure how often a wrong answer is acted on, then linking wrong answers to downstream actions will shrink the interval more than a larger labelled sample will; if you already know that fraction well, spend the effort on the next-largest input.
- Detection cost (8,000) is small next to the harm in this example. That ordering is an artefact of the assumptions, but the comparison is the point: detection spending is justified when it removes more expected harm than it costs. If a check halves h at the same coverage, it saves about half of the direct and indirect terms and is judged against its own cost, not against zero.
What to instrument first
- A labelled production sample every week, stratified by intent, so h comes with an interval and per-intent values.
- An "answer id" on every response, carried into support tickets, feedback buttons and refund records, to estimate a.
- A severity tag on confirmed cases (annoyance, wasted time, money lost, safety or legal exposure). Average cost hides the severe tail.
- A retention comparison between users with and without a confirmed bad answer.
Pitfalls
- Summing an average cost over rare severe events: report the severe tail separately.
- Counting every wrong answer as a cost. Many wrong answers are never acted on; that is what a is for.
- Attributing all churn to hallucinations. Cohort differences include other causes; match on usage and tenure and present the result as an upper estimate.
- Ignoring the cost of over-blocking: refusals and hedges also cost customers, so the cost of the mitigation must include lost helpful answers.
- Presenting a single number. Decision makers need the range and the dominant input.
Unlock Full Question Bank
Get access to all 8 LLM Evaluation and Observability interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.