LLM Evaluation and Observability Questions

Direct answer

Cost per period is requests times tokens per request times price per token, so a spike must come from one of those factors. Investigate in order: confirm it is real and bound it in time, then split the increase by volume (more requests) versus per-request size (more tokens per call) versus price or model mix, then drill into tenant, route, model and prompt version. The split points to the cause: a retry loop or abusive client inflates request counts, a prompt-template change inflates input tokens, a new feature or a changed model inflates output tokens or price.

Order of checks

1. Is it real? Compare ledger counts (the ledger is a table with one row per model call, recording tenant, tokens, model and cost) or gateway counts with the provider's own usage report. A billing-pipeline duplicate, a double-counted retry or a changed rate table looks like a spike but is not.
2. When did it start? Find the minute the curve bends and list every deploy, config change, prompt change, model switch and customer launch near that time.
3. Decompose. Volume versus tokens per request, then by tenant, route, model, prompt version, status code and authenticated versus anonymous traffic (calls that carry a valid API key or login versus calls that carry none).
4. Look at shapes, not averages. Distribution of input tokens per request (a new long tail?), output tokens, requests per client per minute, the share of requests that are retries or timeouts (a retry storm is many clients resending failed or slow requests at once, which multiplies the load).
5. Look at the traffic itself: repeated identical prompts (a loop or a cache that stopped working), unusual user agents or IP ranges, a prompt-injection pattern (input crafted to make the model ignore its instructions, here to make it write a lot), an agent calling itself.
6. Contain first, diagnose second: a rate limit or budget cap on the offending key, a max output token limit, or a rollback of the prompt change.

A computed example

The program builds a baseline day and a spike day from synthetic requests (seeded; tokens are normal draws around a per-tenant mean; the costs use placeholder rates in arbitrary units). In the code, day() makes one row per request: input and output token counts are drawn from a normal distribution around the tenant's mean (clip sets a floor of 50 and 10 so a draw cannot go negative), and cost is tokens times the placeholder price per 1,000 tokens. g0 and g1 group the rows by tenant: n is the request count, tin the mean input tokens per request, cost the total. cmp puts the baseline and spike columns side by side; req_ratio is spike requests divided by baseline requests, input_tok_ratio is spike mean input tokens divided by baseline, and delta_cost is spike cost minus baseline cost. Three causes are planted: tenant c sends four times its usual requests (a retry loop), tenant b's input tokens triple (a prompt template grew), and anonymous traffic grows tenfold (an abuse pattern).

import numpy as np, pandas as pd
rng = np.random.default_rng(0)
PRICE_IN, PRICE_OUT = 1.0, 4.0   # placeholder cost units per 1,000 tokens

def day(n_by_tenant, in_by_tenant, out_by_tenant):
    rows = []
    for t in n_by_tenant:
        n = n_by_tenant[t]
        tin = rng.normal(in_by_tenant[t], in_by_tenant[t]*0.1, n).clip(50)
        tout = rng.normal(out_by_tenant[t], out_by_tenant[t]*0.1, n).clip(10)
        rows.append(pd.DataFrame({"tenant": t, "tin": tin, "tout": tout}))
    d = pd.concat(rows); d["cost"] = d.tin*PRICE_IN/1000 + d.tout*PRICE_OUT/1000
    return d

base = day({"a": 5000, "b": 3000, "c": 2000, "anon": 500}, {"a": 800, "b": 1500, "c": 600, "anon": 700}, {"a": 250, "b": 300, "c": 200, "anon": 250})
# spike: tenant c retry loop (4x requests), tenant b prompt template grew (3x input), anonymous traffic 10x
spike = day({"a": 5000, "b": 3000, "c": 8000, "anon": 5000}, {"a": 800, "b": 4500, "c": 600, "anon": 700}, {"a": 250, "b": 300, "c": 200, "anon": 250})

g0 = base.groupby("tenant").agg(n=("cost", "size"), tin=("tin", "mean"), cost=("cost", "sum"))
g1 = spike.groupby("tenant").agg(n=("cost", "size"), tin=("tin", "mean"), cost=("cost", "sum"))
cmp = g0.join(g1, lsuffix="_base", rsuffix="_spike")
cmp["delta_cost"] = cmp.cost_spike - cmp.cost_base
cmp["req_ratio"] = cmp.n_spike / cmp.n_base
cmp["input_tok_ratio"] = cmp.tin_spike / cmp.tin_base
print(cmp[["cost_base", "cost_spike", "delta_cost", "req_ratio", "input_tok_ratio"]].round(2).sort_values("delta_cost", ascending=False).to_string())
tot = cmp.delta_cost.sum()
print(f"total cost {cmp.cost_base.sum():.2f} -> {cmp.cost_spike.sum():.2f} (x{cmp.cost_spike.sum()/cmp.cost_base.sum():.2f})")
print("share of the increase:", {t: f"{v/tot:.0%}" for t, v in cmp.delta_cost.sort_values(ascending=False).items()})
# volume vs tokens-per-request split for the whole product (cost = requests * cost per request)
n0, n1 = len(base), len(spike); c0, c1 = base.cost.mean(), spike.cost.mean()
vol = (n1-n0)*c0; mix = n1*(c1-c0)
print(f"requests {n0} -> {n1}; cost/request {c0:.5f} -> {c1:.5f}; volume effect {vol:.2f}, per-request effect {mix:.2f}, sum {vol+mix:.2f}")
alt, joint = n0*(c1-c0), (n1-n0)*(c1-c0)
print(f"other ordering: per-request effect at the OLD volume {alt:.2f} + joint term {joint:.2f} = {alt+joint:.2f}")

Output:

        cost_base  cost_spike  delta_cost  req_ratio  input_tok_ratio
tenant                                                               
b         8101.83    17106.81     9004.98        1.0              3.0
c         2800.66    11195.51     8394.85        4.0              1.0
anon       848.91     8499.36     7650.45       10.0              1.0
a         9006.77     9002.74       -4.02        1.0              1.0
total cost 20758.16 -> 45804.42 (x2.21)
share of the increase: {'b': '36%', 'c': '34%', 'anon': '31%', 'a': '-0%'}
requests 10500 -> 21000; cost/request 1.97697 -> 2.18116; volume effect 20758.16, per-request effect 4288.09, sum 25046.25
other ordering: per-request effect at the OLD volume 2144.04 + joint term 2144.04 = 4288.09

Reading the output:

• Total cost rose from 20758.16 to 45804.42, a factor of 2.21. No single cause explains it: b contributes 36% of the increase (input tokens x3.0, request count unchanged), c 34% (requests x4.0, token sizes unchanged), anon 31% (requests x10.0, token sizes unchanged), and a is flat. The ratio columns are what separate the causes: a request ratio far from 1 with a token ratio near 1 is a volume problem, the reverse is a prompt-size problem.
• At product level, cost per request moved from 1.97697 to 2.18116 while requests doubled from 10,500 to 21,000. The volume effect (extra requests at the old per-request cost) is 20758.16 and the per-request effect is 4288.09; they add up to 25046.25, the increase in total cost. The split comes from the identity total cost = requests x cost per request, so the change is $n_1 c_1 - n_0 c_0 = (n_1 - n_0)\,c_0 + n_1\,(c_1 - c_0)$ where $n$ is the request count, $c$ the cost per request, and 0 and 1 mean baseline and spike. The first term prices the extra requests at the old cost per request. The second multiplies the new request count by the change in cost per request, so it also contains the joint effect of extra requests that are each more expensive. That ordering is a convention: pricing the per-request change at the old volume instead gives 2144.04, and the joint term (extra requests times the higher cost) is another 2144.04, together the same 4288.09. The volume effect equals the whole baseline cost, 20758.16, only because requests exactly doubled in this construction (10,500 to 21,000), so $n_1 - n_0 = n_0$; with any other ratio the two numbers differ. Here volume dominates, but the per-request effect is not noise, which is why a single "requests are up" explanation would have missed the prompt change.

Telemetry to inspect

Gateway request counts and status codes; tokens in and out per request; the prompt template and version id; model and fallback model used; retry and timeout counts; cache hit rate; per-key and per-IP request rates; anonymous versus authenticated share; and traces for a few of the most expensive requests.

Hypotheses and who to involve

Involve security early for an unauthenticated public endpoint, customer success when one tenant is the cause (they may be a customer with a bug), and finance if the invoice is affected.

Prevention and pitfalls

Per-key and per-IP rate limits, a hard cap on output tokens, budget alerts per tenant, and an alert on tokens per request by prompt version. Mistakes: reading only the total, assuming "more users" without checking the request ratio, and fixing symptoms (raising the budget) before containing the source.

Direct answer

Build two dashboards on the same trace data (a trace is the record of one request's path through the system, made of spans, one per step such as retrieval or a model call). The reliability dashboard answers "is it up, fast, and within budget?" for SRE and infrastructure owners, and pages people. The product dashboard answers "is it good, fair, safe and liked?" for product owners, and feeds weekly decisions rather than pages. Both drill from a headline number to a slice, to a single trace. Bias, privacy-leak risk and satisfaction live on the product side, behind access controls, because the drill-down ends in user conversations.

Reliability dashboard (SRE, infrastructure)

Terms in the table: a percentile such as p95 is the value that 95 percent of requests beat (p50 is the median, p99 means only 1 request in 100 is slower), which shows the slow tail that an average hides. Time to first token is how long a user waits until the first word of a streamed reply appears. Saturation is how full the system is relative to what it can handle, and queue depth is how many requests are waiting for a free worker (a growing queue means saturation). Rate-limit rejections are requests refused because a quota was exceeded, either ours or the model provider's.

Drill-down: headline chart, then split by version, region and endpoint, then the slowest or failed traces, then the span waterfall, a chart that lays a trace's spans out along a time axis so you can see which step took the time.

Alerting on an SLO (service level objective: a target such as 99.5% of requests succeed). The error budget is 1 minus the target, here 0.005. The burn rate is the observed bad fraction divided by the budget: 1.0 means spending the budget exactly as fast as allowed, and below 1.0 is healthy; there is no universal normal for a latency percentile, so judge p95 against your own latency limit and its history. A robust page requires a short window and a long window to both exceed the threshold, so a one-hour blip does not page and a real incident is not missed. The program below computes it on hand-set hourly data (20,000 requests per hour, an incident starting at hour 7).

Product dashboard (product owners)

Alerts: week-over-week drop in the quality pass rate beyond its interval, a rise in policy violations, a gap between groups that widens, and any leak-detector hit above zero for a restricted category (this one goes to the security owner, not the product channel).

Drill-down: metric, then slice, then a sample of examples with the rubric result, then the full conversation only if the viewer's role allows it.

Access controls on the sensitive panels

• Roles: SREs see operational traces with content redacted; product owners see aggregates and redacted examples; a small named reviewer group (privacy, trust and safety) can open raw conversations, with each open logged and a stated purpose.
• Redaction at ingestion: strip or hash identifiers and personal data before traces reach the dashboard store, and set retention limits.
• Small-cell suppression: do not show a group metric below a minimum count, because a rate over 5 users can identify them. The program uses a floor of 100 for illustration; choose the floor with your privacy owner.
• Leak panel is itself sensitive: store only counts and categories, not the leaked strings, in the shared view.

Worked example

The program has two parts. Part 1 computes the SLO burn rate over one-hour and six-hour windows and pages when both exceed 3 (a threshold chosen here as policy, not a standard). Part 2 builds a small synthetic conversation log, then shows the refusal rate with a Wilson interval (a 95 percent range for the true rate that stays sensible on small counts) per language group, suppressing any group below the floor. The last lines count personal-data pattern hits in four hand-written outputs; the regular expression is a minimal illustration of the kind of detector that feeds the leak panel, covering only a US-style phone fragment, a social-security-like number and a simple email shape, and production detectors must also handle other formats, obfuscation and memorised text.

import numpy as np
import pandas as pd

# ---- Part 1: SLO burn-rate alert for the reliability dashboard (hand-set hourly data) ----
SLO = 0.995                      # target: 99.5% of requests succeed within the latency limit
budget = 1 - SLO                 # allowed bad fraction = 0.005
THRESH = 3.0                     # page when the budget is spent 3x too fast (a policy choice)
req_per_hour = 20_000
bad_per_hour = np.array([60, 80, 70, 90, 75, 85, 70, 400, 450, 420, 430, 410])  # incident starts at hour 7
def burn(bad, window):
    frac = bad[-window:].sum() / (req_per_hour * window)
    return frac / budget         # 1.0 = spending the budget exactly as fast as allowed
for hour in (6, 7, 8, 9, 10, 11):
    b = bad_per_hour[:hour + 1]
    short, long_ = burn(b, 1), burn(b, min(6, len(b)))
    page = short > THRESH and long_ > THRESH   # both windows must exceed the threshold
    print(f"hour {hour:2d}: burn(1h)={short:5.2f} burn(6h)={long_:5.2f} page={page}")

# ---- Part 2: product dashboard slice with a privacy floor (synthetic conversation log) ----
rng = np.random.default_rng(0)
groups = {"lang_a": (900, 0.040, 0.78), "lang_b": (700, 0.045, 0.77), "lang_c": (40, 0.12, 0.60)}
rows = []
for g, (n, refuse_p, sat_p) in groups.items():
    for _ in range(n):
        rows.append((g, int(rng.random() < refuse_p), int(rng.random() < sat_p)))
d = pd.DataFrame(rows, columns=["group", "refused", "thumbs_up_or_resolved"])

K_MIN = 100                      # privacy floor: cells smaller than this are not shown to product viewers
def wilson(k, n, z=1.96):
    p = k / n
    c = (p + z * z / (2 * n)) / (1 + z * z / n)
    h = z * np.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / (1 + z * z / n)
    return c - h, c + h
lo2, hi2 = wilson(2, 10)
print(f"Wilson interval for 2 refusals in 10: [{lo2:.3f}, {hi2:.3f}]; the plain p +/- 1.96*sqrt(p(1-p)/n) lower end would be {0.2 - 1.96 * np.sqrt(0.2 * 0.8 / 10):.3f}")
out = []
for g, x in d.groupby("group"):
    n = len(x)
    if n < K_MIN:
        out.append((g, n, "suppressed (n < %d)" % K_MIN, "", ""))
        continue
    lo, hi = wilson(x.refused.sum(), n)
    out.append((g, n, f"{x.refused.mean():.3f} [{lo:.3f}, {hi:.3f}]", f"{x.thumbs_up_or_resolved.mean():.3f}", ""))
print(pd.DataFrame(out, columns=["group", "n", "refusal rate [95% Wilson]", "satisfaction", ""]).iloc[:, :4].to_string(index=False))

# PII-leak counter: simple pattern count on logged outputs (stub detector, illustrates a count not a guarantee)
import re
outputs = ["Call me at 555-0142 or mail a@b.co", "No personal data here", "ID 123-45-6789 appeared", "Plain answer"]
pat = re.compile(r"\b[0-9]{3}-[0-9]{4}\b|\b[0-9]{3}-[0-9]{2}-[0-9]{4}\b|[A-Za-z0-9._-]+@[A-Za-z0-9.-]+\.[a-z]{2,}")
hits = [bool(pat.search(o)) for o in outputs]
print("outputs with a pattern hit:", sum(hits), "of", len(outputs))

Output:

hour  6: burn(1h)= 0.70 burn(6h)= 0.78 page=False
hour  7: burn(1h)= 4.00 burn(6h)= 1.32 page=False
hour  8: burn(1h)= 4.50 burn(6h)= 1.95 page=False
hour  9: burn(1h)= 4.20 burn(6h)= 2.50 page=False
hour 10: burn(1h)= 4.30 burn(6h)= 3.09 page=True
hour 11: burn(1h)= 4.10 burn(6h)= 3.63 page=True
Wilson interval for 2 refusals in 10: [0.057, 0.510]; the plain p +/- 1.96*sqrt(p(1-p)/n) lower end would be -0.048
 group   n refusal rate [95% Wilson] satisfaction
lang_a 900      0.039 [0.028, 0.054]        0.774
lang_b 700      0.041 [0.029, 0.059]        0.780
lang_c  40      suppressed (n < 100)             
outputs with a pattern hit: 2 of 4

• At hour 7 an incident raised the bad rate to about 2% (400 of 20,000, four times the budget rate). The one-hour burn is already 4.00 at hour 7 and 4.50 at hour 8, but the six-hour window still holds quiet hours, so its burn is only 1.32 and then 1.95, and the page waits. By hand at hour 10: the last six hours (5 to 10) hold 85 + 70 + 400 + 450 + 420 + 430 = 1,855 bad requests out of 120,000, which is 1.55% and a burn of 1.55 / 0.5 = 3.09, above 3 together with the one-hour burn of 4.30. So the first page is at hour 10, the fourth bad hour (7, 8, 9, 10), and it stays on at hour 11. This delay is the price of avoiding false pages; a lower long-window threshold pages earlier and more often.
• The Wilson line shows why that interval is used: for 2 refusals in 10 the plain margin would put the lower end below zero, while the Wilson interval stays between 0 and 1 and is centred slightly toward the middle. It widens as n shrinks, which is why lang_c would be too uncertain to show anyway.
• lang_a and lang_b have refusal rates of 0.039 and 0.041 with overlapping intervals: no evidence of a gap. lang_c, with 40 conversations, is suppressed. Suppression protects individuals but can hide the group at highest risk, so give the restricted reviewer role the pooled figure over a longer period for that group.
• Two of the four sample outputs matched a personal-data pattern, which is a number of the sample, not a rate.

Mistakes to avoid

• One dashboard for everyone: product owners get paged on latency, SREs get raw conversations.
• Averages only. Always show a percentile or a distribution for latency and an interval for rates.
• Bias metrics without group sizes or intervals.
• Satisfaction shown without the number of responses behind it.

Direct answer

Treat each request as a trace (one record of everything that happened to it, made of timed steps called spans) and split the pipeline into a cheap, always-on metrics path and an expensive, selective payload path. Emit a small metadata record for every request, redact and enrich it near the edge, tail-sample the full payloads (decide after the request finishes: keep everything interesting and a small slice of the rest; head sampling, by contrast, decides at the start with a flat rate), keep recent data in a fast store and older data in cheap object storage, and alert from the metrics path in real time. Privacy is a property of the pipeline's shape: redact before storage, encrypt per tenant, and give each tenant its own isolation boundary.

The pipeline, in the order data flows

1. Instrumentation and redaction at the edge. The inference gateway writes one structured event per request (request id, tenant, model version, token counts, latency, status, a prompt-template id) and a separate payload record (prompt and completion text). PII redaction runs inside the gateway process (see the privacy section), so the payload record is already redacted before it leaves the gateway; nothing raw reaches disk anywhere downstream. Instrument with a vendor-neutral tracing standard (for example OpenTelemetry, an open specification and set of libraries for emitting traces, metrics and logs) so the backend can change.
2. Ingestion. Events and redacted payload records go to a durable queue (a message log stored on disk, so events wait safely until consumers read them), so a slow downstream store never slows inference.
3. Enrichment. A stream processor (a service that reads events as they arrive, transforms them and writes them onward) adds derived fields: cost from token counts and a rate table, tenant tier, region, the prompt-template version, and cheap classifier flags (safety trigger, refusal, language). Enrichment stays off the request path.
4. Processing. Two branches: (a) aggregate to metrics (counters, numbers that only go up such as requests served, and histograms, counts of requests per latency bucket from which percentiles are read, keyed by tenant, model, route, status) for dashboards and alerts; (b) tail-sample the payload path.
5. Storage. Hot store: recent metrics and sampled traces, queryable in seconds, retained for days. Cold store: columnar files (formats such as Parquet that store each field together, so a query reads only the columns it needs) in object storage, partitioned by date and tenant, retained for months at far lower cost per gigabyte, queried on demand.
6. Alerting. Alerts read the metrics stream, not the trace store, so they stay fast and cheap even if trace search is slow.

Sizing at the stated volume

The program uses stated, assumed sizes (a 1,200-byte metadata record and a 6,000-byte payload per request) and a peak factor of 4, all assumptions rather than measurements. It then simulates tail sampling on 200,000 synthetic traces: always keep errors, flagged responses and requests slower than 3 seconds; keep 1% of the rest by hashing the trace id, so the decision is deterministic and the same trace is kept or dropped in every service that sees it. It then estimates a population rate from the kept traces, to show why reweighting is needed, and ends with a minimal redaction regex and a per-tenant keyed pseudonym (an HMAC: a hash computed with a secret key, so only a holder of the key can recompute it; the privacy section explains why).

import hashlib, hmac, random, re

# --- capacity arithmetic (assumed sizes are stated here, not measured) ---
SPAN_BYTES_META = 1_200       # assumed: metadata-only span record per request
PAYLOAD_BYTES = 6_000         # assumed: prompt + completion text per request
for rpd in [1_000_000, 50_000_000]:
    avg = rpd / 86_400
    print(f"{rpd:>11,} req/day: avg {avg:8.1f} req/s, peak (x4 assumed) {4*avg:8.1f} req/s, "
          f"metadata {rpd*SPAN_BYTES_META/1e9:6.1f} GB/day, full payload {rpd*(SPAN_BYTES_META+PAYLOAD_BYTES)/1e9:7.1f} GB/day")

# --- tail sampling with a deterministic per-trace decision ---
random.seed(0)
def make_trace(i):
    return {"trace_id": f"t{i:07d}", "tenant": f"tenant{random.randint(1,5)}",
            "error": random.random() < 0.01, "latency_ms": random.lognormvariate(6.5, 0.5),
            "flagged": random.random() < 0.005}
traces = [make_trace(i) for i in range(200_000)]
SLOW_MS, BASE_RATE = 3000, 0.01
def keep(t):
    if t["error"] or t["flagged"] or t["latency_ms"] > SLOW_MS:
        return "always"
    h = int(hashlib.sha256(t["trace_id"].encode()).hexdigest()[:8], 16) / 0xFFFFFFFF
    return "base" if h < BASE_RATE else None
decisions = [keep(t) for t in traces]
n_always = decisions.count("always"); n_base = decisions.count("base")
is_interesting = [t["error"] or t["flagged"] or t["latency_ms"] > SLOW_MS for t in traces]
interesting = sum(is_interesting)
kept_interesting = sum(1 for flag, d in zip(is_interesting, decisions) if flag and d)
print(f"traces {len(traces)}: always-kept {n_always}, base-sampled {n_base}, kept fraction {(n_always+n_base)/len(traces):.4f}")
print(f"interesting traces {interesting}, of which kept {kept_interesting}")

# --- reweighting: estimate the share of ALL traces slower than 1.5 s from the kept traces alone ---
SLOW_Q = 1500
kept = [(t, d) for t, d in zip(traces, decisions) if d]
weight = {"always": 1.0, "base": 1 / BASE_RATE}      # weight = 1 / probability that the trace was kept
true_share = sum(t["latency_ms"] > SLOW_Q for t in traces) / len(traces)
naive = sum(t["latency_ms"] > SLOW_Q for t, d in kept) / len(kept)
weighted = sum(weight[d] for t, d in kept if t["latency_ms"] > SLOW_Q) / sum(weight[d] for t, d in kept)
print(f"share slower than {SLOW_Q} ms: true {true_share:.4f}, naive from kept {naive:.4f}, reweighted {weighted:.4f}")

# --- minimal redaction + tenant-keyed pseudonym (illustration, not a complete PII detector) ---
EMAIL = re.compile(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}")
PHONE = re.compile(r"\+?[0-9][0-9 ()-]{7,}[0-9]")
def redact(text):
    return PHONE.sub("<PHONE>", EMAIL.sub("<EMAIL>", text))
def pseudonym(tenant_key: bytes, user_id: str):
    return hmac.new(tenant_key, user_id.encode(), hashlib.sha256).hexdigest()[:12]
print(redact("Mail bob@example.com or call +1 415 555 0100 about order 123"))
ka, kb = b"key-tenant-a", b"key-tenant-b"
print("same user id, tenant A:", pseudonym(ka, "u42"), "| again:", pseudonym(ka, "u42"), "| tenant B:", pseudonym(kb, "u42"))

Output:

  1,000,000 req/day: avg     11.6 req/s, peak (x4 assumed)     46.3 req/s, metadata    1.2 GB/day, full payload     7.2 GB/day
 50,000,000 req/day: avg    578.7 req/s, peak (x4 assumed)   2314.8 req/s, metadata   60.0 GB/day, full payload   360.0 GB/day
traces 200000: always-kept 3255, base-sampled 2050, kept fraction 0.0265
interesting traces 3255, of which kept 3255
share slower than 1500 ms: true 0.0515, naive from kept 0.0939, reweighted 0.0476
Mail <EMAIL> or call <PHONE> about order 123
same user id, tenant A: 1b9343449285 | again: 1b9343449285 | tenant B: 728e782b8047

Reading the output:

• 1 million requests a day is about 11.6 requests per second on average and 46.3 at the assumed peak: trivial. 50 million a day is about 578.7 per second on average and 2,314.8 at peak, and storing every payload would be 360.0 GB per day against 60.0 GB per day for metadata alone. That 6x gap is why payloads are sampled and metadata is not.
• Tail sampling kept all 3,255 interesting traces (errors, flagged, slow) plus 2,050 base-sampled traces: 2.65% of all traffic, so storage falls by roughly 38x while every failure stays debuggable. Head sampling (deciding at the start with a flat rate) would have thrown away 99% of the rare failures.
• Counters and histograms are never sampled, so rates stay exact. Sampled traces are reweighted by the inverse of their keep probability whenever you estimate a population rate from them: a trace kept with probability 1% stands for 100 traces, so it counts 100 times, while an always-kept trace counts once. The program estimates the share of traces slower than 1.5 s from the kept traces alone. In this run the true share is 5.15%; the kept set read naively says 9.39%, nearly double, because slow traces are over-represented in it; the reweighted estimate is 4.76%, close to the truth, and the remaining gap is sampling noise from the 1%-sampled traces.
• Keep metric label cardinality bounded. Cardinality is the number of distinct values a label takes, and every distinct combination of label values is its own stored time series. Tenant, model and route take a modest number of values and are fine; a request id label would create a new series for every request, so request id and user id belong in traces, not metric labels.

Privacy and tenant isolation

• Redact at the edge, before storage. The email and phone patterns in the program above are a minimal illustration that catches simple email and phone shapes only. A production redactor also needs a named-entity model for names and addresses, checksum validation for card numbers, a leak test set, and a review of what it misses.
• Pseudonymize with a per-tenant key (an HMAC, a keyed hash). The same user maps to the same token inside a tenant (so you can follow a session) and to a different token in another tenant, which stops cross-tenant joining. Keep the keys in a secrets manager.
• Isolation: tenant id on every record, per-tenant encryption keys, partitions and access policies per tenant, and row-level filters (a rule the query layer adds to every query, such as only rows where the tenant is the caller's tenant) enforced by the query layer, not by the dashboard. Large or regulated tenants can get a dedicated bucket.
• Retention: short for payloads (days to weeks), longer for metadata, with a deletion path keyed by the pseudonym.
• Access: payload access is role-gated and audited; most engineers see only metrics and redacted traces.

Synthetic checks and the three signals

Logs answer "what happened to this request", metrics answer "how is the fleet doing", traces answer "where did the time or the error come from". Add synthetic checks: a scheduled set of fixed prompts against the production endpoint, with a quality and latency assertion, so you notice an outage or a quiet model regression even at 3 a.m. when real traffic is low. Alert on page-worthy symptoms (error rate, p99 latency (the latency that 99% of requests beat), token cost per request, refusal rate jump, synthetic-check failure), each with an owner and a runbook link.

Trade-offs and what would flip the design

• Sampling versus completeness: a lower base rate saves money but reduces the data available to build evaluation sets; raise it for new releases and lower it when stable.
• Real-time versus batch: only alerting needs seconds; analysis can wait for minutes or hours, so cold storage queries are fine.
• Build versus buy: a managed tracing backend is faster to start; self-hosting wins at very high volume or strict data residency.
• Common mistakes: storing raw prompts "temporarily" in a log aggregator that has broad access; using user id as a metric label; sampling away the errors; alerting from the sampled store so rates are biased.

Direct answer

All three describe the same request from different angles. A log is a timestamped record of one event with its details, so you can ask "what exactly happened to this request". A trace is the tree of timed steps (spans) one request passed through, so you can ask "where did the time go". A metric is a number aggregated over many requests per time window, so you can ask "is the service healthy right now". Each has a different cost profile, which is why each gets different sampling and retention.

One request, three views

Request flow in this feature: retrieve documents, build the prompt, generate with the model, filter the output, format the reply. The values below are stub data for one request, not measurements of a real system.

• Log: one structured record at the end with route, prompt template version, model version, token counts, finish reason (why generation stopped, for example it ended naturally or hit the length limit), filter verdict, a hashed user id and status. The record in the program reads: trace_id t-123, route support_chat, model m-2025-a, template support_v7, 1,450 input tokens, 210 output tokens, finish_reason stop, filter_verdict pass, status ok. The trace_id field is the join key: searching the trace store for t-123 returns the span tree for the same request. The log answers "which template version produced this reply, and did the filter pass it".
• Trace: a parent span for the whole request and a child span per step, each with a start time and duration; the generation span carries input tokens, output tokens and time to first token (how long the user waits before the first word of the reply appears). It answers "why was this one slow".
• Metrics: latency, time to first token, tokens in and out and decode time per token (the time spent generating each output token after the first), plus error counts, labelled by route and model version (a label is a tag attached to a metric so that you can split it, such as route=support_chat). They answer "is p95 rising across all users", where p95 is the latency that 95% of requests beat, so the slowest 5% take longer.

The second part of the program simulates a day of 1,000,000 requests (assumed volume, 0.5% error rate, lognormal durations: a right-skewed shape in which most requests take a typical time and a long tail takes much longer) and compares keeping every trace with tail-based sampling. Sizes are of the stub records, so real logs that carry prompt and reply text will be far larger. Libraries: numpy 2.1.3.

import os
os.environ["OMP_NUM_THREADS"] = "1"
import json
import numpy as np

# One inference request, as a log record, a trace and metric updates (all values are stub data).
spans = [
 {"name": "request",          "parent": None,       "start_ms": 0,    "dur_ms": 2410},
 {"name": "retrieve",         "parent": "request",  "start_ms": 12,   "dur_ms": 180},
 {"name": "build_prompt",     "parent": "request",  "start_ms": 195,  "dur_ms": 6},
 {"name": "llm_generate",     "parent": "request",  "start_ms": 205,  "dur_ms": 2050,
  "attrs": {"input_tokens": 1450, "output_tokens": 210, "ttft_ms": 640, "model_version": "m-2025-a"}},
 {"name": "output_filter",    "parent": "request",  "start_ms": 2260, "dur_ms": 70},
 {"name": "format_reply",     "parent": "request",  "start_ms": 2335, "dur_ms": 75},
]
log_record = {"ts": "2025-01-01T12:00:00Z", "trace_id": "t-123", "route": "support_chat",
              "model_version": "m-2025-a", "prompt_template": "support_v7", "input_tokens": 1450,
              "output_tokens": 210, "finish_reason": "stop", "filter_verdict": "pass",
              "user_id_hash": "u-8f3a", "status": "ok"}
llm = spans[3]["attrs"]
print("trace: children sum to", sum(s["dur_ms"] for s in spans[1:]), "ms of", spans[0]["dur_ms"], "ms total")
slowest = max(spans[1:], key=lambda s: s["dur_ms"])
print("slowest child span:", slowest["name"], f"({slowest['dur_ms']/spans[0]['dur_ms']:.0%} of request)")
print("metric updates: latency_ms=2410  ttft_ms=%d  tokens_in=%d  tokens_out=%d  decode_ms_per_token=%.1f"
      % (llm["ttft_ms"], llm["input_tokens"], llm["output_tokens"], (spans[3]["dur_ms"] - llm["ttft_ms"]) / llm["output_tokens"]))
log_bytes = len(json.dumps(log_record))
span_bytes = sum(len(json.dumps(s)) for s in spans)
print(f"serialised sizes: log record {log_bytes} B, trace {span_bytes} B ({len(spans)} spans)")

# Tail-based sampling over a synthetic day (assumed volume and error rate).
rng = np.random.default_rng(0)
N = 1_000_000
dur = rng.lognormal(np.log(2.0), 0.5, N)
err = rng.random(N) < 0.005
slow = dur > 6.0
keep_base = rng.random(N) < 0.02
keep = err | slow | keep_base
print(f"\nrequests {N:,}: errors {err.sum():,}, over 6 s {slow.sum():,}")
print(f"traces kept {keep.sum():,} ({keep.mean():.1%}); errors kept {keep[err].mean():.0%}; slow kept {keep[slow].mean():.0%}; "
      f"ordinary kept {keep[~err & ~slow].mean():.1%}")
full_gb = N * span_bytes / 1e9
kept_gb = keep.sum() * span_bytes / 1e9
print(f"trace storage per day: keep-all {full_gb:.2f} GB, tail-sampled {kept_gb:.3f} GB")

# Retention arithmetic (decimal GB)
print(f"logs (every request, redacted) {N*log_bytes/1e9:.2f} GB/day x 30 days = {N*log_bytes*30/1e9:.1f} GB")
print(f"traces 14 days tail-sampled = {kept_gb*14:.2f} GB")
series = 5 * 4 * 3   # 5 metrics x 4 routes x 3 model versions
pts = 365 * 24 * 60
print(f"metrics: {series} series x {pts:,} one-minute points x 16 B = {series*pts*16/1e6:.0f} MB per year")
kept_pts = 14 * 24 * 60 + 351 * 24   # one-minute points for 14 days, then one point per hour for the other 351 days
print(f"metrics with that downsampling: {kept_pts:,} points per series = {series*kept_pts*16/1e6:.1f} MB per year")

Output:

trace: children sum to 2381 ms of 2410 ms total
slowest child span: llm_generate (85% of request)
metric updates: latency_ms=2410  ttft_ms=640  tokens_in=1450  tokens_out=210  decode_ms_per_token=6.7
serialised sizes: log record 275 B, trace 546 B (6 spans)

requests 1,000,000: errors 4,950, over 6 s 14,238
traces kept 38,588 (3.9%); errors kept 100%; slow kept 100%; ordinary kept 2.0%
trace storage per day: keep-all 0.55 GB, tail-sampled 0.021 GB
logs (every request, redacted) 0.28 GB/day x 30 days = 8.2 GB
traces 14 days tail-sampled = 0.29 GB
metrics: 60 series x 525,600 one-minute points x 16 B = 505 MB per year
metrics with that downsampling: 28,584 points per series = 27.4 MB per year

Reading the results

1. The trace finds the slow step. The five child spans sum to 2381 ms of the 2410 ms total, and llm_generate is 85% of the request, with 6.7 ms per generated token after the first. A latency regression is therefore a model-side question here, not retrieval.
2. Metrics stay cheap if labels stay small. 60 series (5 metrics times 4 routes times 3 model versions) at one-minute resolution take about 505 MB per year if every point were kept at one-minute resolution, at an assumed 16 bytes a point. The retention table below keeps one-minute points only for 14 days and then downsamples (older points are averaged into coarser ones, here one point per hour), which cuts the same series to about 27.4 MB per year (28,584 points per series); the 505 MB figure is the upper bound, not the plan. Putting user id or prompt text in a label would multiply the series count by the number of distinct values (cardinality), so those belong in logs and traces.
3. Sampling. Keeping everything costs 0.55 GB of traces per day for the stub. Tail-based sampling decides after the request finishes: keep every error, every request over 6 s, and 2% of the rest. That kept 38,588 of 1,000,000 traces (3.9%), including 100% of errors and 100% of slow requests, and 2.0% of ordinary ones, for 0.021 GB a day. Head-based sampling decides at the start and cannot know that a request will be slow or fail; tail-based sampling needs to hold a trace's spans until it ends.

Proposed sampling and retention

LLM-specific points

• Record the versions: model, prompt template, retrieval index, and filter. Without them a quality change cannot be tied to a deploy.
• Record token counts and finish reason (a reply cut off at the length limit is a failure that has no error code).
• Keep user content out of metrics and treat logged text as sensitive data.
• Quality scores from sampled evaluation belong beside these signals, since a request can be fast, error-free and wrong.

Common mistakes

• Using only metrics and then being unable to explain a single bad request.
• Sampling traces at random so the rare slow ones are missing.
• Logging full prompts forever.
• High-cardinality metric labels.

Direct answer

Observability means you can answer "why did the system do that?" from the data it emits, without shipping new code. For a conventional service, the questions are about health: did the request succeed, how long did it take, what resource was used. For an LLM feature there is a second question that status codes cannot answer: was the output any good? The model can return HTTP 200 with a fluent, confident, wrong answer, so observability for an LLM feature adds the content of each step (prompt, retrieved context, output) and quality signals computed on it, linked to the usual latency and error data.

How it differs from a conventional service

The signals to capture

A trace is the full record of one request; each step inside it is a span (one timed operation with attributes). Capture these per request:

1. Latency per span and end to end, including time to first token (how long until the first word of the reply appears) for streamed replies (replies shown word by word as the model writes them).
2. Errors and retries, including provider rate limits and timeouts.
3. Token counts in and out, and derived cost.
4. Inputs and outputs: the final prompt after template filling, retrieved passages with their IDs and scores, tool calls and results, the reply.
5. Versions: model identifier, prompt template version, index version, sampling settings.
6. Quality signals: groundedness (does the reply stay within the retrieved context), safety flags, format validity, plus user feedback (thumbs, edits, retries, abandonment).
7. User and session identifiers, to find all requests affected by a problem.

A worked example

The program below uses one hand-written trace (timings and token counts are invented inputs) for a retrieval-augmented feature (one that first looks up relevant documents and puts them in the prompt), and six hand-written replies. The groundedness check is a deterministic stand-in: a real system would put a model-based judge or an entailment model (a model that decides whether a passage supports a claim) there. The stand-in only tests that every number in the reply appears in the retrieved context.

How to read the code. Part 1: each span is a tuple (name, start_ms, end_ms, tokens_in, tokens_out), so ("retrieve", 12, 95, 0, 0) means retrieval ran from millisecond 12 to 95 and used no model tokens. The guardrail spans are the automatic input and output checks, and rerank re-orders the retrieved passages with a more careful score. The program prints each span's duration, its share of the 940 ms total and its tokens, then a cost from the token counts. Part 2: each request is a (question, reply, retrieved context) triple, and stub_check pulls every number out of the reply and the context and compares the two sets.

import re
# One request through a retrieval-augmented chat feature, hand-written spans.
# Each tuple is (name, start_ms, end_ms, tokens_in, tokens_out).
spans = [
    ("guardrail_in",   0,  12, 0,   0),
    ("retrieve",      12,  95, 0,   0),
    ("rerank",        95, 160, 0,   0),
    ("llm_call",     160, 910, 1450, 180),
    ("guardrail_out",910, 940, 0,   0),
]
total_ms = spans[-1][2] - spans[0][1]
tin = sum(s[3] for s in spans); tout = sum(s[4] for s in spans)
print("end-to-end ms:", total_ms)
for name, a, b, ti, to in spans:
    print(f"{name:14s} {b-a:4d} ms  {100*(b-a)/total_ms:5.1f}%  tokens in/out {ti}/{to}")
UNIT_IN, UNIT_OUT = 1.0, 3.0   # assumed cost units per 1000 tokens, for illustration only
print("cost units:", round(tin/1000*UNIT_IN + tout/1000*UNIT_OUT, 3))

# Six requests: every one returns HTTP 200. A quality check on the reply tells them apart.
reqs = [
 ("What was Q2 revenue?", "Revenue in Q2 was 4.2 million.", "Q2 revenue: 4.2 million. Q1 revenue: 3.9 million."),
 ("What was Q1 revenue?", "Revenue in Q1 was 3.9 million.", "Q2 revenue: 4.2 million. Q1 revenue: 3.9 million."),
 ("Q2 growth?",           "Growth was 12 percent.",         "Q2 revenue: 4.2 million. Q1 revenue: 3.9 million."),
 ("Q2 revenue?",          "I could not find that.",         "No relevant documents."),
 ("Q2 revenue?",          "Revenue in Q2 was 4.2 million.", "Q1 headcount: 40."),
 ("Q1 revenue?",          "Revenue in Q1 was 3.9 million.", "Q2 revenue: 4.2 million. Q1 revenue: 3.9 million."),
]
def numbers(t): return set(re.findall(r"\d+(?:\.\d+)?", t))
def stub_check(answer, context):
    """Stand-in for a real groundedness judge: every number in the reply must appear in the context."""
    nums = numbers(answer)
    if not nums: return "no_claim"
    return "grounded" if nums <= numbers(context) else "ungrounded"
print()
for q, a, c in reqs:
    print(f"http=200  {stub_check(a, c):10s} {q}")

Output:

end-to-end ms: 940
guardrail_in     12 ms    1.3%  tokens in/out 0/0
retrieve         83 ms    8.8%  tokens in/out 0/0
rerank           65 ms    6.9%  tokens in/out 0/0
llm_call        750 ms   79.8%  tokens in/out 1450/180
guardrail_out    30 ms    3.2%  tokens in/out 0/0
cost units: 1.99

http=200  grounded   What was Q2 revenue?
http=200  grounded   What was Q1 revenue?
http=200  ungrounded Q2 growth?
http=200  no_claim   Q2 revenue?
http=200  ungrounded Q2 revenue?
http=200  grounded   Q1 revenue?

What this shows:

• The trace pins the time: the model call is 750 of 940 ms, so tuning retrieval would barely move the total. A conventional service dashboard would show this too.
• The cost line is in invented relative units (output tokens assumed three times the price of input tokens). The point is that cost comes from the token columns, which a conventional dashboard has no column for.
• All six requests return status 200. The quality check separates them: the "Q2 growth" reply quotes 12 percent that appears nowhere in the context (ungrounded), the fifth reply states Q2 revenue from a context holding only headcount (also ungrounded), and the fourth declines honestly. The stand-in is crude: it counts the digit in a label such as "Q2" as a number, so reply 5's numbers are 2 and 4.2 while its context holds 1 and 40, and reply 1 passes only because the context happens to contain "Q2" and "Q1" too.
• The program does not join anything; the join is done by the person reading the logged trace. Because the trace also stores the retrieved passages, an ungrounded reply can be sorted into two causes: the context held the facts but the reply invented others (a model fault, as with the growth figure), or the context never held the facts (a retrieval fault, as in reply 5, where the passage was about headcount). Without the retrieved text in the trace you could not tell these apart.

Limits and pitfalls

• Quality signals are themselves measurements with error; a stand-in or judge needs calibration against human labels.
• Logging full prompts is a privacy risk; redact, restrict access, and set retention.
• Without version tags on every trace, you cannot tell a regression caused by a prompt edit from one caused by a data change.
• Dashboards of averages hide tails and slices; keep percentiles and group by user segment and topic.