Large Language Model Observability and Evaluation Questions

Covers the end to end product and technical considerations for monitoring, evaluating, and troubleshooting large language model systems. Topics include what observability means for model driven features, which signals to capture such as input provenance, token usage, latency, error modes, and outcome quality, and how to design instrumentation and data contracts that ensure consistent and auditable telemetry. It includes evaluation approaches and metrics such as relevance, accuracy, hallucination rate, calibration, and cost, and the trade offs between human labeling, automated metrics, and model driven judges. Product design aspects cover dashboards, alerts, logging, tracing, debugging interfaces, and developer workflows that make investigation and root cause analysis efficient. Finally this topic addresses operational concerns for an observability platform including storage and cost trade offs, scaling telemetry pipelines, privacy and compliance constraints, and how evaluation and observability feed back into model improvement cycles.

EasyTechnical

27 practiced

Define 'hallucination' in the context of LLMs and provide three lightweight automated signals that could approximate hallucination rate without full human labels. For each automated signal, explain what types of hallucinations it might catch and its limitations.

MediumSystem Design

21 practiced

Design an A/B testing and canary rollout strategy to evaluate a new model version claiming 20% lower hallucination rate but 10% higher latency. Specify target metrics, required sample sizes, traffic allocation plan, monitoring guardrails, fairness checks, cost monitoring, and rollback criteria you would use as TPM.

Sample Answer

**Clarify goals & assumptions**- Goal: verify claim of 20% relative reduction in hallucination rate vs baseline, while assessing 10% higher latency and cost impact.- Assumptions to state up front: baseline hallucination rate (e.g., 10%), baseline mean latency (e.g., 200 ms), significance α=0.05, power=0.8.

**Target metrics**- Primary: hallucination rate (binary per response), mean latency (ms).- Secondary: 95th percentile latency, error rate (5xx), user satisfaction (NPS/QA), cost per request.- Fairness: hallucination & latency by user segment (language, region, device, demographic proxies).

**Required sample sizes (approx)**- For proportions (10% → 8%): with α=0.05, power=0.8, roughly 8k–12k requests per arm. (If baseline differs, recalc.)- For latency mean difference (~10% on 200ms = 20ms): ~1k–3k samples per arm to detect with t-test.- Plan to collect at least max(samples) per arm; stratify to ensure per-segment power for fairness checks.

**Traffic allocation / rollout plan**- Canary phase 0: internal validation + synthetic tests.- Canary 1 (smoke): 1% traffic, 24–48 hrs, compare live metrics.- Canary 2 (small): 5% traffic, 48–72 hrs, run statistical tests.- Canary 3 (ramp): 25% traffic, 72 hrs, include stress windows.- Full rollout: 100% after passing metrics for 7 days.- Use randomized bucketing with sticky user assignment for session consistency.

**Monitoring & guardrails**- Real-time dashboards: hallucination rate, mean & p95 latency, error rate, throughput, cost.- Statistical tests: daily A/B significance (two-sided for hallucination; non-inferiority for latency if required).- Automated alerts: - Hallucination increases >1% absolute or fails to meet ≥20% relative reduction. - Mean latency increase >15% (buffer above claimed 10%). - Error rate spike >2x baseline or absolute >1%. - Per-segment regressions exceeding thresholds (see fairness).- Canary health checks: circuit-breaker to shift traffic away automatically on alert.

**Fairness checks**- Slice metrics by region, language, device, user cohort.- Require no segment to have >10% relative increase in hallucination or latency compared to baseline; apply stricter thresholds for protected groups.- Run manual review of flagged examples from each segment.

**Cost monitoring**- Track cost-per-request and projected monthly spend delta.- Alert when marginal cost increase > budget threshold (e.g., >20% cost uplift) or if cost growth exceeds forecast.- Provide cost/benefit dashboard: expected reduction in downstream costs (e.g., moderation, user support) vs. model CPU cost.

**Rollback criteria**- Immediate rollback if: - Hallucination rate increases vs baseline (absolute) or fails to meet claimed reduction within confidence interval. - Error rate > threshold or spike sustained >15 minutes. - Latency increase >15% causing SLA violations or user-impacted KPIs drop (engagement). - Any segment shows fairness regression beyond preset bounds. - Cost exceeds budget alert and value metrics not improved.- Post-rollback: capture traces, preserve sample set for root-cause, and run post-mortem with engineering and product.

Rationale: staged ramp + statistical power ensures signal detection for rare events (hallucinations) while limiting user impact. Guardrails combine automated controls and human review for fairness and cost trade-offs — appropriate for TPM owning product risk, metrics, and cross-functional coordination.

HardTechnical

30 practiced

You must build a business case to convince executives to invest in a unified LLM observability platform. As TPM, outline the key value levers (incident reduction, cost savings, faster feature launches, customer retention), propose metrics to quantify ROI, and present a prioritized MVP feature list that delivers measurable impact within 3 months.

Sample Answer

**Executive summary (one line)** Investing in a unified LLM observability platform reduces incidents, cuts compute & latency costs, accelerates feature launches, and improves customer retention — producing measurable ROI within 3–9 months.

**Key value levers & why they matter**- Incident reduction: fewer outages, faster MTTR → lower downtime costs, improved SLAs.- Cost savings: detect runaway prompts/models, optimize GPU utilization.- Faster feature launches: confidence to deploy model changes with automated validation.- Customer retention: higher model quality → fewer complaints, higher NPS/renewal.

**Metrics to quantify ROI**- Incidents/month and MTTR (target: -50% incidents, -60% MTTR)- Compute spend on inference/training per model (target: -20% cost)- Time-to-production for model changes (target: -30% cycle)- Churn rate / NPS attributable to model quality (target: +5 NPS, -2% churn)- Business impact formula: Annualized savings = (reduction in downtime hours * $/hr) + compute savings + incremental revenue from retention.

**Prioritized 3-month MVP (impact-first)**1. Core telemetry + anomaly detection (Week 1–6) — logs, latencies, token counts, distribution shifts; alerting for drift and cost spikes. (High impact: incident reduction + cost)2. Per-request lineage & sampling + automated incident playbooks (Week 3–9) — trace inputs→model→config; attach runbooks to alerts for <30 min MTTR. (High impact: MTTR and faster launches)3. Cost dashboard & budget guards (Week 6–12) — per-model spend, throttles, autoscaling suggestions. (Medium impact: direct cost savings)4. Canary rollout + automatic rollback hooks (Week 8–12) — safe deployment gating using observability signals. (High impact: faster, safer launches)

**Success criteria (90 days)**- Detectable reduction in high-severity incidents by 40%- First month compute savings visible (≥10%)- One successful canary rollback prevented a regression- Dashboard adoption: >3 engineering teams using weekly

**Ask**Budget for 3 FTEs (1 PM, 1 SRE/observability engineer, 1 frontend/backend dev), instrumentation credits, and executive support for data access — projected payback <9 months using above metrics.

HardTechnical

27 practiced

A deployed assistant generated content that led to a legal claim of defamation for a customer. As the TPM owning observability and evaluation, outline the immediate incident response (legal, communications, containment), short-term forensic steps, evidence to preserve for regulators, and long-term product and policy changes you would propose to reduce future legal risk.

Sample Answer

**Immediate incident response — legal, communications, containment**1. Legal: Pause related feature via kill-switch; notify in-house counsel and external counsel specializing in defamation/compliance; preserve privilege by using legal-approved channels.2. Communications: Draft coordinated internal and external lines — notify affected customer privately, prepare holding statement for public/press, inform CS and account teams with Q&A and escalation paths.3. Containment: Disable the assistant’s publishing/output to customers while keeping read-only logs; quarantine the model/config and recent prompt templates.

**Short-term forensic steps**- Capture immutable snapshots: model versions, weights/config, inference logs, prompt history, API keys, user session data, config flags, deployment manifests, and container images.- Reproduce inputs in isolated sandbox to trace generation path.- Collect telemetry: upstream data sources, retrieval logs, reranking, grounding/citation traces.- Time-ordered audit trail with checksums and chain-of-custody.

**Evidence to preserve for regulators**- Immutable logs (signed/TLS), reproduction artifacts, retention policy records, decision trees for content filtering, human review actions, escalation tickets, legal notifications, and remediation timelines.

**Long-term product & policy changes**- Technical: add real-time safety filters, provenance metadata, stronger hallucination detection, mandatory human-in-loop for high-risk domains, rate-limited outputs, and enhanced observability dashboards with alerted SLA breaches.- Process: formal risk classification for use-cases, expanded review for high-legal-risk customers, contractual indemnities and clearer ToS.- Governance: cross-functional legal/ethics review gate for releases, incident playbooks, regular tabletop exercises, retention & audit policies, and KPIs (false-positive/negative rates, time-to-contain).I would own prioritization, define acceptance criteria for mitigations, and roadmap these changes with engineering, legal, and customer teams.

EasyTechnical

45 practiced

Explain how you would instrument token usage and cost attribution for a multi-tenant LLM API. Specify what tags/labels to include per request (for example: customer_id, endpoint, model_version, response_tokens), how to handle streaming responses, and how to present per-customer cost insights without leaking other customers' data.

Unlock Full Question Bank

Get access to hundreds of Large Language Model Observability and Evaluation interview questions and detailed answers.

Join thousands of developers preparing for their dream job.