End-to-End ML System Design Questions

End-to-end design of machine learning systems, covering data collection and validation, feature engineering and feature stores, model training and evaluation, deployment and serving architectures, monitoring and incident management, retraining pipelines, data governance, scalability, security, and MLOps practices.

MediumTechnical

31 practiced

Design an automated retraining trigger policy for a production model. The system should consider data drift, model performance degradation, minimum new label count, and calendar-based retraining. Describe detection logic, guardrails to avoid unnecessary retrains, validation steps to vet retrained models, and how to schedule/orchestrate retraining jobs safely.

Sample Answer

Detection logic:- Continuous monitoring pipelines compute metrics per time window (daily/weekly): model performance (AUC/accuracy, calibration), data-distribution stats (feature means, histograms), and label arrival count.- Drift signals: - Feature drift: Population Stability Index (PSI) > 0.1 for any high-impact feature OR Kolmogorov–Smirnov p-value < 0.01. - Label drift: change in label rate > 20% vs rolling baseline. - Performance degradation: drop in primary metric > 3% absolute or > 10% relative vs 7-day rolling baseline, confirmed for 3 consecutive windows. - Minimum new labeled examples: at least N_new (e.g., 5k for large models, configurable per use case).- Calendar trigger: scheduled retrain (e.g., monthly/quarterly) to capture slow seasonal shifts even if no drift flagged.

Guardrails to avoid unnecessary retrains:- Require at least two independent triggers among {feature drift, label drift, performance drop, min labels, calendar} OR one strong trigger (performance drop sustained) + min labels.- Cooldown window: after a retrain, block retrains for T_cool (e.g., 7 days) unless critical regression (>10%) occurs.- Resource-aware gating: check GPU quota and cost budget before starting.- Manual veto: alert ML owner when a retrain would change > K features or model architecture.- Signal debiasing: require anomaly suppression (ignore single-day spikes) and combine statistical and business-rule checks.

Validation steps (vetting retrained models):1. Offline validation: - Train/validate/test with time-aware splits; compute primary and secondary metrics, calibration, and per-segment performance (e.g., by region, device). - Data sanity checks: feature ranges, missingness, label leakage tests. - Robustness tests: adversarial/noise augmentation, out-of-distribution subsets. - Explainability: feature importance shifts reviewed; large unexplained shifts flag human review.2. Gating rules: - New model must beat production on primary metric by a margin delta (e.g., >0.5% absolute) OR show lower latency/resource use while matching performance. - No degradation above tolerance in any critical subpopulation.3. Staging rollout: - Shadow run: serve predictions in parallel for a sample of traffic and compare decisions (no user impact) for 24–72 hours. - Canary rollout: route small % (1–5%) of live traffic; monitor key business and safety metrics for N hours. - Automatic rollback if any monitored metric regresses beyond fail thresholds (e.g., >1% absolute drop in conversion or >5% increase in error rate).4. Human-in-the-loop approval if explainability or fairness checks fail or changes exceed complexity thresholds.

Scheduling and orchestration:- Represent retrain job as idempotent DAG (e.g., Airflow, Kubeflow Pipelines). Stages: data extraction → preprocessing → training → evaluation → explainability checks → deployment (shadow/canary) → promote/rollback.- Use distributed lock/mutex to prevent concurrent retrains for same model; job metadata stores trigger reason and dataset snapshot for reproducibility.- Autoscaling training infra (spot instances with fallback) and resource pre-checks; enforce timeouts and retry limits.- Observability: emit events and dashboards, notify stakeholders at key steps (triggered, started, passed validation, promoted, rolled back).- Auditability: version artifacts (code, data snapshot, model weights, metrics) stored in model registry with lineage and reproducible training recipe.

Why this design:- Combines statistical rigor (PSI/KS) with business-sensitive metrics (performance, label count).- Multi-stage gating and staged rollouts prevent unnecessary or unsafe promotions.- Orchestration ensures repeatability, resource safety, and quick rollback if production impact appears.

MediumSystem Design

33 practiced

Design a monitoring dashboard for ML model health. Describe specific panels and alerts you would include: model performance metrics, input distribution monitoring, feature-level drift, latency and error rates, resource utilization, and business KPIs. For each panel specify the data sources, aggregation window, thresholds for alerts, and playbook actions when an alert fires.

Sample Answer

Requirements & overview:- Real-time + historical view of model health covering accuracy/regression metrics, input/feature drift, latency/errors, infra, and business KPIs. Use streaming telemetry (Kafka), model logs (structured JSON), feature store, metrics DB (Prometheus/Grafana or Datadog), and business DB/warehouse (Redshift/BigQuery). Typical aggregation windows: 1m (real-time), 1h (operational), 24h (trend).

Panels & alerts

1) Model performance panel- Shown: rolling accuracy/AUC, precision/recall, calibration (reliability curve), loss.- Data source: prediction logs + ground-truth labels joined in the labels store.- Aggregation: 1h rolling and 24h trend.- Alert: drop in primary metric >5% absolute over 24h or >10% relative vs baseline.- Playbook: verify label delay; run evaluation on recent labeled batch; check data pipeline; roll back to previous model if degradation confirmed; open incident, notify ML lead & product owner.

2) Input distribution monitoring- Shown: per-feature histograms, categorical value counts, top-k tokens for text.- Data source: feature-store/serving logs.- Aggregation: 1m for spikes, 24h for drift detection.- Alert: KL-divergence or population stability index (PSI) >0.2 for any feature over 24h or sudden spike >3x in 1h.- Playbook: inspect source system logs, validate upstream schema, run sample replay against model, tag suspect data, pause automated retraining.

3) Feature-level drift panel- Shown: per-feature drift score, p-values (KS for numeric, chi-square for categorical), embedding-distance for text/image features.- Data source: stored training distribution vs recent serving window.- Aggregation: 24h vs baseline.- Alert: p-value <0.01 or embedding cosine distance >0.2.- Playbook: run root-cause analysis to find correlated features; retrain candidate model on recent data in staging; set holdout for manual review before promoting.

4) Latency & error rates- Shown: p95/p99 latency, median, request rate, HTTP 5xx/4xx, model inference exceptions.- Data source: API gateway + inference logs + APM (Jaeger/New Relic).- Aggregation: 1m (real-time) and 1h.- Alert: p95 latency > target SLO (e.g., 300ms) or error rate >1% sustained for 5m.- Playbook: auto-scale GPU/serving replicas; check model size or batching config; rollback heavy changes; engage SRE if infra fault.

5) Resource utilization- Shown: GPU/CPU utilization, memory, disk I/O, queue length.- Data source: Prometheus/node exporter + cloud metrics.- Aggregation: 1m.- Alert: GPU util >90% for 10m or OOM events or queue backlog > threshold.- Playbook: autoscale workers, throttle non-critical jobs, investigate memory leaks, schedule maintenance.

6) Business KPIs panel- Shown: conversion rate, revenue per prediction, false positives/negatives cost estimate, user complaints.- Data source: event pipeline + analytics warehouse.- Aggregation: 24h with daily/weekly trend.- Alert: KPI decline >3% week-over-week or cost-per-error exceeds ROI threshold.- Playbook: correlate with model and input alerts; if model is root cause, trigger rollback and A/B test alternative; notify product and commercial teams.

Operational notes & SLAs- Correlation view linking alerts across panels (e.g., drift + KPI drop).- Alert routing: severity page for P1 (immediate paging), P2 (Slack + email), P3 (dashboard ticket).- Maintain runbooks per alert with reproducible queries and sample data, canned rollback steps, and communication templates.- Periodic: weekly drift reports, monthly model retraining cadence, and chaos tests for infra resilience.

MediumSystem Design

34 practiced

Design a serving architecture to host a large transformer model for low-latency text inference with p95 under 100 ms at 200 QPS. Discuss options for batching, model sharding, GPU or CPU serving, quantization, caching of frequent requests, autoscaling, and how to measure and enforce latency SLOs. Include trade-offs between throughput, cost, and latency.

Sample Answer

Requirements & constraints:- Functional: 200 QPS, p95 <100 ms for single-request text inference.- Budgeting: target compute latency budget ≈ 50–80 ms (allow for network + queuing).

High-level architecture:- Front-end API/load‑balancer → inference gateway (gRPC) → scheduler/admission controller → pool of GPU-backed workers (inference processes) → model shards or replicated models → optional cache (Redis) for frequent inputs → observability + autoscaler.

Key design choices and why:1) GPU vs CPU:- Use GPUs (A10/A100/H100 or efficient cloud TPU equivalents) for strict p95; CPUs only if model is tiny or heavily quantized and cost-sensitive.- GPUs provide low single-request latency; multiple concurrent small batches keep utilization.

2) Batching:- Dynamic micro-batching: collect requests up to small max batch (e.g., 8–32) or short timeout (1–5 ms) to increase throughput while bounding latency.- Admission control: refuse or route overflow for strict SLOs to prevent queue amplification.- Trade-off: larger batches → higher throughput and lower amortized cost, but higher tail latency.

3) Model sharding & replication:- Prefer model replication (each GPU hosts full quantized model) for low-latency single-shot inference.- Sharding/model-parallel only if model too large to fit one GPU; then use pipelined parallelism with careful scheduling to avoid added hops (adds latency).- Use multi-instance per GPU (MPS) carefully; isolate critical traffic.

4) Quantization:- Use mixed-precision / int8 or 4-bit quantization + calibration and QA to reduce memory and latency.- Trade-off: slight accuracy loss vs big latency/cost gains. Validate quality-sensitive workloads.

5) Caching:- LRU cache for identical inputs/beam prefixes; TTL and size limits. Edge caching for static prompts.- Use result fingerprints; cache hits return <10 ms.

6) Autoscaling:- Scale based on p95 latency and queue length, not just CPU/GPU utilization.- Fast scale-up: keep warm pool of standby GPUs or container instances to avoid cold-starts.- Scale-down with cool-down windows to avoid oscillation.

7) Measuring & enforcing SLOs:- Instrument end-to-end latency (ingress→egress) and internal breakdown (scheduler, batching delay, model inference).- Continuous synthetic traffic tests (canary) targeting p95. Alert on p95 breaches.- Enforce via admission control: reject/queue/serve degraded-mode (smaller batch, lower-precision model) when approaching SLO breach.- Use rate-limiter per-customer to protect system.

Trade-offs summary:- Latency vs throughput: small batches/replication favor latency; large batches/sharding favor throughput/cost.- Cost vs latency: more replicas/warm GPUs increase cost but reduce p95 risk.- Accuracy vs latency: aggressive quantization lowers latency/cost but risks quality.

Example sizing heuristic:- If a quantized model inference on one GPU is ~20 ms per request (batch=1) and GPU can handle ~50 concurrent inferences → one GPU can serve ~50 QPS with margin. For 200 QPS target, deploy ~5 replicas (plus 1–2 warm spares) and dynamic batching to improve utilization.

Operational best practices:- Canary deploy model changes and quantization variants.- Track both p50/p95/p99 and batching delay distributions.- Implement graceful degradation (smaller batches, lower precision, cached responses) during overload.

HardSystem Design

25 practiced

Design a safe canary rollout and observability strategy for a model update when the primary online metric is noisy and labels arrive with substantial delay. Include shadow testing, offline replay, traffic splitting strategy, statistical tests suitable for noisy metrics, rollback thresholds, and how to use auxiliary signals to make safer decisions.

Sample Answer

Requirements & constraints:- Primary online metric is noisy and ground-truth labels arrive with long delay (days–weeks).- Need safe canary rollout with fast detection of regressions, minimal user impact, ability to rollback quickly, and high confidence before full rollout.

High-level strategy:1) Offline validation (offline replay + shadow testing)2) Gradual live canary with conservative traffic splitting, rich observability, and statistical monitoring3) Decision rules using robust statistical tests + auxiliary signals; pre-defined rollback thresholds and risk budget

Architecture & components:- Offline replay service: replays production request logs through new model to compute proxy metrics and compare to baseline. Capture distributional drift, latency, confidence calibration, and top-k outputs.- Shadow testing layer: route a copy of live traffic to new model (no user-facing effects), store predictions and features; compute real-time mismatch rates and cohort analytics.- Canary router: supports traffic-splitting (percent, user cohorts, region, customer tier), can be toggled quickly.- Observability stack: metric ingestion (Prometheus), event tracing (Jaeger), logging, feature export, dashboards, anomaly detector, drift detector, and alerting.- Label collection pipeline: link delayed labels back to requests to compute delayed true-metric.

Traffic splitting strategy:- Start with 0% live traffic. Stage 1: 100% shadow for T_shadow (e.g., 48–72h) + offline replay across historical windows.- Stage 2: Small canary: 0.1–1% of traffic for a short window (e.g., 24–72h), focused on low-risk cohorts (internal users, synthetic queries, non-monetized users), then 5%, 10%, 25%, 50% increments only after passing checks.- Use progressive rollout gates: shadow checks → small canary → larger canary → ramp to 100%.

Statistical testing for noisy metrics and delayed labels:- Use short-term proxy metrics and distributional checks for early signals: - Calibration drift (Brier score), confidence shift, prediction entropy, top-k overlap with baseline. - Feature distribution drift (KL divergence, population stability index).- For noisy online metrics use sequential testing: - Sequential Probability Ratio Test (SPRT) or Bayesian A/B with hierarchical model to accumulate evidence while controlling Type I/II errors. - Use non-parametric bootstrap on rolling windows to estimate confidence intervals for metric deltas under autocorrelation. - Adjust for temporal correlation using block bootstrap or Newey-West standard errors.- For delayed labels, treat final effect as delayed cumulative metric. Use survival-analysis style accumulation and set interim decision thresholds that are conservative (require stronger proxy signal to permit larger traffic).- Control false discovery by predefining minimum sample sizes, minimum exposure time per stage, and stopping rules.

Rollback thresholds & risk budget:- Define explicit SLAs and rollback criteria: - Immediate rollback (auto): severe production errors (latency > 2x, error rate spike > X%), or safety/regulatory violation. - Near-term rollback: proxy metric deterioration beyond threshold (e.g., >3% relative degradation with p < 0.01 using SPRT) OR auxiliary signals degrade beyond allowed budget. - Delayed rollback: when delayed labels show true-metric drop exceeding pre-agreed delta (e.g., 1–2% absolute or relative), after enough labeled samples reach statistical significance.- Use a risk budget: maximum allowed cumulative negative impact (e.g., 0.5% of conversions lost across all canaries). When budget is exhausted, stop rollouts.

Auxiliary signals to make safer decisions:- Latency, error rates, resource utilization (GPU/CPU/RAM), tail latencies.- Proxy business metrics: click-through, session length, upstream/downstream service failures.- Model-internal signals: confidence/entropy shifts, rejection rates, calibration drift, feature importance shift, hallucination rate (for generative models).- User feedback channels, human-in-the-loop labels, and synthetic tests (golden set).- Product telemetry segmented by cohort (new vs returning users, region, device).

Example decision flow:1. Run offline replay and shadow for 72h. Check: no significant distribution shifts, top-k overlap > 95%, calibration stable, no latency issues.2. Launch 0.5% canary to internal beta users for 24–48h. Monitor proxy signals continuously with SPRT; require p>0.95 for no-regression.3. If pass, move to 5% public canary for 3–7 days; enforce minimum sample size and block-bootstrap CIs for the noisy metric.4. If auxiliary signals show small negative drift but within risk budget, continue; if exceed thresholds, auto-rollback.5. After labels arrive, run final A/B on true metric with pre-specified hypothesis test (two-sided, control of multiple testing via alpha spending) and only promote if significance and business delta thresholds met.

Operational practices:- Automate the pipeline: automated replays, shadow comparison reports, rolling statistical tests, alerting and auto-rollback hooks.- Maintain a canary runbook: owners, escalation, rollback commands, and post-mortem templates.- Instrument for quick forensics: ability to query individual request traces and feature snapshots.- Regularly rehearse rollbacks and chaos drills.

Trade-offs:- Conservative gates slow rollout but reduce user risk.- Heavy shadowing and logging increase cost and storage; prune intelligently and sample.- Sequential/Bayesian tests reduce sample needs but require careful priors and simulation to set thresholds.

This approach balances safety and velocity by using rigorous offline validation, conservative staged rollouts, robust statistical monitoring tuned for noisy/delayed metrics, auxiliary signals to corroborate decisions, and clear rollback policies.

MediumTechnical

29 practiced

Compare common model explainability techniques such as global feature importance, LIME, SHAP, and counterfactual explanations. For each technique explain what questions it answers, computational cost, limitations (e.g., instability, reliance on model assumptions), and production considerations for storing and serving explanations.

Sample Answer

Global feature importance (e.g., permutation importance, feature weights)- Questions answered: Which features drive model predictions overall; relative ranking.- Computational cost: Low–medium. Permutation needs O(#features × inference time); weight-based is constant.- Limitations: Aggregates across population (masks heterogeneity); correlated features distort importance; not causal.- Production: Store per-model and per-version summaries (rankings, confidence intervals). Serve as static metadata via model registry or API.

LIME (local surrogate explanations)- Questions answered: Why did the model predict this label for a specific instance? Provides interpretable local model (linear/tree).- Computational cost: Medium–high per-explanation (generates many perturbed samples × inference calls).- Limitations: Instability (different seeds/perturbations yield different explanations); sensitive to perturbation scheme; surrogate may misrepresent complex models.- Production: Cache explanations for frequent queries; log perturbation seeds and local-surrogate parameters for reproducibility; rate-limit on-demand generation.

SHAP (game-theoretic feature attributions)- Questions answered: Local and global attributions with additive consistency; how each feature contributed to a prediction or aggregated importance.- Computational cost: High for exact SHAP (exponential); TreeSHAP and KernelSHAP optimizations reduce cost (TreeSHAP is fast for tree models; KernelSHAP still expensive).- Limitations: Assumptions about feature independence (KernelSHAP); computational cost; can be dense and hard to communicate.- Production: Precompute and store SHAP vectors for common inputs or use approximate sampling; store background dataset reference; provide online explanation service with batching.

Counterfactual explanations- Questions answered: What minimal changes to inputs would change the model outcome (actionable recourse)?- Computational cost: Variable—solving optimization/search per instance (can be heavy for high-dim models).- Limitations: Multiple valid counterfactuals, plausibility constraints, may suggest unrealistic or infeasible actions; sensitive to constraint modeling.- Production: Store candidate counterfactuals and metadata (distance, plausibility, required features); enforce privacy (counterfactuals may expose sensitive model behavior); provide a constrained generator service with business-rule filters.

Trade-offs and best practices- Use global importance for monitoring and feature engineering; use SHAP for faithful attributions when cost allows; use LIME for quick local intuition but validate stability; use counterfactuals when providing user recourse.- For production: version explanations with model artifacts, store background datasets and random seeds, cache common explanations, monitor explanation drift, and expose explanations via an authenticated, rate-limited API with human-readable summaries and confidence/validity metadata.

Unlock Full Question Bank

Get access to hundreds of End-to-End ML System Design interview questions and detailed answers.

Join thousands of developers preparing for their dream job.