End-to-End ML System Design Questions
End-to-end design of machine learning systems, covering data collection and validation, feature engineering and feature stores, model training and evaluation, deployment and serving architectures, monitoring and incident management, retraining pipelines, data governance, scalability, security, and MLOps practices.
EasyTechnical
33 practiced
List the top production ML metrics you would monitor for a deployed classification model (include data, model, and infra metrics). For each metric explain what a deviation might indicate and a possible remediation step. Consider metrics such as input traffic and feature distributions, label rates, prediction latency, error rates, model accuracy, calibration drift, resource utilization, and request errors.
Sample Answer
Below are key production ML metrics grouped by data, model, and infra/serving, with what deviations may indicate and suggested remediation.Data metrics- Input traffic (requests/sec, user segments): Drop → upstream app bug or client changes; Spike → bot/flooding or experiment. Remediation: verify client deploys, add throttling/filters, roll back change, update alerting.- Feature distributions (per-feature mean, std, percentiles, categorical value rates): Shift → data-drift, new user behavior, bad ETL. Remediation: retrain on new data, investigate ETL, apply transformation fixes, enable online feature validation.- Label rate & delay (label arrival rate, label/feedback latency): Lower rates or delayed labels → feedback loop broken, reduced training signal. Remediation: fix labeling pipeline, use semi-supervised or active learning temporarily, widen monitoring windows.Model metrics- Prediction accuracy / business metric (AUC, precision@k, recall): Degradation → model drift, concept shift, label noise. Remediation: run shadow model, retrain with recent data, rollback to previous model.- Calibration (confidence vs. empirical accuracy): Overconfident/underconfident → miscalibrated probabilities harming decisions. Remediation: recalibrate (Platt scaling / isotonic), retrain with calibration-aware loss.- Error rates by slice (false positive/negative per cohort): Increase in a cohort → bias/regression for subgroup. Remediation: slice-specific analysis, augment data, constrained retraining, apply per-group thresholds.- Prediction distribution (class proportions): Change → upstream population shift or label mismatch. Remediation: investigate upstream changes, consider sample reweighting.Infra / serving metrics- Prediction latency (P50/P95/P99): Increase → model size, hardware contention, cold starts. Remediation: scale replicas, optimize model (quantize/prune), cache results, move to faster hardware.- Resource utilization (CPU/GPU, memory): High → need autoscaling or model optimization; Low with high cost → downscale. Remediation: autoscale policies, optimize batching, right-size instances.- Request errors & availability (5xx rates, timeouts): Rise → serving bug, model crash, networking. Remediation: circuit breakers, health checks, fallback model, investigate logs.- Throughput / queue length: Growing queue → underprovisioned or slow downstream. Remediation: increase workers, enable async processing, backpressure.Notes on alerts & practices- Use threshold + change-rate alerts (absolute + relative) to avoid noise.- Monitor metrics per cohort and per client to catch localized issues.- Maintain canary and shadow deployments with automatic rollback triggers for critical metric regressions.- Log raw inputs and predictions (privacy-compliant) to enable root-cause analysis and quick retraining.
MediumSystem Design
25 practiced
Design a retraining pipeline that is triggered when data drift is detected. Describe components for drift detection, retrain triggering logic (thresholds, windows), automated preprocessing, validation tests, model candidate selection, model promotion/rollback, and human-in-the-loop gates. Explain strategies to avoid retraining on transient noise and how you'd validate a retrained model before deployment.
Sample Answer
Requirements & constraints:- Automatically detect distribution or performance drift, retrain only when meaningful drift persists, validate new model thoroughly, support safe deployment with rollback and human approval.High-level architecture:- Data source → Feature store → Drift detectors + Metrics monitor → Retrain orchestration → Training pipeline → Validation suite → Candidate selection → Deployment (shadow → canary → full) → Continuous monitoringDrift-detection components:- Two families: input/feature drift (unsupervised) and label/performance drift (supervised).- Detectors: PSI/KS/KL for marginals, MMD for multivariate, ADWIN or EDDM for sequential detection, performance monitors (AUC, accuracy, calibration).- Maintain rolling windows: short (last 1 day), medium (7 days), long (30 days). Compute detector stats per window and baseline (training/production baseline).Retrain-triggering logic:- Use ensemble voting + hysteresis: require ≥2 detector signals AND performance degradation (e.g., delta AUC > 2%) sustained across medium and long windows.- Apply cooldown and minimum interval (e.g., no retrain within 7 days) to avoid oscillation.- Thresholds configurable per model; use tiered alerts: monitor-only → automated retrain → human approval.Automated preprocessing & training:- Reuse deterministic feature pipelines (feature store + processing containerized with same code).- Data validation (schema, missingness, drift in cardinality).- Automated hyperparameter tuning with budgeted search; seed runs with current model weights for faster convergence.Validation tests:- Data tests: schema, value ranges, distribution sanity.- Performance tests: holdout evaluation on recent labeled holdout, cross-validation, backtesting on time-sliced data.- Robustness: adversarial/noise injection, calibration, fairness checks.- Regression tests: ensure no metric regressed beyond tolerance compared to production model.- Resource tests: latency, memory, throughput.Model candidate selection:- Rank by primary metric (business KPI) subject to constraints (latency, cost).- Prefer simpler/smaller models if gains marginal.- Keep lineage and reproducibility metadata for each candidate.Deployment, promotion & rollback:- Shadow deploy first (run in parallel on live traffic without affecting decisions); compare metrics.- Canary rollout (1–5% traffic) with automatic guardrails: if any safety metric breaches, auto-rollback.- Full promotion via CI/CD after human sign-off or automated pass.- Instant rollback path to previous model artifact.Human-in-the-loop gates:- Dashboard summarizing drift evidence, training logs, validation metrics, and explanations.- Mandatory human approval for models that change decision thresholds, materially affect fairness, or cost.- Allow fast approvals for low-risk retrains that pass all automated gates.Avoiding retraining on transient noise:- Require persistence across multiple windows and detector agreement.- Use hysteresis and cooldown periods.- Smooth signals with EWMA; set minimum sample-size thresholds.- Apply statistical significance testing and false-positive control (e.g., Bonferroni or FDR for multiple features).Validating before deployment:- Use temporally realistic holdout (no leakage), backtesting, shadow traffic, and canary with automated rollback.- Run A/B experiments measuring business KPIs for a defined period.- Monitor calibration, fairness slices, and downstream impact; only promote when stable.Observability and governance:- Store telemetry, metadata, and drift history; alerting and audit logs; model registry with lineage, metrics, approval records.Trade-offs:- Aggressive retraining responds faster but risks churn; conservative policies reduce noise but may lag. Tune thresholds to business tolerance.This pipeline balances automation with human oversight, using multi-window detection, robust validation, and safe rollout patterns (shadow → canary → full) to avoid reacting to transient noise while ensuring models stay accurate and reliable.
MediumSystem Design
26 practiced
Design a scalable training pipeline for a supervised model on 10 TB of tabular data that requires complex joins and preprocessing. Describe choices for data preprocessing framework (Spark/Beam), feature storage, distributed model training (e.g., XGBoost on Spark, distributed PyTorch), orchestration, resource scheduling, caching, and fault tolerance. Explain trade-offs and cost considerations.
Sample Answer
Requirements & constraints:- 10 TB tabular data, complex joins, supervised training, repeatable/production pipeline, reasonable latency for retraining (daily/weekly), cost-sensitive.High-level design:1) Ingestion & storage- Raw data in cloud object store (S3/GCS) partitioned by time + primary keys. Use Parquet with snappy for columnar reads.2) Preprocessing framework- Spark (Databricks or EMR) for heavy joins and wide-table transformations: mature optimizer, partitioning, broadcast join controls, and ecosystem (MLlib, Spark-accelerated XGBoost). Beam is better if multi-runner portability/streaming required.3) Feature engineering & storage- Compute features in Spark and write to a Feature Store (Feast or in-house) backed by a low-latency DB (Redis/Cassandra) for serving and a columnar storage (Delta Lake / BigQuery) for training materialization. Store both online (key->feature) and offline (feature tables keyed by entity + timestamp).4) Distributed training- For tree models: XGBoost on Spark (sparkxgb) or SageMaker/XGBoost with external memory; use GPU only if using deep nets. For deep / complex models: distributed PyTorch with Horovod or torch.distributed on a Kubernetes or Spark cluster. Use data-parallel with sharded Dataset, S3 streaming, and efficient shuffling.5) Orchestration & scheduling- Airflow/Kubeflow Pipelines for DAGs: ingestion -> preprocess -> feature materialization -> train -> validate -> register model. Use Kubernetes for containerized tasks; MLFlow or model registry for artifacts.6) Resource management & caching- Use autoscaling Spark clusters with right-sizing (spot instances for batch, on-demand for critical jobs). Cache intermediate large joins on persisted Parquet/Delta and in-memory RDD/DataFrame caching for iterative feature computation. Use broadcast joins for small dimension tables and partitioning keys for large joins.7) Fault tolerance & reproducibility- Keep immutable raw data, use checkpoints in Spark, deterministic seeds, store job configs, Docker images, and feature/label versions. Retryable idempotent tasks, speculative execution disabled/enabled per workload.8) Trade-offs & cost- Spark has lower engineering overhead for heavy joins; Beam offers portability. Feature store adds upfront cost but saves serving/consistency later. Spot instances cut compute cost ~50-70% but risk preemption—combine with checkpoints. Using GPU clusters increases accuracy training speed but higher cost; prefer CPU XGBoost for tree models on tabular data. Delta Lake/Parquet reduces storage I/O and query costs.Key metrics to monitor:- Job durations, shuffle/read/write bytes, cache hit rates, training/validation throughput, cost per training run, model performance drift.This design balances scalability, reproducibility, and cost.
MediumTechnical
26 practiced
Case: After deploying a change in the data pipeline, a recommendation model's CTR drops by 8%. Outline a systematic incident investigation plan. What logs, metrics, and artifacts do you check across ingestion, feature computation, model inputs/outputs, serving, and downstream evaluation? Describe containment actions and criteria for rolling back the pipeline or model.
Sample Answer
Start by scoping & timeline: note exact deploy time, traffic/A/B group affected, magnitude (8% CTR) and whether it's steady or transient. Notify stakeholders and put incident channel.Investigation plan (walk forward + backtrace):1. Ingestion- Check logs: ingestion job scheduler (Airflow/Kubernetes) run history, latency, error rates, schema validation failures.- Metrics: input volume, data skew vs baseline, late/duplicate messages, partition key distribution.- Artifacts: raw sample of records before and after deploy, checksum/hashes.2. Feature computation- Logs: feature pipeline job logs, traceback, retries.- Metrics: feature availability (null %), distribution stats (mean, std, percentiles), cardinality changes, freshness/latency.- Artifacts: feature store snapshots, sample feature vectors.3. Model inputs/outputs (offline)- Logs: data-validation (TensorFlow Data Validation) alerts, input schema mismatches.- Metrics: feature correlation shifts, PSI/KL divergence per feature, proportion of OOV tokens/IDs, inference input counts.- Artifacts: batch of model input vectors pre/post-change, model score distributions, calibration plots.4. Serving- Logs: serving infra (gRPC/REST) errors, latency, dropped requests, model load events, container image/tag used.- Metrics: per-route latency/p50,p95, error rates, QPS, model version traffic split, resource saturation (CPU/memory).- Artifacts: request/response samples, serving config (feature transforms applied in-serving vs offline).5. Downstream evaluation & business metrics- Logs: A/B test assignment logs, experiment SDK events.- Metrics: CTR by cohort, click latency, conversion funnel, user segmentation breakdown, hover vs click rates.- Artifacts: experiment analysis notebook, pre/post launch dashboards.Containment actions (in priority order)- Pause affected experiment or route new traffic away from new pipeline/model.- Re-route traffic to the last-known-good model/version (traffic split rollback).- Disable problematic feature(s) (feature flag) if traced to a specific feature causing negative psi or null spike.- Throttle ingestion or rerun backfill/repair corrupted data partitions.Rollback criteria- Immediate rollback if: (a) CTR drop >5% relative and sustained beyond short spike window (e.g., 30–60 minutes) in production cohort; (b) evidence of data corruption (schema mismatch, high nulls/OOV) or serving errors; (c) model serving failures, high error rates, or resource exhaustion.- If uncertain, partial rollback/traffic split: shift 50% traffic to baseline and monitor recovery within a defined SLA window (e.g., 30–60 min). Full rollback when metrics return to baseline and root cause is unresolved.- Document decision, run postmortem, and create action items: fix pipeline, add validations, improve canary tests and monitoring (PSI thresholds, sanity checks, feature-level alerts).Key preventive checks to add: pre-deploy canary with synthetic traffic, automated data/schema validators, feature drift alarms, and playbooks for fast rollback.
HardSystem Design
25 practiced
Design an automated canary deployment process for ML models running in microservices. Include traffic-shifting policies, required statistical tests for health and business metrics, rollback rules, handling of stateful online features, and safe ramp-up limits to minimize user impact. Explain how you'd automate decision-making while preventing false positives.
Sample Answer
Requirements:- Functional: safely deploy new ML model versions with progressive traffic shifts and automated decisioning.- Non-functional: low latency, zero/near-zero user impact, observable metrics, auditable rollbacks, safe for stateful online features.- Scale: support many concurrent canaries, per-model config.High-level architecture:- CI/CD pipeline triggers model build → container image → canary orchestration service.- Traffic router (Envoy/NGINX/Service Mesh) supports weighted routing.- Metrics pipeline: logs, inference metrics, business KPIs → streaming store (Prometheus, Kafka → ClickHouse).- Analysis service runs statistical tests and decides (or suggests) continue/rollback.- Orchestrator enforces rollout plan and rollback.Traffic-shifting policy:- Start at 0.5–1% traffic for 10–30 min (configurable).- Ramp phases: e.g., 1% → 5% → 20% → 50% → 100%. Each phase requires passing health and business checks.- Conservative caps per minute/hour and per-customer (no more than X% of single-user cohort).Statistical tests & health signals:- System health: error rate, latency P95/P99, resource usage — require no statistically significant degradation (one-sided z-test or bootstrap) at alpha=0.01.- Model quality: primary business metric(s) (CTR, conversion, revenue per user) — use uplift tests (difference-in-proportions or t-test/bootstrapped CI). Require lower-bound of uplift > M (practical delta) or at least non-inferiority margin.- Use sequential testing (alpha spending / group sequential tests or Bayesian bandit-style posterior thresholds) to avoid repeated-sampling inflation.- Minimum sample size per phase computed from power analysis; if not met, hold ramp.Rollback rules:- Immediate rollback if critical system metrics cross absolute thresholds (e.g., error rate > baseline + X% or latency increase > Y ms).- Conditional rollback if business metric shows statistically significant negative impact at configured alpha and minimum sample.- Automatic rollback after N failed phases or manual abort by owner.- Graceful drain: route traffic away, finish in-flight requests, rollback model binary, preserve logs for postmortem.Handling stateful online features:- Feature gating: freeze state updates during canary or mirror writes to a shadow store.- Dual-write pattern: new model reads from same feature store but writes to separate namespace until validated.- Feature consistency: ensure keys/versions tagged; use feature versioning and per-request feature snapshotting to avoid train/serving skew.- For streaming features (counters), use deterministic replay or offset-aware reads to keep comparisons fair.Automation and preventing false positives:- Combine frequentist and Bayesian signals: require both (e.g., p < 0.01 AND Bayesian posterior of harm > 90%) to act automatically; otherwise alert humans.- Use sequential testing with alpha correction and enforce minimum sample sizes/power to reduce false alarms.- Ensemble of metrics: require guardrail pass (health) and business pass; weight metrics; use multivariate control charts and anomaly detection to detect drift unrelated to model change.- Implement hysteresis: require sustained degradation across multiple windows (e.g., 3 consecutive intervals) before auto-rollback.- Canary isolation: run multiple simultaneous canaries on disjoint user segments to detect population-specific effects.Operational details:- Config-as-code per model: ramp schedule, metrics, thresholds, min sample size, rollback policies.- Audit logs, notifications, Slack/Runbook integration, and postmortems auto-generated with traces and diffs.- Run periodic simulated canaries on synthetic traffic to validate pipeline.Why this works:- Progressive ramps + statistical rigor minimize user impact and false positives.- Combining health + business metrics and requiring sustained signals avoids knee-jerk rollbacks.- Feature versioning and dual-write protect stateful behavior, ensuring fair comparison.
Unlock Full Question Bank
Get access to hundreds of End-to-End ML System Design interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.