ML Operations & Reliability at Large Scale Questions
Production ML systems lifecycle, including deployment, monitoring, scaling, and reliability practices for machine learning at large-scale platforms. Covers MLOps, model serving architectures, data quality and versioning, feature stores, canary rollouts, incident response, postmortems, and platform reliability considerations for ML workloads serving very high request volumes and large user bases.
EasyTechnical
89 practiced
Compare batch and online (real-time) inference architectures for ML at Netflix scale. For each approach list typical use-cases, latency/throughput characteristics, cost implications, and how you would maintain feature parity between offline training and online serving for both architectures.
Sample Answer
High-level summary:- Batch inference: large-scale, periodic scoring (hours), high throughput, high latency (minutes–hours). Good for offline analytics, retraining labels, nightly recommendations, email campaigns.- Online (real-time) inference: low-latency, per-request scoring (ms–tens of ms), lower throughput per node but elastic; used for UI recommendations, personalization, fraud detection, dynamic UI.Typical use-cases:- Batch: nightly reranks, offline A/B evaluation, feature-generation for training, catalog-wide re-scoring.- Online: homepage ranking, search ranking, personalization, real-time eligibility decisions.Latency / throughput:- Batch: latency tolerant, can process millions of items per job using distributed compute (Spark/Beam). Throughput limited by cluster size; cost amortized.- Online: strict tail-latency SLOs (e.g., 50–200ms). Throughput scaled with autoscaling, caching, and batching at model-server level.Cost implications:- Batch: cheaper per-item for massive jobs (spot VMs, big clusters). Lower operational complexity but longer feedback loop.- Online: higher cost per prediction (replicated low-latency infra, reserved capacity, CDNs, feature-store caches). Costs rise with QPS and strict SLOs.Maintaining feature parity (both architectures):1. Central feature definitions: single source of truth (Feature Store) with documented transforms and schemas.2. Share transformation code: package feature engineering as versioned libraries or embed SQL/Beam pipelines that run both offline and produce online-ready artifacts.3. Online materialization / serving: precompute features in batch and write to low-latency stores (Redis/Cassandra) for online reads; complement with real-time feature pipelines for streaming events.4. Consistent joins & missing-value handling: same null-imputation, aggregations, time-window semantics.5. Schema/versioning & CI: enforce schema checks, contract tests, and unit tests that compare offline vs online feature values on sample keys.6. Shadow testing and calibration: run online model in shadow or log predictions and compare to batch recomputations; monitor drift and parity metrics (per-feature diff distributions).7. Latency-aware approximations: document any online approximations (e.g., truncated windows) and include compensating normalization in training.Result: use batch where scale & cost dominate; use online when latency and personalization need immediacy — but unify feature engineering via a versioned feature-store, shared code, and continuous parity checks to keep models consistent.
EasyTechnical
86 practiced
Describe shadow traffic (traffic mirroring) testing for ML models. Explain how to implement it so shadow models receive live traffic without impacting users, what telemetry to collect (inputs, outputs, latencies), how to avoid side effects to downstream systems, and statistical comparisons you would run on the collected data.
Sample Answer
Shadow traffic (traffic mirroring) testing sends a copy of real production requests to one or more shadow models so you can evaluate them on live data without affecting users.Implementation:- Deploy shadow model(s) behind the same API gateway or service mesh and configure mirroring: for each incoming request, synchronously serve the response from the primary model to the user while asynchronously forwarding a copy to shadows.- Use a non-blocking queue (Kafka, Pub/Sub) or async worker to avoid adding latency to production responses.- Tag each mirrored request with unique IDs, timestamps, model version, and routing metadata.Telemetry to collect:- Inputs (raw request, features used)- Model outputs (predictions, confidences, top-k, scores)- Timing: request receive time, inference start/end, serialization time (so you can separate serving vs model time)- Context: user segment, feature schema/version, model config- Downstream effects: any downstream API calls initiated by model decisions (logged but not executed—see below)- Error logs and resource metrics (CPU/GPU, memory)Avoiding side effects:- Make mirrored requests read-only: do not execute actions (no writes, no emails, no billing). For decision actions, simulate downstream calls or send them to a sandbox endpoint.- Ensure idempotency and masking of personally identifiable info if necessary.- Rate-limit mirror traffic to control costs and avoid saturation.Statistical comparisons:- Compare distributions of inputs (feature drift) using KS test or population stability index.- Compare output distributions (calibration, top-label frequency) with chi-squared or KS.- Compare continuous scores with paired t-test or Wilcoxon signed-rank.- Compute calibration curves, ROC/AUC, precision/recall on labeled subsets or delayed ground-truth.- Track latency differences (paired tests) and tail latency (p95/p99).- Use cohort analyses (by user segment, device, geography) and confidence intervals, and monitor for significant regressions with alerting.This approach gives realistic, low-risk validation of new models before promotion to serving traffic.
EasyTechnical
77 practiced
List practical cloud cost-optimization strategies for serving and training ML at scale: model compression (quantization/distillation), batching of requests, caching expensive feature lookups, right-sizing instances, using spot capacity, multi-tenancy on inference nodes, and storage lifecycle management. For each strategy, describe trade-offs in accuracy, latency, or reliability.
Sample Answer
Here are practical cloud cost-optimization strategies for serving and training ML at scale, with clear trade-offs for accuracy, latency, and reliability:- Model compression (quantization/distillation) - Benefit: reduces model size, memory, and inference cost; enables CPU/TPU inference. - Trade-offs: small accuracy loss (more for aggressive int8/4-bit); may require calibration; can reduce numerical stability for some layers.- Batching of requests - Benefit: higher throughput and GPU utilization, lower cost per inference. - Trade-offs: increases tail latency (added queuing), needs adaptive batching to avoid user-visible delay; requires careful batch-size tuning.- Caching expensive feature lookups - Benefit: avoid repeated DB/feature-store calls, lower egress and compute. - Trade-offs: cache staleness affects model freshness; added complexity for invalidation and consistency.- Right-sizing instances - Benefit: match CPU/GPU/memory to workload to avoid overprovisioning. - Trade-offs: under-sizing causes throttling or OOMs; requires monitoring and profiling to pick correct types.- Using spot/preemptible capacity - Benefit: large savings for training and batch jobs. - Trade-offs: interrupted jobs require checkpointing/restart logic; not suitable for critical low-latency serving.- Multi-tenancy on inference nodes - Benefit: better resource amortization across models/clients. - Trade-offs: noisy-neighbor interference, isolation/security considerations, more complex scheduling and autoscaling.- Storage lifecycle management - Benefit: tier data (hot/cool/archival) to save storage costs for datasets and model artifacts. - Trade-offs: slower access for archived data, egress costs when restoring, and complexity in automating lifecycle policies.For each technique combine monitoring (latency, error-rate, cost), progressive rollout, and A/B testing to measure real impact and catch regressions in accuracy or reliability.
MediumSystem Design
71 practiced
Design a CI/CD workflow for ML that enforces reproducibility and quality gates before production deployment. Include unit and integration tests for feature transforms, data schema checks, model validation tests (performance, fairness), automated canary deployment, and automatic rollback triggers. Describe tooling and GitOps integration points.
Sample Answer
Requirements & constraints:- Immutable reproducibility: versioned data, code, env, and model artifacts- Quality gates: unit/integration tests, data/schema checks, model performance & fairness thresholds- Safe rollout: automated canary with metric-based rollback- GitOps for auditability and traceabilityHigh-level CI/CD pipeline (Git-driven):1. Dev branch -> feature work2. PR triggers CI (GitHub Actions / GitLab CI / Jenkins) - Lint, unit tests for transform logic and utilities (pytest) - Build and publish container image (Docker) with pinned base image and dependency lockfile - Run lightweight transform unit tests with synthetic snapshots3. Model training job (on merged main or via scheduled retrain) - Use DVC or Delta Lake to version training data; record dataset hash - Train reproducibly (fixed seeds, captured env via conda/poetry + Docker) - Log artifacts (model binary, feature transform pipeline, metrics, provenance) to MLflow or Weights & Biases - Produce immutable model artifact with content-hash and metadata (data hash, code commit, hyperparams)4. Validation stage (blocking quality gates) - Data schema and drift checks: Great Expectations / Deequ against dataset snapshot - Feature transform integration tests: run transform pipeline end-to-end on held-out data; compare outputs to golden snapshots or property-based checks - Model validation tests: - Performance: evaluate on test set; require metrics >= threshold (AUC, RMSE, etc.) - Stability: bootstrap/resampling checks - Fairness: group-based metrics (precision/recall parity, false positive/negative parity), statistical tests - Explainability sanity checks (feature importance sign checks) - Register results in MLflow; fail pipeline if any gate fails5. Approval & GitOps manifest update - If validation passes, CI generates a Kubernetes deployment manifest (or KServe/Seldon InferenceService) pinned to the model artifact tag and container image - Commit manifest change to infra Git repo (or open PR). Argo CD / Flux watches repo and applies changes once merged—ensures GitOps6. Canary deployment & monitoring (CD) - Argo Rollouts or Kubernetes + Istio/Linkerd to do weighted traffic split (e.g., 5% canary) - Observability: Prometheus metrics (latency, error rate), custom model metrics (prediction distribution, average score), business metrics logged to downstream systems - Shadow traffic for stricter evaluation if feasible7. Automated gating & rollback - Define SLOs and statistical tests: compare canary vs baseline on key metrics (latency, error, model performance proxies, fairness metrics) over the canary window - Use Prometheus Alertmanager or a control plane (Argo Rollouts + Prometheus) to evaluate thresholds; if any threshold breached or drift detected, trigger automated rollback to previous stable revision - Post-deploy validation job runs continuous checks and stores results for auditTooling suggestions:- Source + CI: GitHub/GitLab + Actions/GitLab CI- Data & artifact versioning: DVC, Delta Lake, or LakeFS; model registry: MLflow/W&B- Tests: pytest (unit), Great Expectations / Deequ (schema), Tox for environments- Containers & serving: Docker images, Kubernetes; model serving: KServe or Seldon Core- GitOps: Argo CD or Flux for manifest sync- Rollouts: Argo Rollouts / Istio + Linkerd for traffic shifting- Monitoring & alerts: Prometheus, Grafana, Alertmanager, plus custom probes for fairness/perf- Orchestration: Airflow / Kubeflow Pipelines for scheduled retrains and validation flowsKey design points & reasoning:- Reproducibility: pin data + code + environment, record hashes and provenance so any production artifact can be rebuilt.- Tests layered: unit tests catch logic bugs early; integration tests validate feature transforms end-to-end; schema checks prevent data surprises; model validations guard business impact & fairness.- GitOps ensures declarative deployments with audit trail and rollbacks via VCS history.- Canary + metric-driven rollback provides safety: small exposure and automatic rollback reduces blast radius.- Keep gates strict in CI; orchestrate long-running validations and fairness analyses asynchronously but require pass before manifest commits.Edge considerations:- Cost/latency of full validation: run quick sanity gates in CI, heavier statistical checks in a validation cluster.- Data privacy for held-out validation: use synthetic or privacy-preserving approaches where needed.- Drift detection post-deploy: automatic retrain triggers when drift exceeds threshold; include human-in-the-loop for risky changes.
HardSystem Design
70 practiced
Design an automated experimentation platform for progressive rollouts that supports streaming metrics ingestion, sequential statistical testing with early-stopping rules, multiple hypothesis correction, automated rollback on adverse safety signals, privacy controls for experimental data, and an auditable experiment history. Describe the data pipelines and monitoring needed for safe automation.
Sample Answer
Requirements & constraints:- Support streaming metrics (low latency), progressive rollouts with automated decisions, sequential tests with early stopping, correct for multiple hypotheses, automatic rollback on safety signals, privacy controls, and an auditable immutable history. Targets: sub-minute decision latency, verifiable audit trail, GDPR/CCPA compliance.High-level architecture:- Ingestion: client SDKs/edge emit events → Kafka (or Kinesis) topics partitioned by user-id/experiment.- Stream processing: Flink/Beam jobs compute real-time aggregated metrics (counts, sums, CTR, revenue) into a metrics store and feature store. Processing includes windowing, deduplication, enrichment (assign experiment bucket), and per-user hashing for privacy.- Metrics store: OLAP time-series DB (ClickHouse/BigQuery) for historical queries + fast in-memory store (Redis) for rolling aggregates.- Experiment controller: service that runs sequential statistical engine, enforces early-stopping rules, applies multiplicity correction, and issues rollout/rollback commands to deployment/orchestration & feature-flag system.- Orchestration & enforcement: feature-flag platform (e.g., LaunchDarkly / internal) + Kubernetes operator to scale/route traffic; canary channels for gradual ramp.- Audit & lineage: append-only event log in Kafka + immutable metadata DB (write-once rows, signed) stored in encrypted BigQuery / data lake; store experiment spec, decisions, seeds, code versions, and model artifacts hashes.Sequential testing & multiple comparisons:- Support both frequentist group/alpha-spending (O’Brien–Fleming, Pocock) and sequential probability ratio test (SPRT), and Bayesian sequential tests (posterior probability thresholds).- Implement alpha-spending framework at the controller level to allocate type-I error over time across analyses.- For multiple concurrent experiments/metrics, apply hierarchical correction: pre-specify metric families and use Benjamini–Hochberg for FDR or Holm-Bonferroni for strict control; incorporate dependency structure via permutation-based adjustment or hierarchical Bayesian model to reduce conservatism.- Early stopping rules: require pre-registered stopping boundary, minimum sample or exposure, and holdout verification window before final promotion. Use decision thresholds + effect size bounds and require contextual checks (consistency across segments).Automated rollback & safety signals:- Define safety SLOs (latency, error-rate, revenue, safety-specific metrics). Stream pipeline continuously evaluates safety metrics against thresholds and triggers a circuit-breaker if thresholds breached.- Rollback flow: experiment controller publishes “rollback” event → feature-flag service immediately toggles traffic → orchestrator shifts traffic to baseline → controller logs action and notifies on-call via pager.- Use progressive rollback: partial rollback first (reduce to 0% for risky segments) before full rollback; require human-in-the-loop override for irreversible actions.Privacy, security & compliance:- Minimize PII: hash/pepper user IDs before ingestion; use deterministic salts scoped per environment.- Differential privacy: add calibrated noise to aggregated metrics where needed, with documented privacy budget accounting in experiment metadata.- Access controls: RBAC on experiment API and metric views; column-level encryption in storage.- Data retention & deletion pipelines integrated with identity deletion requests; audit deletions.Data pipelines & monitoring for safe automation:- Data quality pipeline: schema validation, monitoring for missing partitions, cardinality anomalies, late-arrival rates; auto-quarantine of suspect streams.- Drift detection: online statistical tests (KS, PSI) on feature distributions and covariate balance between variants; alert when drift exceeds thresholds.- Backtesting and simulation: offline replay engine to re-run sequential tests on historical data and stress-test stopping rules and multiplicity corrections; use these to calibrate alpha-spend and thresholds.- Observability: dashboards for per-experiment metrics, decision boundaries, and alerting for safety signals; logs of every controller decision and actor (with signatures).- Chaos & guardrails: synthetic traffic tests and canary failures to validate rollback automation; require kill-switch requiring human acknowledgement for experiments that affect irreversible user state.Auditability & reproducibility:- Store experiment spec, code commit hashes, randomization seeds, metric definitions, and every decision event in append-only store.- Provide replay capability: re-run experiment analysis deterministically from raw events using stored seeds and code versions.- Provide signed export for regulatory audits.Trade-offs & rationale:- Streaming enables fast detection but increases false alarms; controlled via pre-specified minimum exposure and alpha-spending.- Frequentist vs Bayesian: offer both; Bayesian simplifies continuous monitoring but requires priors and stakeholder buy-in. Provide defaults and guardrails.- Multiplicity: family-based correction balances discovery vs conservatism; hierarchical models reduce overcorrection at cost of modeling complexity.Example flow (concrete):1. User request triggers SDK assignment to variant; event sent to Kafka.2. Flink aggregates metric increments into 1-minute rolling windows; writes to Redis and OLAP.3. Controller reads aggregates, updates sequential test statistic (e.g., log-likelihood ratio + alpha-spend), applies BH across metrics, checks safety SLOs.4. If stop/rollback condition met, controller emits rollback event; feature-flag toggles; orchestrator re-routes traffic; controller logs action in immutable audit store and notifies SRE/owners.This design balances automation speed with statistical rigor, privacy, and safety through pre-registration, alpha-spending, multiplicity controls, comprehensive monitoring, and an auditable immutable trail.
Unlock Full Question Bank
Get access to hundreds of ML Operations & Reliability at Large Scale interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.