ML Operations & Reliability at Large Scale Questions
Production ML systems lifecycle, including deployment, monitoring, scaling, and reliability practices for machine learning at large-scale platforms. Covers MLOps, model serving architectures, data quality and versioning, feature stores, canary rollouts, incident response, postmortems, and platform reliability considerations for ML workloads serving very high request volumes and large user bases.
MediumTechnical
71 practiced
Design a canary rollout experiment for a low-latency recommendation model. Specify which business and technical metrics to track, how to perform statistical tests to detect regressions quickly (including sample-size/power reasoning), and guardrail metrics and rollback criteria. Explain how you'd run the experiment to minimize user disruption.
Sample Answer
Clarify goal & constraints- Goal: safely roll a new low-latency recommendation model that may change ranking/CTR while preserving latency and revenue. Constraints: real-time traffic, tight p95 latency SLA.Experiment design (canary + sequential monitoring)- Serve new model to a small subset and ramp: 0% -> 0.5% -> 2% -> 10% -> 50% -> 100%. At each step hold until minimum sample and guardrails pass. Use server-side feature flag + traffic bucketing by user-id for stable assignment.Business metrics to track- Primary: CTR (clicks/impressions) or engagement-rate appropriate to product; Revenue per user (RPU) or conversions per user.- Secondary: Session length, downstream purchases, retention over 7/14 days (as available).Technical metrics (guardrails)- Latency: p50, p90, p95, p99 response times and change vs baseline.- Error rate: request failures, timeout rate.- Resource: CPU/memory per host, queue lengths.- Model correctness: distribution shifts in top-k items, novelty/diversity if relevant.Statistical testing & sample-size/power reasoning- For CTR (a proportion) use two-proportion z-test for fixed-horizon or sequential alternatives: - Desired detectable effect: e.g., 10% relative change. If baseline CTR p0=0.05 and target delta = 0.005 (absolute, 10% relative), with α=0.05, power=0.8, approximate per-arm sample n ≈ 150k–300k impressions (rule of thumb; compute exact with formula or stats package).- For continuous metrics (latency), use Welch’s t-test (or nonparametric Mann–Whitney if skewed) on log-latencies.- Prefer group-sequential (alpha-spending) or Bayesian sequential monitoring to detect regressions fast while controlling false positives. Example: use an O’Brien–Fleming or Pocock spending function to allow multiple looks with controlled family-wise α.- If using Bayesian: monitor posterior probability P(effect < 0) and stop if P(drop > threshold) > 0.95.Quick detection: sequential checks every X minutes (based on traffic) but enforce minimum sample per check (e.g., 10k impressions or 1k users) to avoid noisy alarms.Guardrail thresholds & rollback criteria- Immediate automatic rollback triggers: - Latency p95 increases > 20% or absolute increase > 50 ms vs baseline. - Error/timeout rate exceeds baseline + 0.1 percentage points or doubles. - CPU/memory per-host > baseline + 30% causing downstream backpressure.- Business soft rollback triggers (require human review + automatic rollback if sustained): - CTR drop > 5% relative for two consecutive monitoring windows of sufficient sample. - RPU drop statistically significant (p<0.01) and material (>2%).- Maintain an “escalation” tier: if a soft trigger is hit once, pause ramp and collect full-window data; if persists, rollback.Minimizing user disruption- Start with low-risk segments: internal users, power users, or new users; or non-critical geos.- Shadowing/dark launch: run new model in parallel to collect predictions and differences without serving, to validate ranking distributions and catch safety issues.- Feature-flagged ramp with circuit-breaker: auto-revert to baseline on guardrail breach.- Canary on small percentage and short holds: e.g., hold 1–2 hours at small ramps to gather latency + CTR signals; longer holds for revenue signals (1–3 days) as needed.- Monitor dashboards & alerts: real-time dashboards for guardrails, automated alerts to on-call with playbook.Operational notes- Precompute sample-size using actual impression rates; adapt ramp durations to traffic.- Log detailed events to reproduce failures; include request/response latencies, feature vectors and ranking diffs for postmortem.- Communicate plan & rollback criteria with SRE/product/stakeholders before launch.This approach balances rapid detection (sequential monitoring, strict guardrails) with business sensitivity (statistical power sizing, conservative ramps) while minimizing user risk via shadowing, small initial exposure, and automatic rollback.
EasyTechnical
93 practiced
Define shadow traffic (shadow testing) and explain how you would use it to validate a new ranking model at Netflix without affecting user-facing responses. What are the benefits and what operational or privacy pitfalls should you watch out for when implementing shadow traffic?
Sample Answer
Shadow traffic (shadow testing) is sending live production requests in parallel to a new model/service without using its output for user-facing responses. The production system continues serving the current model; the shadow replica receives identical inputs so you can compare predictions, latency, and downstream metrics offline.How I’d use it at Netflix:- Mirror a sampled subset of real recommendation requests to the new ranking model in a controlled shadow environment.- Log inputs, model scores, ranked lists, inference latency, and counterfactual downstream signals (clicks, play starts) tied by anonymized request IDs.- Run automated A/B-style analyses and offline replay to estimate impact on engagement and revenue without affecting users.- Gradually increase traffic and configuration variations (feature toggles, feature ablations) before any live rollout.Benefits:- Realistic validation on live distribution of inputs and edge cases- Detects regressions in accuracy, latency, memory, and feature handling- Safe way to test instrumentation, feature drift, and integration with feature storesPitfalls and mitigations:- Performance overhead: shadowing increases compute and network load — sample traffic and autoscale shadow infra.- Data leakage/privacy: mirrored requests may contain PII. Anonymize or hash identifiers, apply the same privacy policies, and limit retention.- Observability gaps: ensure consistent logging and deterministic inputs so differences are attributable to the model, not sampling or preprocessing mismatches.- Downstream effects: avoid writing stateful side-effects (notifications, billing) from shadow runs.- Bias in sampling: choose representative sampling to avoid missing rare but important cases.Monitoring, strict access controls, and privacy-preserving logging are essential for safe, effective shadow testing.
HardSystem Design
83 practiced
Design a shadow testing system at Netflix scale where production traffic is forked to experimental models for validation. Explain how to ensure no side-effects, preserve user privacy, aggregate comparison metrics for offline analysis, and limit performance overhead. Also describe policies for sampling and retention of shadow logs.
Sample Answer
Requirements:- Fork a fraction of production requests to experimental models without affecting user-visible behavior (no side-effects).- Preserve PII and user privacy (HIPAA/GDPR-aware).- Collect deterministic comparison metrics for offline analysis at Netflix scale with minimal latency overhead.- Configurable sampling & retention policies.High-level architecture:- Edge proxy (API Gateway) → Primary model (prod) path + non-blocking Shadow Forker → Shadow Queue (Kafka with topic per experiment) → Shadow Inference Pool (autoscaled) → Secure Shadow Logs in object store + metrics pipeline (streaming aggregator).Key components & behavior:1. Shadow Forker: synchronous call to prod model; asynchronously clones request (tokenized/redacted) and metadata to Kafka. No writes allowed from shadow path; aborts/timeout never affect response to user.2. Shadow Inference Pool: runs experimental models in isolated containers, reading from Kafka. Inference is read-only; no downstream service calls. Resource limits (cgroups) prevent noisy neighbors.3. Privacy & PII: enforce automatic PII removal at fork time using field-level schema & deterministic hashing/salting for IDs. Use encryption-at-rest and access-controls; differential privacy or k-anonymity applied to aggregated outputs when required.4. Metrics & aggregation: stream metrics (latency, scores, top-k outputs) to a metrics aggregator (Spark/Beam) that computes pairwise comparisons, uplift, calibration drift. Use deterministic request IDs to join logs with offline labels.5. Performance constraints: fork is non-blocking; add lightweight sampling layer and adaptive rate limiter based on CPU/memory signals. Measure added latency (<1ms) on critical path; target <=1% total CPU overhead.6. Sampling policies: support stratified sampling (by region, user cohort, device), probabilistic sampling, and conditional sampling (e.g., only for rare events). Dynamic sampling rates controlled by experiment metadata.7. Retention & data lifecycle: raw shadow logs kept short (e.g., 7–30 days) in hot storage for debugging; aggregated/ anonymized metrics stored longer (months–years) with retention tiers. Automatic retention enforcement, legal hold exceptions, and periodic privacy audits.8. Security & governance: RBAC for experiment access, signed experiment manifests, audit logs, and automated checks to prevent experiments that attempt write actions or external calls.9. Validation & monitoring: health checks for shadow pool; alert on error rates divergence; rollout gates—start small, monitor metrics, increase traffic.Trade-offs:- Full-fidelity logs increase cost; mitigate via sampling + on-demand replay.- Stronger privacy (DP) reduces signal; balance by isolating non-PII features and using synthetic augmentation.This design ensures zero side-effects, strong privacy guarantees, scalable comparison metrics, controlled overhead, and clear policies for sampling and retention—enabling safe, reproducible model validation at Netflix scale.
HardSystem Design
85 practiced
Design a feature lineage and data provenance system integrated with your model registry and experiment tracking. Describe the data captured at dataset, feature, transformation, and model levels; APIs for querying lineage; storage model; and how this supports reproducibility, audits, and debugging for the Data Science and Legal teams.
Sample Answer
Requirements & constraints:- Capture immutable lineage for datasets, features, transformations, experiments, and models; support search/query; link to code, config, and artifacts; retain provenance for audits (legal) and reproducibility (DS); handle scale (TBs, many versions).- Low-latency queries for debugging; append-only storage for auditability; RBAC and retention policies.High-level architecture:- Ingestion hooks in ETL, feature store, model training pipeline, and model registry emit standardized lineage events to a Lineage Service (event queue → enrichment → lineage store).- UI + GraphQL/REST API + CLI + integrations (notebooks, CI/CD, feature store, model registry).Data captured- Dataset level: dataset_id, version/hash, source URI, ingestion timestamp, schema, row counts, sample hash, checksum, data-owner, access controls, upstream sources, ingestion job id, storage location.- Feature level: feature_id, feature_set_id, expression/SQL/code hash, feature version, input datasets, computation window, materialization schedule, statistics (nulls, cardinality), monitor thresholds, feature-store location.- Transformation level: transform_id, code (or reference to Git commit), dependencies (datasets/features), parameters, container image, runtime environment, execution logs, execution timestamp, deterministic flag, seed.- Model/experiment level: experiment_id, model_id, code repo + commit, hyperparameters, training dataset versions, features & versions used, preproc transforms (ids), training job metadata, random seeds, training metrics, artifact URIs, evaluation dataset versions, deployment snapshot.Storage model- Graph database (e.g., Neo4j or managed graph like AWS Neptune) for connected queries + append-only object store (S3) for artifacts and snapshots + searchable metadata store (Elasticsearch) for fast filters.- Write path: events -> enrichment (resolve IDs, compute checksums) -> persist nodes/edges in graph + metadata in search index + artifacts to object store. All changes are immutable; new versions create new nodes/edges.APIs- REST/GraphQL examples: - GET /lineage/dataset/{id}?depth=2 → returns upstream/downstream graph - GET /lineage/feature/{id}/history → versions + code hashes + stats - POST /lineage/query (DSL) → find models using a dataset version X within timeframe Y - GET /repro/{model_id} → returns reproducibility bundle (dataset versions, feature versions, code commit, container image, run script) - GET /audit/report?model={id}&period=... → compliance report (access, data-provenance, PII flags)- Notebook SDK to attach provenance inline and auto-capture cell->artifact mapping.How this supports reproducibility, audits, debugging- Reproducibility: a single reproducibility bundle reconstructs exact training environment (code commit, container, artifacts, dataset and feature versions, seeds). DS can re-run training deterministically.- Audits (Legal): immutable lineage graph + exportable compliance reports showing PII sources, data owner approvals, retention policy, and access logs. Graph allows demonstrable chain from model prediction back to source data.- Debugging: trace impact of upstream data drift by querying downstream models/features, inspect execution logs and statistics, compare feature versions; fast root-cause: “Which transformation changed cardinality?” or “Which models use feature X that changed on date Y?”Trade-offs & best practices- Use graph DB for relationship queries; replicate metadata to Elasticsearch for scale. Keep artifacts in object store to reduce DB size. Enforce strict schema for emitted events and require code references (git commit) for legal traceability. Monitor size of provenance—prune only after legal retention windows.Example quick flow- ETL job writes dataset v1 -> event emitted with checksum -> feature compute references dataset:v1 -> training job references feature:v2 + dataset:v1 and code@commit -> Lineage Service creates nodes/edges -> Repro API bundles artifacts and returns a single Docker command to re-run.This design balances DS needs for fast debugging/repro and Legal’s need for auditable, immutable provenance.
HardTechnical
64 practiced
You have strict SLOs and a fixed budget. Propose cost-optimization strategies that jointly reduce training, serving, and storage costs for thousands of models while keeping latency and quality SLOs intact. Include techniques like spot instances, mixed precision, distillation, caching, model tiering, and billing attribution per model.
Sample Answer
Requirements & constraints:- Maintain latency and quality SLOs for thousands of models- Fixed budget across training, serving, storage- Need per-model cost visibility for accountabilityHigh-level approach: reduce cost across three domains (training, serving, storage) with coordinated techniques so SLOs stay intact. Prioritize high-impact models for expensive resources and use lightweight options for low-impact ones.Training- Spot / preemptible instances for non-critical or retrainable workloads; checkpoint frequently and implement automated restart/resume to tolerate preemption.- Mixed precision (FP16 / bfloat16) and optimized libraries (NVIDIA apex / TensorFlow mixed precision) to cut GPU time ~1.5–2x with negligible quality loss.- Curriculum: schedule full retrains less frequently; use incremental / warm-start fine-tuning for frequent small updates.- Distillation: produce smaller distilled models from expensive teachers; use teacher only for periodic refreshes.- Training tiering: only top-N revenue/accuracy models get expensive architectures; others get cheaper architectures or transfer-learned variants.Serving- Model tiering & routing: maintain multi-tier model store (tiny, small, medium, large). Route requests via A/B rules: critical customers -> high-quality models; bulk low-priority traffic -> distilled or cached responses.- Caching: result-level cache for repeated queries; embedding cache for similarity searches; TTLs and cache invalidation tied to data drift signals.- Autoscaling + spot-backed inference nodes: use spot instances for stateless inference where latency SLOs permit small tail variability; reserve capacity for critical low-latency paths on on-demand instances.- Quantization & mixed-precision inference: int8 or FP16 where accuracy tests pass.- Batch inference for non-interactive workloads; asynchronous pipelines with retries for throughput efficiency.Storage- Cold/warm storage lifecycle: keep active model binaries and feature stores on fast SSDs; archive older checkpoints/models to cheaper object storage (S3 Glacier or equivalent) with automated restore policies.- Deduplicate artifacts (shared base checkpoints, parameter deltas) and use delta storage for fine-tuned variants.- Compress model artifacts and use content-addressable storage to avoid duplicates.Governance & billing attribution- Tagging and per-model metadata for compute/storage usage; enforce cost-aware CI/CD pipelines that record GPU-hours, inference-requests, storage-GB-days.- Implement chargeback/showback dashboards with cost-per-prediction and cost-per-accuracy-point metrics; use these to drive tier assignment and retire low-value models.- Budget guards: automated policies that throttle non-critical retrains or scale-down serving when forecasts hit budget thresholds.Metrics & validation- Monitor latency P95/P99, accuracy drift, cost per inference, cost per retrain. Setup safety guards: rollback or route to higher-tier model if SLOs breach.- Run offline A/B tests to validate distilled/quantized models maintain required quality before promotion.Trade-offs- Spot instances and aggressive quantization risk higher tail latency or slight accuracy loss—mitigated via hybrid reserve capacity, canary testing, and routing rules.- Distillation and tiering reduce cost but add governance overhead; justify via cost-per-utility metrics.This combined strategy reduces GPU-hours, storage footprint, and serving VM costs while keeping strict SLOs via tiering, validation gates, caching, and per-model billing to optimize decisions.
Unlock Full Question Bank
Get access to hundreds of ML Operations & Reliability at Large Scale interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.