Design, build, deploy, and operate end to end machine learning systems in production. Topics include data ingestion and validation, feature engineering and real time feature computation, training and testing pipelines, model serving and prediction latency optimization, scalability and reliability of infrastructure, and monitoring and observability for data and model performance. Covers detection and handling of data drift and model drift, retraining strategies and automation, versioning and reproducibility for data code and models, experiment tracking and model registries, and practices for continuous integration and continuous delivery in machine learning contexts. At senior and staff levels, expect system level trade offs, designing platform capabilities for multiple teams, debugging production performance regressions, and managing technical debt in machine learning systems.
HardTechnical
53 practiced
Design a production system to detect and mitigate adversarial inputs targeting a vision model. Describe detection approaches (input sanitization, anomaly detection, uncertainty estimation, ensembles), mitigation strategies (reject, fallback model, human review), continuous monitoring, and how to incorporate detected attacks back into model improvement cycles.
Sample Answer
Requirements & constraints:- Real-time vision model in production (low-latency), must detect adversarial inputs (evasion/poisoning), minimize false rejects, provide traceability and feedback to retraining pipelines.Detection approaches:1. Input sanitization- Preprocess inputs with randomized smoothing (pixel-level Gaussian blur + clipping) and JPEG compression to remove high-frequency adversarial perturbations. Run both original and sanitized images to compare logits.2. Anomaly detection- Train an autoencoder or Mahalanobis-distance detector on latent features (e.g., penultimate-layer activations). Flag inputs with high reconstruction error or large distance from class centroids.3. Uncertainty estimation- Use ensemble dropout or deep ensembles to estimate predictive entropy and mutual information; high epistemic uncertainty indicates out-of-distribution/adversarial inputs.4. Ensembles & consistency checks- Maintain a diverse ensemble (architectures, augmentations). Compute consistency score (agreement across models / augmentations). Low agreement triggers alert.Mitigation strategies:- Reject: if confidence < threshold OR anomaly score > threshold, return "unable to classify".- Fallback model: route to a robust, smaller model trained with adversarial training or certified defenses (e.g., randomized smoothing) for conservative predictions.- Human review / escalation: pipeline to queue flagged examples with provenance (input, saliency maps, model logits) for human labelers. Include rate limits and SLA.Continuous monitoring & ops:- Telemetry: log inputs, preprocess steps, ensemble outputs, uncertainty, timestamps, user/session metadata, and downstream impact metrics.- Real-time dashboards & alerts for spikes in anomaly rate, distribution shift, or degrade in calibration.- Canarying: A/B test new detectors/mitigations on shadow traffic before rollout.Feedback loop into model improvement:- Automatic triage: confirmed adversarial examples (human-labeled) go to a secured dataset store with attack metadata.- Retraining: augment training with adversarial examples (PGD, CW, common corruptions) and mix into continual learning schedule; use curriculum (start with constrained attacks).- Model validation: include adversarial robustness benchmarks, OOD tests, and held-out attack types. Track metrics: robust accuracy, false reject rate, calibration, latency.- Versioning & governance: track dataset/model lineage, approvals, and rollbacks. Periodic red-team exercises to discover new attack modes and update detectors.Trade-offs:- Latency vs robustness: ensemble/defense-in-depth increases cost/latency—use fast reject+asynchronous human review for borderline cases.- False positives: tune thresholds with business cost function (misclassification vs. delay).This design balances layered detection, conservative mitigation, operational monitoring, and a closed loop for continuous robustness improvement.
EasyTechnical
73 practiced
Outline steps to run an A/B test for a new ranking model intended to increase conversions. Define control/treatment, success metrics, sample size and power considerations, guardrail metrics, rollout policies, and how you'd avoid contamination or leakage between cohorts.
Sample Answer
1) Define objective & hypotheses- Objective: increase conversions (e.g., purchase rate).- Null: new ranking = no change; Alternative: conversion rate increases.2) Control / Treatment- Control: current production ranking model.- Treatment: new ranking model served to users identically except ranking logic.- Randomize at the user-id (or session) level depending on product semantics; persist assignment.3) Success & guardrail metrics- Primary success metric: conversion rate (e.g., purchases per user or per session) and/or conversion rate within X minutes.- Secondary metrics: click-through rate, average order value, revenue per user.- Guardrails (safety): engagement (time on site), latency, error rate, cancellation/return rate, business KPIs (gross merchandise value). Stop if guardrails degrade beyond pre-defined thresholds.4) Sample size & power- Choose minimum detectable effect (MDE) based on business value (e.g., 2–5% relative lift).- Use baseline conversion p0, desired power (80–90%), alpha (0.05 two-sided), compute sample with standard proportions formula or A/B sample size calculator. Inflate for multiple testing and expected exposure loss.- Consider experiment duration to capture seasonality and user revisit cycles.5) Rollout policy- Start with small % (e.g., 1%) for smoke test, run short sanity checks on guardrails, then ramp to 5%, 25%, 50%, 100% with monitoring windows between steps (e.g., 24–72h).- Use sequential testing controls or pre-registered analysis to avoid p-hacking.6) Avoid contamination/leakage- Randomize at the correct unit (user ID for persistent personalization). Persist assignment in a central service.- Ensure logging isolates impressions/clicks tied to model version.- Prevent cross-over by avoiding allocation changes mid-session; if user sees both variants, consider session-level randomization.- Validate telemetry (no duplicated events, consistent version tags).7) Analysis & decisions- Pre-specify analysis plan, metrics, segments.- Use ITT (intention-to-treat) for unbiased effect estimates.- Check heterogeneity by key segments; verify no data leakage.- If statistically and practically significant and guardrails pass, proceed with full rollout; otherwise iterate.This plan balances statistical rigor, business impact, and production safety.
MediumTechnical
67 practiced
Your team stores terabytes of training data in S3 and needs dataset versioning and reproducibility. Compare DVC, Delta Lake (or Iceberg), and MLflow artifacts for dataset versioning. Discuss pros/cons in storage overhead, metadata tracking, retrieval speed, compatibility with batch/streaming pipelines, and operational effort.
Sample Answer
Situation: We need dataset versioning and reproducibility for terabytes on S3. Below I compare DVC, Delta Lake/Iceberg, and MLflow Artifacts across storage overhead, metadata, retrieval speed, batch/streaming fit, and operational effort — with practical trade-offs for an ML engineer.DVC- Pros: Git-like pointers (small metafiles) referencing S3 objects; minimal local metadata; supports pipelines and reproducible experiments; easy to integrate into CI.- Cons: Storage overhead low (data stays in S3), but DVC stores checksums + refs; retrieval requires downloading full objects (no query engine) so training start-up can be slow for large files. Best for batch workflows. Operational effort: low–medium (manage remote storage, auth, DVC remotes), simple to adopt for teams used to Git.- Good when: datasets are file-oriented (images, parquet blobs) and you want experiment-level reproducibility without changing storage format.Delta Lake / Apache Iceberg- Pros: Table format with transactional metadata, partitioning, schema evolution, and time-travel. Metadata (parquet file lists, manifests) enables fast predicate pushdown, incremental reads, and efficient partial retrieval. Good for both batch and streaming (Iceberg/Delta support CDC/stream sinks).- Cons: Storage overhead: additional manifests/transaction logs (small compared to TBs). Retrieval speed excellent for columnar queries but requires compute engine (Spark, Presto, Flink) and may add cluster cost. Operational effort: higher — need to run metastore/transactional layer or use managed lakehouse; migrations and governance require more ops work.- Good when: you need table semantics, fast selective reads, streaming ingestion, and multi-team analytical access.MLflow Artifacts- Pros: Simple artifact storage (S3) tied to experiments/models. Good for storing fixed dataset snapshots used for a given run; integrates with MLflow tracking.- Cons: Not designed as a dataset-versioning system — lacks table/query semantics, no partitioned reads, metadata limited to run-level artifacts. Storage overhead: you often duplicate whole snapshots (higher costs). Retrieval: download full artifacts; slow at TB scale. Operational effort: low if MLflow is already used, but not scalable for large dataset management.- Good when: small-to-moderate sized dataset artifacts accompany models, or for reproducible examples.Recommendation (practical): - For large, queryable datasets and mixed batch/stream pipelines choose Delta Lake/Iceberg (lakehouse) despite higher ops; it unlocks fast selective reads and streaming. - For file-oriented datasets where teams want Git-like workflows and minimal infra changes, use DVC on top of S3. - Use MLflow Artifacts only for model-attached, limited-size dataset snapshots or as pointers to canonical dataset versions in DVC/Delta.Operational note: many teams combine approaches — maintain canonical tables in Delta/Iceberg for production serving and analytics, and use DVC to snapshot experiments’ input files (with MLflow storing pointers/IDs) to get both performance and experiment reproducibility.
EasyTechnical
71 practiced
Explain the difference between batch and streaming data ingestion in production ML systems. Describe typical use-cases, latency and consistency trade-offs, example technologies (e.g., Kafka, Kinesis, Google Pub/Sub, Airflow), failure/retry behavior, and when you would choose each approach for a user-event pipeline.
Sample Answer
Batch vs streaming ingestion:Batch ingestion collects and processes data in periodic windows (minutes–hours). Streaming (real-time) ingests and processes events continuously with low latency (milliseconds–seconds).Use-cases:- Batch: model training, nightly feature aggregation, large ETL jobs, analytical reports.- Streaming: real-time personalization, fraud detection, online feature updates, monitoring and alerts.Latency & consistency trade-offs:- Batch: higher throughput, simpler exactly-once semantics via bulk writes, eventual freshness; suitable when stale data (minutes/hours) is acceptable.- Streaming: low latency, supports near-real-time decisions, but handling ordering, duplicates, and exactly-once semantics is more complex; may require stateful processing and windowing.Example technologies:- Streaming: Kafka, AWS Kinesis, Google Pub/Sub; stream processors: Flink, Spark Structured Streaming, Kafka Streams.- Batch/orchestration: Airflow, Dataproc/EMR, Cloud Dataflow in batch mode, scheduled Spark jobs.Failure and retry behavior:- Batch: job-level retries, checkpointing via result snapshots, easier idempotency; failures roll back or re-run whole window.- Streaming: consumer offsets, persistent logs, at-least-once default; need deduplication, checkpointing, and state snapshots for exactly-once; retries are per-record or per-partition and must preserve ordering when required.When to choose for a user-event pipeline:- Use streaming when you need immediate responses (real-time recommendations, fraud blocking, session personalization) or to keep online features up-to-date.- Use batch when you can tolerate delay (daily user scoring, retraining, analytics) or when processing large historical windows is more efficient and simpler.- Hybrid: ingest events into a durable streaming layer (Kafka/PubSub), run real-time feature updates for latency-sensitive features and periodic batch jobs for heavy aggregations and model training—this balances freshness, reliability, and engineering complexity.
HardTechnical
68 practiced
For a recommendation system with rapidly changing user preferences, discuss when online learning (continual updates) is preferable to periodic batch retraining. Cover the stability-plasticity dilemma, label availability and delay, fairness concerns, monitoring and rollback, and engineering complexity versus business benefit.
Sample Answer
When user preferences change rapidly (e.g., news feeds, trending e-commerce, real‑time ads), online learning is preferable when the business gains from fast adaptation outweigh the risks and engineering cost. Below is a principled breakdown.When to choose online learning- Fast feedback loop: user signals (clicks, purchases, dwell time) arrive quickly and reliably. Low label delay means model can update with near-real feedback.- High volatility / concept drift: user tastes shift within hours/days (trending topics, seasonal bursts) and stale batch models degrade quickly.- Low latency to value: personalization improvements immediately affect engagement or revenue.Stability–plasticity dilemma- Trade-off: online updates increase plasticity (fast adaptation) but risk catastrophic forgetting and oscillation. Use mechanisms: learning rate schedules, replay buffers (experience replay), regularization (elastic weight consolidation), ensembling (short-term fast model + long-term stable model), or meta-learning to control plasticity.- Hybrid approach: continual small updates to a lightweight layer (bias or user embedding) while keeping core model stable.Label availability and delay- If labels are delayed (e.g., long conversion funnel), naive online updates amplify noise. Prefer batch retrain or use proxy signals for immediate updates and correct with delayed ground truth via importance weighting or delayed correction steps.- For sparse labels, aggregate over sessions or use bandit feedback with exploration strategies rather than supervised online gradient descent.Fairness and safety- Online updates can unintentionally amplify biases or feedback loops (popular items get more exposure). Mitigate with constraints (exposure caps), fairness-aware loss, periodic auditing, and maintain a frozen model shadow to compare distributional shifts.Monitoring and rollback- Production-grade monitoring is essential: drift detectors, cohort-level metrics, causal A/B tests, and shadow testing. Implement fast rollback: versioned models, feature-store snapshots, and automated gates (thresholds on business and stability metrics) to revert updates.Engineering complexity vs business benefit- Online systems add complexity: streaming pipelines, per-user state, rate-limited updates, and extensive testing. Use online learning when incremental revenue/engagement lift justifies this cost; otherwise, prefer frequent batch retrains (daily/hourly) or a hybrid (fast user embeddings online + periodic batch retrain of base model).Concrete pattern- Example hybrid: maintain a stable neural ranking model retrained nightly, serve per-user embeddings updated in real time via lightweight SGD with capped learning rate and periodic reconciliation to the batch model.Conclusion- Prefer online learning when labels are timely, drift is fast, and business impact is immediate; otherwise use hybrid or batch solutions with strong monitoring, fairness guards, and rollback capability.
Unlock Full Question Bank
Get access to hundreds of Production Machine Learning Systems interview questions and detailed answers.
