Machine Learning System Architecture Questions

Design and operational reasoning for end to end machine learning systems covering the full lifecycle from data sources to production serving and maintenance. Key areas include data ingestion and integration, storage choices such as data lakes and data warehouses, data validation cleaning and preprocessing, feature engineering and feature store design, experiment tracking and training infrastructure including distributed training and hyperparameter tuning, model validation evaluation explainability and fairness considerations, model packaging and model registry practices, deployment and serving architectures for batch online streaming and edge inference, monitoring and observability for data quality model performance and drift detection, feedback loops and automated retraining pipelines, model versioning rollback and controlled rollout strategies, and testing continuous integration and continuous delivery for models. Candidates should be able to explain data flow between components choose between batch and real time patterns reason about trade offs among latency throughput cost reliability and accuracy identify bottlenecks and failure modes propose mitigation strategies and name common architectural patterns operational practices and tooling used to build robust scalable and maintainable machine learning pipelines.

EasySystem Design

20 practiced

For a content-recommendation system, describe three inference serving patterns: (1) batch scoring for offline features, (2) synchronous online RPC for real-time ranking, and (3) streaming enrichment for event-driven personalization. For each pattern, explain expected latency/throughput, storage choice, and suitable use-cases.

Sample Answer

**Batch scoring — offline features**

- Description: Periodic (e.g., nightly) job computes heavy features or candidate scores for all users/items using full models (e.g., large GBDT, deep CF). Results materialized into stores.- Latency / throughput: High latency (minutes–hours) but very high throughput (processes millions/billions in a job).- Storage choice: Columnar data lake + OLAP (Parquet on S3, BigQuery) for raw output; serving cache (Redis, Cassandra) for fast lookup of precomputed embeddings/scores.- Use-cases: Cold-start candidate generation, long-term personalization signals, offline metrics, A/B evaluation and training label generation.

**Synchronous online RPC — real-time ranking**

- Description: Low-latency RPC service that computes final ranking with lightweight real-time features (session signals, freshness) using compact models (e.g., logistic/MLP or distilled ranker).- Latency / throughput: Very low latency (1–50 ms p99), moderate throughput (thousands–tens of thousands QPS); autoscaled.- Storage choice: In-memory feature stores (Redis/Memcached), feature vector stores (Faiss for nearest neighbors), model in-process or via fast model server (TorchScript/TensorRT).- Use-cases: Final re-ranking at request time, session-aware personalization, exploration/exploitation balancing.

**Streaming enrichment — event-driven personalization**

- Description: Real-time stream processors (Kafka + Flink/Beam) enrich event streams with features (user state, incremental aggregates) and update materialized views or trigger downstream ML inference.- Latency / throughput: Sub-second to seconds latency, very high throughput (streaming QPS), bounded processing time.- Storage choice: Stream-backed state stores (RocksDB via Flink), scalable key-value stores for incremental state (DynamoDB/Redis), event logs (Kafka).- Use-cases: Real-time counters, TTL-based user features, trend detection, immediate personalization (e.g., recent clicks influence next recommendations), online feature pipelines for online learning.

Trade-offs: balance freshness vs. cost/latency; combine patterns (batch candidates + online re-rank + streaming updates) for best accuracy and performance.

EasyTechnical

17 practiced

List the minimum set of metrics and signals you would monitor right after deploying a regression model to production. Include both data-quality signals and model metrics, and describe what thresholds or alerting rules you would configure to surface problems early without causing alert fatigue.

HardTechnical

18 practiced

Design an online concept-drift detection algorithm with a controlled false positive rate for continuous streams. Describe the statistical test, adaptive windowing strategy, threshold selection approach, and provide Python-style pseudocode for updating statistics and triggering alerts. Discuss computational and memory complexity and how you'd tune sensitivity.

Sample Answer

**Approach (brief)** Use an adaptive-window online test that maintains two adjacent windows (recent W_r and reference W_ref). Compare distributions with a streaming two-sample statistical test based on empirical CDF (streaming KS) or mean/frequency with Bernstein/Hoeffding bounds to guarantee controlled false positive rate α. Adapt window size like ADWIN: shrink when a statistically significant difference is detected, otherwise grow.

**Statistical test & thresholding** - Use a streaming two-sample KS-statistic D = sup_x |F_r(x)-F_ref(x)|. For window sizes n_r, n_ref set threshold using asymptotic/finite-sample bound with controlled α via:

text

threshold = sqrt( -0.5 * log(alpha/2) * (n_r + n_ref) / (n_r * n_ref) )

Plain-English: threshold scales inverse with sqrt(sample size); choose alpha to control FPR. For bounded real-valued features, you can instead use Bernstein/Hoeffding on mean/frequency to get nonparametric guarantees with known bounds.

**Adaptive windowing strategy** - Maintain a buffer, grow W_r one sample at a time. After each append, test all splits of buffer into W_ref | W_r (or use geometric reductions) to find earliest split where test passes; if found, cut buffer at split and declare drift at split point (ADWIN-style).

**Pseudocode (Python-style)**

python

# streaming buffers store sufficient stats (ECDF sketches or histograms)
from collections import deque
W = deque()          # full buffer
max_buffer = MAX_N

def add(x):
    W.append(x)
    if len(W) > max_buffer: W.popleft()
    check_drift()

def check_drift():
    # consider splits; for efficiency test geometric splits
    for split in candidate_splits(W):
        ref = W[:split]; recent = W[split:]
        D = ks_statistic_stream(ref, recent)   # or mean diff with bernstein_bound
        thr = ks_threshold(len(ref), len(recent), alpha)
        if D > thr:
            alert_drift(split)
            # shrink buffer to recent window after split
            while len(W) > len(recent): W.popleft()
            return

**Complexity & memory** - Exact ECDF comparisons cost O(n) per test; testing many splits is O(n^2). Use sketching (q-digests, KLL, histograms) and geometric splits to reduce to O(n log n) or O(log n) per update with memory O(k) where k is sketch size. Mean/frequency tests with incremental sums are O(1) per update, O(1) memory per feature.

**Tuning sensitivity** - Control α to trade FPR vs detection delay. Use bootstrap or held-out stationary stream to calibrate α and sketch resolution. For faster detection, reduce sketch granularity or use smaller min-window sizes; to reduce false alarms, increase alpha stringency or require consecutive positive detections before alert.

Notes: prefer Bernstein bounds when bounded variance known; use multidimensional tests by combining per-feature alarms with Bonferroni or false-discovery control.

HardSystem Design

19 practiced

Design training infrastructure to train a 100-billion-parameter transformer model. Discuss parallelism strategies (data/model/pipeline), parameter sharding approaches (e.g., ZeRO), optimizer checkpointing and offloading, network bandwidth needs, checkpoint frequency, fault tolerance and restart strategy, and cost-reduction techniques (mixed precision, gradient accumulation). Provide a high-level architecture and trade-offs.

Sample Answer

**Clarify requirements & constraints**- Train a 100B-parameter transformer (FP32 ≈ 400 GB weights). Target throughput, budget (cloud vs on-prem), acceptable time-to-solution, and SLAs for checkpoint recovery.

**High-level architecture**- Multi-node GPU cluster (A100/H100); 8–16 GPUs/node; NVLink intra-node, 100–400 Gbps RDMA inter-node.- Use PyTorch + DeepSpeed / Megatron-LM for integration of model/data/pipeline parallelism and ZeRO.

**Parallelism strategy**- Model parallelism: tensor (operator) parallelism across GPUs within a node to split large layers (reduce per-GPU memory and FLOPs).- Pipeline parallelism: split transformer layers into stages across nodes to increase utilization; combine with micro-batching and activation checkpointing.- Data parallelism: sharded data-parallel across pipeline/model slices for gradient aggregation.

**Parameter sharding & ZeRO**- Use ZeRO stage 2/3: shard optimizer states, gradients, and parameters across data-parallel ranks. Stage 3 stores full parameters sharded — essential to fit 100B on fewer GPUs.- Combine with NVIDIA/Azure MVAPICH or NCCL for efficient all-gather during forward/backward.

**Optimizer checkpointing & offloading**- Offload optimizer states/parameters to CPU NVMe when memory limited (ZeRO-Offload). Keep hot tensors (current stage params) on GPU.- Use incremental/streamed checkpoints: shard checkpoints per rank (small files) and optional consolidated checkpoint step for archival.

**Network & checkpoint frequency**- Network: 100–400 Gbps recommended; higher reduces all-reduce and all-gather stalls. Monitor network saturation.- Checkpoint frequency: trade reliability vs. overhead. For long jobs, periodic full checkpoints (e.g., every 1–4 hours) plus lightweight metadata heartbeats. Use delta/incremental checkpoints if training steps large.

**Fault tolerance & restart**- Use rank-level restart with checkpoint inventory; design checkpoint atomicity and consistency via a coordinator (e.g., Redis / S3 manifest).- Fast restart: load sharded checkpoint in parallel; use elastic training (add/remove nodes) if supported.- For transient failures, retry logic at rank level; for node loss, rely on saved checkpoint and reallocate cluster.

**Cost-reduction techniques**- Mixed precision (AMP / BF16): reduces memory and speeds up compute with minimal loss in convergence.- Gradient accumulation: increase effective batch size without more GPUs.- Activation checkpointing: recompute activations to save memory.- Use larger GPU instances for fewer nodes to reduce cross-node comms; preemptible/spot instances with frequent checkpoints to lower cost.

**Trade-offs**- More model parallelism lowers memory per-GPU but increases latency from communication and complexity in implementation.- ZeRO stage 3 minimizes memory but increases I/O for checkpoints and network all-gathers.- Offloading saves GPU memory but adds PCIe/NVMe and CPU overhead.- Pipeline increases utilization but complicates micro-batch tuning and bubble handling.

**Operational notes**- Monitor perf counters: NCCL, PCIe, GPU SM/util, GPU memory.- Automate checkpointing, manifest, and validation; run small-scale smoke tests before full run.- Aim for a balance: ZeRO-3 + tensor parallelism + pipeline + mixed precision + gradient accumulation is common sweet spot for 100B models.

MediumTechnical

17 practiced

Compare grid search, random search, Bayesian optimization, Hyperband, and population-based training for hyperparameter tuning at production scale. For each method detail parallelism capabilities, resource efficiency, how they handle noisy objectives, and scenarios where you would prefer one approach over another.

Sample Answer

**Overview — quick summary**- Grid: exhaustive, deterministic. Random: stochastic uniform sampling. Bayesian (e.g., GP/Tree-structured): model-based sequential; can be batched. Hyperband: multi-fidelity bandit (early-stopping). PBT: asynchronous evolutionary + online adaptation of weights and hyperparams.

**Parallelism**- Grid: trivially parallel (embarrassingly parallel). Scales to many workers but expensive.- Random: trivially parallel, near-linear scaling.- Bayesian: inherently sequential; batch/asychronous variants (q‑EI, BOHB) allow limited parallelism with diminishing info per eval.- Hyperband: good parallelism across brackets and trials; early stopping frees resources.- PBT: highly parallel and asynchronous; population members run concurrently with occasional sync.

**Resource efficiency**- Grid: poor — wastes compute on unpromising combos.- Random: better than grid, especially in high-D spaces.- Bayesian: sample-efficient (fewer full runs) but overhead in modeling; struggles with high noise/dimensions.- Hyperband: high efficiency via multi-fidelity; saves compute by killing bad runs early.- PBT: efficient for nonstationary/online tuning and models benefiting from warm-starts; higher overhead for management.

**Noisy objectives**- Grid/Random: robust but need many repeats.- Bayesian: sensitive to noise; needs noise-aware surrogate or repeated evaluations.- Hyperband: noise can cause premature stopping; mitigated by more bracket redundancy or successive halving parameters.- PBT: resilient — population diversity and ongoing perturbation smooth noise but can propagate noisy lucky runs.

**When to prefer**- Grid: small discrete spaces, sanity checks, reproducibility.- Random: quick baseline, high-dim spaces, massively parallel cheap resources.- Bayesian: expensive evaluations, low-dim/moderate-dim, need sample efficiency.- Hyperband/BOHB: when partial-training fidelity is available (e.g., epochs) and you need efficiency at scale.- PBT: large-scale deep learning where online adaptation helps (LR schedules, augmentation), long training runs, nonstationary objectives.

Practical tip: combine methods (e.g., BO for search space + Hyperband or PBT for budgets) and instrument robust logging/seeded repeats for production reliability.

Unlock Full Question Bank

Get access to hundreds of Machine Learning System Architecture interview questions and detailed answers.

Join thousands of developers preparing for their dream job.