ML Systems Architecture & Components Questions

Design and architecture of production-grade machine learning systems, including data ingestion and preprocessing pipelines, feature stores, model training and validation pipelines, deployment and serving infrastructure, monitoring and observability, model governance, and platform-level concerns such as scalability, reliability, security, and integration with product systems.

EasyTechnical

149 practiced

You need to A/B test two ranking models on an e-commerce site for 2% expected lift. Sketch an experiment design: traffic split, duration estimation, key metrics (primary and secondary), instrumentation needs, and how you would handle metric noise and novelty bias.

Sample Answer

Requirements & goal: detect a 2% relative lift in conversion (or revenue-per-visitor) from new ranking model with high confidence (α=0.05, power=0.8).

Traffic split:- Randomized user-level split 50/50 (A: control ranking, B: treatment ranking). Use user-cookie or logged-in user id to avoid session-level churn. If risk of cross-contamination, consider 60/40 to preserve control traffic.

Duration & sample-size estimate:- Estimate baseline metric mean μ and std σ (e.g., conversion rate ~2%, use historical std of per-user conversion or per-session revenue).- Use standard two-sample t-test / proportion power calc. For small lift (2%) and low base rate, you’ll likely need weeks and O(100k–1M) users per arm. Example formula for proportions: n ≈ 2 * (Z_{1-α/2}+Z_{power})^2 * p(1-p) / (Δp)^2- Run for at least one full business cycle (2–4 weeks) to cover weekday/weekend patterns.

Key metrics:- Primary: revenue-per-visitor (RPV) or conversion rate (choose business-aligned KPI).- Secondary: average order value, click-through-rate, engagement (time on page), add-to-cart rate, bounce rate, downstream retention/return rate, and negative safety metrics (page load, errors).

Instrumentation needs:- Consistent user assignment logging, timestamped exposure events, model-version tags, ranked-list telemetry (positions, item IDs), downstream events (clicks, add-to-cart, purchases), business context (prices, promotions), and experiment metadata.- Ensure deterministic seeding for ranking to reproduce results.- Capture funnel attribution windows (same-session vs 7/30-day conversions).

Handling metric noise:- Use guardrail metrics and variance reduction techniques: stratified randomization, blocking by user cohort, and CUPED (use pre-experiment covariates to reduce variance).- Use robust statistical tests (bootstrapping for revenue) and report confidence intervals, not only p-values. Pre-define minimum detectable effect and stop rules; avoid peeking without correction (alpha spending or sequential tests).

Handling novelty bias:- Run a cooldown period: ignore first N exposures per user (e.g., first week) when users are exploring new results.- Analyze lift over time slices (day 1, 2–7, week 2+) to separate immediate novelty from long-term effect.- If novelty is large but transient, prefer long enough experiment to observe stabilization or run holdout period after treatment to measure persistence.

Decision rules:- Accept treatment if primary metric meets pre-defined MDE with statistical significance and no adverse guardrail violations; otherwise reject or iterate.

EasyTechnical

92 practiced

List the core monitoring/observability signals you would capture for a production classification model serving 10k QPS. Include metrics, logs, traces, and data quality checks. Explain why each signal matters and a simple alerting rule example.

Sample Answer

Metrics (aggregated at 1s/1m and per-model-instance):- QPS / throughput — detect traffic drops/spikes (matters for capacity & pricing). Alert: QPS < 5000 for 5m or >15000 for 1m.- Latency (p50/p95/p99, tail) — user experience and SLO compliance. Alert: p99 latency > 200ms for 5m.- Error rate (5xx, serving exceptions) — failures, model crashes. Alert: error_rate > 0.5% for 2m.- Model accuracy / top-k accuracy / calibration (if ground truth available) — correctness drift. Alert: accuracy drop > 5% vs baseline for 1h.- Model confidence distribution (mean, variance) — detect over/underconfidence or data shift. Alert: mean confidence decrease > 10% vs rolling 24h.- Input feature stats (counts, means, histograms) — detect feature distribution shift. Alert: KL divergence(feature X, baseline) > threshold for 1h.- Prediction distribution (class frequencies) — label/target drift or upstream bugs. Alert: class frequency change > 20% for 30m.- Resource metrics (CPU, GPU, memory, GPU util) — capacity and OOM prevention. Alert: GPU mem used > 90% for 5m.

Logs:- Structured request/response logs (request id, model version, input hashes, prediction, confidence, timestamps) — debugging, replay, auditing.- Error stack traces with context — root-cause analysis.- Inference sampling logs (sampled raw inputs) — human inspection and repro.

Traces:- Request traces across components (API gateway → preprocess → model → postprocess → DB) with spans and timings — pinpoints latency hotspots and retry loops.- Distributed trace IDs tied to logs and metrics — navigate from metric alert to concrete request.

Data quality checks:- Schema validation (types, required fields, missing rate) — reject/buffer bad requests. Alert: missing_rate(feature) > 1%.- Outlier detection per-feature (z-score, IQR) — catch anomalous inputs. Alert: outlier_rate > 0.5%.- Completeness and freshness (lag in labeling pipeline) — ensure ground truth for retraining. Alert: label lag > 24h.- Duplicate detection / replay detection — prevents double-counting.

Why each matters: metrics show system health and SLA/SLO compliance; logs/traces give context to reproduce and fix; data checks detect upstream problems and model drift before user impact.

Example combined alert pattern:- Severity P1: p99 latency > 500ms OR error_rate > 2% OR GPU mem > 95% → Page on-call, rollback traffic.- Severity P2: accuracy drop > 5% OR KL divergence(feature) > threshold → Notify ML team, trigger sampling & investigation pipeline.

Implement alerts with rate windows, noise suppression (prediction of transient spikes), and include runbook links and relevant trace/log links in alert payload.

MediumTechnical

81 practiced

Explain how a feature store supports offline training and online serving consistency using point-in-time correct joins. Provide an example of how you would guarantee point-in-time correctness for user features computed daily when training a model for historical timestamps.

Sample Answer

A feature store ensures offline/online consistency by storing feature values with explicit timestamps (the time the feature value was valid, e.g., feature_event_ts) and metadata (write/ingest time or created_ts). Point-in-time correct joins select, for each training example timestamp, the most recent feature record whose event_ts (and optionally created_ts) is <= that training timestamp—this prevents leakage from future data and matches what the online store will expose at serving time.

Example approach and SQL pattern (user daily aggregated features):- Store features in a table user_daily_features(user_id, feature_date, feature_value, feature_event_ts, created_ts).- Store labels/events in events(user_id, label_ts, label).

To guarantee point-in-time correctness for each historical label_ts, join using the latest feature_event_ts <= label_ts and ensure created_ts <= label_ts (protects against late-arriving corrections):

sql

SELECT e.user_id, e.label_ts, e.label, f.feature_value
FROM events e
LEFT JOIN LATERAL (
  SELECT feature_value
  FROM user_daily_features f
  WHERE f.user_id = e.user_id
    AND f.feature_event_ts <= e.label_ts
    AND f.created_ts <= e.label_ts
  ORDER BY f.feature_event_ts DESC
  LIMIT 1
) f ON true;

Why this works:- feature_event_ts ensures you only use information that would have been known by label_ts.- created_ts prevents using corrections that were created after label_ts (important if your ingestion pipeline backfills or fixes historical data).- Using LATERAL or window functions yields the latest valid row per (user,label_ts).

Operational notes to ensure online/offline parity:- Materialize the same feature logic into online store with identical keys and retention so a live request at time T reads the latest feature with event_ts <= T.- Version feature transformations and store code; use immutable feature views or snapshots for experiments.- Provide time-travel/backfill tools so offline training uses the same materialized aggregates as production, with consistent TTLs and monitoring for late arrivals.

Edge cases:- No matching feature row: define default or indicator for missing.- High-latency backfills: record lineage and prefer created_ts constraints.- Performance: precompute daily aggregates and use partitioning/indexing on (user_id, feature_event_ts) for efficient joins.

MediumSystem Design

80 practiced

You must integrate SHAP-based explanations into a low-latency real-time scoring API. Design an architecture that provides per-request explanations without adding >30ms latency to the 95th percentile of requests. Include caching, approximation strategies, and offline precomputation considerations.

Sample Answer

Requirements & constraints:- Per-request explanation (SHAP-like) must not add >30ms to p95 latency.- Real-time scoring API (high QPS), model may be tree-based or NN.- Explainability fidelity should be acceptable; system should allow full offline explanations.

High-level architecture:- Real-time scoring service (stateless containers behind LB) -> returns prediction + fast explanation.- Explanation service (co-located light-weight or sidecar) -> provides quick approx explanation (<=30ms).- Background/exact explanation pipeline (batch jobs, GPU cluster) -> computes full SHAP values asynchronously and populates caches/DB.- Distributed cache (Redis or memcached) + nearest-neighbor index (FAISS) for cached explanations.- Monitoring + fallback.

Core components & strategies:1. Fast-path approximations (inline, p95 ≤ 30ms)- If model is tree-based: use TreeSHAP optimized C++ implementation (very fast) as first choice.- If model is NN: use a pre-trained surrogate (local linear model or small explainable model) that approximates predictions in local regions; compute SHAP on surrogate (fast).- Use low-sample SHAP: limit Monte Carlo samples (e.g., 8–32) for KernelSHAP-like approximations for NNs to meet latency, calibrated offline.- Precompute and store feature attributions for quantized feature space (see next).

2. Caching & nearest-match:- Deterministic cache key: hash of quantized/rounded feature vector or hashed bucket ID. For common inputs, return precomputed SHAP instantly.- Nearest-neighbor lookup: for unseen inputs, query FAISS over offline precomputed embeddings (e.g., representation from penultimate layer). Return nearest cached explanation and adjust (weighted interpolation) — often high fidelity with little compute.- LRU + TTL policy; item size limited by top-K features only to save memory.

3. Offline precomputation & indexing:- Batch compute exact SHAP values for representative dataset (stratified by segments) using full SHAP with many samples/GPU.- Store: feature vectors, exact SHAP vectors, embeddings, buckets. Build FAISS/Annoy index on embeddings.- Train surrogate explanation models per bucket (local surrogates) and store.

Data flow per request:1. Score request arrives; model returns prediction and embedding.2. Sidecar queries cache by exact quantized key. If hit, return cached SHAP.3. If miss, query FAISS for top-N nearest precomputed examples; interpolate cached SHAPs -> return as quick approx (≤30ms).4. If approximation unacceptable (confidence low), fallback to low-sample SHAP on surrogate or TreeSHAP (ensure still within latency envelope) or queue for async exact computation; mark response with approximation/confidence.

Scalability & performance:- Cache hit and NN lookup are O(1)/O(log n) and meet latency; FAISS tuned for sub-ms p95 with in-memory indices.- Precompute scales horizontally on GPU cluster; incremental updates for model drift.- Use batching for any online SHAP computation; colocate GPUs for heavier cases.

Trade-offs:- Approximate SHAP vs exact: reduce compute and latency at cost of fidelity. Mitigate by measuring fidelity offline and exposing confidence scores.- Cache / quantization reduces accuracy for rare inputs.- Surrogate models require retraining on model updates.

Monitoring & validation:- Track explanation latency (p50/p95), cache hit rate, approximation error (periodically compute exact SHAP for sample of live requests).- Automated alerting when fidelity drops below threshold; retrain surrogates / refresh cache.

This design provides per-request SHAP-like explanations under tight latency by combining native fast implementations (TreeSHAP), approximations (surrogates, low-sample SHAP), caching, and nearest-neighbor reuse of precomputed exact values, with a background pipeline ensuring high-fidelity data and continuous validation.

MediumTechnical

74 practiced

You need to reduce training costs for nightly model retraining using GPUs. Propose a strategy combining spot/interruptible instances, mixed-precision training, gradient accumulation, and job scheduling to save costs while maintaining training reliability. Include expected trade-offs.

Sample Answer

Approach: build a resilient, cost-aware nightly retrain pipeline that prefers spot/interruptible GPUs but falls back to on-demand when needed; reduce GPU time per job with mixed-precision and gradient accumulation; and use smart scheduling and checkpoints to tolerate interruptions.

Design:- Spot-first pool: submit training jobs to a fleet of spot/interruptible GPU instances (diverse AZs and instance types) with aggressive automatic checkpointing (periodic model + optimizer + RNG state).- Checkpointing + resume: save checkpoints every N minutes/steps and on preemption signal so jobs can resume on any instance. Use cloud provider’s preemption notices (e.g., 2-minute warning) to trigger final flush.- Mixed-precision: enable AMP (PyTorch/TensorFlow) to reduce memory and runtime; validate numeric stability with loss scaling and a small validation run nightly.- Gradient accumulation: for large effective batch sizes on smaller/cheaper GPUs, accumulate gradients over k steps so you can use smaller instances without hurting optimization dynamics.- Scheduler: orchestrator (Kubernetes + Kube-batch, Ray, or managed ML job scheduler) that: - prioritizes spot nodes, fits jobs to available resources, - enforces deadlines (e.g., soft deadline for spot; if deadline nears and job not complete, migrate to on-demand to guarantee completion), - autoscaling pools for spot vs on-demand.- Monitoring & fallback: track preemption rates; if spot instability rises beyond threshold increase on-demand fallback ratio.

Trade-offs:- Cost savings: typically 50–80% vs all on-demand depending on spot availability.- Reliability: added complexity (checkpointing, resume, scheduler) and small risk of increased wall-clock time due to preemptions and restarts.- Performance: mixed-precision may need extra validation; gradient accumulation increases CPU/GPU synchronization overhead and slightly increases memory for accumulators and may change training dynamics (requires tuning of learning rate / warmup).- Engineering effort: moderate upfront engineering to implement robust checkpointing and orchestration.

Metrics to monitor: cost per nightly run, time-to-completion, preemption frequency, final model metrics (validation loss/AUC) to ensure no accuracy regression.

Unlock Full Question Bank

Get access to hundreds of ML Systems Architecture & Components interview questions and detailed answers.

Join thousands of developers preparing for their dream job.