ML Systems Architecture & Components Questions
Design and architecture of production-grade machine learning systems, including data ingestion and preprocessing pipelines, feature stores, model training and validation pipelines, deployment and serving infrastructure, monitoring and observability, model governance, and platform-level concerns such as scalability, reliability, security, and integration with product systems.
EasyTechnical
94 practiced
Explain with examples the difference between concept drift and data drift. For each type of drift propose at least two detection methods and one mitigation strategy that could be automated in production (e.g., retrain, feature engineering, cohort-specific models).
Sample Answer
Concept drift vs data drift (short):- Data drift (covariate drift): distribution of input features P(X) changes over time while P(Y|X) stays the same. Example: sensor calibration shifts, suddenly more mobile users (feature: device_type).- Concept drift (label/target drift): the relationship between inputs and label P(Y|X) changes. Example: user behavior changes so the same features predict churn differently (seasonal promotion changes purchase drivers).Detection methods — Data drift (>=2):1. Population Stability Index (PSI) on numeric features: compare baseline vs current buckets; PSI > 0.2 indicates meaningful drift.2. Two-sample statistical tests: Kolmogorov-Smirnov (KS) for continuous features, Chi-square for categorical features.3. Unsupervised distances: monitor feature centroid/embedding shifts via Mahalanobis or KL divergence on feature embeddings.Detection methods — Concept drift (>=2):1. Monitor model performance over time (AUC, accuracy, calibration): statistically significant drop signals concept drift.2. Label-conditioned drift: compare conditional distributions P(X|Y) or use Performance Degradation + Retraining Delay analysis (monitor error residuals for systematic bias).3. Drift detectors on residual stream: ADWIN or Page-Hinkley for online change detection on prediction errors.Automated mitigation strategies (one per drift type, production-ready):- Data drift mitigation (automated): feature revalidation + dynamic preprocessing pipeline. On detected drift, trigger automated feature transformations (re-scaling, outlier handling), update feature normalization parameters, and route affected cohorts to a fallback model or cohort-specific submodel. This can be automated via a CI/CD job that updates feature stores and model input schema.- Concept drift mitigation (automated): automated retrain-and-deploy pipeline. On sustained performance drop (with statistical significance and minimum sample size), kick off continuous training using recent windowed data, validate against holdout and canary-deploy the new model; if new model passes gates, promote to production. Optionally include ensemble weighting that gradually favors recent models.Notes/Best practices:- Combine detectors: data drift alone shouldn’t trigger full retrain; require performance corroboration to avoid unnecessary churn.- Use cohort monitoring: detect drifts affecting subpopulations and apply targeted retrain/cohort-specific models rather than global changes.
HardSystem Design
71 practiced
Design a production-grade ML platform that supports training and serving models across multiple geographic regions while keeping user-facing inference latency under 100ms. Address data replication strategies, model artifact distribution, inference routing and caching, regional feature stores vs global store, failover, compliance with data residency, and cost balancing.
Sample Answer
Requirements & constraints:- Functional: train and serve models across N regions; <100ms user-facing tail latency.- Non-functional: data residency, high availability, cost-efficiency, model consistency, fast deployment.High-level architecture:User → Edge CDN + Global LB → Regional Ingress → Regional API Gateway → Regional Inference Cluster (GPUs/TPUs + autoscale) + Redis/LRU cache → Feature Store(s) → Monitoring & A/B rollout controlTraining Plane: Centralized/global training control with regional trainer workers; Artifact registry + CDN for model distribution.Data replication & residency:- Partition raw user data by legal region. Use write-local, replicate-as-allowed: synchronous local write; asynchronous cross-region replication only for aggregated/allowed datasets.- Global aggregated datasets live in a controlled analytics region with consent; PII tokens remain region-bound.- Use change-data-capture (CDC) + Kafka/GeoMirror for stream replication with filtering and encryption.Feature stores: hybrid- Regional feature stores for low-latency online features (co-located with inference), updated via local stream processors.- Global offline feature store for training, cross-region analytics, and global features constructed under residency policy.Model artifact distribution:- Central artifact registry (signed Docker / ONNX / TorchScript artifacts) + geo-replicated object storage/CDN.- Canary rollout orchestrated by orchestration service (Argo/Spinnaker) which pushes artifacts to regional caches and warms instances.Inference routing & caching:- Use Geo-aware DNS + Anycast LB to route to nearest region.- Edge inference for ultra-low-latency models (small models) deployed at CDN/edge functions.- Regional inference clusters for medium/large models.- Multi-layer cache: edge (CDN) for deterministic responses, regional Redis for hot feature/model outputs, and local L1 in-process cache.- Use adaptive request routing: if regional load high, route to secondary within latency budget; include SLO-aware fallback to smaller distilled model.Failover & consistency:- Active-active serving per region with health probes; stateful data (online features) replicated via async streams and reconciled via periodic checks.- Failover policy: prefer local; if degraded, route to next nearest with SLA telemetry. For strict data residency, failover may be denied—use policy evaluation.Compliance & security:- Data tagging with residency metadata; enforcement via policy engine preventing cross-region movement of protected data.- Encryption at rest and in transit; KMS keys per region; audit logs and consent records.- Provide regional model explainability & data lineage for audits.Cost balancing:- Mix spot/preemptible instances for non-critical training; reserved for steady-serving baseline.- Autoscale with predictive scaling using traffic forecasting.- Model distillation: serve cheaper distilled models for most traffic and only run heavyweight models for high-value requests.- Monitor cost per inference and adjust routing thresholds.Observability & Ops:- End-to-end tracing, p95/p99 latency SLOs per region, model drift monitors, automated rollback.- CI/CD pipelines for model tests, fairness & privacy checks before cross-region deployment.Trade-offs:- Strong consistency vs latency: choose eventual consistency for online features to meet latency goals.- Data-residency strictness increases cost and complexity—mitigate with localized processing and minimal allowed replication.This design ensures sub-100ms inference via geo-routing, edge/regional caches, and model tiering while honoring residency, enabling scalable training and controlled artifact distribution.
EasyTechnical
150 practiced
Describe the trade-offs between computing features on-request (online) versus precomputing them in batch. Discuss latency, storage cost, operational complexity, freshness, and examples where each is preferred. Explain a hybrid architecture that supports both styles and how you would implement cache invalidation and TTLs.
Sample Answer
Computing features on-request (online) vs. precomputing in batch has clear trade-offs:Trade-offs- Latency: Online computes per request → higher inference latency (depending on compute), but can deliver perfectly up-to-date values. Batch yields near-zero read latency since values are read from a store.- Freshness: Online = freshest (real-time); Batch = stale between runs (minutes → hours). Frequency of batch jobs controls staleness.- Storage cost: Batch increases storage (materialized feature store, indices); online needs less storage but may require heavier compute resources or feature-serving microservices.- Operational complexity: Online requires reliable low-latency compute paths, autoscaling, and more integration complexity. Batch requires scheduling, ETL pipelines, data lineage and idempotency handling.- Predictability & repeatability: Batch features are easier to reproduce for training and debugging; online adds nondeterminism unless carefully logged.When to prefer each- Online: user-personalized scores that must reflect recent behavior (real-time fraud signal, session-level features, streaming embeddings).- Batch: heavy aggregation across large history (30-day counts), offline model training features, features that change slowly.Hybrid architecture- Materialized feature store + online feature server. Precompute historical/expensive aggregates in batch and store them (feature DB / Redis for low-latency reads). Compute only highly dynamic or cheap features online and join at serve-time.- Implementation: feature retrieval API merges materialized feature values with live-computed features. Use a consistent schema and feature versioning to ensure training/serving parity.Cache invalidation & TTLs- Use TTLs on cached batch features tuned to their acceptable staleness. For critical features, shorter TTL or event-driven invalidation.- Event-driven invalidation: update/patch cache when source events arrive (CDC or stream processors) to maintain freshness for hot keys.- Versioned keys: include feature version and computation timestamp in cache keys so models can detect staleness and fall back to safe defaults.- Graceful fallback: if cache miss or stale beyond threshold, either compute online or use last-known value + uncertainty signal.Key principles: quantify acceptable staleness per feature, instrument latency and error budgets, and prefer hybrid patterns to balance cost, latency, and freshness.
EasyTechnical
86 practiced
Explain canary and blue-green deployment patterns for ML models: what each pattern achieves, how traffic is split, rollback strategies, and how you would wire telemetry (system and model metrics) to validate a canary before full rollout. Mention shadow deployments and when you'd use them.
Sample Answer
Canary and blue-green are safe rollout patterns for ML model updates.Blue‑Green:- What it achieves: Fast, atomic cutover with near-zero downtime. Two environments: Blue (current) and Green (new).- Traffic split: Initially 100% to Blue; when ready, switch router/load balancer to 100% Green.- Rollback: Immediate: flip back to Blue if problems; retains old environment for quick recovery.- Use when you need quick rollback and environment parity (easier for major infra changes).Canary:- What it achieves: Gradual, controlled exposure of new model to production to limit blast radius.- Traffic split: Start with small % (e.g., 1–5%), monitor, then escalate (10%, 25%, 50%, 100%).- Rollback: Stop escalation and route all traffic back to baseline; can terminate canary instances.- Use when you want to validate model behavior under real traffic with progressive risk.Telemetry & validation (for canary):- System metrics: latency, error rate, resource usage, throughput.- Model metrics: accuracy, AUC, calibration drift, prediction distribution shifts, top‑k/thresholded metrics, business KPIs (CTR, conversion).- Wire telemetry: tag canary requests with metadata; aggregate metrics separately for canary vs baseline; use statistical tests (A/B significance, non-inferiority) and drift detection thresholds; alerting and automated gates to stop or rollback.- Log inputs/outputs for offline analysis and bias checks.Shadow deployments:- What: Send production requests to new model in parallel but don’t use its outputs for serving decisions.- When to use: Safe validation of new models against real input distributions without user impact — useful for feature parity tests, performance comparison, and auditing before any serving traffic.- Combine patterns: run shadow initially, then canary, then blue‑green cutover for full confidence.
HardTechnical
82 practiced
Design a quantization-aware training (QAT) pipeline for deploying 8-bit inference across heterogeneous hardware (ARM CPU, x86 CPU, GPU). Explain how to configure fake-quantization during training, choose per-layer vs per-channel quantization, calibration techniques, evaluate accuracy regression, decide when to use post-training quantization vs QAT, and tools for automating hardware-aware tuning and validation.
Sample Answer
Requirements & goals:- Target: accurate 8-bit inference on ARM/x86 CPU and GPU with minimal accuracy regression (<1% absolute if possible), deterministic latency and memory reduction.- Constraints: heterogeneous operator support, mixed per-layer hardware behaviors, deployment toolchains (TFLite, ONNX Runtime, TensorRT, XNNPACK).High-level QAT pipeline:1. Baseline & profiling- Measure FP32 accuracy and per-layer sensitivity (activation/weight dynamic ranges, MSE/KL divergence, ops unsupported by hardware).- Run representative inference traces on target hardware or emulator to capture real activation ranges and operator fallbacks.2. Decide PTQ vs QAT- Try Post-Training Quantization (PTQ) with advanced calibration first for small models or when accuracy drop is small. Use per-channel weight quant and symmetric/asymmetric activation depending on hw.- If PTQ regression > target, use QAT.3. Fake-quant configuration during QAT- Insert fake-quant nodes on weights and activations (learnable observers for min/max or moving-average with EMA).- Use straight-through estimator (STE) for gradients.- Start from pretrained FP32 weights and fine-tune with a small LR (1e-5–1e-4) for 10–30 epochs depending on data availability.- Use per-channel weight quantization (int8) and per-tensor or per-channel activation depending on ops: prefer per-channel weights for conv/linear; prefer per-tensor activations for hardware lacking per-channel activation support.- Quantization scheme: use symmetric quant for weights (better for GEMM kernels) and asymmetric for activations if hardware expects zero-point.- Bit-width emulation: simulate true-scale zero-point, saturation, and rounding-to-nearest-even.4. Calibration techniques- For PTQ: use KL-divergence or percentile (99.9th) calibration on a representative calibration dataset; consider layerwise MSE for sensitive layers.- For QAT: replace static observers with quantization-aware moving statistics; periodically re-run calibration on hold-out set to refine ranges.- Hybrid: use PTQ calibration to initialize fake-quant ranges before QAT fine-tuning.5. Per-layer vs per-channel tradeoffs- Per-channel weights: better accuracy, slightly higher storage for scales, must ensure runtime supports per-channel (most CPU/GPU runtimes do).- Per-tensor activations: simpler and faster on many HW; per-channel activations give marginal gains rarely worth complexity.- Strategy: default per-channel weights + per-tensor activations; fallback per-tensor weights for hardware without per-channel support.6. Accuracy evaluation & regression handling- Evaluate on representative validation and on-device end-to-end tasks (latency/throughput).- Track layerwise activation histograms and weight quant error; run sensitivity analysis to identify layers needing higher precision (e.g., first/last layers, softmax, layernorm).- Recovery techniques: mixed-precision (keep sensitive layers in FP16/FP32), clip range tuning, bias correction, distillation loss during QAT, temperature scaling for logits.7. Automation & hardware-aware tuning tools- Use tools: Apache TVM autotvm/Ansor, TensorRT QAT/TorchQuant, ONNX Runtime Quantization Tool, Google’s QAT support in TensorFlow Model Optimization, AIMET (Qualcomm/Broadcom), Neural Compressor.- Implement a CI that: - Runs PTQ candidates with calibration, measures accuracy and operator mapping on target. - If failing, kicks off QAT with preconfigured fake-quant policies per target and runs tuned fine-tuning. - Uses autotuners (TVM/Ansor) to optimize kernels per platform, and cross-validates on real hardware.- Integrate hardware-in-the-loop validation to catch operator fallbacks and numerical mismatches.8. Deployment checklist- Verify operator coverage and fallback behaviors per runtime.- Export quantized model in target format (TFLite, ONNX with quant ops, TensorRT engine).- Run end-to-end tests on ARM/x86/GPU: accuracy, latency, memory, thermal throttling.- If discrepancies, iterate: adjust quant scheme, apply mixed-precision, or increase QAT epochs.Key principles:- Start simple (PTQ with good calibration), escalate to QAT when needed.- Prefer per-channel for weights, per-tensor for activations unless hardware supports per-channel activations.- Automate hardware-aware tuning and include on-device validation early to avoid surprises.
Unlock Full Question Bank
Get access to hundreds of ML Systems Architecture & Components interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.