MediumTechnical

58 practiced

Explain the difference between offline evaluation metrics and online metrics in production. Provide an example where offline metrics improved but online metrics regressed, and describe steps you would take to reconcile and prevent this mismatch.

Sample Answer

Offline evaluation metrics are computed on historical labeled data (holdout/validation sets) and measure model performance under controlled, often i.i.d., assumptions (e.g., accuracy, AUC, NLL). Online metrics are observed in production under real user behavior and systems conditions (e.g., click-through rate, conversion, latency, false positive cost) and capture end-to-end impact including feedback loops, latency, UI effects, and population shift.

Example (mismatch): In a recommender system I worked on, an updated ranking model showed a +5% offline AUC and better calibration on the test set, but an A/B test revealed a -3% CTR and lower downstream conversion. Investigation showed the offline set over-represented power users and used richer features available only in batch; in production the model relied on ephemeral features with higher latency and missingness, producing stale or default scores that harmed ranking diversity. Also the new model narrowed recommendations (higher precision but lower exploration), reducing long-term engagement.

How I reconciled it:- Reproduce production inputs: run the model in shadow mode on live traffic to compare scores given real feature availability and latency.- Check feature parity: verify preprocessing, imputation, feature freshness, and schema drift between training and serving.- Analyze slice performance: compare online vs offline by user segments, traffic sources, and contexts to find where regression occurs.- Inspect label/metric mismatch: ensure offline metric correlates with business metric; add proxies (e.g., predicted CTR) or train directly on business objectives when possible.- Run causal/ablation tests: disable new features or use holdout features to isolate causes.

How I’d prevent future mismatch:- Enforce evaluation parity: use the same feature pipeline in offline eval (or simulate missingness/freshness); run offline metrics on "production-like" traces.- Continuous shadow testing: always validate candidate models on live traffic in shadow before rollout.- Use business-aligned objectives: include online KPIs in training/validation or optimize proxy losses (propensity-weighted).- Robust validation: stratified/sliced validation, temporal split to catch non-i.i.d. behavior, and adversarial tests for missing/late features.- Safe rollout: staged rollouts, canarying, and automated rollback triggers tied to key online metrics.- Monitoring and feedback: instrument per-feature missingness, latency, calibration drift, and user-segment metrics, plus automated alerts.

These steps align offline evaluation with online realities and reduce surprises when models reach production.

MediumSystem Design

53 practiced

Design an online model-serving architecture for a recommender that must handle 10,000 QPS with p95 latency under 50ms, and support a 1TB feature store receiving 1M updates daily. Describe components for serving, caching strategy, feature routing and consistency, sharding, autoscaling, and how you ensure feature freshness for near-real-time personalization.

Sample Answer

Requirements (clarify): 10k QPS, p95 <50ms, 1TB feature store with 1M daily updates, near‑real‑time personalization (seconds–minutes), strong availability.

High-level architecture:- Edge/API Gateway: TLS, auth, rate‑limit → load balances to stateless Frontend.- Frontend (k8s): handles request enrichment, routing, fallbacks.- Online Feature Store (hot store): Redis/KeyDB cluster (in‑memory), sharded by user_id. Holds “hot” subset (e.g., 100–200GB). Supports <1ms lookups.- Cold Feature Store: distributed DB (Bigtable/Cassandra/S3 + Parquet) for full 1TB archive.- Stream layer: Kafka for ingest of events/updates; Flink/Beam for feature computation, aggregation, and incremental updates.- Model Serving: fleet of optimized model servers (ONNX/TensorRT / torchserve) behind a predictor service. Models are CPU/GPU packed depending on latency; keep model size & compute so single inference <10ms.- Cache layer: Redis LRU with two-tier: 1) in‑memory hot feature cache, 2) local LRU per model server (process-local) for microsecond hits.- Feature Router: service that decides for a key whether to read from online Redis, or fallback to computed feature from stream / cold store.- Consistency/versioning: features versioned with (feature_name, version, timestamp). Use event time and materialized views so model reads features with consistent snapshot (per-request version tag). Writes are idempotent and ordered by Kafka partition (user_id keying).- Freshness pipeline: stream processing computes incremental updates and writes to Redis with update timestamps. Use "write-through" for critical features (update Redis synchronously where latency allows) and async for high‑volume features. Maintain per-feature TTL (seconds-minutes) and a "staleness" threshold; if feature older than threshold, trigger on‑demand recompute via fast path.- Sharding & scaling: shard Redis and model servers by user_id hash. For 10k QPS, estimate 100 pods at 100 RPS each; autoscale via HPA/HVPA by CPU, latency SLO and custom RPS metric. Redis cluster scales by adding shards; use consistent hashing to minimize rebalancing.- Latency budget (example): network + API gateway 3–5ms, feature fetch (cache hit) 1–3ms, model inference 8–12ms, serialization + routing 2–4ms → p95 <50ms with margin. Ensure >95% cache hit to meet SLO.- Autoscaling: k8s HPA with metrics (RPS, p95 latency), predictive autoscaler using recent traffic patterns, and scale-down safeguards. Redis autoscale with shard addition and rebalancing during low traffic windows.- Reliability & monitoring: SLOs, end‑to‑end tracing, Prometheus + Grafana, alerting on cache-hit ratio, feature freshness, p95 latency. Canary deploys with feature and model gating.- Edge cases/fallbacks: if feature missing or stale, use default imputation or lightweight fallback model; keep offline batch recompute to reconcile drift.

Why this works:- Two-tier feature store keeps hot reads ultra-fast while cold storage holds full history.- Kafka + stream processing ensures near‑real‑time feature updates and deterministic ordering by user.- Caching + sharding ensures throughput and p95 latency at 10k QPS while autoscaling handles spikes.

MediumSystem Design

64 practiced

Design an offline training pipeline that supports nightly retraining with incremental joins of features, ensures reproducibility, and allows fast debug turnaround. Include data lineage, environment reproducibility, artifact storage, schema enforcement, and ways to accelerate iterative debugging of model code.

Sample Answer

Requirements:- Nightly retrain with incremental feature joins (append new data)- Strong reproducibility (data + code + env + config)- Observable data lineage and schema enforcement- Fast developer debug iterations

High-level architecture:1. Orchestration: Airflow or Prefect triggers nightly pipeline + ad-hoc runs.2. Data ingestion & incremental joins: Partitioned raw tables (event_time). Use CDC or daily batch to create feature-store staging tables; perform incremental joins using timestamp-based windowing and upserts.3. Feature Store: Store features with entity keys, feature timestamps, and version IDs (Feast or internal). This enables point-in-time correct training.4. Lineage & Schema: Use a metastore (Data Catalog) capturing table/column lineage; integrate Great Expectations for schema and data quality checks in each step; log lineage events to OpenLineage/Marquez.5. Environment reproducibility: Containerized steps with pinned base images + conda/pip lockfiles; track code commit SHA; run everything via Docker images built by CI and stored in registry.6. Artifact storage: Model artifacts, feature snapshots, and datasets versioned in object storage (S3) with manifests; use MLflow or DVC to record parameters, metrics, artifacts, and model registry entries tied to git SHA and feature-store version.7. Reproducible training: Training job reads exact feature snapshot (by timestamp/version) and code image; record full provenance in MLflow: git commit, docker image, dataset path, hyperparams, seed.8. Fast debug turnaround: - Local/lightweight mode: small-sample "mini-batch" pipeline that runs locally with same code paths and dry-run flags. - Hot-reload: use fast iterative notebooks or lightning-style Trainer with smaller dataset and same feature transforms. - Unit tests for transforms and feature join logic; synthetic data generators to exercise edge cases. - Use cached intermediate artifacts (precomputed joins) so dev runs start from feature snapshots instead of full raw joins.Trade-offs:- Extra storage for snapshots vs faster debugging and reproducibility.- Strong lineage and schema checks add latency but prevent downstream failures.

This design ensures nightly incremental retrain, full lineage and reproducibility, and pragmatic ways to accelerate developer iteration.

MediumTechnical

62 practiced

Explain trade-offs when using model ensembling at inference time versus deploying a single model, focusing on accuracy, latency, cost, and operational complexity. Provide an example where ensembling was beneficial and another where it was harmful in production.

HardTechnical

81 practiced

Propose an architecture to serve an ensemble of models where privacy rules prevent sharing raw user data across individual models. Explain secure computation or architecture patterns you would use (e.g., secure enclaves, federated inference, encrypted feature-sharing, or split-execution), latency implications, and how to validate ensemble outputs without violating privacy constraints.

Sample Answer

Requirements & constraints:- Ensemble of N models owned by different parties; raw user features cannot be shared between models.- Low-latency inference (soft real-time), strong privacy guarantees, ability to validate ensemble accuracy and debug without leaking raw data.

High-level architecture (privacy-preserving ensemble broker):1. Client-side feature extractor: user device or trusted edge service extracts raw features and performs local preprocessing. Sensitive raw data never leaves client.2. Secure execution layer options (choose combination per threat model): - Secure Enclaves (TEEs): client or broker sends encrypted features to each model host’s TEE which decrypts inside enclave, runs model, returns protected score. Good for untrusted hosts but adds enclave startup/IO overhead. - Federated Inference / Split-execution: split models so client computes initial layers; server-side completes forward pass. Only intermediate activations (ideally lower-dimensional, encrypted or obfuscated) are transmitted. - Encrypted feature-sharing using HE/GC/SMPC: use homomorphic encryption or secure multi-party computation to compute model outputs without revealing inputs. Preferable when TEEs unavailable; HE introduces high compute and latency.3. Ensemble aggregator (privacy-preserving): receives per-model predictions as encrypted or attested values. Runs aggregation inside a TEE or via SMPC to produce final prediction and confidence. Optionally apply differential privacy (DP) noise to outputs for audit logs.

Latency trade-offs & mitigations:- TEEs: moderate added latency (ms–tens ms); minimal algorithmic changes. Good throughput with warmed enclaves and pooling.- Split-execution: low network payload if activations small; client compute increases but reduces end-to-end latency.- HE/SMPC: highest latency (×10–100s); use only for low-throughput or offline scenarios.Mitigations: model distillation to smaller student models at edge, quantization, batching for throughput, parallel calls to model hosts, caching frequent results, asynchronous returns with best-effort fallback.

Validation and debugging without leaking raw data:- Use holdout synthetic datasets and joint benchmarks where parties agree on feature schemas; run validations in TEEs or via SMPC revealing only aggregated metrics (AUC, calibration).- Use DP-protected aggregate metrics: add calibrated noise when reporting per-model errors.- Use cryptographic attestations from TEEs to prove models used canonical weights and there was no tampering.- For per-example disagreement analysis, exchange only masked identifiers and hashed feature summaries or saliency attributions computed inside enclaves (no raw features).- Continuous monitoring: drift detection on encrypted feature distributions (histogram sketches via DP) and on ensemble score distributions.

Trade-offs:- TEEs + split-execution balance practicality and latency; HE/SMPC maximize privacy but cost latency.- Choose hybrid: client-side preprocessing + split-execution for latency-critical paths, TEE-based aggregation for trust, and HE/SMPC for high-sensitivity transactions.

This design preserves raw-data boundaries, provides verifiable computation, and offers paths to balance privacy vs latency depending on operational requirements.

Production ML Systems Experience Summary Questions

Sample Answer

Sample Answer

Sample Answer

Sample Answer

Sample Answer

Unlock Full Question Bank