Applied ML to Real-World Problems and Constraints Questions

Practical application of machine learning to solve real-world problems while navigating operational constraints such as latency and compute budgets, data privacy and regulatory requirements, fairness, interpretability, and production readiness. Covers problem formulation, data collection and preprocessing under real-world data limitations, feature engineering, model selection and evaluation for constrained settings, deployment patterns (online vs. batch/offline), monitoring and retraining, ML platform design, and governance for responsible AI.

HardSystem Design

36 practiced

You need to add auditing to your ML pipeline so every production prediction can be traced to the model version, feature snapshot, and raw input provenance. Propose a design for minimal per-request trace metadata and an efficient storage strategy for long-term audits (5+ years) that balances cost and queryability.

Sample Answer

Requirements & constraints:- Trace each prediction to: model version, feature snapshot (values or fingerprint), raw input provenance (source id, timestamp), and request metadata (user id, request id).- Keep per-request payload minimal (low latency, low storage per row) yet reconstructable for audits over 5+ years.- Optimize cost vs. queryability for regulatory investigations.

Design — per-request trace metadata (stored synchronously at prediction time):- event_id (UUID), timestamp, model_id:version, serve_container_id- feature_fingerprint (hash, e.g., SHA256 of serialized feature vector) and feature_store_snapshot_id (pointer to snapshot)- raw_input_pointer: (source_type, source_id, ingestion_batch_id) — small identifiers, not full payload- user_id/requestor_id, decision/outcome, score/confidence- provenance_tags (enum bitfield) for quick filteringRationale: store small primitives + fingerprints/pointers to avoid storing bulky payloads per request.

Feature & raw input archival (asynchronously):- Immutable snapshots per deployment: store full feature vectors and schema as compressed Parquet/ORC in object storage (cold tier) keyed by feature_store_snapshot_id and feature_fingerprint(s). Include manifest with snapshot_id, time, model_version, data hashes.- Raw input blobs: store original payloads or pointers to long-term archive (WORM) with ingestion_batch_id index.

Storage & indexing strategy:- Online OLTP store (e.g., DynamoDB/Postgres) for per-request metadata with TTL disabled; keep last N months for fast queries.- Warm index: nightly ETL to time-partitioned columnar store (e.g., BigQuery, Snowflake, or AWS Athena over S3) with partition keys: date, model_id, snapshot_id; include searchable columns: event_id, user_id, feature_fingerprint.- Cold archive: object storage (S3 Glacier/Archive) for full snapshots/raw blobs with manifests and checksum verification.

Query patterns & optimization:- Fast lookups by event_id → use OLTP store.- Bulk audits (date range, model_version) → run queries on columnar store.- Deep reconstructions → join event row (feature_fingerprint) to feature snapshot files in object storage.

Cost, retention & compliance:- Keep minimal metadata online (months) in low-latency DB; move aggregated/partitioned data to analytics store for 5+ year retention; full payloads compressed in glacier-like cold tier.- Encrypt at rest, apply KMS; sign manifests; use WORM/immutability for compliance.- PII/GDPR: store pointers and hashes; persist raw PII only if legally required and with access controls and consent logging. Support deletion requests by marking records and removing reconstructable raw payloads while keeping irreversibly hashed audit trail.

Operational considerations:- Automate snapshot creation at model deployment; ensure deterministic feature serialization for stable fingerprints.- Include monitoring for missing pointers/hashes; periodically validate checksums.- Provide tooling: quick audit UI to fetch event -> snapshot -> raw blob; automated playbook for investigations.

Trade-offs:- Fingerprints + pointers reduce per-request cost and latency but require reliable snapshoting and retention policies. Storing full payloads per-request simplifies reconstruction at higher storage cost. Choose pointer approach for scale and compliance-aware raw-data retention.

MediumSystem Design

28 practiced

You must implement a lightweight streaming feature materializer that computes aggregations (e.g., count, mean over 24h) from Kafka events and writes to an online key-value store. Outline the streaming framework (e.g., Kafka Streams/Flink), state management, exactly-once semantics, and strategies for backfilling historical aggregates.

Sample Answer

Requirements (clarify): low-latency continuous feature updates, per-entity aggregations (count, 24h mean), at-least-once or exactly-once correctness, write to an online KV store for serving, must support backfill of historical data.

High-level architecture:- Ingress: Kafka topics of raw events- Stream processing: Apache Flink (preferred) or Kafka Streams for simpler setups- State store: Flink keyed state with RocksDB backend + durable checkpointing and changelog- Sink: Online KV store (Redis/KeyDB for low-latency, Cassandra/Bigtable for larger scale)- Backfill: Batch or bounded stream job reading historical partitions

Why Flink:- Robust event-time/windowing and watermarks for 24h windows- Strong stateful processing with RocksDB and scalable keyed state- Built-in checkpointing and exactly-once semantics with two-phase commit for sinks

State management:- Use keyed state per entity (userId, itemId) storing running aggregates and a TTL aligned to 24h- Persist state to RocksDB; enable incremental checkpoints and savepoints for fast recovery- Keep a compact changelog (e.g., Kafka state changelog) or rely on filesystem-based checkpoints (S3)

Exactly-once semantics:- Enable Flink checkpointing with durable state backend and KafkaSource with commit-on-checkpoint- For sink: prefer transactional/atomic writes — either Flink’s two-phase-commit sink (e.g., to Kafka or transactional DB) or implement idempotent writes: - If using Redis: write keys with version stamps (checkpoint id / last-processed-offset) and conditional updates (Lua scripts) to ensure idempotency. - If using Cassandra: use lightweight transactions or upserts with monotonic sequence/version.- Ensure source offsets are committed only after successful checkpoint to guarantee EOS.

Aggregation design:- For sliding/tumbling 24h windows use event-time windowing and incremental pre-aggregation: - Maintain per-entity sliding-window data structure (e.g., time-buckets or tumbling sub-windows) to compute rolling mean/count efficiently and drop expired buckets.- For mean: store (sum, count) per entity and compute mean = sum/count.

Backfilling/historical aggregates:- Option A — Recompute full history (offline): - Run a batch Spark/Flink job that scans historical data, computes aggregates per entity for the 24h windows, writes to the same KV store atomically (write to staging keys then swap). - Use partitioned processing and checkpoints. This is simplest for correctness.- Option B — Replay to streaming job: - Restore job to savepoint, switch Kafka topic to start from historical offsets, run the same Flink pipeline in bounded mode to rebuild state; use savepoints to persist final state. - Careful with duplication: mark backfill runs with version tags and perform atomic swap if writing to serving store.- Hybrid: Precompute heavy history offline, then run streaming job to catch up and keep incremental updates.

Operational considerations:- Schema/versioning for feature keys and metadata (feature name, window, timestamp, version)- Monitoring: metrics for processing lag, checkpoint latency, failed checkpoints, sink write errors, data skew- Fault tolerance: auto-scaling keyed parallelism, hot-key handling (split heavy keys), state size limits and compaction- Testing: end-to-end tests with deterministic sources, chaos tests for failover and exactly-once verification- Security & governance: ensure encryption in transit, RBAC for KV writes, provenance metadata (last-offset, watermark)

Example implementation notes:- In Flink, use KafkaSource (exactly-once via offsets-on-checkpoint), keyedStream.keyBy(entity), processFunction with RocksDB state storing fixed-size ring buffer of sub-window buckets, use timers to evict old buckets, onCheckpointFlush write to KV sink via TwoPhaseCommitSinkFunction or atomic staging+swap.

This design balances correctness (exactly-once), operational simplicity (RocksDB + checkpointing), and practical backfill strategies (offline recompute + streaming catch-up).

HardTechnical

58 practiced

Design an evaluation methodology for rare-event prediction (positive rate ~1/10,000). How do you collect evaluation data, choose statistically meaningful metrics, and compute required sample sizes for detecting improvements? Discuss handling label delay and stratified sampling.

Sample Answer

Situation and goal: We need an evaluation methodology that yields statistically meaningful comparisons for a very rare positive class (~1/10,000). The core challenges are (1) collecting enough labeled positives without prohibitive cost, (2) choosing metrics that reflect business utility and are robust with extremely imbalanced data, (3) calculating sample sizes (especially number of positive events) to detect realistic improvements, and (4) handling label delay and stratified sampling to make estimates unbiased and efficient.

1) Data collection strategy- Enrichment / targeted labeling: perform an initial screening model or heuristics to over-sample likely positives (two-stage sampling). This yields many more positives per labeled unit. Record the sampling probability for each record to allow unbiased reweighting later (inverse-probability weighting, IPW).- Event-driven labeling: prioritize labeling around known triggers (time windows, cohorts) to capture event context.- Backfill with historical data: use historical windows where outcomes have matured to reduce label delay.- Maintain a held-out unbiased evaluation set: periodically label a simple random sample (SRS) of production traffic (even if small) to measure real-world calibration and overall rates.- Label pipeline: track labeling timestamps, label confidence, and cascade for adjudication. For delayed labels, track censoring indicators.

2) Metrics — what to measure and why- Primary: Precision at operational recall/threshold or top-K precision if model used to surface candidates. In rare settings, precision is often most actionable (false positives cost).- Recall (sensitivity): important if missing positives has cost.- FDR (1 - precision) and F1 where appropriate, but F1 can mislead when prevalence is tiny.- Area under Precision-Recall curve (PR-AUC): preferable to ROC-AUC because ROC can be misleading with extreme class imbalance.- Calibration / Probabilistic metrics: Brier score or calibration plots; use IPW to estimate if sample is enriched.- Business utility / expected value: convert model outputs to monetary/operational utility and measure expected utility or net benefit — often more relevant than pure stats.- Uncertainty: report confidence intervals (CIs) for all metrics using analytic approximations or bootstrap with IPW.

3) Sample-size / power planning- Key insight: with rare events the limiting factor is number of positive labels (E). Power to detect differences depends primarily on E, not total N.- For comparing two proportions (e.g., recall r0 → r1), approximate required number of positive events using normal approximation: Let p0 = baseline positive detection rate (e.g., recall), p1 = p0 + Δ. For a two-sided test at α and power 1-β, required positives: E ≈ ( (Z_{1-α/2} * sqrt( p̄(1-p̄) ) + Z_{1-β} * sqrt( p0(1-p0) + p1(1-p1) ) )^2 ) / Δ^2 where p̄=(p0+p1)/2. In practice simplify: required positives scale ≈ (Zsum^2 * p*(1-p)) / Δ^2.- If evaluating precision (a proportion among predicted positives), similar formulas apply but sample size should center on number of predicted positives labeled.- Translate positives to total labels: total required labeled records ≈ E / prevalence for SRS. With enrichment sampling, effective total labels = (E / enrichment_factor) but you must apply IPW when estimating population metrics.- Poisson approximation for extremely small p can be used; event-driven designs (collect until E events observed) are practical.- Example guidance: to detect an absolute improvement in recall from 0.50 → 0.60 (Δ=0.10) with α=0.05, power=0.8, you typically need several hundred positive events. At prevalence 1/10,000 that implies millions of raw samples unless you enrich.

4) Stratified sampling and estimation- Use stratified sampling to reduce variance: strata by model score buckets, covariates correlated with event probability, or time windows. Allocate more labeling budget to high-variance or high-impact strata (Neyman allocation).- When using stratified or enriched sampling, estimate population-level metrics with IPW: weight each labeled record by inverse of its selection probability (or use stratum weights).- For CIs with stratified designs, use appropriate survey-estimation formulas or bootstrap respecting sampling weights and strata.

5) Handling label delay (censoring)- If outcomes take time to reveal, treat data as right-censored. Options: - Use survival-analysis methods (Kaplan–Meier, Cox models) to estimate event probabilities at the evaluation horizon. - Only evaluate on cohorts with fully matured labels (delay window + safety buffer); this reduces bias but increases waiting time. - For near-real-time evaluation, use partially observed labels with censoring adjustment: model label delay distribution and impute probabilities for unlabeled cases; propagate uncertainty into CIs. - For enrichment, ensure delay distributions are similar across strata or adjust for differing delays.- Track time-to-label and include that in stratification to avoid bias.

6) Practical workflow (stepwise)- Define metric(s) tied to business utility (precision@k, recall, expected utility).- Choose test horizon and acceptable Δ to detect (absolute or relative).- Compute required positive events E using formulas above; convert to labeling budget given sampling strategy (SRS vs enriched).- Collect data: run targeted enrichment + SRS holdout; log selection probabilities and timestamps.- Label until E positives are obtained (event-driven) or until budget exhausted.- Estimate metrics with IPW or survey estimators; compute CIs (analytic + bootstrap).- Run sensitivity analyses: alternate thresholds, different enrichment settings, check calibration and delay adjustments.- If comparing two models, use paired evaluation on same labeled set to reduce variance (McNemar or paired proportion tests) when feasible.

7) Robustness and trade-offs- Enrichment reduces labeling cost but requires careful reweighting and increases variance if selection probabilities are small; use moderate enrichment factors.- Event-driven designs reduce time to answer but may bias covariate distributions if not corrected.- Always report both enriched-sample metrics and population-adjusted metrics with uncertainty.

Summary: focus on collecting a sufficient number of positive events (via enrichment + IPW and stratified allocation), choose PR-focused metrics and business-utility metrics, compute sample size centered on required positive count using proportion-power formulas (or event-driven stopping), and correct for stratification and label delay using IPW and survival/censoring methods. This yields unbiased, statistically powered evaluation even when prevalence is ~1/10,000.

HardTechnical

39 practiced

As a senior ML engineer you recommend deprecating a high-cost legacy model used by multiple product teams. Describe how you would build a migration plan: stakeholder communication, phased replacement, fallbacks, metrics to monitor, and minimizing business disruption.

Sample Answer

Situation: At my previous company we had a decade-old recommendation model that was costly to run (GPU-heavy, slow retrain) and produced marginally better business metrics than a new lightweight model we developed. It was used by three product teams and a personalization API; deprecating it risked regressions and upset stakeholders.

Task: I needed to create a migration plan that decommissioned the legacy model safely, minimized business disruption, kept teams informed, and ensured rollback/fallbacks.

Action:- Stakeholder communication - Identified stakeholders: product owners for the three teams, infra/SRE, data engineering, legal/privacy, and exec sponsor. - Ran a kickoff meeting outlining rationale (cost, maintainability), timeline, risk assessment, and clear success criteria. Followed with weekly status updates and a shared runbook and slack channel. - Aligned on approvals required and change windows with product managers.

- Phased replacement - Canary phase: Deployed the new model behind a feature flag; routed 1–5% live traffic per team. Instrumented side-by-side scoring so both models produced outputs for the same requests. - Team-by-team ramp: After validating canary, incrementally increased traffic per product team (5→20→50→100%) with at least 48 hours observation at each step. - Shadow testing before full cutover for non-customer visible products to validate offline metrics.

- Fallbacks and safety - Automatic rollback triggers defined in runbook (e.g., >2% drop in CTR or >5% error rate increase within 24 hours). - A/B testing queue allowed instant switch to legacy model; kept legacy model in standby for 2 months post-cutover. - Graceful degradation: if new model service fails, request routing falls back to cached responses or a rule-based baseline.

- Metrics to monitor - Business metrics: CTR, conversion rate, revenue per user, retention (team-specific KPIs). - Model metrics: latency (p95/p99), error rates, model confidence calibration, feature distribution drift (population stability index), input schema changes. - Engineering metrics: CPU/GPU cost, memory, autoscaling events, SLOs. - Set alert thresholds and automated dashboards per stakeholder.

- Minimizing disruption - Run experiments on a replica of production data first; use canary windows during low-traffic periods. - Provide product teams with a validation report and A/B experiment results highlighting equivalence or improvements. - Offer a rollback playbook and 24/7 on-call coverage during major cutover windows.

Result: Using this approach we reduced serving costs by 60% while maintaining or improving CTR across teams. No major incidents occurred; we completed deprecation with the legacy model removed from production after a two-month standby. The structured communication and clear rollback criteria preserved stakeholder trust and enabled a smooth transition.

Learning: Early alignment on success metrics, automated observability, and preserving a tested fallback are the most important levers to de-risk model deprecation at scale.

MediumSystem Design

28 practiced

Design a feature store to support both offline training and low-latency online retrieval (<10ms) for 100M users and 10k distinct features. Specify components, storage choices for online vs offline, feature materialization workflow, consistency model, and how you'd support feature TTL and backfills.

Sample Answer

Requirements & constraints:- 100M users, 10k features, online retrieval <10ms, support offline batch training, TTL, backfills, strong availability.

High-level architecture:- Ingestion layer (Kafka) → Feature ingestion/transform workers (Flink/Spark Streaming) → Serving store (low-latency key-value DB) + Offline store (columnar data lake / warehouse) → Feature registry & metadata service → Materialization/orchestration service → API gateway for online reads.

Storage choices:- Offline (training): Cloud object store (S3) + table format (Iceberg/Delta) partitioned by date and entity_id; accessible via Spark/BigQuery for large joins and historical snapshots.- Online (serving): Distributed key-value DB with <10ms p99 reads — e.g., DynamoDB/Scylla/Cassandra with SSD nodes or Redis Cluster (if RAM budget allows) using user_id (or entity_id) as primary key and storing feature vector (binary/JSON).

Feature materialization workflow:1. Batch materialization: Periodic Spark jobs read raw events, apply transforms from feature registry, write to offline store and produce per-entity Parquet + incrementally write to serving store via bulk loaders or import APIs.2. Stream materialization: Real-time streaming jobs apply same transforms and upsert to serving store (and append to offline lake).3. Orchestrator (Airflow/Argo) manages full-refreshs, incremental updates, and backfills using consistent feature definitions.

Consistency model:- Offline: Strongly consistent snapshot semantics per training run (time-travel via Iceberg/Delta).- Online: Eventual consistency with causal guarantees; streaming jobs provide near-real-time updates. For reads that require strict freshness, support read-repair or read-through to compute features on-demand from latest events (with higher latency) or maintain versioned features with timestamps.

Feature TTL & versioning/backfills:- Each feature stored with (value, timestamp, version, ttl). Serving layer enforces TTL by expiry or tombstones; materializers emit TTL metadata.- Backfills: Orchestrator triggers historical recompute using the same transforms producing new versioned artifacts. During backfill, use shadow write pattern: write to separate namespace/version; run validation; ramp traffic to the new version via registry switch. Support incremental backfills by partition/date ranges to bound compute.

Operational considerations:- Sharding by entity_id, hot-key mitigation via secondary hashing.- Monitoring: p99 latency, freshness lag, drift, feature registry tests, CI for feature transform unit tests.- Security: IAM, encryption at rest/in transit, access auditing.

Trade-offs:- Redis gives lowest latency but higher cost; DynamoDB/Cassandra cheaper at larger scale with predictable latency.- Eventual consistency keeps throughput low; for stricter correctness add on-demand recompute or synchronous writes at higher latency.

Unlock Full Question Bank

Get access to hundreds of Applied ML to Real-World Problems and Constraints interview questions and detailed answers.

Join thousands of developers preparing for their dream job.