Delivery Marketplace ML Applications Questions
Machine learning use cases common to on-demand delivery marketplaces, covering demand forecasting, driver/courier dispatch and routing, pricing and revenue optimization, recommendations, fraud detection, and real-time optimization. Includes model development, deployment, monitoring, drift handling, and scalability considerations for ML systems operating in a high-velocity, two-sided marketplace.
HardTechnical
33 practiced
Design an embedding-based recommendation architecture to serve millions of users and restaurants on DoorDash. Cover embedding training strategies (co-occurrence, matrix factorization, or deep learning), online retrieval with approximate nearest neighbor indices, memory and latency trade-offs, embedding freshness, and approaches to handle cold-start items and users.
Sample Answer
Requirements & constraints:- Functional: return top-N restaurant recommendations per user in <100ms for millions of users/restaurants.- Non-functional: handle 10k QPS, embedding freshness within minutes, support cold-start, 99.9% availability.High-level architecture:- Offline training pipeline → embedding store + metadata DB → ANN index service (sharded) + feature service → Online scorer & reranker → CDN/cache for hot results → Logging & feedback loop.Embedding training strategies:- Co-occurrence / word2vec-style (skip-gram on user sequences): fast, scalable, captures short-term behavior; good for session-based similarity.- Matrix factorization (ALS/SVD): interpretable, handles global preference signals, efficient for implicit feedback.- Deep learning (two-tower / DSSM / transformer): encode users and restaurants separately (user tower inputs: user history embeddings, recency, context; item tower inputs: menu, cuisine, location, price). Train with contrastive loss (max margin / sampled softmax) or pairwise ranking (BPR). Use hybrid loss combining recall and downstream conversion prediction.- Practical: ensemble embeddings — combine MF for long-term taste + two-tower for contextual signals + session co-occurrence for recency.Online retrieval (ANN):- Build per-embedding-type ANN indices (HNSW / Faiss IVFPQ). Shard indices by geography and verticals to reduce candidate set (geo-filter then ANN).- Two-stage: ANN recall (top-K candidates) → lightweight model rerank using freshness, delivery ETA, inventory, business rules → final ranking model (GBM or small NN).- Use Faiss on GPU for high throughput or HNSW on CPU for low-latency.Memory & latency trade-offs:- Uncompressed dense embeddings: lower latency but high memory; use for hot shards.- Product Quantization / IVFPQ: 8-16x memory reduction, small recall loss; good for cold storage.- HNSW provides highest recall/latency trade-off; tune M / efConstruction / efSearch.- Cache top results per user or segment in Redis; warm hot-item embeddings in memory.- Budget strategy: keep top 10M item embeddings in-memory (high-demand cities) and rest compressed on disk.Embedding freshness:- Nearline: mini-batch updates from streaming events (Kafka → micro-updates) to user embeddings (incremental two-tower user tower) within minutes.- Periodic full retrain: nightly/weekly for item tower and global embeddings.- Use delta indices: apply incremental updates to ANN with live swap to avoid downtime.Cold-start:- Items (new restaurants): compute content-based embeddings from metadata (menu NLP, cuisine, tags, location, images) and map to item-embedding space via trained item-tower; use popularity priors and explore-exploit (epsilon-greedy / contextual bandits) to collect signals quickly.- Users (new users): warm-start via cohorts (demographics, acquisition source), recent session signals, device locale; ask 1-2 onboarding questions or use popular local restaurants + explore policy to learn preferences.- Use meta-learning: train a model to predict embeddings from sparse signals (feature-to-embedding mapper) to handle extreme cold-start.Metrics & monitoring:- Offline: recall@K, MAP, AUC, NDCG on holdout; online: CTR, conversion rate, average order value, delivery SLA impact.- Monitor embedding drift, ANN recall degradation, QPS, p95 latency.- Continuous A/B testing for embedding variants and freshness windows.Key trade-offs:- Freshness vs stability: too-frequent updates cause index churn; use user-embedding micro-updates + periodic index rebuilds.- Memory vs recall: compression reduces memory but may reduce recall—tune PQ bits and reranker strength.- Complexity vs maintainability: ensemble/stacked embeddings improve metrics but require orchestration and robust feature stores.This design yields low-latency, scalable recall with mechanisms for freshness and principled cold-start handling while providing measurable trade-offs for engineering teams.
HardSystem Design
31 practiced
Design a scalable end-to-end ML pipeline to produce and serve demand forecasts per zone across thousands of cities for DoorDash. Discuss data ingestion, feature pipelines, model training cadence (hourly/daily), how to handle sparse zones (aggregation or transfer learning), model selection (global vs local), serving API design, caching, and monitoring/retraining triggers.
Sample Answer
Requirements & constraints:- Produce zone-level demand forecasts (next 1h, 6h, 24h) for thousands of cities, 10k+ zones, sub-minute serving latency, high availability, retrain cadence daily/hourly, handle sparse zones, track business KPIs (MAE, RMSE, coverage), and data freshness.High-level architecture:Event sources (orders, pings, promos, weather, traffic, holidays, POIs) → Ingestion (Kafka) → Raw data lake (S3/HDFS) + streaming feature store (Feast/Redis) → Batch feature pipeline (Spark) → Model training infra (Kubernetes + TF/PyTorch) → Model registry (MLflow) → Serving: prediction service (autoscaled REST/gRPC) + cache (Redis) → Monitoring & retrain orchestration (Airflow/KubeCron + Prometheus + Seldon/Alibi).Data ingestion & feature pipelines:- Streaming: Kafka topics for real-time events, CDC for RDB updates. Stream processors (Flink/Spark Structured Streaming) compute online features (last-5min count, recent velocity).- Batch: Nightly Spark jobs aggregate historical features (seasonality, lag features, zone embeddings, weather encodings). Persist features to feature store with versioning and lineage.- Feature engineering: time features (hour, day-of-week, holiday), rolling windows (15m,1h,6h,24h), external features (weather, promotions), zone metadata embeddings (POI mix, population density).Modeling & training cadence:- Hourly: Retrain or fine-tune a global model using latest data (online learning or incremental updates). Daily: full retrain with expanded window (90 days) and hyperparameter search.- Use hybrid approach: global backbone (spatio-temporal model: Transformer/Temporal Fusion Transformer or Graph Neural Network over zones) that captures shared patterns + light per-zone adapters (linear heads, small MLPs) for local calibration. This balances statistical strength and locality.- Sparse zones: use hierarchical modeling—aggregate into virtual zones (cluster by geography/POI), share global parameters, and apply transfer learning / multi-task learning. Use Bayesian hierarchical priors or domain-adaptive fine-tuning to borrow strength. For extremely sparse zones, serve cluster-level forecasts.Model selection:- Compare: global-only, local-only, hybrid. Preferred: global model with per-zone calibration (fine-tune or scaling factor). Evaluate via backtesting on per-zone metrics and business cost function (under/over-supply asymmetry).- Regular validation: rolling-window CV, backtest on seasonal events, fairness checks.Serving API & caching:- Prediction service exposes gRPC/REST: GetForecast(zone_id, horizon, features_context) → returns quantiles + uncertainty.- Low-latency path uses cached forecasts in Redis with TTL (e.g., hourly). For on-demand real-time recompute, read online features from feature store and call model; limit to low QPS.- Batch push: precompute forecasts for all zones hourly and push to feature store/cache and downstream consumers (dispatch, inventory).- Use CDN or read replicas for high-read volume. Add circuit-breaker fallback to last good forecast.Monitoring & retraining triggers:- Continuous monitoring: data drift (KS/Wasserstein), feature distribution drift, label drift; model performance (MAE, RMSE, CRPS) per zone/cluster; SLA alerts when errors exceed thresholds or business KPIs degrade.- Retrain triggers: automated retrain when sliding-window performance drops by X% vs baseline, major data schema change, or scheduled cadence (hourly incremental, daily full).- Explainability: track feature importances and SHAP drift to detect root causes.- Canary & rollout: shadow mode > canary to 1% > progressive rollout with rollback on degradations.Scalability & trade-offs:- Precompute heavy features & forecasts in batch; keep online path minimal.- Global+local hybrid reduces memory/computation vs full local models while improving per-zone accuracy.- Trade-offs: more frequent retraining costs compute but reduces staleness; caching reduces latency but risks serving stale forecasts—use TTL aligned to acceptable freshness.Operational notes:- Store full lineage in MLflow + feature store for reproducibility.- Use cost-aware loss (asymmetric) if business cost of under-prediction > over-prediction.- Cold-start: initialize new zones with cluster-level model and quick adaptation using few-shot fine-tuning.
HardTechnical
34 practiced
Describe the end-to-end process to validate and certify a fraud detection model for internal audit and regulatory compliance at DoorDash. Cover data lineage, model versioning, reproducibility, threshold selection, human-in-the-loop review policies, logging for audit trails, and how to prepare evidence for auditors.
Sample Answer
Requirements & scope:- Define business objective (reduce chargebacks by X%, maintain false positive rate ≤Y) and regulatory constraints (PCI, privacy/CCPA), SLA for investigation time, and stakeholders (Risk, Legal, Internal Audit).Data lineage and provenance:- Catalog source systems (orders, payments, device, GEO, 3rd‑party signals) in a data catalog (e.g., Amundsen/Collibra) with dataset IDs, owners, schema versions, refresh cadence.- Capture ETL jobs as code (Airflow/DAGs) with commit hashes; record transformations, joins, imputations, and feature engineering steps. Store sample hashes and checksums of raw inputs and derived features for each model run.Model development & versioning:- Use MLflow/Git + DVC to version code, data pointers, and artifacts. Each candidate model is an immutable run with model artifact, params, hyperparams, training dataset pointers, random seeds, and environment spec (conda/docker).- Track performance on held-out validation and backtest time windows; store evaluation metrics and confusion matrices per model version.Reproducibility:- Repro runbook: given model-id + data-snapshot-id + run-id, pipeline can re-create training and scoring. Use containerized CI to re-run end-to-end on demand; retain deterministic seeds and note non-deterministic components.Threshold selection & calibration:- Define operating point using business cost matrix (investigation cost vs fraud loss) and regulatory constraints. Use ROC/precision-recall tradeoffs, expected value optimization (EV = TP*loss_saved - FP*investigation_cost), and select one primary threshold and safe fallback thresholds (conservative/lenient).- Validate thresholds on temporal holdout and stress scenarios (seasonality, promo spikes).Human-in-the-loop & policy:- Define cases for automatic block, manual review, and allowlist. Build triage rules combining model score + rules. Document reviewer workflows, decision codes, SLA, and feedback loop capturing reviewer decisions to retrain models.- Implement periodic reviewer calibration sessions and monitor reviewer accuracy and bias metrics.Logging & audit trails:- For every scored event log: model-id, model-version, data-snapshot-id, feature vector (or hashed/pseudonymized), score, threshold used, decision, reviewer-id (if applicable), timestamps, and downstream action. Store immutable logs (write-once S3 or append-only DB) and index for retrieval.- Retain pipeline logs (ETL runs, infra), CI/CD deploy logs, and access control logs.Validation, testing & bias checks:- Perform backtesting, population stability, adversarial/sensitivity tests, and subgroup fairness analyses. Conduct stress tests (simulated attacks, feature tampering).- Produce Model Risk Assessment (MRA) with assumptions, limitations, and mitigation plans.Preparing evidence for auditors:- Provide a dossier per certified model: - Executive summary: purpose, owners, date, risk rating - Data lineage graph + snapshots (dataset IDs and checksums) - Model version artifacts: code repo commit, environment spec, binary artifact - Training/validation results, calibration plots, threshold decision rationale and cost analysis - Reproducibility runbook with a reproduction test result (CI run id) - Logging schema and sample logs (redacted PII) proving end-to-end traces - Human review policy, reviewer training records, SLA metrics - Change management & deployment history, rollback plan, and post-deployment monitoring metrics (drift, alerting) - Access control and encryption proofs- Offer a live reproduction demo and exportable audit package (PDF + artifact links + checksums).Governance cadence:- Quarterly model risk review, annual recertification, immediate re-evaluation on material data drift, regulatory changes, or major incidents.This process ensures traceability, reproducibility, defensible threshold choices, controlled human oversight, and a complete audit trail suitable for internal audit and regulators.
HardTechnical
32 practiced
Describe an algorithm and provide pseudocode to assign k available drivers to n incoming orders to minimize expected total wait time, considering estimated travel times and existing driver commitments. Discuss complexity and propose heuristics suitable for sub-second decision-making at scale.
Sample Answer
Approach (summary):This is a weighted bipartite matching with dynamic costs: drivers D (k) and orders O (n). Cost c(i,j) = estimated wait time if driver i takes order j = max(0, arrival_time_i_to_pickup_j - order_wait_so_far) plus downstream penalty for driver’s existing commitments (estimated added delay). Exact Hungarian on k×n yields optimal, but O((max(k,n))^3) is too slow at scale. I propose a two-stage approach: compute pairwise costs, then use a fast greedy with local improvement (auction / greedy + 1‑swap) or assign via min-cost flow on pruned graph.Pseudocode (Python-like):Key concepts:- Prune edges using radius or cost threshold to make sparse graph (each order only considers nearest m drivers).- Use fast travel-time estimators (precomputed tiles or ML model) instead of full routing.- Local search (1-swap or small k-swap) often finds near-optimal quickly.- For higher quality under slightly longer latency, use Auction algorithm or incremental min-cost flow on pruned graph.Complexity:- Pairwise cost computation after pruning: O(k * m) where m is drivers considered per order (m << k).- Greedy assignment: O(k * m log(k*m)) for sorting.- Local improvement: O(iterations * k * m).This yields sub-second decisions when m is ~10–50 and travel-time lookups are O(1).Heuristics & production considerations:- Spatial partitioning (geohash) to limit candidates.- Cache travel-time matrix and update incrementally.- Use ML model to predict commitment_penalty and ETA to avoid expensive routing.- Prioritize orders by SLA/priority; preempt low-priority reassignments.- Monitor assignment quality via A/B tests and fallback to exact solver for small batches or offline periodic optimization.Trade-offs:- Greedy+local gives near-optimal with sub-second latency; sacrifices theoretical optimality for scalability.- Pruning risks missing a globally optimal distant driver for high-priority order—mitigate with priority exceptions.
python
# D: list of available drivers with positions/current ETA; O: list of orders with pickup loc/time
# compute travel_time(i,j), commitment_penalty(i,j)
edges = []
for i in D:
for j in O:
eta = travel_time(i, j)
wait = max(0, eta - j.wait_so_far)
penalty = commitment_penalty(i, j) # expected extra wait beyond pickup
cost = wait + penalty
if cost <= COST_THRESHOLD: # pruning
edges.append((i, j, cost))
# Fast assignment: greedy by lowest cost then 1-swap improvement
assign = {}
unassigned_orders = set(O)
# sort edges by cost
edges.sort(key=lambda e: e[2])
for i,j,c in edges:
if i not in assign and j in unassigned_orders:
assign[i] = j
unassigned_orders.remove(j)
# Local improvement: try swapping to reduce total cost
for i1,j1 in list(assign.items()):
for i2 in D:
if i2 not in assign:
# try reassign i1->i2 if lowers total
if cost(i2,j1) + sum_other_costs < current_total:
assign.pop(i1); assign[i2]=j1
# return assignMediumTechnical
31 practiced
Given a stream of events with schema (event_type: 'pickup'/'deliver', order_id, driver_id, event_ts), write Python-like pseudocode to consume events from Kafka, compute per-driver average delivery time in 1-hour tumbling windows, and write the results to Redis for dashboards. Explain handling of late events and state cleanup.
Sample Answer
Approach:Use a Kafka consumer, maintain in-memory per-order state to match pickup→deliver, aggregate per-driver delivery durations into 1-hour tumbling windows, and periodically flush windowed averages to Redis. Use event-time processing with a watermark (allowed lateness) to handle late events and expire state after a retention period.Key points and reasoning:- Use event-time windows (tumbling) so metrics reflect when delivery occurred, not arrival.- Watermark = max_event_time_seen - allowed_lateness; flush when watermark passes window end + allowed lateness.- ALLOWED_LATENESS handles out-of-order/late events; late arrivals within this bound are processed into their window. Beyond it, you can drop, log, or write to a “late-events” store for manual correction.- State cleanup: keep pickups and aggregates with last-update timestamps and evict after STATE_RETENTION to avoid unbounded memory.- For production, use a stream processing framework (Flink, Spark Structured Streaming, Kafka Streams) to get built-in event-time, state TTL, checkpoints, and exactly-once sinks to Redis.
python
# pseudocode Python
from kafka import KafkaConsumer
import redis, time, datetime, heapq, threading
consumer = KafkaConsumer("events", ...)
r = redis.Redis(...)
WINDOW_SIZE = 3600 # seconds
ALLOWED_LATENESS = 300 # seconds
STATE_RETENTION = 24 * 3600 # seconds
# in-memory state:
# pickups: order_id -> (driver_id, pickup_ts)
# aggregates: (driver_id, window_start) -> (sum_durations, count, last_update_ts)
pickups = {}
aggregates = {}
def window_start(ts):
return ts - (ts % WINDOW_SIZE)
def process_event(evt):
etype = evt['event_type']
oid = evt['order_id']
did = evt['driver_id']
ts = int(evt['event_ts']) # epoch seconds
if etype == 'pickup':
pickups[oid] = (did, ts, time.time())
elif etype == 'deliver':
p = pickups.get(oid)
if p and p[0] == did:
duration = ts - p[1]
wstart = window_start(ts)
key = (did, wstart)
s, c, _ = aggregates.get(key, (0,0,0))
aggregates[key] = (s + duration, c + 1, time.time())
# optionally remove pickup since matched
del pickups[oid]
else:
# deliver arrived before pickup or pickup missing -> buffer or store tombstone
# for simplicity, store a small buffer to match later
pickups[oid] = (did, None, time.time()) # None means waiting for pickup
def flush_windows(current_watermark):
# watermark = event_time_now - allowed_lateness
for (did, wstart), (s, c, last) in list(aggregates.items()):
window_end = wstart + WINDOW_SIZE
if current_watermark >= window_end + ALLOWED_LATENESS:
avg = s / c
redis_key = f"driver:{did}:window:{wstart}"
r.hset(redis_key, mapping={'avg_delivery_time': avg, 'count': c, 'window_start': wstart})
r.expire(redis_key, STATE_RETENTION)
del aggregates[(did, wstart)]
def cleanup_state():
now = time.time()
for oid, (_, _, last) in list(pickups.items()):
if now - last > STATE_RETENTION:
del pickups[oid]
for k, (_, _, last) in list(aggregates.items()):
if now - last > STATE_RETENTION:
del aggregates[k]
# main loop (simplified)
watermark = int(time.time()) - ALLOWED_LATENESS
for msg in consumer:
evt = parse(msg)
process_event(evt)
# advance watermark based on max observed event time (omitted: tracking max_event_time)
watermark = int(time.time()) - ALLOWED_LATENESS
flush_windows(watermark)
if int(time.time()) % 60 == 0:
cleanup_state()Unlock Full Question Bank
Get access to hundreds of Delivery Marketplace ML Applications interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.