Delivery Marketplace ML Applications Questions
Machine learning use cases common to on-demand delivery marketplaces, covering demand forecasting, driver/courier dispatch and routing, pricing and revenue optimization, recommendations, fraud detection, and real-time optimization. Includes model development, deployment, monitoring, drift handling, and scalability considerations for ML systems operating in a high-velocity, two-sided marketplace.
HardTechnical
32 practiced
A city shows an unexplained 20% drop in driver acceptance probability overnight. Lay out a step-by-step incident response plan: quick checks, data to inspect (metrics/logs), root-cause hypotheses, short-term mitigations, and metrics to verify resolution. Include categorical causes like outages, incentives change, or model drift.
Sample Answer
Situation summary (goal): unexplained overnight 20% drop in driver acceptance probability (P_accept). Aim: rapidly identify cause, mitigate customer/user impact, and restore baseline while preserving evidence for RCA.1) Immediate quick checks (first 15–30 min)- Verify alerts and incident channel; page on-call SRE, product/ops, marketplace lead, and ML owner.- Check system health dashboards: API latencies, error rates (500/4xx), queue/backpressure, database errors, job failures.- Confirm no scheduled deploys/experiments or config flips in last 24 hours (CI/CD, feature flags, config store).- Check third-party dependencies: SMS/push provider, mapping/geocoding, payments, and telemetry availability.- Look for correlated business signals: surge pricing changes, promotions, region-wide outages (mobile carriers, network).2) Data and logs to inspect (first 30–90 min)- Time-series metrics: P_accept by city, zone, hour; driver online count; new driver logins; offer volume; match latency; trip request rate.- Feature pipeline health: last successful feature calc times, dropped rows, null rates, schema changes.- Model logs: input feature distributions, model confidence, inference errors/exceptions, model-serving latencies, returned scores.- Experiment/flag logs: A/B assignment rates, rollout percentages.- External signals: push delivery rate, push open rate, SMS delivery, notification service logs.- Business telemetry: active incentives, fare adjustments, market ops notes.- Raw event logs for a sample of rejected offers: sequence (offer -> push -> arrival -> driver action) with timestamps.3) Root-cause hypotheses (categorize)- Outage/infra: push/notification provider failures, model-serving endpoint errors, increased latency causing offers to expire before driver sees them.- Incentives/pricing: sudden removal or change of incentives, negative earnings due to pay rollout, surge multiplier bugs.- Experiments/config: a mis-routed experiment or feature-flag flip reducing offer attractiveness (e.g., hide destination info).- Model/feature drift: production input distribution shift (traffic, time-of-day), feature preprocessing bug (nulls/scaling), model weights deployed incorrectly.- Data pipeline corruption: missing/incorrect features (e.g., incorrectly normalized ETA), causing lower predicted acceptance probability.- External/behavioral: weather, local events, regulatory changes, or driver strike decreasing willingness.4) Short-term mitigations (first 1–4 hours) — prefer safety, minimal blast radius- If infra or push failures: temporarily switch to fallback notification channel (SMS) or requeue offers with extended TTL; notify drivers via direct ops channels.- If experiment/config suspected: immediately rollback recent deployments/flags to known-good state (canary first if uncertain).- If model-serving errors or anomalous predictions: switch model endpoint to previous stable model or use a conservative heuristic scoring (fallback rule-based acceptance score).- If incentives/pricing bug: restore previous incentives or add short-term driver bonus in affected zones to restore acceptance.- If pipeline corruption: pause dependent batch jobs, revert to last known-good feature snapshot.- Activate manual escalation and communication: inform ops, CS, and marketing; post in-app notice to riders if dispatch delays expected.5) Verification metrics to monitor for resolution (real-time, 0–24 hours)- Primary: P_accept by city/zone — expect recovery toward baseline within target window (e.g., within 2–6 hours).- Secondary: driver online count, offer-to-accepted latency, push delivery and open rates, model inference success rate, traffic of feature pipeline (row counts).- Business KPIs: completed trips, cancellation rate, rider wait time, driver earnings per hour.- Compare cohorts: affected vs unaffected zones, drivers on previous vs new app versions, drivers receiving promotions vs not.6) RCA follow-up items (post-stabilization)- Root cause confirmation with evidence (logs, diffs, model input distribution delta).- Remediation plan: permanent fix, tests, alerts, and runbooks; add synthetic tests for push + offer flow; add model monitoring (data drift, prediction distribution) and automatic rollback policy.- Process changes: require pre-deploy checks for feature pipeline schema, stronger experiment guardrails, and rollback playbook rehearsals.Key reasoning: prioritize reducing customer impact (fallbacks, incentives, rollbacks) while preserving evidence for a definitive RCA. Use layered hypothesis testing — infra, config, model, data, business — and validate with cohort comparisons and input distribution checks before major changes.
MediumTechnical
34 practiced
Write pseudocode for a greedy online matching algorithm that assigns incoming orders to nearby available drivers subject to: max_pickup_time_seconds, driver_capacity (orders they can batch), and driver acceptance probability threshold. Discuss cases where greedy fails and a better algorithm could be needed.
Sample Answer
Approach: For each incoming order, greedily select the best available driver within max_pickup_time_seconds who has spare capacity and whose acceptance probability ≥ threshold. Score drivers by a weighted function (ETA, remaining capacity, acceptance probability). Assign immediately (online). This is simple, low-latency, and practical for dispatch.Key concepts:- Online decision: immediate assignment without future knowledge.- Scoring balances ETA, remaining capacity, and acceptance probability.- Parameters alpha/beta/gamma tune policy.Time/Space complexity:- Per order: O(D log D) if sorting candidates, or O(D) with single-pass min; D = #drivers. Space O(D) for candidate list.Edge cases:- No candidate within threshold: implement fallback (wait queue, increase radius, guaranteed dispatch with surge).- Rapid driver state changes (concurrency): require atomic updates / optimistic locking.- Acceptance probabilities can be stale—update with live signals.When greedy fails / better algorithms:- High contention: greedy can block better future matchings (e.g., using capacity for low-value order). Use batched matching (periodic windowed optimization) solved via max-weight bipartite matching or min-cost flow to global-optimize assignments.- Stochastic arrivals / uncertainty: use online stochastic optimization or prophet/secretary algorithms with learned thresholds.- Strategic objectives (latency + fairness + revenue): use integer programming / constrained MDP, or RL-based dispatch policies trained in simulation to trade off long-term metrics.- Scalability with complex constraints (multi-stop routes, time windows): use route-aware assignment (VRP heuristics) rather than per-order greedy.Recommendation: Start with greedy for low latency; monitor regret metrics (missed high-value matches). If regret grows or KPIs suffer, migrate to hybrid: short batching windows + fast solver or learned policy (RL/heuristic) for near-optimal online matching.
python
# Pseudocode / Python-like
def greedy_dispatch(order_stream, drivers, max_pickup_time_seconds, driver_capacity, accept_prob_threshold):
# drivers: dict driver_id -> {location, capacity_used, accept_prob, available:bool}
# order_stream: yields orders with {order_id, location, time_received}
for order in order_stream:
candidates = []
for d_id, d in drivers.items():
if not d['available']:
continue
remaining = driver_capacity - d['capacity_used']
if remaining <= 0:
continue
eta = estimate_travel_time(d['location'], order['location'])
if eta > max_pickup_time_seconds:
continue
if d['accept_prob'] < accept_prob_threshold:
continue
# score: lower is better
score = alpha*eta - beta*remaining - gamma*d['accept_prob']
candidates.append((score, d_id, eta))
if not candidates:
handle_unassigned(order) # e.g., queue, expand radius, or notify customer
continue
# pick best (min score)
candidates.sort()
best_score, best_driver_id, best_eta = candidates[0]
assign_order_to_driver(order['order_id'], best_driver_id)
drivers[best_driver_id]['capacity_used'] += 1
if drivers[best_driver_id]['capacity_used'] >= driver_capacity:
drivers[best_driver_id]['available'] = FalseMediumTechnical
39 practiced
Design a safe deployment strategy to roll out a new pricing model that could change customer fees and driver payouts in multiple regions. Include canary/gradual rollouts, monitoring KPIs, automatic rollback criteria, and human approvals for high-risk regions.
Sample Answer
High-level approach: treat the new pricing model as a high-risk feature by using staged rollout, strong monitoring (business + ML), automated safety gates, and manual approvals for sensitive regions.Steps:1. Pre-launch validation- Run offline fairness and impact analysis (customer spend distribution, driver payout delta, churn risk) with historical data and counterfactual simulations.- Shadow mode: run model in production path without affecting prices; log decisions and predicted impacts.2. Canary / gradual rollout- Canary 1 (internal + 0.1% users in low-risk region): enable via feature flag. Use canary cohort for 48–72 hours.- Canary 2 (1% users across few regions): broaden to representative cohorts.- Phased regional rollouts (5% → 20% → 50% → 100%) per region with at least 24–72 hours and stable KPIs between steps.3. Monitoring KPIs (real-time + hourly/daily dashboards)- Business KPIs: revenue per trip, take rate, average customer fee, driver payout, activation/acceptance rate, churn/complaints, booking conversion.- ML/operational KPIs: prediction calibration, input feature drift, model inference latency, error rate.- Safety KPIs: percent of users exceeding X% fee change, drivers with payout drop >Y%.4. Automatic rollback criteria (fail-fast)- Immediate rollback triggers (auto): - Revenue or bookings drop >5% vs baseline in 1 hour - Driver acceptance rate down >10% absolute in 2 hours - Customer complaints/ticket rate >3x baseline - Model input drift beyond threshold (statistical test)- Graceful rollback: disable feature flag to previous pricing logic and notify on-call.5. Human approvals / governance- High-risk regions (regulated markets or regions with >Z% projected negative impact) require explicit approval from Product, Legal, and Ops before progressing beyond Canary 2.- Approval checklist: shadow validation report, KPI baselines, rollback plan, stakeholder sign-off.- Post-rollout review within 7 days for adjustments.6. Additional controls- Rate limiting: cap maximum price change per user/day.- A/B testing for incremental optimization; record causal impact.- Audit logs, explainability reports for ML decisions, and customer/driver communication plan.Why this works: combines statistical validation, staged exposure, automated safety nets, and human judgment where regulation or impact warrants it—minimizing user harm while allowing iterative improvements.
HardSystem Design
33 practiced
You must build a real-time feature that computes average driver ETA to restaurants updated every 10 seconds. Propose a streaming architecture (ingestion, windowing, state store), how to handle late/out-of-order events, and how to ensure low read-latency for the serving layer.
Sample Answer
Requirements & constraints:- Update average driver ETA to each restaurant every 10s, low read latency (<100ms), tolerate out-of-order and late GPS/ETA events, high throughput (millions events/min), fault-tolerant and horizontally scalable.High-level architecture:- Ingestion: Kafka topic per region (partitioned by restaurant_id). Producers (mobile SDKs / driver services) push ETA events (driver_id, restaurant_id, eta_ts, eta_seconds, event_ts).- Stream processor: Apache Flink cluster (or Kafka Streams) consuming Kafka, performing windowing, stateful aggregation, and writing deltas to serving layer.- State store: Flink-managed RocksDB state for per-key aggregates, checkpointed to durable storage (S3).- Serving layer: Materialized view exposing latest averages in low-latency store (Redis or DynamoDB with DAX). Use change-log writes from Flink (exactly-once) to update store.Windowing & aggregation:- Use tumbling windows of 10s aligned to event time to compute per-window avg and count. Maintain incremental aggregates (sum, count) in keyed state for restaurant_id so Flink can update on arrival.- Emit final aggregation at window close and also emit incremental updates every second if UI needs live smoothing.Handling late/out-of-order events:- Use event-time processing with watermarks based on ingestion patterns (e.g., allowed lateness = 30s). Configure periodic watermarks with maxOutOfOrderness.- For events arriving within allowed lateness: update window state and emit correction/update to serving store (idempotent updates).- For very late events beyond allowed lateness: route to a side-output for offline reconciliation/batch job that can backfill historical metrics and optionally trigger corrective deltas if needed.- Use Flink’s exactly-once semantics (checkpointing + Kafka transactions) to avoid double counts.State and correctness:- Store (sum, count, last_update_ts) per restaurant in RocksDB keyed state. Persist checkpoints frequently (e.g., every 5s) and enable incremental snapshots.- Use vector clocks or monotonic sequence numbers per driver event to deduplicate if producers can resend.Low read-latency serving:- Flink writes computed aggregates (or just deltas) to Redis sorted by restaurant_id or to DynamoDB for persistence; updates are applied atomically.- Read path: API gateway + caching layer (edge CDN or local Redis) serving <50ms. Use fan-out writes to update both Redis (fast reads) and a durable store (DynamoDB) for fallback.- For scalability, partition serving keys same as Kafka partitions to co-locate processing and reduce cross-shard writes.Operational considerations & trade-offs:- Tradeoff between allowed lateness and freshness: larger lateness improves accuracy but increases corrections and staleness. Pick based on observed skew.- If strict consistency required, accept slightly higher read latency by reading directly from state via a queryable state or RPC to Flink; otherwise materialized store gives fastest reads with eventual consistency.- Monitor watermark lag, processing latency, and correction rate; implement alerts and autoscaling.Example flow:Driver emits ETA -> Kafka partition -> Flink keyed by restaurant_id -> event-time tumbling 10s window, update RocksDB (sum,count) -> emit avg -> write to Redis and Kafka change-log -> API reads Redis for UI. Late events within allowed lateness update RocksDB and send correction to Redis; beyond lateness go to side-output for offline handling.
MediumTechnical
40 practiced
You must reduce inference cost for a large transformer-based model used in merchant recommendations. Compare three model compression techniques—knowledge distillation, pruning, and quantization—in terms of expected latency, accuracy loss, and implementation complexity for DoorDash production.
Sample Answer
Situation/context: We need to reduce inference cost for a large transformer recommender in DoorDash production while keeping latency tight and recommendation quality high.Comparison (expected latency, accuracy loss, implementation complexity):1) Knowledge Distillation- Latency: Moderate — student model size governs speed; a well-designed smaller transformer (e.g., 3–6 layers) can reduce latency 3–10x relative to teacher.- Accuracy loss: Low-to-moderate — with behavioral + response/feature matching, often retains 90–98% of teacher metrics on ranking/relevance.- Complexity: High (training) / Low (serving) — requires building student architecture, multi-loss training (soft labels, intermediate-layer matching), extra offline compute and hyperparameter tuning. Serving is simple (single smaller model).When to pick: If you can afford offline training and want best quality-speed tradeoff.2) Pruning- Latency: Variable — unstructured pruning reduces flop count but not always wall-clock latency unless libraries/hardware exploit sparsity; structured pruning (heads, layers, neurons) yields reliable latency gains (2–5x).- Accuracy loss: Variable — moderate if aggressively pruned; structured pruning with gradual magnitude or L0 methods keeps loss lower.- Complexity: Medium — requires iterative prune–fine-tune cycles, tooling for structured pruning, and validation that production inference stack uses pruned model efficiently.When to pick: If you need minimal infra change and can target specific modules (e.g., remove attention heads).3) Quantization- Latency: High improvement — INT8 (or mixed INT8/FP16) often gives 2–4x latency and memory reduction; on CPU inference gains can be >4x with optimized kernels.- Accuracy loss: Low (for post-training quantization with calibration) to moderate (aggressive INT4 or quantizing activations); quant-aware training minimizes loss.- Complexity: Low-to-medium — many frameworks offer quant tooling (ONNX, TensorRT, FBGEMM) but you must validate calibrations, per-channel weight quant, and handle operator support; deployment may require new runtime or engine.When to pick: Fastest win for inference latency and cost when runtime supports quantized kernels.Recommendation for DoorDash production:- First pass: apply INT8 quantization (post-training + calibration) on current model to get quick latency/cost wins.- Parallel: train a distilled student model to target production latency with minimal metric regression; use it for high QPS paths.- Use structured pruning selectively for components that hurt latency and when runtime sparsity support exists.- Validate end-to-end: A/B test on business KPIs, monitor tail latency, and ensure reproducibility across hardware (GPU/CPU) and serving stacks.
Unlock Full Question Bank
Get access to hundreds of Delivery Marketplace ML Applications interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.