Ride-Hailing Product Problems & Analytical Approaches Questions

Ride-hailing and on-demand transportation product problems: problem framing, hypothesis generation, and data-driven decision making for marketplace platforms connecting riders and drivers. Covers experimentation design (A/B testing, guardrail metrics, sample-size and power calculations), marketplace health metrics (supply, demand, financial, and user-experience dimensions), funnel and conversion analysis across the request-to-completion flow, feature prioritization frameworks (e.g. RICE), ETA and matching model tradeoffs, and stakeholder alignment to improve rider and driver experience and marketplace efficiency.

MediumTechnical

24 practiced

You want to evaluate a new dispatch policy offline using logged data. Each log row contains context, action taken, action_propensity under the logging policy, and reward (e.g., completed trip, rating). Describe inverse propensity scoring (IPS), direct method (DM), and doubly robust (DR) estimators. Explain their assumptions, pros/cons, variance considerations, and steps you'd take to validate that offline estimates are reliable enough to inform a live rollout.

Sample Answer

Inverse Propensity Scoring (IPS), Direct Method (DM), and Doubly Robust (DR) are standard off-policy evaluation estimators for logged contextual bandit / dispatch data.

IPS- What: Weight each observed reward by inverse of the logging policy propensity: E[r | policy] ≈ (1/N) Σ r_i * (π_new(a_i|x_i) / p_i).- Assumptions: Logging propensities p_i > 0 for actions that π_new would take (support / overlap).- Pros: Unbiased if propensities are correct; simple.- Cons: High variance when propensities are small or π_new differs a lot from logging policy; sensitive to propensity mis-specification.- Variance: Var scales with importance weights; large weights -> unstable estimates. Common fixes: weight clipping, self-normalized IPS.

Direct Method (DM)- What: Fit a model E[r | x,a] (reward model) from logs, then estimate policy value by averaging predicted rewards under π_new: (1/N) Σ Σ_a π_new(a|x_i) * r̂(x_i,a).- Assumptions: Reward model is well-specified / accurate.- Pros: Low variance if model is good; smooth estimates across actions.- Cons: Biased if reward model is misspecified; can underestimate uncertainty.- Variance: Generally low but bias dominates if model wrong.

Doubly Robust (DR)- What: Combines IPS and DM: value ≈ (1/N) Σ [r̂(x_i,π_new) + (π_new(a_i|x_i)/p_i) * (r_i - r̂(x_i,a_i))]. Uses model prediction plus weighted residual.- Assumptions: Consistent if either propensities OR reward model is correct (hence "doubly robust").- Pros: Lower bias than DM when model imperfect, lower variance than IPS; more stable in practice.- Cons: Still vulnerable if both components are poor; importance weights can inflate variance—clipping/self-normalization often applied.- Variance: Improved over IPS but depends on residual variance and weights.

Practical validation steps before rollout1. Data checks: verify logging propensities are recorded and >0 for relevant actions; check distribution overlap between logging and candidate policy.2. Sanity tests / off-policy unit tests: - Recompute known-policy values (replay test) where π_new = logging policy → estimators should match observed average reward. - A/B replay: simulate simple deterministic policies with known outcomes.3. Model evaluation: - For DM/DR: evaluate reward model via cross-validation and calibration; check error on held-out sets and by action strata. - Examine residuals vs. context and actions.4. Robustness checks: - Compare IPS, DM, DR estimates — large disagreements indicate problems. - Sensitivity to weight clipping, self-normalization, or alternative propensity models. - Bootstrap confidence intervals (or asymptotic SE) to quantify uncertainty.5. Subgroup analyses: estimate performance across segments (region, time-of-day, driver supply) to detect heterogenous effects or support gaps.6. Small-scale live experiments: if offline estimates look promising, run a safe, constrained online A/B test (e.g., small percentage, conservative allocation) to confirm uplift and detect unmodeled dynamics.7. Monitoring plan: define metrics, guardrails, and rollback thresholds for rollout.

Summary: Use DR as default for balance of bias/variance, validate propensities and reward model thoroughly, run multiple diagnostics (replay, cross-val, disagreement, bootstrap), and confirm with a small live test before full deployment.

HardTechnical

28 practiced

In two-sided marketplaces, SUTVA is often violated. Compare and contrast these experimental designs to handle interference: cluster randomization by zone, graph-cluster randomization, switchback/stepped-wedge designs, and network-based randomized encouragement. For each, discuss strengths, weaknesses, sample-size implications, and analysis adjustments you would make.

Sample Answer

Brief framing: interference (violations of SUTVA) is ubiquitous in two-sided marketplaces — treatments on one side affect outcomes on other nodes via matching, prices, or visibility. Below I compare four pragmatic designs, focusing on strengths, weaknesses, sample-size implications, and required analysis adjustments.

1) Cluster randomization by zone (geographic or logical market)- Strengths: easy to implement, minimizes cross-cluster interference if clusters are well separated; operationally simple for rollout.- Weaknesses: residual spillovers across borders (edge effects); heterogeneity across clusters creates imbalance risk.- Sample-size: effective sample size equals number of clusters, not individuals — often large ICC (intra-cluster correlation) inflates required clusters; use cluster-level power calculations.- Analysis: cluster-level summaries or mixed models with cluster fixed/random effects; adjust SEs for clustering (cluster-robust or permutation at cluster level); check for spatial spillover with boundary analyses.

2) Graph-cluster randomization (cluster by network communities)- Strengths: respects observed network edges, reducing within-cluster interference; better when interference follows known graph.- Weaknesses: requires good network data and scalable clustering; imperfect community detection leaves cross-cluster ties.- Sample-size: sample size driven by number of graph clusters; clustering quality influences ICC; simulate power under realistic propagation models.- Analysis: randomization-based inference conditional on graph clusters (e.g., permutation across clusters); use exposure mapping (treatment fractions among neighbors) and model-based GEE or hierarchical models; sensitivity to misspecified network.

3) Switchback / stepped-wedge designs (time-based crossover)- Strengths: each unit can serve as its own control improving power; practical when treatments are policy/time-varying.- Weaknesses: temporal carryover and time trends confound effects; interference across units persists if contemporaneous interactions matter.- Sample-size: temporal correlation reduces independent observations; need longer time series and account for autocorrelation in power calculations.- Analysis: difference-in-differences or panel models with unit and time fixed effects; explicitly model carryover (lag terms); cluster SEs by unit and consider permutation across time blocks.

4) Network-based randomized encouragement (instrumental-variable style)- Strengths: randomize encouragement (e.g., messages) rather than treatment; can estimate local average treatment effects when uptake varies and interference operates via encouragement diffusion.- Weaknesses: requires valid IV assumptions (exclusion, monotonicity); partial compliance and complex interference of encouragement itself.- Sample-size: weaker instruments (low encouragement->treatment uptake) inflate variance — use IV power formulas and plan for larger sample.- Analysis: two-stage least squares with network-aware standard errors; define exposure mappings and estimate direct/indirect effects (e.g., via complier average causal effects under interference); conduct falsification and placebo tests.

Cross-cutting practices:- Pre-specify exposure mappings (who is affected by whose treatment).- Use simulation-based power using realistic network/matching models.- Report ITT, complier and spillover (indirect) effects; use randomization inference where possible for valid p-values under interference.- Sensitivity analyses to unmeasured cross-cluster ties, time trends, and compliance.

This set of designs trades operational simplicity against statistical complexity; choose based on where interference primarily flows (geography, explicit network, or time) and budget for more clusters/time or stronger instruments accordingly.

EasyTechnical

21 practiced

As a Data Scientist at Lyft, list and define the top 6 product metrics you would track to evaluate the health of the rider-driver marketplace. For each metric, explain why it's important, what direction indicates improvement, and one guardrail metric you'd monitor to avoid negative side-effects (for example: optimizing completed-rides could reduce average driver earnings or increase cancellations). Consider metrics across supply, demand, financial, and user-experience dimensions and briefly state how you'd compute each from event logs.

Sample Answer

1) Completed Rides / Matching Rate Definition: Number of trips that start and complete divided by ride requests. Why: Core demand fulfillment signal — shows marketplace liquidity. Improvement: higher. Guardrail: Cancellation Rate (if completed rises but cancellations spike, quality suffers). Compute: From event logs count request_id where status == "completed" / count(request_id where status == "requested").

2) Time-to-Pickup (Median ETA realized) Definition: Median time from driver accepted to rider pickup (or from request to pickup if auto-matched). Why: Direct UX latency impacting retention. Improvement: lower. Guardrail: Driver Utilization (avoid reducing time-to-pickup by oversupplying drivers causing low utilization). Compute: median(pickup_time - accept_time) per city/hour from trip events.

3) Driver Supply Active / Utilization Rate Definition: Active drivers online; Utilization = time-with-passenger / time-online. Why: Supply health and driver earnings proxy. Improvement: more active drivers and stable/increasing utilization. Guardrail: Average Earnings per Hour (prevent utilization gains that cut earnings). Compute: count(distinct driver_id where status=="online"); utilization = sum(driver_trip_seconds)/sum(driver_online_seconds).

4) Match Success / Fulfillment Latency (Share of requests matched within X minutes) Definition: Fraction of requests matched to a driver within target minutes. Why: Measures responsiveness and market tightness. Improvement: higher. Guardrail: Surge Frequency (excessive surge may mask unmet demand). Compute: count(requests with matched_time - request_time <= X)/total requests.

5) Gross Booking Value (GBV) and Net Revenue per Trip Definition: Sum of fares (GBV) and company take-rate * GBV (net revenue) per period. Why: Financial health and pricing effectiveness. Improvement: GBV and sustainable net revenue growth. Guardrail: Price Sensitivity / Churn (rising fares hurt retention). Compute: sum(fare_amount) and sum(fare_amount * take_rate) from trip payments table.

6) Rider & Driver Retention (30-day active cohorts) and NPS/CSAT Definition: % of users active again within 30 days; NPS/CSAT from surveys. Why: Long-term health — acquisition efficiency and satisfaction. Improvement: higher retention and NPS. Guardrail: Average Trip Frequency (avoid churn masking by fewer but high-value trips). Compute: cohort(user_signup_week) -> fraction with >0 trips in days 1–30; NPS computed from survey responses linked to trip_id.

For each metric, monitor by city/segment and hour/day granularity; annotate known experiments. Prioritize a single Overall OEC (e.g., weighted combination of completed rides, pickup time, driver earnings, and retention) agreed with stakeholders to guide optimization while using guardrails to prevent adverse side-effects.

HardTechnical

21 practiced

You trained an uplift model using historical promotion data, but treatment assignment in that history was non-random (marketing targeted specific users). Describe methods to debias uplift estimation (propensity scores, inverse-probability weighting, doubly-robust learners, causal forests with orthogonalization), validation strategies using RCT-held-out data or backtests, and operational guardrails to avoid targeting mistakes that could harm metrics or fairness.

Sample Answer

Begin by framing the problem: non-random treatment (targeting) creates selection bias — P(T|X) ≠ random — so naive difference-in-means or predictive models will conflate selection effects with causal effect. I would address this in three layers: estimation methods, validation, and operational guardrails.

Estimation methods (de-biasing)- Propensity-score modeling: estimate e(X)=P(T=1|X) with a well-calibrated classifier (logistic/GBM) and check overlap/positivity. Use trimming where e(X) near 0 or 1 to avoid high variance.- Inverse-probability weighting (IPW): re-weight outcomes by 1/e(X) or 1/(1−e(X)). Stabilize weights (multiply by marginal treatment rate) and clip extreme weights.- Doubly-robust learners: use DR-learner or augmented IPW so consistency holds if either propensity or outcome model is correct. This reduces bias and variance relative to pure IPW.- Orthogonalization / Neyman-orthogonal estimators: residualize outcome and treatment (e.g., T − e(X) and Y − m(X)) to build orthogonal scores; this improves robustness to nuisance estimation.- Causal forests / generalized random forests with honesty and orthogonalization (Athey & Wager): nonparametric heterogeneous treatment effect estimation that internally orthogonalizes and provides variance estimates.- Practical tips: cross-fitting (K-fold) to avoid overfitting nuisance models, hyperparameter tuning on DR loss, check balance after weighting.

Validation strategies- Holdout RCT if available: the gold standard—compare uplift estimates and policy decisions against outcomes in RCT subset; measure calibration of predicted uplift and policy value.- Off-policy policy evaluation / offline A/B: estimate expected policy value using IPS and doubly-robust policy value estimators, with bootstrap CIs and variance estimates.- Backtesting on temporal holdouts: simulate historical rollouts using time-split data and IPS/DR evaluation to detect covariate shift.- Calibration and ranking checks: group users by predicted uplift decile and compare observed uplift (using weighted estimators) to predicted; check coverage of CIs from causal forest.- Sensitivity analyses: vary propensity model, clipping thresholds, and conduct placebo tests (e.g., on pre-treatment outcomes) to detect residual confounding.

Operational guardrails- Conservative rollout: start with small randomized exploration (epsilon-greedy or Thompson sampling) and ramp up as offline and small RCTs validate lift.- Thresholding + caps: require minimum predicted uplift and enforce exposure caps per segment to limit risk and prevent spurious extreme targeting from noisy estimates.- Fairness and safety checks: monitor disparate impact across protected groups; add constraints or regularization to the policy (e.g., demographic parity bounds, minimum treatment rates).- Monitoring: real-time dashboards for key metrics (response, revenue, churn), statistical alarms for deviation from expected uplift, and automated rollback if negative impact detected.- Explainability: log feature-level contributions for high-value targets so product/marketing can review decisions.- Governance: A/B test any automated policy, maintain an experiment registry, and require business sign-off for high-risk segments.

Why these choices: propensity/IPW correct for selection flows but are high-variance; DR and orthogonal methods combine strengths to reduce bias and variance and protect against misspecification; causal forests add nonparametric flexibility and uncertainty quantification. Validation against RCTs or robust off-policy estimators is essential before production, and operational guardrails (small rollouts, clipping, fairness constraints, monitoring) prevent harmful decisions from noisy uplift estimates.

HardTechnical

31 practiced

Your ETA prediction model's error increased by 30% in a single city over the last week. Walk through a diagnostic checklist (data ingestion, feature distributions, label generation, serving vs training differences), propose SQL checks or visualizations to localize the issue, and describe the retrain vs rollback decision process including risk mitigation and communication to ops/PMs.

Sample Answer

Start with a structured diagnostic checklist, then propose concrete checks and visualizations, and finish with a retrain vs rollback decision process plus mitigation and communication.

Diagnostic checklist (priority order)1. Data ingestion & pipeline health- Check recent job failures, late/duplicate batches, schema changes, or source API errors.- SQL: find missing partitions or delayed timestamps:

sql

SELECT date_trunc('hour', event_time) AS hr, COUNT(*) AS rows
FROM raw_trips
WHERE city = 'X'
GROUP BY hr
ORDER BY hr DESC
LIMIT 48;

2. Label generation correctness- Verify labels computed last week (trip_end - trip_start) aren’t biased by timezone shifts, daylight savings, or truncated logs.- SQL: compare label distribution week-over-week:

sql

SELECT week, PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY duration) AS median_dur
FROM (
  SELECT DATE_TRUNC('week', trip_end) AS week, EXTRACT(EPOCH FROM (trip_end - trip_start))/60 AS duration
  FROM processed_labels WHERE city='X'
) t GROUP BY week ORDER BY week;

3. Feature distribution drift- Check numeric and categorical drift (means, quantiles, new categories).- SQL: feature histograms or summaries by week:

sql

SELECT week, AVG(speed), PERCENTILE_CONT(0.9) WITHIN GROUP(ORDER BY speed) AS p90
FROM features WHERE city='X' GROUP BY week;

Visualizations: weekly KDEs, boxplots, population stability index (PSI) per feature, and UMAP/t-SNE embedding of feature vectors to spot clusters.

4. Serving vs training skew- Ensure online feature values match offline training snapshots (freshness, aggregation windows).- SQL: compare feature computation windows:

sql

SELECT feature_name,
  AVG(CASE WHEN source='train' THEN val END) AS train_mean,
  AVG(CASE WHEN source='serve' THEN val END) AS serve_mean
FROM feature_samples
WHERE city='X' GROUP BY feature_name;

5. Labels leakage/regression in ground truth- Check for changes in logging (e.g., new delays causing finish events to be recorded differently).

6. External/contextual causes- Event calendar, roadworks, policy changes, new traffic sensors — correlate error spike with external signals.

Localizing the issue- Compute error by slice: geography (neighborhood), hour-of-day, vehicle type, driver, route length.- Visualization: heatmap of MAE by neighborhood-week and hourly MAE trend lines.

Retrain vs rollback decision process1. Root-cause found and fixable in data (e.g., ingestion bug, label calc bug): rollback model and fix pipeline before retrain. Rationale: model is fine but fed bad data.2. No pipeline bug; features drifted but retraining with recent data corrects behavior: prefer retrain with updated feature engineering, include drifted period as separate segment or adversarial validation. Evaluate offline: backtest (walk-forward) on last 4 weeks; require metrics threshold improvement and stability.3. If uncertain or risk of downstream harm high: rollback to last known-good model while investigating.

Risk mitigation steps- Canary/gradual rollout or shadow serving while retraining.- Add feature gates and alerting for PSI and label-change thresholds.- Add quick-fix rules (bias correction factor by neighborhood/time) if safe.

Communication plan- Immediate: notify Ops and PMs with incident summary — scope (city X), impact (30% MAE increase), initial hypothesis, mitigation (rollback/canary), ETA for next update.- Follow-up: daily status with findings, simulated impact of fixes, expected timeline.- Postmortem: root cause, actions taken, metrics before/after, changes to monitoring, timeline to prevent recurrence.

Example quick checks to run immediately: hourly MAE by zip, count of processed trips vs expected, PSI for top 10 features, and label distribution check. Prioritize rollback if ingestion/label bug confirmed; otherwise run guarded retrain with A/B canary and monitor business KPIs.

Unlock Full Question Bank

Get access to hundreds of Ride-Hailing Product Problems & Analytical Approaches interview questions and detailed answers.

Join thousands of developers preparing for their dream job.