DoorDash Key Metrics & Dashboard Requirements Questions
Defining and standardizing DoorDash KPIs, identifying data sources, calculating metric definitions, data governance, and designing dashboards and reporting pipelines to monitor product and business performance. Includes data visualization best practices, dashboard design, interactivity, drill-down capabilities, and alignment with business goals across operations, product, and marketplace analytics.
HardTechnical
68 practiced
Upstream, the base order event schema changed (new statuses added), and several derived metrics (take-rate, cancellation rate) now disagree with previous values. Design a process to catch such regressions early: automated tests, metric lineage, canary checks, alerting, and rollback strategies. Describe tooling you'd use and indicators that show a breaking change.
Sample Answer
High-level goal: detect upstream schema changes (e.g., new order statuses) before they silently corrupt derived metrics, and enable fast remediation and safe rollback.Process1. Prevent/Detect upstream schema changes- Contract + governance: require schema change PRs, maintainer sign-off, and versioned schema registry (e.g., Confluent Schema Registry for Kafka).- CI gate: schema-change PRs must include an impact analysis (which downstream tables/metrics reference status field).2. Automated tests (pre- and post-deploy)- Unit/regression tests for transformation logic (dbt tests for SQL models): assert expected status enumerations, counts sum to source, no unknown status values.- Data-quality tests (Great Expectations/Deequ): expect distribution of order.status; fail if new/nullable/unknown statuses exceed threshold.- Metric-level tests: golden values for critical metrics (take-rate, cancellation rate) computed on recent snapshot; CI asserts percent-change within tolerance.3. Metric lineage & visibility- Catalog and lineage (Amundsen/Marquez + dbt docs): maintain mapping from source fields → models → metrics → dashboards. On schema change, automatically list impacted metrics and owners.- Owners: notify metric/product owners via Slack/issue template.4. Canary checks & monitoring- Canary pipeline: deploy transformation change to a small shard (region/customer segment) or run in shadow mode; compare metric outputs to baseline.- Runtime checks: automatic canary queries after deploy (compare last-24h vs previous window). Use anomaly detection (Monte Carlo, Datadog/Prometheus APM) to flag statistically significant deviations.- Key indicators: - Sudden non-zero counts of “unknown” or new statuses - Per-status counts diverging >X% (configurable) from baseline - Derived metrics (take-rate, cancellation rate) changing beyond SLA (e.g., >5% absolute or >20% relative) - Downstream SQL errors, increases in nulls or dropped rows5. Alerting and escalation- Multi-channel alerts (PagerDuty/Slack/email) with context: diff of pre/post metric values, lineage, recent deploys.- Auto-create incident with runbook, including rollback steps, owners, and urgency.6. Rollback & remediation- Fast rollback: revert transformation code via CI/CD (git revert) or flip feature flag to use previous stable mapping.- Mitigation: if upstream change is intentional, update transformation mapping and tests, run backfill job to correct historical derived metrics (Airflow triggered job). Prioritize partial backfill for critical windows.- Postmortem: add test cases for the new status in regression suite and update catalog.Tooling summary- Schema registry: Confluent Schema Registry or protobuf- ETL orchestration: Airflow- Transformations + tests + lineage: dbt + dbt docs; Amundsen/Marquez- Data-quality: Great Expectations, Deequ- Monitoring/anomaly detection: Datadog/Prometheus, Monte Carlo, Looker/LookML for metric alerts- CI/CD: GitHub Actions/GitLab + feature flags (LaunchDarkly)- Incident management: PagerDuty, Slack, Confluence runbooksWhy this works- Combines preventive governance, automated tests, and real-time canary/alerting so schema drift is caught at PR, CI, and production stages. Lineage ensures rapid owner notification and focused remediation; rollbacks and backfills limit business impact.
HardTechnical
76 practiced
Design an attribution methodology for DoorDash orders to marketing channels when users have multiple touchpoints across devices and delayed conversions. Compare first-touch, last-touch, time-decay, and data-driven attribution approaches. Propose implementation steps, required data sources (UTM, ad logs, cookies), and validation strategy using incremental experiments.
Sample Answer
Goal: attribute DoorDash orders to marketing channels accurately across devices and delayed conversions so marketing can optimize spend while respecting privacy and scale.High-level approach- Use a hybrid system: deploy a data-driven (causal) model as the long-term truth for planning and budget allocation, backed by pragmatic rules (time-decay + last-touch) for near-real-time reporting and channel-level dashboards.Compare approaches- First-touch: credits the first recorded interaction. Pros: simple, good for awareness metrics. Cons: overweights top-of-funnel, ignores later conversion intents and cross-device friction.- Last-touch: credits final interaction before conversion. Pros: intuitive for optimization, low latency. Cons: biased toward retargeting, ignores assisted channels.- Time-decay: splits credit by recency (exponential or log). Pros: balances early and late influences. Cons: arbitrary decay params, still descriptive not causal.- Data-driven (causal/attribution modeling): uses uplift modeling, multi-touch heterogeneous effects, or Shapley-value–style decomposition with experimentation priors. Pros: estimates marginal impact, better for budget decisions. Cons: needs rich data, compute, and validation via experiments.Implementation steps1. Requirements & constraints: define objectives (ROAS, incremental orders), privacy limits (PII minimization, GA4 cookieless), latency needs.2. Instrumentation: standardize UTM tagging, capture ad logs (impression, click, campaign IDs), server-side event tagging (app open, add-to-cart, order), and deterministic cross-device keys where available (logged-in user IDs).3. Linkage: build deterministic join on user_id for logged-in sessions; probabilistic/linking fallback using hashed device signals and last-touch heuristics for guest flows while respecting privacy.4. Interim reporting: implement time-decay attribution with configurable half-life (e.g., 7 days) for operational dashboards.5. Data-driven model: build uplift models (random forests / causal forest) or use Shapley decomposition on user-level paths aggregated to campaigns. Train on features: exposures, recency, frequency, creative, channel, user propensity, seasonality.6. Experimentation backbone: design and run holdout and geo/auction-randomized incrementality tests to estimate causal lift per channel/campaign and calibrate model priors.Required data sources- UTM parameters, click IDs (gclid, fbclid), campaign metadata- Ad platform logs (impressions, bids, spend, creative ids)- Server-side event logs from app/web (user_id, timestamps, events)- Cookie/device IDs where allowed; hashed identifiers for linking- Conversion outcomes (orders, lifetime value, cancellations)- Contextual data: time, location, promotions, store availabilityValidation strategy- Incrementality experiments: - Holdout experiments: random user-level holdouts for small campaigns to get unbiased lift. - Geo/region experiments for supply-side channels where user-level randomization is hard. - Auction-randomized experiments (if supported) to randomize auctions for ad exposure.- Use experiments to: - Measure channel-level incremental ROAS and calibrate model outputs. - Validate model assumptions (e.g., decay half-life) and Shapley attributions. - Monitor heterogeneity (by cohort, device, new vs returning users).- Ongoing metrics: predicted vs experimental lift mismatch, stability over time, and business KPIs (orders, AOV, retention). Retrain model when drift exceeds threshold.Governance & privacy- Prefer server-side linking on hashed user_id; avoid storing PII in ad logs.- Implement differential privacy / aggregate reporting for sensitive slices.- Maintain an experimentation calendar to avoid contamination and coordinate spend.Outcome- Short term: time-decay reports for ops; experiments to produce channel-level incrementality estimates.- Medium term: deploy calibrated data-driven model for budget allocation and bidding, backed by continuous experimental validation to ensure causal accuracy and business-aligned decisions.
MediumSystem Design
92 practiced
Design a dashboard to monitor promotion and coupon performance on DoorDash. Requirements: attribute orders to promotions, show redemption rate, incremental lift vs baseline, cost per incremental order, and impact on merchant payout. Describe data requirements, attribution windows, cohort comparisons, and recommended visualizations for product and finance stakeholders.
Sample Answer
Requirements (clarify): attribute every order to promo/coupon (single or stacked), show redemption rate, incremental lift vs baseline, cost per incremental order, and merchant payout impact for Product and Finance. Support A/B experiment and ongoing campaigns.Data requirements:- Event streams: impression, promo eligibility, promo click, promo apply, order placed, order completed, refund/cancel, merchant payout amounts, fees, discounts, timestamped and user/order ids.- Promo metadata: promo_id, type (percent/fixed/free-delivery), eligible criteria, channel, budget, start/end, stacking rules.- User/account metadata: user cohort, region, device, first-order date.- Historical baseline dataset for lift calculations.Attribution windows & logic:- Primary attribution: orders where promo was applied at checkout (explicit apply) → direct attribution.- Implied attribution: orders within X hours (configurable, default 7 days) after exposure/eligibility but without explicit apply — flag as influenced using decay weighting.- For stacked promos, assign primary attribution to the promo with highest discount or last-touch depending on business rule; report stacked contributions separately.- Use lookback windows for refunds (30 days) and for incremental measurement align with experiment duration.Cohort comparisons & lift:- For A/B tests: randomize at user level; compare treated vs control on conversion rate, avg orders/user, avg order value.- For observational campaigns: build matched cohorts (propensity score matching) on user, geo, recency, past spend.- Incremental lift = (conversion_treated - conversion_baseline) / conversion_baseline; compute both % lift and absolute incremental orders = (conv_treated - conv_baseline) * N_treated.- Cost per incremental order = (total promo cost + marketing cost + platform subsidy) / incremental orders.Merchant payout impact:- Calculate merchant_revenue_loss = discounts where merchant contributes + platform subsidy portions.- Show net merchant payout change = baseline payout per order vs payout under promo (accounting refunds, increases in order frequency, and any platform fee waivers).Recommended visualizations:- Top line KPIs panel: redemption rate, incremental orders, cost per incremental order, total promo spend, merchant payout delta.- Time series: redemption rate and incremental lift over time with confidence intervals.- Cohort table: treated vs control metrics (CR, AOV, orders/user, lift, p-value).- Waterfall: promo spend → incremental orders → cost per incremental → merchant payout change (breakdown platform vs merchant).- Promo explorer: filter by promo_id/channel/region; Sankey of exposure → apply → order → refund.- Heatmap: promo performance by geo and user cohort.- Experiment diagnostics: randomization balance table, sample sizes, statistical significance markers.- Alerts/thresholds: high cost per incremental order, negative merchant impact beyond threshold.Deliverables & operational notes:- Build ETL to join events into sessionized promo exposure table; compute daily aggregates and store in OLAP.- Expose metrics via BI (Looker/Mode) and an interactive dashboard with exportable CSV.- Provide doc with attribution rules, windows, and assumptions; version metrics for auditability.
EasyTechnical
76 practiced
For different DoorDash stakeholder groups (executives, product, operations, finance), propose SLA categories for metrics freshness (for example: real-time <1min, near-real-time <15min, daily batch). Explain rationale for each category, example metrics in each bucket, and monitoring you'd set up to ensure SLAs are met.
Sample Answer
High-level SLA categories (applied per stakeholder): Real-time (<1 min), Near-real-time (<15 min), Hourly (<1 hr), Daily batch (24 hrs). Rationale: tie freshness to decision speed and operational risk.Executives- Rationale: strategic decisions need stable high-level trends, not second-by-second.- Buckets & examples: - Daily batch: GMV, monthly active users, LTV/CAC - Hourly: daily revenue burn-down during promotions- Monitoring: daily ETL job success alerts, SLA dashboards (job lag, last successful run), anomaly detection on aggregates, weekly data quality reports.Product- Rationale: product iterations require rapid feedback but can tolerate small delays.- Buckets & examples: - Near-real-time: feature engagement, conversion funnel steps, A/B test exposure counts - Hourly: cohort metrics, retention by feature- Monitoring: streaming pipeline health (consumer lag), dashboards with latency gauges, alert on metric staleness or drift, automated QA tests on event schema.Operations- Rationale: operational safety requires near-instant visibility to act on incidents.- Buckets & examples: - Real-time: marketplace supply-demand imbalance, driver/restaurant ETA, order failure rates - Near-real-time: cancellations, acceptance rates- Monitoring: low-latency metrics stream (Prometheus/Kafka), paging alerts for thresholds, SLOs on max allowed lag, end-to-end synthetic transactions, per-partition consumer lag alarms.Finance- Rationale: financials require accuracy and reconciliation—some metrics can be batched, others need quicker visibility for cash flow.- Buckets & examples: - Daily batch: revenue recognition, payouts, fees reconciliation - Hourly: refund spikes, fraud detection signals- Monitoring: job-level reconciliation checks, delayed-run alerts, discrepancy dashboards, data lineage for audits.Cross-cutting monitoring & governance- Define SLAs per metric in a catalog with owner, allowed latency, and severity.- Implement metric freshness pipeline: capture event timestamps, compute ingestion vs event time lag, expose freshness metrics.- Automated alerts: metric-staleness, schema changes, consumer lag, and data drift; runbooks for responders.- Regular reviews: monthly SLA dashboard reviews with stakeholders and continuous improvement backlog.
HardTechnical
91 practiced
You're planning a rollout of a new definition for 'active user' that will change values across ~50 dashboards. As PM, propose a canary and rollback plan: staging approach, how to run parallel definitions, stakeholders to notify, testing steps, migration timeline, rollback criteria, and methods to measure downstream impact before full rollout.
Sample Answer
Situation: We need to change the definition of "active user" which will change values across ~50 dashboards used by ops, finance, marketing and leadership. This is high-risk because decisions (budget, incentives, SLAs) depend on those numbers.Task: As PM I must design a canary & rollback plan that validates the new definition end-to-end, minimizes disruption, and provides clear rollback criteria.Action:- Staging approach: Create a private staging environment that mirrors production metrics pipelines and a slice of production data (1% of users by user_id hash). Deploy the new definition only to staging first and run full pipeline end-to-end.- Parallel definitions: Implement dual-pipeline where both "legacy_active" and "new_active" flags are computed in production for the canary cohort and stored side-by-side. Dashboards should be parameterized to show either metric.- Canary rollout: Start with 1% user slice for 2 weeks, then expand to 5% for 1 week, 20% for 1 week, and 100% pending approvals.- Stakeholders to notify: Analytics/data engineering (owners of ETL & dashboards), BI/dashboard consumers (ops, finance, marketing), legal/compliance if SLAs or incentives affected, customer support, and executive sponsor. Schedule weekly syncs during rollout and a daily channel for incidents.- Testing steps: - Unit tests and data-contract validations in ETL jobs. - Backfill parity checks: run historical re-computation for sample windows and compare distributions. - A/B data comparisons: compute delta, pct-change per dashboard, and statistical significance. - Downstream consumers smoke tests: run queries for top 10 dependent alerts/reports. - QA sign-off checkpoints at each canary stage.- Migration timeline: 6-week plan: week 0 staging & backfill; wk1 canary 1%; wk2 expand 5%; wk3 expand 20%; wk4 run 20% with stakeholder review; wk5 final expand 100% and flip dashboards; wk6 monitor & close.- Rollback criteria: - Absolute or relative thresholds: e.g., any core dashboard metric deviates >10% from expected range without documented cause, or monitoring alerts triggered in downstream systems (billing, incentives), or significant user-impact incidents. - Data integrity failures: pipeline errors, missing partitions, or >0.5% reconciliation mismatches between sources. - Business impact: finance/ops raise concerns impacting revenue/SLAs. If any criterion met, immediately revert dashboard to legacy metric (parameter toggle) and pause further expansion; trigger incident review.- Methods to measure downstream impact before full rollout: - Compare KPIs for canary cohort using both definitions: retention, conversion, revenue per user. - Reconciliation reports per downstream job showing delta and confidence intervals. - Run "shadow runs" of billing/incentive calculations using new definition and compare outputs. - Monitor user-facing alerts, support tickets, and executive dashboards for anomalies. - Use feature flags and dashboard toggles to let consumers preview both metrics for a week and collect feedback.Result/Outcome: This approach provides incremental exposure, dual visibility for stakeholders, automated validation, and clear rollback triggers—minimizing business risk while allowing a safe, data-driven migration.
Unlock Full Question Bank
Get access to hundreds of DoorDash Key Metrics & Dashboard Requirements interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.