DoorDash Data Analyst Interview Preparation Guide - Staff Level

Direct answer

LEAD(start_date) OVER (PARTITION BY user_id ORDER BY start_date) pulls each subscription's next start date into the current row without a self-join, so the gap between one subscription ending and the next beginning is just next_start_date - end_date. A NULL from LEAD means there is no next subscription: label those rows deliberately (commonly "churned"), and pick a business threshold for the gap (say, resumed if the gap is at most 30 days, churned otherwise). The exact same LEAD-over-a-partition idea, applied to a purchases table ordered by purchase_date instead of a subscriptions table, computes time-to-next-purchase for a churn or customer lifetime value (LTV) feature; the one thing that changes is how you treat the rows where LEAD is NULL, because "no next purchase yet" is not the same fact as "this user will never purchase again."

Structured elaboration

• LEAD(col) is the mirror of LAG(col): it reaches forward instead of back, using the same PARTITION BY/ORDER BY contract.
• Gap-based labeling: next_start_date - end_date gives the number of days a customer went without an active subscription; a negative gap means the next subscription started before the current one ended (overlap), which needs its own explicit handling rather than silently falling into "resumed."
• The identical query shape applied to purchases(user_id, purchase_date) gives time_to_next_purchase = LEAD(purchase_date) - purchase_date per purchase, which is a standard input feature for churn models and one building block of an LTV estimate.
• Right-censoring: for a user's most recent purchase, LEAD returns NULL because there simply is no later row in the data you have so far, not because the user is confirmed to never buy again. Treating every NULL as "never" biases the feature toward pessimism for exactly your most recently active users, who are the ones you have had the least time to observe. The standard fix is to compute against an explicit snapshot/as-of date: report "days observed since this purchase with no repeat yet" instead of pretending the time-to-next-purchase is either 0 or infinite.

Worked example

WITH nxt AS (
  SELECT user_id, start_date, end_date,
    LEAD(start_date) OVER (PARTITION BY user_id ORDER BY start_date) AS next_start_date
  FROM subscriptions
)
SELECT user_id, start_date, end_date, next_start_date,
  CASE WHEN next_start_date IS NULL THEN NULL ELSE next_start_date - end_date END AS gap_days,
  CASE
    WHEN next_start_date IS NULL THEN 'churned'
    WHEN (next_start_date - end_date) <= 30 THEN 'resumed'
    ELSE 'churned'
  END AS status
FROM nxt ORDER BY user_id, start_date;

Executed against a 4-row subscriptions sample for two users: user 1 has three subscriptions (Jan 1 to Mar 1, Mar 20 to Jun 1, Sep 1 to Oct 1). The first gap is 19 days (Mar 1 to Mar 20), labeled resumed; the second gap is 92 days (Jun 1 to Sep 1), labeled churned; the third row has no next subscription, so next_start_date and gap_days are NULL and status is churned. User 2's single subscription also has no next row and is labeled churned.

The LTV variant, executed against purchases(user_id, purchase_date) with user 1 buying on Jan 1, Jan 15, and Mar 1, 2024, and a snapshot date of Apr 1, 2024 (2024 is a leap year, so February runs to the 29th; the 46-day gap below only reproduces with that extra day): days_to_next_purchase is 14 (Jan 1 to Jan 15) and 46 (Jan 15 to Mar 1); the Mar 1 purchase has next_purchase_date = NULL and, computed against the Apr 1 snapshot, days_observed_since_this_purchase = 31. That last row is the right-censored one: 31 days with no repeat purchase yet is a very different signal from "31 days and counting, still nothing," and both are different again from a user whose gap of 46 days between purchases is fully observed.

Trade-offs & pitfalls

• The resumed/churned gap threshold is a business decision, not a mathematical one; validate 30 days against the actual distribution of real reactivation gaps rather than hardcoding a round number.
• Conflating "no next row yet" with "will never return" quietly turns an LTV or churn feature into one that systematically under-predicts value for your newest active users, since they have had the least time to generate a "next" row.
• Overlapping subscriptions (next_start_date < end_date) produce a negative gap; decide explicitly whether that counts as resumed, gets clamped to 0, or is flagged as a distinct overlap state, rather than letting it silently pass the <= 30 check.
• LEAD avoids a full self-join scan for this pattern, but still needs an index on (user_id, start_date) (or purchase_date) to stay cheap on large tables, since the engine still has to sort or seek within each partition.

Direct answer

Missing completely at random (MCAR) means the chance a value is missing has nothing to do with any variable in the dataset, observed or not, like a random sensor glitch. Missing at random (MAR) means the chance of missingness depends on OTHER observed variables but not on the missing value itself, like older survey respondents being more likely to skip an income question. Missing not at random (MNAR) means the chance of missingness depends on the value that's actually missing, like high earners specifically declining to report their income.

Why the distinction matters

The mechanism determines whether ignoring the missingness (or a simple imputation) will bias your conclusions. Under MCAR, the observed data is a random subsample of the full data, so simple approaches introduce little bias. Under MAR, you can often correct for the pattern because the information you need to explain the missingness is sitting right there in other columns. Under MNAR, the missingness itself carries information you can't recover just by looking at the rest of the row, since the very thing that's missing is what's driving whether it's missing.

Worked example

Consider a customer survey: a "device model" field missing because of a random logging bug affecting a random 2% of sessions is MCAR. A "household income" field missing more often for older respondents, but at the same rate regardless of their actual income once you condition on age, is MAR: age is observed, and it explains the pattern. A "household income" field missing specifically because high earners are less willing to disclose it, even after accounting for age and every other observed variable, is MNAR: the value's own magnitude is driving whether you see it at all.

Trade-offs and pitfalls

In practice, you rarely get to observe the true mechanism directly since it's a statement about unobserved values, so you're always making an educated diagnosis from indirect evidence (patterns of missingness across other variables, domain knowledge about why people don't report something) rather than a proof. Treating every missingness pattern as MCAR by default because it's the simplest assumption is the single most common way EDA quietly understates how much a dataset's blind spots matter.

Outline — key sections of the growth analytics playbook (short description), one best practice per section, and adoption plan across product & marketing.

1. Goals & Strategy

• Contents: north-star, growth KPIs (activation, retention, LTV, CAC), business hypotheses & success criteria.
• Best practice: map every metric to a clear business question and a decision owner (who acts on the metric).

2. Measurement & Data Model

• Contents: event taxonomy, definitions, data sources, canonical tables, ownership, instrumentation checklist.
• Best practice: enforce a single source of truth with a tracked event catalog and semantic layer (column-level docs + versioning).

3. Experimentation Framework

• Contents: experiment lifecycle, hypothesis template, sample size calc, randomization, analysis plan, rollout rules.
• Best practice: require pre-registration of hypothesis, metrics, and analysis method before running experiments.

4. Analytics & Reporting

• Contents: standard dashboards, cohort/segmentation templates, SQL snippets, automated reports, alerting thresholds.
• Best practice: ship self-serve templates (SQL + BI) that analysts and PMs can adapt—reduce ad-hoc gatekeeping.

5. Tooling & Infrastructure

• Contents: approved stack (data warehouse, ETL, BI, A/B tools), access control, cost guidelines.
• Best practice: provide managed starter projects for each tool so teams can onboard in <2 days.

6. Governance & Quality

• Contents: data review cadence, SLA for fixes, incident playbook, tagging for data owners.
• Best practice: lightweight data contract—changes to schema require consumer sign-off.

7. Playbooks & Experiment Templates

• Contents: ready-to-use templates (hypothesis, dashboard, SQL), win/loss analysis, communication templates.
• Best practice: include examples of 3 past experiments: one success, one null, one failure—highlight learnings.

Adoption plan (how I’d ensure uptake)

• Co-create with stakeholders: run two half-day workshops (product + marketing) to tailor the playbook and name decision owners.
• Quick wins: ship starter templates and one pilot experiment with a PM and marketer to demonstrate cycle time reduction and better decisions.
• Embed in rituals: make playbook artifacts part of sprint planning, PRD checklist, and weekly growth reviews.
• Training + office hours: 60–90 min onboarding sessions + recurring office hours for first 3 months.
• Metrics for adoption: track playbook usage (template copies, dashboard views), percentage of experiments pre-registered, and time from idea to result.
• Governance loop: monthly review to iterate on the playbook from feedback and metrics.

This approach balances rigor with usability so product and marketing can move fast with consistent, trusted measurement.

High-level comparison

• MMM: aggregate, often weekly/monthly time-series regression (econometric). Good for long-term channel ROI, budget-level decisions, and capturing seasonality and macro effects.
• User-level MTA: event-level or user-journey attribution (fingerprinting/IDs). Good for short-term, tactical optimizations (bid/creative), channel sequencing, and incrementality at individual touch level.

Data requirements & granularity

• MMM: aggregated media spend/impressions, sales/revenue time-series, pricing, promos, competitors, macro variables, holiday flags. Granularity: weekly or monthly across geos/products.
• MTA: user identifiers, timestamped ad exposures, impressions/clicks, conversions, device/browser, attribution window. Granularity: impression/session/user-level.

Biases and challenges

• Selection bias: MTA suffers if exposed users differ systematically (targeting bias). MMM mitigates user-level selection but can suffer from omitted-variable bias (missing controls).
• Adstock/decay: MMM explicitly models adstock (lagged carryover, saturation curves). MTA can implicitly capture short conversion lags but often misses long carryover without modeling.
• Measurement bias: MTA impacted by cookie loss, cross-device, walled gardens; MMM less sensitive to ad measurement noise but aggregates away micropatterns.
• Multicollinearity: In MMM, correlated spends across channels require regularization or hierarchical models. In MTA, overlapping touches complicate credit assignment.

Handling seasonality & externalities

• MMM: include seasonal dummies, Fourier terms, fixed effects, holiday indicators, and control variables (price, promos, weather). Can decompose trend/seasonality.
• MTA: adjust for seasonality by conditioning on time/date features or training windows; better combined with uplift/incrementality tests to separate calendar effects.

Using results to optimize budgets

• MMM: provides channel-level ROI and diminishing returns curves → use to reallocate budget across channels, forecast outcomes under scenarios, run budget optimizer (constrained nonlinear program).
• MTA: informs tactical bids, creative, and audience targeting; use incremental lift estimates and lookback windows for bid shading.
• Combine: use MMM to set strategic channel caps and long-term budgets; use MTA to allocate within-channel bids and creatives in real time.

Hybrid approach (practical recipe)

1. Run MMM at geo-week-product level to estimate long-run ROI, adstock, seasonality, and saturation curves.
2. Collect MTA-derived incremental lift or propensity-adjusted conversion rates at campaign/audience level (use uplift tests or causal forests to reduce selection bias).
3. Reconcile:
   • Use MMM to set top-line channel budget and saturation targets.
   • Use MTA signals to distribute that budget to audiences/campaigns and to estimate short-term pacing.
4. Feedback loop: feed aggregated, cleaned MTA conversion counts back into MMM as an additional predictor or to validate adstock/saturation assumptions.
5. Governance: run periodic holdout/incrementality experiments (geo or time-based) to validate both models and recalibrate.

Practical tips for a Data Analyst

• Prioritize data quality: unify spend definitions, map impressions to conversions, align time windows.
• Surface uncertainty: report confidence intervals, sensitivity to adstock and lag specifications.
• Translate to action: present MMM-derived ROI curves plus MTA tactical insights in dashboards with recommended reallocation and expected uplift scenarios.

Situation: Stakeholders need analytics delivered 50% faster without reducing accuracy or clarity.

Roadmap (90–180 day phased plan)
Short-term wins (0–6 weeks)

• Measure baseline: end-to-end SLA per report, time spent per task, and error rate.
• Quick tooling tweaks: add DB indexes, optimize slow SQL (EXPLAIN), limit wide JOINs, cache heavy queries, reduce dashboard default date ranges.
• Process fixes: create a prioritized intake queue and standard request template; enforce “definition of ready” for ad-hoc requests.
• Training: 2 half-day workshops on SQL performance patterns and dashboard best-practices.
  Expected impact: 15–25% faster delivery.

Medium-term automation (6–16 weeks)

• Introduce dbt (or equivalent) for modular, tested data transformations; standardize metric definitions (metrics layer).
• Automate ETL orchestration with Airflow / Prefect; schedule materialized views for heavy aggregates.
• Implement analytics CI: automated testing for dbt models and smoke tests for dashboards.
• Reuse components: SQL macros, parameterized report templates, and a self-serve request portal.
  Expected impact: additional 20–30% reduction in turnaround and fewer rework cycles.

Long-term architectural changes (4–12 months)

• Migrate slow, ad-hoc datasets to a columnar analytics store (e.g., Snowflake/BigQuery) and adopt partitioning/clustering strategy.
• Build a centralized metrics registry and served feature store for cross-team reuse.
• Replace manual exports with embedded analytics / self-serve dashboards and data products.
• Create capacity plan and SLOs for analytics platform.
  Expected impact: sustainable 50%+ improvement, scalable for growth.

Risks & mitigations

• Risk: Data quality regressions when changing pipelines — Mitigate: rollout dbt tests, staged deployments, and shadow runs before cutover.
• Risk: Tooling costs and complexity — Mitigate: run pilots, measure ROI, prioritize high-impact areas first.
• Risk: Team resistance to new processes — Mitigate: involve analysts in design, provide hands-on training, appoint change champions.
• Risk: Over-automation leading to black-box metrics — Mitigate: maintain documentation, versioned metric definitions, and easy lineage tracing.

Metrics to track progress

• Mean time to delivery (MTTD) per report
• % requests fulfilled within SLA
• Rework rate / error rate
• Analyst time spent on maintenance vs. insight generation

This plan balances immediate wins, automated repeatability, and a robust platform to reach and sustain a 50% improvement while preserving data quality.

Direct answer

Use a z-test only when the population standard deviation is genuinely known in advance, which is rare in practice. Use a t-test whenever the standard deviation has to be estimated from the sample itself, which is the normal situation, and this holds regardless of sample size. Sample size affects a different thing: how close the t and z critical values are to each other and how much you can lean on the Central Limit Theorem if the underlying data isn't very normal.

Structured elaboration

Known vs. unknown variance. This is the formal criterion. If $\sigma$ is known (rare outside quality-control settings with a long-established process variance), use z. If $\sigma$ is estimated from the sample as $s$ (the normal case), use t with $df = n-1$; the t-distribution's heavier tails are exactly the correction for the added uncertainty of estimating $\sigma$ rather than knowing it.

Sample size's actual role. As $n$ grows, $t_{n-1}$ converges to $z$, so at large $n$ the choice barely changes the numeric answer, which is why "just use z for n≥30" survives as a practical shortcut even though it's not the formal reason. Separately, larger $n$ also makes the Central Limit Theorem a stronger justification for treating the sampling distribution of the mean as approximately normal even when the raw data isn't, which matters for the validity of either test, not for the t-vs-z choice itself.

Comparing two means: pooled vs. Welch's t. If assuming the two groups have equal population variances, use the standard (pooled) two-sample t-test. If variances might differ, and there's rarely a strong reason to assume they're equal, use Welch's t-test, which does not assume equal variances and adjusts the degrees of freedom accordingly. Welch's costs very little power when variances actually are equal but protects against inflated Type I error when they aren't, which is why it's the safer default.

Robustness. t-tests are reasonably robust to mild-to-moderate non-normality once $n$ is moderate (roughly 30+ per group), thanks to the CLT. They're not robust to strong skew or heavy outliers at small $n$, where a few extreme points can dominate both the mean and the variance estimate.

Worked example: how close t and z actually are, by sample size

(All values from scipy.stats.t.ppf(0.975, df), verified directly.) At $df=5$ the t critical value is about 31% larger than z, meaningfully widening the interval or raising the bar for significance; by $df=60$ the gap has shrunk to about 2%. This is the practical justification behind "large n, t and z are basically the same," even though the theoretically correct reason to pick t is always "σ is estimated," not "n is small."

When variances are unknown and sample sizes are small: the actual procedure

1. Look at the data: a histogram or Q-Q plot per group, and check for obvious outliers.
2. If approximate normality looks plausible, default to Welch's t-test (not pooled, unless there's a specific reason to believe variances are equal, such as both groups measuring the identical underlying process).
3. If normality looks clearly violated, or the sample is extremely small (single digits per group) with visible skew, switch to a nonparametric alternative like the Mann-Whitney U test, or use a bootstrap for the confidence interval and p-value instead of the t-distribution's analytic formula.

Trade-offs & pitfalls

• Defaulting to the pooled t-test "because it's the classic one" without checking the equal-variance assumption is a common shortcut that inflates false positives when variances genuinely differ; Welch's is essentially free insurance against this.
• Small samples with heavy skew or outliers can pass a superficial normality check while still producing an unreliable t-test; this is where nonparametric or bootstrap alternatives earn their keep, not just as a formality but as a real fix.
• The "n≥30 use z" heuristic is useful as a rule of thumb but wrong as a justification; it should never be given as the reason to choose z over t in an interview answer, since the real criterion is whether σ is known.

Approach summary: build an interactive prototype where stakeholders can (a) pick among candidate KPI definitions, (b) see them side‑by‑side across same filters/timeframes, (c) save/restore comparison states, and (d) record preferences/feedback. I’ll show how to implement this in both Tableau and Power BI and note performance/UX considerations.

Tableau (features: parameters, calculated fields, story points, extracts, dashboard actions)

• Definitions as logic: implement each candidate KPI as a calculated field (KPI_A, KPI_B, KPI_C) or store definition logic in a small reference table.
• Toggling: create a Parameter “Slot 1” and “Slot 2” (string or integer mapping to definition names). Create a calculated field that uses CASE/IF to return the selected KPI value:
  • KPI_Slot =
    CASE [Slot Parameter]
    WHEN "A" THEN [KPI_A]
    WHEN "B" THEN [KPI_B]
    END
• Side‑by‑side: place KPI_Slot for Slot1 and Slot2 in a single view (columns), synchronized filters and date context.
• Persisting comparisons: use Story Points to capture selected views + filters, or use Dashboard “Export” (PDF/image) for snapshots. For lightweight persistence, build a parameter actions dashboard where clicking a saved bookmark sets parameters; you can also create dashboard buttons that set parameters via parameter actions.
• Performance: extract data source (Hyper extract) to speed aggregates while prototyping; refresh on cadence.
• Feedback capture: add a web page object that opens a Google Form/Microsoft Form prefilled with comparison identifiers (pass parameters via URL query), or use Tableau Server comments to capture in‑context feedback.

Power BI (features: disconnected tables, measures, bookmarks, personal bookmarks, aggregations/extracts)

• Definitions as measures: create measures for each candidate KPI (KPI_A = ... , KPI_B = ...).
• Toggling: create a disconnected table DefinitionSelector with rows A/B/C. Use slicers allowing multi‑select for comparison slots, or create two slicers bound to duplicate disconnected tables to choose Slot1/Slot2. Implement a generic measure:
  SelectedKPI_Slot1 =
  SWITCH( SELECTEDVALUE(DefinitionSelector1[Name]),
  "A", [KPI_A],
  "B", [KPI_B],
  BLANK() )
• Side‑by‑side: place two cards/columns showing Slot1 vs Slot2 measures; keep page filters synced.
• Persisting comparisons: use Bookmarks to capture the full state (slicer selections, filters) and expose buttons for saved comparisons; in Power BI Service, users can create Personal Bookmarks to persist their view. Use Report-level bookmarks and assign to buttons for stakeholder-shared snapshots.
• Performance: enable aggregations or use import mode (Power BI dataset/Composite) to speed queries; consider incremental refresh for large sources.
• Feedback capture: embed a “Vote” button that opens a Form/Power Automate flow with current slicer values (Power BI supports passing filters in URL). Alternatively use the Comments pane or a Power Automate button that writes preference to a SharePoint/SQL table when clicked.

Key UX notes and trade-offs:

• Use two slot selectors rather than trying to display many candidates; it keeps comparisons readable.
• For multiple candidates at once, a small multiples chart with measures as separate series works but requires careful legend/scale alignment.
• Persisting server-side: Tableau Server and Power BI Service store bookmarks/stories but user-specific persistence differs (Power BI Personal Bookmarks). If enterprise persistence needed, log selections to a table via form/flow.
• Always surface exact definition logic (tooltip or modal) so stakeholders know what they’re comparing.
• Start with extracts/aggregations for interactive speed; later wire to live sources for production.

Example implementation pattern (Power BI DAX snippet):

SelectedKPI = SWITCH( SELECTEDVALUE(DefinitionSelector[Name]),
  "ConversionRate", [ConversionRate_Measure],
  "RevenuePerUser", [RPU_Measure],
  BLANK() )

This design lets stakeholders toggle definitions quickly, compare them directly under identical filters, save/restore meaningful snapshots, and record preferences in a way that’s auditable and actionable.

Direct answer

CUPED (Controlled-experiment Using Pre-Experiment Data) is a variance-reduction technique, published by Microsoft researchers (Deng, Xu, Kohavi and Walker, 2013), that uses a metric measured before the experiment started to strip out some of the between-user noise in the outcome metric, without touching the unbiasedness of the treatment-effect estimate. It requires one thing: a pre-experiment value of a covariate, ideally the same metric you are about to test, measured for the same units, from a period before assignment happened (so it cannot itself have been affected by treatment). The payoff is a tighter estimate of the same treatment effect from the same traffic, which functionally means you can detect the same effect size with less traffic or in less time.

Structured elaboration

What it requires and assumes

• A pre-experiment covariate $X$ for every unit, measured strictly before random assignment (so treatment cannot have influenced it) and ideally correlated with the outcome $Y$ you're testing.
• The scaling coefficient must be estimated without using the treatment label, typically pooled across both arms or from control alone. This is what keeps the adjustment from injecting bias: you are not letting "knowledge of which arm someone is in" leak into how you weight their pre-period value.

How the adjustment is computed, in plain terms

For each unit, adjust its outcome by subtracting a scaled version of how far its pre-experiment covariate was from the population average:

$Y_i^{\text{CUPED}} = Y_i - \theta \left(X_i - \bar{X}\right)$

where the scaling coefficient is the population covariance of $X$ and $Y$ divided by the variance of $X$:

$\theta = \frac{\operatorname{Cov}(X, Y)}{\operatorname{Var}(X)}$

Intuitively: if a user's pre-period activity was above average, some of their post-period activity being above average too is "expected" and not informative about treatment, so it gets subtracted out. The treatment effect is then estimated as the difference in adjusted outcome means between arms, $\bar{Y}^{\text{CUPED}}_{\text{treat}} - \bar{Y}^{\text{CUPED}}_{\text{control}}$, which is algebraically the same as the raw mean difference minus $\theta$ times the pre-period mean difference between arms; since randomization makes that pre-period difference expected to be zero, the adjustment does not shift the expected value of the estimate, it only removes the part of each unit's variance that $X$ already explained.

The resulting variance reduction has a clean form: if $\rho$ is the correlation between the pre-experiment covariate and the outcome, the adjusted estimator's variance is $(1-\rho^2)$ times the unadjusted variance. A stronger pre-period predictor buys a bigger reduction; a weak one buys almost nothing.

What makes a covariate good or poor

What goes wrong with a poor choice

• Using a post-treatment or treatment-adjacent value as $X$ breaks the "cannot be affected by treatment" requirement and reintroduces bias: you would be conditioning on something treatment itself moved, which is a different and more serious problem than losing precision.
• Using a weakly correlated covariate costs almost nothing in bias but also buys almost nothing in variance reduction; the $(1-\rho^2)$ factor is close to 1.
• Using a covariate that is missing for a large share of units (commonly new users with no pre-period history) forces a choice: drop those units (losing sample and possibly biasing toward existing users) or impute a value (introducing a modeling assumption you did not need before).

Worked example

Suppose, for illustration, we assume a pre-experiment 7-day conversion rate correlates with the in-experiment conversion rate at $\rho = 0.6$ for a given product (a hypothetical, stated input, not an empirical claim about any real product). The variance reduction from CUPED is then derived directly:

$\frac{\operatorname{Var}(Y^{\text{CUPED}})}{\operatorname{Var}(Y)} = 1 - \rho^2 = 1 - 0.6^2 = 1 - 0.36 = 0.64$

So the adjusted estimator's variance is 64% of the unadjusted one, a 36% reduction, given only the stated assumption $\rho = 0.6$. If $\rho$ had instead been a weak 0.2, the reduction would be $1 - 0.04 = 0.96$, i.e., only a 4% cut, illustrating why the strength of the pre-period correlation, not just its existence, determines whether CUPED is worth implementing for a given metric.

Trade-offs and pitfalls

• CUPED does not help metrics with no meaningful pre-experiment history, most obviously new-user-only experiments; for those, look at stratification or other covariate-adjustment techniques instead.
• Estimating $\theta$ using data that has already been split by arm (rather than pooled or control-only) risks contaminating the coefficient with treatment information; keep the estimation step blind to arm assignment.
• CUPED reduces variance, it does not change what the true effect is; do not present the tighter confidence interval as evidence the effect itself grew, only that you can now see it more precisely.
• CUPED composes with stratified randomization (apply the adjustment within strata), which is a reason to know both rather than treat them as competing choices.

I would first separate disagreement on the outcome from disagreement on the method. Then I would bring the executives into a short decision session with a one-page brief: the business goal, the options, the trade-offs, and the decision needed. I would make decision rights explicit using RACI, which means Responsible, Accountable, Consulted, and Informed. That way, everyone knows who recommends, who decides, and who simply needs to stay informed.

For example, if one sponsor wants speed, another wants cost savings, and a third wants risk reduction, I would ask which metric is the tie-breaker if they conflict. I would propose a shared scorecard with 2 or 3 measures, such as revenue impact, operational risk, and delivery date, then ask the accountable executive to make the final call in writing.

While the debate is happening, I would keep teams executing on work that is not dependent on the unresolved choice, pause only the parts that could be wasted, and communicate a clear interim plan. The goal is to prevent thrash, protect momentum, and get everyone back to one set of success criteria.

For example, on a customer-onboarding automation initiative, three executives disagreed about halfway through: the VP of Engineering wanted to prioritize system reliability given a recent outage, the VP of Finance wanted to prioritize cost savings from reduced manual onboarding labor, and the VP of Risk wanted to prioritize compliance controls given a pending audit. In the decision session, the RACI mapping made the VP of Product the accountable decision-maker, with all three VPs as consulted. The proposed scorecard used three metrics: onboarding error rate (tied to reliability), manual labor hours saved per month (tied to cost), and number of unresolved audit findings (tied to risk). The accountable VP decided that the audit-findings metric was the tie-breaker for this quarter, since the audit deadline was fixed and immovable, while the reliability and cost metrics would be weighted equally starting the following quarter. While that decision was being finalized in writing, the teams kept building the shared onboarding data pipeline, which every option needed regardless of the outcome, and paused only the specific reporting dashboard whose design depended on which metric ultimately won.

Direct answer

Do not treat this as picking a winner among the three stakeholders' asks; treat it as scoring every candidate metric, including ones nobody explicitly proposed, against a shared rubric (alignment to company goals, actionability, whether it is a leading or lagging signal, and how much it overlaps with something else already on the page), then keep the three with the best combined profile. In practice that usually means keeping one true north-star or engagement metric, one monetization metric like monthly recurring revenue (MRR), and one health metric like retention, while moving a purely acquisition-flavored ask like installs to a channel-specific dashboard instead of the shared homepage.

Structured elaboration

Selection criteria

• Goal alignment: does the metric map to a current company objective (growth, monetization, retention)?
• Actionability: can a team change their behavior based on this number, or is it purely descriptive?
• Leading vs. lagging: does it warn you early, or only confirm what already happened?
• Uniqueness: does it duplicate information another metric on the same dashboard already carries?

Reconciling the conflict

Run a short working session with a representative from each stakeholder group, score every candidate against the four criteria on a simple 1-5 scale, and let the scores, not seniority or volume of asking, decide the final three. When two metrics score close to tied, prefer the one with better actionability, since a dashboard's job is to trigger a decision, not just to report a number.

Worked example

Score four candidates (the three originally requested, plus a genuine north-star candidate nobody named) on a 1-5 scale across the four criteria:

The three highest scores are core-action weekly active users (18), 30-day retention (16), and MRR (15). Installs (12) is the lowest scorer, mainly on actionability (a rising install count by itself does not tell product or customer-success teams what to do), so it moves to marketing's own channel dashboard instead of the shared homepage, while the shared homepage carries the north-star, monetization, and health signal.

Trade-offs & pitfalls

Scoring feels objective but the scores themselves are judgment calls; running the exercise with only one stakeholder group in the room reproduces the original conflict under a veneer of rigor. A common wrong turn is picking three metrics that all lag together (installs, MRR, and a trailing retention number), which looks balanced on paper but gives nobody an early warning signal; deliberately keep at least one leading indicator in the final set. Dropping a stakeholder's requested metric entirely, rather than relocating it to a dashboard where it is actually actionable, tends to resurface the same disagreement at the next planning cycle, so pair the decision with where the dropped metric still lives.