Causal Inference Questions

Direct answer. Test whether the 20% to 23% improvement is statistically real with a difference-in-proportions test accounting for the cohort size (50,000 users), attribute the lift to the initiative specifically by ruling out concurrent confounders (seasonality, other launches in the same window, cohort composition shifts), and build the executive narrative around the point estimate, its confidence interval, and an honest statement of what alternative explanations were checked and ruled out.

Structured elaboration.

1. Statistical significance: with $n=50{,}000$, a 3-percentage-point shift from a 20% base rate has a standard error of roughly $\sqrt{\frac{0.20 \times 0.80}{50{,}000}} \approx 0.0018$, or about 0.18 percentage points; a 3-point shift is roughly 16-17 standard errors, overwhelmingly statistically significant if the comparison is well-formed. The real question is whether the COMPARISON is fair (did this cohort experience only the initiative, or other changes too), not whether 3 points is "big enough."
2. Attribution: check whether a comparable cohort from BEFORE the initiative launched showed similar seasonal movement (a rough difference-in-differences (DiD)-style check even without a formal control group), and whether any other change (a pricing update, a UI redesign) launched in the same window that could independently explain part of the lift.
3. Confidence interval and narrative: report the 3-point lift with its confidence interval (roughly plus or minus 0.35 to 0.5 points at typical 95% coverage, given the standard error above), rather than a bare point estimate, and state explicitly which alternative explanations were checked.

Worked example: a one-page executive narrative. "90-day retention for the initiative cohort improved from 20% to 23% (a 3.0 percentage point lift, 95% CI approximately 2.3 to 3.7 points). We compared this cohort's pre-period seasonal pattern to the prior year's equivalent cohort and found no comparable natural lift, and confirmed no other major launch occurred in this window. We recommend treating this as strong, though not experimentally confirmed, evidence, and suggest a smaller randomized confirmatory test before scaling further investment." A suggested visual: a simple before/after bar chart for the initiative cohort alongside the same before/after comparison for last year's equivalent cohort, side by side, making the "is this just seasonal" comparison visually obvious.

Trade-offs and pitfalls. Statistical significance at this sample size is nearly guaranteed for any real-looking effect size, which means the executive narrative's honesty hinges entirely on the attribution step (ruling out confounders), not on the p-value; leading with "statistically significant" without addressing attribution risks overselling a result that a skeptical stakeholder could easily poke holes in.

Direct answer. The synthetic control method constructs a weighted combination of untreated ("donor") units that closely matches the treated unit's outcome trajectory during the pre-treatment period, then uses that synthetic combination's post-treatment trajectory as the counterfactual for what would have happened to the treated unit without the intervention. It's preferable to difference-in-differences specifically when you have ONE treated unit and no single comparable control unit, but several plausible donor units, none individually a great match, that together can be combined into a better one.

Structured elaboration. The weights on donor units are chosen (typically by minimizing pre-treatment prediction error) so that the weighted combination's pre-treatment trajectory and pre-treatment covariates closely track the treated unit's actual pre-treatment values; unlike DiD, which relies on the parallel-trends assumption holding for a single chosen comparison group, synthetic control's credibility rests on how well the FITTED synthetic series actually matches the real pre-treatment series, which is directly checkable rather than assumed.

Worked example: constructing a synthetic control for a city-level policy change.

1. Select a pool of donor cities that plausibly weren't affected by the policy.
2. Choose pre-treatment predictors: several years of the outcome itself (say, monthly revenue) plus relevant covariates (population, baseline growth rate, category mix).
3. Solve for donor weights (non-negative, summing to 1) that minimize the gap between the treated city's pre-treatment values and the weighted donor average across the chosen predictors.
4. Apply those same weights to the donors' POST-treatment outcomes to construct the synthetic counterfactual.
5. The estimated effect at each post-treatment period is the treated city's actual outcome minus the synthetic control's outcome at that period.
6. Run placebo tests: apply the identical procedure to each DONOR city (pretending each one, in turn, was "treated") and check that the treated city's estimated gap is unusually large compared to the placebo gaps for donors that weren't actually treated, which guards against the result being an artifact of pre-treatment overfitting.

Trade-offs and pitfalls. A synthetic control that fits the pre-treatment period very closely is more credible, but a poor pre-treatment fit (the synthetic series diverges noticeably from the real one even before treatment) should be reported as a limitation, not hidden, since it undermines the core logic the method depends on; results are also sensitive to which donor pool and which pre-treatment predictors were chosen, so pre-registering that choice, or at minimum reporting sensitivity to reasonable alternative donor pools, strengthens the credibility of the estimate.

Direct answer. A short stakeholder narrative for explaining that a cohort-level lift isn't necessarily causal: (1) the users who did the campaign-related action self-selected, or were targeted, in a way that likely differs from the broader population on factors that independently predict the outcome; (2) the timing overlaps with other things that could explain part or all of the lift (seasonality, other launches); (3) a raw before/after or treated/untreated comparison doesn't rule either of these out; (4) here's the concrete evidence we do or don't have that speaks to whether it's real; (5) here's what additional evidence, ideally a proper experiment, would let us say this confidently.

Structured elaboration. The goal of this conversation isn't to argue the campaign definitely didn't work, it's to calibrate confidence honestly and propose a path to a defensible answer, since flatly saying "we can't trust this" without a constructive next step tends to be dismissed by stakeholders who have already decided the number looks good.

Worked example: the actual 3-5 bullet narrative.

• "The users who engaged with this campaign chose to (or were targeted to) opt in, and people who opt into campaigns often already look different from the average user in ways that predict better outcomes on their own, independent of the campaign itself."
• "This campaign ran during [a period with a known seasonal or other confound], so some of the lift we're seeing likely would have happened anyway."
• "A simple before/after comparison for this cohort doesn't let us separate the campaign's real effect from these other factors."
• "To get a confident answer, we'd want to run a proper holdout: randomly withhold the campaign from a comparable slice of users and compare, which would take about [X weeks] and directly answer whether this is real."
• "In the meantime, here's our best honest estimate of the RANGE the effect could plausibly be in, based on what we can currently rule in or out."

Trade-offs and pitfalls. Under pressure to defend a launch or budget decision, there's a real temptation to soften this message into something that sounds more confident than the evidence supports; the discipline of proposing a concrete, time-bounded next step (a holdout test) alongside the honest caveat is what keeps the conversation constructive rather than reading as a pure objection.

Direct answer. Attributing impact across multiple overlapping promotion types to both short-term conversions and long-term retention needs modeling strategies that complement each other's blind spots: incremental holdout experiments give a clean, causally valid TOTAL effect but can't easily decompose it across overlapping promotion types on their own; multi-touch attribution decomposes credit across touchpoints but relies on modeling assumptions rather than randomization; uplift modeling identifies which users respond most to which promotion type, useful for targeting even when the aggregate attribution question is hard to fully resolve.

Structured elaboration.

1. Incremental holdout experiments: randomly withhold a subset of users from ALL promotions (or from specific promotion types, in a factorial design) to measure the true total incremental lift; the factorial version, randomizing exposure to each promotion type somewhat independently, is what lets you decompose the total effect across promotion types with causal validity, at the cost of experimental complexity and some lost revenue from the holdout group during the test.
2. Multi-touch attribution: assign fractional credit to each promotion a converting user was exposed to, using a rule-based (first-touch, last-touch, linear) or data-driven (Shapley-value-style) allocation; this scales to ongoing, non-experimental measurement but doesn't establish that any individual touchpoint was causally necessary, since observed exposure to multiple promotions is itself often correlated with being a high-value user regardless of the promotions.
3. Uplift modeling: even without solving the full attribution problem, an uplift model trained on the holdout experiment's randomized data can identify which user segments respond most to which promotion type, directly useful for future targeting decisions regardless of how the aggregate attribution question resolves.

Worked example. Run a factorial holdout: randomly assign users to full-exposure (all promotions), partial-exposure (only some), and no-exposure groups; the total incremental lift from full versus no-exposure gives a causally clean top-line number, and comparing the partial-exposure cells lets you estimate each promotion type's marginal contribution and any interaction effects (does a coupon plus a merchant discount together produce more than the sum of each alone, suggesting positive interaction, or less, suggesting redundancy).

Trade-offs and pitfalls. The limitations compound across long lookback windows: multi-touch attribution's credit-allocation rule becomes increasingly arbitrary the longer the window (was a touch from three months ago really causally relevant to today's conversion), and holdout experiments become more expensive and slower to read out the longer you need to wait to observe long-term retention; a common practical compromise is a shorter, cleanly-measured holdout for short-term conversion lift, paired with a cohort-tracked (not necessarily experimentally-isolated) view of longer-term retention that's reported with explicitly weaker causal confidence.

Direct answer. Whether the 8% US engagement lift should transport to other markets depends on whether the mechanism driving the effect is likely to be similar elsewhere, not just on the fact that the experiment was well-run; the practical recommendation is a smaller-scale, faster confirmatory test in one or two representative non-US markets before a full global rollout, rather than either blindly extrapolating or re-running the full US-scale experiment everywhere.

Structured elaboration. External validity concerns to check before trusting the transport: does the recommendation algorithm rely on signals (content catalog density, user behavior patterns) that differ meaningfully across markets? Are user demographics, device mix, or competitive landscape different enough that the mechanism behind the 8% lift might not hold? Is there a reason the US result itself might be inflated or unusual (a novelty effect specific to the timing of the US test, a US-specific promotional overlap)? None of these can be answered from the US experiment alone; they require either domain knowledge about the mechanism or additional data from the target markets.

Worked example. If the recommendation algorithm's lift came largely from surfacing under-explored long-tail content, and a target market has a much smaller content catalog, the mechanism plausibly weakens there, and blind rollout risks a much smaller (or even negative, if the algorithm's other trade-offs dominate without a large content pool to draw on) effect. A quick, cheap way to get real signal without a full-scale re-test: run a smaller, shorter experiment in one or two representative non-US markets, sized to detect a materially smaller effect than 8% (say, powered to detect 3%), which is enough to catch a "doesn't transport" result without the cost of a full re-test everywhere.

Trade-offs and pitfalls. The two extreme responses, "the randomized controlled trial (RCT) proved it works, roll out everywhere" and "every market is different, re-run the full experiment everywhere," are both wrong for different reasons: the first ignores that external validity is a real, checkable risk, not a formality, and the second is often needlessly slow and expensive when a smaller confirmatory test in a few representative markets would resolve most of the uncertainty at a fraction of the cost. Multi-country experiment designs that stratify by market characteristics upfront (rather than testing one market at a time sequentially) are the more scalable version of this same logic when you're rolling out across many markets at once.