Causal Inference Questions

Direct answer. A sensitivity analysis quantifies how strong an unmeasured confounder would need to be, in terms of its relationship to both treatment and outcome, before it could fully explain away your observed effect. Two standard tools are Rosenbaum bounds, which ask how much an unmeasured factor could distort the odds of treatment assignment before your conclusion (the p-value or effect direction) becomes unreliable, and the E-value, a more recent and more directly interpretable measure of the minimum strength of association, on the risk-ratio scale, that an unmeasured confounder would need with BOTH treatment and outcome to fully explain away the estimate.

Structured elaboration. The E-value for an observed risk ratio $RR$ (for $RR \geq 1$) is:

$$E = RR + \sqrt{RR \times (RR - 1)}$$

This gives the minimum risk ratio an unmeasured confounder would need to have with BOTH the treatment and the outcome, on this same multiplicative scale, to fully explain away the observed association. A larger E-value means a stronger, harder-to-dismiss result; an E-value close to 1 means even a weak unmeasured confounder could explain the whole effect away.

Worked example. Suppose an observational estimate finds a risk ratio of $RR = 1.5$ for a retention-boosting feature. Computing the E-value:

$$E = 1.5 + \sqrt{1.5 \times 0.5} = 1.5 + \sqrt{0.75} \approx 1.5 + 0.866 = 2.366$$

This means an unmeasured confounder would need to be associated with BOTH feature adoption and retention by a risk ratio of at least about 2.37 (each) to fully explain away the observed 1.5 effect. A plain-language interpretation for a non-technical stakeholder: "for this result to be spurious, there would need to be some unmeasured factor that makes users roughly two and a half times more likely to both adopt the feature and retain, purely by coincidence with the true effect being zero. Given the confounders we've already controlled for, that's a high bar, but not an impossible one, which is why we're still recommending a confirmatory experiment before scaling investment."

Trade-offs and pitfalls. A sensitivity analysis doesn't prove the result is unconfounded, it quantifies how much confounding would be needed to overturn it, which is a meaningfully different and more honest claim; the biggest pitfall is presenting a comfortably large E-value as proof of causality, when a genuinely strong unmeasured confounder (a marketing campaign that happened to target both feature-adopters and high-retention users) could still plausibly exceed even a moderately large threshold.

Direct answer. Propensity score matching (PSM) discards data (unmatched units) to build comparable pairs, while inverse probability weighting (IPW) keeps the full sample and reweights it so that, in expectation, the weighted treated and control groups look balanced. PSM tends to be preferable when overlap is poor and you'd rather trust a smaller, well-matched sample than a reweighted full sample with extreme weights; IPW tends to be preferable when overlap is reasonably good and you want to use all the data, since discarding units in PSM is itself a loss of information and can change the estimand.

Structured elaboration.

Worked example. For a product change rolled out selectively (say, to users who opted into a beta), if propensity scores are well-distributed with good overlap across the population, IPW uses the full sample efficiently. If instead a subset of users has propensity scores near 0 or 1 (near-certain to be treated or not), those units get enormous or near-zero weights under IPW, which can make the estimate unstable; PSM's caliper would simply exclude those units rather than let them dominate through extreme weighting, trading sample size for stability.

Trade-offs and pitfalls. Doubly robust methods exist precisely because both PSM and IPW depend heavily on a well-specified propensity model; a doubly robust estimator combines the propensity approach with an outcome model so that the estimate stays consistent even if one of the two models is misspecified, which is why it's often preferred over either PSM or IPW alone at a senior level, when the additional modeling complexity is worth it.

Direct answer. At least five plausible confounders: (1) tech-savviness, users comfortable customizing settings are also more engaged generally; (2) tenure, longer-tenured users both explore more settings and have had more time to build retention habits; (3) baseline engagement level, highly engaged users are more likely to discover and enable optional features, and engagement itself predicts retention; (4) acquisition channel, users from a high-intent channel (organic search, referral) may both self-select into trying personalization and retain better regardless; (5) device or platform, certain platforms may both surface the personalization option more prominently and independently correlate with retention.

Structured elaboration. Each of these biases the naive comparison the same way: something drives both whether a user enables personalization and how likely they are to retain, so comparing enablers to non-enablers mixes the personalization effect with the effect of whatever selected them into enabling it. The practical test for each candidate is whether removing it (holding it fixed, or matching on it) would plausibly shrink the 30-point gap; if tenure alone explains most of the gap because personalization-enablers are, on average, three times more tenured, that's the dominant confounder to address first.

Worked example. If enablers average 14 months of tenure and non-enablers average 5 months, and tenure alone predicts a large chunk of the retention gap independent of personalization, a naive comparison would attribute most of the 30-point lift to personalization when a large share is really a tenure effect. Matching enablers to non-enablers with similar tenure (and the other candidate confounders) would shrink the observed gap toward the true incremental effect.

Trade-offs and pitfalls. The goal of listing confounders isn't to prove personalization has no effect, it's to size how much of the raw 30 points survives once you account for the obvious selection story; a common mistake is treating "confounders exist" as equivalent to "there's no real effect," when the honest next step is a matched comparison or a randomized encouragement design to isolate the true incremental lift.

Direct answer. A propensity score is the estimated probability that a unit receives the treatment, given its observed covariates. Matching (or weighting) on it helps because it collapses many covariates into a single number: two units with similar propensity scores are, on average, similar across the whole set of covariates that went into the score, so comparing outcomes between treated and control units with similar propensity scores approximates a fair, apples-to-apples comparison, even though treatment wasn't randomly assigned.

Structured elaboration. The key theoretical result (Rosenbaum and Rubin) is that conditioning on the propensity score is, under the same assumptions needed for any observational adjustment (no unmeasured confounders, sufficient overlap), as good as conditioning on the full set of covariates directly. Practically, this matters because matching or adjusting on many covariates directly is hard (curse of dimensionality, sparse matches on any single dimension), while matching on one scalar score is tractable.

Worked example. You want to know whether users who enabled a premium onboarding flow (not randomly assigned) have higher 30-day revenue. Two users, one who enabled the flow and one who didn't, might look very different on ten separate covariates (tenure, device, acquisition channel, prior spend), but if a logistic regression predicting "enabled the flow" from those ten covariates gives both users a propensity score of 0.42, it means the model considers them equally likely to have enabled it given everything it knows, so comparing their actual outcomes is a fairer comparison than comparing two randomly chosen users.

Trade-offs and pitfalls. Matching on the propensity score only controls for confounders that are actually in the model; if an important driver of both enabling the flow and revenue was never measured, matching on the propensity score does nothing to fix that, it only balances the observed covariates. It's also only valid where there's overlap: if almost no non-enablers have a propensity score anywhere near 0.9, you simply can't estimate the effect for that region of the covariate space.

Direct answer. Extrapolating a short experiment's immediate engagement lift to a 6-month retention outcome relies on surrogate endpoints: short-term metrics that are believed to be reliable proxies for the long-term outcome you actually care about. The extrapolation is only as trustworthy as the evidence that the short-term surrogate reliably predicts the long-term outcome for THIS specific intervention, which needs to be checked, not assumed by default.

Structured elaboration. Key assumptions and how to check them:

• Surrogate validity: has this same short-term metric reliably predicted 6-month retention for OTHER similar past interventions? If the company has a history of comparable launches with both short-term and eventual long-term data, checking whether the historical relationship between the two is stable (and not itself confounded by novelty effects) is the strongest available evidence.
• No divergent long-run dynamics: an intervention that produces an immediate engagement spike from novelty, but nothing sustainable, would show a strong short-term surrogate signal that doesn't hold up at 6 months; distinguishing "durable engagement change" from "novelty spike" from the short-term data alone is genuinely hard, which is exactly why extrapolation carries real risk.
• Attrition: if the underlying experiment population itself experiences differential attrition (dropout, churn, or opt-out) between treatment and control before 6 months, that further complicates a clean extrapolation, since the population you're extrapolating for shrinks and changes composition unevenly across arms.

Estimation methods to combine with the surrogate. Difference-in-differences on the intermediate metric trajectory (does the treatment-control GAP grow, stay flat, or shrink over the weeks available so far, which hints at whether it's durable or decaying); survival analysis if retention is naturally modeled as time-to-churn, which handles partial/incomplete follow-up more honestly than a simple binary retained/churned cutoff at 6 months; and uplift modeling, if you specifically want to identify which SEGMENTS show a durable versus a decaying pattern, since the extrapolation risk often isn't uniform across the whole population.

Worked example. A feature shows a 15% week-1 engagement lift. If the treatment-control engagement gap for a comparable past launch was still roughly 60% of its original size by month 3 and roughly 40% by month 6 (a decaying but real and durable pattern, not a full novelty-effect collapse to zero), that historical decay curve is a defensible basis for a conservative 6-month retention extrapolation; without that kind of prior evidence, an honest answer states the extrapolation is speculative and recommends running the experiment longer, or accepting the uncertainty explicitly in the launch decision.

Trade-offs and pitfalls. The single most common mistake is presenting an extrapolated long-term number with the same confidence as a directly measured one; the surrogate-validity evidence (or lack of it) should travel WITH the extrapolated estimate, not get lost once it's summarized in a slide.