Causal Inference Questions

Direct answer. An interrupted time series (ITS) design compares the trajectory of a metric before and after an intervention that applied to everyone at once (so there's no control group), modeling the pre-period trend and level, then testing whether the actual post-period trajectory diverges from what that pre-period trend would have predicted. The break at the intervention date, both a level shift and/or a slope change, is the estimated effect.

Structured elaboration. Because there's no control group, the entire causal claim rests on correctly modeling what the trend WOULD have looked like without the intervention, which makes trend and seasonality adjustment the core of the design rather than an afterthought. A typical specification: $Y_t = \beta_0 + \beta_1 t + \beta_2 \cdot \text{post}_t + \beta_3 \cdot t_{\text{since intervention}} + \varepsilon_t$, where $\beta_2$ captures an immediate level shift at the intervention and $\beta_3$ captures a change in slope afterward; seasonal terms (monthly or weekly dummies, or a Fourier-term representation) are added when the metric has known seasonal structure.

Diagnostics and sensitivity checks.

• Autocorrelation: time series residuals are rarely independent; check the residual autocorrelation function and, if needed, model residuals with an autoregressive term rather than assuming plain OLS-style independent errors, since ignoring autocorrelation understates the standard errors and overstates confidence in the break.
• Trend stability: fit the pre-period trend on a few different pre-period window lengths and check the estimated break is stable, not an artifact of exactly where the pre-period window starts.
• Placebo break dates: apply the same model with a FAKE break date somewhere in the pre-period; a spurious "significant break" at a date nothing happened is a red flag about the model's specification.

Worked example. For removing a free tier, daily active users (DAU) had been growing at roughly 0.3% per day for six months before the change. After the removal, the level drops sharply and the growth rate flattens to roughly 0.05% per day. The ITS model would show a significant negative $\beta_2$ (the immediate drop) and a significant negative $\beta_3$ (the growth-rate slowdown), separately quantifying the immediate shock from the longer-run trajectory change, which matters for deciding whether the effect is a one-time adjustment or a lasting structural shift.

Trade-offs and pitfalls. ITS versus difference-in-differences (DiD) is largely a data-availability question: use ITS when no comparable untreated group exists (a universal, simultaneous rollout), and prefer DiD whenever a genuine comparison group is available, since DiD's parallel-trends assumption is generally easier to defend than ITS's implicit assumption that the pre-period trend, once modeled, would have continued unchanged.

Direct answer. A sensitivity analysis quantifies how strong an unmeasured confounder would need to be, in terms of its relationship to both treatment and outcome, before it could fully explain away your observed effect. Two standard tools are Rosenbaum bounds, which ask how much an unmeasured factor could distort the odds of treatment assignment before your conclusion (the p-value or effect direction) becomes unreliable, and the E-value, a more recent and more directly interpretable measure of the minimum strength of association, on the risk-ratio scale, that an unmeasured confounder would need with BOTH treatment and outcome to fully explain away the estimate.

Structured elaboration. The E-value for an observed risk ratio $RR$ (for $RR \geq 1$) is:

$$E = RR + \sqrt{RR \times (RR - 1)}$$

This gives the minimum risk ratio an unmeasured confounder would need to have with BOTH the treatment and the outcome, on this same multiplicative scale, to fully explain away the observed association. A larger E-value means a stronger, harder-to-dismiss result; an E-value close to 1 means even a weak unmeasured confounder could explain the whole effect away.

Worked example. Suppose an observational estimate finds a risk ratio of $RR = 1.5$ for a retention-boosting feature. Computing the E-value:

$$E = 1.5 + \sqrt{1.5 \times 0.5} = 1.5 + \sqrt{0.75} \approx 1.5 + 0.866 = 2.366$$

This means an unmeasured confounder would need to be associated with BOTH feature adoption and retention by a risk ratio of at least about 2.37 (each) to fully explain away the observed 1.5 effect. A plain-language interpretation for a non-technical stakeholder: "for this result to be spurious, there would need to be some unmeasured factor that makes users roughly two and a half times more likely to both adopt the feature and retain, purely by coincidence with the true effect being zero. Given the confounders we've already controlled for, that's a high bar, but not an impossible one, which is why we're still recommending a confirmatory experiment before scaling investment."

Trade-offs and pitfalls. A sensitivity analysis doesn't prove the result is unconfounded, it quantifies how much confounding would be needed to overturn it, which is a meaningfully different and more honest claim; the biggest pitfall is presenting a comfortably large E-value as proof of causality, when a genuinely strong unmeasured confounder (a marketing campaign that happened to target both feature-adopters and high-retention users) could still plausibly exceed even a moderately large threshold.

Direct answer. Whether the 8% US engagement lift should transport to other markets depends on whether the mechanism driving the effect is likely to be similar elsewhere, not just on the fact that the experiment was well-run; the practical recommendation is a smaller-scale, faster confirmatory test in one or two representative non-US markets before a full global rollout, rather than either blindly extrapolating or re-running the full US-scale experiment everywhere.

Structured elaboration. External validity concerns to check before trusting the transport: does the recommendation algorithm rely on signals (content catalog density, user behavior patterns) that differ meaningfully across markets? Are user demographics, device mix, or competitive landscape different enough that the mechanism behind the 8% lift might not hold? Is there a reason the US result itself might be inflated or unusual (a novelty effect specific to the timing of the US test, a US-specific promotional overlap)? None of these can be answered from the US experiment alone; they require either domain knowledge about the mechanism or additional data from the target markets.

Worked example. If the recommendation algorithm's lift came largely from surfacing under-explored long-tail content, and a target market has a much smaller content catalog, the mechanism plausibly weakens there, and blind rollout risks a much smaller (or even negative, if the algorithm's other trade-offs dominate without a large content pool to draw on) effect. A quick, cheap way to get real signal without a full-scale re-test: run a smaller, shorter experiment in one or two representative non-US markets, sized to detect a materially smaller effect than 8% (say, powered to detect 3%), which is enough to catch a "doesn't transport" result without the cost of a full re-test everywhere.

Trade-offs and pitfalls. The two extreme responses, "the randomized controlled trial (RCT) proved it works, roll out everywhere" and "every market is different, re-run the full experiment everywhere," are both wrong for different reasons: the first ignores that external validity is a real, checkable risk, not a formality, and the second is often needlessly slow and expensive when a smaller confirmatory test in a few representative markets would resolve most of the uncertainty at a fraction of the cost. Multi-country experiment designs that stratify by market characteristics upfront (rather than testing one market at a time sequentially) are the more scalable version of this same logic when you're rolling out across many markets at once.

Direct answer. A propensity score is the estimated probability that a unit receives the treatment, given its observed covariates. Matching (or weighting) on it helps because it collapses many covariates into a single number: two units with similar propensity scores are, on average, similar across the whole set of covariates that went into the score, so comparing outcomes between treated and control units with similar propensity scores approximates a fair, apples-to-apples comparison, even though treatment wasn't randomly assigned.

Structured elaboration. The key theoretical result (Rosenbaum and Rubin) is that conditioning on the propensity score is, under the same assumptions needed for any observational adjustment (no unmeasured confounders, sufficient overlap), as good as conditioning on the full set of covariates directly. Practically, this matters because matching or adjusting on many covariates directly is hard (curse of dimensionality, sparse matches on any single dimension), while matching on one scalar score is tractable.

Worked example. You want to know whether users who enabled a premium onboarding flow (not randomly assigned) have higher 30-day revenue. Two users, one who enabled the flow and one who didn't, might look very different on ten separate covariates (tenure, device, acquisition channel, prior spend), but if a logistic regression predicting "enabled the flow" from those ten covariates gives both users a propensity score of 0.42, it means the model considers them equally likely to have enabled it given everything it knows, so comparing their actual outcomes is a fairer comparison than comparing two randomly chosen users.

Trade-offs and pitfalls. Matching on the propensity score only controls for confounders that are actually in the model; if an important driver of both enabling the flow and revenue was never measured, matching on the propensity score does nothing to fix that, it only balances the observed covariates. It's also only valid where there's overlap: if almost no non-enablers have a propensity score anywhere near 0.9, you simply can't estimate the effect for that region of the covariate space.

Direct answer. With treatment $A$, outcome $Y$, a confounder $Z$ (which affects both $A$ and $Y$), a mediator $M$ on the path $A \to M \to Y$, and a collider $C$ caused by both $A$ and $Y$: the backdoor criterion says you must adjust for $Z$ (it opens a non-causal "backdoor" path between $A$ and $Y$ that needs to be blocked), and you must NOT adjust for $M$ (it's on the causal path itself; controlling for it would remove part of the very effect you're trying to measure) or $C$ (conditioning on a collider opens a new spurious association rather than blocking one).

Structured elaboration. The backdoor criterion, informally: a set of variables satisfies it for estimating the effect of $A$ on $Y$ if it blocks every "backdoor path" (a path into $A$ that isn't through the causal arrows you care about) while not including any descendant of $A$ on the causal path to $Y$. Here, $Z \to A$ and $Z \to Y$ forms exactly such a backdoor path; adjusting for $Z$ blocks it. $M$ is a "front-door" variable on the causal path from $A$ to $Y$, not a backdoor variable, so it's excluded. $C$ is a collider: since $A \to C \leftarrow Y$, $C$ is not a path INTO $A$ at all, and conditioning on a collider (by adjusting for it, or by any implicit selection that filters on it) creates a spurious association between $A$ and $Y$ that wasn't there before, because you'd be selecting a biased subsample where $A$ and $Y$ appear artificially linked through their shared effect.

Worked example. In a promotion-click-purchase scenario: promotion increases the chance of a click, and clicking increases the chance of a purchase; income independently increases both the chance of seeing/engaging with the promotion AND the chance of purchasing. To estimate the causal effect of the promotion on purchase, adjust for income (a backdoor confounder), but do NOT adjust for click even though it sits between promotion and purchase; click is a mediator of the very effect (promotion causes purchase partly BY causing clicks), so controlling for it would strip out that channel and understate the promotion's true effect. Similarly, for a loyalty program where users self-select into enrollment based on pre-existing purchase frequency (a confounder) and enrollment also increases engagement which independently affects frequency (a mediator on the path), the same logic applies: adjust for pre-existing purchase frequency, not for post-enrollment engagement.

Trade-offs and pitfalls. The most common real-world mistake is reflexively "controlling for everything available in the dataset," which silently includes mediators and colliders alongside genuine confounders; sketching the DAG (directed acyclic graph) first, even informally, is what turns "which variables go in the model" from an intuition call into a mechanical, checkable decision.