Causal Inference Questions

Direct answer. Whether the 8% US engagement lift should transport to other markets depends on whether the mechanism driving the effect is likely to be similar elsewhere, not just on the fact that the experiment was well-run; the practical recommendation is a smaller-scale, faster confirmatory test in one or two representative non-US markets before a full global rollout, rather than either blindly extrapolating or re-running the full US-scale experiment everywhere.

Structured elaboration. External validity concerns to check before trusting the transport: does the recommendation algorithm rely on signals (content catalog density, user behavior patterns) that differ meaningfully across markets? Are user demographics, device mix, or competitive landscape different enough that the mechanism behind the 8% lift might not hold? Is there a reason the US result itself might be inflated or unusual (a novelty effect specific to the timing of the US test, a US-specific promotional overlap)? None of these can be answered from the US experiment alone; they require either domain knowledge about the mechanism or additional data from the target markets.

Worked example. If the recommendation algorithm's lift came largely from surfacing under-explored long-tail content, and a target market has a much smaller content catalog, the mechanism plausibly weakens there, and blind rollout risks a much smaller (or even negative, if the algorithm's other trade-offs dominate without a large content pool to draw on) effect. A quick, cheap way to get real signal without a full-scale re-test: run a smaller, shorter experiment in one or two representative non-US markets, sized to detect a materially smaller effect than 8% (say, powered to detect 3%), which is enough to catch a "doesn't transport" result without the cost of a full re-test everywhere.

Trade-offs and pitfalls. The two extreme responses, "the randomized controlled trial (RCT) proved it works, roll out everywhere" and "every market is different, re-run the full experiment everywhere," are both wrong for different reasons: the first ignores that external validity is a real, checkable risk, not a formality, and the second is often needlessly slow and expensive when a smaller confirmatory test in a few representative markets would resolve most of the uncertainty at a fraction of the cost. Multi-country experiment designs that stratify by market characteristics upfront (rather than testing one market at a time sequentially) are the more scalable version of this same logic when you're rolling out across many markets at once.

Direct answer. With noncompliance in both directions, use the original random assignment (not actual treatment received) as the basis for the Intent-to-Treat (ITT) estimate, and use it as an instrument for actual treatment receipt to estimate the Complier Average Causal Effect (CACE, also called the Local Average Treatment Effect, LATE), which isolates the effect specifically for the "compliers," the subgroup whose actual treatment status follows their assignment.

Structured elaboration. ITT compares outcomes by original assignment, regardless of what treatment was actually received; it answers "what's the effect of OFFERING the treatment," which is often the policy-relevant question, but understates the effect of the treatment itself when compliance is imperfect (some assigned-treatment users never got it, diluting the measured effect toward zero). CACE/LATE uses the Wald estimator, dividing the ITT effect on the outcome by the ITT effect on actual treatment receipt (the "first stage"):

$$\widehat{\text{LATE}} = \frac{\bar{Y}_{assigned=1} - \bar{Y}_{assigned=0}}{\bar{D}_{assigned=1} - \bar{D}_{assigned=0}}$$

where $Y$ is the outcome and $D$ is an indicator for actually receiving treatment. This scales the diluted ITT effect back up by the compliance rate, recovering the effect specifically among compliers, those who would take the treatment if assigned to it and not take it if assigned to control.

Assumptions. The exclusion restriction requires that assignment affects the outcome ONLY through actual treatment receipt (being assigned to treatment can't itself change behavior except by changing whether you receive the treatment); monotonicity requires no "defiers" (nobody who would take the treatment specifically BECAUSE they were assigned to control, and refuse it specifically because they were assigned to treatment).

Worked example. With 1,000 assigned to treatment (700 actually received it, 300 didn't) and 1,000 assigned to control (50 got access anyway through leakage), suppose the treatment-assigned group's outcome averages 0.42 versus 0.38 for control-assigned (an ITT effect of 0.04). The first stage (compliance) difference is $0.70 - 0.05 = 0.65$. The LATE/CACE estimate is $0.04 / 0.65 \approx 0.062$, meaningfully larger than the raw ITT of 0.04, because it rescales for the fact that only 65 percentage points of the assignment difference actually translated into treatment-receipt difference.

Trade-offs and pitfalls. LATE is specifically the effect for compliers, not for the whole population; if compliers are systematically different from always-takers or never-takers (say, compliers are unusually price-sensitive, which is why they only take the treatment when nudged into it by assignment), the LATE doesn't generalize to "the effect on everyone," and that scoping should be stated explicitly rather than reported as if it were the ATE.

Direct answer. A confounder affects both the treatment and the outcome from outside the causal chain you care about, and biases the raw comparison if you don't control for it. A mediator sits on the causal path between the treatment and the outcome, meaning it's part of how the treatment produces its effect, not a distortion of it. A collider is a variable that is caused by both the treatment and the outcome (or by two things you're comparing), and conditioning on it (filtering, or accidentally selecting on it) creates a spurious association where none existed.

Structured elaboration. The practical consequence differs sharply: you generally must adjust for confounders (or they bias your estimate), you generally must NOT adjust for mediators if you want the total causal effect (adjusting for them would remove part of the very effect you're measuring), and you must NOT adjust for or select on colliders (doing so introduces bias that wasn't there before). Getting this backwards, adjusting for a mediator or a collider as if it were a confounder, is one of the most common causal-inference mistakes in practice.

One-sentence examples.

• Confounder: a user's baseline engagement level drives both whether they enable a premium feature and their retention, so it must be controlled for.
• Mediator: a UI redesign increases retention by increasing time-on-page; time-on-page is on the causal path, so controlling for it would hide part of the redesign's true effect.
• Collider: among users who reached the checkout page (a variable caused jointly by interest in the product and by the marketing channel that brought them there), interest and channel quality can appear artificially related, even if they're independent in the full population.

Trade-offs and pitfalls. The single most valuable habit is sketching a quick causal diagram before deciding what to adjust for, since "control for everything available" is not a safe default; over-controlling for mediators or colliders can introduce or hide bias just as badly as under-controlling for a real confounder.

Direct answer. Propensity score matching (PSM) estimates each unit's probability of receiving treatment given its observed covariates, then pairs treated and control units with similar scores so that the comparison approximates a randomized one. Done well, it turns "compare adopters to non-adopters" into "compare adopters to non-adopters who looked equally likely to adopt," which removes bias from the observed covariates driving self-selection.

Structured elaboration.

1. Covariate selection: include variables that plausibly affect both treatment and outcome (true confounders); avoid including anything downstream of treatment (mediators), which would bias the estimate the other direction.
2. Estimate the score: usually logistic regression, though any well-calibrated classifier works; the goal is a good conditional-probability estimate, not necessarily a high-accuracy classifier.
3. Choose a matching algorithm and choose among them deliberately: nearest-neighbor matching (pair each treated unit to its closest control) is simple but can leave large gaps if matches are poor; caliper matching adds a maximum-distance threshold so bad matches are dropped rather than forced; kernel matching uses a weighted average of nearby controls instead of a single match, trading some bias for lower variance. As a rule of thumb, use caliper matching when you have plenty of controls and want to guard against poor matches, and kernel matching when treated units are scarce and you can't afford to discard many.
4. Check match quality: compare standardized mean differences (SMD) on every covariate before and after matching; SMD under roughly 0.1 on each covariate is a common bar for "adequately balanced."
5. Estimate the average treatment effect on the treated (ATT) and its uncertainty: compute the outcome difference across matched pairs, and use a bootstrap or a matching-aware variance estimator (naive standard errors understate uncertainty because they ignore the matching step).

Worked example. Estimating the effect of a premium onboarding flow on 30-day revenue: fit a logistic regression for flow-enablement using tenure, acquisition channel, prior spend, and device type; match enablers to non-enablers with the closest propensity score within a 0.05 caliper; check that the SMD on prior spend, the covariate most likely to dominate self-selection, drops from roughly 0.4 pre-match to under 0.1 post-match; compute the ATT as the average revenue difference across the ~1,800 matched pairs that survived the caliper, with a bootstrap confidence interval.

Trade-offs and pitfalls. Two practical failure modes recur: first, PSM only balances covariates you actually measured, so a strong unmeasured confounder (say, marketing intent that never got logged) leaves the estimate biased no matter how tight the matches look; a sensitivity analysis for unobserved confounding is the honest follow-up. Second, discarding unmatched units (poor overlap, low prior spend or high tenure with no comparable match) changes the population the estimate applies to, from "everyone" to "the matchable subpopulation," which should be stated explicitly rather than glossed over.

Direct answer. A sensitivity analysis quantifies how strong an unmeasured confounder would need to be, in terms of its relationship to both treatment and outcome, before it could fully explain away your observed effect. Two standard tools are Rosenbaum bounds, which ask how much an unmeasured factor could distort the odds of treatment assignment before your conclusion (the p-value or effect direction) becomes unreliable, and the E-value, a more recent and more directly interpretable measure of the minimum strength of association, on the risk-ratio scale, that an unmeasured confounder would need with BOTH treatment and outcome to fully explain away the estimate.

Structured elaboration. The E-value for an observed risk ratio $RR$ (for $RR \geq 1$) is:

$$E = RR + \sqrt{RR \times (RR - 1)}$$

This gives the minimum risk ratio an unmeasured confounder would need to have with BOTH the treatment and the outcome, on this same multiplicative scale, to fully explain away the observed association. A larger E-value means a stronger, harder-to-dismiss result; an E-value close to 1 means even a weak unmeasured confounder could explain the whole effect away.

Worked example. Suppose an observational estimate finds a risk ratio of $RR = 1.5$ for a retention-boosting feature. Computing the E-value:

$$E = 1.5 + \sqrt{1.5 \times 0.5} = 1.5 + \sqrt{0.75} \approx 1.5 + 0.866 = 2.366$$

This means an unmeasured confounder would need to be associated with BOTH feature adoption and retention by a risk ratio of at least about 2.37 (each) to fully explain away the observed 1.5 effect. A plain-language interpretation for a non-technical stakeholder: "for this result to be spurious, there would need to be some unmeasured factor that makes users roughly two and a half times more likely to both adopt the feature and retain, purely by coincidence with the true effect being zero. Given the confounders we've already controlled for, that's a high bar, but not an impossible one, which is why we're still recommending a confirmatory experiment before scaling investment."

Trade-offs and pitfalls. A sensitivity analysis doesn't prove the result is unconfounded, it quantifies how much confounding would be needed to overturn it, which is a meaningfully different and more honest claim; the biggest pitfall is presenting a comfortably large E-value as proof of causality, when a genuinely strong unmeasured confounder (a marketing campaign that happened to target both feature-adopters and high-retention users) could still plausibly exceed even a moderately large threshold.