Causal Inference Questions

Direct answer. Average Treatment Effect (ATE) is the average effect of the treatment if everyone in the population received it, compared to if no one did. Average Treatment Effect on the Treated (ATT) is the average effect specifically among the people who actually received the treatment, comparing their outcome to what it would have been had they not received it. For a feature only 10% of users adopt spontaneously, ATT is almost always the more useful number for a promotion or rollout decision, because it answers "what did the treatment do for the people who took it" rather than "what would it do if we forced it on the 90% who currently choose not to," and those two groups may respond very differently.

Structured elaboration. ATE and ATT coincide when treatment assignment is random (a coin flip doesn't correlate with who benefits more), but diverge whenever adoption is self-selected, which is the normal case for an optional feature: users who adopt spontaneously often differ systematically from those who don't, in ways that plausibly correlate with how much the feature helps them. If you're deciding whether to keep promoting the feature to the same self-selecting population, ATT tells you what it's actually doing for them. If you're deciding whether to force the feature on the 90% who haven't adopted, neither ATE nor ATT alone answers that; you'd want the effect specifically among the non-adopters (sometimes called ATU, the average treatment effect on the untreated), since they may be the low performers who chose not to adopt for reasons that make it less likely to help them too.

Worked example. In-app notification digests are adopted spontaneously by 10% of users. Among adopters, weekly sessions are 12% higher than a matched estimate of what they'd be without the digest (that's the ATT). If you're deciding "should we keep this feature and keep letting interested users find it," ATT of +12% directly answers that. If you're deciding "should we auto-enable it for the other 90%," ATT tells you nothing reliable about that group, since they self-selected out, possibly because it wouldn't help them as much.

Trade-offs and pitfalls. A common mistake is quoting an ATT from an opt-in population as if it were the ATE that would apply if the feature were force-enabled for everyone; the two can differ substantially, and reporting one as if it were the other overstates the expected impact of a broader rollout.

Direct answer. Check overlap (also called common support) by comparing the distributions of propensity scores between the treated and control groups. If both groups span a similar range of scores, with enough control units near every treated unit's score and vice versa, comparisons are trustworthy. If treated units cluster at scores where almost no control units exist (or the reverse), the propensity model is effectively extrapolating, and the resulting comparison isn't supported by real data.

Structured elaboration. Two practical checks: (1) plot the propensity score distributions for both groups on the same axis and visually inspect where they do and don't overlap; (2) after matching, report how many treated units were dropped for lack of a close enough control match (a high drop rate is itself a symptom of poor overlap). A stricter numeric check is to look at the propensity score range where both groups have reasonable density, sometimes trimmed to, say, the 5th to 95th percentile of the overlapping region, and restrict the analysis to that region rather than the full sample.

Worked example. Suppose you're comparing users who enabled a feature (treated) against those who didn't (control) using a propensity model built from tenure and engagement. If nearly all treated users have propensity scores between 0.6 and 0.95, but 90% of control users have scores under 0.3, only a thin sliver of control users (those with scores above 0.6) is actually comparable to the treated group; the average treatment effect on the treated (ATT) estimate would then only be credible for that sliver, not for the typical treated user.

Trade-offs and pitfalls. It's tempting to fix poor overlap by widening the caliper or accepting worse matches, but that just trades an honest "we can't estimate this for the full population" for a quietly biased estimate. The more defensible response is to redefine the estimand (report the ATT only for the region of common support, and say so explicitly) or collect more data in the poorly-overlapped region.

Direct answer. With staggered adoption and heterogeneous treatment effects across cohorts, the standard two-way fixed-effects (two-way fixed effects (TWFE)) difference-in-differences (DiD) estimator is a weighted average of many pairwise cohort comparisons, and some of those implicit comparisons use ALREADY-treated units, at a later stage of their own treatment effect, as if they were valid controls for a newly-treated cohort; when effects genuinely differ across cohorts, some of these implicit weights end up NEGATIVE, meaning the overall TWFE estimate can be biased in magnitude, and in extreme cases even in sign, relative to any reasonable notion of the "true" average treatment effect.

Structured elaboration. The core mechanical issue: TWFE's single treatment coefficient is forced to summarize a comparison that should really vary by cohort and by calendar time into one number, and the regression math accomplishes this by implicitly weighting different two-group, two-period comparisons by their relative sample size and variance, a weighting scheme that has nothing to do with how much you'd WANT to weight each cohort's effect, and can assign negative weight to comparisons involving already-treated units as controls.

Modern estimators that correct this. The Callaway and Sant'Anna estimator explicitly computes group-time average treatment effects for each cohort at each point in event time, using only not-yet-treated (or never-treated) units as valid comparisons at each step, then lets you aggregate those clean group-time effects into whatever overall summary (an overall average treatment effect on the treated (ATT), an event-study path) you actually want, with an aggregation scheme you control rather than one implicit in the regression. The Sun and Abraham estimator similarly uses an interaction-weighted approach that avoids using already-treated units as controls, producing a clean event-study plot of effects by time-since-adoption. Both trade some additional estimation complexity and a need for a reasonably large number of cohorts and never-or-not-yet-treated observations, for a materially more defensible estimate than plain TWFE whenever effects are genuinely heterogeneous across adoption timing.

Trade-offs and pitfalls. The corrected estimators generally need MORE structure to specify correctly (you have to decide how to aggregate the group-time effects into a summary number, which is itself a meaningful choice about what question you're answering) than a single plain TWFE regression, and they need enough never-treated or not-yet-treated observations at each relevant point to be well-powered; for panels with very few cohorts or almost universal, simultaneous adoption, plain TWFE's bias risk is smaller to begin with, and the added complexity of a modern estimator may not be worth it.

Direct answer. Mediation analysis decomposes a treatment's total effect on an outcome into a direct effect (the part not going through the named mediator) and an indirect effect (the part that flows specifically through the mediator), which matters whenever the mechanism, not just the existence, of an effect is the question you actually care about.

Structured elaboration. The sequential ignorability assumption requires two things: the treatment is as-good-as-randomly assigned with respect to the outcome (the usual causal assumption), AND the mediator is as-good-as-randomly assigned with respect to the outcome, GIVEN the treatment and any measured pre-treatment covariates (a stronger, second assumption that's often the weaker link in practice, since the mediator usually wasn't itself randomized). Estimation typically proceeds by fitting a mediator model (predicting the mediator from treatment and covariates) and an outcome model (predicting the outcome from treatment, the mediator, and covariates), then combining them to decompose the total effect: the indirect effect is roughly the mediator model's treatment coefficient times the outcome model's mediator coefficient, and the direct effect is the outcome model's treatment coefficient with the mediator held fixed.

Pitfall: post-treatment confounding of the mediator-outcome relationship. If something occurring AFTER treatment affects both the mediator and the outcome (a post-treatment confounder), the sequential ignorability assumption is violated even if the treatment itself was randomized, because the mediator-to-outcome relationship is now confounded by something the treatment itself may have influenced. This is a genuinely hard problem to fully resolve, since a post-treatment confounder can't simply be "controlled for" the way a pre-treatment confounder can, controlling for it risks the same mediator-adjustment trap discussed for DAGs (adjusting for something on or affected by the causal path).

Worked example. For a treatment T (a UI change) affecting an outcome Y (purchases) through a mediator M (increased engagement), if the mediator model shows engagement rises by 0.3 units per unit of treatment, and the outcome model shows purchases rise by 2 units per unit of engagement (holding treatment fixed), the indirect effect is roughly $0.3 \times 2 = 0.6$ units of purchase increase running through engagement; if the total effect of the UI change on purchases is 1.0 unit, the direct effect (not through engagement) is roughly $1.0 - 0.6 = 0.4$ units.

Trade-offs and pitfalls. Mediation analysis answers a genuinely different and often more decision-relevant question than a plain treatment-effect estimate ("is the effect real") because it tells you WHERE the effect is coming from, which shapes what to invest in next (double down on driving the mediator, versus the treatment has value beyond that one mechanism); the cost is a stronger, harder-to-verify assumption set than a simple total-effect estimate requires.

Direct answer. Off-policy (counterfactual) evaluation estimates how a NEW policy would have performed, using only data logged under an OLD policy, without deploying the new policy live; the main families of estimators are importance-weighting approaches (reweighting logged outcomes by how differently the new policy would have acted compared to the old one), direct/model-based approaches (predicting the new policy's outcome from a learned reward model), and doubly robust approaches (combining both, for the same robustness benefit as in the treatment-effect setting).

Structured elaboration.

• Importance-weighting family (inverse propensity scoring and its variants): reweight each logged (context, action, reward) triple by the ratio of the new policy's probability of taking that action to the old (logging) policy's probability of having taken it; this is unbiased under the right assumptions but can have very high variance when the two policies disagree substantially on which actions to take, since the ratio can become extreme. A self-normalized variant reduces this variance at the cost of a small bias, a common and often worthwhile trade in practice.
• Direct/model-based method: train a model predicting reward from context and action using the logged data, then use it to predict what the new policy's expected reward would be; lower variance than importance weighting, but biased if the reward model is misspecified, and it can look deceptively confident even when wrong.
• Doubly robust estimator: combines both, consistent if either the importance weights or the reward model is correctly specified, generally the preferred choice when feasible.

Where each can fail. Importance-weighting-based methods fail when the logging policy has essentially zero probability of ever taking an action the new policy strongly prefers (no support for that action in the logged data at all, an overlap violation exactly analogous to poor propensity-score overlap in the observational treatment-effect setting); the direct method fails silently when the reward model is systematically wrong in a way that isn't obvious from its training performance; and ALL of these methods degrade when non-stationarity means the logged data no longer reflects current user behavior (the world the old policy was logged in has meaningfully changed by the time you're evaluating the new one), since the entire framework assumes the logging data still represents the current population and context distribution.

Trade-offs and pitfalls. Off-policy evaluation is attractive because it avoids the cost and risk of a live deployment for the new policy, but it should be treated as a SCREENING step, ruling out clearly bad candidate policies cheaply, rather than a substitute for a final live validation before full rollout, precisely because the overlap and non-stationarity failure modes can silently produce a confidently wrong estimate that looks reasonable.