Causal Inference Questions

Direct answer. Off-policy (counterfactual) evaluation estimates how a NEW policy would have performed, using only data logged under an OLD policy, without deploying the new policy live; the main families of estimators are importance-weighting approaches (reweighting logged outcomes by how differently the new policy would have acted compared to the old one), direct/model-based approaches (predicting the new policy's outcome from a learned reward model), and doubly robust approaches (combining both, for the same robustness benefit as in the treatment-effect setting).

Structured elaboration.

• Importance-weighting family (inverse propensity scoring and its variants): reweight each logged (context, action, reward) triple by the ratio of the new policy's probability of taking that action to the old (logging) policy's probability of having taken it; this is unbiased under the right assumptions but can have very high variance when the two policies disagree substantially on which actions to take, since the ratio can become extreme. A self-normalized variant reduces this variance at the cost of a small bias, a common and often worthwhile trade in practice.
• Direct/model-based method: train a model predicting reward from context and action using the logged data, then use it to predict what the new policy's expected reward would be; lower variance than importance weighting, but biased if the reward model is misspecified, and it can look deceptively confident even when wrong.
• Doubly robust estimator: combines both, consistent if either the importance weights or the reward model is correctly specified, generally the preferred choice when feasible.

Where each can fail. Importance-weighting-based methods fail when the logging policy has essentially zero probability of ever taking an action the new policy strongly prefers (no support for that action in the logged data at all, an overlap violation exactly analogous to poor propensity-score overlap in the observational treatment-effect setting); the direct method fails silently when the reward model is systematically wrong in a way that isn't obvious from its training performance; and ALL of these methods degrade when non-stationarity means the logged data no longer reflects current user behavior (the world the old policy was logged in has meaningfully changed by the time you're evaluating the new one), since the entire framework assumes the logging data still represents the current population and context distribution.

Trade-offs and pitfalls. Off-policy evaluation is attractive because it avoids the cost and risk of a live deployment for the new policy, but it should be treated as a SCREENING step, ruling out clearly bad candidate policies cheaply, rather than a substitute for a final live validation before full rollout, precisely because the overlap and non-stationarity failure modes can silently produce a confidently wrong estimate that looks reasonable.

Direct answer. Randomly assigning units to treatment and control makes the two groups statistically identical, on average, in every characteristic, both the ones you can measure and the ones you can't, so any difference in outcomes between the groups can be attributed to the treatment rather than to some pre-existing difference between the people who got it and the people who didn't.

Structured elaboration. Without randomization, whoever ends up in the "treatment" group typically got there for a reason (they opted in, they were more engaged already, an algorithm targeted them), and that reason is very often also related to the outcome, which is exactly what confounding is. Randomization breaks that link mechanically: a coin flip has no relationship to a user's baseline behavior, so it can't be a confounder. This is why "the two groups look the same on every metric we checked before the experiment started" is treated as a sanity check on randomization rather than the reason it works: it works because of the coin flip, not because you happened to verify balance afterward.

Assumption that must still hold. Randomization on its own does not guarantee a valid estimate: you still need the Stable Unit Treatment Value Assumption to hold, meaning one unit's assignment doesn't affect another unit's outcome (no spillovers or interference between users), and that the treatment a unit is assigned to is the treatment it actually receives (no meaningful non-compliance). If a treated user's friends are also affected because they interact with the treated user, the clean "coin flip removes confounding" logic can still be undermined by that interference.

Trade-offs and pitfalls. The most common misstatement to correct here is "randomization guarantees the two groups are identical," when what it actually guarantees is that they're identical in expectation; any single randomized split can still show some imbalance by chance, especially with a small sample, which is exactly why pre-registered balance checks and adequate sample sizes matter even in a randomized design.

Direct answer. With staggered adoption and heterogeneous treatment effects across cohorts, the standard two-way fixed-effects (two-way fixed effects (TWFE)) difference-in-differences (DiD) estimator is a weighted average of many pairwise cohort comparisons, and some of those implicit comparisons use ALREADY-treated units, at a later stage of their own treatment effect, as if they were valid controls for a newly-treated cohort; when effects genuinely differ across cohorts, some of these implicit weights end up NEGATIVE, meaning the overall TWFE estimate can be biased in magnitude, and in extreme cases even in sign, relative to any reasonable notion of the "true" average treatment effect.

Structured elaboration. The core mechanical issue: TWFE's single treatment coefficient is forced to summarize a comparison that should really vary by cohort and by calendar time into one number, and the regression math accomplishes this by implicitly weighting different two-group, two-period comparisons by their relative sample size and variance, a weighting scheme that has nothing to do with how much you'd WANT to weight each cohort's effect, and can assign negative weight to comparisons involving already-treated units as controls.

Modern estimators that correct this. The Callaway and Sant'Anna estimator explicitly computes group-time average treatment effects for each cohort at each point in event time, using only not-yet-treated (or never-treated) units as valid comparisons at each step, then lets you aggregate those clean group-time effects into whatever overall summary (an overall average treatment effect on the treated (ATT), an event-study path) you actually want, with an aggregation scheme you control rather than one implicit in the regression. The Sun and Abraham estimator similarly uses an interaction-weighted approach that avoids using already-treated units as controls, producing a clean event-study plot of effects by time-since-adoption. Both trade some additional estimation complexity and a need for a reasonably large number of cohorts and never-or-not-yet-treated observations, for a materially more defensible estimate than plain TWFE whenever effects are genuinely heterogeneous across adoption timing.

Trade-offs and pitfalls. The corrected estimators generally need MORE structure to specify correctly (you have to decide how to aggregate the group-time effects into a summary number, which is itself a meaningful choice about what question you're answering) than a single plain TWFE regression, and they need enough never-treated or not-yet-treated observations at each relevant point to be well-powered; for panels with very few cohorts or almost universal, simultaneous adoption, plain TWFE's bias risk is smaller to begin with, and the added complexity of a modern estimator may not be worth it.

Direct answer. A checklist for responsibly reporting causal claims from observational or quasi-experimental analyses should require, at minimum: which method was used and why it fits this situation, the specific identifying assumption it depends on, the diagnostic evidence for that assumption (not just that one was run, but what it showed), and an explicit confidence level with a matched recommended next action.

Structured elaboration.

Diagnostics to require by method: Difference-in-differences (DiD) needs a pre-trend comparison (and ideally a placebo test); propensity score matching (PSM) needs a balance table (standardized mean differences before/after matching) and an overlap check; regression discontinuity design (RDD) needs a manipulation test (such as McCrary) around the cutoff. Every write-up should name the specific method, state its core assumption in plain language, and show the diagnostic evidence for it, not just assert that "robustness checks were performed."

Worked example. A regression report claims "Feature X increases retention by 8%." The checklist would require: what method (a plain regression, or an identified quasi-experimental design)? What confounders were controlled for, and were mediators or colliders accidentally included? Was a pre-trend or balance diagnostic run, and what did it show? Given that, what confidence level does this claim deserve, and does the recommended action (say, a full rollout) match that level, or does it overreach given a "medium" confidence result?

Trade-offs and pitfalls. A checklist can become a compliance exercise that gets filled out mechanically without changing the actual rigor of the analysis; the fix is tying confidence levels directly to specific, checkable diagnostic evidence (not to the analyst's own subjective confidence), and requiring the recommended action to visibly match the stated confidence level rather than defaulting to "ship it" regardless.

Direct answer. At least five plausible confounders: (1) tech-savviness, users comfortable customizing settings are also more engaged generally; (2) tenure, longer-tenured users both explore more settings and have had more time to build retention habits; (3) baseline engagement level, highly engaged users are more likely to discover and enable optional features, and engagement itself predicts retention; (4) acquisition channel, users from a high-intent channel (organic search, referral) may both self-select into trying personalization and retain better regardless; (5) device or platform, certain platforms may both surface the personalization option more prominently and independently correlate with retention.

Structured elaboration. Each of these biases the naive comparison the same way: something drives both whether a user enables personalization and how likely they are to retain, so comparing enablers to non-enablers mixes the personalization effect with the effect of whatever selected them into enabling it. The practical test for each candidate is whether removing it (holding it fixed, or matching on it) would plausibly shrink the 30-point gap; if tenure alone explains most of the gap because personalization-enablers are, on average, three times more tenured, that's the dominant confounder to address first.

Worked example. If enablers average 14 months of tenure and non-enablers average 5 months, and tenure alone predicts a large chunk of the retention gap independent of personalization, a naive comparison would attribute most of the 30-point lift to personalization when a large share is really a tenure effect. Matching enablers to non-enablers with similar tenure (and the other candidate confounders) would shrink the observed gap toward the true incremental effect.

Trade-offs and pitfalls. The goal of listing confounders isn't to prove personalization has no effect, it's to size how much of the raw 30 points survives once you account for the obvious selection story; a common mistake is treating "confounders exist" as equivalent to "there's no real effect," when the honest next step is a matched comparison or a randomized encouragement design to isolate the true incremental lift.