Causal Inference Questions

Direct answer. When a randomized experiment isn't possible, the right identification strategy depends on what natural variation exists in your data: instrumental variables (IV) need something that shifts treatment as-if-randomly without directly affecting the outcome; regression discontinuity (RDD) needs a sharp eligibility cutoff (a score, date, or threshold); difference-in-differences (DiD) needs a group that got treated and a comparable group that didn't, observed both before and after; and synthetic control needs a single treated unit (a market, a city) and enough untreated comparison units to construct a synthetic counterfactual from a weighted combination of them.

Structured elaboration.

Worked example. A pricing change rolls out in one country but not a comparable neighboring one at the same time: DiD is a natural first choice if pre-trends are parallel. If instead the price only applies above a specific score threshold used for a discount, RDD is the natural fit. If neither a clean control group nor a sharp cutoff exists, but you have a plausible instrument (say, a policy that shifted eligibility in a way unrelated to the outcome except through treatment), IV becomes the option, with instrument strength as the make-or-break diagnostic. If you have one affected market and many unaffected but comparable markets with rich pre-period history, synthetic control lets you construct a tailored counterfactual for that one market.

Trade-offs and pitfalls. Every one of these methods can produce a confident-looking point estimate that's wrong if its core assumption fails silently; the discipline that separates a credible quasi-experimental analysis from a plausible-looking one is running the specific falsification check for the specific method (parallel pre-trends for DiD, first-stage strength for IV, the manipulation test for RDD, pre-period fit for synthetic control) and being willing to report "I don't trust this identification" when the check fails. When several weak signals point the same direction, and no single design is airtight, combining evidence from multiple small experiments and observational analyses, and being explicit about how much you trust each, is often more honest than picking one method and reporting it as definitive.

Direct answer. A checklist for responsibly reporting causal claims from observational or quasi-experimental analyses should require, at minimum: which method was used and why it fits this situation, the specific identifying assumption it depends on, the diagnostic evidence for that assumption (not just that one was run, but what it showed), and an explicit confidence level with a matched recommended next action.

Structured elaboration.

Diagnostics to require by method: Difference-in-differences (DiD) needs a pre-trend comparison (and ideally a placebo test); propensity score matching (PSM) needs a balance table (standardized mean differences before/after matching) and an overlap check; regression discontinuity design (RDD) needs a manipulation test (such as McCrary) around the cutoff. Every write-up should name the specific method, state its core assumption in plain language, and show the diagnostic evidence for it, not just assert that "robustness checks were performed."

Worked example. A regression report claims "Feature X increases retention by 8%." The checklist would require: what method (a plain regression, or an identified quasi-experimental design)? What confounders were controlled for, and were mediators or colliders accidentally included? Was a pre-trend or balance diagnostic run, and what did it show? Given that, what confidence level does this claim deserve, and does the recommended action (say, a full rollout) match that level, or does it overreach given a "medium" confidence result?

Trade-offs and pitfalls. A checklist can become a compliance exercise that gets filled out mechanically without changing the actual rigor of the analysis; the fix is tying confidence levels directly to specific, checkable diagnostic evidence (not to the analyst's own subjective confidence), and requiring the recommended action to visibly match the stated confidence level rather than defaulting to "ship it" regardless.

Direct answer. Selection bias occurs when the process that determines who ends up in your sample (or who ends up treated) is itself related to the outcome, producing a non-representative sample; confounding occurs when a third variable independently affects both treatment and outcome. They often co-occur but are distinct mechanisms: selection bias is about who you observe or compare, confounding is about a shared cause distorting a comparison you can otherwise observe cleanly.

Structured elaboration and worked example. A company surveys customers who are still active to ask about satisfaction with a support interaction; customers who had a bad experience and churned are systematically excluded from the sample (they're gone), so the surviving sample is non-representative, satisfaction estimates from it will look better than the true population average, purely from who's left to survey. This is selection bias, distinct from a confounder like "account tenure" that might separately bias a comparison between two support channels within the surviving sample.

Correction methods. Weighting methods (like the inverse probability weighting used for selection above) can correct for selection ON OBSERVED characteristics that predict both selection and the outcome. The Heckman correction is a classic two-stage approach specifically designed for this shape of problem: model the selection process explicitly (a "selection equation" predicting who ends up in the sample) alongside the outcome equation, and use the selection model to correct the outcome estimate for the resulting non-random sample.

When selection bias is impossible to fully correct. If the very thing that determines selection is unmeasured and strongly related to the outcome (churned customers had systematically different reasons for churning that you have no proxy for), no reweighting or Heckman-style correction can fix it, because both require some observed variable that predicts selection well enough to model it. In that case the honest conclusion is that the estimate is only valid for the surviving/selected population, not the true target population, and that limitation should be stated rather than papered over with a correction that assumes more than the data supports.

Trade-offs and pitfalls. Reaching for "Heckman correction" as a default fix without checking whether its own exclusion-restriction-style assumption (a variable that predicts selection but not the outcome directly) is actually satisfiable is a common overreach; when no such variable exists, the correction is formally identified but practically fragile.

Direct answer. For each unit, the potential-outcomes framework posits two potential outcomes: $Y_i(1)$, what would happen to unit $i$ if treated, and $Y_i(0)$, what would happen if not treated. We can never observe both for the same unit at the same time (a user either sees the feature or doesn't), which is why this is called the "fundamental problem of causal inference": the individual causal effect $Y_i(1) - Y_i(0)$ is never directly observable, only estimable in aggregate under the right conditions.

Structured elaboration. This framework guides hypothesis formation by forcing precision: instead of vaguely asking "does the feature help retention," you're asking "what is $E[Y_i(1) - Y_i(0)]$" for a specific population, which immediately raises the right follow-up questions (which population? averaged how?). Randomization is what makes this estimable: by assigning units to treatment or control via a coin flip, the observed outcome for treated units, on average, stands in for the unobservable $Y_i(1)$ across the whole population, and the observed outcome for control units stands in for $Y_i(0)$, because the coin flip ensures neither group is systematically different in any way related to the potential outcomes themselves.

Worked example. For a checkout redesign, unit $i$'s potential outcomes are "would $i$ convert if shown the new design" ($Y_i(1)$) and "would $i$ convert if shown the old design" ($Y_i(0)$). We only ever observe one of these per user. Randomly assigning 50% of users to each version means the observed conversion rate in the treated group is an unbiased estimate of $E[Y_i(1)]$ across the whole randomized population, and the control group's rate estimates $E[Y_i(0)]$; their difference estimates the average treatment effect, even though no single user's individual effect was ever directly measured.

Trade-offs and pitfalls. The framework's power is also its limit: it formalizes exactly what causal inference can and can't claim, an AVERAGE effect across a population, not a guarantee about any specific individual, which is worth stating explicitly when someone asks "so did the feature help ME" from a group-level result.

Direct answer. Whether the 8% US engagement lift should transport to other markets depends on whether the mechanism driving the effect is likely to be similar elsewhere, not just on the fact that the experiment was well-run; the practical recommendation is a smaller-scale, faster confirmatory test in one or two representative non-US markets before a full global rollout, rather than either blindly extrapolating or re-running the full US-scale experiment everywhere.

Structured elaboration. External validity concerns to check before trusting the transport: does the recommendation algorithm rely on signals (content catalog density, user behavior patterns) that differ meaningfully across markets? Are user demographics, device mix, or competitive landscape different enough that the mechanism behind the 8% lift might not hold? Is there a reason the US result itself might be inflated or unusual (a novelty effect specific to the timing of the US test, a US-specific promotional overlap)? None of these can be answered from the US experiment alone; they require either domain knowledge about the mechanism or additional data from the target markets.

Worked example. If the recommendation algorithm's lift came largely from surfacing under-explored long-tail content, and a target market has a much smaller content catalog, the mechanism plausibly weakens there, and blind rollout risks a much smaller (or even negative, if the algorithm's other trade-offs dominate without a large content pool to draw on) effect. A quick, cheap way to get real signal without a full-scale re-test: run a smaller, shorter experiment in one or two representative non-US markets, sized to detect a materially smaller effect than 8% (say, powered to detect 3%), which is enough to catch a "doesn't transport" result without the cost of a full re-test everywhere.

Trade-offs and pitfalls. The two extreme responses, "the randomized controlled trial (RCT) proved it works, roll out everywhere" and "every market is different, re-run the full experiment everywhere," are both wrong for different reasons: the first ignores that external validity is a real, checkable risk, not a formality, and the second is often needlessly slow and expensive when a smaller confirmatory test in a few representative markets would resolve most of the uncertainty at a fraction of the cost. Multi-country experiment designs that stratify by market characteristics upfront (rather than testing one market at a time sequentially) are the more scalable version of this same logic when you're rolling out across many markets at once.