Causal Inference Questions

Direct answer. When a randomized experiment isn't possible, the right identification strategy depends on what natural variation exists in your data: instrumental variables (IV) need something that shifts treatment as-if-randomly without directly affecting the outcome; regression discontinuity (RDD) needs a sharp eligibility cutoff (a score, date, or threshold); difference-in-differences (DiD) needs a group that got treated and a comparable group that didn't, observed both before and after; and synthetic control needs a single treated unit (a market, a city) and enough untreated comparison units to construct a synthetic counterfactual from a weighted combination of them.

Structured elaboration.

Worked example. A pricing change rolls out in one country but not a comparable neighboring one at the same time: DiD is a natural first choice if pre-trends are parallel. If instead the price only applies above a specific score threshold used for a discount, RDD is the natural fit. If neither a clean control group nor a sharp cutoff exists, but you have a plausible instrument (say, a policy that shifted eligibility in a way unrelated to the outcome except through treatment), IV becomes the option, with instrument strength as the make-or-break diagnostic. If you have one affected market and many unaffected but comparable markets with rich pre-period history, synthetic control lets you construct a tailored counterfactual for that one market.

Trade-offs and pitfalls. Every one of these methods can produce a confident-looking point estimate that's wrong if its core assumption fails silently; the discipline that separates a credible quasi-experimental analysis from a plausible-looking one is running the specific falsification check for the specific method (parallel pre-trends for DiD, first-stage strength for IV, the manipulation test for RDD, pre-period fit for synthetic control) and being willing to report "I don't trust this identification" when the check fails. When several weak signals point the same direction, and no single design is airtight, combining evidence from multiple small experiments and observational analyses, and being explicit about how much you trust each, is often more honest than picking one method and reporting it as definitive.

Direct answer. A correlation only tells you that two things move together; it says nothing about why. Three general mechanisms besides direct causation can produce a correlation: a shared confounder that drives both variables, reverse causation (the effect actually precedes the presumed cause), and plain coincidence in a noisy sample. The next concrete step is not to argue in the abstract but to ask what would happen to the correlation if you could intervene on one variable while holding everything else fixed, which is exactly what an experiment (or a well-designed quasi-experiment) lets you check.

Structured elaboration. Walk a stakeholder through each mechanism with a question they can ask themselves:

• Confounding: is there a third factor plausibly driving both series? (Ice cream sales and drowning both rise with summer heat.)
• Reverse causation: could the "outcome" actually be causing the "cause"? (Struggling users might seek out a help feature, so help-feature usage correlates with struggling, not the other way around.)
• Coincidence: how much data is this based on, and did you check other time periods or segments? A correlation computed on one quarter of noisy data is weak evidence on its own.

Once you've ruled these out qualitatively, the real test is whether the relationship survives an intervention: randomize who gets the treatment, or find a situation where assignment was effectively random (a policy cutoff, a staggered rollout) and see if the effect holds.

Worked example. Suppose weekly push notifications sent and weekly active users (WAU) are strongly positively correlated ($r = 0.82$) over the last year. A marketer wants to conclude notifications drive WAU. Before agreeing: check for a shared confounder (both notification volume and WAU tend to rise around product launches and holidays), check reverse causation (the team may deliberately send more notifications because WAU dipped, to try to recover it, which would produce a correlation with the wrong sign of causality), and check the sample (is $r=0.82$ stable across quarters, or driven by one holiday spike?). The decisive next step is a holdout: stop sending notifications to a random 10% of users for two weeks and compare WAU between that holdout and the rest.

Trade-offs and pitfalls. The most common mistake is treating "we found a plausible confounder" as proof the relationship isn't causal at all, when the correct conclusion is usually "we can't tell yet, here's how we'd find out." A second pitfall is stopping at qualitative reasoning when a cheap experiment or natural-experiment check is available; qualitative reasoning is a screening step, not a substitute for evidence.

Direct answer. Because signup was voluntary, treated users aren't a random sample of the target population, they're systematically younger, which biases a naive average treatment effect if age (or whatever drove the younger skew) also affects the outcome. Inverse probability weighting (IPW), post-stratification, or a transportability adjustment can each recover an estimate that better represents the intended target population rather than just the self-selected sample.

Structured elaboration.

• Inverse probability weighting: model the probability of selecting into the sample (using age and other observed characteristics), then weight each observed unit by the inverse of that probability, so underrepresented groups (older users, in this case) count more heavily in the weighted average, correcting for the skew.
• Post-stratification/reweighting: split the sample into strata by age band, compute the effect within each stratum, then combine using the age distribution of the TARGET population (not the sample) as weights.
• Transportability: a more general framework for formally stating what additional assumptions are needed to transport an effect estimated in a selected sample to a different target population, useful when the selection mechanism is complex or only partially known.

Worked example. If the sample skews toward users under 30 (60% of the sample vs. 35% of the full user base), and the treatment effect is meaningfully different for younger versus older users, a raw average over the sample overweights the under-30 effect relative to the population you actually care about. Reweighting each age stratum's estimated effect by the population's true age distribution (35% weight for under-30, not 60%) produces an estimate that better reflects the effect for the population as a whole, assuming age captures the main driver of who self-selected in.

Trade-offs and pitfalls. All three corrections only fix selection ON OBSERVED characteristics; if something unmeasured (unusual motivation, prior exposure to a similar product) drove both the decision to sign up and the outcome, none of these techniques touch that, and the corrected estimate is still biased, just less so than the raw sample average. It's also worth being explicit that these corrections change WHAT population the estimate describes (the full target population versus the self-selected sample), which is exactly the kind of nuance interviewers probe for.

Direct answer. A natural experiment is a situation where some external, non-experimenter-controlled event or rule assigns different conditions to different units in a way that approximates random assignment, even though no one deliberately randomized anything. Its value is that it can support causal claims with observational data, IF the "as-if-random" assumption genuinely holds, which needs to be checked rather than assumed just because the situation is convenient.

Worked example. A regulatory change requiring a minimum age for a promotional discount takes effect on a specific date in one jurisdiction but not a neighboring one with similar demographics and market conditions; because the timing and jurisdiction of the regulation weren't chosen based on anticipated business outcomes, the regulation's rollout approximates random assignment of "exposed to the new rule" versus "not exposed," letting an analyst compare outcomes across the two jurisdictions as if the assignment had been randomized.

Structured elaboration: validating the as-if-random assumption. The natural experiment doesn't get a free pass just because it feels external; the standard checks:

• Balance check: compare pre-period covariates (and, where possible, pre-period outcome trends) across the "as-if-treated" and "as-if-control" groups; systematic pre-existing differences undermine the as-if-random claim.
• Placebo/falsification test: check for a "jump" at dates or thresholds where nothing actually happened, similar to a regression discontinuity design (RDD)'s manipulation checks; a spurious effect appearing where it shouldn't casts doubt on the design.
• Mechanism plausibility: articulate specifically why the timing/threshold that created the natural experiment was unrelated to anticipated outcomes; if the regulation's effective date was itself chosen in response to early signs of the trend you're studying, the "natural" experiment isn't actually exogenous.

Trade-offs and pitfalls. Natural experiments are attractive because they're cheap (no experiment to run) and can address questions a randomized controlled trial (RCT) couldn't ethically or practically test, but the "as-if-random" claim is doing all the causal work, and it's exactly the kind of assumption that's easy to assert and hard to fully verify; the strongest natural-experiment analyses spend as much effort validating that assumption as they do estimating the effect itself.

Direct answer. Randomly assigning units to treatment and control makes the two groups statistically identical, on average, in every characteristic, both the ones you can measure and the ones you can't, so any difference in outcomes between the groups can be attributed to the treatment rather than to some pre-existing difference between the people who got it and the people who didn't.

Structured elaboration. Without randomization, whoever ends up in the "treatment" group typically got there for a reason (they opted in, they were more engaged already, an algorithm targeted them), and that reason is very often also related to the outcome, which is exactly what confounding is. Randomization breaks that link mechanically: a coin flip has no relationship to a user's baseline behavior, so it can't be a confounder. This is why "the two groups look the same on every metric we checked before the experiment started" is treated as a sanity check on randomization rather than the reason it works: it works because of the coin flip, not because you happened to verify balance afterward.

Assumption that must still hold. Randomization on its own does not guarantee a valid estimate: you still need the Stable Unit Treatment Value Assumption to hold, meaning one unit's assignment doesn't affect another unit's outcome (no spillovers or interference between users), and that the treatment a unit is assigned to is the treatment it actually receives (no meaningful non-compliance). If a treated user's friends are also affected because they interact with the treated user, the clean "coin flip removes confounding" logic can still be undermined by that interference.

Trade-offs and pitfalls. The most common misstatement to correct here is "randomization guarantees the two groups are identical," when what it actually guarantees is that they're identical in expectation; any single randomized split can still show some imbalance by chance, especially with a small sample, which is exactly why pre-registered balance checks and adequate sample sizes matter even in a randomized design.