Causal Inference Questions

Direct answer. A confounder affects both the treatment and the outcome from outside the causal chain you care about, and biases the raw comparison if you don't control for it. A mediator sits on the causal path between the treatment and the outcome, meaning it's part of how the treatment produces its effect, not a distortion of it. A collider is a variable that is caused by both the treatment and the outcome (or by two things you're comparing), and conditioning on it (filtering, or accidentally selecting on it) creates a spurious association where none existed.

Structured elaboration. The practical consequence differs sharply: you generally must adjust for confounders (or they bias your estimate), you generally must NOT adjust for mediators if you want the total causal effect (adjusting for them would remove part of the very effect you're measuring), and you must NOT adjust for or select on colliders (doing so introduces bias that wasn't there before). Getting this backwards, adjusting for a mediator or a collider as if it were a confounder, is one of the most common causal-inference mistakes in practice.

One-sentence examples.

• Confounder: a user's baseline engagement level drives both whether they enable a premium feature and their retention, so it must be controlled for.
• Mediator: a UI redesign increases retention by increasing time-on-page; time-on-page is on the causal path, so controlling for it would hide part of the redesign's true effect.
• Collider: among users who reached the checkout page (a variable caused jointly by interest in the product and by the marketing channel that brought them there), interest and channel quality can appear artificially related, even if they're independent in the full population.

Trade-offs and pitfalls. The single most valuable habit is sketching a quick causal diagram before deciding what to adjust for, since "control for everything available" is not a safe default; over-controlling for mediators or colliders can introduce or hide bias just as badly as under-controlling for a real confounder.

Direct answer. For each unit, the potential-outcomes framework posits two potential outcomes: $Y_i(1)$, what would happen to unit $i$ if treated, and $Y_i(0)$, what would happen if not treated. We can never observe both for the same unit at the same time (a user either sees the feature or doesn't), which is why this is called the "fundamental problem of causal inference": the individual causal effect $Y_i(1) - Y_i(0)$ is never directly observable, only estimable in aggregate under the right conditions.

Structured elaboration. This framework guides hypothesis formation by forcing precision: instead of vaguely asking "does the feature help retention," you're asking "what is $E[Y_i(1) - Y_i(0)]$" for a specific population, which immediately raises the right follow-up questions (which population? averaged how?). Randomization is what makes this estimable: by assigning units to treatment or control via a coin flip, the observed outcome for treated units, on average, stands in for the unobservable $Y_i(1)$ across the whole population, and the observed outcome for control units stands in for $Y_i(0)$, because the coin flip ensures neither group is systematically different in any way related to the potential outcomes themselves.

Worked example. For a checkout redesign, unit $i$'s potential outcomes are "would $i$ convert if shown the new design" ($Y_i(1)$) and "would $i$ convert if shown the old design" ($Y_i(0)$). We only ever observe one of these per user. Randomly assigning 50% of users to each version means the observed conversion rate in the treated group is an unbiased estimate of $E[Y_i(1)]$ across the whole randomized population, and the control group's rate estimates $E[Y_i(0)]$; their difference estimates the average treatment effect, even though no single user's individual effect was ever directly measured.

Trade-offs and pitfalls. The framework's power is also its limit: it formalizes exactly what causal inference can and can't claim, an AVERAGE effect across a population, not a guarantee about any specific individual, which is worth stating explicitly when someone asks "so did the feature help ME" from a group-level result.

Direct answer. Check overlap (also called common support) by comparing the distributions of propensity scores between the treated and control groups. If both groups span a similar range of scores, with enough control units near every treated unit's score and vice versa, comparisons are trustworthy. If treated units cluster at scores where almost no control units exist (or the reverse), the propensity model is effectively extrapolating, and the resulting comparison isn't supported by real data.

Structured elaboration. Two practical checks: (1) plot the propensity score distributions for both groups on the same axis and visually inspect where they do and don't overlap; (2) after matching, report how many treated units were dropped for lack of a close enough control match (a high drop rate is itself a symptom of poor overlap). A stricter numeric check is to look at the propensity score range where both groups have reasonable density, sometimes trimmed to, say, the 5th to 95th percentile of the overlapping region, and restrict the analysis to that region rather than the full sample.

Worked example. Suppose you're comparing users who enabled a feature (treated) against those who didn't (control) using a propensity model built from tenure and engagement. If nearly all treated users have propensity scores between 0.6 and 0.95, but 90% of control users have scores under 0.3, only a thin sliver of control users (those with scores above 0.6) is actually comparable to the treated group; the average treatment effect on the treated (ATT) estimate would then only be credible for that sliver, not for the typical treated user.

Trade-offs and pitfalls. It's tempting to fix poor overlap by widening the caliper or accepting worse matches, but that just trades an honest "we can't estimate this for the full population" for a quietly biased estimate. The more defensible response is to redefine the estimand (report the ATT only for the region of common support, and say so explicitly) or collect more data in the poorly-overlapped region.

Direct answer. Controlling for a variable that sits ON the causal path between treatment and outcome (a post-treatment variable, or more specifically here, a mediator) removes part of the very effect you're trying to measure, because you're statistically holding constant something the treatment itself changes; the resulting "controlled" coefficient understates (or in some cases completely masks) the treatment's true total effect, even though the regression output looks perfectly well-specified.

Structured elaboration. The core question to ask the analyst: for each control variable, is it something that existed and was determined BEFORE the treatment (a legitimate pre-treatment confounder, safe to control for), or could the treatment itself plausibly have influenced it (a post-treatment variable, dangerous to control for as if it were a confounder)? A control variable measured after treatment started is not automatically a mediator (some post-treatment variables genuinely are unaffected by the treatment), but it needs a specific causal argument for why it's safe, not just a timestamp check.

Questions to ask the analyst. "Is this control variable measured before or after the treatment was applied?" "Could the treatment plausibly have changed this control variable's value?" "If we dropped this control, does the treatment coefficient change substantially, and in which direction?" A large, systematic shift in the treatment coefficient when a suspected post-treatment variable is added is itself diagnostic: it suggests the control is absorbing part of the treatment's real effect rather than isolating a confound.

Worked example. A regression estimating the effect of a loyalty program on total spend controls for "number of app sessions in the following month." If the loyalty program itself drives more app sessions (which is plausible, and part of HOW the program increases spend), controlling for sessions removes that channel from the estimated effect, understating the program's true total impact on spend; the coefficient on the loyalty program in this specification answers a narrower question ("holding session count fixed, does the program still increase spend"), not the total causal effect a PM actually wants to know.

Trade-offs and pitfalls. Post-treatment bias is genuinely easy to introduce by accident, since "control for more variables" feels like careful analysis by default, and the regression's statistical output gives no automatic warning that a control is actually a mediator; the discipline of explicitly checking each control's timing and plausible causal relationship to the treatment, before trusting the regression, is what catches it.

Direct answer. Propensity score matching (PSM) estimates each unit's probability of receiving treatment given its observed covariates, then pairs treated and control units with similar scores so that the comparison approximates a randomized one. Done well, it turns "compare adopters to non-adopters" into "compare adopters to non-adopters who looked equally likely to adopt," which removes bias from the observed covariates driving self-selection.

Structured elaboration.

1. Covariate selection: include variables that plausibly affect both treatment and outcome (true confounders); avoid including anything downstream of treatment (mediators), which would bias the estimate the other direction.
2. Estimate the score: usually logistic regression, though any well-calibrated classifier works; the goal is a good conditional-probability estimate, not necessarily a high-accuracy classifier.
3. Choose a matching algorithm and choose among them deliberately: nearest-neighbor matching (pair each treated unit to its closest control) is simple but can leave large gaps if matches are poor; caliper matching adds a maximum-distance threshold so bad matches are dropped rather than forced; kernel matching uses a weighted average of nearby controls instead of a single match, trading some bias for lower variance. As a rule of thumb, use caliper matching when you have plenty of controls and want to guard against poor matches, and kernel matching when treated units are scarce and you can't afford to discard many.
4. Check match quality: compare standardized mean differences (SMD) on every covariate before and after matching; SMD under roughly 0.1 on each covariate is a common bar for "adequately balanced."
5. Estimate the average treatment effect on the treated (ATT) and its uncertainty: compute the outcome difference across matched pairs, and use a bootstrap or a matching-aware variance estimator (naive standard errors understate uncertainty because they ignore the matching step).

Worked example. Estimating the effect of a premium onboarding flow on 30-day revenue: fit a logistic regression for flow-enablement using tenure, acquisition channel, prior spend, and device type; match enablers to non-enablers with the closest propensity score within a 0.05 caliper; check that the SMD on prior spend, the covariate most likely to dominate self-selection, drops from roughly 0.4 pre-match to under 0.1 post-match; compute the ATT as the average revenue difference across the ~1,800 matched pairs that survived the caliper, with a bootstrap confidence interval.

Trade-offs and pitfalls. Two practical failure modes recur: first, PSM only balances covariates you actually measured, so a strong unmeasured confounder (say, marketing intent that never got logged) leaves the estimate biased no matter how tight the matches look; a sensitivity analysis for unobserved confounding is the honest follow-up. Second, discarding unmatched units (poor overlap, low prior spend or high tenure with no comparable match) changes the population the estimate applies to, from "everyone" to "the matchable subpopulation," which should be stated explicitly rather than glossed over.