Causal Inference Questions

Direct answer. Collider bias is a spurious association created by conditioning on (filtering by, or controlling for) a variable that is caused by both of the variables you're studying. It's different from confounding in the direction of the arrows: a confounder causes both variables and biases you if you DON'T adjust for it; a collider is caused BY both variables and biases you if you DO condition on it (or implicitly select on it).

Structured elaboration. The mechanism: if $A$ and $B$ are truly independent, but both influence $C$, then within any fixed value of $C$, learning something about $A$ tells you something about $B$ (because if $C$ is fixed and $A$ was unusually low, $B$ must have been unusually high to still produce that value of $C$). This creates a correlation between $A$ and $B$ that exists ONLY because you restricted the sample by $C$.

Worked example. Suppose a company evaluates employees on both technical skill and communication skill, and only promotes people who score highly on at least one (promotion is the collider, caused by both skills). If you then look only at promoted employees and ask "is technical skill correlated with communication skill," you may find a NEGATIVE correlation, even if the two skills are genuinely independent in the full employee population: among promoted people, someone with mediocre technical skill almost certainly needed strong communication skill to get promoted (and vice versa), which mechanically induces the negative relationship purely from the promotion filter.

Diagnostics and design choices. To detect it: ask whether your dataset is implicitly filtered on something that both your variables of interest could plausibly influence (application funnels, "reached checkout," "was retained," are common colliders in product analytics). To avoid it: analyze the full population rather than a filtered subpopulation when possible, or explicitly model the selection mechanism (inverse-probability-of-selection weighting) if you must work with filtered data.

Trade-offs and pitfalls. Collider bias is counterintuitive precisely because "controlling for more variables" is usually good instinct in modeling; here, the instinct actively backfires, and the fix is knowing which variables NOT to touch, which is why sketching the causal structure before deciding what to include matters as much for colliders as it does for confounders.

Direct answer. Selection bias occurs when the process that determines who ends up in your sample (or who ends up treated) is itself related to the outcome, producing a non-representative sample; confounding occurs when a third variable independently affects both treatment and outcome. They often co-occur but are distinct mechanisms: selection bias is about who you observe or compare, confounding is about a shared cause distorting a comparison you can otherwise observe cleanly.

Structured elaboration and worked example. A company surveys customers who are still active to ask about satisfaction with a support interaction; customers who had a bad experience and churned are systematically excluded from the sample (they're gone), so the surviving sample is non-representative, satisfaction estimates from it will look better than the true population average, purely from who's left to survey. This is selection bias, distinct from a confounder like "account tenure" that might separately bias a comparison between two support channels within the surviving sample.

Correction methods. Weighting methods (like the inverse probability weighting used for selection above) can correct for selection ON OBSERVED characteristics that predict both selection and the outcome. The Heckman correction is a classic two-stage approach specifically designed for this shape of problem: model the selection process explicitly (a "selection equation" predicting who ends up in the sample) alongside the outcome equation, and use the selection model to correct the outcome estimate for the resulting non-random sample.

When selection bias is impossible to fully correct. If the very thing that determines selection is unmeasured and strongly related to the outcome (churned customers had systematically different reasons for churning that you have no proxy for), no reweighting or Heckman-style correction can fix it, because both require some observed variable that predicts selection well enough to model it. In that case the honest conclusion is that the estimate is only valid for the surviving/selected population, not the true target population, and that limitation should be stated rather than papered over with a correction that assumes more than the data supports.

Trade-offs and pitfalls. Reaching for "Heckman correction" as a default fix without checking whether its own exclusion-restriction-style assumption (a variable that predicts selection but not the outcome directly) is actually satisfiable is a common overreach; when no such variable exists, the correction is formally identified but practically fragile.

Direct answer. An instrumental variable (IV) is a variable that shifts treatment without having any other path to the outcome, letting you extract a causal effect even when unobserved confounding makes a direct comparison untrustworthy. For it to be valid, it needs three properties: it must actually move the treatment (relevance), it must affect the outcome only through the treatment and not through any other channel (the exclusion restriction), and it must not itself be driven by the same unobserved factors that drive the outcome (exchangeability, sometimes called as-if-random assignment of the instrument).

Structured elaboration and candidate instruments.

• Server-load-driven delay in sending a reminder email: the delay is caused by infrastructure load, plausibly unrelated to a specific user's purchase propensity (supports exchangeability), it clearly shifts whether/when a user receives the reminder (relevance), and as long as the delay itself has no other channel to affect purchases besides through receiving the email later (exclusion), it's a workable instrument. The risk to check: does high server load correlate with high traffic days that independently affect purchase behavior?
• A policy change that shifted eligibility as-if-randomly: for example, a threshold that changed for reasons unrelated to individual user behavior. Relevance is clear if eligibility strongly predicts uptake; exclusion requires that the policy change didn't also coincide with some other simultaneous change affecting the outcome.
• Geographic or time-based rollout staggering driven by operational constraints (which warehouse or region got a new feature first, for reasons like engineering capacity rather than user characteristics): relevance holds if the rollout order predicts exposure; exchangeability requires the rollout order wasn't itself chosen based on expected performance in that region.

For estimation, two-stage least squares (2SLS) regresses the treatment on the instrument (and any exogenous controls) in the first stage, then regresses the outcome on the predicted treatment from that first stage. The coefficient on predicted treatment is the IV estimate.

Worked example. Formally, with instrument $Z$, treatment $D$, and outcome $Y$:

$$\hat{D} = \hat{\pi}_0 + \hat{\pi}_1 Z \quad \text{(first stage)}$$ $$Y = \beta_0 + \beta_1 \hat{D} + \varepsilon \quad \text{(second stage)}$$

The first-stage F-statistic on $Z$ is the standard instrument-strength diagnostic; a common rule of thumb flags instruments weaker than roughly F=10 as unreliable, since a weak first stage amplifies any small violation of exclusion into a large bias in $\beta_1$.

Trade-offs and pitfalls. Exclusion is the assumption that's almost never directly testable, since it requires arguing there's no OTHER path from the instrument to the outcome, which is a claim about the world rather than something the data alone can confirm; the strongest IV answers name the specific alternate channel a skeptic would raise and explain why it's implausible, rather than asserting exclusion by fiat.

Direct answer. Check overlap (also called common support) by comparing the distributions of propensity scores between the treated and control groups. If both groups span a similar range of scores, with enough control units near every treated unit's score and vice versa, comparisons are trustworthy. If treated units cluster at scores where almost no control units exist (or the reverse), the propensity model is effectively extrapolating, and the resulting comparison isn't supported by real data.

Structured elaboration. Two practical checks: (1) plot the propensity score distributions for both groups on the same axis and visually inspect where they do and don't overlap; (2) after matching, report how many treated units were dropped for lack of a close enough control match (a high drop rate is itself a symptom of poor overlap). A stricter numeric check is to look at the propensity score range where both groups have reasonable density, sometimes trimmed to, say, the 5th to 95th percentile of the overlapping region, and restrict the analysis to that region rather than the full sample.

Worked example. Suppose you're comparing users who enabled a feature (treated) against those who didn't (control) using a propensity model built from tenure and engagement. If nearly all treated users have propensity scores between 0.6 and 0.95, but 90% of control users have scores under 0.3, only a thin sliver of control users (those with scores above 0.6) is actually comparable to the treated group; the average treatment effect on the treated (ATT) estimate would then only be credible for that sliver, not for the typical treated user.

Trade-offs and pitfalls. It's tempting to fix poor overlap by widening the caliper or accepting worse matches, but that just trades an honest "we can't estimate this for the full population" for a quietly biased estimate. The more defensible response is to redefine the estimand (report the ATT only for the region of common support, and say so explicitly) or collect more data in the poorly-overlapped region.

Direct answer. Extrapolating a short experiment's immediate engagement lift to a 6-month retention outcome relies on surrogate endpoints: short-term metrics that are believed to be reliable proxies for the long-term outcome you actually care about. The extrapolation is only as trustworthy as the evidence that the short-term surrogate reliably predicts the long-term outcome for THIS specific intervention, which needs to be checked, not assumed by default.

Structured elaboration. Key assumptions and how to check them:

• Surrogate validity: has this same short-term metric reliably predicted 6-month retention for OTHER similar past interventions? If the company has a history of comparable launches with both short-term and eventual long-term data, checking whether the historical relationship between the two is stable (and not itself confounded by novelty effects) is the strongest available evidence.
• No divergent long-run dynamics: an intervention that produces an immediate engagement spike from novelty, but nothing sustainable, would show a strong short-term surrogate signal that doesn't hold up at 6 months; distinguishing "durable engagement change" from "novelty spike" from the short-term data alone is genuinely hard, which is exactly why extrapolation carries real risk.
• Attrition: if the underlying experiment population itself experiences differential attrition (dropout, churn, or opt-out) between treatment and control before 6 months, that further complicates a clean extrapolation, since the population you're extrapolating for shrinks and changes composition unevenly across arms.

Estimation methods to combine with the surrogate. Difference-in-differences on the intermediate metric trajectory (does the treatment-control GAP grow, stay flat, or shrink over the weeks available so far, which hints at whether it's durable or decaying); survival analysis if retention is naturally modeled as time-to-churn, which handles partial/incomplete follow-up more honestly than a simple binary retained/churned cutoff at 6 months; and uplift modeling, if you specifically want to identify which SEGMENTS show a durable versus a decaying pattern, since the extrapolation risk often isn't uniform across the whole population.

Worked example. A feature shows a 15% week-1 engagement lift. If the treatment-control engagement gap for a comparable past launch was still roughly 60% of its original size by month 3 and roughly 40% by month 6 (a decaying but real and durable pattern, not a full novelty-effect collapse to zero), that historical decay curve is a defensible basis for a conservative 6-month retention extrapolation; without that kind of prior evidence, an honest answer states the extrapolation is speculative and recommends running the experiment longer, or accepting the uncertainty explicitly in the launch decision.

Trade-offs and pitfalls. The single most common mistake is presenting an extrapolated long-term number with the same confidence as a directly measured one; the surrogate-validity evidence (or lack of it) should travel WITH the extrapolated estimate, not get lost once it's summarized in a slide.