Causal Inference Questions

Direct answer. The implementation fits stabilized inverse-probability-of-treatment weights from the given propensity scores, computes a weighted mean difference as the average treatment effect (ATE), clips extreme weights to control variance, and returns a bootstrap 95% confidence interval.

Approach. Stabilized weights use the marginal treatment probability in the numerator (rather than 1), which keeps the weight distribution tighter than unstabilized weights while preserving the same expectation; clipping bounds how much any single observation can dominate the weighted average.

import numpy as np

def ipw_ate(treatment, outcome, propensity, clip=0.01, n_boot=1000, seed=42):
    treatment = np.asarray(treatment, dtype=float)
    outcome = np.asarray(outcome, dtype=float)
    propensity = np.clip(np.asarray(propensity, dtype=float), clip, 1 - clip)

    def stabilized_ate(t, y, ps):
        p_treat = t.mean()  # marginal probability of treatment, for stabilization
        w1 = p_treat / ps
        w0 = (1 - p_treat) / (1 - ps)
        weighted_mean_treated = np.sum(t * w1 * y) / np.sum(t * w1)
        weighted_mean_control = np.sum((1 - t) * w0 * y) / np.sum((1 - t) * w0)
        return weighted_mean_treated - weighted_mean_control

    ate = stabilized_ate(treatment, outcome, propensity)

    rng = np.random.default_rng(seed)
    n = len(treatment)
    boot_ates = np.empty(n_boot)
    for b in range(n_boot):
        idx = rng.integers(0, n, size=n)
        boot_ates[b] = stabilized_ate(treatment[idx], outcome[idx], propensity[idx])

    ci_low, ci_high = np.percentile(boot_ates, [2.5, 97.5])
    se = float(np.std(boot_ates, ddof=1))
    return {"ate": float(ate), "se": se, "ci_95": (float(ci_low), float(ci_high))}

Key points. Clipping propensity scores to [clip, 1-clip] before computing weights, rather than clipping the weights directly, keeps the stabilization numerator and denominator consistent; the bootstrap resamples the ORIGINAL (treatment, outcome, propensity) triples together, preserving each unit's own propensity score rather than resampling propensity scores independently, which would break the correlation structure the estimator depends on.

Complexity. $O(n)$ per bootstrap resample and $O(n \cdot n_{boot})$ overall, dominated by the bootstrap loop; for large $n$ and $n_{boot}=1000$, this is the practical bottleneck rather than the single weighted-mean computation.

Edge cases. If clip is set too small (say, 0.0001) on a dataset with genuinely near-0 or near-1 propensity scores, individual weights can still be extreme enough to make a handful of observations dominate the estimate; a diagnostic worth running alongside this function is the effective sample size after weighting ($\left(\sum w_i\right)^2 / \sum w_i^2$), which drops sharply when a few large weights are doing most of the work. An all-treated or all-control bootstrap resample (possible by chance with small $n$) would produce a degenerate stabilized_ate call; for small samples, checking that both groups are non-empty in each resample is worth adding.

Direct answer. SHAP values and feature importance describe which features a MODEL relied on to make its predictions, which is a statement about the model's learned associations, not about what actually causes the outcome in the real world; treating them as causal hypotheses risks two specific pitfalls: confounded features looking important purely because they correlate with the true driver, and features that are themselves DOWNSTREAM of the outcome (leakage) appearing highly "important" for entirely non-causal reasons.

Structured elaboration and mitigations.

1. Confounded importance: a feature can show high SHAP importance simply because it correlates strongly with the true causal driver, without itself having any causal effect (the same shared-confounder logic as any correlation-versus-causation trap, now operating inside a model's learned weights). Mitigation: treat a high-importance feature as a HYPOTHESIS to test, not a conclusion, and validate the top candidates with an actual causal method (an experiment, or an observational method with a defensible identification strategy) before acting on the "why" of a model failure.
2. Leakage-driven importance: a feature that's actually a consequence of the outcome (or of something downstream of it) can show artificially high importance if it snuck into the training data, which reflects a data-pipeline bug more than any real signal, causal or otherwise. Mitigation: for any surprisingly high-importance feature, explicitly check its timing and causal plausibility (could this feature have been observed BEFORE the outcome occurred, given how the pipeline actually works) before trusting it as even a correlational signal, let alone a causal one.

Worked example. A model failing to correctly classify a subset of transactions shows high SHAP importance for "customer support contacts in the last 7 days." Interpreted causally, this might suggest customer support behavior is somehow driving the model's failures. The more likely explanation, worth checking before acting: customers who experienced the SAME underlying issue causing the model to fail also tend to contact support (a shared cause, not support contacts causing the failure), or worse, the support-contact feature was logged partially AFTER the failure event, making it a leakage artifact rather than a real predictive signal at all.

Trade-offs and pitfalls. SHAP and feature importance remain genuinely useful for GENERATING causal hypotheses efficiently, scanning many features for candidates worth investigating further is faster than brainstorming from scratch, the mistake is treating the output as the answer rather than as the starting point for a separate causal validation step.

Direct answer. An interrupted time series (ITS) design compares the trajectory of a metric before and after an intervention that applied to everyone at once (so there's no control group), modeling the pre-period trend and level, then testing whether the actual post-period trajectory diverges from what that pre-period trend would have predicted. The break at the intervention date, both a level shift and/or a slope change, is the estimated effect.

Structured elaboration. Because there's no control group, the entire causal claim rests on correctly modeling what the trend WOULD have looked like without the intervention, which makes trend and seasonality adjustment the core of the design rather than an afterthought. A typical specification: $Y_t = \beta_0 + \beta_1 t + \beta_2 \cdot \text{post}_t + \beta_3 \cdot t_{\text{since intervention}} + \varepsilon_t$, where $\beta_2$ captures an immediate level shift at the intervention and $\beta_3$ captures a change in slope afterward; seasonal terms (monthly or weekly dummies, or a Fourier-term representation) are added when the metric has known seasonal structure.

Diagnostics and sensitivity checks.

• Autocorrelation: time series residuals are rarely independent; check the residual autocorrelation function and, if needed, model residuals with an autoregressive term rather than assuming plain OLS-style independent errors, since ignoring autocorrelation understates the standard errors and overstates confidence in the break.
• Trend stability: fit the pre-period trend on a few different pre-period window lengths and check the estimated break is stable, not an artifact of exactly where the pre-period window starts.
• Placebo break dates: apply the same model with a FAKE break date somewhere in the pre-period; a spurious "significant break" at a date nothing happened is a red flag about the model's specification.

Worked example. For removing a free tier, daily active users (DAU) had been growing at roughly 0.3% per day for six months before the change. After the removal, the level drops sharply and the growth rate flattens to roughly 0.05% per day. The ITS model would show a significant negative $\beta_2$ (the immediate drop) and a significant negative $\beta_3$ (the growth-rate slowdown), separately quantifying the immediate shock from the longer-run trajectory change, which matters for deciding whether the effect is a one-time adjustment or a lasting structural shift.

Trade-offs and pitfalls. ITS versus difference-in-differences (DiD) is largely a data-availability question: use ITS when no comparable untreated group exists (a universal, simultaneous rollout), and prefer DiD whenever a genuine comparison group is available, since DiD's parallel-trends assumption is generally easier to defend than ITS's implicit assumption that the pre-period trend, once modeled, would have continued unchanged.

Direct answer. Causal forests and double/debiased machine learning (DML) both use flexible machine-learning models to estimate causal effects while avoiding the bias that naively plugging ML predictions into a causal estimate would introduce; causal forests directly target heterogeneous treatment effects (a tree-based method that splits on covariates specifically to maximize detected treatment-effect heterogeneity), while DML more generally uses ML models for nuisance functions (the propensity and outcome models) combined with an orthogonalized estimating equation that's robust to moderate errors in those nuisance models.

Structured elaboration. The key theoretical idea behind DML is Neyman orthogonality: constructing the estimating equation so that its sensitivity to SMALL errors in the nuisance function estimates is zero to first order, which is why flexible, regularized ML models (which have some estimation error by construction) can still be plugged in without that error directly biasing the causal estimate, as long as cross-fitting is used (fitting nuisance models on one fold, evaluating the causal estimate on a held-out fold) to prevent the same data from both fitting and evaluating the nuisance models.

Identification assumptions. Both methods still rely on the same fundamental assumptions as any observational causal estimate: unconfoundedness (all relevant confounders are measured and included as covariates) and overlap (every unit has a non-zero, non-one probability of treatment given its covariates); flexible ML modeling improves how well you can estimate the NUISANCE functions given the covariates you have, but it does nothing to address an unmeasured confounder that was never captured in the data at all.

Practical tuning and interpretation considerations. Causal forests need enough data per "leaf" (terminal split) to estimate a treatment effect reliably within it, which limits how fine-grained the heterogeneity detection can be with a modest sample; honest splitting (using separate subsamples to determine the tree structure versus to estimate effects within it) is what protects against the model finding spurious subgroup effects purely from overfitting the same data twice. Cross-fitting and sample splitting generally cost some statistical efficiency (using less data per step) in exchange for this robustness, a trade-off that's usually worth it once you're using flexible enough models that overfitting is a real risk.

Trade-offs and pitfalls. These methods are frequently mistaken for a way to "solve" confounding through modeling sophistication; they don't relax the unconfoundedness assumption at all, they only make the NUISANCE-function estimation step more flexible and less bias-prone than a naive plug-in approach, which is a meaningfully narrower (though still valuable) claim than ‘machine learning fixes causal inference.’

Direct answer. For each unit, the potential-outcomes framework posits two potential outcomes: $Y_i(1)$, what would happen to unit $i$ if treated, and $Y_i(0)$, what would happen if not treated. We can never observe both for the same unit at the same time (a user either sees the feature or doesn't), which is why this is called the "fundamental problem of causal inference": the individual causal effect $Y_i(1) - Y_i(0)$ is never directly observable, only estimable in aggregate under the right conditions.

Structured elaboration. This framework guides hypothesis formation by forcing precision: instead of vaguely asking "does the feature help retention," you're asking "what is $E[Y_i(1) - Y_i(0)]$" for a specific population, which immediately raises the right follow-up questions (which population? averaged how?). Randomization is what makes this estimable: by assigning units to treatment or control via a coin flip, the observed outcome for treated units, on average, stands in for the unobservable $Y_i(1)$ across the whole population, and the observed outcome for control units stands in for $Y_i(0)$, because the coin flip ensures neither group is systematically different in any way related to the potential outcomes themselves.

Worked example. For a checkout redesign, unit $i$'s potential outcomes are "would $i$ convert if shown the new design" ($Y_i(1)$) and "would $i$ convert if shown the old design" ($Y_i(0)$). We only ever observe one of these per user. Randomly assigning 50% of users to each version means the observed conversion rate in the treated group is an unbiased estimate of $E[Y_i(1)]$ across the whole randomized population, and the control group's rate estimates $E[Y_i(0)]$; their difference estimates the average treatment effect, even though no single user's individual effect was ever directly measured.

Trade-offs and pitfalls. The framework's power is also its limit: it formalizes exactly what causal inference can and can't claim, an AVERAGE effect across a population, not a guarantee about any specific individual, which is worth stating explicitly when someone asks "so did the feature help ME" from a group-level result.