Causal Inference Questions

Direct answer. Extrapolating a short experiment's immediate engagement lift to a 6-month retention outcome relies on surrogate endpoints: short-term metrics that are believed to be reliable proxies for the long-term outcome you actually care about. The extrapolation is only as trustworthy as the evidence that the short-term surrogate reliably predicts the long-term outcome for THIS specific intervention, which needs to be checked, not assumed by default.

Structured elaboration. Key assumptions and how to check them:

• Surrogate validity: has this same short-term metric reliably predicted 6-month retention for OTHER similar past interventions? If the company has a history of comparable launches with both short-term and eventual long-term data, checking whether the historical relationship between the two is stable (and not itself confounded by novelty effects) is the strongest available evidence.
• No divergent long-run dynamics: an intervention that produces an immediate engagement spike from novelty, but nothing sustainable, would show a strong short-term surrogate signal that doesn't hold up at 6 months; distinguishing "durable engagement change" from "novelty spike" from the short-term data alone is genuinely hard, which is exactly why extrapolation carries real risk.
• Attrition: if the underlying experiment population itself experiences differential attrition (dropout, churn, or opt-out) between treatment and control before 6 months, that further complicates a clean extrapolation, since the population you're extrapolating for shrinks and changes composition unevenly across arms.

Estimation methods to combine with the surrogate. Difference-in-differences on the intermediate metric trajectory (does the treatment-control GAP grow, stay flat, or shrink over the weeks available so far, which hints at whether it's durable or decaying); survival analysis if retention is naturally modeled as time-to-churn, which handles partial/incomplete follow-up more honestly than a simple binary retained/churned cutoff at 6 months; and uplift modeling, if you specifically want to identify which SEGMENTS show a durable versus a decaying pattern, since the extrapolation risk often isn't uniform across the whole population.

Worked example. A feature shows a 15% week-1 engagement lift. If the treatment-control engagement gap for a comparable past launch was still roughly 60% of its original size by month 3 and roughly 40% by month 6 (a decaying but real and durable pattern, not a full novelty-effect collapse to zero), that historical decay curve is a defensible basis for a conservative 6-month retention extrapolation; without that kind of prior evidence, an honest answer states the extrapolation is speculative and recommends running the experiment longer, or accepting the uncertainty explicitly in the launch decision.

Trade-offs and pitfalls. The single most common mistake is presenting an extrapolated long-term number with the same confidence as a directly measured one; the surrogate-validity evidence (or lack of it) should travel WITH the extrapolated estimate, not get lost once it's summarized in a slide.

Direct answer. Validating that a deployed model, rather than something else, caused a KPI change means you need a comparison point that shows what would have happened without the model, not just a before/after look at the metric. The most reliable way is a randomized holdout: keep a small, randomly-selected slice of traffic on the old system (or no model) and compare their outcomes to the rest.

Structured elaboration. When a true holdout wasn't run, the fallback is to rule out the obvious alternative explanations one at a time: check whether the KPI moved for a comparable segment that wasn't exposed to the model (a natural control group, if one exists); check the timing precisely, does the KPI inflect exactly at deployment, or does it start moving before or gradually drift, which would suggest something else; and check whether any other change (a concurrent marketing push, a seasonal effect, a separate product launch) coincided with the deployment window.

Worked example. After deploying a new ranking model, weekly conversion rose 3%. Before crediting the model: check a small percentage of users still served by the old ranking as an informal control (if they also show a 3% lift, something else, not the model, drove it); check the exact date the lift starts against the exact deployment date (a lift starting a week before deployment rules out the model as the cause); and check the launch calendar for other concurrent changes.

Trade-offs and pitfalls. A common mistake is treating "the metric improved after deployment" as sufficient evidence on its own; without a control group of some kind, even a well-timed inflection is still just correlational evidence, and the strength of the causal claim should be stated proportionate to how good the available control actually is.

Direct answer. Difference-in-differences (DiD) estimates a causal effect by comparing the change over time in a treated group to the change over the same period in an untreated (control) group, rather than comparing their levels directly. The key identifying assumption, parallel trends, is that absent the treatment, the treated and control groups would have moved in parallel (the same trend), so any divergence after treatment starts can be attributed to the treatment itself.

Structured elaboration. Formally, the DiD estimate is:

Two empirical checks for parallel trends before trusting this:

1. Pre-period trend plot: plot the outcome for both groups over several periods before treatment starts; if the two lines track each other closely (even if at different levels), that's supportive evidence. A visible divergence before treatment is a red flag that the groups were already on different paths.
2. Placebo (falsification) test: pretend the treatment started at an earlier, fake date within the pre-period, and re-run the DiD estimator; a non-zero "effect" at a date before treatment actually started suggests the parallel-trends assumption is shaky.

Worked example. A price change is implemented in Country A on a specific date, while comparable Country B has no change. Revenue in Country A was tracking closely with Country B for the six months before the change (supporting parallel trends). After the change, Country A's revenue growth diverges from Country B's by 4 percentage points; that divergence, not the raw before/after change in Country A alone, is the DiD estimate of the price change's effect, because it nets out whatever common trend (seasonality, macro conditions) both countries were already sharing.

Trade-offs and pitfalls. Parallel trends is an assumption about what would have happened, which can never be directly verified for the post-period; pre-trend similarity is suggestive, not proof. A second common pitfall: if the treated and control groups have different sizes, or the treatment effect is heterogeneous across adoption cohorts staggered over time (a related but distinct issue), the standard DiD regression can be biased even when parallel trends genuinely holds for each individual pair of periods.

Direct answer. Simulating a staggered-adoption setting shows that the standard two-way fixed-effects (two-way fixed effects (TWFE)) difference-in-differences (DiD) estimator can be badly biased, sometimes even the wrong sign, when treatment effects are heterogeneous across adoption cohorts, because TWFE implicitly uses ALREADY-treated units as part of the comparison group for later-treated units, contaminating the estimate with their own (different) treatment effect. An event-study specification with cohort-specific effects, or a stacked-DiD approach, avoids this by never comparing an already-treated unit's later periods against another cohort as if it were a valid control.

Approach. Simulate units adopting treatment at different times with a treatment effect that GROWS over cohorts (later cohorts get a larger true effect), fit plain TWFE, and separately fit a stacked-DiD estimator that only ever compares each treated cohort to not-yet-treated units.

import numpy as np
import pandas as pd
import statsmodels.formula.api as smf

def simulate_staggered_adoption(n_units=300, n_periods=10, seed=42, ramp=1.0):
    rng = np.random.default_rng(seed)
    adoption_period = rng.choice([3, 5, 7, np.inf], size=n_units, p=[0.25, 0.25, 0.25, 0.25])

    rows = []
    for i in range(n_units):
        unit_fe = rng.normal(0, 1)
        adopt_t = adoption_period[i]
        for t in range(n_periods):
            treated = t >= adopt_t
            rel_time = (t - adopt_t) if treated else 0.0
            # dynamic, GROWING effect the longer a unit has been treated: this is the specific
            # DGP shape that makes TWFE's implicit already-treated-as-control comparisons biased
            effect = ramp * rel_time if treated else 0.0
            y = unit_fe + 0.5 * t + effect + rng.normal(0, 0.5)
            rows.append({"unit": i, "t": t, "treated": int(treated), "adopt_t": adopt_t, "y": y, "rel_time": rel_time})
    return pd.DataFrame(rows)

def twfe_estimate(df):
    model = smf.ols("y ~ treated + C(unit) + C(t)", data=df).fit()
    return float(model.params["treated"])

def stacked_did_estimate(df):
    # for each finite-adoption cohort, build a clean sub-panel vs never-treated units only,
    # then pool and estimate once with cohort-specific time effects
    cohorts = sorted(c for c in df["adopt_t"].unique() if np.isfinite(c))
    stacks = []
    for stack_id, adopt_t in enumerate(cohorts):
        sub = df[(df["adopt_t"] == adopt_t) | (df["adopt_t"] == np.inf)].copy()
        sub["stack"] = stack_id
        stacks.append(sub)
    stacked = pd.concat(stacks, ignore_index=True)
    stacked["unit_stack"] = stacked["unit"].astype(str) + "_" + stacked["stack"].astype(str)
    model = smf.ols("y ~ treated + C(unit_stack) + C(t):C(stack)", data=stacked).fit()
    return float(model.params["treated"])

df = simulate_staggered_adoption()
twfe_result = twfe_estimate(df)
stacked_result = stacked_did_estimate(df)

# ground truth: the true average treatment effect on the treated (ATT) is the average of the dynamic effect actually realized
# across all treated unit-periods (known exactly here because we generated the data)
true_att = float((1.0 * df.loc[df["treated"] == 1, "rel_time"]).mean())
print({"true_att": true_att, "twfe_estimate": twfe_result, "stacked_did_estimate": stacked_result})

Running this (VERIFIED by execution, seed=42) prints {'true_att': 2.333, 'twfe_estimate': 0.925, 'stacked_did_estimate': 2.050}: TWFE recovers less than 40% of the true average effect on the treated, while the stacked estimator recovers roughly 88% of it, on this specific dynamic-effect DGP. The bias is largest precisely when the effect GROWS over time since adoption, since that is exactly the situation where an already-treated unit's later, larger effect contaminates its use as an implicit control for a later-adopting cohort.

Key points. TWFE implicitly weights each cohort-pair comparison by group size and variance in a way that can assign NEGATIVE weight to some genuinely-positive comparisons when effects are heterogeneous across cohorts and adoption timing, which is the mathematical root of the bias; the stacked-DiD approach sidesteps this by constructing separate treated-versus-never-treated comparisons for each cohort and combining them, so a later, already-treated cohort is never used as an implicit control for an earlier one.

Complexity. The stacked approach's unit_stack interaction can materially inflate the number of fixed-effect parameters (each unit appears once per cohort-stack it participates in as a control), which is the practical cost of the correction; for large panels, this can noticeably increase compute versus plain TWFE.

Edge cases. With very few never-treated units (or none, an "all eventually treated" panel), the stacked-DiD approach as written needs a different comparison group (using not-yet-treated units at each specific point in event time, rather than a permanently never-treated group), which is the more general Callaway-Sant'Anna-style construction; this simulation's simplification (a dedicated never-treated group) works cleanly for this demonstration but would need generalizing for a real, fully-staggered panel with no never-treated units at all.

Direct answer. Off-policy (counterfactual) evaluation estimates how a NEW policy would have performed, using only data logged under an OLD policy, without deploying the new policy live; the main families of estimators are importance-weighting approaches (reweighting logged outcomes by how differently the new policy would have acted compared to the old one), direct/model-based approaches (predicting the new policy's outcome from a learned reward model), and doubly robust approaches (combining both, for the same robustness benefit as in the treatment-effect setting).

Structured elaboration.

• Importance-weighting family (inverse propensity scoring and its variants): reweight each logged (context, action, reward) triple by the ratio of the new policy's probability of taking that action to the old (logging) policy's probability of having taken it; this is unbiased under the right assumptions but can have very high variance when the two policies disagree substantially on which actions to take, since the ratio can become extreme. A self-normalized variant reduces this variance at the cost of a small bias, a common and often worthwhile trade in practice.
• Direct/model-based method: train a model predicting reward from context and action using the logged data, then use it to predict what the new policy's expected reward would be; lower variance than importance weighting, but biased if the reward model is misspecified, and it can look deceptively confident even when wrong.
• Doubly robust estimator: combines both, consistent if either the importance weights or the reward model is correctly specified, generally the preferred choice when feasible.

Where each can fail. Importance-weighting-based methods fail when the logging policy has essentially zero probability of ever taking an action the new policy strongly prefers (no support for that action in the logged data at all, an overlap violation exactly analogous to poor propensity-score overlap in the observational treatment-effect setting); the direct method fails silently when the reward model is systematically wrong in a way that isn't obvious from its training performance; and ALL of these methods degrade when non-stationarity means the logged data no longer reflects current user behavior (the world the old policy was logged in has meaningfully changed by the time you're evaluating the new one), since the entire framework assumes the logging data still represents the current population and context distribution.

Trade-offs and pitfalls. Off-policy evaluation is attractive because it avoids the cost and risk of a live deployment for the new policy, but it should be treated as a SCREENING step, ruling out clearly bad candidate policies cheaply, rather than a substitute for a final live validation before full rollout, precisely because the overlap and non-stationarity failure modes can silently produce a confidently wrong estimate that looks reasonable.