Causal Inference Questions

Direct answer. Average Treatment Effect (ATE) is the average effect of the treatment if everyone in the population received it, compared to if no one did. Average Treatment Effect on the Treated (ATT) is the average effect specifically among the people who actually received the treatment, comparing their outcome to what it would have been had they not received it. For a feature only 10% of users adopt spontaneously, ATT is almost always the more useful number for a promotion or rollout decision, because it answers "what did the treatment do for the people who took it" rather than "what would it do if we forced it on the 90% who currently choose not to," and those two groups may respond very differently.

Structured elaboration. ATE and ATT coincide when treatment assignment is random (a coin flip doesn't correlate with who benefits more), but diverge whenever adoption is self-selected, which is the normal case for an optional feature: users who adopt spontaneously often differ systematically from those who don't, in ways that plausibly correlate with how much the feature helps them. If you're deciding whether to keep promoting the feature to the same self-selecting population, ATT tells you what it's actually doing for them. If you're deciding whether to force the feature on the 90% who haven't adopted, neither ATE nor ATT alone answers that; you'd want the effect specifically among the non-adopters (sometimes called ATU, the average treatment effect on the untreated), since they may be the low performers who chose not to adopt for reasons that make it less likely to help them too.

Worked example. In-app notification digests are adopted spontaneously by 10% of users. Among adopters, weekly sessions are 12% higher than a matched estimate of what they'd be without the digest (that's the ATT). If you're deciding "should we keep this feature and keep letting interested users find it," ATT of +12% directly answers that. If you're deciding "should we auto-enable it for the other 90%," ATT tells you nothing reliable about that group, since they self-selected out, possibly because it wouldn't help them as much.

Trade-offs and pitfalls. A common mistake is quoting an ATT from an opt-in population as if it were the ATE that would apply if the feature were force-enabled for everyone; the two can differ substantially, and reporting one as if it were the other overstates the expected impact of a broader rollout.

Direct answer. When a randomized experiment isn't possible, the right identification strategy depends on what natural variation exists in your data: instrumental variables (IV) need something that shifts treatment as-if-randomly without directly affecting the outcome; regression discontinuity (RDD) needs a sharp eligibility cutoff (a score, date, or threshold); difference-in-differences (DiD) needs a group that got treated and a comparable group that didn't, observed both before and after; and synthetic control needs a single treated unit (a market, a city) and enough untreated comparison units to construct a synthetic counterfactual from a weighted combination of them.

Structured elaboration.

Worked example. A pricing change rolls out in one country but not a comparable neighboring one at the same time: DiD is a natural first choice if pre-trends are parallel. If instead the price only applies above a specific score threshold used for a discount, RDD is the natural fit. If neither a clean control group nor a sharp cutoff exists, but you have a plausible instrument (say, a policy that shifted eligibility in a way unrelated to the outcome except through treatment), IV becomes the option, with instrument strength as the make-or-break diagnostic. If you have one affected market and many unaffected but comparable markets with rich pre-period history, synthetic control lets you construct a tailored counterfactual for that one market.

Trade-offs and pitfalls. Every one of these methods can produce a confident-looking point estimate that's wrong if its core assumption fails silently; the discipline that separates a credible quasi-experimental analysis from a plausible-looking one is running the specific falsification check for the specific method (parallel pre-trends for DiD, first-stage strength for IV, the manipulation test for RDD, pre-period fit for synthetic control) and being willing to report "I don't trust this identification" when the check fails. When several weak signals point the same direction, and no single design is airtight, combining evidence from multiple small experiments and observational analyses, and being explicit about how much you trust each, is often more honest than picking one method and reporting it as definitive.

Direct answer. Measurement error in a confounder or the treatment adds noise that generally biases a causal effect estimate toward zero (attenuation bias) when the error is classical (random, unrelated to the true value), but can bias the estimate in either direction, and in unpredictable ways, when the error is systematic (correlated with the true value or with other variables in the model).

Structured elaboration. Classical measurement error means the observed value equals the true value plus independent random noise; when this happens in the TREATMENT variable, the estimated coefficient on treatment is attenuated (pulled toward zero) in a mathematically predictable way, because the noisy treatment variable is a weaker proxy for the true treatment than the true treatment itself would be. When measurement error occurs in a CONFOUNDER instead, the confounder is only partially "controlled for" (the adjustment is diluted by the noise), leaving RESIDUAL confounding in the treatment estimate even after nominally controlling for it, which can bias the treatment effect estimate in either direction depending on the confounder's true relationship with treatment and outcome. Systematic measurement error, where the error itself correlates with the true value or with something else in the model (a device that consistently under-reports for heavy users, say), doesn't follow the clean "attenuation toward zero" pattern at all, and can inflate, deflate, or even flip the sign of an estimate.

Mitigation approaches.

1. Validation data: if a subset of observations has both the noisy and a "gold standard" accurate measurement, that subset can be used to estimate and correct for the measurement error's structure (a validation or calibration study).
2. Instrumental variable for the mismeasured variable: an instrument correlated with the TRUE value of the mismeasured variable but uncorrelated with the measurement error itself can recover a consistent estimate, using the same 2SLS logic as ordinary IV estimation.
3. Regression calibration: model the relationship between the noisy proxy and the true value (using external or partial validation data) and use the predicted "de-noised" value in place of the raw noisy measurement in the main analysis.

Worked example. Ad click tracking on iOS undercounts clicks due to known tracking loss (attribution restrictions), and the undercounting rate isn't uniform, it's worse for privacy-conscious users, who may also differ systematically in purchase behavior. This is closer to systematic than purely classical error, which means a simple "the estimate is probably attenuated, so the true effect is somewhat larger" adjustment isn't safely applicable; the more defensible response is either validation data (comparing tracked clicks to a platform-reported ground truth for a subsample) or an alternative measurement channel less affected by the same tracking loss.

Trade-offs and pitfalls. The comforting rule "measurement error just attenuates toward zero, so my true effect is probably even bigger" only applies cleanly to classical error in the treatment variable; applying that same comforting logic to systematic error, or to measurement error in a confounder, is a common and consequential overreach.

Direct answer. For each unit, the potential-outcomes framework posits two potential outcomes: $Y_i(1)$, what would happen to unit $i$ if treated, and $Y_i(0)$, what would happen if not treated. We can never observe both for the same unit at the same time (a user either sees the feature or doesn't), which is why this is called the "fundamental problem of causal inference": the individual causal effect $Y_i(1) - Y_i(0)$ is never directly observable, only estimable in aggregate under the right conditions.

Structured elaboration. This framework guides hypothesis formation by forcing precision: instead of vaguely asking "does the feature help retention," you're asking "what is $E[Y_i(1) - Y_i(0)]$" for a specific population, which immediately raises the right follow-up questions (which population? averaged how?). Randomization is what makes this estimable: by assigning units to treatment or control via a coin flip, the observed outcome for treated units, on average, stands in for the unobservable $Y_i(1)$ across the whole population, and the observed outcome for control units stands in for $Y_i(0)$, because the coin flip ensures neither group is systematically different in any way related to the potential outcomes themselves.

Worked example. For a checkout redesign, unit $i$'s potential outcomes are "would $i$ convert if shown the new design" ($Y_i(1)$) and "would $i$ convert if shown the old design" ($Y_i(0)$). We only ever observe one of these per user. Randomly assigning 50% of users to each version means the observed conversion rate in the treated group is an unbiased estimate of $E[Y_i(1)]$ across the whole randomized population, and the control group's rate estimates $E[Y_i(0)]$; their difference estimates the average treatment effect, even though no single user's individual effect was ever directly measured.

Trade-offs and pitfalls. The framework's power is also its limit: it formalizes exactly what causal inference can and can't claim, an AVERAGE effect across a population, not a guarantee about any specific individual, which is worth stating explicitly when someone asks "so did the feature help ME" from a group-level result.

Direct answer. Compute a difference-in-differences estimate in SQL using indicator columns for treatment group and post-period, an interaction term for the DiD coefficient itself, and cluster standard errors at the unit level the treatment was actually assigned at, here the store or region level, rather than assuming each row is an independent observation.

Approach. Build a flags table joining users to their treatment status and to a pre/post indicator relative to the treatment start date, aggregate to a group-period level, then compute the DiD estimate directly (or export the group-period panel to a regression tool that supports clustered standard errors, since raw SQL alone doesn't compute a clustered variance estimate).

-- 1. Build user-level treatment flag (assumes a separate treatment_users(user_id) table, or a cohort column)
WITH flagged AS (
  SELECT
    m.user_id,
    u.cohort,
    (u.cohort = 'treatment') AS is_treated,
    (m.event_date >= DATE '2025-01-01') AS is_post,
    m.conversion
  FROM metrics m
  JOIN users u ON u.user_id = m.user_id
),

-- 2. Aggregate to the four DiD cells: treated x post, treated x pre, control x post, control x pre
cell_means AS (
  SELECT
    is_treated,
    is_post,
    AVG(CASE WHEN conversion THEN 1.0 ELSE 0.0 END) AS conv_rate,
    COUNT(*) AS n
  FROM flagged
  GROUP BY is_treated, is_post
)

-- 3. Compute the DiD point estimate directly from the four cell means
SELECT
  (MAX(CASE WHEN is_treated AND is_post THEN conv_rate END)
   - MAX(CASE WHEN is_treated AND NOT is_post THEN conv_rate END))
  -
  (MAX(CASE WHEN NOT is_treated AND is_post THEN conv_rate END)
   - MAX(CASE WHEN NOT is_treated AND NOT is_post THEN conv_rate END))
  AS did_estimate
FROM cell_means;

For clustered standard errors: export a store-period-level panel (store_id, is_post, conv_rate) and run the regression conv_rate ~ is_treated + is_post + is_treated:is_post with standard errors clustered on store_id in a stats tool (statsmodels' cov_type='cluster', or R's feols with a cluster argument); the interaction coefficient IS the DiD estimate, and the clustered standard error accounts for the fact that observations within the same store are correlated over time, which a naive standard error would understate.

Key points. The DiD estimate is exactly the double-difference of the four cell means; the SQL above computes it directly, but does NOT produce a valid standard error, clustering has to happen in a regression framework that supports a specified clustering variable, not in raw aggregate SQL.

Complexity. The aggregation is a single pass over the metrics table, $O(n)$; the join to users is $O(n)$ with an index on user_id.

Edge cases. If treatment and control groups have very different sizes (as with a rollout to one store versus 50 stores), the four cell means are still valid, but the DiD's precision depends heavily on the smaller group's sample size; unequal date ranges of data availability for treated versus control groups (one store launched later, so its pre-period is shorter) would bias the pre-period cell mean if not handled by restricting all groups to a common, symmetric pre/post window around the treatment date.