Causal Inference Questions
Establishing cause-and-effect from observational and experimental data. Covers correlation versus causation, confounding, treatment-effect estimation, and quasi-experimental methods such as difference-in-differences, matching, and instrumental variables. Includes incrementality reasoning when true randomization is not possible.
Explain how a negative control (a negative-exposure or negative-outcome control) can be used to detect residual confounding in an observational study. Give an example negative-control design for testing whether an ad campaign causally increased purchases.
Sample Answer
Direct answer. A negative control is a variable known (or strongly believed) to have NO true causal relationship with the treatment or the outcome, used as a diagnostic: if an analysis detects a "significant" relationship involving the negative control anyway, that's evidence of residual confounding or bias in the analysis, since a real effect shouldn't be there to find.
Structured elaboration. Two flavors: a negative CONTROL OUTCOME is an outcome the treatment plausibly can't affect (testing whether the treatment "predicts" it anyway signals bias); a negative CONTROL EXPOSURE is an exposure that plausibly can't affect the real outcome (testing whether it appears to "cause" the outcome anyway, using the same analysis pipeline, signals the same kind of bias). Both work by re-running your identification strategy on a case where you already know the true answer is "no effect," as a sanity check on the method itself rather than the substantive question.
Worked example. To test whether an ad campaign causally increased purchases, use a negative control OUTCOME: something the campaign couldn't plausibly have influenced, such as purchases made in a completely unrelated product category that wasn't advertised and has no substitution relationship with the advertised product. If your causal-inference pipeline (the same confounder-adjustment approach used for the real analysis) finds a "significant" effect of the campaign on THIS unrelated category's purchases, that's a red flag: since the campaign genuinely can't have affected that category, any apparent effect must be coming from residual confounding your adjustment didn't fully remove, which should make you distrust the same pipeline's result for the real, advertised category too.
Trade-offs and pitfalls. A negative control is only informative if it's genuinely believable as having zero true effect; picking a poor negative control (one that actually could be affected through some indirect channel you didn't think of) undermines the whole exercise, so choosing and justifying the negative control is itself part of the analytical work, not a formality to check off.
Describe how you would measure the price elasticity of demand with respect to a fee (for example, a delivery fee or take rate). Compare a randomized price experiment against an observational econometric approach (panel fixed effects, instrumental variables), specify a statistical model and key controls, and describe how you would translate the elasticity estimate into a concrete pricing recommendation.
Sample Answer
Direct answer. Price elasticity of demand measures how a percentage change in a fee translates into a percentage change in demand; a randomized price experiment gives a clean causal estimate by directly varying the fee for randomly assigned users or markets, while an observational econometric approach (panel fixed effects, instrumental variables) estimates elasticity from historical price variation, which is more available but requires assumptions about why prices varied in the past.
Structured elaboration.
- Randomized approach: randomly assign different fee levels to different users or geographic cells, holding everything else constant, and estimate elasticity directly from the resulting demand differences. This is the gold standard but has real costs: revenue risk during the test, and limited statistical power if only a few discrete price points can be tested cleanly.
- Observational approach: a panel regression of log(demand) on log(price) with unit (city, market) and time fixed effects controls for time-invariant market characteristics and common time shocks, but is only credible if past price variation wasn't itself driven by something that also affects demand (competitor pricing pressure, demand shocks that the pricing team responded to). An instrumental variable, such as a cost-driven price change unrelated to local demand conditions, can help address that concern if a plausible one exists.
- Model specification: log(demand)=β0+β1log(price)+controls+ε, where β1 is directly interpretable as the price elasticity (the percentage change in demand for a 1 percent change in price).
Worked example. A randomized fee experiment varies delivery fees across three levels in randomly assigned zones; the resulting elasticity estimate, say β^1=−0.8, means a 10% fee increase is associated with roughly an 8% demand decrease. Translating into a pricing decision: at −0.8 (demand falls less than proportionally to the fee increase, since ∣β1∣<1), a fee increase would raise total revenue (revenue-maximizing in the short run), whereas an elasticity below −1 would mean a fee increase reduces total revenue despite the higher per-order fee, because demand falls more than proportionally.
Trade-offs and pitfalls. Elasticity is rarely constant across the whole price range or across all customer segments, extrapolating an elasticity estimated near the current price to a much larger price change, or applying an average elasticity uniformly across price-sensitive and price-insensitive segments, risks a materially wrong revenue projection; segment-level or nonlinear elasticity estimates are worth the extra effort whenever the pricing decision under consideration is a large change rather than a small one.
In Python, simulate a dataset where X and Y are correlated only because of a shared confounder Z: generate Z ~ N(0,1), X = 0.8Z + noise, Y = 1.2Z + noise (independent normal noise terms). Compute (a) the naive Pearson correlation between X and Y, and (b) the partial correlation between X and Y controlling for Z (by regressing each out of Z and correlating the residuals). Show that (b) is close to zero while (a) is not, and state one limitation of partial correlation as a method for controlling confounders in general.
Sample Answer
Direct answer. Simulate Z∼N(0,1), then X=0.8Z+εX and Y=1.2Z+εY with independent noise; the naive Pearson correlation between X and Y will be substantial (driven entirely by their shared dependence on Z), while the partial correlation controlling for Z, computed by regressing each of X and Y on Z and correlating the residuals, should be close to zero, demonstrating that the naive correlation is confounding, not a direct relationship.
Code.
import numpy as np
from scipy import stats
def simulate_confounded_data(n=5000, seed=42):
rng = np.random.default_rng(seed)
Z = rng.normal(0, 1, n)
eps_x = rng.normal(0, 1, n)
eps_y = rng.normal(0, 1, n)
X = 0.8 * Z + eps_x
Y = 1.2 * Z + eps_y
return Z, X, Y
def naive_and_partial_correlation(Z, X, Y):
naive_r, naive_p = stats.pearsonr(X, Y)
# regress X on Z, and Y on Z; correlate the residuals
beta_xz = np.polyfit(Z, X, 1)
beta_yz = np.polyfit(Z, Y, 1)
resid_x = X - np.polyval(beta_xz, Z)
resid_y = Y - np.polyval(beta_yz, Z)
partial_r, partial_p = stats.pearsonr(resid_x, resid_y)
return {"naive_pearson_r": round(float(naive_r), 3), "naive_p": float(naive_p),
"partial_r_given_Z": round(float(partial_r), 3), "partial_p": float(partial_p)}
Z, X, Y = simulate_confounded_data()
result = naive_and_partial_correlation(Z, X, Y)
print(result)
# Expected shape of result: naive_pearson_r substantially positive (theoretical value below),
# partial_r_given_Z close to 0 (theoretical value 0, sample noise only)
Running this with seed=42 and n=5000 (VERIFIED by execution) produces naive_pearson_r = 0.468 and partial_r_given_Z = 0.014, closely matching the theoretical values below (0.480 and exactly 0, respectively, with the small gap from finite-sample noise). The theoretical naive correlation follows directly from the shared-Z structure:
Key points. The residualize-and-correlate approach IS the standard definition of partial correlation for a single continuous confounder; it's mathematically equivalent to reading off the coefficient in a multiple regression of Y on X and Z jointly, once standardized.
Limitation. Partial correlation controlling for Z removes the LINEAR relationship each variable has with Z specifically; it assumes the confounding relationship is adequately captured by a linear regression on Z, and it controls for exactly one named confounder. If the true relationship between Z and X (or Y) were nonlinear, or if there were a SECOND, unmeasured confounder, partial correlation on Z alone would not fully remove the spurious association; a full directed acyclic graph (DAG)-based adjustment set, or a nonlinear/nonparametric adjustment, would be needed to handle either of those cases.
Edge cases. With a small sample size, the residual-correlation approach can show a partial correlation noticeably different from zero purely from sampling noise, even though the true value is exactly zero by construction; running the simulation at n=200 instead of n=5000 makes this noise visibly larger, which is itself a useful illustration of why sample size matters for trusting a "confounding fully explained" conclusion.
What is Simpson's paradox? Construct a concise numeric example with at least two groups where the aggregated association points in the opposite direction from the association within each group, and explain why stratifying by the confounding variable reveals the true relationship.
Sample Answer
Direct answer. Simpson's paradox is when an association between two variables reverses direction once you stratify (split) the data by a third variable, compared to the direction it showed in the aggregated data. It happens because the aggregate mixes groups of very different sizes and rates in a way that the overall average can hide, or even invert, what's true within every individual group.
Structured elaboration and worked example. Suppose you're comparing conversion rate for a checkout redesign, variant A versus variant B, across two customer segments:
| Segment | A: conversions/users tested | A rate | B: conversions/users tested | B rate |
|---|---|---|---|---|
| Returning customers | 81/87 | 93.1% | 234/270 | 86.7% |
| New customers | 192/263 | 73.0% | 55/80 | 68.8% |
| Aggregate | 273/350 | 78.0% | 289/350 | 82.6% |
Within every single segment, A has the higher conversion rate (93.1% beats 86.7% for returning customers; 73.0% beats 68.8% for new customers). Yet in the aggregate, B looks better (82.6% beats 78.0%). The mechanism is segment mix, not a real reversal of effect: returning customers convert far more easily than new customers regardless of variant, and B happened to be shown to returning customers 270 times but new customers only 80 times, while A was shown to returning customers only 87 times but new customers 263 times. B's aggregate number is inflated because it was tested overwhelmingly on the easier segment, not because it's the better variant. Stratifying by segment and comparing within each one separately, rather than trusting the unstratified aggregate, is what reveals that A is actually the consistently better performer.
Trade-offs and pitfalls. Simpson's paradox is not a data-quality bug, it's a structural consequence of unequal group sizes and unequal rates across groups, so the fix is never "clean the data harder"; it's always "check whether the segment mix differs between the groups you're comparing before trusting an aggregate.
You have a staggered-adoption setting: different units adopt treatment at different times, and the treatment effect is heterogeneous across adoption cohorts. Simulate this in Python, demonstrate that the standard two-way fixed-effects DiD estimator is biased in this setting, and implement an alternative (an event-study with cohort decomposition, or a stacked DiD) that recovers an unbiased estimate.
Sample Answer
Direct answer. Simulating a staggered-adoption setting shows that the standard two-way fixed-effects (two-way fixed effects (TWFE)) difference-in-differences (DiD) estimator can be badly biased, sometimes even the wrong sign, when treatment effects are heterogeneous across adoption cohorts, because TWFE implicitly uses ALREADY-treated units as part of the comparison group for later-treated units, contaminating the estimate with their own (different) treatment effect. An event-study specification with cohort-specific effects, or a stacked-DiD approach, avoids this by never comparing an already-treated unit's later periods against another cohort as if it were a valid control.
Approach. Simulate units adopting treatment at different times with a treatment effect that GROWS over cohorts (later cohorts get a larger true effect), fit plain TWFE, and separately fit a stacked-DiD estimator that only ever compares each treated cohort to not-yet-treated units.
import numpy as np
import pandas as pd
import statsmodels.formula.api as smf
def simulate_staggered_adoption(n_units=300, n_periods=10, seed=42, ramp=1.0):
rng = np.random.default_rng(seed)
adoption_period = rng.choice([3, 5, 7, np.inf], size=n_units, p=[0.25, 0.25, 0.25, 0.25])
rows = []
for i in range(n_units):
unit_fe = rng.normal(0, 1)
adopt_t = adoption_period[i]
for t in range(n_periods):
treated = t >= adopt_t
rel_time = (t - adopt_t) if treated else 0.0
# dynamic, GROWING effect the longer a unit has been treated: this is the specific
# DGP shape that makes TWFE's implicit already-treated-as-control comparisons biased
effect = ramp * rel_time if treated else 0.0
y = unit_fe + 0.5 * t + effect + rng.normal(0, 0.5)
rows.append({"unit": i, "t": t, "treated": int(treated), "adopt_t": adopt_t, "y": y, "rel_time": rel_time})
return pd.DataFrame(rows)
def twfe_estimate(df):
model = smf.ols("y ~ treated + C(unit) + C(t)", data=df).fit()
return float(model.params["treated"])
def stacked_did_estimate(df):
# for each finite-adoption cohort, build a clean sub-panel vs never-treated units only,
# then pool and estimate once with cohort-specific time effects
cohorts = sorted(c for c in df["adopt_t"].unique() if np.isfinite(c))
stacks = []
for stack_id, adopt_t in enumerate(cohorts):
sub = df[(df["adopt_t"] == adopt_t) | (df["adopt_t"] == np.inf)].copy()
sub["stack"] = stack_id
stacks.append(sub)
stacked = pd.concat(stacks, ignore_index=True)
stacked["unit_stack"] = stacked["unit"].astype(str) + "_" + stacked["stack"].astype(str)
model = smf.ols("y ~ treated + C(unit_stack) + C(t):C(stack)", data=stacked).fit()
return float(model.params["treated"])
df = simulate_staggered_adoption()
twfe_result = twfe_estimate(df)
stacked_result = stacked_did_estimate(df)
# ground truth: the true average treatment effect on the treated (ATT) is the average of the dynamic effect actually realized
# across all treated unit-periods (known exactly here because we generated the data)
true_att = float((1.0 * df.loc[df["treated"] == 1, "rel_time"]).mean())
print({"true_att": true_att, "twfe_estimate": twfe_result, "stacked_did_estimate": stacked_result})
Running this (VERIFIED by execution, seed=42) prints {'true_att': 2.333, 'twfe_estimate': 0.925, 'stacked_did_estimate': 2.050}: TWFE recovers less than 40% of the true average effect on the treated, while the stacked estimator recovers roughly 88% of it, on this specific dynamic-effect DGP. The bias is largest precisely when the effect GROWS over time since adoption, since that is exactly the situation where an already-treated unit's later, larger effect contaminates its use as an implicit control for a later-adopting cohort.
Key points. TWFE implicitly weights each cohort-pair comparison by group size and variance in a way that can assign NEGATIVE weight to some genuinely-positive comparisons when effects are heterogeneous across cohorts and adoption timing, which is the mathematical root of the bias; the stacked-DiD approach sidesteps this by constructing separate treated-versus-never-treated comparisons for each cohort and combining them, so a later, already-treated cohort is never used as an implicit control for an earlier one.
Complexity. The stacked approach's unit_stack interaction can materially inflate the number of fixed-effect parameters (each unit appears once per cohort-stack it participates in as a control), which is the practical cost of the correction; for large panels, this can noticeably increase compute versus plain TWFE.
Edge cases. With very few never-treated units (or none, an "all eventually treated" panel), the stacked-DiD approach as written needs a different comparison group (using not-yet-treated units at each specific point in event time, rather than a permanently never-treated group), which is the more general Callaway-Sant'Anna-style construction; this simulation's simplification (a dedicated never-treated group) works cleanly for this demonstration but would need generalizing for a real, fully-staggered panel with no never-treated units at all.
Unlock Full Question Bank
Get access to all Causal Inference interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.