Experiment Design, Analysis, and Causal Methods Questions

Design and analysis of experiments and causal inference methods for when randomization is not possible. Candidates should know strategies to ensure randomization and evaluate experiment quality compute sample size and minimum detectable effect select and interpret primary and guardrail metrics and design appropriate test duration. Analysis skills include hypothesis testing p values confidence intervals effect size estimation variance estimation and variance reduction segmentation and interaction analysis and robust reporting of uncertainty. This topic covers observational and quasi experimental approaches such as propensity score matching difference in differences and regression discontinuity how to reason about confounding and selection bias and when to prefer a quasi experimental approach over a randomized test. Candidates should be able to translate causal conclusions into actionable guidance recommend follow up analyses and triangulate evidence across methods.

HardTechnical

41 practiced

Describe mediation analysis for decomposing total effect into direct and indirect effects through a mediator (e.g., product UI → engagement → purchases). List assumptions (sequential ignorability), outline the estimation steps, and discuss pitfalls like post-treatment confounding.

Sample Answer

Mediation analysis decomposes a total causal effect of an exposure X (product UI) on an outcome Y (purchases) into: (1) direct effect of X on Y not through mediator M (engagement), and (2) indirect effect transmitted via M. Formal targets: average total effect (ATE) = E[Y|do(X=1)] − E[Y|do(X=0)], natural direct effect (NDE) and natural indirect effect (NIE) where NIE = ATE − NDE.

Key assumptions (Sequential ignorability):1. No unmeasured confounding of X→Y: Y(x,m) ⟂ X | C.2. No unmeasured confounding of X→M: M(x) ⟂ X | C.3. No unmeasured confounding of M→Y given X: Y(x,m) ⟂ M | X, C.4. Positivity and consistency.These collectively are often called sequential ignorability; C are observed pre-treatment covariates.

Estimation steps:1. Specify models: e.g., M ~ f1(X,C) (linear/GLM) and Y ~ f2(X,M,C) including interaction X*M if plausible.2. Fit models and use parametric g-computation / mediation formula: compute counterfactual expectations E[Y|X=x, M=m, C] and integrate over the distribution of M under X=x'.3. Alternatively use inverse probability weighting (IPW) for M, or targeted maximum likelihood estimation (TMLE) for double robustness.4. Bootstrap for CIs.

Pitfalls & diagnostics:- Post-treatment confounding: variables affected by X that also affect M and Y (mediator–outcome confounders that are descendants of X) violate assumptions; adjusting for them induces bias. Use instrumental variables or longitudinal mediation models (sequential g-estimation) if present.- Model misspecification: interaction terms and nonlinearity change decomposition (on non-linear scales, NDE+NIE may not sum to ATE).- Unmeasured confounding: sensitivity analysis (e.g., VanderWeele’s bounds, bias parameters) is essential.- Measurement error in M attenuates indirect effect.Practical advice: pre-register causal graph, adjust only for pre-treatment covariates, test robustness with alternative estimators and sensitivity analyses, and report assumptions transparently.

MediumTechnical

25 practiced

Compare propensity score matching (PSM) and inverse probability weighting (IPW). For a product change rolled out selectively, when would PSM be preferable, when would IPW be preferable, and what are the main diagnostics and pitfalls of each approach?

Sample Answer

Brief definitions:- Propensity score matching (PSM) pairs treated and control units with similar estimated propensity to be treated, estimating average treatment effect on the treated (ATT) by comparing matched outcomes.- Inverse probability weighting (IPW) uses the propensity score to weight units so the weighted control group resembles the treated group (or vice versa), estimating ATE or ATT depending on weights.

When to prefer PSM:- When you care specifically about comparability of treated units (ATT) and want a transparent matched sample for diagnostics and storytelling.- When treatment is sparse or there is good overlap but you prefer to exclude poor matches rather than rely on extreme weights.- When model misspecification risk exists and you want to rely on local balance via calipers.

When to prefer IPW:- When estimating population-average effects (ATE) or using the full sample is important.- When you have many covariates and want to adjust continuously rather than discard units.- When sample sizes are large and stable weight estimation is feasible.

Key diagnostics:- Overlap / common support: propensity score histograms/density plots by group; check regions with no overlap.- Covariate balance: standardized mean differences (SMDs) before and after matching/weighting; aim for SMD < 0.1.- For IPW: examine weight distribution (max, mean, variance), effective sample size, and influence diagnostics.- For PSM: check matched-pair balance, percent of treated matched, and residual bias within calipers.

Main pitfalls:- Both rely on unconfoundedness (no unobserved confounders) — cannot fix hidden bias.- PSM pitfalls: loss of sample size and external validity if many treated are unmatched; poor caliper choice can introduce bias; matching on propensity alone can hide imbalance on important covariates unless checked.- IPW pitfalls: extreme weights amplify variance and sensitivity to propensity model misspecification; trimming or stabilized weights help but change estimand; positivity violations produce unstable or biased estimates.

Practical recommendations:- Always inspect overlap first. If overlap is poor and you want ATT, prefer matching with calipers and report matched sample characteristics. If overlap is reasonable and you need ATE/use covariate adjustment flexibly, prefer IPW with stabilized weights plus trimming and robust variance estimation. Complement either approach with doubly robust methods (AIPW) and sensitivity analyses for unmeasured confounding.

HardTechnical

29 practiced

Explain two approaches for sensitivity analysis to quantify the effect of unobserved confounding on an estimated treatment effect: Rosenbaum bounds and the E-value. Provide an example interpretation of each and how a product team should use the results in decision making.

Sample Answer

Rosenbaum bounds and the E-value are two complementary sensitivity-analysis tools for assessing how strong an unobserved confounder would need to be to explain away an observed treatment effect.

Rosenbaum bounds (matched/observational setups):- What it is: A nonparametric approach that quantifies how large an unmeasured binary confounder’s impact on treatment odds must be to change inference. It introduces Γ (Gamma): the factor by which two matched units could differ in odds of treatment because of unobserved confounding.- Interpretation example: If Γ = 1.5 is the threshold where the treatment’s p-value exceeds 0.05, an unobserved confounder would need to make one unit 50% more likely to receive treatment than its match to overturn significance.- How product teams use it: If plausible unmeasured biases (from domain knowledge) are much smaller than Γ, decision-makers can be confident in the effect. If plausible biases approach or exceed Γ, delay launch or run further experiments / collect additional covariates.

E-value (risk-ratio based, for point estimates like relative risks or odds ratios):- What it is: The minimum strength of association (on the risk ratio scale) that an unmeasured confounder would need, with both treatment and outcome, to reduce the observed effect to the null.- Interpretation example: Observed RR = 0.6 (protective). E-value = 2.5 means an unmeasured confounder would need a risk ratio ≥2.5 with both treatment and outcome, beyond measured covariates, to explain away the RR.- How product teams use it: Compare the E-value to known confounders’ associations. If known plausible confounders have associations much smaller than 2.5, the effect is robust; otherwise prioritize randomized tests or targeted data collection.

Practical guidance:- Use both methods: Rosenbaum bounds when you have matched or paired analyses; E-value for regression/ratio estimates.- Report the sensitivity numbers, plausible confounder candidates, and conduct targeted checks (negative controls, bounding with plausible bias).- Treat sensitivity analysis as decision support—not proof—guiding whether to act, run an A/B test, or invest in better measurement.

MediumTechnical

24 practiced

Design an experiment to evaluate a new search ranking algorithm where some users are logged in and others are anonymous. Decide on the randomization unit (user, session, request), discuss the pros/cons, propose primary and guardrail metrics, and outline how to compute sample size given baseline CTR and desired MDE.

Sample Answer

Clarify objective: measure causal effect of a new search ranking on user engagement (CTR) while avoiding bias from logged-in vs anonymous behavior.

Randomization unit — recommendation:- Primary: user-level randomization for logged-in users; session-level for anonymous users (hybrid).Pros/cons:- User-level: pros — avoids cross-contamination across sessions, captures persistent behavior; cons — requires identity, slower ramp if many anonymous users.- Session-level: pros — feasible for anonymous users, faster allocation; cons — risk of contamination if user sees both variants in different sessions.- Request-level: pros — maximal statistical power; cons — high risk of interference (same user sees inconsistent rankings), unrealistic UX inconsistency, violates SUTVA.Choose hybrid: assign stable treatment to logged-in user IDs; for anonymous, assign per-session with tracking cookie and expire after session. Ensure logging of assignment and stickiness windows.

Metrics:- Primary metric: per-search click-through rate (CTR) = clicks / searches. Use user-averaged CTR as main estimator to reduce heavy-tail influence.- Secondary metrics: mean clicks per search position, dwell time after click, query success rate (no reformulation), conversion rate (if applicable).- Guardrails: overall search latency, zero-results rate, query abandonment rate, user retention (7/28-day), and negative-tail CTR (for important segments).

Sample size:- For binary/ratio CTR approximate with proportions: required n per arm (searches or users) for detecting absolute MDE Δ with power 1-β and α:n = [ (Z_{1-α/2} sqrt(2 p (1-p)) + Z_{1-β} sqrt(p1(1-p1)+p2(1-p2)) )^2 ] / Δ^2Simpler approx when p1≈p2≈p: n ≈ 2 * (Z_{1-α/2}+Z_{1-β})^2 * p(1-p) / Δ^2Example: baseline CTR p=0.10, want 10% relative MDE → Δ=0.01 (absolute). For α=0.05 (Z=1.96), power=0.8 (Z=0.84):n ≈ 2*(1.96+0.84)^2 *0.1*0.9 / 0.01^2 ≈ 2*(7.84)*0.09 /1e-4 ≈ 1.411e5 searches per arm → ~141k searches/arm.Adjust for clustering (user-level): multiply by design effect = 1 + (m-1)ρ where m = avg searches per user, ρ = ICC. If interactions are correlated, increase n.

Analysis notes:- Pre-specify analysis (intention-to-treat), stratify by logged-in vs anonymous, and run heterogeneity checks.- Use sequential monitoring with alpha spending if checking early.- Monitor guardrails in near-real time and stop if severe regressions.

MediumTechnical

34 practiced

Explain Regression Discontinuity Design (RDD). Provide a toy example where RDD is appropriate (e.g., users above a threshold receive premium access), describe sharp vs fuzzy RDD, and list the key assumptions required for causal interpretation.

Unlock Full Question Bank

Get access to hundreds of Experiment Design, Analysis, and Causal Methods interview questions and detailed answers.

Join thousands of developers preparing for their dream job.