Experiment Design, Analysis, and Causal Methods Questions

Design and analysis of experiments and causal inference methods for when randomization is not possible. Candidates should know strategies to ensure randomization and evaluate experiment quality compute sample size and minimum detectable effect select and interpret primary and guardrail metrics and design appropriate test duration. Analysis skills include hypothesis testing p values confidence intervals effect size estimation variance estimation and variance reduction segmentation and interaction analysis and robust reporting of uncertainty. This topic covers observational and quasi experimental approaches such as propensity score matching difference in differences and regression discontinuity how to reason about confounding and selection bias and when to prefer a quasi experimental approach over a randomized test. Candidates should be able to translate causal conclusions into actionable guidance recommend follow up analyses and triangulate evidence across methods.

MediumTechnical

28 practiced

Describe how causal forests (generalized random forests) estimate heterogeneous treatment effects. In what situations would you prefer causal forests over simple subgroup analysis, and what diagnostics would you run to ensure reliable conditional average treatment effect (CATE) estimates?

Sample Answer

Causal forests (generalized random forests) estimate heterogeneous treatment effects by combining the idea of propensity-robust splitting with honest, local estimation. Key idea:- Build many trees where splits are chosen to maximize treatment effect heterogeneity (e.g., using criteria based on treatment vs control score differences) rather than prediction MSE.- Use "honest" estimation: each tree is grown on one sample and treatment effects are estimated on a separate out-of-bag sample to avoid overfitting.- At prediction time, a test unit’s neighborhood is the set of training units that share leaves across trees; the CATE is the weighted average of within-leaf treatment-control differences (often using doubly robust or orthogonalized outcomes to remove confounding).

When to prefer causal forests over simple subgroup analysis:- When heterogeneity is high-dimensional and interactions matter (many covariates, unknown functional forms).- When you want continuous, individualized CATEs rather than coarse pre-specified groups.- When sample sizes per subgroup would be too small for reliable estimates.Simple subgroup analysis is fine for a few, pre-specified, interpretable splits and when transparency is priority.

Diagnostics to ensure reliable CATE estimates:- Overlap/positivity checks: verify common support of propensity scores across treated and control.- Calibration: check average predicted CATE equals ATE in aggregate or via grouping predicted CATE quantiles and comparing observed subgroup effects.- Placebo/null tests: apply estimator where treatment is randomized or on pre-treatment outcomes to check zero effects.- Covariate balance within leaf/neighborhood: ensure important covariates aren’t imbalanced.- Honesty and sample-splitting sensitivity: confirm results stable across random seeds and splits.- Variable importance and partial dependence plots to interpret drivers of heterogeneity.- Confidence intervals and coverage: use built-in forest standard errors, and bootstrap or jackknife to validate coverage.

EasyTechnical

24 practiced

Explain cluster-randomized experiments: when they are necessary for AI experiments that change backend infrastructure, how to compute the design effect given ICC and average cluster size, and provide a numerical example: ICC=0.01, avg cluster size=40, base sample size per arm (user-level) 10,000 — what is effective sample size?

MediumTechnical

24 practiced

You observe that users who adopt Feature X are systematically different (more engaged) than non-adopters. Outline steps to detect selection bias, propose methods to mitigate it (e.g., matching, IV, DiD), and explain diagnostic checks you would run to justify causal claims from observational data.

Sample Answer

Approach: treat adoption of Feature X as a treatment and ask whether observed engagement differences are causal or driven by selection bias/confounding. Steps to detect, mitigate, and validate causal claims:

1) Detect selection bias- Compare pre-treatment covariates: baseline engagement, tenure, demographics, device, traffic source. Large differences suggest selection.- Visualize propensity scores and overlap (histograms, KDE) to check common support.- Balance tests (standardized mean differences) on covariates.- Placebo/falsification checks: test for “effects” on outcomes measured before adoption.

2) Mitigation methods (with intuition and when to use)- Propensity score matching / weighting: model P(adopt|X) (logistic/GBM/NN), match or IPW to balance observables. Good when unconfoundedness plausible.- Covariate adjustment using doubly robust estimators (AIPW) to reduce model dependence.- Difference-in-differences (DiD): if adoption timing varies, compare pre/post trends vs control; use event-study to check parallel trends.- Instrumental variables (IV): find an instrument that shifts adoption (e.g., randomized rollout, eligibility, UI experiment) that affects adoption but not engagement directly.- Regression discontinuity: if adoption threshold exists (quota, score).- Synthetic control for aggregate-level treatments.

3) Diagnostics to justify causal claims- Covariate balance after adjustment (SMD < 0.1).- Overlap: no extreme propensity weights; truncate if needed.- Sensitivity analysis to unobserved confounding (Rosenbaum bounds, E-value).- Placebo outcomes and lead-lag tests in DiD/event-study: no pre-treatment effect.- Check robustness across models (matching, IPW, AIPW, parametric regression) and subsets.- Heterogeneity analysis: ensure effects not driven by one subgroup.- If IV used, test instrument relevance (F-stat) and argue exclusion with domain knowledge; run overidentification tests if multiple instruments.

Example application: model propensity with a gradient-boosted tree on user history, match 1:1 with caliper, run AIPW for ATT, run event-study to confirm no pre-trends, compute E-value to report how strong an unmeasured confounder would need to be to negate the effect.

Conclusion: combine good design (exploit quasi-experiments), rigorous adjustment, and multiple diagnostics to build credible causal claims from observational data.

EasyTechnical

26 practiced

Explain the difference between simple randomization and stratified (blocked) randomization. For an AI-powered feed experiment, give a practical example of when stratification is strongly preferred and why. Explain potential pitfalls if strata are not pre-specified correctly.

EasyTechnical

25 practiced

List and briefly describe three variance-reduction techniques commonly used in online experiments (e.g., covariate-adjusted analysis, CUPED, blocking). For each technique give one practical limitation or assumption and an example scenario where it yields the largest gains.

Unlock Full Question Bank

Get access to hundreds of Experiment Design, Analysis, and Causal Methods interview questions and detailed answers.

Join thousands of developers preparing for their dream job.