Statistical Foundations for Experimentation Questions

Core statistical concepts and inference needed to design analyze and interpret experiments. Topics include hypothesis testing p values confidence intervals Type One and Type Two errors the relationship between sample size variability and interval width statistical power minimum detectable effect and effect size versus practical significance. Candidates should be able to choose and explain common statistical tests such as t tests and chi square tests contrast Bayesian and frequentist approaches at a conceptual level and describe variance estimation and variance reduction techniques. The topic covers corrections for multiple comparisons sequential testing and the risks of peeking and p hacking common misconceptions about p values and limitations of inference such as confounding and selection bias. Candidates should also be able to translate statistical findings into clear language for non technical stakeholders and explain uncertainty and limitations.

HardTechnical

81 practiced

You are considering stopping an experiment early based on a surrogate metric (e.g., add-to-cart) that correlates with the final outcome (revenue) but is imperfect. Explain statistical risks of stopping on a surrogate and propose a validated framework for using surrogates for early stopping that addresses validation, pre-specification, adjusted alpha allocation, and calibration to avoid biased inference.

Sample Answer

Stopping on a surrogate (e.g., add-to-cart) risks biased decisions because the surrogate is an imperfect proxy: it can inflate Type I error (false positives), misestimate treatment effect on the true outcome (revenue), and introduce selection bias if stopping depends on noisy early signals. Key statistical risks: regression-to-the-mean, measurement error attenuation, mediator vs. surrogate confusion, and multiplicity from interim looks.

Validated framework for safe early stopping on surrogates:

1) Validation (offline): use historical experiments to quantify the surrogate’s fidelity. Estimate correlation and predictive R² of surrogate → revenue, fit joint models (surrogate and outcome) and compute the surrogate threshold effect (minimum surrogate change implying non-zero revenue effect). Perform cross-validated calibration and mediation analysis to separate direct vs. mediated effects.

2) Pre-specification: in the analysis plan declare the surrogate, the stopping criterion (e.g., predictive probability of revenue lift > X), timing of interim looks, and minimum sample size before any stop. Lock scripts and thresholds before peeking.

3) Adjusted alpha allocation: treat surrogate looks as interim analyses for the primary hypothesis. Use group-sequential or alpha-spending procedures (O’Brien–Fleming, Pocock) on the joint test, or apply Bonferroni/Šidák across looks. Alternatively, use conditional error-rate control: compute conditional Type I error for stopping on surrogate and adjust final test threshold to preserve overall alpha for revenue.

4) Calibration & simulation: run experiment-level simulations using historical variance, surrogate-revenue mapping, and measurement noise to estimate operating characteristics (Type I, Type II, bias) under plausible effects. Calibrate stopping thresholds so that final inference on revenue is unbiased or bias-corrected (e.g., inverse-probability weighting, hierarchical Bayesian shrinkage using the surrogate as predictor).

Operational rules: require minimum predictive strength (e.g., surrogate R² > threshold), mandate a final analysis on revenue regardless of surrogate stop (with adjusted alpha), and report adjusted confidence intervals that account for early stopping. Combine frequentist alpha-spending with Bayesian predictive stopping for transparency: stop early only when both procedures indicate high probability of meaningful revenue lift.

This approach controls false discovery, quantifies bias, preserves inferential validity, and keeps decisions actionable while acknowledging surrogate imperfection.

HardTechnical

50 practiced

You run automated experiments producing thousands of correlated metrics across product areas. Propose a scalable statistical approach to control false discoveries that accounts for metric correlation and hierarchical grouping (for example, metrics grouped by product area). Discuss methods like hierarchical Benjamini-Hochberg, knockoffs, and empirical Bayes, and trade-offs between power and Type I error control.

Sample Answer

Situation: You have thousands of correlated metrics organized by product area and need scalable false-discovery control that respects correlation and hierarchy while keeping power.

Proposal (overview):1. Treat the problem as hierarchical multiple testing: test families (product areas) first, then metrics within rejected families. Combine a global FDR target with within-family control to limit wasted follow-ups.

Recommended pipeline (practical, scalable):- Preprocessing: compute standardized test statistics (z-scores) per metric; estimate correlation structure within and across groups (block-covariance by product area) using shrinkage estimators (Ledoit–Wolf) or factor models to keep it scalable.- Stage 1 — Group-level testing: compute a group-level statistic (e.g., Simes or aggregated z via Stouffer with estimated covariances). Apply Benjamini–Hochberg (BH) to group p-values to select product areas for follow-up.- Stage 2 — Within-group testing: for each selected area, apply BH on its metric p-values but use procedures that account for dependence: - Hierarchical BH (Yekutieli & Benjamini variants) that controls FDR across levels. - Or use the grouped Benjamini–Hochberg with adjusted alpha allocation (e.g., allocate more alpha to larger/more promising groups using weights proportional to expected signal).

Methods to consider and trade-offs:- Hierarchical BH: simple, well-understood, and scalable. Controls FDR under positive dependence; may be conservative under arbitrary dependence, losing power.- Knockoffs (model-X knockoffs): provides exact FDR control even with arbitrary feature correlation for variable selection tasks. Strength: powerful and robust to dependence. Weakness: requires building a good generative model for covariates (metrics) and typically targets variable selection in regression settings — heavier engineering and less direct when metrics are heterogeneous or non-Gaussian.- Empirical Bayes (EB)/local FDR: model the z-score distribution as a mixture of null and alternative (with correlation modeled via factor structure). EB can borrow strength across metrics and groups, increasing power and producing local FDR estimates for prioritization. Trade-off: relies on correct mixture modeling and may under-control frequentist FDR if model misspecified; however, use conservative thresholds or calibrate via permutations to mitigate risk.

Scalable hybrid recommendation:- Start with hierarchical BH for initial automated pipelines (fast, low complexity).- Parallel track empirical-Bayes modeling: fit a two-groups model with factor-adjusted null (e.g., FDRreg or ash with factor correction) to get local FDRs and prioritization scores; validate calibration using permutations or negative controls.- For high-stakes decisions or when you can model metric covariates well, use knockoffs or knockoff-based variable selection inside an experiment-specific regression to get rigorous FDR control.

Implementation notes:- Use block/low-rank covariance estimates to scale to thousands of metrics.- Use permutations/negative controls to empirically estimate null behavior and calibrate procedures.- Report both adjusted p-values (BH) and local FDR scores so stakeholders see both formal control and prioritized lists.- Monitor Type I errors via periodic audits (holdout experiments or synthetic nulls).

Practical trade-offs summary:- Simplicity & speed: hierarchical BH (less power under complex dependence).- Power & borrowing strength: Empirical Bayes (higher power, model risk).- Rigorous control under arbitrary dependence: Knockoffs (engineering cost, modeling covariates).Choose a hybrid: hierarchical BH for routine runs + EB-prioritization and knockoffs selectively for critical analyses; validate with permutations/controls to maintain trust in Type I control.

HardTechnical

59 practiced

Design a stopping rule for an experiment that expects 500,000 users per arm but will be monitored daily. Provide an alpha-spending plan using an O'Brien-Fleming or Pocock-style approach for up to 30 interim looks. Explain qualitatively how the nominal p-value thresholds change early versus late and how to implement this in practice without introducing bias from ad-hoc peeking.

Sample Answer

Framework / assumptions:- Two-arm A/B, two-sided overall alpha = 0.05, max information = N = 500k users per arm (final look).- Up to 30 interim looks (roughly daily monitoring), information fraction t_k = n_k / N (use cumulative users seen).- Use Lan–DeMets alpha-spending with either O’Brien–Fleming (OBF)-style or Pocock-style spending function to allocate alpha across looks. Pre-specify everything and lock code/plan.

Alpha-spending plan (how to compute):1. Choose spending function: - OBF-style: very small early spending, most alpha reserved for the end. Good when you want strong protection against false positives on early unstable signals. - Pocock-style: more uniform spending, lower final boundary but more chance to stop earlier.2. Define information times t_1,..,t_30 (can be calendar-based but apply actual information fraction = observed cumulative sample / N at each look).3. Use a library (R gsDesign::gsDesign or east, or python gs-algorithms) to compute nominal two-sided z or p thresholds for each t_k under chosen spending function (Lan–DeMets approximation). Save the table of nominal p-value thresholds.

Qualitative thresholds (example intuition):- OBF-style: first few looks have extremely strict nominal p-values (practically impossible to cross unless huge effect) — e.g., look 1: p << 1e-4; middle looks relax (look 15: p ~0.001–0.01); final look ≈ 0.049 (close to nominal 0.05).- Pocock-style: thresholds are roughly constant across looks — each look requires a moderate p-value (e.g., ~0.005–0.02 depending on number of looks), meaning easier to stop earlier but lower final p than 0.05.

Implementation in practice (prevent ad-hoc-peeking bias):- Pre-specify and document: - total alpha, spending function, max looks, how information fraction is computed, analysis metric and test statistic, primary population (e.g., all users, exclusions). - exact code to compute thresholds (commit to repo with hash), and the table of nominal p thresholds keyed by actual information fraction or look index.- At each daily check: - compute the current information fraction t and the test statistic. - map t to the pre-computed nominal threshold (interpolate between pre-computed t_k if necessary using the same spending function code). - Compare observed p to that nominal p — stop only if p <= nominal p at that look.- Use an independent monitor or blinded analyst to perform looks and make stopping decision, and log each look (timestamp, n, metric estimate, p, decision).- Never change the spending function, alpha, or number of looks after seeing interim results.

Practical tips:- If daily numbers are noisy, consider grouping looks (e.g., every 3 days) to reduce multiple tiny-information checks and false alarms.- Simulate operating characteristics (power, expected sample size, type I error) under plausible effect sizes before rollout.- Use two-sided or one-sided boundaries consistent with business question; adjust alpha accordingly.

Why this avoids bias:- Pre-specifying the alpha-spending and mapping by information fraction guarantees the familywise type I error is controlled regardless of how many times you look, preventing inflated false positives from ad-hoc peeking. Independent monitoring and audit logs add governance for business stakeholders.

HardTechnical

50 practiced

Contrast Bayesian and frequentist approaches conceptually and with a small numeric example. Suppose control has 52 successes out of 1000 and treatment has 64 successes out of 1000. Using independent Beta(1,1) priors for each conversion rate, compute (or describe how to compute) the posterior probability that treatment conversion is higher than control, and compare that decision to a frequentist two-sided proportion test p-value and conclusion.

Sample Answer

Conceptual contrast- Frequentist: parameters are fixed; inference via sampling distributions of estimators. You test H0 (no difference) and compute p-values; cannot assign probability to hypotheses.- Bayesian: parameters are random with prior distributions; update to posterior given data and can compute direct probabilities like Pr(p_t > p_c). Decisions incorporate prior beliefs and loss functions.

Numeric setup- Control: 52/1000 → likelihood Binomial(1000, p_c)- Treatment: 64/1000 → Binomial(1000, p_t)- Priors: independent Beta(1,1) (uniform)

Bayesian posterior- Posterior for p_c ~ Beta(1+52, 1+948) = Beta(53,949)- Posterior for p_t ~ Beta(1+64, 1+936) = Beta(65,937)- We want Pr(p_t > p_c). Compute by Monte Carlo sampling or analytic Beta-Beta integration.

Example Monte Carlo (python):

python

import numpy as np
from scipy.stats import beta
N=2000000
pt = beta.rvs(65,937,size=N)
pc = beta.rvs(53,949,size=N)
prob = (pt>pc).mean()
prob

This yields approximately 0.87 (≈0.873).

Frequentist two-sided test- p_hat_t=0.064, p_hat_c=0.052, diff=0.012- Pooled p≈116/2000=0.058; SE ≈ sqrt(p*(1-p)*(1/1000+1/1000)) ≈ 0.01044- z ≈ 0.012/0.01044 ≈ 1.15 → two-sided p ≈ 0.25 (fail to reject H0)

Comparison & decision- Bayesian: posterior Pr(p_t > p_c) ≈ 0.87 → moderately strong evidence that treatment is better; you can map to decision via expected loss (e.g., choose treatment if gain × probability > cost).- Frequentist: p≈0.25 → no statistically significant difference at conventional α (0.05).- They differ because Bayesian gives direct probability about the parameter (and uses prior + observed data), while frequentist tests assess extremeness under H0. For business decisions, report the posterior probability and expected effect size (mean diff ≈ 0.012) and decide using the cost/benefit trade-off rather than p-value alone.

HardTechnical

65 practiced

You have sparse per-country experiment data with noisy effect estimates. Describe a hierarchical Bayesian model that pools information across countries to estimate country-level treatment effects with partial pooling. Specify a plausible model structure (likelihood and priors), explain how shrinkage improves estimates, and outline fitting options (MCMC versus variational inference) and diagnostic checks.

Unlock Full Question Bank

Get access to hundreds of Statistical Foundations for Experimentation interview questions and detailed answers.

Join thousands of developers preparing for their dream job.