Statistical Foundations for Experimentation Questions

Core statistical concepts and inference needed to design analyze and interpret experiments. Topics include hypothesis testing p values confidence intervals Type One and Type Two errors the relationship between sample size variability and interval width statistical power minimum detectable effect and effect size versus practical significance. Candidates should be able to choose and explain common statistical tests such as t tests and chi square tests contrast Bayesian and frequentist approaches at a conceptual level and describe variance estimation and variance reduction techniques. The topic covers corrections for multiple comparisons sequential testing and the risks of peeking and p hacking common misconceptions about p values and limitations of inference such as confounding and selection bias. Candidates should also be able to translate statistical findings into clear language for non technical stakeholders and explain uncertainty and limitations.

MediumTechnical

63 practiced

You're testing a feature and suspect it works differently for new versus returning users. Describe how to detect treatment-by-subgroup interactions: which statistical tests or models to use (interaction terms in regression, stratified estimators), how to control for multiple testing when exploring many subgroups, and how to validate heterogeneity findings to avoid false discoveries.

Sample Answer

**Approach overview**- Treat this as testing for heterogeneous treatment effects (HTE). Combine confirmatory inference (pre-specified) with exploratory discovery plus strict validation.

**Detecting interactions**- Start with an interaction regression:

text

Y = β0 + β1*T + β2*NewUser + β3*(T * NewUser) + X'γ + ε

Plain English: β3 estimates the difference in treatment effect between new vs returning users. Use robust SEs and check model fit.

- Use stratified estimators: estimate ATE separately in each subgroup with covariate adjustment (IPW/augmented IPW) to reduce bias.

- Use flexible HTE methods when many covariates matter: causal forest, X-learner, T-learner. They estimate individual treatment effects and can test subgroup differences.

**Multiple testing control**- If testing many subgroups, control false discoveries: prefer Benjamini–Hochberg (FDR) for exploratory, Bonferroni for conservative confirmatory.- Use hierarchical or multilevel models to borrow strength and reduce multiplicity (partial pooling).

**Validation to avoid false discoveries**- Pre-register key subgroup hypotheses when possible.- Use split-sample: discovery on one fold, confirm on held-out fold (or cross-fit for machine learners).- Perform permutation tests / placebo treatments to assess calibration.- Check robustness to covariate adjustment, continuous definitions (age cutpoints), and model class.- Report effect sizes, CIs, and smallest detectable effects; emphasize practical significance.

**Practical example**- Pre-specify new vs returning as primary subgroup, run interaction regression with covariate adjustment, correct p-values, then if exploratory causal forest suggests other subgroups, validate those in a held-out sample with FDR control.

MediumTechnical

59 practiced

Write a Python function sample_size_two_sample_t(d, sigma, alpha=0.05, power=0.8, ratio=1.0) that returns the required sample size per group for a two-sample t-test detecting Cohen's d effect size d given pooled standard deviation sigma. Use a normal-approximation or t-approximation and document assumptions in comments. Explain numeric choices and edge cases in a short docstring.

Sample Answer

**Approach — brief**Use a normal-approximation for a quick closed-form estimate:n0 = 2 * (z_{1-alpha/2} + z_{power})^2 / d^2Then optionally refine using the t-distribution / noncentral t to account for small samples by iterating n until the achieved power (via scipy.stats.nct) meets target.

python

from math import ceil
from scipy.stats import norm, nct, t

def sample_size_two_sample_t(d, sigma, alpha=0.05, power=0.8, ratio=1.0):
    """
    Return required sample size per group for two-sample (equal-variance) t-test.
    - d: Cohen's d (difference in means divided by pooled sigma)
    - sigma: pooled standard deviation (kept for API parity; d already unitless)
    - alpha: two-sided significance level
    - power: desired power (1 - beta)
    - ratio: n2 / n1 (only supports ratio==1.0 currently)
    Numeric choices:
      - start from normal-approx (fast closed form), then refine with noncentral t.
    Edge cases:
      - d == 0 -> raises ValueError
      - ratio != 1 -> NotImplementedError (extension possible)
      - requires scipy
    Assumptions:
      - equal variances, two-sided test, independent groups.
    """
    if d == 0:
        raise ValueError("Effect size d must be non-zero.")
    if ratio != 1.0:
        raise NotImplementedError("Only equal group sizes (ratio==1) supported.")
    # Normal approximation (per-group)
    z_alpha = norm.ppf(1 - alpha/2)
    z_power = norm.ppf(power)
    n0 = 2 * (z_alpha + z_power)**2 / (d**2)
    n = max(4, int(ceil(n0)))  # start at least small positive integer
    # Refine using noncentral t iterative search
    while True:
        df = 2 * n - 2
        # two-sided critical t
        t_crit = t.ppf(1 - alpha/2, df)
        # noncentrality parameter for difference of means with equal n:
        ncp = d * (n**0.5 / (2**0.5))  # because d = (mu1-mu2)/sigma, pooled sigma
        # achieved power = Pr(|T_df(ncp)| > t_crit) = upper tail + lower tail
        # nct.cdf gives CDF of noncentral t; compute two-sided power:
        power_achieved = 1 - (nct.cdf(t_crit, df, ncp) - nct.cdf(-t_crit, df, ncp))
        if power_achieved >= power or n > 1e6:
            return n
        n += 1

Explanation of key choices:- Normal approx gives a fast starting point; iterative nct refinement corrects for finite-sample t critical values.- Using d (unitless) makes sigma optional; kept for interface symmetry.Complexity: loop increases n until power satisfied; typical iterations small. Edge cases: very small d yields large n; if scipy unavailable, fallback to normal formula only.

EasyTechnical

60 practiced

You ran an experiment comparing two treatments and computed a 95% confidence interval for the difference in conversion rates: [-0.02, 0.05]. Explain in clear non-technical language what this interval means, what conclusions you can draw about statistical significance and practical significance, and what next steps you would recommend to a product manager. Be explicit about the repeated-sampling interpretation.

HardTechnical

50 practiced

Derive the variance of the difference-in-means estimator for a completely randomized experiment under the Neyman finite-population framework. Then show the standard large-sample sample-size formula for detecting an absolute difference delta with power 1 - beta and significance alpha. State all assumptions and show intermediate algebraic steps.

Sample Answer

**Assumptions**- Finite population of size N with fixed potential outcomes {Y_i(1), Y_i(0)} (Neyman); treatment assigned by complete randomization with n1 treated, n0 control, n1+n0=N.- Define sample finite-population variances: S1^2 = (1/(N-1)) Σ_i (Y_i(1)-Ȳ(1))^2, S0^2 similarly, and Sτ^2 for individual treatment effects τ_i = Y_i(1)-Y_i(0).- Two-sided test, large-sample normal approximation.

**Variance derivation (intermediate steps)**The difference-in-means estimator:

τ̂ = Ȳ_obs(1) - Ȳ_obs(0)

Under randomization, Var(τ̂) equals sampling variances of group means minus covariance from finite population:

Var(τ̂) = (1 - n1/N) * S1^2 / n1  +  (1 - n0/N) * S0^2 / n0  -  Sτ^2 / N

Plain-English: first two terms are finite-population corrections for mean estimators; the final term subtracts covariance induced by paired potential outcomes. A conservative estimator drops the negative term:

Var(τ̂) ≤ (1 - n1/N) * S1^2 / n1  +  (1 - n0/N) * S0^2 / n0

**Large-sample sample-size formula**For two-sided test of H0: τ=0 vs |τ|=δ, using normal approx and assuming equal group variances σ^2 and equal n per arm (n1=n0=n), required per-arm sample size is:

n = 2 * σ^2 * ( z_{1-α/2} + z_{1-β} )^2  /  δ^2

If variances unequal, replace 2σ^2 by σ1^2 + σ0^2. This follows from setting

Power ≈ Φ( (δ) / sqrt( (σ1^2+σ0^2)/n ) - z_{1-α/2} ) = 1-β

and solving for n.

**Remarks / Practical notes**- In finite N with non-negligible sampling fraction, include (1 - n/N) corrections.- Conservative practice uses pooled or pilot-based σ^2; Neyman unbiased variance estimator uses observed group sample variances.

HardTechnical

48 practiced

Design an analysis pipeline to discover and validate heterogeneous treatment effects (HTE) in large-scale experiments. Include exploratory discovery tools (causal trees/forests, uplift models), honest estimation techniques (sample-splitting, cross-fitting), multiplicity control for subgroup claims, and steps to produce calibrated subgroup average treatment effects for product teams. Specify evaluation metrics for subgroup policy value.

Sample Answer

**Overview & goals**Design a reproducible pipeline that (1) discovers potential heterogeneous treatment effects (HTE) at scale, (2) provides honest / calibrated subgroup ATE estimates, and (3) reports statistically-controlled claims and policy-value metrics for product teams.

**1) Data & requirements**- Define estimand (ATE conditional on X), treatment assignment mechanism (RCT vs observational; record propensity scores), outcome definition and precommitment of covariates.- Partition: holdout evaluation set (20%), discovery+estimation pool (80%).

**2) Exploratory discovery (honest discovery)**- Use causal trees/causal forests and uplift models to propose candidate subgroups: - Fit causal forests (grf) on discovery folds to estimate CATE(x). - Fit uplift classifiers (meta-learners: T-learner / X-learner / DR-learner) to rank heterogeneity.- Use stability selection: repeat discovery across bootstraps / folds; keep subgroups/features stable above threshold.

**3) Honest estimation: sample-splitting & cross-fitting**- For each candidate subgroup, perform honest ATE estimation by sample-splitting: - Use K-fold cross-fitting: train nuisance models (propensity, outcome) on K-1 folds, predict on held fold, compute doubly robust (DR) CATE estimates on held fold; rotate and aggregate.- Report subgroup Average Treatment Effect (SATE) with influence-function-based standard errors.

**4) Multiplicity control for claims**- For multiple subgroup claims apply: - Benjamini–Hochberg FDR control on p-values of SATEs for discovery-driven lists. - For post-hoc localized claims, use selective inference / holdout-based confirmatory tests (e.g., closed testing) to control family-wise error. - Optionally apply knockoff-style feature selection for FDR control when selecting covariates generating subgroups.

**5) Calibration & reporting for product teams**- Calibration: compare predicted CATE quantiles to empirical SATEs in heldout eval set; produce calibration plots and reliability metrics (ECE for CATE).- Produce calibrated subgroup ATEs: use isotonic regression or Platt scaling on predicted CATEs vs observed SATEs in holdout.- Report: subgroup definition, n, treatment rate, SATE ± CI, p-value (multiplicity-adjusted), calibration score, stability score.

**6) Evaluation metrics for subgroup policy / value**- Policy value: evaluate a decision policy π(x)=1{predicted CATE(x)>τ} on heldout eval set: - Estimated policy value V̂(π) using inverse-propensity-weighted / DR policy-value estimator with CIs. - Uplift@k (mean treatment effect among top-k% recommended). - Regret vs optimal policy (approx): V̂(opt) − V̂(π). - Precision in HTE discovery (PHE): fraction of discovered subgroups with positive true SATE. - Calibration error, coverage of CIs, and false discovery rate.

**7) Practical considerations & trade-offs**- RCT vs observational: rely on strong ignorability; if not, incorporate IV or front-door strategies.- Scalability: use distributed training for forests/GRFs or XGBoost-based DR learners.- Interpretability: prefer shallow causal trees for product-facing rules; use forests for ranking then translate to human-readable rules.

**Example workflow**- Split data 80/20. On 80% run 5× bootstrap causal-forest discovery → select 10 stable subgroups → cross-fit DR estimates per subgroup → adjust p-values via BH → calibrate CATEs on 20% holdout → report SATEs, V̂(π) and uplift@10%.

This pipeline yields honest, reproducible subgroup claims, calibrated CATEs for product decisions, and policy-value metrics to quantify business impact.

Unlock Full Question Bank

Get access to hundreds of Statistical Foundations for Experimentation interview questions and detailed answers.

Join thousands of developers preparing for their dream job.