A/B Test Design & Statistical Rigor Questions

Designing and statistically defending a controlled online experiment: framing a testable hypothesis, defining control and treatment variants, choosing the randomization unit, setting the primary success metric, and computing sample size, power, and minimum detectable effect. Covers the statistical foundations that make a readout trustworthy, including hypothesis testing, p-values, confidence intervals, statistical vs practical significance, and Type I/II error. Emphasizes avoiding the common pitfalls that invalidate a test, such as peeking, multiple-comparison inflation, underpowered designs, and how test duration and stopping rules affect the validity of conclusions.

HardTechnical

48 practiced

How would you estimate the causal effect of a feature on retention when there is substantial noncompliance (users randomized to treatment may not receive it) and attrition over time? Discuss intention-to-treat (ITT), complier-average causal effect (CACE) via instrumental variables, inverse-probability-of-censoring weighting (IPCW) for attrition, and survival-model adjustments.

Sample Answer

**High-level approach**Estimate effect at multiple levels: report ITT as policy-relevant baseline, then recover causal per-complier effect via IV/CACE, while handling time-varying attrition with IPCW and/or survival models. Combine methods and run sensitivity checks.

**1) ITT (intention-to-treat)**- Estimate difference in retention between randomized treatment and control groups regardless of uptake.- Pros: unbiased under randomization, robust to noncompliance/attrition mechanisms that are independent of assignment.- Report as primary, policy-relevant effect: "effect of offering the feature."

**2) CACE via instrumental variables**- Use random assignment Z as instrument for receipt D to recover effect among compliers.- Wald (two-stage least squares) estimator for a continuous/linear setting:

text

CACE = ( E[Y | Z=1] - E[Y | Z=0] ) / ( E[D | Z=1] - E[D | Z=0] )

- Assumptions: relevance (Z affects D), exclusion (Z affects Y only through D), monotonicity (no defiers), stable unit treatment value (SUTVA).- Implementation: 2SLS with cluster-robust SEs; include baseline covariates to improve precision.

**3) Handling attrition: IPCW**- If dropout is informative, weight observed data by inverse probability of being uncensored conditional on history.- IPCW weight for subject i at time t:

text

w_i(t) = 1 / P( observed through t | covariates, treatment, past )

- Fit a flexible dropout model (Cox, pooled logistic, or ensemble) to estimate censoring probabilities; stabilize weights to reduce variance.- Use weights in ITT and IV (weighted 2SLS) to correct for attrition under the assumption of missing at random conditional on included covariates.

**4) Survival-model adjustments**- Model retention as time-to-event with censoring (e.g., Cox or parametric survival).- For noncompliance, use structural models: rank-preserving structural failure time models or two-stage IV for survival (e.g., control function or additive hazards IV).- Example: fit weighted Cox with IPCW and instrumented treatment indicator (via two-stage residual inclusion) to estimate hazard ratios among compliers.

**5) Practical pipeline & diagnostics**- Step 1: Estimate and report ITT (unweighted); show balance and first-stage.- Step 2: Estimate CACE with 2SLS; check F-statistic (>10) and monotonicity plausibility.- Step 3: Model censoring, compute IPCW (stabilize/truncate), re-estimate ITT and IV using weights.- Step 4: Complement with survival-IV methods for time-varying hazards.- Diagnostics: sensitivity analyses (bounds, selection models), placebo outcomes, vary covariates in censoring model, weight truncation thresholds, and simulation-based calibration.

**6) Key trade-offs and assumptions**- IPCW relies on MAR conditional on modeled covariates — include rich time-varying predictors.- IV identifies only complier effect; not generalizable to all users.- Survival IV methods require stronger modeling choices and careful SE estimation (bootstrap).

Wrap-up: present ITT, then CACE via IV, correct for attrition with IPCW, and use survival-IV techniques for time-to-event outcomes — always report assumptions, diagnostics, and sensitivity bounds.

MediumTechnical

39 practiced

Explain alpha-spending for group-sequential testing. For a trial with up to 4 interim looks, describe how you compute stopping boundaries under Pocock and O'Brien-Fleming styles and how those choices affect early-stopping probability and overall power.

Sample Answer

**Brief definition / purpose**Alpha-spending formalizes how the total type I error α is allocated across interim looks so the cumulative chance of a false positive remains α. We work on information time t_k (proportion of maximum information at look k) and specify an alpha‑spending function A(t) (nondecreasing, A(1)=α). Lan–DeMets implements this flexibly and produces critical boundaries by inverting joint normal crossing probabilities.

**How to compute boundaries for 4 looks**Common workflow (applies to both styles)- Choose information times t1..t4 (e.g., equally spaced 0.25,0.50,0.75,1.0).- Choose spending function A(t) (Pocock-style or O’Brien–Fleming-style).- For each look k compute incremental spend α_k = A(t_k) − A(t_{k−1}) (with A(0)=0).- Solve for critical z_k such that the multivariate normal probability of crossing any earlier boundary equals ∑_{i≤k} α_i. Practically this is solved numerically (e.g., using gsDesign or mvtnorm) because Zs are correlated with corr(Z_i,Z_j)=sqrt(t_i/t_j).

Pocock-style (rough characterization)- The Pocock design aims for roughly equal critical values across looks. In alpha‑spending terms it uses a more front-loaded spending than strict linear but results in nearly constant z_crit at each look — i.e., z1≈z2≈z3≈z4 (a single threshold solved so overall α is preserved).- Implementation: pick a Pocock-type spending function (Lan–DeMets approximation) or directly solve for a constant z_p such that P(max_k Z_k > z_p) = α under null.

O’Brien–Fleming (OBF) style- OBF is very conservative early and spends almost no alpha until late looks. One canonical Lan–DeMets spending: A(t) = 2*(1 − Φ(z_{α}/sqrt(t))) which yields very large early z and a final z close to the fixed-sample boundary.- Compute α_k = A(t_k)−A(t_{k−1}) and numerically invert joint normal to get z_k (z1 ≫ z4).

**Effects on early-stopping probability and power**- Pocock: lower early critical thresholds than OBF → higher probability of stopping early for moderate true effects; but to preserve overall α it raises the final critical value slightly compared to OBF, so maximum (final-look) power can be a bit lower than OBF for effects detectable only at full information.- OBF: very small early α spend → low early-stop probability unless effect is large; final boundary is close to fixed-sample threshold so OBF tends to retain near‑maximum fixed-sample power while protecting type I error strongly early.- Trade-offs: Pocock favors trialist who values chance to stop early (favors resource saving, ethics), OBF favors maintaining fixed-sample power and conservative early claims.

**Practical notes**- Use standard software (gsDesign, rpact, East) to get exact z_k and sample sizes.- Always report information times, spending function, and resulting z_k and actual cumulative alpha spent.

HardTechnical

65 practiced

SUTVA is violated in your social network experiment (partial interference). Propose an estimator and identification strategy for average direct and spillover effects when interference is confined within clusters. Include assumptions, how to map exposures, weighting/HT estimators, and variance estimation that accounts for clustering.

Sample Answer

**Clarify target & assumptions**

- Goal: identify average direct effect (ADE) and average spillover effect (ASE) under partial interference (interference only within clusters).- Key assumptions: - Partial interference: clusters independent; no interference across clusters. - Known randomization/propensity model for treatment vector within each cluster (can be fully randomized, stratified, or Bernoulli with known p). - Positivity: each exposure cell has nonzero probability. - Stable cluster membership and no hidden versions of treatment beyond exposure mapping.

**Exposure mapping**

- Define an exposure mapping e_i = e(Z_i, Z_{-i,c}) that summarizes how others in i’s cluster affect i. Common choices: - e = (Z_i, k) where k = number or proportion of treated neighbors in cluster excluding i. - e ∈ {0, 1} × {0,...,m} or {0,1}×[0,1].

This reduces unit-level potential outcomes to Y_i(e).

**Identification**

- Because clusters are independent and the randomization mechanism is known, the average potential outcome for exposure e is identified by the weighted expectation over observed outcomes among units with e: - Let π_{i}(e) = P(E_i = e) under the assignment mechanism. - Then E[Y_i(e)] = E[ Y_i 1{E_i = e} / π_i(e) ] (identifiable via Horvitz–Thompson).

**Estimator: Horvitz–Thompson for clusters**

- Let clusters indexed by c = 1..C, units i∈c. Define indicator I_{ic}(e)=1{E_{ic}=e}.- HT estimator of mean outcome under exposure e:

text

mu_hat(e) = (1 / N) * sum_{c=1}^C sum_{i in c} Y_{ic} * I_{ic}(e) / pi_{ic}(e)

where N = total # units (or use average per-cluster normalization).

- ADE (direct effect) comparing e=(1,k) vs e=(0,k):

text

ADE(k) = mu_hat(1,k) - mu_hat(0,k)

- ASE comparing different neighborhood exposure levels for the same own-treatment:

text

ASE(t; k1,k0) = mu_hat(t,k1) - mu_hat(t,k0)

**Weighting / Hájek variant**

- To improve stability use Hájek (ratio) estimator:

text

mu_haj(e) = sum_i Y_i * I_i(e) / pi_i(e)  /  sum_i I_i(e) / pi_i(e)

which estimates mean among units under exposure e.

**Variance estimation accounting for clustering**

- Clusters are independent, so aggregate influence across clusters. Use cluster-robust variance for HT/Hájek: - For HT, influence for cluster c:

text

U_c(e) = sum_{i in c} ( Y_{ic} * I_{ic}(e) / pi_{ic}(e) )  -  mu_hat(e) * n_c_contrib

(choose normalization consistent with mu_hat). - Estimate Var(mu_hat(e)) as

text

V_hat = (1 / N^2) * sum_{c=1}^C ( U_c(e) )^2

- For contrasts (ADE/ASE), compute covariance between exposure estimators similarly and use delta method: Var(ADE)=Var(mu(1,k))+Var(mu(0,k)) - 2 Cov(...), with cluster-level covariances estimated by summing cluster contributions.

- For Hájek, linearize estimator and compute cluster-sum of linearized residuals to form variance; in practice use plug-in linearization or bootstrap at cluster level (clustered nonparametric bootstrap) — recommended when pi_i complicated.

**Practical notes & pitfalls**

- If assignment mechanism depends on cluster-level covariates, include correct pi_i(e) conditioning on those covariates or stratify.- Ensure exposure mapping is pre-specified and interpretable.- If some exposure cells rare, combine bins or regularize (use Hájek or model-assisted estimators).- Consider augmented estimators (IPW + outcome regression) to gain efficiency and robustness; variance via sandwich/clustered robust SE.

This strategy yields identified, unbiased (HT) estimates for direct and spillover effects under partial interference with valid cluster-robust variance accounting for within-cluster dependence.

HardTechnical

50 practiced

You must estimate direct and spillover effects on a large-scale social graph with highly skewed degree distribution. Propose a randomization and estimation strategy—options include graph cluster randomization, independent-set sampling, or edge-based assignment—and describe how to compute required sample size or number of clusters given target MDEs and desired power.

Sample Answer

**High-level approach (goal)** Estimate direct and spillover effects robustly on a heavy-tailed social graph by (a) choosing a randomization that controls interference, (b) using estimators that separate direct vs. neighborhood exposure, and (c) computing sample size via analytic + simulation power accounting for network dependence and degree skew.

**Design options & recommendation**- Graph cluster randomization (cluster by community / METIS / Louvain): good when spillovers are mostly within communities. Reduces cross-cluster contamination but yields few, large clusters → high design effect if clusters vary in size.- Independent-set sampling (randomly pick nodes with no edges between treated nodes): targets direct effects with minimal immediate spillovers; efficient when feasible on sparse parts of graph.- Edge-based assignment (treat edges or randomize half-edges): useful to estimate dyadic spillovers and local treatment effects; can be combined with node-level assignment.

For a highly skewed degree distribution:- Stratify by degree quantiles; oversample high-degree nodes or treat them separately (e.g., always-control or separate analysis) to reduce variance and bias.- Use degree-normalized exposure models and inverse-probability weights to correct estimator bias.

**Estimators**- Use exposure mapping: define Z_i (own treatment) and G_i (fraction / count of treated neighbors). Fit linear-in-exposures model (or nonparametric contrasts) to estimate direct effect β and spillover γ.- Use Hajek/IPW or generalized propensity scores to account for unequal assignment probabilities from clustering/stratification.- For inference, use randomization-based permutation tests constrained by the design or use clustered robust SEs with network-aware variance estimators (e.g., Aronow-Samii).

**Sample size / number of clusters**Start with a standard difference-in-means formula, then inflate by a design effect (DE) capturing intra-cluster correlation or network-induced dependence.

- Basic per-arm sample size (ignoring dependence):

n0 = ( (z_{1-alpha/2} + z_{1-beta})^2 * (sigma_t^2 + sigma_c^2) ) / (MDE^2)

Plain-English: required independent units per arm given variance and target MDE.

- Cluster trials: replace by number of clusters m using ICC (rho) and average cluster size s:

DE = 1 + (s - 1) * rho
n_effective = n0 * DE
clusters per arm m = n_effective / s

Intuition: dependence within clusters inflates variance; with skewed cluster sizes, use a weighted DE: 1 + (CV_s^2 + 1) * (s_bar - 1) * rho (adjust for cluster size variability).

- For network interference, replace ICC with network correlation parameter (rho_net) estimated from pilot or literature; for estimands involving neighbor exposure, compute variance of exposure G_i and use delta-method to propagate variance into estimator of γ.

**Practical workflow**1. Pilot/simulate on observed graph: define plausible β, γ; simulate assignments under candidate designs; estimate power and bias.2. Choose stratified graph-clusterization minimizing cross-cluster edges; set cluster sizes to balance power vs contamination.3. Compute analytic sample sizes using formulas above with rho_net from pilot, then validate with Monte Carlo sims across parameter grid (β, γ, degree heterogeneity).4. Pre-specify estimators, weighting, and permutation inference consistent with randomization.

**Key trade-offs**- Fewer large clusters → more contamination control but fewer independent units (lower power).- Independent-set reduces interference but may undersample hubs (bias/inefficient for spillovers).- Simulations are essential with heavy tails; analytic approximations guide but do not replace simulation-based power.

HardTechnical

49 practiced

Online experiments arrive continuously. Propose a procedure to control the false discovery rate in a streaming setting (for example, using alpha-investing, LORD, or SAFFRON). Explain the mechanics of maintaining and updating 'alpha-wealth' or thresholds over time, provide pseudocode for the update rule, and discuss parameter choices to balance discovery rate versus risk.

Sample Answer

**Goal & high-level approach**

Use an online FDR procedure (e.g., LORD or SAFFRON) to allocate testing thresholds adaptively via an "alpha-wealth" budget that grows when discoveries are made and decays when tests consume budget. This balances long-run FDR control with power in streaming experiments.

**Mechanics**

- Maintain W_t (alpha-wealth) at time t.- For each incoming p-value p_t, set test level α_t = f(W_{t-1}, t) (deterministic function used by LORD/SAFFRON).- If p_t ≤ α_t → reject (discovery): update W_t = W_{t-1} - α_t + reward (often a fixed or schedule-based gain).- Else → accept: W_t = W_{t-1} - α_t (or with small rebate depending on variant).- Ensure W_t ≥ 0 by design.

**Pseudocode (LORD++-style schedule)**

python

# parameters: w0 initial wealth, gamma sequence gamma[t], b reward per discovery
W = w0
for t, p in enumerate(stream, start=1):
    alpha_t = W * gamma[t]        # allocate proportional to wealth
    if p <= alpha_t:
        reject = True
        W = W - alpha_t + b       # reward on discovery
    else:
        reject = False
        W = W - alpha_t
    W = max(W, 0)

SAFFRON uses a candidacy threshold λ and only spends wealth on non-candidate tests; update rules allocate alpha to candidate indices and give larger rewards for early discoveries.

**Parameter choices & trade-offs**

- w0 (initial wealth): larger w0 boosts early power but increases short-run FDR risk; choose modest (e.g., 0.05).- gamma sequence: heavy-tail (gamma_t ∝ 1/(t log^2 t)) preserves long-run control; faster-decay favors early tests.- reward b (or target q): ties to desired FDR level q; larger reward increases discovery throughput but must respect theoretical bounds.- λ in SAFFRON: higher λ makes more p's candidates (increasing power) but risks spending wealth on weak signals.

Practical tips: simulate realistic p-value streams (null/non-null mix) to tune w0, gamma, b. Monitor W_t and effective alpha_t; adapt conservative defaults in production.

Unlock Full Question Bank

Get access to hundreds of A/B Test Design & Statistical Rigor interview questions and detailed answers.

Join thousands of developers preparing for their dream job.