Experimentation Methodology and Rigor Questions

Focuses on rigorous experimental methodology and advanced testing approaches needed to produce reliable, actionable results. Topics include statistical power and minimum detectable effect trade offs, multiple hypothesis correction, sequential and interim analysis, variance reduction techniques, heterogenous treatment effects, interference and network effects, bias in online experiments, two stage or multi component testing, multivariate designs, experiment velocity versus validity trade offs, and methods to measure business impact beyond proximal metrics. Senior level discussion includes designing frameworks and practices to ensure methodological rigor across teams and examples of how to balance rapid iteration with safeguards to avoid false positives.

MediumTechnical

56 practiced

You have high-dimensional user features and want to discover heterogeneous treatment effects. Compare decision-tree uplift methods, causal forests, and meta-learners (T-, S-, X-learners) in terms of bias-variance trade-offs, interpretability, scalability, and validation strategies. Include practical steps to avoid overfitting when searching for subgroups.

Sample Answer

**Summary comparison**

- **Decision-tree uplift (CT-AU, UpliftDT)** - Bias-variance: low bias for strong discrete interactions, high variance in high-dim sparse settings. - Interpretability: high—easy subgroup rules. - Scalability: fast single-tree; ensembles slower. - Validation: use out-of-bag uplift or doubly-robust uplift estimates.

- **Causal forests** - Bias-variance: designed to reduce bias via honest splitting and aggregation; lower variance than single trees, but can still overfit if signals are weak. - Interpretability: medium—variable importance and leaf summaries, but rules less transparent. - Scalability: scalable with parallel trees; memory-heavy for very high-dim. - Validation: honest estimation (sample-splitting), calibration via doubly-robust ATE on held-out folds.

- **Meta-learners (T-, S-, X-)** - Bias-variance: depends on base learners. T-learner high variance if separate models; S/X borrow strength to reduce variance and bias under imbalance. - Interpretability: depends on base models (GLMs vs. forests). - Scalability: flexible—scales with chosen learners. - Validation: cross-fitting, targeted regularization, and doubly-robust scoring.

**Practical validation & overfitting avoidance (high-dim focus)**

1. Honest estimation / sample-splitting: separate data for model selection, leaf/split creation, and treatment effect estimation. 2. Cross-fitting: K-fold cross-fitting to debias nuisance estimates. 3. Regularization & feature selection: L1/elastic-net or supervised PCA for nuisance models; use domain constraints for splits. 4. Use doubly-robust or augmented inverse propensity scoring to evaluate CATEs on held-out data. 5. Calibration checks: test subgroup ATEs with permutation / placebo tests and bootstrap CIs. 6. Prune search space: require minimum leaf size, penalize complexity (honest penalty), and adjust for multiple testing (FDR or sequential testing). 7. Report sensitivity: overlap, unconfoundedness assumptions, and heterogeneous effect stability across folds.

These choices balance bias-variance: use causal forests or X-learner with strong regularization and honest validation for robust discovery; use uplift trees when interpretability and actionable rules matter.

MediumTechnical

64 practiced

Explain alpha-spending for group-sequential testing. For a trial with up to 4 interim looks, describe how you compute stopping boundaries under Pocock and O'Brien-Fleming styles and how those choices affect early-stopping probability and overall power.

Sample Answer

**Brief definition / purpose**Alpha-spending formalizes how the total type I error α is allocated across interim looks so the cumulative chance of a false positive remains α. We work on information time t_k (proportion of maximum information at look k) and specify an alpha‑spending function A(t) (nondecreasing, A(1)=α). Lan–DeMets implements this flexibly and produces critical boundaries by inverting joint normal crossing probabilities.

**How to compute boundaries for 4 looks**Common workflow (applies to both styles)- Choose information times t1..t4 (e.g., equally spaced 0.25,0.50,0.75,1.0).- Choose spending function A(t) (Pocock-style or O’Brien–Fleming-style).- For each look k compute incremental spend α_k = A(t_k) − A(t_{k−1}) (with A(0)=0).- Solve for critical z_k such that the multivariate normal probability of crossing any earlier boundary equals ∑_{i≤k} α_i. Practically this is solved numerically (e.g., using gsDesign or mvtnorm) because Zs are correlated with corr(Z_i,Z_j)=sqrt(t_i/t_j).

Pocock-style (rough characterization)- The Pocock design aims for roughly equal critical values across looks. In alpha‑spending terms it uses a more front-loaded spending than strict linear but results in nearly constant z_crit at each look — i.e., z1≈z2≈z3≈z4 (a single threshold solved so overall α is preserved).- Implementation: pick a Pocock-type spending function (Lan–DeMets approximation) or directly solve for a constant z_p such that P(max_k Z_k > z_p) = α under null.

O’Brien–Fleming (OBF) style- OBF is very conservative early and spends almost no alpha until late looks. One canonical Lan–DeMets spending: A(t) = 2*(1 − Φ(z_{α}/sqrt(t))) which yields very large early z and a final z close to the fixed-sample boundary.- Compute α_k = A(t_k)−A(t_{k−1}) and numerically invert joint normal to get z_k (z1 ≫ z4).

**Effects on early-stopping probability and power**- Pocock: lower early critical thresholds than OBF → higher probability of stopping early for moderate true effects; but to preserve overall α it raises the final critical value slightly compared to OBF, so maximum (final-look) power can be a bit lower than OBF for effects detectable only at full information.- OBF: very small early α spend → low early-stop probability unless effect is large; final boundary is close to fixed-sample threshold so OBF tends to retain near‑maximum fixed-sample power while protecting type I error strongly early.- Trade-offs: Pocock favors trialist who values chance to stop early (favors resource saving, ethics), OBF favors maintaining fixed-sample power and conservative early claims.

**Practical notes**- Use standard software (gsDesign, rpact, East) to get exact z_k and sample sizes.- Always report information times, spending function, and resulting z_k and actual cumulative alpha spent.

MediumTechnical

74 practiced

A product change increases click-through rate by 10% but observed downstream revenue does not change. Describe methodologies to quantify true business impact beyond proximal metrics: attribution windows, funnel experiments, instrumental variables, mediation analysis, and structural/econometric models. Describe when each is appropriate.

Sample Answer

**Approach (framework)**Start by treating the observed 10% CTR lift as a proximal change; ask whether and how that should causally affect downstream revenue. Use a mix of quasi-experimental and structural methods to test mechanisms, timing, and heterogeneity, and to estimate long-run impact under explicit assumptions.

**Attribution windows**- What: Extend conversion windows (short → long) and measure lagged conversions per cohort.- When: Use when conversions may be delayed (e.g., subscription, multi-step purchase).- Example: Kaplan–Meier style survival curves of time-to-purchase for treatment vs control.- Pros/cons: Simple, minimal assumptions; can miss indirect/longer-term effects.

**Funnel (sequential) experiments**- What: Run randomized experiments at downstream funnel stages (e.g., checkout discounts, re-targeting) or factorial designs that randomize the original change and downstream nudges.- When: Use to isolate where drop-off occurs and to estimate mediated effects via manipulations.- Example: 2×2 RCT: CTR-change on/off × re-target email on/off to measure interaction on revenue.

**Instrumental variables (IV)**- What: Use exogenous variation correlated with CTR but not with unobserved determinants of revenue (e.g., rollout timing, A/B assignment as instrument).- When: Use when randomization for revenue is unavailable or contamination exists.- Example: Use random assignment to treatment as instrument for click exposure when clicks themselves are endogenous.- Pros/cons: Estimates local average treatment effect; requires strong exclusion restriction.

**Mediation analysis**- What: Decompose total effect into direct and indirect (via clicks) paths using causal mediation frameworks (potential outcomes, sequential ignorability).- When: Use to quantify how much of any revenue change should be attributable to CTR changes under ignorability assumptions.- Example: Estimate natural indirect effect of treatment → CTR → revenue controlling for confounders.- Pros/cons: Sensitive to mediator-outcome confounding; can be combined with sensitivity analysis.

**Structural / econometric models**- What: Build dynamic demand or conversion models (state transitions, latent propensity) to simulate long-run revenue under counterfactual policies.- When: Use when mechanisms are complex, effects delayed, or you need policy simulations (pricing, lifetime value).- Example: Hidden Markov model of browsing → click → add-to-cart → purchase with parameters estimated by ML/EM.- Pros/cons: Powerful for extrapolation; requires model validation and robustness checks.

**Practical recipe**1. Check extended attribution windows and heterogeneity.2. Run targeted funnel experiments or factorial RCTs where feasible.3. Use IV for endogeneity issues; perform mediation analysis with sensitivity bounds.4. If long-run policy decisions depend on estimates, build and validate a structural model and stress-test assumptions.

This combination provides both minimally-assumption checks and principled causal estimates suitable for research-grade conclusions.

MediumTechnical

61 practiced

Explain the CUPED (Controlled-experiment Using Pre-Experiment Data) variance reduction technique. Derive the optimal control variate coefficient theta = Cov(X,Y) / Var(X) given pre-experiment covariate X and outcome Y, and describe how to implement CUPED in an A/B testing pipeline. Discuss failure modes when X is affected by treatment.

Sample Answer

**Brief definition**CUPED uses a pre-experiment covariate X (correlated with outcome Y) as a control variate to reduce variance of estimated treatment effects without bias when X is unaffected by treatment.

**Derivation (optimal theta)**We define adjusted outcome:

text

Y_adjusted = Y - theta * (X - E[X])

Minimise Var(Y_adjusted) w.r.t. theta. Var(Y - theta X) = Var(Y) - 2 theta Cov(X,Y) + theta^2 Var(X). Set derivative = 0:

text

-2 Cov(X,Y) + 2 theta Var(X) = 0

text

theta* = Cov(X,Y) / Var(X)

Intuition: project Y onto X and remove predictable component.

**Implementation in A/B pipeline**- Choose pre-period metric X (same definition as Y) measured before randomization.- Estimate theta* from pooled pre-period data (or from control-only if concerned about leakage).- Compute Y_adjusted per user: Y - theta* (X - mean(X)).- Run A/B analysis (difference-in-means, t-test, regression) on Y_adjusted; compute standard errors accounting for clustering if needed.- Validate: check balance of X post-randomization and pre-period correlations.

**Failure modes when X is affected by treatment**- If X is influenced by treatment (post-treatment covariate), adjustment induces bias (collider/mediation). Theta estimated using pooled data can remove part of treatment effect or create spurious differences.- Remedies: use strictly pre-treatment measurements; if impossible, use instrumental-variable style methods or ANCOVA with caution and sensitivity analyses, or estimate theta using only pre-treatment period or historical cohorts.

HardTechnical

57 practiced

You must estimate direct and spillover effects on a large-scale social graph with highly skewed degree distribution. Propose a randomization and estimation strategy—options include graph cluster randomization, independent-set sampling, or edge-based assignment—and describe how to compute required sample size or number of clusters given target MDEs and desired power.

Sample Answer

**High-level approach (goal)** Estimate direct and spillover effects robustly on a heavy-tailed social graph by (a) choosing a randomization that controls interference, (b) using estimators that separate direct vs. neighborhood exposure, and (c) computing sample size via analytic + simulation power accounting for network dependence and degree skew.

**Design options & recommendation**- Graph cluster randomization (cluster by community / METIS / Louvain): good when spillovers are mostly within communities. Reduces cross-cluster contamination but yields few, large clusters → high design effect if clusters vary in size.- Independent-set sampling (randomly pick nodes with no edges between treated nodes): targets direct effects with minimal immediate spillovers; efficient when feasible on sparse parts of graph.- Edge-based assignment (treat edges or randomize half-edges): useful to estimate dyadic spillovers and local treatment effects; can be combined with node-level assignment.

For a highly skewed degree distribution:- Stratify by degree quantiles; oversample high-degree nodes or treat them separately (e.g., always-control or separate analysis) to reduce variance and bias.- Use degree-normalized exposure models and inverse-probability weights to correct estimator bias.

**Estimators**- Use exposure mapping: define Z_i (own treatment) and G_i (fraction / count of treated neighbors). Fit linear-in-exposures model (or nonparametric contrasts) to estimate direct effect β and spillover γ.- Use Hajek/IPW or generalized propensity scores to account for unequal assignment probabilities from clustering/stratification.- For inference, use randomization-based permutation tests constrained by the design or use clustered robust SEs with network-aware variance estimators (e.g., Aronow-Samii).

**Sample size / number of clusters**Start with a standard difference-in-means formula, then inflate by a design effect (DE) capturing intra-cluster correlation or network-induced dependence.

- Basic per-arm sample size (ignoring dependence):

n0 = ( (z_{1-alpha/2} + z_{1-beta})^2 * (sigma_t^2 + sigma_c^2) ) / (MDE^2)

Plain-English: required independent units per arm given variance and target MDE.

- Cluster trials: replace by number of clusters m using ICC (rho) and average cluster size s:

DE = 1 + (s - 1) * rho
n_effective = n0 * DE
clusters per arm m = n_effective / s

Intuition: dependence within clusters inflates variance; with skewed cluster sizes, use a weighted DE: 1 + (CV_s^2 + 1) * (s_bar - 1) * rho (adjust for cluster size variability).

- For network interference, replace ICC with network correlation parameter (rho_net) estimated from pilot or literature; for estimands involving neighbor exposure, compute variance of exposure G_i and use delta-method to propagate variance into estimator of γ.

**Practical workflow**1. Pilot/simulate on observed graph: define plausible β, γ; simulate assignments under candidate designs; estimate power and bias.2. Choose stratified graph-clusterization minimizing cross-cluster edges; set cluster sizes to balance power vs contamination.3. Compute analytic sample sizes using formulas above with rho_net from pilot, then validate with Monte Carlo sims across parameter grid (β, γ, degree heterogeneity).4. Pre-specify estimators, weighting, and permutation inference consistent with randomization.

**Key trade-offs**- Fewer large clusters → more contamination control but fewer independent units (lower power).- Independent-set reduces interference but may undersample hubs (bias/inefficient for spillovers).- Simulations are essential with heavy tails; analytic approximations guide but do not replace simulation-based power.

Unlock Full Question Bank

Get access to hundreds of Experimentation Methodology and Rigor interview questions and detailed answers.

Join thousands of developers preparing for their dream job.