Experimentation Methodology and Rigor Questions

Focuses on rigorous experimental methodology and advanced testing approaches needed to produce reliable, actionable results. Topics include statistical power and minimum detectable effect trade offs, multiple hypothesis correction, sequential and interim analysis, variance reduction techniques, heterogenous treatment effects, interference and network effects, bias in online experiments, two stage or multi component testing, multivariate designs, experiment velocity versus validity trade offs, and methods to measure business impact beyond proximal metrics. Senior level discussion includes designing frameworks and practices to ensure methodological rigor across teams and examples of how to balance rapid iteration with safeguards to avoid false positives.

HardTechnical

81 practiced

You run a multi-component experiment where components interact non-additively. How would you detect and quantify interaction effects? Propose an efficient sample allocation strategy to detect both main effects and interactions with minimal total sample size.

Sample Answer

Framework — clarify goal: detect/estimate both main effects and non-additive (interaction) effects with minimal total n while controlling Type I/II errors. I recommend factorial-design + targeted power calculations + adaptive allocation.

1) Model and estimand- Fit a linear model with interaction terms: y = μ + ∑αi Xi + ∑βij Xi Xj + ε (or generalized linear model for non-Gaussian outcomes). Test βij for interactions; estimate magnitudes and CIs.

2) Design choices- Full factorial (2^k or multi-level) gives orthogonality and unbiased estimates; variance of an interaction estimate for a 2×2 balanced design: Var(β12) = σ^2(1/n11+1/n12+1/n21+1/n22). For equal n per cell, Var = 4σ^2/n_cell; SE = 2σ/√n_cell. Use this to compute per-cell n for desired power: n_cell = (2σ (z_{1-α/2}+z_{power}) / δ)^2 where δ is the minimum interaction effect size to detect.- If k large, use fractional factorial of appropriate resolution (III–V) to reduce cells while ensuring interactions of interest are estimable (resolution V to separate main effects and two-way interactions).

3) Efficient sample allocation- Start balanced across chosen factorial cells for orthogonality (minimizes covariances). If prior beliefs suggest some interactions are more plausible/important, use Neyman allocation across cells: allocate n_cell ∝ σ_cell * |contrast_sensitivity| to minimize variance of targeted contrasts.- Use D-optimal or A-optimal experimental design when continuous factors or uneven variance: solve for design weights that minimize determinant/trace of parameter covariance.- Use sequential/adaptive approach: run an initial pilot (small balanced allocation) to estimate σ and promising interactions; then re-allocate remaining budget to cells that most reduce variance of the key βij (adaptive optimal allocation). Include pre-specified stopping rules to control multiplicity.

4) Analysis & multiplicity- Use linear-model inference, report interaction effect sizes with CIs. Adjust for multiple tests (BH/FDR or hierarchical testing focusing first on main effects then interactions).- For complex noise or heteroskedasticity, use mixed models or Bayesian hierarchical modelling to borrow strength across cells — reduces required n.

5) Practical recipe (summary)- If k ≤ 4 and budget allows: balanced full factorial; compute per-cell n from SE formula.- If k large: choose fractional factorial (resolution V if two-way interactions matter), run small balanced pilot, then adaptive Neyman/D-optimal re-allocation focusing on promising interactions.- Use model-based inference (frequentist or Bayesian) and correct for multiple comparisons.

This approach balances statistical power for interactions with economy of samples by combining orthogonal design, optimal allocation, and adaptive reallocation guided by pilot estimates.

HardTechnical

76 practiced

Explain intention-to-treat (ITT) versus per-protocol (PP) analysis in the context of model rollouts where some users do not receive the treatment due to system errors. When would ITT be preferred and how would you adjust analysis to estimate Complier Average Causal Effect (CACE)?

Sample Answer

Situation: In model rollouts some users randomized to treatment never actually receive it due to system errors (non-compliance). Choosing analysis affects bias and interpretability.

Definition and contrast:- Intention-to-treat (ITT): Analyze outcomes by original random assignment regardless of whether each user actually received the model. ITT estimates the effect of being assigned to the treatment policy (policy effect including implementation failures). It preserves randomization, is unbiased for the assignment effect, and reflects real-world rollout impact (including failures).- Per-protocol (PP): Analyze only users who adhered to their assigned protocol (e.g., treatment-assigned who actually received it, control-assigned who were not treated). PP targets the effect of actually receiving the treatment but breaks randomization and is susceptible to selection bias because compliance may correlate with outcomes.

When ITT is preferred:- When you want an unbiased estimate of policy effect under real operational conditions (including system errors).- For primary analysis in randomized rollouts, regulatory reporting, and when non-compliance is related to system reliability rather than treatment efficacy.- When preserving randomization and avoiding confounding is critical.

Estimating Complier Average Causal Effect (CACE):- CACE (aka Local Average Treatment Effect) estimates the causal effect among compliers — those who would receive treatment if assigned.- Use random assignment as an instrument for actual treatment received (instrumental variables approach). Key assumptions: random assignment (instrument relevance), exclusion restriction (assignment affects outcome only via receiving treatment), and monotonicity (no “defiers”).- Wald (IV) estimator in binary assignment/treatment: CACE = (E[Y | Z=1] - E[Y | Z=0]) / (E[D | Z=1] - E[D | Z=0]) where Z = assignment, D = actual receipt, Y = outcome.- Implementation steps: 1. Compute ITT numerator: difference in mean outcomes by assignment. 2. Compute compliance rate difference: difference in treatment receipt by assignment. 3. Divide to get CACE; compute standard errors via delta method or two-stage least squares (2SLS).- Use 2SLS for covariate adjustment: first-stage regress D on Z (+ covariates), second-stage regress Y on predicted D.- Check assumptions and report sensitivity analyses (e.g., bounds, violations of exclusion, heterogeneous effects).

Practical considerations:- Report both ITT (primary) and CACE (secondary) with clear assumptions.- If non-compliance is large, CACE helps understand per-user efficacy but depends on IV assumptions—be explicit about plausibility.- Provide confidence intervals and robustness checks (placebo tests, alternative instruments, subgroup analyses).

MediumTechnical

65 practiced

Your experimentation platform runs hundreds of metrics per experiment (clicks, time on site, revenue, etc.). Describe the trade-offs between per-metric Bonferroni correction, BH-FDR, and adjusting analysis pipelines by pre-specifying primary and secondary metrics. Provide a recommended practical policy for the platform.

Sample Answer

Clarify goal: minimize false discoveries that lead to bad product decisions while preserving power to detect meaningful effects. With hundreds of metrics you must balance strict control, interpretability, and operational practicality.

Trade-offs- Per-metric Bonferroni (family-wise error rate control): - Pros: Strong control of any false positive; easy to explain. - Cons: Very conservative when testing many metrics (e.g., alpha/100 → alpha/0.01), large loss of power, likely to miss real effects; impractical for exploratory metrics.- Benjamini–Hochberg FDR: - Pros: Controls expected proportion of false discoveries; much more powerful for large sets of correlated tests; well-suited for exploratory analytics. - Cons: Allows some proportion of false positives (q-level must be chosen); assumptions about dependency affect guarantees; less intuitive to non-statisticians.- Pre-specifying primary vs secondary metrics: - Pros: Enables targeted strict control for business-critical metrics (clear decision rules), preserves power on a small set; aligns experiments with product goals; improves accountability. - Cons: Requires discipline; mis-specification can blind you to important side effects; secondary/exploratory signals must still be analyzed properly.

Recommended practical policy (operational, explainable)1. Pre-specify a small set (1–5) of primary metrics per experiment with business-owner sign-off. Use strong error control for this family: - Apply Holm or Bonferroni across primaries (Holm is uniformly more powerful than Bonferroni). - Report adjusted p-values and minimum detectable effects used in sample-size calc.2. Treat the remaining metrics as exploratory and analyze under FDR: - Run BH-FDR (or BH adaptive) at q = 0.05 for batches of related metrics (grouped by outcome type) rather than across all 100s to respect dependence structure. - Report FDR-adjusted p-values, effect sizes, CIs, and a flag indicating exploratory status.3. Hierarchical approach / gatekeeping: - If any primary metric crosses the primary threshold, allow follow-up confirmatory testing on pre-specified secondary metrics with controlled alpha transfer (gatekeeping) to limit multiplicity inflation.4. Engineering & governance: - Automate adjustment in the platform, show both raw and adjusted summaries, and require an “accepted evidence” workflow for shipping (who signs off, required corroborating metrics). - Log all tests, timestamps, and metric definitions to avoid p-hacking.5. Power & minimum effect sizes: - Require power calculations on primaries; if expected effects are small, consider consolidating correlated metrics or using composite metrics to improve power.6. Education & defaults: - Default experiments to the policy above; provide clear UI labels: Primary (confirmatory), Secondary (confirmatory if gated), Exploratory (FDR). - Train PMs/engineers on interpretation: FDR discoveries need replication or product validation before rollout.

Why this balances trade-offs- Primaries get strict, interpretable decisions with controlled Type I risk.- BH-FDR preserves discovery potential across many exploratory metrics without exploding false positives.- Gatekeeping and logging prevent misuse and ensure that operational decisions rely on robust, pre-agreed evidence.

HardTechnical

117 practiced

Explain fractional factorial designs and orthogonal arrays for scaling multivariate experiments. When is a fractional design appropriate, how do aliasing and confounding affect interpretability, and what are the practical limitations in an online product context?

Sample Answer

Fractional factorial designs and orthogonal arrays let you explore many factors without testing every combination. A full factorial with k binary factors needs 2^k runs; fractional designs use a carefully chosen subset (a 1/2, 1/4, ... fraction) that preserves orthogonality for selected effects. Orthogonal arrays are combinatorial tables (e.g., OA(16,5,2,2)) that ensure balanced, independent estimation of main effects and some interactions.

When appropriate:- Use a fractional design when the number of factors makes a full factorial infeasible (budget, user traffic, time).- Use if you believe higher-order interactions are negligible and main effects (± low-order interactions) are primary drivers.- Useful for screening: identify the few factors that matter before a follow-up full/response-surface study.

Aliasing and confounding:- In a fraction, some effects are aliased (confounded) with others — e.g., a main effect might be aliased with a two-way interaction depending on the resolution. Resolution III: mains confounded with two-way; Resolution IV: mains clear, two-ways may be confounded with each other; Resolution V: two-ways clear.- Aliasing limits interpretability: a detected effect could be any member of its alias set. You must decide assumptions (e.g., negligible interactions) or run follow-ups (fold-over, added runs) to de-alias.

Practical limitations in online product contexts:- Heterogeneous traffic and time-varying effects violate randomization assumptions; need stratification and blocking.- Multiple comparisons and exposure frequency: metric sensitivity and sample size constraints may make detecting small effects hard.- User experience continuity: rapidly exposing users to many variants risks churn; implement targeting rules and safeguards.- Implementation complexity: coordinating feature flags, instrumentation, and ensuring orthogonality across rollout segments is operationally heavy.- Ethical/metric contamination: learning effects, carryover, and interference between users can bias estimates.

Best practices: choose an appropriate resolution OA, pre-specify which interactions you’ll assume negligible, compute power for targeted effects, monitor balance continuously, and plan follow-up runs or fold-over designs to resolve critical aliasing.

MediumTechnical

69 practiced

An A/B test reports p=0.03 and the estimated relative lift is 0.5% on conversion. How would you interpret this result as an AI Engineer? List the immediate diagnostic checks you would run before recommending rollout.

Unlock Full Question Bank

Get access to hundreds of Experimentation Methodology and Rigor interview questions and detailed answers.

Join thousands of developers preparing for their dream job.