Experimentation Methodology and Rigor Questions

Focuses on rigorous experimental methodology and advanced testing approaches needed to produce reliable, actionable results. Topics include statistical power and minimum detectable effect trade offs, multiple hypothesis correction, sequential and interim analysis, variance reduction techniques, heterogenous treatment effects, interference and network effects, bias in online experiments, two stage or multi component testing, multivariate designs, experiment velocity versus validity trade offs, and methods to measure business impact beyond proximal metrics. Senior level discussion includes designing frameworks and practices to ensure methodological rigor across teams and examples of how to balance rapid iteration with safeguards to avoid false positives.

HardTechnical

68 practiced

Explain the design effect and how to adjust sample size for cluster-randomized trials. Define the intraclass correlation coefficient (ICC), derive the design effect 1 + (m - 1) * ICC where m is cluster size, and show with numbers how cluster randomization inflates required sample size compared to individual randomization.

Sample Answer

Definition and intuition- The intraclass correlation coefficient (ICC, commonly ρ) measures similarity of outcomes within the same cluster: ρ = σ_b^2 / (σ_b^2 + σ_w^2), where σ_b^2 is the between-cluster variance and σ_w^2 the within-cluster variance. ρ ranges 0 (no within-cluster correlation) to 1 (identical within cluster).- The design effect (DE) quantifies how much clustering inflates the variance (and thus required sample size) relative to individual randomization.

Derivation of DE = 1 + (m − 1)ρ- Consider a cluster of size m. For an individual outcome Y_ij = μ + b_j + e_ij with Var(b_j)=σ_b^2, Var(e_ij)=σ_w^2 and Cov(Y_ij,Y_ik)=σ_b^2 for i≠k.- The variance of the cluster mean Ȳ_j = (1/m) Σ_i Y_ij is: Var(Ȳ_j) = (1/m^2)[ m(σ_b^2+σ_w^2) + m(m−1)σ_b^2 ] = (σ_b^2+σ_w^2)/m + (m−1)σ_b^2/m = (σ_b^2+σ_w^2)/m * [1 + (m−1)σ_b^2/(σ_b^2+σ_w^2)]- Replace ρ = σ_b^2/(σ_b^2+σ_w^2) to get: Var(Ȳ_j) = (σ_total^2 / m) * [1 + (m−1)ρ]- Compared to an independent sample of m individuals (variance σ_total^2/m), the multiplicative inflation is DE = 1 + (m−1)ρ.

Adjusting sample size- For a target sample size n_individual per arm under individual randomization, the required number of individual observations under cluster randomization is n_cr = DE * n_individual.- More usefully, if clusters have fixed size m, clusters per arm = (DE * n_individual) / m.

Numeric examples- Suppose an individually randomized trial needs n_individual = 200 per arm.- Case A: m = 10, ρ = 0.02 → DE = 1 + (10−1)*0.02 = 1 + 0.18 = 1.18. - Required individuals per arm: 200 * 1.18 = 236. - Clusters per arm = 236 / 10 ≈ 23.6 → 24 clusters of size 10 per arm.- Case B: m = 10, ρ = 0.05 → DE = 1 + 9*0.05 = 1.45. - Required individuals per arm: 200 * 1.45 = 290. - Clusters per arm = 29 clusters per arm.- Larger m or larger ρ increases DE substantially; e.g., m=20, ρ=0.05 → DE=1+19*0.05=1.95 (~double sample).

Practical notes- ICCs in practice are often small (0.01–0.05) but still materially increase required sample and cost.- If cluster sizes vary, use the effective m ≈ (Σm_i)^2 / (k Σ m_i^2) or adjust DE using mean and coefficient of variation of cluster sizes.- When planning, estimate ICC from pilot data or literature and perform sensitivity analyses across plausible ρ and m to ensure adequate power.

MediumTechnical

57 practiced

You need to measure the long-term retention effect of a new onboarding flow expected to influence 3-month retention. Design an experiment: describe cohort definitions, metric(s) to use (short-term and long-term), sample size/time horizon trade-offs, how to handle seasonality and delayed effects, and how you'd analyze and report results to product and finance stakeholders.

Sample Answer

Situation: Product plans a new onboarding flow expected to improve 3‑month retention. I’d design an A/B experiment that measures both short and long-term effects and produces actionable results for product and finance.

Cohort definitions:- Randomize new users at signup into Control (current onboarding) and Treatment (new flow) with user-level holdout. Record assignment, signup timestamp, acquisition source, device, and country.- Define cohorts by signup week to enable cohort analysis and control seasonality.

Metrics:- Short-term (leading indicators): Day-1 activation rate (completed key steps), Day-7 retention, percent completing onboarding milestones. These provide early signals.- Long-term (primary): 3‑month retention (binary: user returns and performs key action within 90±7 days) and survival curve / time-to-churn (Kaplan‑Meier) to capture retention dynamics and delayed effects.- Business metrics for finance: LTV uplift projection over 12 months using cohort-level revenue per user and retention curves.

Sample size / time horizon trade-offs:- Power to detect a realistic uplift (e.g., 5% relative increase in 3‑month retention) drives sample size; since 3‑month outcome is slow, compute launch-size required or run the experiment for fewer users but longer time. Use interim power calculations on Day‑7 leading metrics to stop early for futility or safety, but avoid peeking on primary 3‑month metric to control Type I error.

Handling seasonality and delayed effects:- Run for multiple signup weeks (≥4–8) and stratify randomization by source/geo to balance seasonality.- Use cohort-week fixed effects in regression and include time covariates.- Model delayed effects with survival analysis and time-varying hazard models; check for treatment effect emergence after initial weeks.

Analysis and reporting:- Primary analysis: pre-specified intent-to-treat comparison of 3‑month retention using logistic regression adjusted for stratification covariates; present absolute and relative lift, 95% CIs, and p-values. Complement with Kaplan‑Meier plots and Cox proportional hazards for time-to-churn.- Secondary: lift on leading metrics, funnel conversion, and projected LTV impact (bootstrapped).- Sensitivity checks: per-protocol analysis, heterogeneity by channel/device, and adjustment for missing data (inverse probability weighting).- Reporting to product and finance: concise executive summary (primary result, CI, p-value, and projected revenue impact), visualization (retention curves, weekly cohort heatmap), recommended decision (rollout, iterate, or kill) with risk assessment and expected ROI and timeline. Include technical appendix with methodology, power calculations, and robustness checks.

MediumTechnical

75 practiced

Describe sequential analysis and alpha-spending approaches for repeated looks at an experiment. Explain the differences between O'Brien-Fleming and Pocock boundaries, and why alpha spending controls Type I error. In what production situations would you allow interim looks, and how would you document stopping rules?

Sample Answer

Sequential analysis allows evaluating an experiment at multiple interim looks without inflating the Type I error. Alpha-spending frameworks formalize how much of the total Type I error (α, e.g., 0.05) is "spent" at each look via a spending function (Lan–DeMets), so the cumulative chance of a false positive across all looks stays ≤ α.

Key ideas:- Fixed-sample testing assumes one look; repeated looks require adjusted critical boundaries.- Alpha-spending defines thresholds for test statistics at each look so the sum of incremental α’s equals the nominal α.

O'Brien–Fleming vs. Pocock:- O’Brien–Fleming: very conservative early (very high critical z for early looks), almost equal to fixed-sample at the final look. Good when you want to avoid early false positives and prefer evidence accumulation.- Pocock: uses a constant (relatively liberal) boundary across looks; easier to stop early but costs power at the final analysis.- Trade-off: O’Brien–Fleming minimizes type I risk early at cost of requiring stronger early evidence; Pocock balances stopping probability across looks.

Why alpha-spending controls Type I error:- Spending functions map look times to cumulative α such that the union bound over sequential tests is controlled; sequential boundaries are derived so the overall probability of ever crossing a boundary under the null equals the pre-specified α.

Production usage (when to allow interim looks):- High-revenue features where early wins/losses affect business risk (pricing, checkout flow).- Safety-critical or regulatory contexts where early stopping for harm is required.- Long-running experiments where waiting wastes opportunity or resources.- Fast-fail validation for expensive treatments (large infra cost).

How to document stopping rules (must be pre-specified and versioned):- State the objective and primary metric, null/alternative, and target α and power.- Specify alpha-spending function (e.g., Lan–DeMets with OBF-like spending) and maximum number of interim looks or calendar schedule (or information fraction rule).- Define exact statistical stopping criteria (test statistic thresholds, sample sizes or information fractions at each look).- Describe permissible adaptations (sample-size re-estimation, multiple comparisons handling), decision actions on stopping, and roles/approval flow.- Log each interim analysis, code, random seeds, and keep analyses blinded where appropriate; preregister in experiment tracking system and include business owners in sign-off.

This approach preserves statistical validity while aligning experimental practice with business risk and operational constraints.

HardTechnical

73 practiced

Given a corpus of past experiments, design an approach to detect signs of p-hacking or data-snooping (e.g., many p-values just below 0.05, suspicious streaks of positives). Describe statistical tests or visualizations you would use, and propose retrospective corrections and organizational safeguards to reduce p-hacking.

Sample Answer

Framework / goal- Detect statistical signs of p-hacking in a corpus of past experiments, quantify likelihood, propose retrospective corrections and operational safeguards.

1) Data ingestion & preprocessing- Collect per-experiment: test statistics, p-values, sample sizes, effect sizes, direction, two- vs one-sided tests, timestamps, analyst, code repo links, stopping rules.- Standardize (convert reported p-values to z-scores or t-statistics when possible) and exclude clearly non-comparable tests (different hypotheses).

2) Exploratory visualizations & diagnostics- P-value histogram (0–1, fine bins): look for spikes just below 0.05 and depletion above.- Zoomed histogram in [0, 0.1] to reveal bunching around 0.049–0.05.- P-curve (distribution of significant p-values): under honest effects, should be right-skewed when true effects exist; flat or left-skew suggests p-hacking or selective reporting (use Simonsohn et al. tests).- Z-curve / z-density: fit a mixture model of null and non-null z-scores (Schimmack’s z-curve) to estimate expected discovery rate and selection bias.- Q-Q plot of p-values against Uniform(0,1): departures indicate selection.- Time-series of positive result rate per analyst/experimenter and streak analysis (runs test, CUSUM) to find suspicious clustering.- Funnel plots of effect size vs standard error: asymmetry may indicate small-study effects or selective reporting.

3) Statistical tests / metrics- Excess of p-values just under threshold: compare observed count in (0.045,0.05] vs expectation under smooth null using local density estimation; use bootstrapped CI and binomial test.- P-curve test (right-skewness): test significance of evidential value; software implementations exist.- Caliper test: likelihood-ratio comparing smooth density vs bump at threshold.- Estimation of pi0 (proportion true nulls) with Storey’s estimator; compare to observed discovery rate to estimate selection bias.- Meta-analytic heterogeneity / publication bias tests: Egger’s regression on funnel asymmetry.- False discovery rate recalibration with Empirical Bayes (Efron): estimate local fdr for each study.- For streaks: permutation of sign labels across time to get distribution of longest run; compute p-value for observed run length.

4) Retrospective corrections- Weighting by selection model: apply trim-and-fill or selection models to adjust pooled effect sizes.- Empirical Bayes deconvolution to shrink effect sizes and adjust reported p-values / local fdr.- Report adjusted FDR and provide corrected confidence intervals accounting for estimated selection bias.- Flagged-experiment label: annotate experiments with an audit score (composite of p-bunching, suspicious runs, lack of preregistration, atypical stopping rules).

5) Organizational safeguards- Mandatory preregistration of analysis plan and primary outcomes in internal registry before data access.- Enforce one-click provenance: store code, seed, dataset snapshot, and notebook in immutable audit logs (git + artifact storage).- Require specification of stopping rules and power calculations; block optional interim peeking unless group-sequential plan documented.- Blind analysis where possible (e.g., permute labels until pipeline fixed).- Code review & statistical review for experiments that will be used for business decisions; random audits of a sample of experiments.- Training on multiple testing, p-hacking risks, and incentives aligned to quality (reward replication & preregistration).- Dashboard monitoring: automated alerts for suspicious patterns (high density just below thresholds, unusually high positive rate for an analyst).

6) Implementation & metrics of success- Build a scheduled pipeline producing p-value diagnostics, z-curve estimates, analyst streak reports, and audit scores.- KPIs: reduction in flagged experiments over time, increased preregistration rate, stabilization of p-value distribution toward expected shapes, improved replication rate.

Why this works- Combines descriptive, inferential, and model-based methods to detect both blatant and subtle selection.- Retrospective adjustments mitigate bias in aggregated estimates; safeguards change incentives and reduce future p-hacking.

HardSystem Design

68 practiced

Design a constrained multi-armed bandit algorithm where the reward is revenue and the organization requires: (1) a cap on maximum regret compared to a safe baseline, (2) fairness across user segments, and (3) the ability to pause and audit decisions. Describe algorithmic choices (e.g., conservative Thompson sampling, constrained optimization), architecture, and monitoring.

Sample Answer

Requirements & constraints:- Functional: maximize revenue per impression while ensuring decisions never exceed a predefined regret cap relative to a safe baseline policy, guarantee fairness across user segments (e.g., parity in conversion lift or exposure), and support pausing + full auditability.- Non‑functional: low-latency decision service (ms), reproducible logs, explainability for audits.

Algorithmic design:1. Core learner: Bayesian bandit (Thompson Sampling) optimizing revenue per arm with priors seeded from historical data for fast convergence.2. Safety layer (regret cap): Conservative Thompson Sampling (CTS) — draw samples, but constrain selection to arms whose posterior-estimated expected revenue >= baseline revenue − allowable regret margin with high confidence (e.g., lower credible bound > baseline − δ). If no arm satisfies, fallback to baseline.3. Fairness constraints: incorporate group-aware post-processing or constrained optimization: - Maintain per-segment posteriors; compute expected uplift per segment. - Solve a small constrained allocation LP each decision epoch: maximize expected revenue subject to (a) expected regret budget over horizon, (b) fairness constraints (e.g., minimum exposure or minimum relative uplift per segment), (c) capacity constraints. - Alternatively use Lagrangian relaxation to combine fairness penalty into acquisition function; tune multiplier via dual updates to enforce constraints.4. Auditability & pause: every decision includes context, posterior samples, constraint checks, LP solution, chosen arm ID, and seed; write to immutable event store.

Architecture:- Data layer: streaming events (impressions, conversions, revenue) → preprocessing (feature enrichment, cohort/segment mapping) → real-time metric aggregator.- Model service: Thompson Sampling posterior updater (online/batch hybrid), priors ingest. Exposes decision API.- Safety & Allocator service: consumes posterior summaries, baseline estimates, business-configured regret cap and fairness constraints; runs constraint solver (LP or tiny convex solver) and returns safe allocation.- Decision router: enforces latency SLAs; caches latest allocations; calls fallback baseline if paused or safety fails.- Storage & audit: append-only event log (Kafka + S3), snapshot DB for posteriors, versioned model/artifact registry.- Control plane: UI for setting caps, pausing/unpausing, toggling fairness constraints, and running offline audits; role-based access control.

Monitoring & observability:- Metrics: cumulative regret vs baseline, per-segment revenue uplift, exposure share per segment, conversion rate, posterior uncertainty, fraction of decisions falling back to baseline, solver constraint violations.- Alerts: automatic alert when cumulative regret approaches cap threshold (e.g., 80%), any fairness constraint breach > tolerance, model drift signal (posterior variance increasing), or data pipeline lag.- Dashboards: live cohort-level revenue and fairness heatmaps; downloadable audit reports for any time window (decisions + explanations + raw events).- Offline validation: A/B sandboxing and synthetic stress tests to verify caps under adversarial distributions; backtest regret & fairness on historical logs.- Explainability: per-decision short rationale (baseline comparison, constraint binding status) for auditors.

Operational considerations:- Warm start with historical logging and offline simulation to set priors and tune δ and Lagrange multipliers.- Conservative default: tighter regret margin at launch, loosen after proven behavior.- Legal & ethics: store PII separately and use hashed cohort IDs in audit reports.- Testing: holdout segments, chaos tests (pause/unavailable services), and safety fault-injection.

Why this design:- CTS + constrained allocator gives provable safety (never exceed regret cap with configured confidence) while allowing exploration where safe.- Group-aware posteriors + constrained optimization enforce fairness explicitly rather than heuristically.- Modular architecture separates learning, safety, and serving for auditability, easier certification, and controlled pausing.

Unlock Full Question Bank

Get access to hundreds of Experimentation Methodology and Rigor interview questions and detailed answers.

Join thousands of developers preparing for their dream job.