Statistical Foundations for Experimentation Questions

Core statistical concepts and inference needed to design analyze and interpret experiments. Topics include hypothesis testing p values confidence intervals Type One and Type Two errors the relationship between sample size variability and interval width statistical power minimum detectable effect and effect size versus practical significance. Candidates should be able to choose and explain common statistical tests such as t tests and chi square tests contrast Bayesian and frequentist approaches at a conceptual level and describe variance estimation and variance reduction techniques. The topic covers corrections for multiple comparisons sequential testing and the risks of peeking and p hacking common misconceptions about p values and limitations of inference such as confounding and selection bias. Candidates should also be able to translate statistical findings into clear language for non technical stakeholders and explain uncertainty and limitations.

HardSystem Design

54 practiced

Design an automated monitoring system to detect signs of p-hacking or selective reporting across hundreds of experiments (e.g., an excess of p-values just below 0.05, frequent stopping at significance). What metrics, statistical tests (e.g., p-curve, density tests), and operational safeguards would you implement to detect and discourage questionable research practices?

Sample Answer

Requirements & constraints- Detect signals of p-hacking/selective reporting across ~100s experiments daily.- Low false-positive rate; explainable alerts for investigation.- Near real-time ingestion from experimentation platform + metadata (pre-registered analysis plan, stopping rules, interim looks, sample size, randomization, outcome definitions).- Audit trail, dashboards, and operational safeguards (prevention, not just detection).

High-level architectureExperimentation Platform → ETL → Statistical Monitoring Service → Anomaly Scoring/Alerting → Investigation Dashboard & Audit Logs- Storage: event store of experiment metadata and per-analysis results (p-values, effect sizes, SE, sample sizes, timestamps).- Compute: batch and streaming jobs to compute metrics/tests.- Alerting: tiered (info / investigate / escalated).

Core metrics & tests1. P-value distribution checks - P-curve analysis: test whether distribution of significant p-values is right-skewed (indicative of evidential value) versus excess right-bunching just below 0.05. - Binned density test around 0.05: compare counts in bins (e.g., 0.045–0.05 vs 0.05–0.055) using chi-square / bootstrap to detect spikes. - McCrary-style density test for discontinuities near thresholds.2. Excess significance / expected vs observed - Estimate expected proportion significant given observed power (z-curve or Empirical Bayes). Flag higher-than-expected significant rate in groups (team, metric).3. Sequential stopping signals - Look for patterns: p-value trajectory over time; frequent stopping immediately after crossing 0.05; clustering of interim analyses ending at significance. - Implement conditional power and group-sequential modeling to detect improbable stopping patterns.4. Outcome switching & selective reporting - Cross-check reported primary outcome vs preregistered; flag mismatches. - Compare number of reported experiments vs planned; missing null-result reports.5. Meta-analytic diagnostics - Funnel plots for small-study effects, trim-and-fill asymmetry tests.6. Bayesian checks - Bayes factor distributions, S-values (surprisal) to complement p-value anomalies.

Operational scoring & thresholds- Compute per-experiment and cohort anomaly scores combining multiple signals (p-curve skewness, density discontinuity z, excess-significance z, stopping-likelihood). Calibrate with historical data to set thresholds.- Use false discovery rate control across many hypothesis tests (Benjamini-Hochberg) to limit noisy alerts.

Workflow & safeguardsPrevention:- Enforce preregistration and required analysis plan upload before enrollment starts.- Lock analysis code or require tracked branches; discourage post-hoc changes without justification.- Offer built-in sequential analysis templates (group-sequential, alpha spending) with automatic boundaries so stopping is legitimate.Detection & response:- Automatic low-severity alerts for teams (educational) when minor anomalies appear.- Escalation to data governance for persistent or high-score anomalies with audit packet (p-value timeline, preregistration diffs, raw summary stats).- Mandatory reproducibility check for escalated cases (re-run analysis from raw data).- Periodic anonymized reporting to leadership to detect systemic issues.

Implementation considerations & trade-offs- Privacy & legal: limit access to raw data; store hashed experiment identifiers for cross-team aggregation.- False positives: calibrate on simulated p-hacking and honest variations; provide human-in-the-loop review.- Gaming risk: designers may hide practices; keep some monitoring opaque and audit sampling randomized.- Scalability: compute heavy tests (empirical Bayes) in nightly batch; lightweight density checks streaming.

What I’d deliver first (MVP)- Ingestion + archive of p-values/metadata, dashboard with p-value histogram and density-around-0.05 detector, preregistration mismatch checker, alert email workflow. Iterate by adding empirical Bayes, stopping-pattern models, and automated reproducibility playbook.

This design balances statistical rigor (multiple orthogonal tests) with operational safeguards (prevention, education, audit) to both detect and discourage p-hacking in a scalable, explainable way.

HardTechnical

60 practiced

Your organization tracks 5 primary KPIs and 20 secondary metrics. Propose a gatekeeping strategy to control overall false positives while allowing useful exploration of secondary metrics. Explain sequential gatekeeping, hierarchical testing, and trade-offs in terms of power and operational complexity.

Sample Answer

Framework summary (goal): Protect overall Type I rate for business-critical primary KPIs while permitting discovery in 20 secondary metrics with controlled inflation and operationally practical rules.

Proposal — two-layer sequential/hierarchical gatekeeping1. Define Families- Family P: 5 primary KPIs (high priority).- Family S: 20 secondary metrics (exploratory but useful).

2. Alpha allocation & gatekeeping- Global α = 0.05.- Primary family gets an initial allocation αP = 0.04 (reserve αS = 0.01 for secondaries). Use a gate: only if Family P shows significant results (per family test) do we release more α to Family S.- Within Family P, apply a fixed-sequence or Holm step-down across the 5 KPIs ranked by business importance. Fixed-sequence is most powerful when ordering is correct; Holm is safer if order uncertain.

3. Sequential gatekeeping mechanics- Test Family P first. If any hypothesis in P is rejected, transfer pre-specified α (e.g., up to αS) to Family S based on a redistribution rule (fallback procedure or graphical approach) so S’s tests gain power only when primaries indicate signal.- Within Family S, use a hierarchical testing: group secondaries into subfamilies (e.g., UX, performance, engagement) and apply FDR control (Benjamini–Hochberg) within each subfamily after gate opens to allow exploration while controlling expected false discovery proportion.

Operational details and examples- Example: P has KPI1..KPI5. Use Holm with αP=0.04. If KPI2 and KPI4 rejected, release αR = 0.01 to S. In S, run BH at αR across subfamilies or run per-subfamily BH with proportional splitting.- Maintain preregistered ordering, transfer rules, and reporting templates. Log all tests and alpha flows.

Trade-offs- Power: Conservative global control reduces power for secondaries unless primaries show effect. Fixed-sequence maximizes power for higher-priority tests but risks missing signals if order wrong. Using fallback/graphical approach is flexible but slightly reduces power vs unconstrained testing.- Complexity: Graphical/fallback procedures and multi-level allocation are more complex to implement and require governance, code for alpha bookkeeping, and analyst training. Simpler: single gate (only open S if any P significant) easier but less nuanced.- Practical compromise: Use automated pipeline (alpha accounting library), preregistered families/subfamilies, and a reporting rubric that separates confirmatory (gate-controlled) results from purely exploratory (uncorrected or FDR-controlled) findings.

Why this is appropriate- Protects business-critical KPIs first, provides controlled exploration for secondaries, and is implementable with standard statistical tools (Holm, BH, graphical/fallback gatekeeping). It balances Type I control, interpretability, and usable discovery.

HardTechnical

69 practiced

A retrospective analysis shows users exposed to a promotion have higher revenue, but assignment wasn't randomized. Describe causal inference strategies you could use to estimate the treatment effect while accounting for confounding: propensity score matching, inverse probability weighting, and instrumental variables. For each, outline key assumptions and diagnostics you'd run.

Sample Answer

Brief framing: Because treatment wasn't randomized, we must rely on methods that make different assumptions to recover causal effects. I'll outline each approach, its key identifying assumptions, practical steps, and diagnostics.

1) Propensity Score Matching (PSM)- Idea: model P(Treatment|X) and match treated to similar controls on the propensity score to approximate randomization on observables.- Key assumptions: Conditional ignorability (no unmeasured confounders given X), positivity/overlap (0<e(X)<1).- Practical steps: fit logistic/boosted tree propensity model; choose matching algorithm (1:1, caliper, nearest, kernel); estimate ATT or ATE.- Diagnostics: check covariate balance pre/post via standardized mean differences (SMD) and love plots; inspect propensity score overlap/density plots; assess matched sample size and effective sample. Sensitivity: Rosenbaum bounds or tipping-point analyses for unobserved confounding; vary caliper/algorithm and show robustness.

2) Inverse Probability Weighting (IPW)- Idea: weight units by 1/e(X) or 1/(1-e(X)) so weighted sample mimics randomized population.- Key assumptions: same ignorability and positivity as PSM; correct propensity model specification.- Practical steps: compute stabilized weights to reduce variance; use weighted regression to estimate effect and robust SEs; possibly doubly robust estimation (AIPW).- Diagnostics: examine weight distribution (max weight, variance), effective sample size, truncation/trimming thresholds (e.g., trim extreme weights at 99th percentile); check covariate balance after weighting (weighted SMDs); compare results with doubly robust estimator.

3) Instrumental Variables (IV)- Idea: use instrument Z that affects assignment but only affects outcome through treatment; estimate causal effect for compliers (LATE).- Key assumptions: relevance (Z strongly predicts treatment), exclusion restriction (no direct path Z→Y), independence (Z independent of potential outcomes), monotonicity (no defiers) often required.- Practical steps: use two-stage least squares (2SLS) or IV methods (for binary treatment, use control function or TSLS with linear probability caution); estimate LATE and report first-stage F-stat (>10 suggests strong instrument).- Diagnostics: first-stage strength (F-stat), overidentification tests (Sargan/Hansen) if multiple instruments to test exclusion, placebo tests (check instrument balance on pre-treatment covariates), falsification outcomes unaffected by treatment. Sensitivity: explore bounds, alternative instruments, and plausibility arguments for exclusion.

Cross-cutting practices:- Pre-register analysis plan, include rich covariates and baseline outcomes, report ATT vs ATE distinction, present multiple methods and triangulate results.- Run placebo/time-window checks, subgroup analyses, and sensitivity/tipping-point analyses to quantify robustness to unmeasured confounding.

HardSystem Design

68 practiced

Design a sequential testing plan that controls Type I error at 5% but allows continuous daily monitoring. Compare and justify use of alpha-spending (e.g., O'Brien–Fleming), Pocock boundaries, and Bayesian monitoring. Describe implementation details and how you'd integrate this into an automated monitoring system.

Sample Answer

Requirements:- Control family-wise Type I error ≤ 0.05 under continuous daily looks.- Allow near-real-time stopping for efficacy or futility.- Scalable, auditable, automated daily pipeline with alerts and dashboard.

High-level design:User data ingestion → Preprocessing & test-statistic engine → Sequential boundary evaluator → Logging/audit store → Dashboard & alerting.

Key components:1. Ingestion & ETL: daily rollup, validation, de-duplication. Store immutable snapshots for reproducibility.2. Test-statistic engine: computes chosen test statistic (e.g., Z) and variance using pre-specified estimators; supports stratification and covariate adjustment.3. Boundary module: - Alpha-spending (O’Brien–Fleming): conservative early looks, alpha(t) defined by spending function; implement numeric inversion to get rejection critical values at each information fraction. - Pocock: equal critical value across looks — easier but uses more alpha early so lower final power for fixed max looks. - Bayesian monitoring: posterior probability thresholds (e.g., P(effect>0|data)>0.995) or Bayes factor; no strict frequentist alpha control but can be calibrated to have similar Type I via simulation.4. Simulator & calibration: perform pre-deployment simulation under H0/H1 to set spending function parameters, max information, and assess Type I/II, accounting for dependence across looks.5. Orchestration & automation: scheduled job (daily) triggers pipeline, records decisions, posts alerts; include manual override workflow and required approvals for stopping decisions.6. Audit & reproducibility: store code version, random seeds, snapshots, and all intermediate results.

Comparison & justification:- O’Brien–Fleming: best when many looks are planned; highly conservative early—low false alarm early while retaining power; recommended when continuous monitoring but desire strict 5% control.- Pocock: simpler, easier to implement; more liberal early — increases chance of early stop but reduces final power for same max information; choose if early decisions are prioritized.- Bayesian: flexible, naturally handles continuous looks without alpha inflation, intuitive posterior statements; to guarantee frequentist 5% control requires calibration via simulation or using hybrid decision rules (e.g., require posterior + frequentist boundary).

Implementation details:- Pre-specify max information or maximum sample/time horizon; continuous daily monitoring requires mapping cumulative information fraction each day.- Compute critical Z_t from chosen spending function: use alpha_t = s(t) then critical = inverse_normal(1 - alpha_t/2) for two-sided.- For Bayesian: define priors, inference method (closed-form conjugate or MCMC/variational), and precompute thresholds via Monte Carlo to control Type I.

Operational considerations:- Maintain change-control and approvals; lock analysis script once trial starts.- Monitoring dashboard with plots: Z over time, spending curve, posterior probability, decision suggestions, and full audit trail.- Testing: unit tests for stats code, integration tests with synthetic streams, daily dry-run mode.- Alerts: tiered (informational, recommend stop, require manual review). Provide explanation and reproducible notebook link in alerts.

Trade-offs:- OBF + alpha-spending best for strict Type I control with continuous looks.- Pocock simpler but more aggressive early.- Bayesian provides interpretability and operational simplicity for continuous monitoring but needs calibration for regulatory/frequentist guarantees.

This design balances statistical rigor (alpha control via spending), operational automation, auditability, and flexibility to support Bayesian or frequentist policies depending on stakeholder needs.

HardTechnical

66 practiced

You receive experimental results showing a small positive effect estimate with wide confidence intervals. Formulate a decision policy that combines statistical evidence, expected business value, rollout cost, and risk tolerance. Explain how you would compute or approximate the Expected Value of Sample Information (EVSI) to decide whether to run a larger follow-up experiment or to roll out.

Unlock Full Question Bank

Get access to hundreds of Statistical Foundations for Experimentation interview questions and detailed answers.

Join thousands of developers preparing for their dream job.