A/B Test Design & Statistical Rigor Questions

Designing and statistically defending a controlled online experiment: framing a testable hypothesis, defining control and treatment variants, choosing the randomization unit, setting the primary success metric, and computing sample size, power, and minimum detectable effect. Covers the statistical foundations that make a readout trustworthy, including hypothesis testing, p-values, confidence intervals, statistical vs practical significance, and Type I/II error. Emphasizes avoiding the common pitfalls that invalidate a test, such as peeking, multiple-comparison inflation, underpowered designs, and how test duration and stopping rules affect the validity of conclusions.

MediumTechnical

46 practiced

Design a canary and ramp plan for a feature that reduces page load time but may break older browsers. Propose a staged rollout percentages and durations, list metrics to check at each stage, and define automated stop/rollback rules. Explain reasoning for choose percent steps and minimum observation windows.

Sample Answer

Plan overview:- Goal: Roll out a page-load improvement that may break older browsers. Use feature-flag + canary ramp with automated monitoring and rollback.

Staged rollout:- 0% → 1% (24 hrs) — sanity canary- 1% → 5% (48 hrs)- 5% → 20% (72 hrs)- 20% → 50% (5 days)- 50% → 100% (7 days)

Metrics to monitor (real-time dashboards + 1-hr / 24-hr aggregates):- Front-end: page load time (median, 90th, 99th), Time to First Byte- Errors: JS exceptions by browser/version, resource failed loads, HTTP 4xx/5xx- User behavior: bounce rate, session length, pages per session, conversion funnel (primary KPI)- Business: revenue per user, checkout success rate (if relevant)- Infrastructure: request rate, latency, error budget

Automated stop/rollback rules (evaluate continuously, trigger immediate rollback if any true):- Absolute thresholds: - JS error rate for affected older browsers increases >200% vs baseline and >0.5% absolute - Conversion rate drops >5% relative to baseline for 24-hr rolling window - Bounce rate increases >10% absolute- Statistical/significance rules: - Any metric shows a regression with p < 0.01 and minimum sample size (e.g., 1,000 sessions or 100 events)- Safety floor: - If combined errors + conversion impact cause projected revenue loss >X/day (configurable), rollback immediately

Reasoning for steps & windows:- Small initial % protects users while collecting signal from diverse real traffic. 1% finds high-severity bugs; 5–20% captures browser-segmentation issues. Larger steps need longer windows to observe low-frequency failures and business impact. Minimum windows balance signal-to-noise: 24–72 hrs covers daily usage cycles; 5–7 days at higher ramps capture weekly patterns and edge-case behaviors.

Implementation notes for BI:- Build real-time dashboard with cohort by flag exposure, browser/version, geography.- Automate alerts and a rollback webhook from monitoring to feature-flag system.- Log exposures for postmortem and to compute uplift by segment.

HardTechnical

49 practiced

You need to measure long-term retention impact from a content change but cannot run a long-duration experiment. Propose an analytical strategy using observational causal methods (e.g., synthetic controls, matching, regression discontinuity if applicable) and explain assumptions required to trust the conclusions.

Sample Answer

Approach summary:Use observational causal methods to estimate long-term retention lift from a content change using shorter-term or cross-sectional data. Combine synthetic control (for aggregate-level rollout) with user-level matching / difference-in-differences (DiD) to triangulate results and surface robustness.

Step-by-step strategy:1. Define outcome and horizon proxies: primary outcome = retention at 90/180 days. If you lack that horizon, use validated short-term proxies (e.g., 7/14/30-day retention, engagement depth) and model their relationship with long-term retention using historical cohorts.2. Aggregate-level (Synthetic Control): If content change affected some regions/pages/time windows, build a synthetic control for the treated unit using weighted combination of donor units (pre-treatment period fit on trends and predictors). Estimate post-period gap as causal effect on retention proxy or modeled long-run retention.3. User-level (Matching + DiD): For individual exposure, create propensity scores using pre-treatment covariates (activity, demographics, device, seasonality). Match treated users to controls (e.g., nearest neighbor or coarsened exact matching). Run DiD on matched sample to adjust for time trends.4. Regression-based adjustment: Use outcome regression (e.g., mixed-effects model) including user fixed effects, time fixed effects, and interaction of treatment with time to project long-run retention.5. Validation & sensitivity: (a) Pre-treatment placebo tests and permutation of treatment times. (b) Check pre-trend balance visually and statistically. (c) Vary donor pool, matching calipers, and functional forms. (d) Use bounding methods (Rosenbaum/Frank) to assess unobserved confounding.

Key assumptions to trust conclusions:- No unobserved time-varying confounders correlated with treatment and outcome (synthetic control and DiD rely on stable trends absent treatment).- For matching: conditional ignorability given observed covariates — all major confounders measured and balanced.- For proxy modeling: short-term proxies reliably predict long-term retention (validated on historical cohorts).- SUTVA: no interference between treated and controls (or model spillovers explicitly).

Deliverables and BI integration:- Dashboard showing pre/post trends, synthetic control fit, cohort projections to long-run retention, sensitivity panels (placebo tests, alternative specs).- One-page executive summary with effect estimate range, confidence intervals, and key assumption risks.

Limitations and mitigations:- If strong time-varying confounders exist, supplement with instrumental variables (if valid instrument) or present estimates as bounds. Be transparent about uncertainty; provide policy recommendations that account for risk (e.g., phased rollout + monitoring).

MediumTechnical

51 practiced

Describe the problem of 'peeking' (continuous monitoring) in experiments and compare approaches to allow interim looks while controlling type I error: simple Bonferroni, alpha-spending functions (Pocock, O'Brien-Fleming), group sequential designs, and Bayesian monitoring. For a BI monitoring dashboard where PMs look at results daily, recommend a practical approach and explain how you'd enforce it.

Sample Answer

Peeking (continuous monitoring) inflates type I error because repeated tests against the same null increase the chance of a false positive. To allow interim looks while controlling error, common approaches:

- Simple Bonferroni: divide overall alpha by number of looks. Easy but conservative when many looks; power loss is large for frequent monitoring.- Alpha‑spending functions: allocate total alpha across time as data accrues. Pocock spends more alpha early (roughly equal critical values across looks) — more liberal early, lower final threshold. O’Brien–Fleming spends almost no alpha early and most later — conservative early, minimal power loss overall.- Group sequential designs: pre-specify number/timing of interim analyses and corresponding boundaries (can be derived from Pocock/O’Brien–Fleming). Powerful and statistically principled but requires planning of look schedule and sample sizes.- Bayesian monitoring: uses posterior probabilities or Bayes factors to decide stopping. No frequentist type I control by default but can be calibrated to have controlled false-positive rates; more flexible for continuous monitoring and decision-focused metrics.

Recommendation for a BI dashboard where PMs look daily:- Use an alpha‑spending/group sequential approach with a practical compromise: treat daily dashboard views as informal exploration, but enforce formal stopping only at pre-specified analysis points (e.g., weekly or at n reached). Implement O’Brien–Fleming–style spending if you want to preserve power (conservative early), or Pocock if early detection is prioritized.How to enforce:- Technical: expose only derived “exploratory” metrics on daily dashboards with clear labels; block "formal significance" flags until a scheduled analysis job runs that applies the alpha‑spending boundaries. Automate scheduled hypothesis tests in ETL/analysis pipelines that log look counts and compute adjusted p‑values.- UI/Process: visual cues (orange “exploratory” banner), require a form/approval to convert an exploratory finding into a trial result, and maintain audit logs of all interim looks.- Governance: document monitoring plan in analytics playbook, communicate to PMs, and require replication on subsequent pre-registered checkpoints before product decisions.

This balances statistical rigor with practical day‑to‑day exploration.

MediumTechnical

40 practiced

You're designing an experiment to change weekend pricing where user behavior differs by day-of-week and seasonality is strong. Describe how you would account for seasonality and day-of-week effects in both randomization and analysis. Propose a sampling timeframe and statistical model (e.g., regression with fixed effects) to obtain unbiased estimates.

Sample Answer

Approach summary- Randomize within strata that capture the main cyclical patterns so treatment and control are balanced by day-of-week and seasonal window, then adjust for any residual seasonality in analysis using fixed effects and time trends.

Randomization & sampling timeframe- Unit: user-week (or user-day if pricing applies per visit). Randomize users into treatment/control within each day-of-week stratum (e.g., assign separate random seeds for Sat vs Sun) and within coarse seasonal buckets (month or 2-week window) so each stratum has balanced arms.- Sampling window: at least 8–12 full weeks (preferably spanning months) to capture weekday/weekend cycles and short seasonal shifts; ensure inclusion of any known seasonal events (holidays) or run separate analyses for holiday windows.- Pre-period: collect 2–4 weeks baseline for covariate adjustment and pre-trend checks.- Blocking ensures balance; monitor daily allocation and traffic to detect drift.

Statistical model (unbiased estimate + heterogeneity)- Primary model (continuous outcome, e.g., revenue per user-day) — OLS with clustered SE: Outcome_ijt = β0 + β1 * Treat_i + γ_dow[t] + δ_week[t] + X_i0⋅θ + (Treat_i × γ_dow[t]) + ε_ijt - i = user, j = day, t = time index - γ_dow[t]: day-of-week fixed effects (7 dummies) to absorb systematic day differences - δ_week[t]: week fixed effects (or date fixed effects) to capture seasonality and shocks - X_i0: pre-period covariates (past behavior) to improve precision - Interaction Treat × dow to estimate different weekend effects (important if treatment targets weekend pricing) - Cluster standard errors at the user level (or at the date level if interference across users possible)- Binary outcome: use logistic regression or linear probability model with the same fixed effects; LPM makes fixed effects interpretation simpler.- If rollout is staggered or time-varying adoption, use difference-in-differences with time and day fixed effects.

Checks & robustness- Pre-trend test using baseline: ensure no differential trends.- Balance checks within each block (dow × month).- Sensitivity: alternative seasonality controls (spline on time, month dummies, holiday indicators), and model with user fixed effects to control unobserved time-invariant heterogeneity.- Power: simulate treatment effects accounting for within-user correlation and day-of-week variance to set sample size and test length.- Multiple comparisons: correct when testing many day-specific effects (e.g., Bonferroni or FDR).

Why this works- Blocking by day-of-week and season reduces random imbalance up-front. Day and week/date fixed effects control remaining systematic variability and shocks, interactions reveal heterogeneous weekend responses, and clustering accounts for correlation—together yielding unbiased, precise estimates.

MediumTechnical

51 practiced

You run many tests and often analyze multiple metrics and segments per experiment. Explain approaches to correct for multiple comparisons, including Bonferroni, Holm-Bonferroni, and Benjamini-Hochberg (FDR). For a growth team looking for signals across many noisy metrics, which correction would you recommend and why?

Sample Answer

Start by clarifying the family of comparisons (all metrics x segments you’ll treat as one hypothesis family) and the cost of false positives vs false negatives for your team.

Bonferroni- What: multiply each p-value by m (or divide alpha by m). Controls the family-wise error rate (FWER): probability of any false positive ≤ alpha.- Pros: simple, very conservative, works with arbitrary dependence.- Cons: high false-negative rate when m is large or effects are small.

Holm–Bonferroni- What: step-down procedure: sort p-values, compare the smallest to alpha/m, next to alpha/(m−1), etc.- Pros: still controls FWER but uniformly more powerful (less conservative) than Bonferroni.- Cons: can still be too strict for large-scale screening.

Benjamini–Hochberg (BH, controls FDR)- What: sort p-values p(1) … p(m) and find largest k with p(k) ≤ (k/m)·q; declare first k significant. Controls the expected proportion of false discoveries (FDR).- Pros: much higher power for discovery; appropriate when some false positives are acceptable but you want many true signals.- Cons: controls FDR under independence or certain dependence structures; gives expected proportion rather than absolute guarantee.

Recommendation for a growth BI team- Use a hybrid governance: pre-specify a small set of primary metrics (apply Holm–Bonferroni or no correction) to protect critical decisions; for broad exploratory scans across many noisy metrics/segments, use BH (FDR) at a modest q (e.g., 5–10%) to surface signals.- Complement with practical safeguards: require minimum effect size, replicate promising signals in holdout experiments, report adjusted p-values / q-values on dashboards, and flag exploratory vs confirmatory results.- Optionally consider Storey’s q-value or hierarchical grouping (group-wise BH) to improve power while maintaining interpretability.

This approach balances discovery (finding growth opportunities) with risk control and gives stakeholders clarity about which findings are exploratory vs decision-ready.

Unlock Full Question Bank

Get access to hundreds of A/B Test Design & Statistical Rigor interview questions and detailed answers.

Join thousands of developers preparing for their dream job.