A/B Test Design & Statistical Rigor Questions

Designing and statistically defending a controlled online experiment: framing a testable hypothesis, defining control and treatment variants, choosing the randomization unit, setting the primary success metric, and computing sample size, power, and minimum detectable effect. Covers the statistical foundations that make a readout trustworthy, including hypothesis testing, p-values, confidence intervals, statistical vs practical significance, and Type I/II error. Emphasizes avoiding the common pitfalls that invalidate a test, such as peeking, multiple-comparison inflation, underpowered designs, and how test duration and stopping rules affect the validity of conclusions.

MediumTechnical

45 practiced

Describe how you would detect novelty effects or time-varying treatment effects in an experiment that runs for multiple weeks. Which visualizations, models, and statistical checks would you run to distinguish a genuine effect from novelty or novelty decay? Explain how you might adjust experiment duration or analysis to handle these dynamics.

Sample Answer

**Approach summary**Describe and test time-varying treatment effects with visual diagnostics, flexible time models, and robustness checks to separate a true treatment signal from short-lived novelty.

**Visualizations**- Daily/weekly ATE with 95% CIs (points + error bars) and a smoothed trend (LOESS/GAM).- Cumulative incremental lift over time to see stabilization vs front-loaded spikes.- Separate plots for user cohorts (new vs returning) and by engagement deciles.- Heatmap of effect by week-of-exposure × cohort to spot decay patterns.

**Models & tests**- Mixed-effects model with time and time×treatment interaction: outcome ~ treatment * f(time) + (1 | user). Interaction p-value tests time-varying effect.- GAM or spline for nonparametric time dynamics and treatment×spline(time).- Exponential decay model: effect(t) = beta * exp(-lambda * t); estimate lambda to quantify decay.- Bayesian structural time-series / CausalImpact to control trends and detect transient effects.- Changepoint detection (CUSUM, PELT) on daily ATE to find abrupt novelty spikes.- Placebo tests: assign fake treatment dates or shuffle treatment labels by day to check false positives.

**Statistical checks**- Pre-treatment parallel trends and balance by time strata.- Permutation tests aggregated by time-blocks to preserve autocorrelation.- Check heterogeneity: is effect concentrated in first-week users?- Analyze retention of effect after excluding first N days (sensitivity).

**Adjusting duration & analysis**- Pre-specify primary analysis window; require run until cumulative ATE CI stabilizes or decay parameter converges.- If strong novelty detected, extend experiment until post-novelty plateau (use sequential monitoring with alpha spending).- Segment users: estimate long-term vs short-term cohorts separately; report both.- Use power calculations for time-to-stabilize: simulate decay curves to set minimal duration.

**Reasoning**Combining visuals, flexible models, and placebo/changepoint checks distinguishes transient novelty from sustained treatment effects and guides duration decisions grounded in observed stabilization or estimated decay rates.

MediumSystem Design

41 practiced

Design an experiment to evaluate a personalized recommendation algorithm. The algorithm tailors ranking per user. Discuss randomization choices: should you randomize model assignment per user, randomize per impression, or randomize ranking perturbations? Address stratification, consistency across sessions, SUTVA concerns, logging needs for unbiased ATE/CATE estimation, and offline simulation considerations.

Sample Answer

**Goal & metrics**- Estimate ATE (average effect on click/conversion/engagement) and CATE (heterogeneous effects by user segments) for a personalized ranker vs baseline.

**Randomization choices**- Per-user assignment (recommended primary): randomize users to Model A or B and keep assignment consistent across sessions. This yields unbiased ATE under stable assignment, preserves user experience, and avoids interference across treatments.- Per-impression assignment: higher statistical power but violates personalization consistency and increases SUTVA risk (user behavior can change within-session).- Ranking-perturbation (bandit-style): useful for online learning/exploration; harder to interpret causal ATE without careful logging and may bias estimates.

**Stratification & blocking**- Stratify randomization on baseline covariates (e.g., past engagement decile, device, geography) or use blocking to ensure balance and improve precision for CATE.

**Consistency & SUTVA**- Ensure assignment persistence per user (cookie or account id) to reduce within-user interference. Explicitly consider spillovers (e.g., if one user’s feed affects others), and if present instrument for interference or adopt cluster-level randomization.

**Logging for unbiased estimation**- Log deterministic assignment id, timestamp, user id, session id, full presented ranking, model version, probability of any randomization, candidate-level features, and observed outcomes. Record seeds for any randomized ranking perturbations to enable inverse-propensity weighting and reproducible offline replay.

**Estimation**- Use ITT and inverse-propensity-weighted estimators for ATE. For CATE, fit X-learner/causal forest using logged features and activities; include robust standard errors clustered by user.

**Offline simulation & validation**- Replay experiments on logged policy data (with propensities) to estimate variance and bias; run counterfactual simulation (IPS & SNIPS) and synthetic users to stress-test SUTVA violations. Conduct A/A tests to validate randomization pipeline.

Outcome: prioritize per-user randomized assignment with stratification, comprehensive logging (including propensities and full rankings), cluster-aware analysis, and offline replay to ensure unbiased, actionable ATE/CATE estimates.

HardTechnical

48 practiced

Propose an organization-wide experimentation governance model for a growing company. Include roles and responsibilities (data scientists, product, engineers, QA), pre-registration standards, experiment cataloging, metric-definition registry, peer review process, reproducibility requirements, and mechanisms for sharing learnings and failed experiments.

Sample Answer

**Overview — my goal**Design a lightweight, scalable experimentation governance that balances scientific rigor with product velocity so ML teams can safely test, learn, and ship.

**Roles & responsibilities**- Applied Scientists: define hypotheses, statistical design, metric definitions, run analyses, document reproducibility artifacts (data snapshot, seeds, code).- Product Managers: define business hypothesis, success criteria, risk tolerances, rollout plan.- Engineers: implement treatment, instrumentation, rollout gating, ensure testability and observability.- QA: validate treatment correctness, data collection, and edge-case behaviors.- Experiment Review Board (cross-functional): approves high-risk experiments and enforces standards.

**Pre-registration standards**- Mandatory preregistration form: hypothesis, primary/secondary metrics (referenced by metric-registry IDs), sample size & power calc, guardrails, duration, segmentation plan, rollback criteria, data access paths, and reproducibility checklist.

**Experiment catalog & metric-registry**- Central catalog (searchable) with metadata, status, owner, versioned code links, and results.- Metric registry: canonical definitions, SQL/feature code snippets, ownership, expected distribution and sensitivity, alert thresholds.

**Peer review & reproducibility**- Lightweight peer review for every experiment: code + analysis reviewed by at least one scientist and one engineer before kickoff.- Reproducibility artifacts required: data snapshot or deterministic data pipeline, notebook + unit tests, seed control, Dockerized analysis environment, CI run that reproduces key figures.

**Sharing learnings**- Post-mortem template mandatory for every experiment (success/failure, effect sizes, bias analysis, next steps).- Quarterly “labs” forum to present surprising failures and learnings; internal searchable lessons repository tagging failed hypotheses and implementation lessons to reduce repeat mistakes.

This framework preserves rapid iteration while enforcing scientific standards appropriate for production ML decisions.

EasyTechnical

47 practiced

Compare and contrast three bucket assignment strategies: hashing-based deterministic bucketing, stateful server-side assignment, and client-side cookie assignment. Discuss pros and cons for cross-device consistency, ability to reassign users later, handling of experiment overlap, and impact on analysis validity.

Sample Answer

**Overview (short)** I'll compare hashing-based deterministic bucketing, stateful server-side assignment, and client-side cookie assignment across four dimensions: cross-device consistency, reassignability, experiment overlap, and analysis validity.

**Hashing-based deterministic bucketing** - Cross-device consistency: High if you hash a persistent user identifier (user_id); low if you hash device_id. Use stable salt/versioning to preserve determinism. - Reassignability: Hard — changing bucket boundaries or salt can remap many users; use migration flags or layered mapping to reassign gradually. - Experiment overlap: Easy to coordinate by namespace + hash ranges; deterministic exclusion rules work well. - Analysis validity: Strong for intent-to-treat when ID is stable; watch for ID leakage and missing IDs causing bias.

**Stateful server-side assignment** - Cross-device consistency: Excellent — one canonical store maps user -> treatment across devices. - Reassignability: Flexible — you can update assignments atomically for targeted segments or rollouts. - Experiment overlap: Requires central orchestration to prevent conflicts; easier to enforce complex mutual-exclusion rules. - Analysis validity: High control reduces misclassification, but store errors or propagation delays can introduce bias; needs guarantees on read/write atomicity and logging.

**Client-side cookie assignment** - Cross-device consistency: Poor — cookies are per-browser/device unless combined with login. - Reassignability: Limited — cannot centrally force change; relies on cookie expiry or client script to reassign. - Experiment overlap: Hard to coordinate; overlapping experiments can occur unless client enforces rules fetched from server. - Analysis validity: Risky — cookie loss, ad-blockers, or multiple devices create attrition and measurement error. Use for low-risk UI tweaks only.

**Practical guidance (Applied Scientist lens)** - Prefer server-side/stateful for experiments needing cross-device consistency, precise targeting, and low measurement error. - Use hashing for scalable, low-latency assignment when stable user IDs exist and you can tolerate some remapping complexity. - Use client cookies only for lightweight or exploratory tests; always instrument robust logging and post-hoc assignment reconstruction where possible.

EasyBehavioral

42 practiced

Behavioral: Tell me about a time when you advocated for stricter experimental rigor (e.g., pre-registration, holdouts, correction for multiple tests) on a team that preferred rapid iteration. What was the situation, what actions did you take, and what were the outcomes?

Unlock Full Question Bank

Get access to hundreds of A/B Test Design & Statistical Rigor interview questions and detailed answers.

Join thousands of developers preparing for their dream job.