Experiment Design and Execution Questions

Covers end to end design and execution of experiments and A B tests, including identifying high value hypotheses, defining treatment variants and control, ensuring valid randomization, defining primary and guardrail metrics, calculating sample size and statistical power, instrumenting events, running analyses and interpreting results, and deciding on rollout or rollback. Also includes building testing infrastructure, establishing organizational best practices for experimentation, communicating learnings, and discussing both successful and failed tests and their impact on product decisions.

EasyTechnical

45 practiced

For a recommender system UI change, define 3 primary metrics and 3 guardrail metrics you would set before launching an A/B experiment. Explain why each metric is appropriate and how you would instrument them. The scenario: the change shows a new 'recommended for you' carousel on the homepage for logged-in users.

Sample Answer

Primary metrics (what success looks like)1) Carousel Click-Through Rate (CTR) — fraction of homepage sessions where a user clicks any item in the carousel.- Why: Directly measures engagement with the new surface and signal that recommendations attract attention.- Instrumentation: Log an "exposure" event when carousel rendered (with variant_id, user_id, timestamp) and a "carousel_click" event with item_id. Compute CTR = clicks/exposures by cohort via the A/B framework.

2) Downstream Conversion Rate (e.g., add-to-cart or content-view-to-watch) for clicked items.- Why: Engagement is only valuable if it drives downstream goals (revenue or retention).- Instrumentation: Propagate recommender context (recommendation_id, item_id, exposure_id) through click to conversion events and attribute conversions to the last-recommendation touch; compute conversion per exposed user and per click.

3) Weekly Active User (WAU) retention lift or session length per user.- Why: Checks whether the carousel improves longer-term engagement, not just immediate clicks.- Instrumentation: Track user sessions, link exposure cohorts to retention windows (D7/D28) and average session duration per user.

Guardrail metrics (prevent harm)1) Homepage engagement displacement (clicks on other homepage elements).- Why: Ensure carousel isn't cannibalizing higher-value content.- Instrumentation: Compare click rates on other modules between variant and control.

2) System performance: page load latency and error rate for homepage.- Why: New UI can hurt UX and cause churn.- Instrumentation: Client-side performance metrics (TTFB, JS render) tagged by variant; monitor 95th percentile latency and JS errors.

3) Diversity/novelty and filter-bubble signal (e.g., distribution entropy across item categories).- Why: Avoid narrowing recommendations that reduce long-term satisfaction or introduce bias.- Instrumentation: Calculate category/item entropy and fraction of repeated items across exposures per user; alert if significant drop vs control.

Analysis notes:- Always log exposure with experiment metadata, user bucketing, and deterministic assignment. Use unit-of-analysis = user (or user-session) and predefine statistical tests, sample size, and success criteria. Monitor metrics in near real-time and run sequential checks with multiple-test correction for guardrails.

HardTechnical

51 practiced

Compare sequential testing under a frequentist alpha-spending approach with a fully Bayesian decision framework for early stopping. Discuss how each controls error rates, how priors affect outcomes, and practical considerations for automated stopping in high-velocity growth experiments.

Sample Answer

Sequential frequentist (alpha‑spending) and fully Bayesian decision frameworks approach early stopping from different philosophical and operational angles. Briefly:

Error control- Alpha‑spending: You predefine an overall Type I error (e.g., 0.05) and allocate it across interim looks via an alpha‑spending function (O’Brien–Fleming, Pocock). This guarantees frequentist control of false positives under repeated sampling—if you follow the plan, long‑run Type I error is bounded regardless of stopping rule details that conform to the design.- Bayesian decision: There is no built‑in frequentist Type I guarantee. Instead you compute posterior probabilities or expected utilities and stop when a decision rule (e.g., P(effect>0|data)>0.99) or expected loss threshold is met. Error rates (false positives/negatives) can be evaluated by simulation under assumed priors and data generating processes.

Role of priors- In frequentist alpha‑spending priors don’t enter; operating characteristics are defined by design. They are robust to prior misspecification.- In Bayesian methods priors directly affect posteriors and stopping: informative priors speed decisions and shrink extreme estimates; weak/noninformative priors make Bayesian behavior more like repeated sampling but can inflate misleading posteriors at early looks. Carefully chosen hierarchical priors can regularize and improve out‑of‑sample performance; sensitivity analysis and prior predictive checks are essential.

Practical considerations for automated stopping in high‑velocity experiments- Precommitment vs flexibility: alpha‑spending requires pre-specification of look times or an adaptive alpha schedule; this is excellent for regulatory or product teams demanding formal Type I guarantees. It’s less flexible for irregular traffic or many simultaneous experiments.- Computational automation: Bayesian rules integrate naturally with automated pipelines because decisions are posterior‑based and can accommodate unequal look spacing, continuous monitoring, and multiple metrics via multivariate models. But you must: (a) choose priors and validate via extensive simulation; (b) calibrate decision thresholds to meet business tolerances for false discovery; (c) guard against peeking with weak priors by simulating the true operating curve.- Multiple experiments and multiplicity: For many concurrent tests, frequentist approaches rely on familywise or FDR controls layered on top; Bayesian hierarchical modeling shares information across experiments and can implicitly control multiplicity but requires careful evaluation to avoid information leakage.- Latency, cost, and risk tradeoffs: Define utility or loss (opportunity cost of delayed rollout vs cost of false positive). Decision‑theoretic Bayesian stopping naturally optimizes expected utility; alpha‑spending optimizes error control but needs separate business‑value framing.

Recommendation for ML engineers- If regulatory/frequentist guarantees are required or stakeholders demand strict Type I control, use alpha‑spending with preplanned looks and simulate realistic traffic patterns.- If you need flexible, continuous, automated stopping across many experiments and can invest in simulation and prior elicitation, adopt a Bayesian decision framework with hierarchical priors and explicit utility/loss functions; calibrate thresholds to business risk and run prior/posterior predictive checks.- In practice a hybrid works well: use Bayesian models for estimation and decision utilities, but calibrate thresholds so simulated frequentist error rates meet policy. Automate monitoring, logging of decision triggers, and periodic audit via replayed historical data.

MediumTechnical

55 practiced

Your PM wants the ability to stop an experiment early if it looks promising. Describe safe ways to support early stopping without inflating Type I error. Compare alpha-spending methods, sequential hypothesis testing, and Bayesian approaches and recommend a practical approach for production experiments.

Sample Answer

Situation: PM wants the ability to stop promising experiments early to ship improvements faster, but we must avoid inflating Type I error (false positives) that would harm model quality and business trust.

Key approaches and comparison:

- Alpha‑spending (group sequential methods) - What: Pre-specify interim looks and allocate the overall alpha (e.g., O’Brien–Fleming or Pocock spending functions) so cumulative Type I error stays at desired level. - Pros: Frequentist guarantees on Type I error; well-understood; implementation via fixed-time looks or information-based looks. - Cons: Requires pre-registration of looks or information thresholds; less flexible if team wants ad‑hoc peeks.

- Sequential hypothesis testing (e.g., SPRT, fixed-sequence sequential tests, always-valid p-values / confidence sequences) - What: Continuously monitor using stopping boundaries derived from likelihood ratios or martingale-based confidence sequences. - Pros: Supports continuous monitoring with formal error control; tighter sample efficiency; always-valid p-values are robust to optional stopping. - Cons: More complex math; may require independence assumptions or careful handling of dependent metrics.

- Bayesian approaches - What: Compute posterior probabilities of improvement; stop when posterior exceeds a decision threshold. - Pros: Intuitive decision-making, naturally handles accumulating evidence, accommodates prior knowledge. - Cons: No frequentist Type I error guarantee unless calibrated (e.g., through Bayes factors or simulation). Priors can bias results; regulators/stakeholders may distrust purely Bayesian stopping.

Recommendation for production experiments:1. Use an always-valid/martingale-based monitoring approach (confidence sequences or sequential tests) or alpha‑spending if you need simpler tooling and fixed interim schedule. Both give formal control over Type I error under optional stopping.2. Operational rules: - Pre-register primary metric, minimum detectable effect, maximum sample size or information target, and stopping criteria (boundaries). - Blind secondary/ad-hoc peeks to decision-makers to avoid biased interpretation. - Run pre-deployment simulations (A/B test Monte Carlo) to validate Type I/II error under realistic traffic patterns and metric autocorrelation.3. Practical stack: - If you need continuous real-time stopping: implement always-valid p-values/confidence sequences (e.g., using e‑processes or beta-binomial CS for binary metrics). - If you prefer scheduled checks: use an alpha‑spending function (O’Brien–Fleming) integrated into the experiment platform. - Optionally combine Bayesian dashboards for intuition but require frequentist-based decision rule for releases.

Example stopping rule (binary metric, continuous monitoring): use a beta-binomial confidence sequence and stop when the one-sided lower bound for treatment uplift exceeds the minimum business-relevant lift and the probability of Type I error (as calibrated by simulations) remains below alpha=0.05.

Why this balances safety and speed:- Always-valid/sequential methods preserve Type I error under optional stopping, enabling faster true-positive detection.- Pre-registration and simulation enforce rigor; Bayesian views can inform but shouldn’t replace deployed decision thresholds without calibration.

This approach gives the PM early-stopping capability while preserving statistical guarantees needed for production-grade decisions.

MediumTechnical

48 practiced

Explain how baseline conversion rate, variance, and targeted minimum detectable effect (MDE) interact to determine required sample size. Use a short numeric example showing how sample size changes if baseline is 1% vs 20% for the same absolute MDE.

HardTechnical

51 practiced

Case study/dataset: You have the following aggregate results from an A/B test run for 30 days: Control: users=200k, conversions=20k. Treatment: users=200k, conversions=21k. Provide a step-by-step analysis plan to determine if the treatment is truly beneficial and production-ready. Include sanity checks, significance test, uplift CI, power retrospective, metric segmentation, and pre-trend checks.

Sample Answer

1) Clarify objective & metric- Primary metric: conversion rate (binary). Confirm business impact, risk tolerance, success threshold (absolute or relative uplift).

2) Sanity checks / data integrity- Verify user counts deduped and mutually exclusive; check randomization key distribution across users, cohorts, devices.- Check instrumentation: no missing events, conversion definition stable, no delayed attribution differences.- Check temporal balance: users/day similar across arms.

3) Point estimate & significance- Compute rates: control 10.0% (20k/200k), treatment 10.5% (21k/200k) → absolute uplift 0.5pp, relative +5%.- Perform two-proportion z-test (or Fisher/chi-square) to get p-value.- Report test statistic, p-value, and whether it crosses alpha (e.g., 0.05).

4) Uplift confidence interval- Compute 95% CI for difference in proportions (use pooled SE or Wilson if needed). Report CI in absolute and relative terms to assess practical significance.

5) Retrospective power & minimum detectable effect (MDE)- Calculate achieved power given observed effect and sample sizes; compute MDE at planned alpha/power. If power <80%, treat result as inconclusive even if p<0.05.

6) Check for peeking & multiple comparisons- Confirm no interim peeking or adjust for it; if multiple metrics/segments tested, control FDR or Bonferroni.

7) Metric segmentation / heterogeneity- Segment by device, geography, new vs. returning users, traffic source, time of day. Compute per-segment rates and CIs; check consistency and interaction effects (logistic regression with treatment*segment).- Flag segments with opposite effects or high variance—investigate.

8) Pre-trend and temporal diagnostics- Run pre-period balance on similar metrics (prior week/month) to ensure no pre-existing drift.- Time series of daily conversion rate and cumulative z-score; look for post-randomization anomalies, launch-day spikes, or novelty effects.

9) Secondary metrics & safety checks- Check engagement, retention, revenue, error rates, latency, resource usage. Ensure no negative trade-offs.

10) Robustness tests- Run alternative analyses: bootstrap CI, Bayesian credible interval, regression adjusted for covariates to reduce variance.- Run per-user vs per-session definitions.

11) Production-readiness & rollout plan- If effect is statistically and practically significant and safety checks pass: propose staged rollout (e.g., 10% → 50% → 100%) with monitoring alerts on key metrics and kill-switch.- If inconclusive: extend experiment or increase sample (compute additional sample needed), or run targeted follow-ups on promising segments.

12) Documentation & monitoring- Document analysis, assumptions, and decision criteria. Deploy real-time dashboards and automated anomaly detection for first 2-4 weeks of rollout.

Deliverables: p-value, uplift CI, retrospective power & MDE, segment table, pre-trend plots, regression/robustness results, recommended rollout and monitoring checklist.

Unlock Full Question Bank

Get access to hundreds of Experiment Design and Execution interview questions and detailed answers.

Join thousands of developers preparing for their dream job.