A and B Test Design Questions

Designing and running A and B tests and split tests to evaluate product and feature changes. Candidates should be able to form clear null and alternative hypotheses, select appropriate primary metrics and guardrail metrics that reflect both product goals and user safety, choose randomization and assignment strategies, and calculate sample size and test duration using power analysis and minimum detectable effect reasoning. They should understand applied statistical analysis concepts including p values confidence intervals one tailed and two tailed tests sequential monitoring and stopping rules and corrections for multiple comparisons. Practical abilities include diagnosing inconclusive or noisy experiments detecting and mitigating common biases such as peeking selection bias novelty effects seasonality instrumentation errors and network interference and deciding when experiments are appropriate versus alternative evaluation methods. Senior candidates should reason about trade offs between speed and statistical rigor plan safe rollouts and ramping define rollback plans and communicate uncertainty and business implications to technical and non technical stakeholders. For developer facing products candidates should also consider constraints such as small populations cross team effects ethical concerns and special instrumentation needs.

EasyTechnical

55 practiced

A product change aims to increase revenue per session but may hurt long-term retention. Explain how you would choose a primary metric and guardrail metrics for the experiment. Include time horizons, aggregation windows, and how you would weigh short-term gains against potential long-term harm.

Sample Answer

Primary metric:- Choose a user-level, value-focused metric that captures both short-term gain and long-term harm. For this scenario I’d use 90-day incremental revenue per user (∆Revenue/user at 90d) as the primary metric. Reason: the product change targets revenue per session but retention impact unfolds over weeks — 90d balances signal and business value and converts sessions into lasting value.

Guardrail metrics:- 7-day and 28-day retention rate (cohort retention) — detect early churn signals.- Revenue per session (immediate treatment effect) — confirms the proximal mechanism.- DAU/MAU or weekly active users — monitors engagement.- Churn rate and repeat-session probability within 28d.- User satisfaction/CSAT or NPS if available — qualitative guardrail.

Time horizons & aggregation:- Short-term: per-session and per-user aggregated over 0–7 days (detect immediate lift).- Mid-term: cohort metrics at 28 days (early retention).- Long-term: cohort revenue and retention at 90 days (primary evaluation).- Use user-level aggregation (each user contributes one datapoint: revenue over window) to avoid inflating significance via sessions.

Weighing short-term vs long-term:- Pre-specify decision rules and thresholds before launching (e.g., require ≥5% lift in 90d revenue with p<0.05 OR if 90d is inconclusive, require short-term lift >10% AND no >2 percentage-point drop in 28d retention).- Compute risk-adjusted expected value: short-term incremental revenue × expected user lifetime multiplier minus projected LTV loss from retention declines.- Power the experiment for the smallest meaningful detectable effect on 90d revenue; run monitoring with interim checks only for safety (guardrail breaches trigger pause/rollback).- Use cohort / survival analysis to model long-term impact beyond observation window and run sensitivity analysis.

Operational notes:- Log user IDs, exposure, session timestamps, and events to compute per-user aggregates.- Correct for multiple comparisons when evaluating many guardrails.- Automate alerts for guardrail breaches and run a post-mortem if trade-offs are accepted.

HardTechnical

55 practiced

Design a hierarchical/Bayesian model to detect heterogeneous treatment effects across 100 geographic regions while borrowing strength. Describe priors, pooling strategy, and how to report posterior summaries to business stakeholders.

Sample Answer

Goal: estimate region-specific treatment effects τ_r for r=1..100 while borrowing strength via partial pooling and quantifying uncertainty for business decisions.

Model (continuous outcome y_irt for unit i in region r, t∈{0,1} treatment):y_irt ~ Normal(μ_r + τ_r * T_irt + X_irt β, σ_y)τ_r ~ Normal(τ_pop, σ_τ) (hierarchical pooling)μ_r ~ Normal(μ_pop, σ_μ) (region baselines)β ~ Normal(0, 1) (standardized covariates)

Hyperpriors (weakly informative, encourage shrinkage but allow heterogeneity):τ_pop ~ Normal(0, 1)μ_pop ~ Normal(0, 5)σ_τ ~ Half-Student-t(3, 0, 1) (or Half-Cauchy(0,1))σ_μ ~ Half-Student-t(3, 0, 2)σ_y ~ Half-Student-t(3, 0, 2)

Rationale:- Partial pooling via τ_r ~ Normal(τ_pop, σ_τ) shrinks noisy region estimates toward the global mean; σ_τ quantifies true heterogeneity.- Weakly informative priors (scaled after standardizing outcome/covariates) stabilize estimates, avoid pathological posteriors, and let data dominate.- Use robust priors (Student-t) on scales to tolerate outlier regions.

Computation & diagnostics:- Fit with HMC/NUTS (Stan/PyMC) for accurate posteriors; for scale consider ADVI or MAP + Laplace as approximations but validate with HMC on a subset.- Run posterior predictive checks, R-hat, effective sample sizes, and sensitivity checks to prior choices (e.g., tighter vs looser σ_τ priors).

What to report to stakeholders:- For each region: posterior mean(τ_r), 90% credible interval, and P(τ_r > 0) or P(τ_r > business-relevant delta). Present these in a sortable table.- Global summaries: posterior of τ_pop and σ_τ (interpretable as average effect and across-region variability).- Decision metrics: expected revenue uplift = E[τ_r]*exposure_r, probability of positive ROI, and expected value of targeted rollout.- Visuals: forest plot of τ_r with credible intervals, heatmap across geography, and posterior predictive checks showing model fit.- Communicate uncertainty: emphasize intervals and probabilities rather than point estimates; show sensitivity to priors and covariate adjustments.- Actionable rules: e.g., "target regions with P(τ_r > δ) > 0.9 and expected uplift > $X" and quantify risk of false positives.

Caveats & extensions:- If spillovers or non-exchangeability exist, include spatial correlation (τ ~ GP or CAR prior) or allow region-level covariates to explain heterogeneity.- For binary outcomes use logistic hierarchical model with analogous priors and report risk differences or odds ratios on interpretable scales.- Consider hierarchical shrinkage priors (e.g., horseshoe) if expecting sparse large effects.

HardSystem Design

42 practiced

Your experiment platform supports many concurrent experiments. Describe how you would design treatment assignment and bucketing to minimize interference when users may be in several experiments simultaneously. Discuss trade-offs between orthogonalization and combinatorial explosion of experiment cells.

Sample Answer

Requirements and constraints:- Functional: assign treatments deterministically and reproducibly; support many concurrent experiments per user; enable experiment overlap while minimizing interference (treatment-by-treatment interactions).- Non-functional: low-latency assignment (<ms), high throughput, auditability, and statistical power guarantees.

High-level architecture:1. Bucketing service (deterministic hashing + namespace)2. Experiment registry (metadata: unit, traffic allocation, priority, orthogonalization rules)3. Assignment engine (layered/randomization logic)4. Telemetry & interaction detector (log exposures, run interaction tests)

Design approach (components & flow):- Namespace + hash-based bucketing: compute user_id → 64-bit hash → map into namespaces (feature area or product surface). Each experiment claims a namespace slice (range). Deterministic mapping ensures reproducibility and low latency.- Two-level bucketing: global user-level bucket for coarse segmentation (e.g., 10000 buckets) then experiment-level allocation inside claimed buckets. This supports stable splits and stratification by covariates (device, geography).- Layered randomization with priority & orthogonalization rules: - Default: independent hashing per experiment (fast, simple) — allows users in many experiments; assumes small interaction effects. - For critical experiments where interference unacceptable, declare mutual-exclusion (orthogonalization): assign users exclusively to one of the set using a shared namespace and partitioning logic (round-robin or prioritized claim). This avoids contamination but reduces reachable sample for each experiment.- Factorial / combinatorial cells only for planned interaction testing: for pairs/sets of experiments where interaction is suspected, create a combined experiment that explicitly randomizes orthogonally into all combinations (2^k cells). Limit k (usually 2) to avoid explosion.- Meta-experiment registry enforces constraints: capacity quotas, max concurrent orthogonalized experiments, and automatic fallbacks (e.g., if no capacity left, place new experiment in independent mode).

Trade-offs — orthogonalization vs combinatorial explosion:- Orthogonalization (mutual exclusion): pros — eliminates treatment interference, simpler analysis; cons — reduces eligible traffic per experiment (sample size), increases engineering coordination, and can delay launches.- Full factorial combinatorics: pros — directly estimates interaction effects and causal attribution; cons — cell count grows exponentially (2^k), leading to impractical sample and storage needs; complex analysis and monitoring.Practical compromise:- Use independent assignments by default; require orthogonalization only for high-risk experiments (e.g., revenue-impacting, shared model endpoints).- Use targeted factorial designs: predefine pairs of experiments where interaction matters and allocate a small stratified subsample for full factorial testing.- Instrumentation: always log cross-experiment exposures and include interaction terms in analysis models (regression with indicators and interaction terms). Use hierarchical models or inverse probability weighting to adjust for overlap when full orthogonalization isn't possible.

Scalability & operational considerations:- Keep assignment logic stateless and rely on cached experiment metadata. Use consistent hashing to rebalance without mass reassignment.- Monitoring: automated checks for imbalance, unexpected interactions, and power calculators that account for overlap.- Governance: experiment registration enforces orthog. rules, minimum ramp sizes, and expiration to limit namespace fragmentation.

Summary:Default to lightweight independent bucketing for scale; reserve orthogonalization and factorial designs for cases needing unbiased interaction estimates. Combine deterministic hashing, namespaces, and a registry that enforces constraints; always log exposures so analysts can detect and adjust for interference.

MediumTechnical

56 practiced

You must run an experiment for a developer-facing feature exposed to a small active population (e.g., enterprise beta users). Propose experimental designs and statistical techniques to get useful insights despite limited sample size and cross-team dependencies.

Sample Answer

Approach summary: with a small, enterprise beta population, prioritize designs that increase statistical power and extract maximum signal per user while accounting for cross-team dependencies. Combine within-subjects designs, Bayesian inference, hierarchical modeling, and careful operational controls.

Design options:- Within-subjects / crossover: expose each customer to baseline and treatment periods (randomize order, washout). This reduces between-customer variance and increases power.- Paired A/B on feature-usage units: if features trigger per-request/model-call effects, randomize at request or repo level within customers to create many paired observations.- Staggered rollout / stepped-wedge: gradually introduce treatment across teams/customers and use time as control — useful when you must coordinate releases across teams.- Synthetic controls / counterfactual modeling: build customer-specific baseline forecasts (time-series or ML models) to compare observed post-treatment behavior.

Statistical techniques:- Bayesian hierarchical models: pool information across customers but allow customer-level effects; yields probabilistic effect distributions and credible intervals that work well with small N.- Sequential/Bayesian updating: analyze as data arrives, stop when posterior probability passes decision thresholds; avoids fixed-sample-size power issues.- Variance reduction: include strong covariates (pre-period metrics, product usage embeddings) and use ANCOVA-style adjustment to reduce residual variance.- Nonparametric tests & permutation methods: compute exact p-values on small samples when distributional assumptions fail.- Bootstrapping for CI estimation on per-customer metrics.

Handling cross-team dependencies:- Contract an experiment API/feature flag layer so teams can opt-in to controlled rollout windows; coordinate canary windows to isolate confounders.- Instrument and surface per-team rollout metadata; include team/time fixed effects in models.- Use staged experiments: run an internal pilot, fix integration issues, then run the formally analyzed beta.

Practical metrics and checks:- Define proximal metrics (usage, latency, error rates) and leading indicators rather than rare business outcomes.- Pre-register analysis plan, decision thresholds (e.g., posterior probability > 0.95), and minimal detectable effect to avoid p-hacking.- Run sensitivity analyses (different priors, excluding top/bottom customers) and report uncertainty.

Why this works: within-subjects and hierarchical Bayes maximize information per customer; covariate adjustment and paired tests reduce noise; sequential/Bayesian decision rules adapt to small samples; stepped/staggered designs and careful instrumentation mitigate cross-team confounding.

HardTechnical

49 practiced

You observe a small but statistically significant uplift in the A/B test for conversion rate driven entirely by a tiny subset of heavy users. Describe how you'd investigate whether the effect is real, robust, or an artifact, and how to present actionable conclusions to stakeholders.

Sample Answer

Framework: treat this as a robustness + causal inference investigation, then produce an actionable recommendation with clear uncertainty.

Investigation steps1. Validate randomization & data quality- Check assignment balance on pre-period metrics and covariates (demographics, historical conversion). Run t-tests/KS tests and standardized mean differences.- Ensure no logging or bucketing bugs that could route heavy users disproportionately.

2. Convert-to-user-level and re-run- Aggregate conversions to the user level (not event-level) to avoid heavy-user domination. Compute per-user conversion rates; compare means between groups using bootstrap and cluster-robust SEs.

3. Subgroup and influence analysis- Identify heavy-user threshold (e.g., 95th percentile); compute contribution-to-uplift by percentile buckets.- Run leave-one-out / leave-group-out influence: drop top N users or top X% and re-estimate effect.- Compute influence functions or use robust estimators (median difference, trimmed mean, Winsorization).

4. Statistical robustness- Run permutation tests on user-level data (random reassignments) to get null distribution.- Adjust for multiple comparisons if you looked across many subgroups (BH/FDR).- Estimate effect heterogeneity via regression with treatment × heavy_user indicator, include covariates; check interaction significance.

5. Practical checks- Examine user journeys of heavy users for behavioral or product differences — are they bots, internal users, premium accounts?- Check duration: is uplift sustained over time or a short spike?

Presentation to stakeholders- One-slide executive summary: headline (effect size, CI), who drives it (X% of users cause Y% of uplift), recommendation (rollout, iterate, or hold), and risk level.- Support with 2 visuals: cumulative contribution curve (percentile vs. contribution) and user-level treatment-control distribution with robustness overlays (trimmed/winsorized/median).- Provide concrete options with estimated impact and confidence: e.g., - Proceed to targeted rollout for heavy users (expected lift A, risk B) - Abort global rollout; run follow-up experiment restricting to non-heavy users - Fix data/segmentation issues and re-run

Actionable next steps- If effect driven by valid heavy users: run experiments targeted at that cohort, instrument long-term metrics.- If artifact: correct logging/randomization, re-run test.- Always implement monitoring: automated alerts for heavy-user-driven swings and rerun user-level permutation tests periodically.

This approach quantifies whether the uplift is real, explains its origin, and gives stakeholders clear trade-offs and a path forward.

Unlock Full Question Bank

Get access to hundreds of A and B Test Design interview questions and detailed answers.

Join thousands of developers preparing for their dream job.