A and B Test Design Questions

Designing and running A and B tests and split tests to evaluate product and feature changes. Candidates should be able to form clear null and alternative hypotheses, select appropriate primary metrics and guardrail metrics that reflect both product goals and user safety, choose randomization and assignment strategies, and calculate sample size and test duration using power analysis and minimum detectable effect reasoning. They should understand applied statistical analysis concepts including p values confidence intervals one tailed and two tailed tests sequential monitoring and stopping rules and corrections for multiple comparisons. Practical abilities include diagnosing inconclusive or noisy experiments detecting and mitigating common biases such as peeking selection bias novelty effects seasonality instrumentation errors and network interference and deciding when experiments are appropriate versus alternative evaluation methods. Senior candidates should reason about trade offs between speed and statistical rigor plan safe rollouts and ramping define rollback plans and communicate uncertainty and business implications to technical and non technical stakeholders. For developer facing products candidates should also consider constraints such as small populations cross team effects ethical concerns and special instrumentation needs.

EasyTechnical

48 practiced

A product manager asks you to 'increase engagement' with a new homepage module. Describe how you would choose a primary metric and at least two guardrail metrics. For each metric, specify the unit of analysis (user, session), numerator/denominator definitions, aggregation window, and why it aligns or protects product/business goals.

HardTechnical

60 practiced

In an experiment you evaluate 20 metrics across engagement, revenue, and technical guardrails. Explain practical approaches to control false discoveries at the metric level while preserving power: hierarchical testing, metric families with corrections, pre-specification and gating, and how to implement these in a policy for an analytics team.

Sample Answer

Situation: We run an A/B test that reports 20 metrics across three domains: engagement (8), revenue (7), technical guardrails (5). We need to limit false discoveries while keeping power so product decisions are reliable.

Approach summary (why and how)- Hierarchical testing (gatekeepers): Test broad hypotheses first (family-level), then drill down only if the family shows signal. This preserves power because you avoid multiplicity for families that show no effect. Example: test “any engagement metric changed” using an omnibus test (e.g., pooled z or Simes). If significant at α=0.05, then test the 8 engagement metrics with multiplicity correction.- Metric families with corrections: Group metrics by domain and apply a family-wise correction within each family. For strict control use Bonferroni/Holm (controls FWER), for better power use Benjamini–Hochberg (BH) to control FDR at q=0.05. Example: for 8 engagement metrics use BH to keep FDR 5% among those.- Pre‑specification and gating: Pre-register primary vs exploratory metrics and analysis plan before seeing results. Primary metrics get priority (no or minimal correction), secondary metrics are tested conditionally (gating), exploratory metrics are labeled and reported without formal claims.- Practical trade-offs: Use BH within families when you want discovery (higher power). Use Bonferroni/Holm for guardrails (low tolerance for false positives).

Implementation in an analytics policy (practical checklist)1. Pre-registration: Every experiment must include a signed experiment spec listing primary metric(s), families, statistical tests, sample size and stopping rules, before deployment.2. Family definitions: Define metric families (engagement, revenue, guardrails) and the order of gates (e.g., guardrails first—if violated, stop rollout).3. Gating logic: - Gate 0: Safety check (guardrails): apply Bonferroni or Holm across 5 guardrail metrics at α=0.05. Any failure pauses rollout. - Gate 1: Primary family tests: omnibus test per family at α=0.05. If non-significant, report no further claims. - Gate 2: Within-family testing: apply BH (q=0.05) across family metrics if omnibus positive; otherwise treat as exploratory.4. Reporting and labels: Results labeled as “Confirmed” (passed pre-spec + corrections), “Conditional” (passed family gate but exploratory), or “Exploratory” (not pre-specified).5. Automation: Implement pipeline that: - Runs pre-specified tests and corrections automatically (SQL/stats scripts) - Produces reproducible notebook with p-values, adjusted p-values, effect sizes, CIs - Flags guardrail violations and triggers alerts6. Governance and audit: Weekly review board for experiments that produce business-critical decisions; store specs and outputs in versioned repo for audit.7. Power calculations: Compute sample sizes per primary metric and for family-level omnibus where possible; simulate correlation among metrics to estimate effective multiplicity and adjust sample sizes.8. Education: Document examples (e.g., Simpson’s paradox, correlated metrics) and hold training on interpreting adjusted p-values and FDR vs FWER trade-offs.

Concrete example (numbers)- Guardrails (5): use Holm at α=0.05 → conservative stop if any adjusted p < 0.05.- Engagement (8): omnibus Simes at α=0.05 → if significant, run BH at q=0.05 across the 8 to report discoveries.- Revenue (7): pre-specify one primary revenue metric (no correction), remaining six use BH within revenue family.

Key rationale: gating + family corrections reduces multiple-testing burden only where signal exists, preserving power. Pre-specification prevents p-hacking and gives priority to business-critical metrics. Automation and governance ensure consistent, auditable decisions.

MediumTechnical

60 practiced

Compare the use of a two-sample t-test versus a non-parametric test (e.g., Mann-Whitney U) for analyzing A/B test metrics. When would you prefer each approach, and how do skewed distributions and outliers (e.g., revenue per user) affect your choice and interpretation?

Sample Answer

Situation: You're comparing A vs B on a metric like revenue per user (RPU). The key decision is whether to test differences in means (two-sample t-test) or use a non‑parametric alternative (Mann–Whitney U).

Direct answer / guidance:- Two‑sample t-test: tests difference in group means (assuming independent samples). It is appropriate when sample sizes are moderate-to-large and the sampling distribution of the mean is approximately normal (Central Limit Theorem), or when data are roughly symmetric and variances are similar. It gives interpretable effect in dollars (mean difference) which stakeholders often want.- Mann–Whitney U: tests whether one distribution tends to yield larger values than the other (often framed as medians under similar shapes). It is robust to non-normality and outliers and useful when distributions are skewed or ordinal. It does NOT directly test mean differences.

How skew and outliers affect choice and interpretation:- Heavy skew / long tails (common for RPU): means can be dominated by a few high spenders. A t-test may be misleading if tails are extreme and sample sizes are small. With large N, t-test often remains OK for inference on means, but the mean itself may be a poor summary of central tendency.- Outliers: inflate variance, reduce t-test power and can distort estimates. Mann–Whitney is less sensitive to extreme values but shifts the null hypothesis to stochastic dominance rather than mean equality.

Practical recommendations for a data analyst:- Always visualize (histograms, boxplots, ECDFs) and report both mean and median plus IQR.- If primary metric is business-relevant total or average revenue, prefer robust estimation: consider log-transform (then t-test on log-values → compares geometric means), or winsorize/trim, or use bootstrap CIs for mean differences (nonparametric bootstrap handles skew/outliers).- If you care about median or rank effects, use Mann–Whitney (or bootstrap median CI). For small samples with unknown distribution, prefer nonparametric tests or permutation tests.- Check variance equality; if violated, use Welch’s t-test (doesn’t assume equal variances).- Report effect sizes and confidence intervals, not just p-values, and explain what hypothesis each test answers (mean difference vs. rank/median shift).

Example: For RPU with N=10k per arm and heavy right tail, run Welch’s t-test on raw means and bootstrap the mean difference; also report median difference and a Mann–Whitney test. If log-transform makes distributions symmetric and interpretation acceptable, use t-test on logs for stabilized variance.

HardTechnical

55 practiced

Discuss advantages and disadvantages of adopting a Bayesian framework for A/B testing in a fast-paced growth environment. Include how you would specify priors for conversion rates, interpret posterior probabilities (e.g., probability treatment > control), handle multiple looks, and present Bayesian results to non-technical stakeholders.

Sample Answer

Advantages: Bayesian A/B testing gives direct probabilistic statements stakeholders prefer (e.g., “there’s a 92% probability treatment beats control”), naturally handles small samples via priors, and supports truly sequential monitoring without complex alpha-spending rules if decisions are based on posterior + loss. It also easily extends to hierarchical models (multi-armed tests, segments).

Disadvantages: Results depend on priors (must justify them), pure Bayesian stopping rules may not satisfy teams expecting frequentist Type I control, and communicating subjective priors and credible intervals can confuse non-technical audiences.

Specifying priors for conversion rates:- Use Beta-Binomial conjugacy. For an uninformative prior use Beta(1,1). - If historical baseline exists (e.g., baseline p0 = 0.05 and you want prior weight equivalent to 200 observations), set alpha = p0*N = 10, beta = (1-p0)*N = 190 → Beta(10,190). - Prefer weakly informative priors; run sensitivity checks with alternative priors (flat, skeptical, enthusiastic) and report robustness.

Interpreting posteriors:- Compute posterior for p_control ~ Beta(a_c + successes_c, b_c + failures_c) and p_treatment similarly.- Report P(p_treatment > p_control) = ∫∫ I(p_t>p_c) f(p_t) f(p_c) dp_t dp_c (Monte Carlo estimate). - Report posterior mean uplift, 95% credible interval for uplift, and probability uplift > MDE (minimum detectable/economic effect). Frame as business risk: e.g., “90% chance of >0%, 65% chance of >0.5% (our MDE).”

Handling multiple looks:- Bayesian updating is coherent under optional stopping for the posterior, but if stakeholders care about frequentist error rates, pre-specify a decision rule or loss function (e.g., stop when P(treatment>control) > 0.95 and expected loss of wrong decision < threshold). Alternatively calibrate stopping rule via simulation to estimate frequentist Type I/II properties and share those results.

Presenting to non-technical stakeholders:- Lead with clear business statements: probability treatment outperforms, expected absolute revenue impact, recommended action. - Use intuitive visuals: overlay posterior density curves, histogram of simulated uplift, and a simple gauge showing P(treatment>control) and P(uplift> MDE). - Include short sensitivity summary (what changes if prior differs) and recommended next steps (rollout, more data, segment follow-up). - Always pair probability statements with practical consequences (e.g., expected incremental conversions and revenue) so decisions map to business impact.

MediumTechnical

43 practiced

Two teams launch overlapping experiments on the same user population and features that may interact. Describe experimental designs, analysis techniques, and governance policies to handle overlapping experiments safely (e.g., orthogonal assignment, exclusion rules, factorial design). Include operational suggestions to reduce accidental conflicts.

Sample Answer

Situation: Two product teams run experiments that target the same users and features, risking interaction effects that invalidate results.

Designs:- Orthogonal assignment (preferred): assign users to experiment A/B and experiment B/C independently using deterministic bucketing (hash on user_id) so cells are orthogonal; ensures all combinations exist.- Factorial design: formally treat experiments as factors (2x2, etc.) to estimate main effects and interactions directly.- Exclusion rules: when interaction risk is high and sample is limited, exclude overlapping users from one experiment to preserve power.- Sequential/holdout: run experiments at different times if interaction can't be modeled.

Analysis techniques:- Use regression with interaction terms (Y ~ A + B + A*B) or two-way ANOVA to quantify interaction and main effects.- Pre-specify metrics and run stratified analyses by cell; report cell-level conversions and CIs.- Adjust power calculations for multiple cells and plan sample sizes accordingly.- Run A/A tests and balance checks; correct for multiple comparisons (Bonferroni/FDR) when many overlaps exist.- Always compute lift within each orthogonal cell; if interaction term significant, prioritize cell-level decisions.

Governance & operational suggestions:- Central experiment registry with metadata (owner, start/end, target population, risk level). Require reviewers sign-off for overlapping targets.- Experiment calendar and automated conflict detection (alerts when two experiments target same population/feature).- Standardized sample bucketing library (shared hashing function, feature-flag integration) to ensure consistent assignment across teams.- Pre-analysis plans stored with experiments; require power calculation and interaction-risk assessment.- Guardrail dashboards for key metrics and an automated kill-switch if adverse signals appear.- Post-experiment cross-team debriefs to surface unexpected interactions and update playbooks.

As a data analyst I’d enforce pre-registration, implement regression/ANOVA pipelines that output cell-level results and interaction p-values, and build dashboards/alerts so teams detect and resolve overlaps before they bias decisions.

Unlock Full Question Bank

Get access to hundreds of A and B Test Design interview questions and detailed answers.

Join thousands of developers preparing for their dream job.