A/B Test Results Analysis Questions

Learn to analyze results from A/B tests and experiments. Understand key statistics: sample size, statistical significance, confidence intervals, and p-values at a practical level (not deep theory). Practice interpreting test results: 'Is this difference real or just noise?' Learn common mistakes: stopping tests early, p-hacking, and running too many tests simultaneously.

MediumTechnical

94 practiced

Revenue per user is highly skewed with many zeros and a few large purchases. Describe possible analysis approaches such as log transformation, trimmed means, winsorization, nonparametric tests, and bootstrapping. For each approach note interpretation and trade-offs for business stakeholders.

Sample Answer

When revenue per user (RPU) is highly skewed with many zeros and a few very large purchases, choose an analysis that matches the question stakeholders care about (typical user, total revenue, uplift, risk). Below are common approaches, what they estimate, and trade-offs.

1) Log transformation (e.g., log(1+RPU))- Interpretation: Stabilizes variance and makes distribution closer to symmetric so means and regressions are less dominated by outliers. Effects become multiplicative (percent changes).- Trade-offs: Harder to interpret in original dollars; zeros handled with offset (log1p) which biases small values slightly. Not appropriate if stakeholders want total-dollar impacts.

2) Trimmed means (drop top/bottom x%)- Interpretation: Estimates “typical” revenue excluding extreme tails; good for understanding central tendency of active users.- Trade-offs: Simple and intuitive for stakeholders, but discards real revenue events (high spenders) that drive business value. Choice of trimming fraction can be arbitrary and needs justification.

3) Winsorization (cap extremes at percentile)- Interpretation: Reduces influence of extreme values by capping them; preserves sample size while limiting outlier leverage.- Trade-offs: Keeps high-value users in analysis but alters their magnitude—must communicate that totals are modified. Less arbitrary than removing, but still a tuning parameter.

4) Nonparametric tests (Mann–Whitney, Kolmogorov–Smirnov)- Interpretation: Compare distributions without assuming normality—test for shifts in ranks or full-distribution differences.- Trade-offs: Detects differences in median/shape, not mean revenue; may miss business-relevant differences in total revenue. Good for robust significance statements when assumptions fail.

5) Bootstrapping (resample to get CI for mean/median)- Interpretation: Empirical sampling distribution gives confidence intervals for mean, median, or other metrics even with skewness and zeros.- Trade-offs: Computationally heavier but straightforward; when estimating mean, bootstrap CIs will reflect heavy-tail uncertainty (wide intervals) which is realistic. Must ensure resampling strategy (user-level bootstrap vs transaction-level) matches inference target.

Practical recommendations for BI stakeholders:- Define the metric to drive decisions: average revenue per user (ARPU) for budgeting vs median/typical-user for behavioral insight vs total revenue for financials.- Present multiple metrics: mean (with bootstrap CI), median, and a winsorized mean or trimmed mean. Show distribution plots (histogram + percentiles) and cumulative contribution of top X% users.- Be explicit about trade-offs and how each metric answers a different business question; provide dollar-equivalent interpretations (e.g., “winsorized mean = $X; top 1% contribute Y% of total”) so stakeholders can act on results.

HardTechnical

81 practiced

Your analytics organization runs 50 experiments per quarter and tracks 20 metrics per experiment. Propose an operational approach to control false discoveries across experiments and metrics, including grouping hypotheses into families, applying FDR adjustments, and communicating adjusted results to product teams while preserving signal.

Sample Answer

Approach summary: create a repeatable pipeline that groups related hypotheses into families, computes p-values/effect sizes consistently, applies FDR procedures per-family (and a higher-level correction when needed), and surfaces adjusted results plus effect sizes/CI in dashboards so product teams can act without losing signal.

Operational steps1. Define families up-front (policy): e.g., per-experiment families (20 metrics tested within one experiment) or per-metric families across experiments (same KPI across 50 experiments). Prefer grouping by scientific question — most common: family = all metrics tested within one experiment (keeps context local, preserves power).2. Compute test statistics: standardize A/A checks, use regression/ATE with robust SE; produce p-values, two-sided effects, and 95% CIs. For correlated metrics, estimate covariance (bootstrap or permutation) to get joint null behavior.3. Apply FDR:- Primary: Benjamini–Hochberg (BH) per-family to control FDR at α (e.g., 0.05) across the 20 hypotheses in that experiment.- If families are many and tests correlated across families, use hierarchical FDR: first test family-level null (e.g., Simes or aggregated p-value) then apply BH within families that pass. This preserves signal and reduces multiplicity loss.- For strong dependence or heavy tails, use Benjamini–Yekutieli (BY) or permutation-based FDR estimation (q-values).4. Operationalize in ETL/CI:- Automate computation scripts (Python/R) that ingest experiment results, run tests, apply BH/hierarchical procedure, and output adjusted p-values (q-values), effect sizes, and adjusted CIs where possible.- Store results in a results table: experiment_id, metric, raw_p, q_value, effect, ci_lower, ci_upper, family_id, decision (discover/non-discover).5. Communicate to product teams:- In dashboards, show: effect size + 95% CI, raw p, q_value, and a simple decision tag (e.g., “significant at FDR 5%”).- Use visualization: volcano plot (effect vs -log10(q)), p-value histogram, and small explainer: what family means and the chosen FDR level.- Give playbook: For metrics flagged significant, recommend next steps (replicate, monitor, rollout). For borderline q (e.g., 0.05–0.15), suggest follow-up experiments rather than immediate action.6. Preserve signal:- Use per-experiment families to avoid diluting power across unrelated tests.- Use adaptive BH (storey q-value) or hierarchical testing to recover signal when many nulls are true.- Where metrics are highly correlated, consider testing latent constructs (PCA) or selecting a primary KPI per experiment to focus decisions.7. Monitoring & governance:- Monthly audit: track FDR realized (proportion of discoveries that replicate in holdout or follow-ups).- Version policy: allowed family definitions, α levels, and exceptions (e.g., safety metrics always tested with stricter thresholds).Example (pseudo-Python outline):

python

# compute raw p-values per metric
# apply BH per family
from statsmodels.stats.multitest import multipletests
qvals = multipletests(raw_ps, alpha=0.05, method='fdr_bh')[1]

This approach balances control of false discoveries, operational simplicity for BI dashboards, and retention of actionable signal by grouping sensibly and using hierarchical/adaptive FDR when appropriate.

MediumTechnical

99 practiced

Your analytics team monitors 20 metrics across 10 experiments each month. Explain the difference between family-wise error rate control and false discovery rate control, and recommend a practical multiple testing correction strategy for reporting to stakeholders.

HardTechnical

90 practiced

Explain how hierarchical or mixed effects models can be used to estimate treatment effects across many segments when individual segment sample sizes are small. Describe model setup, intuition behind partial pooling, and how BI analysts should interpret shrunk estimates versus raw per-segment estimates.

HardTechnical

139 practiced

Your business cannot randomize rollout of a pricing change. Propose quasi-experimental designs to estimate causal impact such as difference-in-differences, synthetic controls, and regression discontinuity, and for each describe key assumptions, required data, and diagnostics to validate assumptions.

Sample Answer

I would propose three quasi-experimental designs appropriate when randomized rollout isn’t possible: difference-in-differences (DiD), synthetic control (SC), and regression discontinuity (RD). For each I describe core idea, key assumptions, required data, and diagnostics you can run in a BI environment.

1) Difference-in-differences (DiD)- Idea: Compare pre/post change in outcome for treated group vs. control group to net out common trends.- Key assumptions: Parallel trends — in absence of treatment, treated and control would follow same trend. No time-varying confounders that differentially affect groups at intervention.- Required data: Panel (time series) at user/segment/store/product level with treatment indicator, outcome (revenue, conversion), covariates and many pre-period observations.- Diagnostics: - Visual pre-trend plots (multiple pre-periods) by group. - Event-study coefficients (leads/lags) to test pre-treatment leads ~0. - Placebo treatment dates and placebo groups. - Balance checks on covariates and robustness to adding controls / flexible time trends.

2) Synthetic control (SC)- Idea: Build a weighted combination of untreated units that best reproduce treated unit’s pre-treatment trajectory; use as counterfactual post-treatment.- Key assumptions: Weighted donor pool can reproduce treated pre-treatment path (no unobserved shocks post that only affect treated).- Required data: Long pre-treatment time series for treated unit and a reasonably large donor pool of untreated units with same covariates/outcomes.- Diagnostics: - Pre-fit goodness-of-fit (RMSPE) — low pre-period RMSPE is necessary. - Placebo/test-in-space: apply synthetic control to donor units and compare post/pre RMSPE ratios. - Sensitivity to donor pool (leave-one-out) and to covariates used for matching. - Check covariate balance between treated and synthetic control.

3) Regression discontinuity (RD)- Idea: Use a cutoff-based assignment (e.g., price change applies when metric > threshold) and compare observations just above vs. below the threshold — local randomization.- Key assumptions: No manipulation of running variable around cutoff; conditional continuity of potential outcomes at threshold.- Required data: High-resolution observations around cutoff, running variable, outcome, covariates. Large sample density near cutoff.- Diagnostics: - McCrary density test for manipulation of running variable. - Covariate continuity checks: test whether covariates jump at cutoff. - Vary bandwidths and polynomial orders; robust local linear estimates with triangular kernel. - Placebo cutoffs and falsification windows.

Practical BI considerations- Pre-aggregate to the level of inference (user, store, SKU) and keep raw-level for robustness.- Report effect sizes with confidence intervals and robustness table in dashboards.- Combine methods when possible (e.g., DiD with synthetic control weights) and document assumptions clearly for stakeholders.- Automate diagnostics (pre-trend plots, placebo tests, McCrary) so decision-makers see credibility checks alongside estimated impacts.

Unlock Full Question Bank

Get access to hundreds of A/B Test Results Analysis interview questions and detailed answers.

Join thousands of developers preparing for their dream job.