A/B Test Design & Statistical Rigor Questions

Designing and statistically defending a controlled online experiment: framing a testable hypothesis, defining control and treatment variants, choosing the randomization unit, setting the primary success metric, and computing sample size, power, and minimum detectable effect. Covers the statistical foundations that make a readout trustworthy, including hypothesis testing, p-values, confidence intervals, statistical vs practical significance, and Type I/II error. Emphasizes avoiding the common pitfalls that invalidate a test, such as peeking, multiple-comparison inflation, underpowered designs, and how test duration and stopping rules affect the validity of conclusions.

HardTechnical

42 practiced

Describe how to instrument, monitor, and analyze an A/B test using Amplitude or Mixpanel as the analytics back-end. Cover experiment event instrumentation, calculating treatment assignment exposure, power and minimum detectable effect (MDE), detecting Sample Ratio Mismatch (SRM), and how you'd report results with confidence intervals and practical significance.

Sample Answer

**Brief framing (Design Researcher lens)** I treat A/B tests as mixed quantitative + qualitative experiments: instrumentation must capture who saw what, how they engaged, and key UX signals so results map to design insights and decisions.

**Event instrumentation (Amplitude/Mixpanel)** - Track one canonical exposure event (e.g., experiment_view) with properties: experiment_id, variant, assignment_method, timestamp, and context (page, device, user_cohort). - Track outcome events (e.g., sign_up, task_complete, dropoff_step) and user-level attributes (user_id, anonymous_id). - In Amplitude/Mixpanel use the SDK to set a persistent user property "experiment_variant" so downstream events can be attributed. Log both impression and conversion events to avoid misattribution.

**Calculating treatment assignment exposure** - Define exposure as first qualifying impression within analysis window. Count unique users with impression_event + variant property. Use cohorts in Amplitude/Mixpanel to filter by experiment_id + variant and dedupe by user_id.

**Power and MDE** - Precompute sample size using baseline conversion p0, desired power (1-β, e.g., 0.8), alpha (e.g., 0.05), and target MDE. Use online calculators or:

text

SE = sqrt( p1*(1-p1)/n1 + p2*(1-p2)/n2 )

Plain English: SE is the standard error of difference in proportions; solve for n given z-scores for alpha/power and target effect.

**Detecting SRM** - Compare observed variant counts to expected using chi-square or binomial test. Flag p < 0.01. Investigate randomization bug, instrumentation lag, or interference. In Mixpanel/Amplitude export counts by bucket and run a simple binomial test.

**Reporting results** - Report raw counts, conversion rates, absolute and relative lift, 95% confidence intervals, p-values, and practical significance (business/UX impact). Compute CI for difference:

text

CI_lower = (p1 - p2) - z * SE
CI_upper = (p1 - p2) + z * SE

- Translate numbers into user-centered implications (e.g., “this variant reduces task time by Xs for 10% of users — expect Y fewer support tickets/month”). Visualize with cohort breakdowns and retention curves. Call out limitations (SRM, non-compliance, novelty effects) and recommend follow-ups (qual studies on notable segments, longer run for engagement metrics).

Example: In Amplitude I’d create an experiment cohort for each variant, export user-level exposures, calculate SRM, run power check, compute CIs, and add qualitative session recordings for anomalous segments to explain why a lift occurred.

Unlock Full Question Bank

Get access to hundreds of A/B Test Design & Statistical Rigor interview questions and detailed answers.

Join thousands of developers preparing for their dream job.