InterviewStack.io LogoInterviewStack.io

A/B Testing and Optimization Methodology Questions

Discuss your approach to designing, running, and analyzing A/B tests (randomized controlled experiments) to optimize a product or business metric. Cover experiment design fundamentals: forming a testable hypothesis, choosing the unit of randomization, selecting a primary metric plus guardrail and secondary metrics, and estimating sample size and statistical power. Explain how you interpret results (p-values, confidence intervals, statistical versus practical significance) and avoid common pitfalls (novelty effects, peeking, SUTVA violations, confounding, seasonality). Discuss how you prioritize testing opportunities and build a testing roadmap. Ground your answer with concrete examples from your own experience, whether that is testing content elements (headlines, messaging, CTAs, visual design), conversion flows (checkout, signup), pricing, or feature rollouts.

MediumTechnical
41 practiced
Explain the Bonferroni correction and the Benjamini-Hochberg (BH) procedure for multiple comparisons. In an experimentation program that tracks 20 secondary metrics per test, describe how each method affects Type I and Type II error rates and when you would prefer BH over Bonferroni. Provide a short example showing corrected p-value thresholds.
HardTechnical
51 practiced
When thousands of experiments run concurrently across product surfaces, overlapping treatments can interact and bias estimates (cross-experiment interference). Describe methods to detect interference (co-occurrence matrices, interaction tests), measure its impact, and mitigate it at scale: namespace bucketing, enforced orthogonality, hierarchical or multilevel models to estimate interactions, and organizational policies for overlapping experiments.
HardTechnical
47 practiced
List signs that p-hacking or researcher degrees of freedom exist in an experimentation program (many p-values just below 0.05, selective reporting, post-hoc subgrouping) and propose governance policies and technical controls to prevent them: experiment pre-registration, central experiment registry with pre-specified metrics, locked analysis pipelines or notebooks, audit logs, role-based access, and automated checks for multiple-testing abuses.
MediumTechnical
46 practiced
Given table events(user_id INT, event_type VARCHAR, variant VARCHAR, occurred_at DATE), write an ANSI SQL query to compute 7-day retention per variant for users whose first event occurred between '2025-01-01' and '2025-01-07'. Output: cohort_date, variant, users_in_cohort, retained_day_7_count, retention_rate. Use window functions or CTEs and explain assumptions about time zones and missing data.
HardTechnical
54 practiced
You are asked to build a 12-month experimentation roadmap aligned with company OKRs. Draft the components: prioritized list of experiments (quick wins vs strategic bets), scheduling and resource allocation, infrastructure/tech debt work (e.g., instrumentation, analytics), measurement plan and KPIs for the experimentation program (e.g., cycle time, % significant tests, quality of experiment registry), and stakeholder communication cadence. Explain trade-offs when capacity is limited.

Unlock Full Question Bank

Get access to hundreds of A/B Testing and Optimization Methodology interview questions and detailed answers.

Sign in to Continue

Join thousands of developers preparing for their dream job.