A/B Testing and Optimization Methodology Questions

Discuss your approach to designing, running, and analyzing A/B tests (randomized controlled experiments) to optimize a product or business metric. Cover experiment design fundamentals: forming a testable hypothesis, choosing the unit of randomization, selecting a primary metric plus guardrail and secondary metrics, and estimating sample size and statistical power. Explain how you interpret results (p-values, confidence intervals, statistical versus practical significance) and avoid common pitfalls (novelty effects, peeking, SUTVA violations, confounding, seasonality). Discuss how you prioritize testing opportunities and build a testing roadmap. Ground your answer with concrete examples from your own experience, whether that is testing content elements (headlines, messaging, CTAs, visual design), conversion flows (checkout, signup), pricing, or feature rollouts.

HardTechnical

46 practiced

You're testing a factorial experiment with 3 headline lengths x 2 CTAs x 2 visuals (12 cells). Describe statistical approaches to analyze main effects and interactions while controlling for multiple comparisons. Discuss FWER control (Bonferroni or closed testing), hierarchical testing of hypotheses, and more scalable approaches when the number of contrasts grows.

Sample Answer

**Approach overview**Start with an ANOVA-style factorial model to estimate main effects and interactions:- Fit linear model: outcome ~ headline + CTA + visual + headline:CTA + headline:visual + CTA:visual + headline:CTA:visual- Use Type III sums of squares if unbalanced.

**Primary inference strategy (FWER-focused)**- Global omnibus F-tests for each family (all mains; all two-way interactions; three-way). Only if omnibus significant, proceed to pairwise contrasts — this is hierarchical (gatekeeping).- For family-wise control use closed testing or Bonferroni: - Bonferroni: simple but conservative when many contrasts. - Closed testing: test all intersection hypotheses; more powerful but computationally heavier — can be implemented with sequential rejection (Marcus et al.) for moderate numbers of contrasts.

**Planned contrasts & hierarchical testing**- Pre-specify a small set of primary contrasts (e.g., compare short vs medium vs long headlines via linear trend; CTA A vs B; visuals combined). Test primaries with strict alpha (e.g., Bonferroni across primaries). Secondary/exploratory contrasts are tested only if primaries significant (gatekeeping), reducing multiplicity burden.

**Scalable alternatives when contrasts grow**- Use global methods that are less conservative: - Westfall–Young permutation-based maxT: empirically controls FWER and leverages dependence among tests. - Multilevel (mixed) models with partial pooling: estimate effects and shrink noisy contrasts; focus on posterior intervals (Bayesian hierarchical models) with multiplicity implicitly handled via shrinkage. - FDR control (Benjamini–Hochberg) for exploratory sets where some false discoveries are acceptable. - Tukey or Scheffé for all-pairwise comparisons: Tukey for balanced pairwise, Scheffé for arbitrary contrasts (conservative).

**Practical recommendations (Applied Scientist lens)**- Pre-register primary hypotheses and use hierarchical gatekeeping to protect power.- Use omnibus tests first; follow with permutation-based maxT or closed testing for confirmatory claims.- For large-scale or continuous factorial expansions, prefer hierarchical Bayesian models or mixed-effects with shrinkage — they scale and produce actionable estimates rather than many isolated p-values.- Report effect sizes, CIs, and adjusted p-values; simulate power under realistic effect sizes to choose sample size and multiplicity strategy.

EasyTechnical

45 practiced

In a product experimentation context, define A/B testing and explain the null and alternative hypotheses, Type I and Type II errors, and statistical power. Illustrate each concept with a concrete example from a headline A/B test that measures click-through rate (CTR): what would a Type I error mean here, and what would 'power' represent for the team?

EasyTechnical

43 practiced

Describe A/A tests versus A/B and multi-armed experiments. When would you run an A/A test on headline copy or layout? What specific signals or anomalies (data-quality or implementation issues) would cause you to stop and investigate during an A/A test?

HardTechnical

41 practiced

Implement in Python an alpha-spending sequential stopping rule using an O'Brien-Fleming boundary for a pre-specified number of interim looks k. Create a function obrien_fleming_bounds(alpha, k) that returns nominal two-sided significance thresholds (alpha_i for each look). You may use scipy.stats.norm.ppf for quantiles. Document assumptions, limitations, and how the boundaries should be used in automated monitoring.

Sample Answer

**Approach (brief)** Use the O'Brien–Fleming alpha-spending rule which allocates cumulative alpha at look i as 2*(1 - Phi( z_alpha/ sqrt(t_i) )) inverted; equivalently compute nominal two-sided alpha_i for each look so that the overall two-sided Type I error is alpha across k looks with equally spaced information times t_i = i/k. We'll compute boundaries (two-sided) by solving for z_i such that cumulative spent equals the OBF spending function: alpha_spent(t) = 2*(1 - Phi( z_alpha / sqrt(t) )). Then nominal alpha_i = 2*(1 - Phi(z_i)), where z_i is boundary at look i. Implementation uses root finding per look.

python

import math
from scipy.stats import norm
from scipy.optimize import brentq

def obrien_fleming_bounds(alpha, k):
    """
    Returns list of two-sided nominal alphas for k interim looks (including final).
    Assumes equally spaced information times t_i = i/k and overall two-sided alpha.
    """
    if k < 1 or alpha <= 0 or alpha >= 1:
        raise ValueError("k must be >=1 and 0<alpha<1")
    t = [(i+1)/k for i in range(k)]
    z_alpha = norm.ppf(1 - alpha/2)
    def spent_at_t(z, tt):
        # given boundary z at information fraction tt, OBF cumulative spent:
        return 2 * (1 - norm.cdf(z / math.sqrt(tt)))
    bounds = []
    # For each look solve for z_i such that spent_at_t(z_i, t_i) == target_spent_i
    # target_spent_i is the OBF cumulative alpha at t_i; but OBF defines target via z_alpha
    for tt in t:
        target = spent_at_t(z_alpha, tt)
        # z decreases as target increases; search z in (0, 10)
        z_i = brentq(lambda z: spent_at_t(z, tt) - target, 1e-8, 10.0)
        alpha_i = 2 * (1 - norm.cdf(z_i))  # nominal two-sided alpha at this look
        bounds.append(alpha_i)
    return bounds

**Explanation & usage**- Each alpha_i is the nominal two-sided significance threshold to apply at look i; compare two-sided p-value <= alpha_i to stop. Internally this corresponds to z boundary z_i = norm.ppf(1 - alpha_i/2).- Complexity: O(k * root_iters), negligible for typical k.

**Assumptions & limitations**- Assumes equally spaced information fractions. For unequal spacing, replace t accordingly.- Uses asymptotic normality; valid for large-sample test statistics (z-statistics).- Nominal alphas are two-sided. For one-sided testing adapt alpha allocations.- Numerical root-finding may need bounds tuning for extreme alphas.

**Automated monitoring recommendations**- Pre-compute bounds and store z thresholds. At each look compute test z (or p); if |z| >= z_i (or p <= alpha_i) trigger stop. Log look number, information fraction, test statistic, and decision. Include guardrails for late looks and adjustments if info fractions deviate from plan.

EasyTechnical

59 practiced

Explain the multiple comparisons problem when testing 20 headline variants simultaneously. If you run 20 independent tests each at alpha = 0.05, how many false positives do you expect on average? Suggest at least two concrete statistical approaches to control for multiple comparisons and briefly explain trade-offs.

Unlock Full Question Bank

Get access to hundreds of A/B Testing and Optimization Methodology interview questions and detailed answers.

Join thousands of developers preparing for their dream job.