A/B Test Design & Statistical Rigor Questions

Designing and statistically defending a controlled online experiment: framing a testable hypothesis, defining control and treatment variants, choosing the randomization unit, setting the primary success metric, and computing sample size, power, and minimum detectable effect. Covers the statistical foundations that make a readout trustworthy, including hypothesis testing, p-values, confidence intervals, statistical vs practical significance, and Type I/II error. Emphasizes avoiding the common pitfalls that invalidate a test, such as peeking, multiple-comparison inflation, underpowered designs, and how test duration and stopping rules affect the validity of conclusions.

EasyTechnical

40 practiced

What is Sample Ratio Mismatch (SRM)? Describe how you detect SRM, list common root causes (technical and design), and outline steps you would take when SRM appears during a running experiment.

Sample Answer

Sample Ratio Mismatch (SRM) occurs when the observed assignment counts across experiment variants differ significantly from the expected randomization ratio (e.g., 50/50). It’s a red flag for randomization or data-collection problems that can bias results.

How I detect SRM- Automated monitoring: compute observed vs expected counts each day.- Statistical test: chi-squared goodness-of-fit or binomial test to get a p-value for deviation.- Visualization: time-series of assignment proportions to spot when mismatch began.- Segment checks: compare counts across platforms, regions, or user cohorts to localize the issue.

Common root causes- Technical - Instrumentation bugs (event dropped, duplicate assignment, wrong experiment key) - SDK/version drift or rollout differences between clients - Race conditions in assignment logic or caching causing sticky-but-wrong assignments - Data pipeline loss or ETL filtering- Design / Product - Non-random enrollment rules (e.g., eligibility logic applied after assignment) - Quotas, throttling, or ramp rules in the experiment platform - Bot traffic, test accounts, or user-level deduplication disparity - Cross-device identity resolution inconsistencies

Steps to take when SRM appears (triage + remediation)1. Pause critical decisions — don’t trust effect estimates until resolved.2. Confirm: rerun the chi-squared test and inspect time-series to identify start time.3. Scope: break down by platform, version, country, acquisition source, and user-id hashing to localize the source.4. Check instrumentation and logs: assignment service logs, SDK versions, rollout configs, and recent deploys.5. Validate randomization: sample raw assignment calls to ensure deterministic hashing works.6. Remediate: fix bug (code or config), remove problematic cohorts, or stop sending new users to the experiment.7. Recover data: if feasible, replay logs or reprocess ETL to recover lost events; otherwise mark affected period as invalid.8. Post-mortem: document root cause, add monitoring alerts, increase test coverage, and update runbooks.

Key principle: act fast, scope precisely, and avoid making business decisions from biased data.

MediumTechnical

40 practiced

Your web experiment bucketing is cookie-based, and your mobile app uses device_id. After backend identity merging, many users appear in both variants causing contamination. How would you resolve identity and reassign treatment where necessary without invalidating prior experiments? Propose a future-proof bucketing strategy that avoids double-counting and cross-device contamination.

Sample Answer

Situation: Cookie-based web buckets and device_id mobile buckets were independently assigned. After backend identity merging, many users show exposures in both variants, contaminating experiments.

Plan to resolve current contamination without invalidating prior experiments:1. Preserve raw exposure logs (timestamp, variant, assignment_id, identifier_type, identifier_value). Do not delete historical data.2. Build a post-merge mapping table user_id <- {cookie_ids, device_ids} with merge timestamps and confidence.3. For historical analyses, treat exposures according to the chronology and merge policy: - If both exposures occurred before identity merge, keep both but mark as "distinct identifiers"; run sensitivity analyses (intention-to-treat keeping original assignment vs decontaminated per-user). - If one exposure occurred after another and the merge time indicates same person, classify a single canonical exposure using a deterministic rule (e.g., earliest exposure wins for assignment) while preserving originals for audit.4. For ongoing experiments, reassign treatment only for future exposure. Do not retroactively change what users saw. Instead, collapse duplicate exposures into a single user-level exposure for analysis using a canonicalization rule and record which exposure was used.

Future-proof bucketing strategy to avoid double-counting/cross-device contamination:1. Use a unified, stable identifier (canonical_user_id) at bucketing time whenever available. For anonymous contexts, generate a durable assignment_id tied to device+browser but designed to be reconciled later.2. Deterministic hashing: bucket on hash(canonical_user_id, experiment_id, salt, assignment_epoch) so assignment is stable across devices once user is identified.3. Assignment service: centralize a low-latency assignment/lookup service that: - Looks up canonical_user_id from incoming identifier(s) - Returns deterministic variant and assignment_id - Logs identifier->assignment mapping4. Versioned assignments and metadata: include experiment_version and assignment_epoch so you can change bucketing logic safely in future without mixing cohorts.5. Exposure deduplication at analysis time: treat exposures per canonical_user_id within the experiment window. Use rules: prefer authenticated canonical_id; if multiple, use earliest assignment unless experiment design specifies otherwise.6. Instrument identity events and propagate merge events to assignment service to retroactively link logs for analysis (not to change past user-visible treatments).

Statistical safeguards:- Report both ITT (using original assignments) and decontaminated per-user analyses; use sensitivity checks.- If contamination is substantial, consider excluding contaminated users or using inverse-probability weighting to correct estimates.

Why this works:- Preserves experiment integrity (what users actually saw).- Enables correct user-level analysis after identity resolution.- Centralized deterministic bucketing prevents future cross-device splits once users are unified, while versioning maintains reproducibility.

HardTechnical

51 practiced

Your primary metric is time-to-first-purchase. Explain how you would analyze A/B test data using survival analysis: Kaplan-Meier estimates, log-rank test, Cox proportional hazards model, and how to handle censoring and reporting lift in hazard or median time terms.

Sample Answer

First clarify the objective and data:- Metric: time-to-first-purchase measured from exposure/randomization to event (purchase).- Define event indicator (1=purchase observed, 0=censored). Censor at end-of-follow-up or loss-to-follow-up.- Use intent-to-treat population and preserve randomization.

1) Kaplan–Meier (KM)- Estimate survival function S(t)=P(T>t) per arm to visualize probability of not yet purchased over time.- Plot KM curves with confidence bands and number at risk table; useful to inspect early/late separation and median times.- Code (lifelines):

python

from lifelines import KaplanMeierFitter
kmf = KaplanMeierFitter()
kmf.fit(durations=times_control, event_observed=events_control, label='control')
ax = kmf.plot_survival_function()
kmf.fit(times_treatment, events_treatment, label='treatment')
kmf.plot_survival_function(ax=ax)

2) Log-rank test- Nonparametric test comparing whole survival curves (null: no difference in hazard functions).- Use when proportional hazards roughly hold; returns p-value for overall difference.- In Python: lifelines.statistics.logrank_test.

3) Cox proportional hazards model- Fit Cox model to estimate hazard ratio (HR) between treatment and control, possibly adjusting for covariates (age, channel, prior behavior).- Interpret HR: HR<1 means treatment reduces instantaneous hazard of remaining event-free (i.e., increases rate of purchase). Example: HR=0.8 → 20% higher instantaneous purchase rate in treatment (1/0.8 interpretation depends on coding).- Code:

python

from lifelines import CoxPHFitter
df = pd.DataFrame({...})  # columns: duration, event, treatment, covariates
cph = CoxPHFitter()
cph.fit(df, 'duration', event_col='event')
cph.print_summary()

- Check proportional hazards assumption (Schoenfeld residuals). If violated, use time-varying covariates, stratified Cox, or split time windows.

4) Handling censoring- Right-censoring is standard; include censored observations in KM and Cox.- If administrative censoring differs by arm, ensure follow-up windows are equal or adjust via inverse-prob weighting.- For heavy loss-to-follow-up, perform sensitivity analyses (best/worst-case, multiple imputation).

5) Reporting lift and practical metrics- Primary inferential: report HR with 95% CI and p-value (e.g., HR=0.75, 95% CI [0.62–0.90], p=0.002).- Translate to user-friendly metrics: - Percent change in instantaneous purchase rate: (1 − HR)*100% (if HR<1 and treatment coded 1 reduces time). - Median time-to-event: report medians per arm and difference/ratio with CIs from KM (e.g., median_control=14 days, median_treatment=10 days → ~29% reduction). - Absolute probability improvement at specific time points: ΔS(t)=S_control(t)−S_treatment(t) → difference in cumulative purchase probability by day 7/30.- Example phrasing: "Treatment reduced median time-to-first-purchase from 14 to 10 days (median ratio 0.71), HR=0.75 (95% CI 0.62–0.90), meaning a 25% higher instantaneous purchase rate."

6) Robustness and presentation- Show KM plots with number-at-risk, HR and medians in table, log-rank p-value, covariate-adjusted Cox results, PH diagnostics.- Sensitivity: subgroup analyses, truncated follow-up, competing risks if other terminal events exist.- Emphasize practical significance (e.g., revenue lift from earlier purchases) and uncertainty.

This workflow preserves censoring information, leverages both descriptive (KM) and inferential (log-rank, Cox) methods, adjusts for confounders, checks assumptions, and reports both statistical and business-relevant effects.

HardTechnical

45 practiced

How would you estimate long-term customer lifetime value (LTV) from a 30-day experiment? Discuss modeling choices for extrapolation (parametric survival, bootstrapping, Bayesian hierarchical), biases introduced by short windows, use of retention cohorts, and ways to quantify uncertainty in LTV forecasts.

Sample Answer

Situation: You have only 30 days of experimental data but need to estimate long-term LTV. Approach: treat the 30‑day window as censored observation and combine cohort-based descriptive analysis with parametric and probabilistic extrapolation plus robust uncertainty quantification.

Modeling choices for extrapolation:- Parametric survival models (Weibull, Gompertz, Exponential): fit time‑to-churn on observed durations and extrapolate hazard/retention curves. Good when you believe a functional form (e.g., decreasing hazard). Use AIC/BIC and visual PPCs to pick family.- Semi/nonparametric (Kaplan–Meier + spline hazards): avoid rigid shape assumptions; use spline or piecewise-constant hazard for more flexible extrapolation.- Bayesian hierarchical models: pool information across cohorts or segments (e.g., acquisition channel, geography) to borrow strength and propagate parameter uncertainty. Priors regularize implausible long tails.- Bootstrapping: nonparametric resampling of users/cohorts to capture sampling variability; combine with model-based extrapolation to capture model + sample uncertainty.

Biases from short windows:- Left-censoring of long lifetimes -> optimistic/uncertain tails- Early adopter behavior: conversion/retention in first 30 days may not represent steady-state- Seasonal/cohort effects missed (holiday cohorts bias)- Treatment effects that manifest after 30 days are invisible

Use of retention cohorts:- Build weekly/daily cohorts from acquisition date; compute retention curves and per-user revenue within cohort.- Examine cohort heterogeneity; stratify models (hierarchical) by cohort covariates to avoid aggregation bias.- Use cohort extrapolation: fit survival within each cohort and aggregate weighted by expected cohort sizes.

Quantifying uncertainty:- Posterior predictive intervals from Bayesian models (e.g., 90% credible intervals for LTV).- Parametric bootstrap: resample parameters from fitted covariance matrix and recompute LTV.- Nonparametric bootstrap at user/cohort level to capture sampling variance.- Scenario analysis: pessimistic/median/optimistic hazard shapes; sensitivity to tail assumptions (truncate at plausible max lifetime).- Report expected LTV plus percentile bands and decomposition of uncertainty (sampling vs model vs structural).

Practical steps:1. Exploratory cohort retention and revenue-per-user curves.2. Fit several models (Weibull, spline, Bayesian hierarchical) and compare fits and forecasts.3. Use bootstrapping and Bayesian posteriors to produce prediction intervals.4. Communicate assumptions (tail behavior, cohort representativeness), report multiple scenarios, and update estimates as more data arrives.

This combination yields defensible LTV estimates, explicit uncertainty, and a clear plan to refine forecasts as longer-term data accrues.

MediumTechnical

52 practiced

Implement the Benjamini-Hochberg (BH) false discovery rate procedure in Python. Input: array-like p_values and target_fdr q. Output: boolean array is_significant and array of adjusted p-values (BH-adjusted). Use standard sorting-based BH and handle edge cases.

Sample Answer

Approach: Use the standard sorting-based Benjamini–Hochberg procedure: sort p-values ascending, compute BH thresholds (i/m)*q, find largest i with p_(i) <= (i/m)q, mark those as significant; compute adjusted p-values (p_adj) as the minimum of (m/i)*p_(i) enforced to be monotonic non-decreasing when mapped back to original order and capped at 1. Handle NaNs and edge q values.

python

import numpy as np

def benjamini_hochberg(p_values, q=0.05):
    """
    p_values: array-like of p-values (floats). NaNs allowed (treated as non-significant, p_adj = NaN).
    q: target FDR in (0,1]
    Returns: is_significant (bool array), p_adj (float array)
    """
    p = np.asarray(p_values, dtype=float)
    m = p.size

    if not (0 < q <= 1):
        raise ValueError("q must be in (0, 1].")

    # Prepare outputs
    p_adj = np.full(m, np.nan)
    is_significant = np.zeros(m, dtype=bool)

    # Indices of non-NaN p-values
    valid = ~np.isnan(p)
    if not np.any(valid):
        return is_significant, p_adj

    pv = p[valid]
    m_valid = pv.size

    # Sort p-values and keep indices
    order = np.argsort(pv)
    pv_sorted = pv[order]

    # BH adjusted (initial)
    ranks = np.arange(1, m_valid + 1)  # 1-based ranks
    q_factors = (m_valid / ranks) * pv_sorted  # (m/i) * p_i

    # Enforce monotonicity: p_adj_sorted[i] = min_{j>=i} q_factors[j]
    p_adj_sorted = np.minimum.accumulate(q_factors[::-1])[::-1]
    p_adj_sorted = np.minimum(p_adj_sorted, 1.0)

    # Map adjusted p-values back to original positions among valid entries
    p_adj_valid = np.empty(m_valid)
    p_adj_valid[order] = p_adj_sorted
    p_adj[valid] = p_adj_valid

    # Determine significance: find largest i where p_sorted[i] <= (i/m_valid)*q
    thresholds = (ranks / m_valid) * q
    passed = pv_sorted <= thresholds
    if np.any(passed):
        max_i = np.max(np.nonzero(passed)[0])  # zero-based index
        # All p_sorted[0..max_i] are significant
        sig_sorted = np.zeros_like(pv_sorted, dtype=bool)
        sig_sorted[: max_i + 1] = True
        sig_valid = np.empty(m_valid, dtype=bool)
        sig_valid[order] = sig_sorted
        is_significant[valid] = sig_valid

    return is_significant, p_adj

Key points:- Time complexity O(m log m) due to sorting; space O(m).- Edge cases: handles NaNs, q outside (0,1] raises error, returns no-significance when no valid p-values.- Adjusted p-values are monotonic and capped at 1.0. Alternative: use independent vs. positive dependence assumptions (storey method) if desired.

Unlock Full Question Bank

Get access to hundreds of A/B Test Design & Statistical Rigor interview questions and detailed answers.

Join thousands of developers preparing for their dream job.