A/B Test Design & Statistical Rigor Questions

Designing and statistically defending a controlled online experiment: framing a testable hypothesis, defining control and treatment variants, choosing the randomization unit, setting the primary success metric, and computing sample size, power, and minimum detectable effect. Covers the statistical foundations that make a readout trustworthy, including hypothesis testing, p-values, confidence intervals, statistical vs practical significance, and Type I/II error. Emphasizes avoiding the common pitfalls that invalidate a test, such as peeking, multiple-comparison inflation, underpowered designs, and how test duration and stopping rules affect the validity of conclusions.

HardTechnical

36 practiced

Hard: Implement a regression-based analysis (in Python/pandas or pseudocode) that estimates treatment effect with interaction terms for segments (e.g., country, device), includes robust standard errors clustered by user, and produces a table of segment-level treatment effects with CIs. Describe assumptions and limitations.

Sample Answer

Approach: fit a linear regression of outcome on treatment, segment indicators, and treatment×segment interactions; compute clustered (by user) robust standard errors; extract segment-level treatment effects = coef(treatment) + coef(treatment:segment) with CIs via linear combinations.

python

import pandas as pd
import statsmodels.formula.api as smf
import statsmodels.api as sm
import numpy as np

# df columns: user_id, outcome, treatment (0/1), segment (categorical, e.g., country/device)
df['segment'] = df['segment'].astype('category')
# use treatment as numeric 0/1
model = smf.ols('outcome ~ treatment * C(segment)', data=df).fit(
    cov_type='cluster', cov_kwds={'groups': df['user_id']}
)

# base treatment effect is coef of treatment (for baseline segment)
params = model.params
cov = model.cov_params()

# get list of segments and baseline
segments = df['segment'].cat.categories.tolist()
baseline = segments[0]

results = []
for seg in segments:
    if seg == baseline:
        # effect = coef(treatment)
        coef_name = 'treatment'
        eff = params[coef_name]
        se = np.sqrt(cov.loc[coef_name, coef_name])
    else:
        inter_name = f'treatment:C(segment)[T.{seg}]'
        eff = params['treatment'] + params.get(inter_name, 0.0)
        # variance of sum: Var(treat)+Var(inter)+2*Cov
        var = (cov.loc['treatment','treatment'] +
               cov.loc.get(inter_name, inter_name) +
               2*cov.loc.get('treatment', inter_name))
        se = np.sqrt(var)
    ci_low = eff - 1.96*se
    ci_high = eff + 1.96*se
    results.append({'segment': seg, 'effect': eff, 'se': se, 'ci_low': ci_low, 'ci_high': ci_high})

seg_table = pd.DataFrame(results)
print(seg_table)

Key concepts:- Interaction terms allow heterogeneous treatment effects by segment.- Clustered SEs by user account for intra-user correlation (repeated observations).

Assumptions:- Linearity and additive effects within the model form.- No unobserved confounding (treatment as-good-as-random or controlled covariates).- Sufficient cluster count and cluster sizes for reliable clustered SEs.

Limitations:- If treatment assignment is biased, estimates are biased—consider IVs or propensity-score adjustments.- Small number of clusters undermines clustered SE validity (use wild bootstrap if few clusters).- Nonlinear heterogeneous effects not captured—consider subgroup-specific models or hierarchical/mixed models for shrinkage.

EasyTechnical

39 practiced

Explain bias and confounding with examples relevant to content A/B tests (seasonality, marketing campaigns, segmented traffic). Describe methods to mitigate confounding such as blocking, stratification, time-window controls, and pre-post adjustments.

Sample Answer

Bias: systematic difference between measured treatment effect and the true causal effect. In A/B tests bias arises when randomization or measurement is compromised. Example: running an experiment over the December holidays will inflate engagement in both arms; if one arm receives more exposure to holiday-themed creatives, the measured lift is biased by seasonality.

Confounding: a variable that affects both assignment and outcome, producing spurious associations. Examples in content A/B tests:- Seasonality: weekday vs weekend traffic has different baseline engagement; if treatment traffic is heavier on weekends, you’ll conflate treatment with day-of-week effects.- Marketing campaigns: a paid campaign that sends users to treatment pages will increase conversions independent of the treatment.- Segmented traffic: mobile users behave differently than desktop users; if assignment disproportionately lands mobile users into one arm, device is a confounder.

Mitigation methods:- Blocking (randomized blocks): group users by known confounders (date, device, user cohort) and randomize within each block. Ensures balance on those factors.- Stratification: analyze and report treatment effects within strata (e.g., mobile vs desktop, new vs returning). Combine with weighted averaging to get an overall effect while preserving subgroup balance.- Time-window controls: run parallel experiments across identical time windows for both arms; avoid starting/stopping arms at different times. Use mirrored windows to remove time trends.- Pre-post adjustments (covariate adjustment / ANCOVA): use pre-experiment metrics (historical engagement) as covariates in regression to improve precision and correct for baseline imbalances. For stepped interventions, difference-in-differences (pre/post relative change vs control) helps remove persistent trends.Practical tips:- Monitor assignment balance on key covariates continuously.- Log exposures to external campaigns and either exclude or include campaign indicators in models.- Pre-register primary metric and analysis plan to avoid post-hoc bias.These approaches together reduce bias from known confounders; unknown confounders require cautious interpretation, sensitivity analyses, and possibly longer randomized windows.

HardTechnical

39 practiced

Design and validate an uplift modeling approach for targeting treatment to users most likely to benefit. Outline data collection (treatment labels, features), model architecture choices, evaluation metrics (Qini, uplift curves), offline validation strategy, and how you'd safely roll this into production experiments.

Sample Answer

Start by clarifying objective: maximize incremental outcome (Y) caused by applying treatment vs control for each user (individual treatment effect, ITE). Then design data, models, validation, and safe rollout.

Data collection- Treatment labels: ideally from randomized controlled trials (A/B test) to avoid confounding. Log assignment, compliance, and exposure timestamps. If using observational data, collect rich instrumental variables and compute propensity scores.- Outcomes: primary business KPI (binary or continuous) measured post-treatment with appropriate window.- Features: pre-treatment covariates (demographics, behavior history, recency/frequency, device, segment). Include temporal/contextual features and interaction candidates. Store feature provenance and versions.

Model architecture choices- Baselines: two-model (T-learner: separate outcome models for treatment/control), S-learner (single model with treatment indicator), and meta-learners (X-learner for imbalanced treatment ratios).- Advanced: causal forests / generalized random forest (grf), uplift trees, or doubly-robust learners (DR-learner) for bias reduction. For large-scale/complex features, use representation-learning (TARNet/Dragonnet) with treatment embedding and targeted regularization.- Practical choice: start with X-learner + causal forest for interpretability and an NN-based DR-learner for scale.

Evaluation metrics- Uplift curve and Qini curve; compute Qini coefficient and area under uplift curve (AUUC). Report uplift@k (top k% targeted).- Policy value: expected incremental gain if applying treatment to top-N users.- Calibration: check that predicted uplift ranks align with observed uplift via binned estimates.- Statistical uncertainty: bootstrap / confidence intervals for Qini and uplift@k; permutation tests for significance.

Offline validation strategy- Use cross-validation stratified by treatment to preserve treatment/control ratios.- Use out-of-fold predictions to compute unbiased uplift metrics.- Check covariate balance and overlap; limit extrapolation where propensity is near 0/1.- Conduct sensitivity analyses: estimate with propensity-weighted models, run placebo tests (treatment label shuffled), and subgroup analyses.- Sample size/power: compute detectable uplift given baseline conversion and desired minimum detectable effect to size experiments.

Safe production rollout- Start with shadow mode: score users and compare predicted uplift with randomized holdout outcomes without acting.- Run a constrained randomized experiment (target-policy A/B test): apply model to a portion (e.g., 5-10%) while keeping randomized control elsewhere; measure actual incremental impact and key safety metrics (metrics degradation, fairness).- Progressive ramp: increase traffic in steps with automated checks and kill-switch if adverse outcomes exceed thresholds.- Monitoring & observability: monitor online policy value, calibration drift, feature distribution shift, treatment assignment leakage, and business KPIs. Log exposures and outcomes for continual offline retraining.- Retraining & governance: schedule periodic retrain with newest labeled data, maintain model lineage, and run bias/fairness audits.

Why this approach- Randomized labels + doubly-robust learners minimize bias; Qini/AUUC and policy value directly measure business impact; shadowing and small randomized rollouts reduce risk. Combining interpretable causal forests with scalable DR/NN models balances explainability and performance.

HardTechnical

46 practiced

Case study/dataset: You have the following aggregate results from an A/B test run for 30 days: Control: users=200k, conversions=20k. Treatment: users=200k, conversions=21k. Provide a step-by-step analysis plan to determine if the treatment is truly beneficial and production-ready. Include sanity checks, significance test, uplift CI, power retrospective, metric segmentation, and pre-trend checks.

Sample Answer

1) Clarify objective & metric- Primary metric: conversion rate (binary). Confirm business impact, risk tolerance, success threshold (absolute or relative uplift).

2) Sanity checks / data integrity- Verify user counts deduped and mutually exclusive; check randomization key distribution across users, cohorts, devices.- Check instrumentation: no missing events, conversion definition stable, no delayed attribution differences.- Check temporal balance: users/day similar across arms.

3) Point estimate & significance- Compute rates: control 10.0% (20k/200k), treatment 10.5% (21k/200k) → absolute uplift 0.5pp, relative +5%.- Perform two-proportion z-test (or Fisher/chi-square) to get p-value.- Report test statistic, p-value, and whether it crosses alpha (e.g., 0.05).

4) Uplift confidence interval- Compute 95% CI for difference in proportions (use pooled SE or Wilson if needed). Report CI in absolute and relative terms to assess practical significance.

5) Retrospective power & minimum detectable effect (MDE)- Calculate achieved power given observed effect and sample sizes; compute MDE at planned alpha/power. If power <80%, treat result as inconclusive even if p<0.05.

6) Check for peeking & multiple comparisons- Confirm no interim peeking or adjust for it; if multiple metrics/segments tested, control FDR or Bonferroni.

7) Metric segmentation / heterogeneity- Segment by device, geography, new vs. returning users, traffic source, time of day. Compute per-segment rates and CIs; check consistency and interaction effects (logistic regression with treatment*segment).- Flag segments with opposite effects or high variance—investigate.

8) Pre-trend and temporal diagnostics- Run pre-period balance on similar metrics (prior week/month) to ensure no pre-existing drift.- Time series of daily conversion rate and cumulative z-score; look for post-randomization anomalies, launch-day spikes, or novelty effects.

9) Secondary metrics & safety checks- Check engagement, retention, revenue, error rates, latency, resource usage. Ensure no negative trade-offs.

10) Robustness tests- Run alternative analyses: bootstrap CI, Bayesian credible interval, regression adjusted for covariates to reduce variance.- Run per-user vs per-session definitions.

11) Production-readiness & rollout plan- If effect is statistically and practically significant and safety checks pass: propose staged rollout (e.g., 10% → 50% → 100%) with monitoring alerts on key metrics and kill-switch.- If inconclusive: extend experiment or increase sample (compute additional sample needed), or run targeted follow-ups on promising segments.

12) Documentation & monitoring- Document analysis, assumptions, and decision criteria. Deploy real-time dashboards and automated anomaly detection for first 2-4 weeks of rollout.

Deliverables: p-value, uplift CI, retrospective power & MDE, segment table, pre-trend plots, regression/robustness results, recommended rollout and monitoring checklist.

HardTechnical

42 practiced

You observe a small but statistically significant uplift in the A/B test for conversion rate driven entirely by a tiny subset of heavy users. Describe how you'd investigate whether the effect is real, robust, or an artifact, and how to present actionable conclusions to stakeholders.

Sample Answer

Framework: treat this as a robustness + causal inference investigation, then produce an actionable recommendation with clear uncertainty.

Investigation steps1. Validate randomization & data quality- Check assignment balance on pre-period metrics and covariates (demographics, historical conversion). Run t-tests/KS tests and standardized mean differences.- Ensure no logging or bucketing bugs that could route heavy users disproportionately.

2. Convert-to-user-level and re-run- Aggregate conversions to the user level (not event-level) to avoid heavy-user domination. Compute per-user conversion rates; compare means between groups using bootstrap and cluster-robust SEs.

3. Subgroup and influence analysis- Identify heavy-user threshold (e.g., 95th percentile); compute contribution-to-uplift by percentile buckets.- Run leave-one-out / leave-group-out influence: drop top N users or top X% and re-estimate effect.- Compute influence functions or use robust estimators (median difference, trimmed mean, Winsorization).

4. Statistical robustness- Run permutation tests on user-level data (random reassignments) to get null distribution.- Adjust for multiple comparisons if you looked across many subgroups (BH/FDR).- Estimate effect heterogeneity via regression with treatment × heavy_user indicator, include covariates; check interaction significance.

5. Practical checks- Examine user journeys of heavy users for behavioral or product differences — are they bots, internal users, premium accounts?- Check duration: is uplift sustained over time or a short spike?

Presentation to stakeholders- One-slide executive summary: headline (effect size, CI), who drives it (X% of users cause Y% of uplift), recommendation (rollout, iterate, or hold), and risk level.- Support with 2 visuals: cumulative contribution curve (percentile vs. contribution) and user-level treatment-control distribution with robustness overlays (trimmed/winsorized/median).- Provide concrete options with estimated impact and confidence: e.g., - Proceed to targeted rollout for heavy users (expected lift A, risk B) - Abort global rollout; run follow-up experiment restricting to non-heavy users - Fix data/segmentation issues and re-run

Actionable next steps- If effect driven by valid heavy users: run experiments targeted at that cohort, instrument long-term metrics.- If artifact: correct logging/randomization, re-run test.- Always implement monitoring: automated alerts for heavy-user-driven swings and rerun user-level permutation tests periodically.

This approach quantifies whether the uplift is real, explains its origin, and gives stakeholders clear trade-offs and a path forward.

Unlock Full Question Bank

Get access to hundreds of A/B Test Design & Statistical Rigor interview questions and detailed answers.

Join thousands of developers preparing for their dream job.