A and B Test Design Questions

Designing and running A and B tests and split tests to evaluate product and feature changes. Candidates should be able to form clear null and alternative hypotheses, select appropriate primary metrics and guardrail metrics that reflect both product goals and user safety, choose randomization and assignment strategies, and calculate sample size and test duration using power analysis and minimum detectable effect reasoning. They should understand applied statistical analysis concepts including p values confidence intervals one tailed and two tailed tests sequential monitoring and stopping rules and corrections for multiple comparisons. Practical abilities include diagnosing inconclusive or noisy experiments detecting and mitigating common biases such as peeking selection bias novelty effects seasonality instrumentation errors and network interference and deciding when experiments are appropriate versus alternative evaluation methods. Senior candidates should reason about trade offs between speed and statistical rigor plan safe rollouts and ramping define rollback plans and communicate uncertainty and business implications to technical and non technical stakeholders. For developer facing products candidates should also consider constraints such as small populations cross team effects ethical concerns and special instrumentation needs.

HardTechnical

89 practiced

Implement in Python a simplified group-sequential early stopping rule using an alpha-spending function (e.g., O'Brien-Fleming approximation). Your function will receive running cumulative successes and totals for control and treatment at each interim and should return stop-for-efficacy, stop-for-futility, or continue while attempting to control overall alpha=0.05. Explain assumptions and limitations in comments.

Sample Answer

To solve this, use an alpha-spending O'Brien–Fleming (Lan-DeMets) approximation to compute, at each interim, the cumulative alpha spent and corresponding two-sided critical z. Compute a z-statistic for difference in proportions (two-sample z-test, pooled variance). Stop for efficacy if |z| > z_crit. For futility, use a simple conditional-power rule: project final z if current effect persists to planned final sample; if conditional power < threshold (e.g., 0.2), stop for futility.

python

import math
from scipy.stats import norm

def obrien_fleming_alpha_spent(info_frac, alpha=0.05):
    """
    Lan-DeMets O'Brien-Fleming approximate spending:
    alpha(t) = 2 * (1 - Phi( Z_{1-alpha/2} / sqrt(t) ))
    where t in (0,1] is information fraction.
    """
    if info_frac <= 0:
        return 0.0
    z_half = norm.ppf(1 - alpha / 2.0)
    spent = 2.0 * (1.0 - norm.cdf(z_half / math.sqrt(info_frac)))
    return min(max(spent, 0.0), alpha)

def two_sample_z(success_a, n_a, success_b, n_b):
    # pooled proportion
    p_pool = (success_a + success_b) / (n_a + n_b)
    se = math.sqrt(p_pool * (1 - p_pool) * (1.0 / n_a + 1.0 / n_b))
    if se == 0:
        return 0.0
    p_diff = (success_b / n_b) - (success_a / n_a)
    return p_diff / se  # positive means treatment > control

def sequential_decision(cum_success_ctrl, cum_n_ctrl, cum_success_trt, cum_n_trt,
                        planned_total_ctrl, planned_total_trt,
                        interim_index, total_interims,
                        alpha=0.05, futility_cp_thresh=0.2):
    """
    Returns: 'stop-for-efficacy', 'stop-for-futility', or 'continue'
    Assumptions & limitations (in-code comment):
    - Uses two-sided O'Brien-Fleming alpha spending approximation.
    - Assumes information fraction ~ (current total N) / (planned final total N).
    - Uses two-sample pooled z-test (large-sample approx).
    - Futility rule: conditional power under current observed effect projecting to final sample.
      This is a pragmatic approximate rule (not exact group-sequential beta spending).
    - Planned totals must be > current totals. If equal, treat as final analysis.
    - Does not adjust for covariates, unequal allocation beyond planned, or multiplicity beyond interims.
    """
    # compute information fraction (based on total subjects)
    current_info = (cum_n_ctrl + cum_n_trt)
    planned_info = (planned_total_ctrl + planned_total_trt)
    info_frac = min(current_info / max(planned_info, 1), 1.0)

    # alpha spent so far at this info fraction
    alpha_spent = obrien_fleming_alpha_spent(info_frac, alpha=alpha)
    # two-sided critical z at this interim
    z_crit = norm.ppf(1 - alpha_spent / 2.0)

    z = two_sample_z(cum_success_ctrl, cum_n_ctrl, cum_success_trt, cum_n_trt)

    # Efficacy stopping (two-sided)
    if abs(z) >= z_crit:
        return 'stop-for-efficacy'

    # Futility: compute conditional power projecting current observed effect to planned final sample
    # approximate final standard error assuming planned sample sizes
    p_ctrl = cum_success_ctrl / cum_n_ctrl if cum_n_ctrl > 0 else 0.0
    p_trt = cum_success_trt / cum_n_trt if cum_n_trt > 0 else 0.0
    observed_diff = p_trt - p_ctrl

    # final SE under pooled variance estimated from current pooled p
    pooled_p = (cum_success_ctrl + cum_success_trt) / max(current_info, 1)
    se_final = math.sqrt(pooled_p * (1 - pooled_p) * (1.0 / planned_total_ctrl + 1.0 / planned_total_trt))
    if se_final == 0:
        return 'continue'

    # expected final z if observed_diff persists
    expected_final_z = observed_diff / se_final

    # conditional power approx: probability that final |Z| > z_final_crit given current estimate
    # For simplicity use final alpha ~ overall alpha at info_frac=1
    alpha_final = obrien_fleming_alpha_spent(1.0, alpha=alpha)
    z_final_crit = norm.ppf(1 - alpha_final / 2.0)
    # approximate as one-sided normal tail considering effect direction
    # use two-sided threshold converted to one-sided by symmetry
    cp = 1.0 - norm.cdf(z_final_crit - abs(expected_final_z))  # approx
    if cp < futility_cp_thresh:
        return 'stop-for-futility'

    return 'continue'

Key points:- Time complexity O(1) per interim. Space O(1).- Edge cases: very small sample sizes violate normal approx; pooled variance zero when proportions 0/1.- Alternatives: use exact sequential tests (e.g., Pocock boundaries, fully specified alpha/beta spending, Bayesian predictive probability) or simulate operating characteristics to calibrate futility threshold.

HardTechnical

58 practiced

In an A/B test some users assigned to treatment do not receive it (noncompliance). Explain intention-to-treat (ITT) vs per-protocol analyses and show how instrumental variables (using assignment as instrument) can estimate the Local Average Treatment Effect (LATE). Include key assumptions and formulas.

Sample Answer

Situation: In an A/B test with imperfect compliance, some users assigned to Treatment (Z=1) don’t receive it (D=0); some assigned to Control (Z=0) might receive treatment. We want causal effect of actual treatment (D) on outcome Y.

Intention-to-treat (ITT)- Definition: ITT = E[Y | Z=1] − E[Y | Z=0]. - Interpretation: causal effect of being assigned to treatment (policy effect). Preserves randomization, unbiased for assignment effect even with noncompliance.- Use when policy/assignment is the estimand or when compliance is endogenous.

Per-protocol (PP)- Definition: compares outcomes among those who actually received treatment vs those who didn’t (e.g., E[Y | D=1] − E[Y | D=0] or among compliers only). - Problem: selection bias — D is not randomized, so PP is generally biased unless compliance is random or you adjust for confounders correctly.

Instrumental variables and LATE- Use assignment Z as an instrument for actual treatment D to recover causal effect among compliers (those who take treatment if and only if assigned).- Wald (scalar) estimator / LATE formula: LATE = (E[Y | Z=1] − E[Y | Z=0]) / (E[D | Z=1] − E[D | Z=0]) Numerator = ITT on outcome; denominator = ITT on treatment (first-stage).- Equivalently estimate by Two-Stage Least Squares (2SLS): Stage 1: D_i = π0 + π1 Z_i + v_i Stage 2: Y_i = β0 + β_IV D̂_i + ε_i β_IV converges to LATE.

Key assumptions for validity1. Relevance: Cov(Z, D) ≠ 0 (first-stage nonzero): assignment affects probability of receiving treatment.2. Exclusion restriction: Z affects Y only through D (no direct effect of assignment on outcome).3. Independence (Random assignment): Z ⟂ (Y(1), Y(0), D(1), D(0)) — assignment randomized.4. Monotonicity (No defiers): D_i(1) ≥ D_i(0) for all i (assignment never reduces treatment take-up). This yields interpretable LATE among compliers.5. SUTVA: no interference and well-defined treatments.

Interpretation and caveats- LATE applies only to compliers — may not generalize to always-/never-takers.- If instrument weak (small first-stage), IV estimates are noisy and biased in finite samples — check F-statistic (>10 rule).- If exclusion fails (e.g., assignment changes behavior beyond treatment), IV biased.- Report ITT alongside LATE; provide first-stage, compliance rates, SEs (delta method or 2SLS robust SEs), and sensitivity analyses.

MediumTechnical

52 practiced

When is a difference-in-differences (DiD) approach preferable to an A/B test? Describe DiD assumptions (including parallel trends) and explain how you would validate those assumptions in a product experiment context where randomization is impossible.

Sample Answer

Difference-in-differences (DiD) is preferable to an A/B test when you cannot randomize treatment — e.g., a product change rolled out to one region, or policy applied to a subset of users — and you have pre- and post-treatment data for treated and comparable control units.

Core DiD assumptions and how to validate them in a product context:

1) Parallel trends: in absence of treatment, treated and control groups would have followed the same outcome trajectory.- Validate: plot pre-treatment time series (event-study / leads) to visually check parallelism; formally test with regressions including group×time interactions on pre-period data; run placebo DiD by pretending treatment happened earlier and checking for no effect.- If trends differ, adjust: include time-varying covariates, use group-specific linear trends, or apply synthetic control / propensity-score matching to build a better control.

2) No concurrent shocks differentially affecting groups:- Validate: audit product logs and release calendar for other changes; check other outcomes unaffected by treatment; run falsification tests on metrics that shouldn’t move.

3) Stable composition / no differential attrition:- Validate: compare covariate distributions and retention rates pre/post; reweight observations or use inverse-propensity weighting if composition shifts.

4) SUTVA/no spillovers:- Validate: examine geographic or social network links for contamination; if spillovers exist, model partial interference or use cluster-level DiD.

Practical checks and robustness:- Event-study coefficients (lead and lag plot) to show pre-trend flatness.- Covariate balance and matching, and DiD with covariate controls.- Placebo outcomes and permutation tests (randomly assign treatment times/groups) to get empirical null distribution.- Sensitivity analyses: alternative windows, functional forms, clustered SEs.

Example: A feature launched only in Country A. I’d compare engagement trends in Country A vs similar countries for several months before and after; run event-study, placebo rollout, add country-specific trends, and if needed synthesize a control from a weighted combination of other countries. If assumptions hold and robustness checks pass, DiD provides credible causal estimates when A/B testing is infeasible.

MediumTechnical

52 practiced

List the instrumentation and data-quality checks (unit tests, integration tests, SQL assertions, real-time monitoring) you would implement before trusting A/B test results. For each check describe why it matters and what alert or remediation you would configure if it fails.

Sample Answer

Here’s a comprehensive checklist I’d implement before trusting A/B results, with why each matters and the alert/remediation I’d configure.

1) Unit tests for data pipelines- Why: Validate transformations, edge-case handling, schema expectations (types, non-nullable fields).- Alert/Remediation: CI failure blocks deployment; notify engineers with failing test logs and rollback to last green version.

2) Integration tests / end-to-end smoke tests- Why: Ensure event ingestion → processing → analytics tables flow works (no dropped events, time skew).- Alert/Remediation: Run nightly; alert on mismatch counts (>1% drift) to on-call data engineer; auto-run re-ingest for transient failures or pause experiments if persistent.

3) SQL assertions on key metrics- Why: Sanity-checks on experiment tables: total user count, assignment balance, impressions > clicks, no negative metrics.- Alert/Remediation: Scheduled assertions (hourly); send pager if violated. Remediation: freeze experiment traffic, kick off data replay and root-cause investigation.

4) Randomization & assignment integrity checks- Why: Verify treatment assignment is stable, random, and independent of key covariates (no covariate imbalance).- Alert/Remediation: Real-time imbalance alert (e.g., standardized mean differences > 0.1); roll back new allocation and investigate hashing/segmentation bug.

5) Exposure / event completeness monitoring- Why: Ensure exposure events correspond to downstream metric events (exposure→conversion linkage).- Alert/Remediation: Alert on exposure-to-event ratio drop; trigger synthetic traffic test to validate pipeline and fall back to logging mode.

6) Duplicate & timestamp checks- Why: Prevent double-counting and ensure correct attribution windows.- Alert/Remediation: Alert on duplicate rate or large timestamp skew; deduplication job and pause analysis until corrected.

7) Drift & data-distribution monitoring- Why: Detect upstream product changes or instrumentation regressions that shift distributions.- Alert/Remediation: Drift alert (KL divergence threshold); flag analyses as “at risk,” notify product/eng, and consider covariate adjustment or hold conclusions.

8) Statistical test guardrails & power checks- Why: Prevent underpowered or peeking-driven false positives.- Alert/Remediation: Auto-check sample size/power before publishing; if underpowered, mark as inconclusive and recommend running longer.

9) Canary & synthetic-user tests- Why: Validate real-time pipeline and analytics correctness end-to-end.- Alert/Remediation: If canary fails, rollback recent deployments and pause experiment traffic.

For each alert I’d include context (affected experiment ID, timestamps, delta metrics), automated data snapshot for debugging, and runbook with immediate actions and escalation path. This combination prevents trusting flawed signals and enables fast remediation.

MediumTechnical

94 practiced

For a developer-facing SDK change that reduces API latency but could raise error rates, propose a structured set of primary, secondary, and guardrail metrics. Describe how to instrument these metrics in CI, staging, and production and how to detect regressions early.

Sample Answer

Framework: define objectives (reduce latency) and risks (higher error rates), then assign metrics into Primary / Secondary / Guardrail, instrument at all lifecycle stages, and use statistical detection with automated gating.

Primary metrics (business impact)- P95 API latency (ms) — target reduction- Success rate (%) — overall request success

Secondary metrics (diagnostics)- P50/P90 latency, mean latency- Time-to-first-byte (TTFB)- Retry rate per client- Error breakdown by type (400/500/timeout)- Throughput (requests/sec)

Guardrail metrics (safety)- Client-side exception rate (SDK throws)- API schema/contract violations- CPU/memory per SDK instance- Latency tail for specific customers (top 5%)

Instrumentation across environments- CI: run synthetic unit + integration tests with deterministic mocks; capture latency and error counts; enforce unit-test coverage; fail build on functional regressions.- Staging: deploy canary SDK to sampled production-like traffic or replay recorded traces; run load tests and chaos tests; collect full metric set; require statistical gates.- Production: phased rollout (e.g., 5% → 25% → 100%) with canary monitoring, feature flags, and A/B assignment; aggregate metrics in time-series DB and trace with distributed tracing (OpenTelemetry).

Regression detection & gating- Use statistical tests: for large samples, two-sample t-test or Mann-Whitney for latency; proportion z-test for success rate. Define minimum detectable effect and power.- Apply automated SLO gates: e.g., block promotion if P95 not improved by X% or if success rate drops >0.1% absolute or relative error rate increases by >25% with p<0.01.- Alerting: anomaly detection (rolling baselines, seasonality-aware) + burn-in period checks. On canary failure, auto-rollback and open ticket with diagnostics (traces, top error stacks, affected customers).- Post-deploy: run retrospective A/B analysis, monitor for delayed regressions, and log client-side telemetry to catch SDK-specific failures.

This approach balances latency gains against reliability using measurable gates, staged rollout, and statistical rigor.

Unlock Full Question Bank

Get access to hundreds of A and B Test Design interview questions and detailed answers.

Join thousands of developers preparing for their dream job.