A/B Test Design & Statistical Rigor Questions

Designing and statistically defending a controlled online experiment: framing a testable hypothesis, defining control and treatment variants, choosing the randomization unit, setting the primary success metric, and computing sample size, power, and minimum detectable effect. Covers the statistical foundations that make a readout trustworthy, including hypothesis testing, p-values, confidence intervals, statistical vs practical significance, and Type I/II error. Emphasizes avoiding the common pitfalls that invalidate a test, such as peeking, multiple-comparison inflation, underpowered designs, and how test duration and stopping rules affect the validity of conclusions.

HardSystem Design

38 practiced

Design an experimentation platform that supports 100M MAU and thousands of concurrent experiments. Requirements: deterministic cross-platform randomization, low-latency treatment evaluation, experiment configuration service, consistent exposure logging, experiment conflict detection (prevent overlapping changes), offline metrics computation pipeline, and rollback automation. Describe architecture components, data flow, storage choices for exposure and metrics, and operational considerations for scale and reliability.

Sample Answer

Requirements clarification:- Functional: deterministic cross-platform randomization (same bucket across web/mobile/backend), low-latency eval, config service, consistent exposure logging, conflict detection, offline metrics pipeline, rollback automation.- Non-functional: 100M MAU, thousands of concurrent experiments, <10ms eval P95, 99.99% availability for eval.

High-level architecture:- Experiment Config Service (central source of truth): stores experiment definitions, targeting rules, variants, start/end, preconditions, priority/ownership metadata. Exposes versioned configs via API and push to CDN/KV.- Deterministic Randomization Library: client SDKs + server-side lib implementing hashed bucketing (HMAC with experiment-id + user-id + salt) to ensure consistent bucketing across platforms and tie-break by priority. Local evaluation for low latency.- Config Distribution: configs published to a global CDN + regional in-memory KV (e.g., Redis/managed cache) for fast reads; SDKs poll/subscribe to updates; configs are versioned & signed.- Exposure Logging: Clients send exposure events to an ingestion gateway (edge collectors) which add config-version, bucket, timestamp, and signature. Edge buffers to message queue (Kafka) for exactly-once semantics (idempotence keys).- Conflict Detection & Guardrails: Pre-deployment static analyzer in Config Service checks overlapping changes to the same metric/feature flag scope; runtime guard uses priority and circuit-breaker rules to block overlapping impactful changes.- Offline Metrics Pipeline: Kafka → stream processors (Flink/Beam) for sessionization, enrichment (user cohorts, bucketing), and materialize exposures and events into OLAP store (BigQuery/ClickHouse) for daily aggregations and metric computation. Support attribution windows and statistical tests.- Rollback Automation: Monitoring rules on metrics and SLOs trigger automatic rollback via Config Service API (revert to previous config version); can be staged (percent rollbacks). Human-in-the-loop alerts with one-click rollback.Storage choices:- Configs: strongly-consistent datastore (Cloud Spanner / Postgres) for writes + CDN/cache for reads.- Exposure logs: append-only Kafka + cold storage (S3) + OLAP (ClickHouse/BigQuery) for fast queries.- Derived metrics: time-partitioned OLAP tables, pre-aggregations for common metrics.Operational considerations:- Scale: partition Kafka by user-id, autoscale collectors, regional caches. Client-side evaluation reduces load on central services.- Reliability: multi-region replication, signed configs, feature flags to disable SDKs, chaos testing.- Accuracy: enforce idempotent exposure keys, deterministic hashing, consistent timezone/windowing in pipeline.- Security & Governance: RBAC in Config Service, audit logs, and experiment lifecycle workflows (approve/QA/launch).This design balances low-latency local evaluation, deterministic cross-platform bucketing, reliable logging for offline metrics, and automation for safe rollouts and fast rollback.

MediumTechnical

52 practiced

Explain sequential monitoring and 'peeking' in the context of A/B testing. Outline at least two statistical approaches that allow interim looks (for example, alpha-spending boundaries like Pocock or O'Brien-Fleming, and Bayesian monitoring). Describe pros and cons of each and how you would operationalize a safe stopping rule in a company dashboard.

Sample Answer

Sequential monitoring refers to looking at A/B test results before the planned end — "peeking." Uncontrolled peeking inflates Type I error (false positives) because repeated looks increase the chance of seeing a spurious significant result. To allow interim looks safely, teams use statistical procedures that control the overall error rate or use Bayesian decision rules.

Frequentist alpha-spending (e.g., Pocock, O’Brien–Fleming)- Idea: Pre-specify number/times of interim looks and an alpha spending function that allocates the total Type I error across looks.- Pocock: gives roughly equal critical values across looks (easier early stopping but higher early false positive risk).- O’Brien–Fleming: very conservative early (requires extreme effect to stop early) and near-normal at the final look.- Pros: Controls family-wise Type I error; familiar to statisticians; easy to report p-values.- Cons: Requires pre-specification of look times or information fractions; slightly less flexible; implementation must account for correlated looks and sample size information.

Bayesian monitoring- Idea: Compute posterior probability that variant is better; stop when posterior crosses a decision threshold (e.g., P(effect>0)>99% or P(loss>threshold)<1%).- Pros: Intuitive decision probabilities; natural for continuous monitoring; easy to incorporate prior knowledge and business loss functions.- Cons: Results depend on priors and threshold choices; less familiar to some stakeholders; calibration needed to map to frequentist error rates if required.

Operationalizing a safe stopping rule in a company dashboard1. Policy first: Define experiment charter template requiring pre-specified primary metric, minimum detectable effect, max duration, and allowed interim look policy (alpha-spending function or Bayesian thresholds).2. Automation: - Enforce locking: only allow interim results through a controlled dashboard API that applies the chosen statistical procedure to compute stopping criteria. - Log every look (timestamp, sample sizes, decision statistics) to an audit table.3. Decision logic: - For alpha-spending: compute current information fraction and corresponding critical value; flag “stop for success/futility” only when crossing boundary. - For Bayesian: compute posterior and compare to pre-agreed thresholds and expected loss.4. Governance: - Require approval workflow for early stops (data scientist + PM sign-off) and require re-testing or holdout verification for large business-impact decisions. - Display clear UI signals: “Continue,” “Stop for Success,” “Stop for Futility,” with explanation, confidence, and sample sizes.5. Monitoring and education: - Track operating characteristics (simulations) of chosen policy under realistic traffic to communicate false-positive and false-negative rates to stakeholders.

Summary: Choose the method that balances organizational tolerance for risk and need for flexibility (O’Brien–Fleming-like if you want conservative early stopping; Bayesian for continuous, decision-focused monitoring). Encode rules in the dashboard, log every look, and require clear sign-off to avoid ad hoc peeking.

EasyTechnical

45 practiced

Describe the conceptual steps you would take to estimate required sample size for an A/B test. Explain the roles of baseline rate, minimum detectable effect (MDE — absolute vs relative), alpha (type I error), and power, and list two rules-of-thumb you could use when a calculator is not available.

MediumTechnical

38 practiced

You are testing 5 treatment variants and monitoring 8 metrics. Explain the multiple comparisons problem and discuss three correction strategies (Bonferroni, Benjamini-Hochberg FDR, hierarchical testing). For a PM running product experiments, recommend a pragmatic approach and justify the trade-offs between controlling false positives and preserving power.

EasyTechnical

49 practiced

Describe criteria you would use to decide whether to run an A/B test versus doing an observational study, user interviews, or a pilot rollout. Provide two concrete product scenarios where an A/B test would be inappropriate, and recommend the alternative evaluation method for each scenario with a brief justification.

Unlock Full Question Bank

Get access to hundreds of A/B Test Design & Statistical Rigor interview questions and detailed answers.

Join thousands of developers preparing for their dream job.