A/B Test Design & Statistical Rigor Questions

Designing and statistically defending a controlled online experiment: framing a testable hypothesis, defining control and treatment variants, choosing the randomization unit, setting the primary success metric, and computing sample size, power, and minimum detectable effect. Covers the statistical foundations that make a readout trustworthy, including hypothesis testing, p-values, confidence intervals, statistical vs practical significance, and Type I/II error. Emphasizes avoiding the common pitfalls that invalidate a test, such as peeking, multiple-comparison inflation, underpowered designs, and how test duration and stopping rules affect the validity of conclusions.

MediumTechnical

49 practiced

You're adding human moderation labels into an experiment to assess model safety. Design the instrumentation to measure labeling latency, label quality (inter-annotator agreement), and its impact on primary metrics. How would label latency distort online metric interpretation and how to correct for it?

Sample Answer

Instrumentation design- Data to log per item: unique id, content metadata (type, language, user cohort), model score/prediction, experiment arm, human label, labeler id, label timestamp (assigned, started, submitted), label confidence, adjudication flag, gold-check flag, and any moderation action taken with action timestamp.- Store events in append-only streams (e.g., Kafka) and a joinable analytics table for time-series queries.

Measuring labeling latency- Compute latency = submitted_time - assigned_time. Report distribution (p50/p90/p99), and split by labeler, content type, experiment arm.- Bucket latency (0-1m,1-5m,5-30m,30m+) and track throughput and backlog.- Use survival analysis (Kaplan–Meier) to model time-to-label and estimate censoring for items still unlabeled.

Measuring label quality / inter-annotator agreement- Use redundant labeling: sample X% of items labeled by 3+ annotators.- Compute Fleiss’ kappa for multi-rater and Cohen’s kappa for pairwise; also report percent agreement and label distribution confusion matrices.- Maintain periodic gold-sets and compute accuracy, precision/recall per labeler; use Krippendorff’s alpha if labels are ordinal/multiple categories.- Track drift: agreement over time and by content segment.

Measuring impact on primary metrics- Capture primary online metrics (CTR, retention, policy-violation rate) with timestamps and link to moderation events.- Instrument causal chain: content served -> user action -> post-hoc label -> moderation action (if any) with action_time.

How label latency distorts online metric interpretation- Latency causes time-dependent selection and censoring: unsafe content may be served and measured before labels/adjudication arrive, inflating or deflating observed violation rates in a time window. If one arm has longer labeling latency, its violations appear lower in short-term metrics (survivor bias).- Example: Arm A faster labeling flags and removes bad content within 5 minutes; Arm B flags in 60 minutes. A 10-minute online metric snapshot will undercount Arm B violations, biasing treatment effect.

Corrections and mitigations- Lag-aware metrics: compute metrics both unadjusted and lag-adjusted. Define an observation window longer than max expected latency (or report metric as function of time-since-serve).- Backfill: retroactively update historical metrics when labels arrive; store metric as function(time, label_arrival_time).- Censoring adjustment: apply inverse-probability-of-censoring weighting using survival model for label arrival times to estimate true violation rates at evaluation cutoff.- Counterfactual imputation: train a labeler-proxy model to predict labels for unlabeled items with calibrated probabilities; use these to estimate expected counts and uncertainty until human labels arrive.- Stratified analysis: report treatment effects conditioned on latency buckets; control for latency in regression (e.g., logistic regression with latency as covariate) to isolate model effect.- A/B fairness: ensure randomization before labeling assignment; monitor and enforce balanced labeling capacity across arms to avoid systematic latency differences.

Operational best practices- Prioritize critical items (risk scoring) to reduce bias on high-impact events.- Monitor instrumentation health: missing timestamps, partial joins.- Automate backfill and provide dashboards showing both real-time and lag-adjusted metrics with confidence intervals.

This approach quantifies latency and quality, reveals distortion mechanisms, and provides principled statistical corrections so online metrics reflect true model safety.

MediumTechnical

75 practiced

You must evaluate a new developer-facing API feature used by ~200 monthly active power users. Standard A/B testing lacks power. Describe alternative evaluation strategies (e.g., paired within-user experiments, qualitative feedback, synthetic load tests, Bayesian priors, sequential testing) and explain trade-offs.

Sample Answer

With ~200 MAU power users, classical A/B tests often lack statistical power. Here are practical alternatives, why they help, and trade-offs — focused on developer-facing API evaluation and measurable adoption/quality signals.

1) Paired within-user experiments- Approach: each user tries both versions (A then B or vice‑versa), measure per-user differences (latency, error rate, task completion).- Why: reduces between-user variance, increases power with small N.- Trade-offs: order effects (use crossover/randomize order), learning effects for novel APIs.

2) Sequential (adaptive) testing- Approach: run a test that evaluates continuously and stops when evidence reaches threshold (e.g., alpha-spending).- Why: efficient use of data; can stop early.- Trade-offs: must control false positives, predefine stopping rules; operational complexity.

3) Bayesian analysis with informative priors- Approach: incorporate prior knowledge (previous similar features, expert beliefs) into posterior estimates.- Why: yields probabilistic statements (e.g., 90% chance improvement >X) and works with small samples.- Trade-offs: sensitivity to priors — require transparency and sensitivity analysis.

4) Synthetic load and fuzz testing- Approach: automated scripts, simulated clients, edge-case inputs to stress correctness and performance.- Why: catches reliability and scale issues without user volume.- Trade-offs: may not reflect real developer usage patterns or discover usability friction.

5) Qualitative research & mixed methods- Approach: targeted interviews, UX sessions, code walkthroughs, shadowing, collect annotated logs/snippets.- Why: reveals pain points, misunderstandings, and feature discoverability — high signal per user.- Trade-offs: not statistically generalizable; requires careful synthesis.

6) Instrumented telemetry + user-level A/B with enrichment- Approach: collect rich metrics (latency, retries, feature flags, retention) and combine with qualitative tags.- Why: multi-metric view increases sensitivity; can build composite endpoints.- Trade-offs: complexity in metric definitions and multiple hypothesis corrections.

Recommended practical plan:- Start with synthetic tests to validate correctness and performance.- Run paired within-user crossover with balanced order for key quantitative metrics.- Use Bayesian analysis to report probabilistic effects and run sensitivity checks on priors.- Complement with 10–20 in-depth interviews and code session recordings to interpret behavior.- Pre-register analysis plan and stopping rules to avoid bias.

This combination balances statistical rigor, operational feasibility, and developer-experience insights for small-sample, developer-facing features.

MediumTechnical

44 practiced

Compute the Minimum Detectable Effect (MDE) for a binary metric given sample size per group = 50,000, baseline rate = 0.02, alpha = 0.05, power = 0.8. Show the formula or steps and interpret what that MDE means for product decision-making.

HardTechnical

39 practiced

You must audit a triaged set of past experiments for p-hacking and data-snooping. Describe a practical checklist and statistical tests you would run to detect suspicious patterns (e.g., multiple peeks, selective metric reporting, unrealistic power). How would you report findings?

Sample Answer

Start with scope & metadata collection- Gather experiment registry: hypothesis, primary/secondary metrics, sample size target, start/end dates, randomization method, stopping rules, pre-registered analysis, code/scripts, raw and aggregated logs, and who ran peeks/analyses.

Practical checklist (audit steps)1. Pre-registration check: is there a timestamped protocol? If not, flag.2. Timeline consistency: compare preregistered start/end to data timestamps.3. Stopping rule adherence: look for interim analyses or sample-size changes.4. Multiple testing & metric selection: verify all tracked metrics are reported; check for post-hoc promotion of secondary metrics to primary.5. Peeking / sequential looks: scan for repeated significance checks over time.6. Randomization integrity: test for covariate balance over time and across segments.7. Data handling: check for exclusion criteria, imputation, filtering applied post-hoc.8. Power plausibility: compare claimed power / effect-size to observed variance and sample sizes.9. Code reproducibility: re-run analysis scripts to confirm reported p-values and CIs.

Statistical tests & heuristics- P-curve and z-curve analysis: detect right-skew or excess just-below-0.05 mass.- Uniformity test on p-values for null effects (Kolmogorov–Smirnov).- Sequential analysis indicators: runs test on p-values over time; cumulative Z-statistics plotted vs sample size to spot stopping at significance.- Excess of marginally significant results: histogram of p-values in [0.04,0.05] vs expectation (chi-square).- Covariate balance tests: standardized mean differences and time-series drift tests.- Bunching tests on reported effect sizes (heaping) and impossible precision.- Recompute power: post-hoc observed power is misleading; instead compute minimum detectable effect (MDE) and compare to reported claims.

Practical reproducibility steps- Reproduce key results from raw data and scripts.- Run robustness checks: alternate preprocessing, multiple comparison corrections (Bonferroni, Benjamini-Hochberg), and mixed-effects models if clustered data.- Simulate expected p-value distribution under reported process (including peeking) to compare.

Reporting findings- Executive summary (1 page): scope, main red flags (high/medium/low), recommended actions.- Technical appendix: full checklist, raw evidence (plots: p-curve, cumulative Z, p-value histograms), reproduced analyses, code, and recommended corrective procedures.- Remediation plan: retract/annotate suspect results, rerun analyses with pre-specified corrections (adjust for peeks, multiple comparisons), and require preregistration + automated logging for future experiments.- If misconduct suspected, recommend escalation and preserve chain-of-custody for data and scripts.

Key principle: prioritize reproducibility — provide reproducible code + visual diagnostics so non-technical stakeholders can see where practices departed from acceptable standards.

HardTechnical

39 practiced

Design a governance framework to reduce p-hacking and false positives across teams. Include practices such as experiment pre-registration, blinded analysis, experiment registry, code reviews, automated checks, and post-experiment audits. How would you balance these controls with team velocity?

Sample Answer

Situation: As an AI engineer responsible for experiments across multiple product teams, I faced rising false-positive rates and noisy decisions because teams ran ad-hoc tests without consistent standards.

Framework (high-level goals):- Increase reproducibility and statistical rigor- Prevent p-hacking and bias- Preserve velocity with lightweight, automated controls

Core practices and how they fit together:1. Experiment pre-registration: mandatory brief template (hypothesis, primary/secondary metrics, sample size calc, stopping rule, data exclusions) stored in a centralized Experiment Registry (accessible UI + API). Enforce preregistration before traffic allocation.2. Blinded analysis: analysts get anonymized/held-out labels or shuffled assignment during model tuning and metric selection. Only unblind after agreed analysis plan executed.3. Experiment Registry: single source of truth with metadata, owners, status, links to code, dataset snapshots, and preregistration. Integrates with CI and deployment pipelines.4. Code reviews & reproducible artifacts: all experiment code (data pulls, metric calculations, visualization notebooks) in PRs with checklist: fixed seeds, data snapshot path, idempotency. Reviewers include a statistician or senior engineer rotation.5. Automated checks: CI gates that verify preregistration exists, validate sample-size calculations (basic power checks), detect multiple-hypothesis testing without correction, ensure logging of randomization keys, and run unit tests for metric computations.6. Post-experiment audits: periodic random audits and targeted audits for surprising large effects. Audit playbook checks code, registry, and raw data to confirm adherence.7. Governance board & lightweight escalation: small cross-functional review committee for disputed or high-impact experiments; provide quick advisory reviews.

Balancing with velocity:- Make controls proportionate: lightweight preregistration for low-risk A/B tests; full preregistration + blinded analysis for high-impact or external-facing results.- Automate everything to minimize manual work (templates, CI checks, registry integrations).- Provide developer-friendly tools: one-click preregistration from Jupyter, auto-fill sample size calculators, experiment templates.- Offer fast-track: if experiment passes automated checks and is low-risk, allow immediate launch; else require short advisory review (24–48h SLA).- Incentivize compliance: show teams dashboards of reproducibility score, tie recognition to good practices rather than punitive measures.

Metrics and rollout:- Track false-positive rate, preregistration compliance, audit failure rate, and median time-to-launch.- Pilot with two teams for 6 weeks, iterate templates and automation, then scale.

Trade-offs:- Some friction vs. faster but noisy decisions; mitigate by automation and proportional controls.- Blinding may slow iteration for model tuning—use staged workflows: internal tuning on synthetic/holdout, then blinded confirmatory runs.

This framework enforces rigor where it matters, uses automation to keep overhead low, and creates cultural incentives for reproducible, trustworthy experimentation.

Unlock Full Question Bank

Get access to hundreds of A/B Test Design & Statistical Rigor interview questions and detailed answers.

Join thousands of developers preparing for their dream job.