A and B Test Design Questions

Designing and running A and B tests and split tests to evaluate product and feature changes. Candidates should be able to form clear null and alternative hypotheses, select appropriate primary metrics and guardrail metrics that reflect both product goals and user safety, choose randomization and assignment strategies, and calculate sample size and test duration using power analysis and minimum detectable effect reasoning. They should understand applied statistical analysis concepts including p values confidence intervals one tailed and two tailed tests sequential monitoring and stopping rules and corrections for multiple comparisons. Practical abilities include diagnosing inconclusive or noisy experiments detecting and mitigating common biases such as peeking selection bias novelty effects seasonality instrumentation errors and network interference and deciding when experiments are appropriate versus alternative evaluation methods. Senior candidates should reason about trade offs between speed and statistical rigor plan safe rollouts and ramping define rollback plans and communicate uncertainty and business implications to technical and non technical stakeholders. For developer facing products candidates should also consider constraints such as small populations cross team effects ethical concerns and special instrumentation needs.

MediumTechnical

83 practiced

During an experiment you discover conversion events dropped by ~30% in one region due to a JavaScript release. As the PM, outline immediate remediation actions (technical and communications), how you'd decide whether to abort the experiment vs salvage data, and steps to rebuild confidence in experiment instrumentation going forward.

Sample Answer

Situation: A new JS release coincided with ~30% drop in conversion events in one region during an experiment.

Immediate actions (technical):- Halt further deployments and flag the release as suspect.- Roll back the JS change to the previous stable version in that region (or disable feature flag) to stop additional data loss.- Triage with engineering: capture logs, error rates, network failures, and client telemetry to confirm causal link and scope (browsers, devices, AB groups).- Run a quick-side validation: replay synthetic conversions and check instrumentation endpoints (beacon delivery, batching, retries).- If rollback not possible immediately, deploy a hotfix that restores event firing or retries.

Immediate actions (communications):- Notify stakeholders (growth, analytics, engineering, legal, comms) with a concise incident brief: impact, region, ETA for next update.- Post a customer-facing notice only if user-facing functionality was degraded; otherwise internal updates every 30–60 mins until resolved.- Document findings in a shared incident channel and record decisions.

Decide abort vs salvage:- Abort experiment if: data in affected cohort is biased (behavior changed due to instrumentation failure), key metrics are underreported, or recovery gaps make treatment/control incomparable.- Salvage if: failure is limited to event delivery but relative differences between variants remain unbiased and you can reliably correct counts (e.g., using server-side logs, backup telemetry) with documented adjustments.- Use criteria: percent of affected traffic, ability to correct/estimate missing events with confidence, duration of outage relative to experiment length, and statistical power remaining.

Rebuild confidence in instrumentation:- Postmortem: root cause, timeline, and actionable fixes.- Add automated end-to-end synthetic tests that run per region and browser (conversion flows + instrumentation verification).- Implement monitoring/alerts for drops in event volumes, delivery latencies, and schema validation.- Introduce canary rollout for JS changes with targeted metrics guardrails and automatic pause on anomalies.- Create playbook for experiment data contamination and a checklist for approving future releases.- Share results and improvements with stakeholders to restore trust.

This approach balances immediate containment, data integrity decisions grounded in measurable criteria, and systemic fixes to prevent recurrence.

MediumTechnical

42 practiced

An engineer recommends blocking all experiments until instrumentation is perfect, while the growth team pushes to run experiments now to capture momentum. As the PM, how would you mediate this conflict, set acceptable risk thresholds, prioritize instrumentation fixes versus experiment velocity, and communicate a decision to both teams?

Sample Answer

Situation: At my previous company, growth wanted to launch a set of aggressive A/B tests to capitalize on seasonal demand while engineering pushed to halt experiments until analytics and event tracking were “perfect.” The conflict threatened both time-to-market and data integrity.

Task: As PM I needed to mediate, set an acceptable risk threshold, prioritize work so we could move quickly but safely, and clearly communicate the decision to both teams and stakeholders.

Action:- Clarify goals and constraints: I ran a 30‑minute meeting to align on the experiment objectives (primary metric, minimum detectable effect), deadlines (seasonal window), and what “perfect instrumentation” meant (missing events, latency, identity stitching).- Quantify acceptable risk: Together we defined concrete thresholds — e.g., ≤5% expected measurement bias, event loss <1% for primary metrics, and ability to detect a 3% lift with 80% power. If thresholds couldn’t be met, experiments would be limited to lower-risk hypotheses.- Triage fixes vs velocity: I worked with Eng to classify instrumentation work into (A) blocking for primary metric integrity, (B) recommended improvements that can be mitigated, (C) non-blocking future work. We agreed to fix all (A) items immediately, defer (C), and mitigate (B) via workarounds (sampling, redundant events).- Risk-reducing tactics: We used feature flags, smaller canary cohorts, shortened experiment horizons, and synthetic or backfilled data validation to validate measurement before scaling. We also created a lightweight measurement plan template that must be approved before any experiment runs.- Prioritization and roadmap: I reallocated two engineer-days to urgent instrumentation (A) work and negotiated a parallel track so growth could run low-risk experiments in small cohorts while fixes progressed.- Communication: I sent a summary to both teams and leadership: the decision, the agreed risk thresholds, the triage list with owners and timelines, and the fallback (pause or scale-back) triggers. I scheduled a daily sync until primary fixes were closed and a postmortem after the first experiment.

Result: We launched prioritized experiments in controlled cohorts within the seasonal window, achieved measurable results with acceptable confidence, and completed instrumentation fixes within two sprints. Engineers felt their quality concerns were respected; growth preserved momentum. The measurement checklist reduced future disputes.

This approach balances business urgency with data integrity by making risks explicit, quantifying acceptable thresholds, and using mitigations (canaries, flags, sampling) to keep both velocity and quality.

EasyTechnical

50 practiced

In simple language, explain what a p-value represents in the context of an A/B test. Provide one common misinterpretation you often hear from stakeholders, and write one concise sentence you would use to correct that misinterpretation for a non-technical executive.

MediumTechnical

53 practiced

Explain sequential monitoring and 'peeking' in the context of A/B testing. Outline at least two statistical approaches that allow interim looks (for example, alpha-spending boundaries like Pocock or O'Brien-Fleming, and Bayesian monitoring). Describe pros and cons of each and how you would operationalize a safe stopping rule in a company dashboard.

Sample Answer

Sequential monitoring refers to looking at A/B test results before the planned end — "peeking." Uncontrolled peeking inflates Type I error (false positives) because repeated looks increase the chance of seeing a spurious significant result. To allow interim looks safely, teams use statistical procedures that control the overall error rate or use Bayesian decision rules.

Frequentist alpha-spending (e.g., Pocock, O’Brien–Fleming)- Idea: Pre-specify number/times of interim looks and an alpha spending function that allocates the total Type I error across looks.- Pocock: gives roughly equal critical values across looks (easier early stopping but higher early false positive risk).- O’Brien–Fleming: very conservative early (requires extreme effect to stop early) and near-normal at the final look.- Pros: Controls family-wise Type I error; familiar to statisticians; easy to report p-values.- Cons: Requires pre-specification of look times or information fractions; slightly less flexible; implementation must account for correlated looks and sample size information.

Bayesian monitoring- Idea: Compute posterior probability that variant is better; stop when posterior crosses a decision threshold (e.g., P(effect>0)>99% or P(loss>threshold)<1%).- Pros: Intuitive decision probabilities; natural for continuous monitoring; easy to incorporate prior knowledge and business loss functions.- Cons: Results depend on priors and threshold choices; less familiar to some stakeholders; calibration needed to map to frequentist error rates if required.

Operationalizing a safe stopping rule in a company dashboard1. Policy first: Define experiment charter template requiring pre-specified primary metric, minimum detectable effect, max duration, and allowed interim look policy (alpha-spending function or Bayesian thresholds).2. Automation: - Enforce locking: only allow interim results through a controlled dashboard API that applies the chosen statistical procedure to compute stopping criteria. - Log every look (timestamp, sample sizes, decision statistics) to an audit table.3. Decision logic: - For alpha-spending: compute current information fraction and corresponding critical value; flag “stop for success/futility” only when crossing boundary. - For Bayesian: compute posterior and compare to pre-agreed thresholds and expected loss.4. Governance: - Require approval workflow for early stops (data scientist + PM sign-off) and require re-testing or holdout verification for large business-impact decisions. - Display clear UI signals: “Continue,” “Stop for Success,” “Stop for Futility,” with explanation, confidence, and sample sizes.5. Monitoring and education: - Track operating characteristics (simulations) of chosen policy under realistic traffic to communicate false-positive and false-negative rates to stakeholders.

Summary: Choose the method that balances organizational tolerance for risk and need for flexibility (O’Brien–Fleming-like if you want conservative early stopping; Bayesian for continuous, decision-focused monitoring). Encode rules in the dashboard, log every look, and require clear sign-off to avoid ad hoc peeking.

HardTechnical

42 practiced

Design a decision framework that a PM can use when multiple primary and guardrail metrics exist and sometimes conflict. Include options like hierarchical gating (primary metric first then guardrails), weighted scoring, Pareto-front decisions, and thresholding on practical significance. Describe how you would choose weights or thresholds and how to communicate uncertainty and trade-offs to stakeholders.

Sample Answer

Start by clarifying objectives and metric types- Primary metrics: direct business/user outcomes (e.g., revenue, DAU, retention).- Guardrails: safety/quality constraints (latency, error rate, compliance).- Clarify scale, direction (higher-is-better or lower-is-better), and unit (absolute vs relative change).

Decision framework (process)1. Define decision scope & success criteria: which primary metrics must improve and which guardrails are non-negotiable.2. Apply hierarchical gating first for hard constraints: - If a guardrail breach risks safety, compliance, or legal exposure, block rollout immediately. - Otherwise, allow trade-offs among soft guardrails.3. Use weighted scoring for trade-offs among remaining metrics: - Normalize metrics to comparable scales (z-scores or % change vs baseline). - Choose weights based on business impact, short-term vs long-term priorities, and stakeholder input (e.g., revenue 0.4, retention 0.3, NPS 0.2, latency 0.1). - Derive weights via: executive alignment workshop, historical sensitivity analysis (elasticities), and A/B meta-analysis.4. Complement with Pareto-front analysis: - Plot metric pairs or multi-dimensional projections to identify non-dominated options; prefer Pareto-efficient choices.5. Threshold on practical significance: - Set minimum effect sizes (e.g., 1–2% lift, or minimum absolute change) informed by cost-benefit and sample-size power calculations. - Use confidence intervals and Minimum Detectable Effect (MDE) to avoid acting on noise.

Choosing weights & thresholds- Combine qualitative prioritization (strategy decks) with quantitative evidence: - Backtest on historical experiments to estimate downstream impact per unit change. - Use stakeholder-driven utility functions when data is sparse. - Revisit weights quarterly as strategy changes.

Communicating uncertainty & trade-offs- Present: point estimates, 95% CI, and probability of meeting thresholds (Bayesian posterior or frequentist p-values + power).- Visuals: scorecards, Pareto plots, and “what-if” sensitivity charts showing how decisions change under alternative weights.- Recommendation format: preferred action, secondary options, required mitigations, and clear callouts for hard guardrails.- State assumptions, data quality limits, and how further experimentation or ramping will reduce uncertainty (staged rollout with monitoring and kill-switch).

Operationalize- Encode rules into launch playbooks: gating rules, weight table, MDE thresholds, and monitoring dashboards with alerts.- Regularly review after post-launch data and update weights/thresholds based on learning.

This framework balances safety (gating), nuance (weighted scoring & Pareto), and statistical rigor (practical significance + uncertainty) and makes trade-offs transparent to stakeholders.

Unlock Full Question Bank

Get access to hundreds of A and B Test Design interview questions and detailed answers.

Join thousands of developers preparing for their dream job.