A/B Test Results Analysis Questions

Learn to analyze results from A/B tests and experiments. Understand key statistics: sample size, statistical significance, confidence intervals, and p-values at a practical level (not deep theory). Practice interpreting test results: 'Is this difference real or just noise?' Learn common mistakes: stopping tests early, p-hacking, and running too many tests simultaneously.

EasyTechnical

84 practiced

Explain the difference between statistical significance and practical (business) significance in experiments. Provide a concrete example where a result is statistically significant but not worth shipping, and describe what additional analyses or business questions a PM should consider before deciding.

EasyTechnical

97 practiced

What is an A/A test and why would a product team run one? Describe how you would design and interpret an A/A test, what duration and sample size considerations you'd set, and what actions you'd take if the A/A test shows a statistically significant difference.

MediumTechnical

67 practiced

During an A/B experiment you detect differential attrition: treatment group has 12% dropout while control has 5%. Explain why differential attrition threatens internal validity and list at least four methods to assess whether attrition biases the result and mitigation strategies a PM could use.

Sample Answer

Differential attrition (12% treatment vs. 5% control) threatens internal validity because it can break randomization: if the people who drop out differ systematically by observed or unobserved characteristics that also affect the outcome, the remaining samples are no longer comparable and estimated treatment effects may be biased (confounded by who stayed).

How to assess whether attrition biases results (at least four methods):- Compare baseline covariates between dropouts and non-dropouts and across arms (age, prior behavior, geography). If dropouts differ on predictors of outcome, bias is likely.- Timing and pattern analysis: plot dropout over time and by cohort to see whether attrition spikes align with treatment delivery or bugs (suggests treatment-driven attrition).- Balance test of pre-treatment outcomes: check whether pre-period metrics remain balanced after attrition; imbalance indicates selection.- Sensitivity / worst-case bounds: run Lee bounds or worst-case imputation to quantify how large attrition-driven bias would need to be to overturn results.- Reweighing / adjustment: estimate propensity-to-stay and apply inverse probability weighting (IPW) or use multiple imputation for missing outcomes; compare adjusted vs. unadjusted estimates.- Instrumental / complier analysis: if you can define compliance, estimate complier average causal effect (CACE) rather than per-protocol.- Examine dropout reasons qualitatively: instrument short surveys or logs to capture whether dropouts were due to product, external factors, or random.

Mitigation strategies a PM should use:- Stop and investigate: pause rollout if attrition seems caused by the feature (e.g., critical bug or UX regression).- Collect dropout reasons and capture minimal outcome proxies (e.g., server logs, last action) to reduce missingness.- Re-analyze with conservative approaches (intention-to-treat, worst-case bounds, IPW, multiple imputation) and report ranges/uncertainty to stakeholders.- Run targeted follow-up experiments or A/A tests, or re-randomize with fixes to confirm effects.- Segment results and prioritize decisions on robust subgroups where attrition is low; avoid broad product decisions from biased estimates.- Document attrition and chosen adjustments in experiment reporting to keep stakeholders informed.

As a PM, combine quantitative checks with rapid qualitative investigation, present adjusted estimates alongside unadjusted, and delay irreversible rollouts until you’ve ruled out attrition-driven bias or fixed the cause.

HardTechnical

67 practiced

Define the Stable Unit Treatment Value Assumption (SUTVA) and give examples of interference in social or viral products (e.g., invitations, feeds, recommendations). Propose experimental designs to handle interference (cluster randomization, network experiments) and discuss the trade-offs of each approach from a product and statistical perspective.

Sample Answer

SUTVA (Stable Unit Treatment Value Assumption) states two things: (1) no interference between units — a unit’s potential outcomes depend only on its own assigned treatment, and (2) a single version of each treatment — treatments are well-defined and consistent. Violations occur in social/viral products where one user’s treatment affects others.

Examples of interference:- Invitations: Sending invites to user A increases sign-ups among A’s contacts (spillover).- Feeds: Showing a promoted post to A changes engagement for A’s friends (visibility cascades).- Recommendations: Personalization for A shifts item popularity, changing click-through rates for others.

Experimental designs to handle interference:1) Cluster randomization: Randomize treatment at group level (e.g., households, communities, social graph clusters). - Product trade-offs: Easier to implement and communicate; may require changing UX for whole groups. - Statistical trade-offs: Reduces interference across clusters but suffers from lower effective sample size, higher variance, and potential cluster-level confounding; requires ICC-aware power calculations.

2) Network experiments (exposure mappings / partial interference designs): Randomize individuals but model peer exposures (e.g., assign treatment probabilities to neighbors, or use dilution designs). Use stratified randomization on network metrics and estimate direct and spillover effects via contrasts of exposure conditions. - Product trade-offs: More granular; preserves individual-level rollout; requires tracking network structure and more complex rollout logic. - Statistical trade-offs: Enables estimation of both direct and indirect effects but needs stronger assumptions about exposure mapping, suffers from complexity in inference (dependence, heterogeneous exposure) and often requires specialized estimators (randomization inference, GEE, or Bayesian hierarchical models).

Practical guidance:- Map key interference pathways early; if strong local spillovers exist, prefer cluster randomization for operational simplicity and clear product impact signals.- If you need individual-level personalization insights and can instrument network exposures, use network experiments with pre-specified exposure conditions and rigorous power/inference plans.- Combine approaches: pilot network-aware randomizations to estimate spillover magnitude, then scale with cluster designs if spillovers threaten A/B validity.

HardTechnical

147 practiced

For experiments that touch checkout and payments, design automated guardrail metrics and a rollback strategy. Specify three guardrail metrics (with units), threshold types (relative or absolute), monitoring cadence, and automated actions (alert vs automatic rollback). Explain trade-offs between sensitivity and false alarms.

Sample Answer

Situation: Running experiments that change checkout or payments requires tight automated guardrails to protect revenue and user trust. Below is a practical design with three guardrail metrics, monitoring, automated actions, and rollback strategy, plus trade-offs on sensitivity vs false alarms.

Guardrail 1 — Payment Success Rate- Metric: Payment success rate (%) measured per hour- Threshold type: Relative drop (>= 1.5% absolute drop or >= 5% relative drop vs baseline, whichever triggers first)- Cadence: 5–15 minute aggregations, hourly evaluation- Action: Immediate alert to on-call payments engineer + automated rollback if drop sustained for 30 minutes (>= threshold on 3 consecutive hourly windows)

Guardrail 2 — Checkout Completion Conversion- Metric: Checkout completion rate (sessions → purchase) (percentage points)- Threshold type: Relative (>= 10% relative decrease vs control cohort)- Cadence: 15-minute aggregation, evaluated every 15 minutes- Action: Alert to PM/eng + soft traffic reduction (automatically scale experiment traffic down by 50%) if threshold exceeded for 2 consecutive windows; full rollback if persists for 2 hours

Guardrail 3 — Failed Charge / Error Volume- Metric: Absolute failed charge count / minute and error-rate per 1000 payment attempts- Threshold type: Absolute (>= 5 failed charges/min OR error-rate > 10 per 1,000)- Cadence: 1-minute streaming; real-time evaluation- Action: Immediate automatic rollback (100% traffic to control) + high-priority paged alert

Rollback strategy (automatic + human-in-loop):1. Automatic rollback triggers for severe, high-confidence signals (payment failures, fraud spikes). System routes all traffic to control; engineers and PMs are paged.2. For medium-severity or ambiguous signals, automated mitigation reduces exposure (traffic throttling) and pages team for investigation before full rollback.3. Post-rollback: run a root-cause checklist (logs, error traces, A/B cohort breakdowns), run a postmortem within 48 hours, and require a staged re-launch with smaller % and additional monitoring.

Trade-offs — sensitivity vs false alarms:- More sensitive thresholds detect real regressions faster but increase false positives, causing unnecessary rollbacks, developer context switching, and lost learning. Less sensitive thresholds reduce noise but risk slow detection of real customer/monetary harm.- Use hybrid rules: absolute thresholds for catastrophic signals (fail fast) and relative thresholds with sustained-confirmation windows for degradations to avoid reacting to noise.- Complement automated rules with cohort-aware baselining (time-of-day, payment method, geography) to reduce spurious alarms.

Operational notes:- Implement circuit-breaker with cooldown to prevent flip-flopping.- Log every automated action and require post-incident sign-off to re-enable experiment.- Regularly tune thresholds using historical A/B data and tabletop failure drills.

Unlock Full Question Bank

Get access to hundreds of A/B Test Results Analysis interview questions and detailed answers.

Join thousands of developers preparing for their dream job.