Result Interpretation & Decision Making Questions

Learn to interpret experiment results: 'This is statistically significant. Should we ship it?' Consider practical significance (is the improvement large enough to matter?), business impact (does it align with goals?), and risks (could it have negative second-order effects?). Practice decision frameworks: ship if significant and directionally positive? Roll out gradually to de-risk? Run additional confirmatory experiments?

MediumTechnical

29 practiced

You're evaluating a time-limited promotion that could attract fraud. What analyses and metrics would you run to estimate fraud risk and its impact on measured uplift? How should fraud risk change the decision to ship or roll out the promotion further?

Sample Answer

Approach (overview): treat fraud as an adversarial bias that inflates observed uplift. I’d quantify incidence, estimate its contribution to uplift, run sensitivity analyses, and integrate uncertainty into go/no-go decisions.

Analyses & metrics:- Basic signals: conversion rate, average order value (AOV), promo redemptions per account, redemptions per IP/device, new-account share, time-to-redemption distribution.- Fraud fingerprints: high-frequency redemptions, identical payment instruments, device-ID churn, geolocation clusters, mismatched billing/shipping, velocity rules.- Comparative metrics: uplift on all users vs. uplift after removing suspected-fraud cohort; ratio of suspicious redemptions to total redemptions; lift-per-cohort (new vs. existing users).- Statistical checks: compare variance between treatment/control (Levene), run permutation tests excluding suspicious events, and compute adjusted uplift with bootstrapped CIs.- Modeling: train a supervised fraud model (or use rules) to score events; estimate expected true uplift = observed uplift - expected fraud value (sum(score * txn_value)). Run Bayesian hierarchical model to propagate uncertainty.- Attack simulation: stress-test by injecting synthetic fraud at various rates to see breakpoint where ROI flips negative.

Decision rules for rollout:- If estimated fraud contribution is small relative to uplift and adjusted uplift CI > business threshold → proceed full rollout with monitoring.- If fraud materially reduces or reverses uplift (adjusted uplift ≤ 0) or uncertainty is high → halt or limit to a small, monitored pilot while hardening controls.- If moderate risk: staged rollout with stricter fraud controls (rate limits, device/email verification, CAPTCHAs) and clear kill-switch metrics (e.g., suspicious-redemptions > X% or delta AOV > Y).

Operationalize:- Real-time dashboards for fraud signals, automated alerts, and daily recalculation of adjusted uplift.- Close loop with product/ops to deploy mitigations and re-run analyses.

This balances empirical estimation, uncertainty, and operational safeguards so the promotion can be scaled only when genuine uplift outweighs fraud risk.

MediumTechnical

30 practiced

You're tasked to design an A/B test to detect a 2% relative lift in conversion from baseline 4%, with 80% power and alpha=0.05. Describe how to compute the required sample size, what formulas or approximations you'd use, and which assumptions you must validate before launching.

Sample Answer

Approach (brief): use a two-proportion z-test (normal approximation) to compute per-group sample size for detecting p2 = p1*(1+rel_lift) with power 80% and alpha 0.05. Key formula (equal-sized groups):

n = [ (z_{1−α/2} * sqrt(2 p̄(1−p̄)) + z_{power} * sqrt(p1(1−p1)+p2(1−p2)) )^2 ] / (p2 − p1)^2

where p̄ = (p1 + p2)/2, z_{1−α/2}=1.96, z_{power}=0.84.

Concrete numbers (baseline p1=0.04, rel lift 2% → p2=0.0408, Δ=0.0008):- p̄≈0.0404 → compute gives n ≈ 950,000 per group → ≈1.9M total (so very large).

Python implementation:

python

import math
from scipy.stats import norm

p1 = 0.04
p2 = p1 * 1.02
alpha = 0.05
power = 0.8
z_alpha = norm.ppf(1 - alpha/2)
z_power = norm.ppf(power)
p_bar = (p1 + p2) / 2
num = (z_alpha * math.sqrt(2 * p_bar * (1 - p_bar)) + z_power * math.sqrt(p1*(1-p1) + p2*(1-p2)))**2
n_per_group = num / (p2 - p1)**2
print(n_per_group, "per group")

Assumptions to validate before launch:- Binary conversions ~ Bernoulli, independence of users, randomization integrity (no bias).- Stable baseline (no major seasonality or product changes).- No interference/SUTVA (treatment of one user doesn't affect others).- Expected variance matches model; check pre-experiment variance.- No peeking/multiple testing (plan analysis or use alpha spending).- Sufficient experiment runtime to reach required sample without external changes.

Alternatives if sample size infeasible:- Target a larger effect size, use more sensitive metric (continuous value), pool segments, run longer, or apply sequential methods with adjusted error control, or use Bayesian A/B testing with priors.

MediumTechnical

43 practiced

A product manager prefers Bayesian credible intervals while analytics reports p-values. Explain differences in interpretation and practical pros/cons of Bayesian A/B testing versus frequentist methods when making business decisions under uncertainty.

Sample Answer

Credible intervals vs p-values — interpretation- Bayesian credible interval (e.g., 95% CI): given the observed data and prior, there is a 95% probability the true parameter lies in this interval. It's a direct probability statement about the parameter.- Frequentist p-value: probability of observing data at least as extreme as observed assuming the null hypothesis is true. It does NOT give the probability the null is true or an interval for the parameter.

Practical pros/cons for A/B testing

Bayesian (pros)- Direct decision quantities: posterior distributions give probability treatment is better by X, expected uplift, probability of beating business thresholds — easier to communicate to PMs.- Natural sequential testing: you can peek and stop without complex alpha correction (if decision rule is Bayesian), enabling faster experiments.- Incorporates prior knowledge: useful when historical data exists or when small samples; hierarchical models pool information across segments.- Rich uncertainty quantification: full posterior supports risk-aware decisions (e.g., decision-theoretic expected value).

Bayesian (cons)- Requires choosing priors — can be controversial; needs sensitivity checks.- More computationally intensive for complex models (MCMC), though approximate methods (variational, analytic conjugates) mitigate this.- Not universally accepted in regulated settings where frequentist thresholds are standard.

Frequentist (pros)- Simple, well-understood workflows: t-tests, z-tests, clear sample-size formulas.- Widely accepted benchmarks (p<0.05) and tooling; often quicker for simple experiments.- Inference doesn't require subjective priors.

Frequentist (cons)- P-values are commonly misinterpreted; they don’t tell probability of benefit.- Sequential peeking inflates Type I error unless adjusted (alpha spending, Bonferroni, sequential tests).- Confidence intervals have an indirect interpretation (coverage over repeated samples), which is less intuitive for single-study business decisions.

Practical recommendation for product decisions- If stakeholders want direct probability statements and you can specify reasonable priors, use Bayesian A/B testing for clearer risk/utility decisions (e.g., “80% chance revenue increases by ≥1%”).- If you need fast, standard tests and regulatory acceptance, frequentist methods are fine, but enforce pre-registration, proper sample-size planning, and sequential correction.- Regardless of approach: report effect sizes, uncertainty (posterior or CI), decision thresholds tied to business metrics (expected loss/gain), and run sensitivity checks (priors, stopping rules).

EasyTechnical

32 practiced

Describe how to compute and interpret absolute lift and relative lift in an A/B experiment. Use this example: control conversion = 5.0%, treatment conversion = 5.5%. Explain when absolute vs relative lift matters more for business decisions and how to present both to stakeholders.

EasyTechnical

33 practiced

List the properties of a good primary metric for a growth experiment. For a mobile onboarding experiment, propose a primary metric and two secondary metrics, justify your choices, and explain how you'd instrument them to avoid ambiguity.

Unlock Full Question Bank

Get access to hundreds of Result Interpretation & Decision Making interview questions and detailed answers.

Join thousands of developers preparing for their dream job.