Hypothesis Testing and Inference Questions

Fundamental framework and application of hypothesis testing and statistical inference. Topics include formulating null and alternative hypotheses, understanding Type I and Type II errors, interpreting p values and confidence intervals, selecting and applying common tests such as t tests, chi square tests, analysis of variance, and non parametric alternatives, checking test assumptions, and discussing statistical versus practical significance. Candidates should explain power, significance levels, effect sizes, and common pitfalls such as misinterpreting p values or violating independence assumptions. At more advanced levels, discuss limitations of null hypothesis significance testing, alternatives such as Bayesian inference, and guidance for when different approaches are appropriate.

MediumTechnical

26 practiced

You observe conversion counts for three landing pages: A = 120/1,000, B = 150/1,000, C = 135/1,000. As a BI analyst, select and justify the appropriate hypothesis test to determine whether conversion rates differ across pages. Describe test statistic, assumptions, and what post-hoc analysis you would run if the test is significant.

EasyTechnical

30 practiced

Your weekly BI report runs dozens of hypothesis tests. Explain the multiple comparisons problem, contrast family-wise error rate control with false discovery rate (FDR) control, and recommend a practical strategy for a BI team that wants to surface meaningful signals but avoid too many false alarms.

MediumSystem Design

25 practiced

Design an A/B test for a new checkout flow: define the primary and secondary metrics, unit of randomization, how to calculate sample size (include formula and input assumptions), experiment duration, and a principled stopping rule. Discuss trade-offs between speed and statistical rigor.

Sample Answer

Requirements & constraints:- Goal: Measure impact of new checkout flow on revenue and conversion without harming user experience.- Traffic: desktop + mobile; expected baseline conversion rate (CR) = 8%; average order value (AOV) = $60.- Risk tolerance: avoid >1% absolute drop in CR.

High-level design:- Primary metric: Revenue per user (RPU) over a session (captures conversion + AOV).- Secondary metrics: Conversion rate (orders / sessions), AOV, checkout completion time, error rate, refund rate, and retention (next-week purchase).- Unit of randomization: User-level (cookie / logged-in user id) to avoid cross-contamination across sessions/devices (if cross-device identity available, use that).

Sample size calculation:- Want to detect a minimum detectable effect (MDE) of 3% relative uplift in RPU.- Use two-sample z-test approximation. For continuous RPU, convert to standardized effect using sd(RPU).- Formula (for proportions simplified to CR): n_per_variant = [ (Z_{1-α/2} + Z_{1-β})^2 * (p0(1-p0) + p1(1-p1)) ] / (p1 - p0)^2 For RPU (continuous): n = 2 * (Z_{1-α/2} + Z_{1-β})^2 * σ^2 / Δ^2- Assumptions example: α=0.05 (Z=1.96), power=80% (Z=0.84), baseline RPU = $4.8 (8% * $60), σ ≈ $10 (empirical), Δ = 3% * 4.8 = $0.144.- Plugging: n ≈ 2*(2.8)^2*100 / 0.0207 ≈ 2*7.84*100/0.0207 ≈ 156800/0.0207 ≈ ~7.6M users per variant — indicates high variance; consider focusing on CR or using log-transform / variance reduction.

Experiment duration:- Run for at least time to accumulate n and cover weekly cycles (min 2 full weeks, preferably 4) to smooth seasonality. If required sample is large, prioritize metrics with lower variance (CR) to get power sooner.

Stopping rule (principled):- Pre-register primary metric, MDE, α, power, analysis method.- No peeking: do not stop early based on interim p-values unless using alpha spending (e.g., O’Brien-Fleming) or sequential methods (e.g., group sequential, Bayesian stopping with pre-specified posterior thresholds).- Accept early stopping only if using adjusted boundaries: e.g., O’Brien-Fleming with interim analyses at 50% and 100% information, or Bayesian posterior P(Δ>0) > 0.99 to declare win, <0.01 to declare loss.

Trade-offs (speed vs rigor):- Speed: shorter tests by raising MDE, lowering power, or testing on CR (lower variance) — faster signals but higher chance of missing small true effects.- Rigor: strict α, pre-registration, long duration, controlling for multiple metrics — slower, higher resource but robust decisions.- Practical compromise: use RPU as business objective but run powered test on CR (or log-RPU) for speed, and inspect secondary metrics; use sequential methods to allow safe early stopping.

Implementation notes:- Ensure instrumentation, consistent logging, and guardrails (rollback if error rate spikes).- Report effect sizes, CIs, not just p-values, and present dashboards showing incremental revenue, lift, and sample demographics.

HardTechnical

30 practiced

You only have aggregated monthly means and counts per segment (no per-user data). Which hypothesis tests remain valid, how would you estimate standard errors, and what assumptions must hold for inference? Provide at least two approaches to approximate per-user variance or to run conservative tests from aggregated data.

Sample Answer

Short answer: many standard tests that require independent observations can still be approximated from aggregated-monthly means and counts, but you must be explicit about assumptions (no within-month correlation structure, homoskedasticity or known cluster sizes). You can get valid inference for differences of means between segments using pooled-variance t-tests or z-tests if you can estimate the per-user variance; otherwise run conservative tests using aggregated-level variances (which overstate p-values).

What remains valid- Tests on group/month means (t-test, z-test, linear regression) are valid if the monthly means are independent and each monthly mean is formed from many independent users (CLT applies).- Time-series or paired tests are valid at the monthly-aggregate level if you model autocorrelation.

How to estimate standard errors and assumptions- If you have monthly mean m_j and count n_j for segment j, the SE of m_j equals sqrt(Var_per_user / n_j). So you need an estimate of Var_per_user or assumptions to bound it.- Key assumptions: per-user observations within a month are i.i.d. within segment, users’ contributions average out (central limit), no severe ecological bias (segment composition stable), and independence across months unless you model serial correlation.

Two practical approaches

1) Recover per-user variance from external or historical data- Use past datasets or a sample with per-user records to estimate Var_per_user by segment; plug into SE = sqrt(sigma^2 / n_j). This yields near-standard inference (t/z). If only a proxy is available, treat it as sensitivity analysis: report results across plausible sigma values.

2) Conservative tests from aggregated data (no per-user sigma)- Use the between-month variability of the monthly means as an upper-bound variance: estimate Var(m_j) across months and use SE_est = sqrt(Var(m_j)). When comparing two segments, use Welch’s t with aggregated-mean variances. This is conservative because Var(m_j) = sigma^2/n_j + month-level heterogeneity; using it inflates SE and reduces Type I error.- Alternatively, perform permutation/bootstrap at the monthly-aggregate level: shuffle month labels across segments (preserving counts) to create null distribution for mean differences. This produces valid p-values under exchangeability of months.

Extra considerations- Adjust for unequal n_j across months (use weighted means).- If serial correlation exists, use cluster-robust SEs at month-level or model AR(1) on aggregated means.- Always report assumptions, run sensitivity checks over sigma, and present conservative and less-conservative results so stakeholders understand robustness.

MediumTechnical

42 practiced

You're building an executive dashboard to present A/B test results. Describe how you would present p-values, confidence intervals, effect sizes, and uncertainty visually and textually so that executives can make informed decisions. Include at least two widget ideas (e.g., impact bar with CI, risk band) and how you would annotate practical guidance.

Sample Answer

Goal: make statistics actionable and non-misleading for executives — show magnitude, direction, and confidence, plus a clear recommendation and next steps.

High-level layout:- Top-line summary card per KPI: point estimate (effect size, e.g., +3.2%), 95% CI, simple verdict (Recommend / Monitor / Stop), and a single sentence explanation (e.g., “Estimated +3.2% conversion; 95% CI [-0.4%, +6.8%]; 78% probability of improvement — consider rollout to 25% more users and remeasure”).

Widget 1 — Impact bar with CI- Visual: horizontal bar centered on baseline = 0. Point estimate as thick marker; 95% CI as a darker band; 50% credible region (or SE) as a lighter band around it. Use diverging color: green for positive, red for negative, grey if crosses zero.- Interactivity: hover shows numeric effect, CI bounds, p-value, sample sizes, and power.- Annotation: label “Practical impact” with interpretation: “If true effect = upper bound, revenue +$X/mo; if true effect = lower bound, revenue -$Y/mo.” Include recommended action based on business thresholds.

Widget 2 — Risk band / Probability-of-harm strip- Visual: continuous probability density along effect axis (from Bayesian posterior or bootstrap). Shade area left of 0 (risk) in red with percent label e.g., “12% chance of hurting conversions”.- Show vertical lines for business minimum detectable effect (MDE) and zero; annotate “Acceptable risk threshold = 5%” if applicable.- Add a small gauge showing “Probability of being better than baseline” (e.g., 78%) and “Probability > MDE” (e.g., 42%).

Supplemental visuals- Funnel or cohort table showing sample size, event counts, and realized conversion rates — lets execs inspect data sufficiency.- Trend + cumulative effect chart to show stability over time (helps detect peeking/early stopping issues).

Textual cues & annotations (for every KPI)- Plain-language headline: “Result: Likely positive / Inconclusive / Likely negative”- Key stats line: Effect = X% (95% CI [a%, b%]), p = 0.03, N_control = xx, N_variant = yy, power = zz%- Business translation: “Estimated monthly revenue change = +$12k (95% CI -$1.5k to +$25k)”- Decision rule: map statistical outcome to action (example): - Recommend rollout if P(effect > 0) > 95% OR P(effect > MDE) > 80% AND power >= 80% - Recommend further testing if CI crosses 0 and business impact uncertain - Stop if P(effect < 0) > 90% or expected loss > threshold- Caveats: note multiple testing, duration bias, segments with heterogeneity; link to methods doc.

Design & implementation tips- Prefer probability statements and intervals over raw p-values; show p-value but de-emphasize it visually.- Use Bayesian posterior/bootstrapping for intuitive probabilities if possible; otherwise compute “probability of positive effect” via sampling.- Color/code consistently and avoid misleading precision (round to 1–2 significant digits).- In Tableau/Power BI: use parameter controls for MDE and sample-size calculators, tooltips for revenue translation, and parameterized bands for CI shading.

This layout ensures executives see magnitude, uncertainty, and practical trade-offs at a glance, with clear, actionable recommendations.

Unlock Full Question Bank

Get access to hundreds of Hypothesis Testing and Inference interview questions and detailed answers.

Join thousands of developers preparing for their dream job.