Hypothesis Testing and Inference Questions

Fundamental framework and application of hypothesis testing and statistical inference. Topics include formulating null and alternative hypotheses, understanding Type I and Type II errors, interpreting p values and confidence intervals, selecting and applying common tests such as t tests, chi square tests, analysis of variance, and non parametric alternatives, checking test assumptions, and discussing statistical versus practical significance. Candidates should explain power, significance levels, effect sizes, and common pitfalls such as misinterpreting p values or violating independence assumptions. At more advanced levels, discuss limitations of null hypothesis significance testing, alternatives such as Bayesian inference, and guidance for when different approaches are appropriate.

EasyTechnical

28 practiced

Define Type I and Type II errors in the context of product experiments. Give a concrete example for each (e.g., shipping a feature that seems to lift conversion but harms business or failing to ship a beneficial feature). Explain how sample size, significance level, and effect size influence both error types.

EasyTechnical

32 practiced

When should you use a one-sample t-test, a two-sample (unpaired) t-test, and a paired t-test? For each test: describe the unit-of-analysis, an example product experiment that fits, and the main assumption that must hold.

MediumTechnical

26 practiced

List the diagnostics and formal tests you would run to check the assumptions of a t-test (normality, equal variances, independence). For each failing diagnostic, propose practical remedies and explain when to prefer a transformation, a robust estimator, or a nonparametric test.

Sample Answer

**Overview — tests to run**- Normality: Shapiro–Wilk (small/medium n), QQ plot, histogram, Anderson–Darling. - Equal variances (homoscedasticity): Levene’s test (median), Bartlett’s test (normality assumed), Fligner–Killeen (robust). - Independence: Durbin–Watson (time-series residuals), autocorrelation function (ACF)/Ljung–Box, runs test, inspect study/design for clustering.

**If normality fails**- Remedies: - Transformation (log, sqrt, Box–Cox): use when failure is due to skewness and interpretability remains (e.g., positive measures like income). - Robust estimator: use Welch’s t if slight nonnormality with unequal Ns; use trimmed-mean t or bootstrap CI when outliers distort mean. - Nonparametric test: Wilcoxon/Mann–Whitney or permutation test when distributional shape differs strongly or sample size small.- Prefer transform when model interpretability on transformed scale is acceptable and variance stabilizes; prefer robust/perm tests when outliers or heavy tails predominate.

**If variances unequal**- Remedies: - Use Welch’s t (adjusted df) — first choice for unequal variances. - Transform if variance related to mean (e.g., log for multiplicative variance). - Use nonparametric or permutation test if variance inequality plus nonnormality.- Prefer Welch’s t for standard procedures; transform if it also fixes skewness.

**If independence fails**- Remedies: - Model dependency explicitly: mixed-effects model or generalized estimating equations for clustered/longitudinal data. - Use block/permutation tests that respect dependency structure or cluster-robust SEs. - Avoid simple t-tests when correlation exists.- Always prefer modeling dependency rather than transformations; robust SEs/permutation are practical when model complexity is constrained.

**Practical workflow (Applied Scientist)**1. Visual checks (QQ, residual plots), then formal tests. 2. If fail, try transformation that makes sense for domain; reassess diagnostics. 3. If still failing or uninterpretable, switch to robust estimators or permutation tests; model dependency with mixed models when needed.

Rationale: transformations fix mean–variance relationships; robust methods reduce influence of outliers/heavy tails; nonparametric/permutation ensure valid inference without parametric assumptions.

HardTechnical

31 practiced

You run many sequential experiments on overlapping user populations. Describe statistical and operational strategies to control Type I error and estimate true effects across this temporal/overlap structure. Address issues such as correlated tests, data leakage, user re-randomization, and when to use hierarchical models or alpha-spending approaches.

Sample Answer

**Situation & goal (one line)** Running many sequential A/B tests on overlapping users requires combining statistical controls (to keep Type I error low despite correlated, repeated tests) with operational rules (to prevent leakage and re-randomization) and estimation methods that pool information to recover true effects.

**Statistical strategies**- Pre-specify a family of hypotheses and choose error control: family-wise error (Bonferroni/closed testing) for conservative guarding; FDR (Benjamini–Hochberg) when many discoveries expected. If tests are correlated, use dependency-aware corrections (Benjamini–Yekutieli) or permutation-based nulls that preserve dependence.- Sequential monitoring: apply alpha‑spending functions (e.g., O’Brien–Fleming for late spending, Pocock for even spending) or group sequential tests to allocate cumulative Type I budget across looks. This prevents inflation from repeated peeks.- Use multivariate or joint testing when experiments overlap heavily: model test statistics jointly (multivariate normal with estimated covariance) and do simultaneous inference (e.g., Holm/Westfall procedures) to account for correlation.- Prefer Bayesian or hierarchical models for estimation: hierarchical (partial pooling) shrinks noisy per-experiment effects toward a global mean, improving MSE and giving coherent posterior probabilities for decision rules—especially useful when effects are small and correlated over time.- Meta-analysis / random-effects models to combine sequential runs or cohorts; use time-varying hierarchical models to capture drift.

**Operational controls**- Stable assignment: avoid re-randomizing users between active experiments. Use persistent assignment keys (hashed user_id + experiment namespace + start date) so each user has deterministic bucket.- Experiment graph & blocking: declare mutually exclusive treatments or orthogonalize features; create holdout cells for clean baselines.- Prevent data leakage: isolate logging pipelines per experiment, tag events with assignment version, and freeze analysis windows; use intention-to-treat analyses.- Washout windows & eligibility rules: prevent contamination by excluding recently exposed users or using cooldown periods before re-enrollment.- Accurate lineage & registry: register experiments, hypotheses, analysis plan, and alpha budget centrally; track overlapping cohorts to decide simultaneous vs. sequential handling.

**Handling user overlap, re-randomization, correlated tests**- If re-randomization unavoidable, treat repeated assignments as repeated‑measures: include user random effects, use clustered SE or mixed models to account for within-user correlation.- Estimate covariance of test statistics empirically (via jackknife/block bootstrap preserving user-level dependence) and feed to joint tests or multiplicity adjustments.- When experiments share users and metrics, compute effective number of independent tests (e.g., eigenvalue methods) to scale corrections less conservatively.

**When to use hierarchical models vs. alpha‑spending**- Use hierarchical (Bayesian or frequentist mixed) models when goal is accurate estimation and shrinkage improves power/estimates across many small experiments or cohorts. Hierarchical models naturally borrow strength across time, handle user-level repeated measures, and give posterior probabilities for decisions—good for continual learning pipelines.- Use alpha‑spending/group sequential methods when strict frequentist Type I control for hypothesis testing decisions is required (regulatory or business-critical thresholds) and when you need to allow interim looks with guaranteed error control.

**Example policies (operational + statistical)**- Register experiment, allocate alpha budget from central pool.- If overlapping users: (a) if hypothesis-critical, reserve a fresh holdout or block overlapping allocation; (b) otherwise, allow overlap but analyze with hierarchical model and joint covariance correction.- For interim peeks: use O’Brien–Fleming spending with pre-specified look times; otherwise adopt Bayesian decision threshold (e.g., posterior prob > 0.975) mapped to equivalent alpha.

**Practical tips**- Log user exposures and timestamps; enforce deterministic assignment; compute user-level aggregates before analysis; run cluster-robust or mixed-effect models; simulate pipelines to estimate FWER/FDR under realistic overlap scenarios before deployment.

This combination—policy (assignment, registry), correct multiplicity procedures (alpha‑spending or dependency-aware corrections), and hierarchical modeling for estimation—controls Type I risk operationally and statistically while giving more accurate effect estimates under overlap and temporal dependence.

HardTechnical

46 practiced

Leadership question: As an applied-science lead, outline a program to establish experiment governance (protocols, training, reviews) that minimizes false discoveries and improves reproducibility across teams. Include concrete elements: experiment registry, pre-registration templates, code review policies, metrics monitors, post-experiment audits, mentoring plans, and measurable KPIs to track success.

Unlock Full Question Bank

Get access to hundreds of Hypothesis Testing and Inference interview questions and detailed answers.

Join thousands of developers preparing for their dream job.