Hypothesis and Test Planning Questions

End to end practice of generating clear testable hypotheses and designing experiments to validate them. Candidates should be able to structure hypotheses using if change then expected outcome because reasoning ground hypotheses in data or qualitative research and distinguish hypotheses from guesses. They should translate hypotheses into experimental variants and choose the appropriate experiment type such as A and B tests multivariate designs or staged rollouts. Core skills include defining primary and guardrail metrics that map to business goals selecting target segments and control groups calculating sample size and duration driven by statistical power and minimum detectable effect and specifying analysis plans and stopping rules. Candidates should be able to pre register plans where appropriate estimate implementation effort and expected impact specify decision rules for scaling or abandoning variants and describe iteration and follow up analyses while avoiding common pitfalls such as peeking and selection bias.

HardTechnical

72 practiced

Design an experiment where the primary outcome is user retention measured as time-to-churn. Explain how you would use survival analysis methods such as Kaplan-Meier curves and Cox proportional hazards models in the analysis plan, handle right censoring, compute a sample size for detecting a target hazard ratio, and translate hazard ratio results into interpretable business impact for stakeholders.

Sample Answer

**Overview / goal**Design an A/B experiment where primary outcome is time-to-churn (days until user becomes inactive). Use survival methods to estimate retention curves, test treatment effect, and quantify business impact.

**Design**- Randomize new users to Control vs Treatment; record start date, churn date (event) or last-observed active date (censor).- Pre-specify follow-up window (e.g., 180 days) and covariates (cohort, device, region).- Primary estimand: hazard ratio (HR) of churn T vs C.

**Analysis plan**- Descriptive: plot Kaplan–Meier curves for each arm; report median time-to-churn and survival probabilities at business-relevant times (30/90/180d).- Hypothesis test: log-rank test for global difference.- Adjusted analysis: Cox proportional hazards model with covariates: - Fit: hazard(t|X) = h0(t) * exp(beta * Treatment + gamma'X). - Interpret exp(beta) as HR.

**Right censoring**- Treat users still active at last follow-up as right-censored.- Ensure non-informative censoring: monitor differential loss-to-follow-up; perform sensitivity analyses (e.g., inverse-prob weighting).- If heavy interval censoring (infrequent pings), use interval-censored methods.

**Assumption checks & robustness**- Check proportional hazards via Schoenfeld residuals; if violated, estimate time-varying HR (interaction with log time) or use restricted mean survival time (RMST) difference.- Bootstrap CIs for medians and RMST.

**Sample size (detect target HR)**- Use Schoenfeld approximation: required number of events D:

text

D = ( (Z_{1-alpha/2} + Z_{1-beta})^2 ) / ( (log HR)^2 * p * (1-p) )

- Where p is proportion randomized to treatment; then enroll N = D / expected event rate within follow-up.- Intuition: smaller HR (closer to 1) requires more events; longer follow-up or higher-risk cohorts reduce N.

**Translating HR to business impact**- Convert HR to absolute retention differences: use baseline KM S0(t) and compute S1(t) = S0(t)^{HR?} (approx via Cox) or predict survival from Cox to get survival probabilities at key times.- Compute incremental retained users at time t = N_users * (S1(t) - S0(t)).- Estimate incremental lifetime value = incremental retained users * average revenue per retained user (over horizon) — present as expected revenue uplift and CI using delta method or bootstrap.- Provide scenarios: conservative, best, worst.

**Example**- Baseline 90-day retention 40%. Cox HR=0.8 (20% reduction in churn hazard) → predict 90-day retention ~ 46% (approx +6 pp). For 10k new users, ~600 additional retained at 90d. Multiply by ARPU to get revenue.

This plan provides statistically rigorous inference, checks assumptions, handles censoring, and converts HRs into actionable business KPIs.

MediumTechnical

74 practiced

You observe a small average lift in an experiment. Describe a principled approach to search for heterogeneous treatment effects across cohorts such as geography, device, and new versus returning users while controlling false discoveries. Include discussion on pre-specification, multiplicity correction, and using hierarchical models for shrinkage.

Sample Answer

**Approach overview**

Pre-specify cohorts and estimands- Before peeking, list subgroup axes (e.g., region, device, new/returning), the exact cohorts (e.g., top 10 countries), and the metric/interaction to test (ATE difference or treatment×cohort coefficient). Limit number to what's actionable and powered.

Stage 1 — global check and interaction test- Run an omnibus interaction test (e.g., likelihood ratio or permutation test) to detect any heterogeneity. If no signal, avoid exhaustive slicing.

Multiplicity control and exploratory protocol- Split analysis into confirmatory (pre-specified) and exploratory sets. For confirmatory subgroup tests use family-wise or FDR control: - Use Benjamini–Hochberg to control FDR when many correlated subgroup tests are run. - Use Bonferroni for a small, strict family to control FWER.- For exploratory findings, label them hypothesis-generating and validate on a fresh holdout experiment or temporal split.

Hierarchical / multilevel modeling for shrinkage- Fit a hierarchical model: subgroup effect ~ Normal(mu, tau), observations ~ Normal(effect_subgroup, sigma_subgroup). This partially pools estimates toward the global mean, reducing noise-driven extremes.- Advantages: lower MSE, credible intervals that account for multiplicity implicitly, and ability to borrow strength for small cohorts.- Report posterior summaries: posterior mean, 95% credible intervals, and posterior probability that effect > minimum detectable/business-relevant threshold.

Practical checks and reporting- Check power: small average lift implies low SNR; emphasize minimum detectable effect per cohort.- Visualize: forest plots with shrinkage estimates, and calibration of p-values/posterior probabilities.- Reproducibility: document pre-specification, adjustment method, and whether result is confirmatory or exploratory; follow up top exploratory signals with targeted experiments.

This pipeline balances discovery with principled multiplicity control and uses hierarchical models to produce stable, interpretable subgroup estimates.

MediumTechnical

66 practiced

You must test a new pricing message but it must be randomized at city level because implementation differs by local teams. Baseline conversion is 5% and you estimate intra-class correlation (ICC) of 0.02. Explain how cluster randomization affects required sample size, show the design effect formula, and outline steps to compute the number of clusters required per arm, making reasonable assumptions about cluster size.

Sample Answer

**Brief answer / intuition**

Cluster randomization inflates the required sample size because observations within the same city are correlated. The inflation factor is the design effect (DE), which multiplies the individual-randomization sample size.

**Design effect formula**

text

DE = 1 + (m - 1) * ICC

Plain-English: m = average cluster (city) size; ICC = intra-class correlation.

**Worked example (numbers & steps)**- Goal: detect an absolute lift from 5.0% to 6.0% (delta = 0.01), two-sided alpha = 0.05, power = 80% (Z = 1.96, 0.84).- Compute required individuals per arm under individual randomization using two-proportion normal approximation: - p1 = 0.05, p2 = 0.06, p̄ = 0.055 - This yields n_ind ≈ 8,149 per arm (calculation shown below if needed).- Assume ICC = 0.02 and reasonable average city size m = 2,000 users during the experiment. - DE = 1 + (2000 - 1)*0.02 ≈ 40.98 - Adjusted individuals per arm = n_ind * DE ≈ 8,149 * 40.98 ≈ 334,000 - Clusters required per arm K = (n_ind * DE) / m ≈ 334,000 / 2,000 ≈ 167 cities per arm

**Practical steps you’d take**1. Choose meaningful detectable effect (absolute or relative) and power/alpha.2. Compute n_ind per arm for individual randomization (two-proportion formula).3. Estimate ICC from historical A/B tests or pilot; pick realistic m (avg city users).4. Compute DE and adjusted total individuals; convert to clusters per arm: K = (n_ind * DE) / m.5. Sensitivity: vary ICC and m to produce a table of required K; if K too large, consider increasing delta, running longer (increase m), or using covariate adjustment/stratified randomization to reduce effective ICC.

**Notes / trade-offs**- High ICC or small m makes experiments very expensive in clusters. Covariate adjustment, matching cities, or pairwise randomization can substantially reduce required clusters.

MediumTechnical

99 practiced

Compare frequentist sequential testing with alpha-spending to a Bayesian decision-theory approach for experiment stopping and scaling. For a growth metric that can cause costly negative outcomes, outline practical decision thresholds for each approach and describe how you would validate either approach using simulation.

Sample Answer

**Compare approaches — high level**

- Frequentist (alpha‑spending sequential testing): control long‑run Type I error by allocating a total alpha (e.g., 0.05) across interim looks using an alpha‑spending function (O’Brien‑Fleming for conservative early looks, Pocock for more even spending). Decisions are p‑value based: stop for efficacy if interim p < boundary; stop for harm if p_harm < boundary. - Bayesian decision‑theory: combine posterior distributions with an explicit loss/utility function that encodes costs of negative outcomes and benefits of scaling. Decisions minimize posterior expected loss: choose stop, scale, or continue based on comparative expected utilities.

**Practical thresholds (growth metric with costly negative outcomes)**

- Frequentist practical policy - Overall alpha = 0.05, looks every 2 weeks. Use O’Brien‑Fleming spending to keep early thresholds strict. - Stop for harm early: two‑sided test but flag harm if p_harm < 0.001 at early looks, relaxing to p_harm < 0.025 at final. Stop for benefit with symmetric thresholds. - Add minimum sample/observation floor before first look to avoid volatility.

- Bayesian practical policy - Define delta_harm = minimal clinically/operationally relevant negative effect (e.g., -0.5% conversion). - Stop for harm if posterior P(effect < delta_harm) > 0.95 OR if expected loss of continuing > expected loss of stopping (quantify cost per negative event × expected additional events). - Scale if posterior P(effect > delta_benefit) > 0.99 and expected gain outweighs rollout cost. - Use informative/deliberate priors reflecting historical variability; run sensitivity checks.

**Why these thresholds**- Frequentist thresholds guarantee long‑run alpha control; OBF preserves low false alarms early when data are sparse. - Bayesian thresholds directly trade off business costs and uncertainties; probability thresholds (0.95/0.99) are intuitive and map to risk tolerance; expected‑loss criterion aligns with business impact.

**Validation via simulation (how I would validate)**

1. Define scenarios: null, small positive, large positive, small negative (costly), rare catastrophic harm. Include temporal patterns and heteroskedasticity reflecting real data. 2. Encode experiment process: sampling cadence, look schedule, minimum sample per look, data noise, user heterogeneity, delayed effects. 3. For each policy (alpha‑spend variant, Bayesian with chosen priors/loss), simulate N trials (e.g., 50k) and collect: - Type I error, power, false stop for harm, time-to-decision, average sample size, expected cumulative loss/regret, economic ROI. 4. Calibrate thresholds: adjust alpha‑spend or Bayesian probability/cost weights to meet constraints (e.g., max acceptable probability of catastrophic rollout, budgeted sample size). 5. Sensitivity analysis: vary priors, misspecification, effect heterogeneity, delayed effects; report robustness. 6. Operational check: run “post‑deployment” sims where a decision to scale produces downstream user impacts to measure realized business cost.

**Closing practical notes**- For an applied scientist: implement simulators in Python (NumPy/Numba), parallelize, and supply dashboards of operating characteristics. - Choose approach by constraints: use frequentist when regulatory alpha guarantees are required; use Bayesian decision‑theory when business costs can be quantified and faster, utility‑aware decisions are preferred.

HardTechnical

67 practiced

You run a multivariate test with 7 elements and several derived metrics, creating hundreds of comparisons. Describe a principled analysis pipeline to control false discoveries: define families of hypotheses, propose hierarchical or gatekeeping testing, explain FDR procedures appropriate here, and suggest a protocol for pre-specified confirmatory tests versus exploratory analyses. Discuss tradeoffs between power and Type I error control.

Sample Answer

**Brief summary of goals**Control false discoveries across hundreds of comparisons from a 7‑element multivariate test while preserving power for pre‑specified confirmatory tests and enabling principled exploration.

**1) Define families of hypotheses**- Family A (confirmatory): a small set (pre‑registered) of primary metrics and contrasts (e.g., revenue, conversion for element combos).- Family B (secondary confirmatory): key secondary metrics pre‑specified.- Family C (exploratory): all remaining derived metrics and post‑hoc contrasts.

Separating families limits multiplicity correction burden and enforces transparency.

**2) Hierarchical / gatekeeping testing**- Gate 1: Test Family A using strong control (e.g., Holm or BY if dependence unknown). Only if Gate 1 contains rejections proceed to Gate 2.- Gate 2: Test Family B conditional on Gate 1 results (gatekeeping preserves Type I error across families).- Family C handled separately as exploratory with looser control (see below).

This hierarchy protects primary inferences and prioritizes decision rules.

**3) FDR procedures appropriate here**- For large sets within a family (especially exploratory), use Benjamini–Hochberg (BH) FDR under positive dependence; use Benjamini–Yekutieli (BY) if arbitrary dependence is suspected (more conservative).- If test statistics have known correlation structure (e.g., overlaps among derived metrics), estimate covariance and apply adaptive BH (Storey’s q‑value with pi0 estimation) or use knockoffs / permutation-based FDR to better control error with higher power.- When testing contrasts across elements, consider hierarchical FDR (e.g., Yekutieli’s hierarchical BH) to exploit tree structure (element → metric).

**4) Pre‑specified confirmatory vs exploratory protocol**- Pre-register Family A and B: hypotheses, directionality, analysis script, sample size and stopping rules. Apply strict multiplicity correction (familywise error for top decisions).- Exploratory (Family C): mark as hypothesis‑generating; control FDR (BH) and report q‑values, effect sizes, and replication plan. Do not treat discoveries here as definitive without confirmatory follow‑up.

**5) Practical steps / pipeline**- Pre-specify families and analysis code; lock primary analysis before unblinding.- Compute test statistics and p‑values; estimate dependence via permutation/bootstrap.- Apply gatekeeping/hierarchical corrections for confirmatory families.- Apply BH/Storey q‑values or permutation FDR for exploratory sets; report adjusted p/q, confidence intervals, and minimum detectable effects.- For top exploratory hits, run pre‑planned replication experiments or A/A tests.

**6) Tradeoffs: power vs Type I error**- Strong FWER control (Bonferroni/Holm/BY) minimizes false positives but reduces power—risk missing actionable effects.- FDR control (BH/Storey) increases power but accepts a controlled proportion of false discoveries—appropriate for exploration or when many tests are expected.- Hierarchical/gatekeeping improves power for prioritized hypotheses by not diluting α across all tests.- Use dependence‑aware or permutation procedures to reclaim power when tests are correlated.

**Takeaway**Predefine families, prioritize confirmatory hypotheses with gatekeeping and stricter error control, use FDR and dependence‑aware methods for large exploratory sets, and require replication for exploratory findings. This balances actionable decisions with controlled false discovery in a production‑oriented applied science setting.

Unlock Full Question Bank

Get access to hundreds of Hypothesis and Test Planning interview questions and detailed answers.

Join thousands of developers preparing for their dream job.