Hypothesis and Test Planning Questions

End to end practice of generating clear testable hypotheses and designing experiments to validate them. Candidates should be able to structure hypotheses using if change then expected outcome because reasoning ground hypotheses in data or qualitative research and distinguish hypotheses from guesses. They should translate hypotheses into experimental variants and choose the appropriate experiment type such as A and B tests multivariate designs or staged rollouts. Core skills include defining primary and guardrail metrics that map to business goals selecting target segments and control groups calculating sample size and duration driven by statistical power and minimum detectable effect and specifying analysis plans and stopping rules. Candidates should be able to pre register plans where appropriate estimate implementation effort and expected impact specify decision rules for scaling or abandoning variants and describe iteration and follow up analyses while avoiding common pitfalls such as peeking and selection bias.

HardTechnical

62 practiced

An experiment shows +4% lift in a vanity engagement metric (p=0.02) but revenue per user decreased by 3% (p=0.06). Describe how you'd combine this evidence to make a rollout decision. Include how you'd estimate expected business impact, run simulations for Type S/M errors, and decide whether to run follow-ups or a staged rollout.

Sample Answer

Approach: treat both metrics jointly (vanity engagement up +4%, p=0.02; revenue/user down −3%, p=0.06). I’d quantify expected business impact, assess uncertainty (including Type S: wrong sign, and Type M: exaggerated effect), simulate posterior distributions, and choose between follow-up experiment, staged rollout, or reject.

1) Estimate expected business impact- Convert % changes to dollars: ΔRevenue_per_user = −3% ± SE; ΔEngagement = +4% ± SE. If ARPU = $X and N users, expected ΔRevenue = N * ARPU * E[Δ%].- Build a simple value function that weights revenue change heavily and engagement as potential leading indicator: Value = N * ARPU * Δ% + α * N * engagement_lift (choose α from historical conversion of engagement→revenue).

2) Bayesian/posterior simulation + Type S/M- Use observed point estimates and SEs (from experiment) to form normal posteriors for each metric (or t if small n). Simulate many draws to compute: - Probability revenue truly decreased P(ΔRevenue < 0) - Probability sign error for engagement (very small here) and for revenue (Type S) - Type M: distribution of true effect conditional on significance to see exaggeration.

Example Python simulation:

python

import numpy as np
np.random.seed(1)
# inputs (replace with real SEs)
mu_rev, se_rev = -0.03, 0.018  # observed -3%, p~0.06
mu_eng, se_eng = 0.04, 0.015   # observed +4%, p~0.02
draws = 200000
rev = np.random.normal(mu_rev, se_rev, draws)
eng = np.random.normal(mu_eng, se_eng, draws)
# Type S: prob revenue <=0 (true negative)
prob_rev_down = (rev < 0).mean()
# Type M: expected true magnitude given observed significance (rev p<0.05)
prob_rev_sig = (rev < -1.96*se_rev).mean()
expected_mag_if_sig = rev[rev < -1.96*se_rev].mean()
prob_rev_down, prob_rev_sig, expected_mag_if_sig

3) Decision logic- If P(revenue decrease > threshold) is high (e.g., >70%) or expected dollar loss > business tolerance, do not roll out.- If revenue downturn probability moderate (30–70%) but engagement lift is valuable and correlated with future revenue: run targeted/staged rollout (e.g., 5–10% of users) while instrumenting downstream funnels, LTV, and retention for longer horizon.- If posterior shows high Type M risk (effect may be smaller than observed) — favor follow-up with larger sample or pre-specified secondary metrics.- If small absolute dollar risk and engagement strategically important, consider A/B/n where variant is tweaked to preserve revenue (e.g., reduce friction causing revenue loss).

4) Follow-ups & monitoring- Pre-register primary business metric (revenue per user) and power calculations to detect minimum detectable effect (MDE) aligned with business impact.- Staged rollout with early stopping rules, sequential monitoring (alpha spending), and sentinel metrics (conversion, checkout funnel) to abort quickly if revenue drops.- Report decisions with simulated expected value and downside risk (percentiles), and recommend remediation experiments if revenue drop persists (e.g., UI adjustments, targeted segments).

This approach quantifies trade-offs, accounts for uncertainty (Type S/M), and yields a defensible, risk-aware rollout plan.

MediumBehavioral

70 practiced

Tell me about a time you recommended stopping an experiment early due to safety concerns, negative guardrails, or clear futility. Describe the context, how you evaluated the evidence, how you communicated to stakeholders, and what processes you changed afterwards to reduce recurrence.

Sample Answer

Situation: At my previous company I owned an A/B experiment for a new onboarding flow that used progressive profiling and an aggressive email cadence. After a week and ~20% of planned traffic, product noticed a jump in user complaints and a decline in 7-day retention in the variant.

Task: My responsibility was to determine whether the experiment should continue and to advise stakeholders on risk.

Action:- I pulled realtime telemetry (CTR, retention, complaint rate, support tickets) and compared the variant to control using pre-registered metrics and sequential analysis (alpha-spending) to avoid peeking bias.- The variant had a statistically significant increase in complaint rate (risk ratio 1.8, p<0.01 under sequential test) and a clear negative lift in 7-day retention beyond our harm threshold (absolute drop >2 percentage points).- I escalated immediately: prepared a one-page brief with plots, confidence intervals from bootstrap resampling, and simulated impact at full scale to show expected user harm and cost.- Convened a 30-minute decision call with PM, engineering, legal, and customer support. I recommended pausing the experiment pending rollback because evidence met our pre-defined safety guardrails (complaint rate and retention).- After agreement, engineering rolled back the variant within hours; we monitored recovery.

Result:- Stopping prevented widespread dissatisfaction and reduced potential churn exposure. Support volume returned to baseline within 48 hours.- To prevent recurrence I led updates to experiment governance: we added explicit safety guardrails to experiment specs (absolute thresholds for complaints, latency, and retention), required interim analysis plans with sequential testing when peeking was likely, and implemented automated alerts in our dashboard that notify stakeholders when any safety metric crosses thresholds.- I also ran a short workshop with PMs and engineers on designing ethical, low-risk experiments.

Learning: Quantitative evidence + clear, pre-agreed safety rules enables fast, cross-functional decisions and preserves user trust.

MediumTechnical

57 practiced

Compare Bayesian A/B testing to frequentist p-value-based testing for a company that runs many short experiments. Discuss pros and cons of each approach and operational changes needed to adopt Bayesian decision rules (e.g., posterior probability thresholds instead of p-values).

Sample Answer

Bayesian vs Frequentist for many short experiments — concise comparison and operational implications.

Definitions & intuition:- Frequentist (p-values): Tests null hypothesis; p-value = probability of observing data as extreme assuming no effect. Decisions use fixed alpha (e.g., 0.05).- Bayesian: Computes posterior distribution for treatment effect given prior + data. Decision uses posterior probability (e.g., P(effect > 0) = 0.95) or expected loss.

Pros — Frequentist:- Well-understood, simple thresholds, widespread tooling and regulatory acceptance.- If experiments run with pre-specified sample sizes and strict stopping rules, error rates (Type I/II) are controlled.

Pros — Bayesian:- Naturally handles sequential looks (no alpha inflation if decision rule uses posterior directly).- Provides intuitive statements stakeholders like: “90% probability treatment is better by >1%.”- Better with small/short experiments via informative priors; yields full distribution for business-relevant metrics (e.g., lift, ROI).

Cons — Frequentist:- P-values misinterpreted; optional stopping and multiple testing inflate false positives unless corrected (Bonferroni, FDR, sequential tests).- Hard to incorporate prior business knowledge.

Cons — Bayesian:- Requires priors (can be subjective); poor priors bias results.- Less familiar to stakeholders; needs careful calibration and simulation to guarantee business-level error rates.

Operational changes to adopt Bayesian decision rules:1. Define decision criteria: e.g., stop & ship if P(lift > MROI) ≥ 0.95, or compute expected value of shipping. Make thresholds explicit and linked to business impact.2. Choose and document priors: use weakly informative or historical-data-derived priors; run sensitivity analyses.3. Simulation & calibration: run Monte Carlo to estimate operating characteristics (false positive/negative rates) under realistic traffic and multiple experiments.4. Reporting & tooling: update dashboards to show posterior distributions, credible intervals, and probability of meeting business thresholds instead of p-values.5. Stopping & multiplicity: allow continuous monitoring but simulate to ensure acceptable error rates for your decision rule; adopt hierarchical models to share strength across many short experiments and control FDR.6. Governance & education: train PMs and stakeholders on interpreting posterior probabilities and expected loss; codify experiment lifecycle (pre-registration, metrics, priors).7. Infrastructure: implement Bayesian analysis pipelines (PyMC, Stan, TensorFlow Probability) with automated reporting and reproducible pipelines.

Recommendation: For an environment with many short, sequential experiments, Bayesian approaches (with calibrated priors and simulation-backed thresholds) often give more actionable, interpretable decisions and tolerate flexible stopping. However, adopt gradually: parallel-run Bayesian and frequentist analyses, build trust via calibration, and document decision rules tied to business value.

EasyTechnical

66 practiced

What is sample ratio mismatch (SRM)? Describe a simple statistical test to detect SRM in an experiment, list common causes of SRM, and outline immediate remediation steps if you detect SRM before starting analysis.

EasyTechnical

74 practiced

Why is randomization necessary in experiments? Describe SUTVA (stable unit treatment value assumption) and give an example of interference where SUTVA may be violated (for example, social features such as referrals). Suggest one experimental design approach to mitigate interference.

Unlock Full Question Bank

Get access to hundreds of Hypothesis and Test Planning interview questions and detailed answers.

Join thousands of developers preparing for their dream job.