Statistical Inference and Hypothesis Testing Questions
Reasoning about uncertainty in data and drawing formal conclusions from samples. Covers probability rules and common distributions, the Central Limit Theorem, sampling, standard error, confidence intervals, and Bayesian reasoning, together with the significance-testing framework: null and alternative hypotheses, p-values, statistical power, Type I and Type II errors, effect sizes, and choosing the right test (t-test, chi-square, non-parametric). Emphasizes correctly interpreting statistical results and avoiding common misreadings of significance in business and product contexts rather than memorizing formulas.
In a linear regression model predicting revenue per user, explain the difference between a confidence interval for the mean response at a particular x* and a prediction interval for a new observation at x*. Provide formulas (assuming homoscedastic Gaussian errors) and describe how interval widths change with sample size and residual variance.
Sample Answer
Direct answer
A confidence interval at x∗ captures uncertainty about the average outcome for that x value (a population quantity); a prediction interval at x∗ captures uncertainty about one new individual observation at that x value. The prediction interval is always wider because it has to account for both the uncertainty in estimating the regression line and the irreducible noise around that line, while the confidence interval only accounts for the first source.
Structured elaboration
Setup. Simple linear regression y=β0+β1x+ε, ε∼N(0,σ2) i.i.d. (homoscedastic Gaussian errors). Let y^(x∗)=β^0+β^1x∗, s2=RSS/(n−2), and Sxx=∑i(xi−xˉ)2.
Confidence interval for the mean response E[y∣x∗]:
y^(x∗)±tn−2,α/2sn1+Sxx(x∗−xˉ)2This only reflects how precisely you've pinned down the regression line itself, since E[y∣x∗] is a fixed number, not a random draw.
Prediction interval for a new observation ynew at x∗:
y^(x∗)±tn−2,α/2s1+n1+Sxx(x∗−xˉ)2The extra "1" inside the square root is exactly σ2 worth of individual-observation noise that a single future draw carries and that never goes away, regardless of how much data you have.
How the widths change:
| Factor | Confidence interval | Prediction interval |
|---|---|---|
| Sample size n→∞ | Width →0 | Width → a floor set by 2zα/2σ (never zero) |
| Residual variance s2 up | Wider (linear in s) | Wider, and dominates once 1≫1/n |
| x∗ far from xˉ | Wider (via (x∗−xˉ)2/Sxx) | Wider, but the effect is proportionally smaller since the "1" term dominates |
The practical takeaway: growing the sample only tightens the CI meaningfully; it barely tightens the PI, because the PI's floor is set by σ itself, not by how well you know β0,β1.
Worked example
Simulated revenue-per-user regression (pinned seed, n=40):
import numpy as np
from scipy import stats
rng = np.random.default_rng(7)
n = 40
x = rng.uniform(1, 10, n)
y = 5.0 + 3.0 * x + rng.normal(0, 4.0, n) # true b0=5, b1=3, sigma=4
xbar, Sxx = x.mean(), np.sum((x - x.mean())**2)
b1 = np.sum((x - xbar) * (y - y.mean())) / Sxx
b0 = y.mean() - b1 * xbar
resid = y - (b0 + b1 * x)
s = np.sqrt(np.sum(resid**2) / (n - 2))
xstar = 8.0
yhat = b0 + b1 * xstar
tcrit = stats.t.ppf(0.975, n - 2)
se_mean = s * np.sqrt(1/n + (xstar - xbar)**2 / Sxx)
se_pred = s * np.sqrt(1 + 1/n + (xstar - xbar)**2 / Sxx)
print(f"fitted: b0={b0:.3f}, b1={b1:.3f}, s={s:.3f} (xbar={xbar:.3f}, Sxx={Sxx:.2f})")
print(f"yhat(x*=8.0) = {yhat:.3f}, t_crit(df={n-2}, .975) = {tcrit:.3f}")
print(f"95% CI for mean response: [{yhat-tcrit*se_mean:.3f}, {yhat+tcrit*se_mean:.3f}] half-width = {tcrit*se_mean:.3f}")
print(f"95% PI for new observation: [{yhat-tcrit*se_pred:.3f}, {yhat+tcrit*se_pred:.3f}] half-width = {tcrit*se_pred:.3f}")
Output:
fitted: b0=2.027, b1=3.569, s=3.218 (xbar=5.238, Sxx=282.42)
yhat(x*=8.0) = 30.577, t_crit(df=38, .975) = 2.024
95% CI for mean response: [29.091, 32.063] half-width = 1.486
95% PI for new observation: [23.895, 37.259] half-width = 6.682
The prediction interval is about 4.5 times wider than the confidence interval at the same x∗, entirely because of the extra "1" term, since both intervals use the same y^(x∗), s, tn−2, n, and (x∗−xˉ)2/Sxx.
Trade-offs & pitfalls
The most common mistake in practice is reporting a confidence interval when the actual question is about a single future user (e.g. "what revenue should we expect from a new customer with these attributes") - that calls for a prediction interval, and using the CI there systematically understates risk. The reverse mistake, reporting a PI when summarizing the average effect across a segment, needlessly widens the story and can make a real, precisely-estimated average look uncertain. Both formulas also assume homoscedastic Gaussian errors; if the residual spread grows with x (heteroscedasticity) or is skewed, these closed-form intervals are wrong in both center and width, and a weighted-least-squares model or a bootstrap-based interval that estimates residual spread locally is the more defensible choice.
Under what conditions can a Poisson distribution approximate a Binomial distribution? Provide the mathematical rule (in terms of n and p) and give a practical example (for instance, defects per kilometer or rare server errors) with numeric justification.
Sample Answer
Direct answer
A Binomial(n,p) distribution approximates a Poisson(λ) distribution when n is large, p is small, and their product λ=np stays moderate and roughly fixed as n grows. Practically: n in the hundreds or more, p under about 0.05, with np somewhere in the single digits to low tens works well; this is exactly the regime of "rare events counted over many independent opportunities," like defects per kilometer of cable or errors across many server requests.
Structured elaboration
The mathematical rule
As n→∞, p→0, with np=λ held fixed,(kn)pk(1−p)n−k⟶k!λke−λIntuition for why: with p small, the chance of two or more "successes" landing on the same trial-count window becomes negligible, so the combinatorial factor (kn) and the (1−p)n−k term simplify. In particular, (1−p)n≈e−np=e−λ for small p, which is the Poisson kernel's leading term.
Practical thresholds
| Guideline | Typical cutoff |
|---|---|
| Sample size | n≳50-100 |
| Success probability | p≲0.05 |
| Mean (best single check) | np≲5-10 for a tight approximation |
The mean check (np moderate) is the most reliable single diagnostic, since n and p can each individually look "extreme" while their product is not in the rare-event regime.
Worked example
Rare server errors: 10,000 requests in an hour, each independently erroring with probability p=0.0003.
n=10,000,p=0.0003,λ=np=3Compare the probability of zero errors under each model:
PBinomial(X=0)=(1−p)n=(0.9997)10000 PPoisson(X=0)=e−λ=e−3import math
n, p = 10000, 0.0003
lam = n*p # 3.0 (up to floating-point noise)
p0_binom = (1-p)**n # 0.04976466474747146
p0_pois = math.exp(-lam) # 0.049787068367863965
abs(p0_binom - p0_pois) # 2.24e-05
Both models give ≈0.0498 for the probability of a clean hour, agreeing to four significant figures despite one requiring the exact binomial formula and the other only needing λ=3. For an applied model of rare server errors or defect counts, this means you can drop the exact binomial machinery (which needs both n and p tracked separately) and work directly with the single-parameter Poisson(λ), which is simpler to fit, simpler to reason about (mean = variance = λ), and simpler to compose across time windows (Poisson counts over disjoint intervals just add their λ's).
Trade-offs & pitfalls
- The approximation breaks down when p isn't actually small, even if n is large - e.g. a coin-flip-like process with p=0.4 over 1000 trials is nowhere near Poisson-shaped regardless of how large n is; check p, not just n.
- It also breaks down at small n even with small p, if that makes λ=np too small to be a meaningful count (e.g., n=20,p=0.01→λ=0.2) - the approximation quality depends on the full regime, not any single threshold in isolation.
- Real event counts are often overdispersed relative to Poisson. Poisson forces mean = variance; if observed variance is substantially larger than the mean (common when the underlying rate itself varies, e.g., error rates spike during incidents rather than staying constant), a negative binomial model fits better and a plain Poisson will understate uncertainty.
- Always validate numerically for decisions that matter, the way the worked example does above, rather than trusting the heuristic thresholds blindly - the thresholds are guidelines, not guarantees, and the actual approximation error is cheap to check directly.
You're analyzing thousands of features for correlation with churn. Design a practical pipeline to conduct high-dimensional hypothesis testing that scales, controls the false discovery rate, and produces an actionable ranked list of candidate features for downstream validation. Address p-value computation, multiple-testing adjustment, dependence between tests, and reporting conventions.
Sample Answer
Direct answer
Compute a p-value per feature with a test suited to its type, adjust the whole set for multiple testing with a false-discovery-rate procedure (Benjamini-Hochberg by default, Benjamini-Yekutieli if dependence is severe and arbitrary), and output a ranked table of feature, effect size, adjusted q-value, and a stability flag so downstream reviewers can prioritize which candidates to validate experimentally. The core design tension is that thousands of correlated tests break the independence assumption behind naive FDR control, so the pipeline needs an explicit step to handle dependence rather than ignoring it.
Structured elaboration
Pipeline steps
- Data prep. Standardize feature types, handle missingness and rare categories, record the sample size backing each test (tests with wildly different N need to be flagged, since power varies a lot across them).
- Per-feature test selection. Logistic regression coefficient (Wald z-test) for continuous or binary predictors with relevant covariates included; chi-square or Fisher's exact for small-count categorical features; rank-sum test as a nonparametric fallback when model assumptions fail badly.
- P-value computation. Prefer the fast asymptotic p-value from the GLM fit. For features where sample size is small or independence is questionable, fall back to a permutation-based null: shuffle the churn label within relevant strata (to preserve covariate structure) and compute the empirical p-value from the permutation distribution.
- Multiple-testing adjustment.
- Dependence handling.
- Ranking and reporting.
Multiple-testing adjustment (step 4)
Benjamini-Hochberg (BH) controls the expected false discovery rate under independence or positive dependence (technically PRDS - positive regression dependency: roughly, no test's result is negatively correlated with another's, so a discovery on one feature never makes a discovery on a different feature more suspicious), which covers most feature-vs-label correlation setups reasonably well:
reject H(1),…,H(k) where k=max{i:p(i)≤miq}with p(1)≤⋯≤p(m) the sorted p-values and q the target FDR level.
Dependence handling (step 5)
- If dependence is arbitrary (not PRDS), switch to Benjamini-Yekutieli, which divides the threshold by ∑i=1m1/i - more conservative but valid under any dependence structure.
- Cluster correlated features (hierarchical clustering on correlation or mutual information) and report one representative per cluster rather than every member, to avoid a ranked list dominated by ten near-duplicate correlated features.
- Complement with a sparse multivariate model: LASSO with stability selection (refit an L1-penalized regression on many bootstrap resamples of the data, and treat the fraction of resamples in which a feature's coefficient survives at nonzero as its stability score) as a second signal - features that are both univariately significant and have a high stability score across resamples are the strongest candidates.
Ranking and reporting (step 6)
Output table: feature name, direction and standardized effect size (odds ratio or Cohen's d), raw p-value, BH q-value, sample size, correlation-cluster ID, stability score (fraction of resamples selecting the feature), recommended action.
Worked example
Twelve features tested for churn correlation, sorted p-values, target FDR q=0.10:
| Rank i | p(i) | Threshold miq | Reject? |
|---|---|---|---|
| 1 | 0.0009 | 0.0083 | yes |
| 2 | 0.0031 | 0.0167 | yes |
| 3 | 0.0084 | 0.0250 | yes |
| 4 | 0.0110 | 0.0333 | yes |
| 5 | 0.0190 | 0.0417 | yes |
| 6 | 0.0270 | 0.0500 | yes |
| 7 | 0.0410 | 0.0583 | yes |
| 8 | 0.0580 | 0.0667 | yes |
| 9 | 0.1200 | 0.0750 | no |
| 10 | 0.2600 | 0.0833 | no |
| 11 | 0.4400 | 0.0917 | no |
| 12 | 0.8100 | 0.1000 | no |
import numpy as np
p = np.array([0.0009,0.0031,0.0084,0.011,0.019,0.027,0.041,0.058,0.12,0.26,0.44,0.81])
m, q = len(p), 0.10
order = np.argsort(p)
p_sorted = p[order]
ranks = np.arange(1, m+1)
below = p_sorted <= ranks/m*q
k_max = np.max(np.where(below)[0]) + 1 if below.any() else 0
# k_max = 8, cutoff p_(8) = 0.058 -> 8 of 12 features rejected at FDR 10%
The largest i where p(i)≤miq is i=8 (since 0.058≤0.0667, while rank 9 fails: 0.12>0.075). BH's rule is to take this largest passing rank as the cutoff and reject every hypothesis at or below it - ranks 1 through 8 here - an 8/12 discovery rate at 10% expected FDR. In general the cutoff rank is not simply "the first rank that fails"; it is the largest rank that still satisfies its own threshold, so a single non-monotonic dip partway through the sorted list does not necessarily stop the procedure early.
Trade-offs & pitfalls
- BH under strong negative dependence is not guaranteed valid. If features are engineered to be strongly negatively correlated (rare in practice for churn features, but possible with ratio features), fall back to Benjamini-Yekutieli or a permutation-based q-value.
- Statistical significance at scale is easy; practical significance is the actual bottleneck. With enough rows, thousands of features will show "significant" but tiny effect sizes. Always sort the final list by effect size within the FDR-passing set, not purely by p-value.
- Correlated feature clusters inflate apparent discoveries. Ten near-duplicate engineered features (e.g., 7-day, 14-day, 30-day versions of the same signal) will often all pass together; the clustering step exists specifically to avoid handing downstream teams ten redundant "validated" features.
- Permutation nulls are expensive at scale. Reserve them for features that fail the asymptotic-test assumptions rather than running them universally; screen with the fast GLM p-value first, then permute only the borderline or assumption-violating cases.
Design a testing pipeline for a platform that runs thousands of experiments and reports hundreds of metrics per experiment. The pipeline must control false discoveries while remaining interpretable to product teams. Propose statistical procedures, the data infrastructure needed, and reporting conventions. Discuss computational scalability and monitoring strategies.
Sample Answer
Direct answer
At the scale of thousands of experiments and hundreds of metrics each, the design has to separate metrics into tiers by decision importance (a handful of primary, ship/no-ship metrics per experiment vs. many secondary and exploratory ones), apply a stricter false discovery control to the important tier and a looser, FDR-based control to the exploratory tier, and back all of it with an immutable experiment registry so results are reproducible and auditable. Interpretability comes from tiering and clear reporting conventions, not from picking one universal statistical procedure for every metric.
Structured elaboration
Pipeline flow
flowchart LR
A[Experiment registry: pre-registered plan, metrics, hypotheses] --> B[Metric pipeline: batch-compute per experiment]
B --> C{Metric tier}
C -->|Primary, ~1-3 per experiment| D[Per-experiment alpha, no correction needed within this small set]
C -->|Secondary/exploratory, hundreds| E[Benjamini-Hochberg within the metric family]
D --> F[Results store: adjusted + unadjusted, effect size, confidence interval]
E --> F
F --> G[Dashboard: verdict, q-value, effect size by segment]
F --> H[Continuous monitoring: realized FDR, power, registry-plan mismatches]
Statistical procedures
- Primary metrics (the one to three metrics that actually decide ship/no-ship for a given experiment) are tested at the experiment's own pre-registered α, since there are only a few of them per experiment and they were chosen before the data was seen; multiplicity correction here is unnecessary if the primary set stays small and is genuinely fixed in advance.
- Secondary and exploratory metrics (the hundreds of additional metrics reported per experiment) are controlled with Benjamini-Hochberg (BH) within that family, since the goal there is to surface promising leads for follow-up, not to make a single high-stakes decision, and FDR control keeps a bounded, known fraction of those leads spurious.
- Cross-experiment aggregation (e.g. "how many experiments this quarter showed a significant lift on metric X") is its own separate family and needs its own correction; don't let a metric's significance in one experiment's local family imply anything about a cross-experiment claim without re-testing at that level.
Data infrastructure
- An experiment registry that stores the hypothesis, the declared primary and secondary metrics, the randomization scheme, and the analysis plan before the experiment starts, and treats that record as immutable once registered.
- Raw event logging (not just pre-aggregated metrics), so any later question about what actually happened can be answered by recomputation rather than trusting a snapshot.
- Reproducible, versioned pipelines (a workflow orchestrator with pinned code, data snapshots, and seeds) so a reported result can be regenerated exactly.
Reporting conventions
- One decision-relevant verdict per primary metric per experiment (ship / hold / inconclusive), always paired with its effect size and confidence interval, not the p-value alone.
- Secondary and exploratory metrics reported with their BH-adjusted q-value alongside the raw p-value, explicitly labeled as hypothesis-generating rather than confirmatory.
- A short plain-language note translating each significant primary result into business terms, so the interpretability requirement is met at the reporting layer, not just the statistical one.
Worked example
A single experiment reports 12 exploratory secondary metrics with unadjusted p-values: 0.0009,0.006,0.041,0.09,0.14,0.19,0.27,0.33,0.41,0.58,0.71,0.85. Comparing Bonferroni (family α=0.05) against BH (target FDR q=0.10):
| Procedure | Threshold rule | Rejections |
|---|---|---|
| Bonferroni | p≤0.05/12=0.00417 | 1 (p=0.0009 only) |
| Benjamini-Hochberg | largest k with p(k)≤(k/12)(0.10) | 2 (p=0.0009 and p=0.006) |
(both computed directly from the sorted p-values and the two threshold rules above)
At platform scale, this difference compounds: across thousands of experiments each reporting hundreds of exploratory metrics, a flat Bonferroni-style correction across everything would leave the exploratory tier almost powerless, while a per-experiment BH pass on just that tier keeps a known, bounded false-discovery share without needing an ever-shrinking per-test threshold as the platform grows.
Computational scalability
- Metric computation is embarrassingly parallel per experiment and per metric; pre-aggregate with a batch or streaming pipeline, and only run the multiplicity correction step (cheap, since it's just a sort and threshold comparison) once all p-values for a family are in.
- The correction step itself is O(mlogm) per family (dominated by sorting the p-values), trivial compared to the cost of computing the underlying statistics; scalability bottlenecks are almost always in the metric computation and data pipeline, not the multiplicity correction.
- Reserve heavier machinery (empirical-Bayes shrinkage, hierarchical pooling across similar experiments) for cases where the added interpretability or power is worth the added modeling and validation cost; default to plain BH within each experiment's exploratory family as the baseline that always works.
Monitoring strategies
- Backtest the pipeline on historical experiments with known outcomes (including synthetic A/A tests, where the two arms are identical) to confirm the realized false-positive rate matches what the correction claims.
- Continuously track the realized discovery rate over time; a sudden jump is a signal of either a real shift in how many effects are real, or a broken pipeline inflating false positives.
- Alert on registry-plan mismatches (a primary metric changed after the experiment started, or a metric reported that wasn't pre-registered), since that's the most common way multiplicity control gets silently defeated.
Trade-offs & pitfalls
- Applying a single strict correction (Bonferroni across everything, primary and secondary combined) is the safest failure mode but starves the exploratory tier of power at scale; applying no correction anywhere invites exactly the selective-reporting problem multiplicity control exists to prevent. The tiered approach is a deliberate middle ground, not a shortcut.
- Letting a metric drift from "exploratory" to "primary" after seeing a promising result defeats the whole design; the tier has to be fixed by the pre-registration, not re-assigned post hoc.
- BH's guarantee weakens under strong negative dependence between metrics; correlated metrics (which are common when many metrics are derived from the same few underlying events) may need a dependence-aware variant or an empirical-Bayes model rather than plain BH.
- None of this replaces basic experiment hygiene: a broken randomization or a biased sample invalidates every downstream statistical control, no matter how carefully the multiplicity correction is designed.
Design an A/B testing platform for an online service that supports continuous monitoring with optional stopping while controlling the Type I error rate at 0.05. Describe the statistical approaches you would support, the logging and data infrastructure needed, how to scale to millions of users and thousands of concurrent experiments, and trade-offs between statistical guarantees and engineering complexity.
Sample Answer
Direct answer
Support continuous monitoring by replacing the fixed-sample p-value with a sequentially valid statistic (alpha-spending group-sequential tests, MaxSPRT for streaming binary outcomes, or e-values, also called always-valid p-values: a running statistic that stays a valid bound on the false-positive rate no matter when, or how many times, you look at it, unlike a p-value computed for one pre-planned sample size), and build the data path as a streaming pipeline that maintains compact running summaries per experiment rather than reprocessing raw events on every check. The core trade-off is that the more flexible the statistical guarantee (arbitrary stopping, arbitrary look times), the more the engineering has to guarantee correct incremental state tracking rather than relying on a simple batch recompute.
Structured elaboration
Statistical approaches to support
| Method | Stopping flexibility | Notes |
|---|---|---|
| Group-sequential (O'Brien-Fleming / Pocock boundaries) | Fixed, pre-planned interim looks | Simple to explain; requires committing to a look schedule up front |
| Alpha-spending (Lan-DeMets) | Looks at arbitrary, unplanned times | Spends a total α budget across looks via a spending function; most flexible frequentist option |
Mechanism note: O'Brien-Fleming and Pocock construct their boundaries by allocating a shrinking (O'Brien-Fleming) or uniform (Pocock) share of the total significance budget across a small, pre-specified number of interim looks, so the same overall Type I error rate as a single fixed-sample test is preserved even though the data get checked more than once.
| MaxSPRT | Continuous monitoring of a binary/count endpoint | Closed-form likelihood-ratio boundary; well suited to conversion-style metrics |
| E-values / always-valid p-values (test martingales) | Fully arbitrary optional stopping, including adversarial | Type I control holds for any stopping rule (Ville's inequality); typically the least engineering-coordination overhead per look, since no pre-committed schedule is needed |
| Bayesian sequential (posterior probability / Bayes factor thresholds) | Arbitrary stopping, interpretability trade-off | Needs prior management and, if a frequentist-style guarantee is required, calibration back to a Type I rate |
Recommended default: alpha-spending plus MaxSPRT for the majority of standard conversion-metric experiments (familiar to most analysts, closed-form), with e-values available as an opt-in for teams that need genuinely unplanned, high-frequency monitoring without coordinating a look schedule.
System architecture
flowchart LR
U["User event"] --> K["Event stream"]
K --> AGG["Streaming aggregator"]
AGG --> STORE[("Time-partitioned OLAP store")]
STORE --> SEQ["Sequential test engine"]
SEQ --> DASH["Experiment dashboard"]
SEQ --> ALERT["Stopping-rule alert"]
CAT[("Experiment catalog")] --> SEQ
AGG --> AUDIT[("Audit log")]
- Event stream: every assignment and conversion event carries
experiment_id,user_id,variant,timestamp, and a schema version, written idempotently (so a retried event doesn't double-count). - Streaming aggregator: maintains, per experiment-variant cell, only the sufficient statistics the chosen sequential method needs (counts, running likelihood ratio, or running e-value) - not raw events. This keeps the state that needs re-reading on every look small and bounded regardless of how much traffic the experiment has accumulated.
- Experiment catalog: an immutable manifest per experiment recording the chosen method, its parameters (spending function, priors, minimum detectable effect), and a fixed random seed for any resampling-based component, so any later recomputation is reproducible.
- Sequential test engine: reads the current sufficient statistics plus the manifest, evaluates the stopping rule, and either emits "keep going," "stop, ship," or "stop, no effect."
- Audit log: every evaluation (not just terminal decisions) is logged, so a disputed or unexpected stopping decision can be reconstructed exactly from the manifest and the statistic history.
Scaling to millions of users and thousands of concurrent experiments
- Shard aggregation by
experiment_idso experiments don't contend with each other; within an experiment, pre-aggregate at a coarse time bucket (e.g., per-minute) rather than tracking every individual event in the hot path. - Because the sequential methods above only need running sufficient statistics (a handful of numbers per variant cell), the state per experiment stays small and bounded even as traffic grows - the aggregation problem scales with the number of experiments, not the number of events, once past the streaming aggregation layer.
- Serve dashboards and "should I stop" checks from the pre-aggregated OLAP layer rather than re-scanning raw events, so a check on one experiment doesn't compete for the same resources as ingesting new events for another.
Trade-offs between statistical guarantees and engineering complexity
| Approach | Statistical flexibility | Engineering cost |
|---|---|---|
| Fixed-sample (no monitoring) | None - must wait for pre-committed N | Lowest: simple batch job |
| Group-sequential | Limited to planned looks | Moderate: must track look count and enforce the schedule |
| Alpha-spending / MaxSPRT | Arbitrary look timing | Moderate-high: must track cumulative spent alpha or LR state correctly, no double-spending |
| E-values / always-valid | Fully arbitrary, adversarial-safe stopping | Lowest ongoing coordination burden per look (no schedule to enforce), but requires getting the martingale construction right up front, and is a less familiar tool for most analysts to reason about and communicate |
Worked example
Illustrating the alpha-spending idea with a simple linear spending function f(t)=α⋅t over the planned exposure fraction t∈[0,1], and a target total α=0.05: at 25% of planned exposure the cumulative alpha budget spent is 0.05×0.25=0.0125; at 50% it's 0.025; at 100% the full 0.05 has been allocated. Each interim look consumes only the incremental alpha available since the last look (not the cumulative total again), which is what prevents repeated looks from inflating the overall Type I rate above 0.05. This is a design-time property of the spending function, not a claim about any specific running experiment's realized numbers, since actual per-look alpha consumption depends on the observed test statistic at each look, not just elapsed time.
Trade-offs & pitfalls
- Multiple metrics and multiple concurrent experiments each need their own multiple-testing correction, on top of the sequential-monitoring correction. Sequential validity controls Type I error within one metric's repeated looks; it does not automatically control the false-discovery rate across multiple metrics or multiple simultaneous experiments touching overlapping users.
- Overlapping user populations across experiments are a data-integrity risk, not just a statistical one. Without deliberate traffic allocation or explicit interaction modeling, two experiments sharing users can contaminate each other's estimates; track exposure sets explicitly in the experiment catalog.
- E-values are the most flexible but the least familiar. Analysts and stakeholders need a translation layer ("this e-value crossing 1/α means the same thing as p < α, but validly under any stopping rule") or the tool will be distrusted regardless of its correctness.
- Reproducibility depends entirely on the manifest. If the spending function, priors, or seeds aren't pinned and versioned per experiment at launch time, a later audit of "why did this stop when it did" cannot be answered even with perfect logs of the raw events.
Unlock Full Question Bank
Get access to all Statistical Inference and Hypothesis Testing interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.