Statistical Inference and Hypothesis Testing Questions

Direct answer

P-hacking is any practice, deliberate or not, of searching over many possible analyses, metrics, subgroups, cutoffs, models, and selectively reporting the one that crosses a significance threshold. It breaks the guarantee that a p-value under 0.05 means "only a 5% chance of seeing this by chance," because that guarantee was only ever a promise about one pre-specified test, not about the best of many attempts.

Structured elaboration

How it happens in practice

• Testing many metrics (conversion, revenue, retention, time-on-page) and reporting only whichever one turned out significant.
• Slicing the same data into many subgroups (by country, device, cohort, day of week) until one slice looks significant, then presenting that slice as the headline finding.
• Peeking at results repeatedly and stopping the moment p < 0.05 appears, sometimes called optional stopping.
• Trying several reasonable-looking exclusion or data-cleaning rules and keeping the one that produces significance.

None of these require bad intent. A team that genuinely believes it's just being thorough by checking 15 metrics is p-hacking exactly as much as someone doing it cynically; the inflation comes from the number of looks, not the motive.

Why this inflates the true false-positive rate above the nominal alpha

Each individual test, run and interpreted in isolation, does have a 5% false-positive rate under its own null. The problem is the family-wise error rate: the probability that at least one of k independent tests falsely rejects its null, even when every single null is true, is

because the probability that all k tests correctly fail to reject is $(1-\alpha)^k$ under independence, and "at least one rejects" is the complement of that.

Worked example

With alpha = 0.05 per test:

With just 5 metrics or subgroups tested independently at alpha = 0.05 each, there's already a 22.6% chance that something looks significant purely by chance, not 5%. At 20 tests, for example one significance test per subgroup across 4 countries and 5 device types, the chance of at least one false "win" is over 64%, meaning it's more likely than not that something will look reportable even if nothing real is happening. Reporting only the metric or slice that crossed the threshold, while staying silent about the others that didn't, is p-hacking: it turns a controlled 5% risk into an uncontrolled 22 to 64% risk while still labeling the result "p < 0.05."

Trade-offs and pitfalls

• Pre-register the primary metric and the analysis plan, including which subgroups, if any, will be tested, before looking at data. Anything examined afterward is exploratory, not confirmatory, and should be labeled that way in the report.
• If multiple metrics or subgroups genuinely need testing, control for it: Bonferroni (alpha divided by k per test) is simple but conservative; Benjamini-Hochberg FDR control is the more common choice when running many tests and the goal is to bound the expected proportion of false discoveries rather than the probability of any single false discovery.
• A subtle wrong turn: applying the multiplicity correction only across the metrics that ended up being reported, rather than across everything that was actually looked at. That undercounts k and still overstates significance.
• The fix is not "never look at subgroups." Exploratory subgroup analysis is legitimate and valuable for generating hypotheses; the failure mode is presenting an exploratory finding with the same confidence and framing as a pre-registered confirmatory one.

Direct answer

A production experimentation platform needs six cooperating components: deterministic randomization/assignment, reliable event instrumentation, a metrics computation pipeline, an experiment registry, sequential-analysis support for safe early stopping, and alerting feeding a dashboard. The statistical safeguards (consistent hashing, pre-registration, alpha-spending (giving each early look at the results only a small slice of the total false-positive budget, saving most of it for the planned end, instead of letting every look risk the full 5%), multiplicity correction) matter as much as the engineering scalability, because a platform that scales but lets teams p-hack their way to significance is worse than no platform at all.

Structured elaboration

flowchart TD
    A[Client SDKs: web, mobile, server] --> B[Assignment Service: hashed bucketing]
    B --> C[Experiment Registry: definitions, owners, status]
    A --> D[Event Ingestion: Kafka]
    D --> E[Metrics Pipeline: stream and batch aggregation]
    E --> F[Metrics Store: OLAP warehouse]
    F --> G[Sequential Analysis Engine: alpha-spending, always-valid p-values]
    G --> H[Alerting: guardrail breach, drift, imbalance]
    F --> I[Experimenter Dashboard]
    C --> I
    G --> I
    H --> I

1) Randomization / assignment. Requirement: the same user gets the same bucket every time, across services, without a database lookup on the hot path. Implementation: consistent hashing of (experiment_id, user_id) into [0, 1), compared against the bucket boundaries defined in the registry. Scalability: this makes assignment a pure function, cacheable and computable client-side or at the edge. Statistical safeguard: log the assignment (and its salt/version) with every downstream event so re-computation is auditable, and run an automatic sample-ratio-mismatch (SRM) check (chi-square goodness-of-fit against the intended split) as a standing guardrail on every live experiment.

2) Instrumentation & event collection. Requirement: high-fidelity, schema-validated events with the experiment variant attached. Implementation: client/server SDKs emit to a durable log (Kafka), with a schema registry enforcing event contracts. Scalability: tiered storage (hot for recent, cold for historical replays). Safeguard: automatic drift detection on event volume and shape (a sudden 30% drop in events for one arm is more likely a broken SDK than a real effect).

3) Metrics computation pipeline. Requirement: compute the pre-registered metric set (funnel rates, revenue, latency) per experiment per day, both near-real-time (streaming) and definitively (nightly batch reconciliation). Scalability: stream aggregation (Flink/Spark Structured Streaming) feeding a columnar OLAP store, materialized views for dashboard reads. Safeguard: every metric has a single canonical definition owned by the registry, not redefined ad hoc per dashboard, which is what prevents the "we got a different p-value depending on which team computed it" failure mode.

4) Experiment registry. Requirement: canonical, versioned, immutable-once-started record of hypothesis, primary/secondary metrics, guardrails, start/end date, owner. Scalability: a metadata store (Postgres) behind a cached API with RBAC. Safeguard: registering the primary metric and analysis plan before the experiment starts, and disallowing edits to the primary metric mid-flight, is what makes the eventual p-value meaningful rather than the product of post-hoc metric shopping.

5) Sequential analysis support. Requirement: let teams check results early without inflating the false-positive rate, since naive repeated peeking at a fixed-horizon test can push the effective alpha well above 0.05. Implementation: group-sequential boundaries (O'Brien-Fleming, which spends very little alpha early and most near the planned end) or always-valid p-values via a mixture sequential probability ratio test (a sequential test that stays valid to check after every single observation, by averaging the likelihood ratio over a range of plausible effect sizes instead of committing to one fixed alternative up front, the way a classical fixed-horizon test does). Scalability: the engine only has to track a small running statistic per experiment (the cumulative log-likelihood ratio, or the current position against the O'Brien-Fleming boundary), so its compute and memory cost scale with the number of concurrent experiments, not with raw event volume, which is what keeps re-checking hundreds of experiments' boundaries every few minutes cheap. Safeguard: the dashboard shows the sequential boundary the current z-score must cross, not the naive fixed-sample threshold, so "stop early because it looks significant" cannot silently break the guarantee.

6) Alerting. Requirement: catch instrumentation failures and guardrail breaches (e.g. crash rate, latency) automatically, distinct from the primary-metric result. Implementation: streaming anomaly detection on SRM, event volume, and guardrail metrics, routed to the experiment owner. Safeguard: alert on effect size and statistical significance jointly, since with hundreds of concurrent experiments a significance-only threshold will fire constantly on noise.

7) Dashboard. Requirement: show the primary metric with its sequential-boundary-aware confidence interval, guardrail status, and SRM check, not just a binary "winning/losing" badge. Scalability: dashboard reads are served from the pre-aggregated OLAP layer the metrics pipeline already materializes, not a re-scan of raw events, so one experiment's dashboard load doesn't compete with another's even as the number of concurrently viewed experiments grows into the hundreds. Safeguard: surface corrected p-values for any secondary/segment cuts a user requests interactively, so ad hoc segment-slicing doesn't get reported with the same confidence as the pre-registered primary result.

Worked example

Running hundreds of concurrent experiments creates a multiplicity problem at the company level even when each individual test is well-designed. With 200 simultaneously live experiments, each independently tested at $\alpha = 0.05$, the expected number of false "wins" purely from chance, assuming for simplicity that every experiment's true effect is actually zero, is:

Ten false positives a month across the company is a very different risk profile than the 5% figure any single team sees on their own dashboard, and it is why the platform needs a company-level false-discovery view, not just a per-experiment significance check. This is also the concrete argument for the sequential-analysis engine: if teams are additionally peeking at results daily instead of waiting for a single fixed-horizon read, the effective per-experiment false-positive rate under naive repeated testing climbs above the nominal 5% with every additional look, which would push that company-wide false-win count higher than the naive calculation above. Group-sequential boundaries exist specifically to keep the realized rate at the stated 5% despite the peeking, by spending the alpha budget across looks instead of allowing each look its own full 5%.

Trade-offs & pitfalls

A common wrong turn is building the metrics pipeline first and bolting sequential-analysis safeguards on later; retrofitting always-valid inference onto a system that was built assuming fixed-horizon testing usually means every historical experiment's reported p-value is technically invalid under the peeking behavior that already happened, and there is no clean way to retroactively fix an already-reported result. Another pitfall is over-indexing on engineering scalability (more Kafka partitions, faster dashboards) while under-investing in the registry's guardrails; a platform that computes fast, wrong answers is more dangerous than a slow, correct one because speed increases the temptation to ship on early, unadjusted reads. Finally, a platform that only tracks per-experiment significance and has no company-level false-discovery dashboard will systematically over-trust its own "winning" experiments as the number of concurrent tests grows, exactly the failure mode the worked example above quantifies.

Direct answer

Ratio metrics like revenue per user are dangerous because they collapse two moving parts (a numerator and a denominator) into one number, and because revenue-like numerators are almost always heavy-tailed: a handful of large purchases can dominate the mean and make an experiment look like a clear win or loss when the typical user didn't change at all. The fix is to report a robust summary alongside the mean (median, trimmed mean, or a capped/winsorized version), define the denominator precisely and consistently across arms, and segment before concluding "the metric moved."

Structured elaboration

Pitfalls of ratio metrics

Robust alternatives and preprocessing

• Median or other percentiles (P50, P90): insensitive to a handful of extreme values, at the cost of ignoring exactly how much the tail moved.
• Trimmed or winsorized mean: cap or remove the top X% before averaging; keeps more information than the median while limiting whale influence.
• Log-transform (e.g. $\log(1 + \text{revenue})$): compresses the tail and stabilizes variance for a mean-based test; interpret results multiplicatively (geometric-mean-like) rather than additively.
• Fixed, pre-registered denominator: define "user" identically in both arms (e.g. "assigned at randomization," not "active during the window") so the denominator can't drift with treatment-induced behavior.

Analyzing and communicating results

1. Pre-specify the primary metric and its exact numerator/denominator/transform before the experiment runs.
2. Report mean, median, and a capped/winsorized mean together, plus a bootstrap CI for each (since ratio-metric sampling distributions are rarely well approximated by a simple closed-form CI).
3. Check whether the treatment changed the user mix (e.g. share of new vs. returning users); if so, present both the raw aggregate and a mix-adjusted (segment-weighted) comparison.
4. In the writeup, lead with whichever statistic best reflects the decision at hand, but always show all of them side by side so a stakeholder can see whether a "win" is broad-based or driven by a few large accounts.

Worked example

Simulated per-user revenue in a control arm (mostly small or zero purchases), and a treatment arm identical except for one added large ("whale") purchase, pinned seed:

import numpy as np
rng = np.random.default_rng(seed=99)

n = 1000
control = rng.choice([0, 0, 0, 5, 10, 20], size=n,
                      p=[0.5, 0.2, 0.1, 0.1, 0.06, 0.04]).astype(float)
treatment = control.copy()
rng.shuffle(treatment)
treatment[0] += 5000  # one large purchase added

mean_c, median_c = control.mean(), np.median(control)   # 2.195, 0.0
mean_t, median_t = treatment.mean(), np.median(treatment)  # 7.195, 0.0

lift_mean = (mean_t - mean_c) / mean_c * 100  # 227.8%

cap = np.percentile(control, 99)              # 20.0
treat_wins = np.clip(treatment, None, cap)
lift_winsorized = (treat_wins.mean() - mean_c) / mean_c * 100  # 0.7%

The mean shows a 227.8% lift in revenue per user (from 2.195 to 7.195), which looks like a spectacular win. The median lift is 0%: the typical user's behavior is completely unchanged. Winsorizing at the control arm's 99th percentile (capping treatment at 20) brings the "lift" down to 0.7%, essentially nothing. All three numbers come from the exact same underlying data; the only difference is one added $5,000 purchase in a sample of 1,000 users. Reporting the mean alone would tell a story about broad-based product improvement that the data doesn't actually support.

Trade-offs & pitfalls

• Don't pick whichever statistic tells the story you want. Report mean, median, and a robust variant together as standard practice, not as a reaction to a surprising result.
• Winsorizing/trimming thresholds should be set from a pre-specified rule (e.g. control-arm percentile), not tuned after seeing the treatment data, or you've reintroduced exactly the kind of post-hoc flexibility that inflates false positives.
• Log-transforms complicate interpretation for stakeholders; a "12% lift in log-revenue" needs translation into a business-meaningful statement (roughly a 12% multiplicative lift in revenue, not an additive one), and zero values break a plain log transform (hence $\log(1+x)$).
• If most of a real lift genuinely comes from a small number of high-value users, that's not automatically a false positive. It changes what to do next: consider whether that's the intended (e.g. a premium-tier feature) or unintended (e.g. one enterprise account skewing an SMB experiment) outcome, rather than assuming the whale should always be excluded.
• Segment-level Simpson's paradox checks matter as much as the tail-robustness fixes. A metric can be tail-robust and still misleading if the treatment changed who's in the denominator.

Quick answer

Cohen's $d$ is the standardized mean difference: $d = (\bar x_1 - \bar x_2) / s_{pooled}$, the gap between the two groups' averages expressed in units of their pooled standard deviation, so $d = 0.3$ means the two groups differ by three-tenths of a standard deviation on whatever metric is being compared. To find the sample size needed to detect that effect with 80% power at $\alpha = 0.05$ in a two-sample t-test, use the standard normal approximation that trades off the critical value for significance against the value needed for power, both scaled by how small the effect is. The result, worked by hand below, is about 175 people per group, roughly 350 total.

Assumptions

• Observations are independent, both between and within groups (no repeated measurement of the same unit).
• The outcome is approximately normally distributed in each group, or the sample is large enough that the sampling distribution of the mean difference is close to normal via the Central Limit Theorem.
• The two groups have equal (or near-equal) variance, since $d$ is defined relative to a single pooled standard deviation; if variances differ meaningfully, the formula needs a Welch-style adjustment.
• The test is two-sided; a one-sided test would use $z_{1-\alpha}$ instead of $z_{1-\alpha/2}$ and requires a smaller sample for the same power.
• Group sizes are equal ($n_1 = n_2$); unequal allocation changes the formula's leading constant.

Formula and derivation

The approximate required sample size per group for a two-sided, two-sample t-test is:

where $z_{1-\alpha/2}$ is the standard normal critical value for the significance level, and $z_{1-\beta}$ is the standard normal value corresponding to the desired power. Intuitively: the numerator captures how far apart the rejection boundary and the true effect need to sit to both control false positives and reliably detect a real effect; the denominator shrinks the required sample as the effect gets easier to see (bigger $d$).

Plugging in $\alpha = 0.05$ (two-sided) and power $= 0.80$:

import numpy as np
from scipy import stats

alpha, power, d = 0.05, 0.80, 0.3
z_a = stats.norm.ppf(1 - alpha / 2)
z_b = stats.norm.ppf(power)
n_approx = 2 * (z_a + z_b) ** 2 / d ** 2
print(f"z_a={z_a:.4f}, z_b={z_b:.4f}, n_per_group={n_approx:.2f}")
# z_a=1.9600, z_b=0.8416, n_per_group=174.42

Round up: 175 per group, about 350 total.

Trade-offs and pitfalls

• This is a normal approximation, not the exact answer. The exact calculation uses the noncentral t-distribution, since with finite samples the test statistic isn't exactly normal. Confirming with software shows the approximation slightly underestimates the requirement here:

import numpy as np
from scipy import stats

d, alpha = 0.3, 0.05

def power_two_sample_t(n, d, alpha=0.05):
    df = 2 * n - 2
    nc = d * np.sqrt(n / 2)
    tcrit = stats.t.ppf(1 - alpha / 2, df)
    beta = stats.nct.cdf(tcrit, df, nc) - stats.nct.cdf(-tcrit, df, nc)
    return 1 - beta

for n in [175, 176]:
    print(f"n={n} per group -> exact power={power_two_sample_t(n, d, alpha):.4f}")
# n=175 per group -> exact power=0.7991
# n=176 per group -> exact power=0.8014

At exactly 175 per group the true power is 79.91%, just under the 80% target; 176 is the first sample size that actually clears it. The normal approximation is close enough to plan with, but a careful answer flags that the exact figure (computed via software such as R's power.t.test or Python's statsmodels) can differ by a handful of units, and that's the number to use for a final commitment.

• Power is far more sensitive to the assumed $d$ than to $\alpha$ or the power target, because $n$ scales with $1/d^2$; a $d$ that's overestimated by 20% overstates the achievable power substantially at the planned $n$.
• Equal-variance and equal-group-size assumptions rarely hold exactly in practice; if allocation is unequal (e.g. 70/30 due to cost or ethics), the required total sample size goes up even for the same effect.

Direct answer

Compute a p-value per feature with a test suited to its type, adjust the whole set for multiple testing with a false-discovery-rate procedure (Benjamini-Hochberg by default, Benjamini-Yekutieli if dependence is severe and arbitrary), and output a ranked table of feature, effect size, adjusted q-value, and a stability flag so downstream reviewers can prioritize which candidates to validate experimentally. The core design tension is that thousands of correlated tests break the independence assumption behind naive FDR control, so the pipeline needs an explicit step to handle dependence rather than ignoring it.

Structured elaboration

Pipeline steps

1. Data prep. Standardize feature types, handle missingness and rare categories, record the sample size backing each test (tests with wildly different N need to be flagged, since power varies a lot across them).
2. Per-feature test selection. Logistic regression coefficient (Wald z-test) for continuous or binary predictors with relevant covariates included; chi-square or Fisher's exact for small-count categorical features; rank-sum test as a nonparametric fallback when model assumptions fail badly.
3. P-value computation. Prefer the fast asymptotic p-value from the GLM fit. For features where sample size is small or independence is questionable, fall back to a permutation-based null: shuffle the churn label within relevant strata (to preserve covariate structure) and compute the empirical p-value from the permutation distribution.
4. Multiple-testing adjustment.
5. Dependence handling.
6. Ranking and reporting.

Multiple-testing adjustment (step 4)

Benjamini-Hochberg (BH) controls the expected false discovery rate under independence or positive dependence (technically PRDS - positive regression dependency: roughly, no test's result is negatively correlated with another's, so a discovery on one feature never makes a discovery on a different feature more suspicious), which covers most feature-vs-label correlation setups reasonably well:

with $p_{(1)} \leq \dots \leq p_{(m)}$ the sorted p-values and $q$ the target FDR level.

Dependence handling (step 5)

• If dependence is arbitrary (not PRDS), switch to Benjamini-Yekutieli, which divides the threshold by $\sum_{i=1}^m 1/i$ - more conservative but valid under any dependence structure.
• Cluster correlated features (hierarchical clustering on correlation or mutual information) and report one representative per cluster rather than every member, to avoid a ranked list dominated by ten near-duplicate correlated features.
• Complement with a sparse multivariate model: LASSO with stability selection (refit an L1-penalized regression on many bootstrap resamples of the data, and treat the fraction of resamples in which a feature's coefficient survives at nonzero as its stability score) as a second signal - features that are both univariately significant and have a high stability score across resamples are the strongest candidates.

Ranking and reporting (step 6)

Output table: feature name, direction and standardized effect size (odds ratio or Cohen's d), raw p-value, BH q-value, sample size, correlation-cluster ID, stability score (fraction of resamples selecting the feature), recommended action.

Worked example

Twelve features tested for churn correlation, sorted p-values, target FDR $q = 0.10$:

import numpy as np
p = np.array([0.0009,0.0031,0.0084,0.011,0.019,0.027,0.041,0.058,0.12,0.26,0.44,0.81])
m, q = len(p), 0.10
order = np.argsort(p)
p_sorted = p[order]
ranks = np.arange(1, m+1)
below = p_sorted <= ranks/m*q
k_max = np.max(np.where(below)[0]) + 1 if below.any() else 0
# k_max = 8, cutoff p_(8) = 0.058 -> 8 of 12 features rejected at FDR 10%

The largest $i$ where $p_{(i)} \leq \frac{i}{m}q$ is $i=8$ (since $0.058 \le 0.0667$, while rank 9 fails: $0.12 > 0.075$). BH's rule is to take this largest passing rank as the cutoff and reject every hypothesis at or below it - ranks 1 through 8 here - an 8/12 discovery rate at 10% expected FDR. In general the cutoff rank is not simply "the first rank that fails"; it is the largest rank that still satisfies its own threshold, so a single non-monotonic dip partway through the sorted list does not necessarily stop the procedure early.

Trade-offs & pitfalls

• BH under strong negative dependence is not guaranteed valid. If features are engineered to be strongly negatively correlated (rare in practice for churn features, but possible with ratio features), fall back to Benjamini-Yekutieli or a permutation-based q-value.
• Statistical significance at scale is easy; practical significance is the actual bottleneck. With enough rows, thousands of features will show "significant" but tiny effect sizes. Always sort the final list by effect size within the FDR-passing set, not purely by p-value.
• Correlated feature clusters inflate apparent discoveries. Ten near-duplicate engineered features (e.g., 7-day, 14-day, 30-day versions of the same signal) will often all pass together; the clustering step exists specifically to avoid handing downstream teams ten redundant "validated" features.
• Permutation nulls are expensive at scale. Reserve them for features that fail the asymptotic-test assumptions rather than running them universally; screen with the fast GLM p-value first, then permute only the borderline or assumption-violating cases.