Statistical Inference and Hypothesis Testing Questions

Direct answer

The p-value tells you whether the observed difference is unlikely to be pure chance under "no effect." The confidence interval (CI) tells you the range of effect sizes the data are consistent with. Effect size tells you how big the difference actually is, in units the business cares about. You need all three together: a tiny p-value with a tiny effect size is not a reason to act, and a wide confidence interval is a warning that the point estimate alone is not precise enough to bet on.

Structured elaboration

P-value

• What it is: the probability of seeing data this extreme (or more) if there were truly no difference between the groups.
• What it tells you: whether "no effect" is a poor explanation for what you observed.
• What it does NOT tell you: the probability the treatment works, or how large the effect is. A p-value of 0.001 and a p-value of 0.04 can come from effects of the same practical size, just with different sample sizes or noise.

Confidence interval

• What it is: a range of effect sizes that are plausible given the data and the model, at a chosen confidence level (typically 95%).
• What it tells you: both the size of the estimated effect and how precisely it's been measured. A narrow interval means the data pin the effect down tightly; a wide one means there's a lot of remaining uncertainty.
• What it does NOT tell you: it is not literally "a 95% probability the true value is in this specific interval." the 95% describes the long-run behavior of the procedure across repeated experiments, not a probability statement about this one realized interval.

Effect size

• What it is: the actual magnitude of the difference, absolute (percentage points) or relative (percent lift).
• What it tells you: whether the change is worth the engineering cost and rollout risk, independent of whether it's statistically significant.
• What it does NOT tell you: on its own, whether the estimate is reliable. An effect size without a confidence interval could be pure noise.

Worked example

A checkout test: baseline click-through $p_0 = 0.080$, treatment $p_1 = 0.086$, $n = 20{,}000$ per arm.

Pooled test statistic:

95% CI for the absolute difference (unpooled standard error):

(all values computed directly from these formulas with scipy.stats.norm)

Summary sentence for the PM: "Click-through rose from 8.0% to 8.6%, a 7.5% relative lift (p = 0.030). We're 95% confident the true absolute lift is somewhere between 0.06 and 1.14 percentage points. It's a real improvement, though the interval is wide enough that the low end is a modest win, not a blockbuster."

Trade-offs & pitfalls

• Presenting only the p-value invites the "significant equals big and certain" misread. Always pair it with the interval and the effect size in the business's own units.
• A p-value just under 0.05 with a confidence interval that barely excludes zero (as in this example, lower bound 0.0006) is a different story than a p-value of 0.0001 with a tight interval far from zero. Treat "significant" as a single bit of information, not the whole picture.
• Wide confidence intervals are common with realistic sample sizes and should be surfaced, not hidden. Narrowing the interval requires either more data or a less noisy metric, not a different way of describing the same data.

Direct answer

For a two-sided test comparing two proportions with baseline $p_0=0.05$, target $p_1=0.06$ (20% relative lift), $\alpha=0.05$, and power $=0.80$, the required sample size is about 8,158 per arm, computed from the standard normal-approximation formula for a two-proportion test. Unequal allocation and continuous metrics use the same underlying logic (compare the standardized difference against critical values from the desired error rates) but change the variance term and, for unequal group sizes, the optimal split between arms.

Structured elaboration

Formula (equal allocation, two-sided)

where $\bar p = (p_0+p_1)/2$, $z_{1-\alpha/2}=1.9600$ for $\alpha=0.05$, and $z_{1-\beta}=0.8416$ for power $=0.80$.

Numeric steps

(all steps reproduce exactly with plain arithmetic; verified with python3)

So you need about 8,158 users per arm, 16,320 total.

Unequal allocation

With allocation ratio $r = n_1/n_0$ (treatment vs. control), the sample size for the control arm becomes:

with $\bar p = (p_0 + r\,p_1)/(1+r)$. The allocation ratio that minimizes the total sample size $n_0+n_1$ for a fixed variance target (Neyman allocation) is:

For this example, $p_0(1-p_0)=0.0475$ and $p_1(1-p_1)=0.0564$, so $r^{*}=\sqrt{0.0564/0.0475}=1.090$, close to 1: with proportions this close together, equal allocation is already near-optimal, and there's little total-sample-size benefit to skewing the split. Unequal allocation becomes worth doing when one arm is deliberately capped (e.g. a risky treatment held to 10% of traffic); the cost is a larger total sample size than the balanced design would need for the same power.

Continuous metrics

Replace the proportion-variance terms with the outcome variance $\sigma^2$ (estimated from historical data or a pilot):

where $\Delta$ is the absolute mean difference you want to detect. Use a pooled or historical estimate of $\sigma$, and consider winsorizing or a log transform first if the metric is heavy-tailed, since $\sigma^2$ from raw heavy-tailed data can be dominated by a handful of extreme values and understate how many "typical" observations you'll actually need.

Trade-offs & pitfalls

• For small $p_0$ or small $\Delta$, the normal approximation underlying this formula can be inaccurate; check that $n p_0$ and $n(1-p_0)$ are both comfortably above about 5-10, or fall back to an exact binomial/simulation-based power calculation.
• The unequal-allocation Neyman ratio only minimizes total sample size; it does not account for a fixed traffic budget per arm or for practical constraints like a hard cap on treatment exposure, so it should be treated as a starting point, not an automatic answer.
• Continuous-metric sample sizes are only as good as the variance estimate feeding them; an outdated or unrepresentative $\sigma$ estimate (from before a product change, or from a different user segment) will make the planned sample size wrong in either direction.
• None of these formulas account for multiple comparisons, expected attrition, or metric seasonality; pad the computed $n$ for real-world dropout and plan the run to cover at least one full weekly cycle regardless of what the raw sample-size number implies about duration.

Direct answer

Use Levene's test (or its more robust variant, Brown-Forsythe) to compare the variance of conversion rates between mobile and desktop users, since it doesn't require normally distributed data. Variance differences matter beyond the mean because a segment with high variance is heterogeneous: some users convert constantly, others never do, which changes how you should target, personalize, and size experiments for that segment.

Structured elaboration

Candidate tests

Practical steps

1. Define the outcome as a per-user rate over a fixed window (e.g., fraction of sessions in the last 30 days that converted), so each user contributes one continuous-ish value rather than a single binary draw.
2. Visualize with boxplots or violin plots per segment before testing, to catch obvious outliers or bimodal patterns.
3. Run Levene's (center='median', which is the Brown-Forsythe variant) as the primary test; corroborate with a bootstrap CI on the variance ratio if sample sizes are small.

Why variance differences matter for product decisions

• Heterogeneity signals a mixed population. High variance on desktop, for instance, can mean a small set of power users driving most conversions while the majority rarely convert - a segmentation opportunity that a mean-only comparison would completely miss.
• It breaks pooled-variance test assumptions. A standard two-sample t-test assumes equal variances; if desktop and mobile differ substantially, use Welch's t-test for the mean comparison instead of the pooled-variance version, or the mean comparison's p-value will be miscalibrated.
• It changes experiment design. A higher-variance segment needs more samples to detect the same effect size in an A/B test, and may benefit from stratified randomization or variance-reduction techniques (CUPED, regression adjustment). CUPED (Controlled-experiment Using Pre-Experiment Data) works by using each user's own pre-experiment value of the same metric as a covariate, subtracting out the part of their outcome variance that has nothing to do with the treatment; regression adjustment does the same thing more generally with any pre-treatment covariate, not just the metric's own history.

Worked example

Simulate per-user conversion rates for 300 mobile and 300 desktop users (window-based rate, values in [0,1]), designed so the means are close but the spread differs substantially:

import numpy as np
from scipy import stats

rng = np.random.default_rng(42)
mobile  = np.clip(rng.beta(2, 10, 300), 0, 1)   # tighter distribution around ~0.16
desktop = np.clip(rng.beta(0.6, 3, 300), 0, 1)  # same-ish mean, much heavier spread

mean_mobile, var_mobile   = mobile.mean(), mobile.var(ddof=1)      # 0.1622, 0.01041
mean_desktop, var_desktop = desktop.mean(), desktop.var(ddof=1)    # 0.1690, 0.03420

stat, p_value = stats.levene(mobile, desktop, center='median')
# stat = 36.03, p_value = 3.38e-09

The means differ by less than a point (0.162 vs 0.169) - a standard t-test on means alone would show little of interest. But the desktop variance is over 3x the mobile variance, and Levene's statistic of 36.03 gives $p \approx 3.4 \times 10^{-9}$, decisively rejecting equal variances. To stakeholders: "Mobile and desktop convert at similar average rates, but desktop conversion is far more volatile across users - a small group of desktop power users is likely driving most of the volume, while typical mobile users behave more uniformly."

Trade-offs & pitfalls

• Mean-only comparisons can completely miss this. A team that only runs a t-test on means and stops there would report "no meaningful difference" on this exact data, while the underlying user behavior is structurally different between segments.
• Levene's test doesn't tell you why variance differs. Follow up with quantile analysis or subgroup decomposition (e.g., split desktop by tenure or engagement) to find what's driving the spread.
• Small samples make variance tests noisy. Variance estimates are less stable than mean estimates at a given n; prefer the bootstrap CI on the variance ratio when segment sizes are in the low hundreds or below.
• Transform before testing if the metric is extremely skewed. For highly skewed per-user rates, a log or arcsine-square-root transform can stabilize the comparison, but transform consistently across both groups and interpret the transformed scale correctly when reporting.

State the null as "no difference in the metric between the arms" and let the alternative's direction match what the team will actually act on. A one-sided test is defensible only when a decrease in the metric would never change the decision (you would never ship a change that hurts the metric, regardless of a "significant" drop); otherwise use two-sided. The relative cost of a false positive (shipping a change that does not really help, or worse) versus a false negative (missing a real improvement) should directly set alpha and target power, and therefore the sample size and rollout gate.

Structured elaboration

1. Formulating H0 and H1

Let p be the true population rate of the metric of interest (for example, conversion or click-through). H0 states there is no effect:

$H_0: p_{\text{treatment}} = p_{\text{control}}$

If the team will only ship on evidence of improvement:

$H_1: p_{\text{treatment}} > p_{\text{control}} \quad \text{(one-sided)}$

If a regression is also actionable (you would roll back on a significant drop, or the change touches something safety- or compliance-sensitive):

$H_1: p_{\text{treatment}} \neq p_{\text{control}} \quad \text{(two-sided)}$

The population is the unit actually randomized (usually users or sessions in the experiment's traffic), not "everyone who could ever use the product." Say which population the inference is scoped to.

2. One-sided vs two-sided: how to decide

The one-sided vs two-sided choice must be locked before looking at results. Choosing it after peeking at the sign of the effect inflates the true Type I error rate above the stated alpha.

3. Costs, alpha, power, and the rollout

• Alpha (P(reject H0 given H0 true)) is the cost of a false positive: shipping a change that does nothing or hurts, paid in engineering effort, UX regression risk, or reduced trust in future experiments. Lower alpha (e.g. 0.01 instead of 0.05) when shipping is expensive to reverse.
• Power, 1 minus beta (P(reject H0 given H1 true)), is protection against a false negative: missing a real improvement. Higher target power (0.8 to 0.9) when the upside is large or a missed win is costly to the roadmap.
• Both feed the required sample size directly, so this calibration is not just philosophical: it changes how long the test needs to run.

Worked example

A team is testing a checkout change against baseline click-through p0 = 0.10, and wants to detect a 5% relative lift (p1 = 0.105) with alpha = 0.05 and power = 0.80. The required sample size per arm:

$n = \frac{(z_{crit} + z_{\beta})^2 \left[p_0(1-p_0) + p_1(1-p_1)\right]}{(p_1 - p_0)^2}$

With z_beta = 0.8416 (power 0.80):

• Two-sided (z_crit = z_{0.025} = 1.9600): n is about 57,760 per arm
• One-sided (z_crit = z_{0.05} = 1.6449): n is about 45,498 per arm

(both values computed directly from the closed-form expression above with python3; the two differ only in z_crit)

The one-sided design needs about 21% fewer users to reach the same power, which is the concrete return on committing to "we only act on improvement." If the team is not actually willing to make that commitment, that saved sample size is not real, because they will end up wanting to look at the other tail anyway.

Trade-offs & pitfalls

• Picking "one-sided" purely to shrink the sample size, without a genuine commitment to ignore harm, is the most common misuse; it quietly weakens protection against shipping something worse.
• Alpha and power are symmetric-looking numbers with asymmetric consequences: they should be set from the business cost of each error type, not left at textbook defaults (0.05 / 0.80) by convention.
• Rejecting H0 tells you the observed difference is unlikely under "no effect," not that the effect is large enough to matter. Rollout decisions need the estimate and its confidence interval, not just the p-value.
• Failing to reject H0 is not evidence of no effect; it may just mean the test was underpowered for the true effect size.

Direct answer

Familywise error rate (FWER) is the probability of making at least one Type I error (false positive) anywhere across a family of tests; controlling it is strict and conservative. False discovery rate (FDR) is the expected proportion of your rejected hypotheses that are actually false positives; controlling it allows some false positives as long as their share of all discoveries stays bounded. Bonferroni controls FWER by dividing alpha by the number of tests; the Benjamini-Hochberg (BH) procedure controls FDR by comparing sorted p-values to an increasing threshold. Bonferroni fits confirmatory, high-stakes decisions; BH fits exploratory, large-scale scans where you can tolerate a controlled fraction of false leads in exchange for more power.

Structured elaboration

Formal definitions

Let $V$ be the number of false rejections and $R$ the total number of rejections out of $m$ tests.

The two procedures

Why FDR keeps more power

Bonferroni's per-test threshold $\alpha/m$ is flat: every test, no matter how strong its evidence, is held to the same tiny bar. BH's threshold $(k/m)\alpha$ grows with rank $k$, so the smallest p-values in a batch face a much less punishing cutoff than the largest, which is what lets it recover more true discoveries at the same nominal error budget, at the cost of controlling a rate rather than an absolute occurrence.

Worked example

Ten tests, sorted p-values: $0.001, 0.008, 0.012, 0.020, 0.031, 0.045, 0.06, 0.09, 0.20, 0.55$. Target $\alpha = 0.05$.

Bonferroni: per-test threshold $= 0.05/10 = 0.005$. Only $p_{(1)}=0.001$ clears it. 1 rejection.

Benjamini-Hochberg: compare each sorted $p_{(k)}$ to $(k/10)(0.05)$:

The largest rank where the p-value is still below its threshold is $k=4$, so BH rejects the 4 smallest p-values. 4 rejections, all four of which were also below their individual rank-scaled threshold (verified directly from the table above; no p-value beyond rank 4 satisfies the condition).

Same data, same nominal 0.05: Bonferroni finds 1 discovery, BH finds 4.

Trade-offs & pitfalls

• Bonferroni's per-test threshold gets punishing fast as $m$ grows; in a scan of hundreds of metrics it can leave you with zero discoveries even when several effects are real.
• BH's guarantee is about the dependence structure of the p-values: under strong negative dependence between tests it can undercontrol FDR unless you switch to the Benjamini-Yekutieli variant, which divides the threshold by $\sum_{i=1}^m 1/i$ and is correspondingly more conservative.
• "Controls FDR at 5%" does not mean any single reported discovery has a 5% chance of being false. It means, averaged across all discoveries in this batch, about 5% of them are expected to be false. Treating an individual finding's inclusion in the rejected set as proof is a common misreading.
• Neither procedure fixes an underpowered study. Both operate on the p-values you already have; if the true effects are small relative to your noise, tightening or loosening the correction won't manufacture power you didn't design in.