Statistical Inference and Hypothesis Testing Questions

Direct answer

The null hypothesis (H0) is the "nothing changed" default you assume true until the data gives strong enough evidence to reject it. The alternative hypothesis (H1, or Ha) is the specific claim you're trying to find evidence for. A two-tailed test's alternative allows the effect to go either direction, so an increase or a decrease both count as evidence against H0. A one-tailed test's alternative commits in advance to only one direction, and only evidence in that direction can ever reject H0, however extreme the result is in the other direction, a one-tailed test treats it as not significant.

Structured elaboration

Defining the two hypotheses

• H0: the status-quo claim, typically "no difference" or "no effect," for example mu_new equals mu_old, or p_new equals p_old. It's what you'd believe by default absent evidence otherwise.
• H1 (or Ha): the claim there is reason to suspect and want the data to support. Hypothesis testing is structured to make H0 the thing you disprove, not the thing you prove; you never "accept H0," you either reject it or fail to reject it.

One-tailed versus two-tailed

A two-sided alternative is symmetric: H1 says the parameter is simply not equal to the null value, so evidence in either direction, higher or lower, counts against H0. A one-sided alternative picks a direction in advance, for example H1 says the new value is greater than the old one, and the entire significance level (say 5%) is allocated to that one tail of the distribution. That gives more power to detect an effect in that specific direction for the same sample size, at the cost of being structurally blind to an effect in the other direction.

Worked example

Testing whether a new checkout flow changes conversion rate, where p_old is the current rate and p_new is the new flow's rate:

Testing whether it changes the metric in either direction, appropriate when you'd act on either a lift or a drop, for example rolling back a regression just as readily as shipping an improvement:

$$H_0: p_{\text{new}} = p_{\text{old}}, \qquad H_1: p_{\text{new}} \neq p_{\text{old}}$$

Testing whether it increases the metric, appropriate only when a decrease and "no effect" would be handled identically, for example neither would ship:

$$H_0: p_{\text{new}} \le p_{\text{old}}, \qquad H_1: p_{\text{new}} > p_{\text{old}}$$

Which to choose, and why

Choose the two-sided test whenever a result in either direction would change the decision, which is the common case in product work: a checkout redesign that decreases conversion is just as actionable, roll it back, as one that increases it, so a one-sided test built only to detect increases could let a real, harmful regression show up as "not significant" simply because the test structurally can't reject in that direction. Choose a one-sided test only when a result in the "wrong" direction is genuinely not actionable, and that decision was made before seeing the data. For example, a pure cost-reduction change to backend infrastructure where the only actions are "doesn't hurt the metric" versus "measurably hurts it," never "significantly helped," might reasonably justify a directional test. That's a narrow case, though, and switching to one-sided after peeking at a result that's "almost significant" two-sided is a form of p-hacking, not a legitimate use of the one-sided test.

Trade-offs and pitfalls

• One-sided tests have more power for a fixed alpha and sample size, but that power gain exactly matches giving up the ability to detect the opposite effect. It isn't a free lunch, it's a bet on direction made in advance.
• Choosing the tail after seeing which direction the data leans is a common way to quietly gain extra apparent significance without disclosing it. Always fix the direction, or commit to two-sided, before collecting data, ideally in writing.
• Default to two-sided unless there is a specific, pre-committed, defensible reason not to. When a one-sided test is used, say so explicitly in the write-up along with the pre-registered direction and rationale, so a reader can judge whether the choice was made honestly.

Direct answer

Familywise error rate (FWER) is the probability of making at least one Type I error (false positive) anywhere across a family of tests; controlling it is strict and conservative. False discovery rate (FDR) is the expected proportion of your rejected hypotheses that are actually false positives; controlling it allows some false positives as long as their share of all discoveries stays bounded. Bonferroni controls FWER by dividing alpha by the number of tests; the Benjamini-Hochberg (BH) procedure controls FDR by comparing sorted p-values to an increasing threshold. Bonferroni fits confirmatory, high-stakes decisions; BH fits exploratory, large-scale scans where you can tolerate a controlled fraction of false leads in exchange for more power.

Structured elaboration

Formal definitions

Let $V$ be the number of false rejections and $R$ the total number of rejections out of $m$ tests.

$$FWER = P(V \ge 1) \qquad\qquad FDR = E\left[\frac{V}{\max(R,1)}\right]$$

The two procedures

Why FDR keeps more power

Bonferroni's per-test threshold $\alpha/m$ is flat: every test, no matter how strong its evidence, is held to the same tiny bar. BH's threshold $(k/m)\alpha$ grows with rank $k$, so the smallest p-values in a batch face a much less punishing cutoff than the largest, which is what lets it recover more true discoveries at the same nominal error budget, at the cost of controlling a rate rather than an absolute occurrence.

Worked example

Ten tests, sorted p-values: $0.001, 0.008, 0.012, 0.020, 0.031, 0.045, 0.06, 0.09, 0.20, 0.55$. Target $\alpha = 0.05$.

Bonferroni: per-test threshold $= 0.05/10 = 0.005$. Only $p_{(1)}=0.001$ clears it. 1 rejection.

Benjamini-Hochberg: compare each sorted $p_{(k)}$ to $(k/10)(0.05)$:

The largest rank where the p-value is still below its threshold is $k=4$, so BH rejects the 4 smallest p-values. 4 rejections, all four of which were also below their individual rank-scaled threshold (verified directly from the table above; no p-value beyond rank 4 satisfies the condition).

Same data, same nominal 0.05: Bonferroni finds 1 discovery, BH finds 4.

Trade-offs & pitfalls

• Bonferroni's per-test threshold gets punishing fast as $m$ grows; in a scan of hundreds of metrics it can leave you with zero discoveries even when several effects are real.
• BH's guarantee is about the dependence structure of the p-values: under strong negative dependence between tests it can undercontrol FDR unless you switch to the Benjamini-Yekutieli variant, which divides the threshold by $\sum_{i=1}^m 1/i$ and is correspondingly more conservative.
• "Controls FDR at 5%" does not mean any single reported discovery has a 5% chance of being false. It means, averaged across all discoveries in this batch, about 5% of them are expected to be false. Treating an individual finding's inclusion in the rejected set as proof is a common misreading.
• Neither procedure fixes an underpowered study. Both operate on the p-values you already have; if the true effects are small relative to your noise, tightening or loosening the correction won't manufacture power you didn't design in.

Direct answer

A Type I error is rejecting a true null hypothesis (a false positive: concluding there's an effect when there isn't one); a Type II error is failing to reject a false null (a false negative: missing a real effect). The significance level alpha ($\alpha$) is the Type I error rate you're willing to accept; beta ($\beta$) is the Type II error rate, and power ($1-\beta$) is the chance of detecting a true effect. The right alpha is not a universal default. It should reflect how expensive a false positive is relative to a false negative for the specific decision being made.

Structured elaboration

The error matrix

Business examples

• Type I error (false positive) is costly: shipping an expensive personalization system because a test showed a "significant" lift that was actually noise. You pay ongoing engineering and infrastructure cost, plus the opportunity cost of the team not working on something that actually helps.
• Type II error (false negative) is costly: killing a checkout redesign because the test result wasn't significant, when the redesign truly would have lifted conversion. You lose the revenue and any competitive advantage from never shipping it.

Setting alpha and beta from cost, not convention

For a fixed sample size, $\alpha$ and $\beta$ trade off against each other: tightening $\alpha$ (fewer false positives) pushes power down (more false negatives) unless you also grow $n$. The textbook defaults ($\alpha=0.05$, power $=0.80$) are a starting point, not a business decision. Set $\alpha$ lower when a false positive is expensive to reverse (a payments flow, a compliance-adjacent surface, a change that's hard to roll back); accept a higher $\alpha$, or invest in a larger sample, when missing a true win is the more expensive mistake.

Worked example

A team wants to detect a 2 percentage-point lift on a checkout metric, baseline $p_0=0.20$, $p_1=0.22$, with $n=2{,}000$ users already collected per arm (fixed. This cohort won't grow before the decision is needed).

Holding $n$ fixed, power at two different alpha values (two-sided z-test for two proportions):

(computed from the standard two-proportion power formula with scipy.stats.norm)

Insisting on $\alpha=0.01$ for this "high-stakes" surface, without collecting more data, drops power from 34% to 15%: an 85% chance of missing a real 2-point lift entirely. If the team genuinely needs $\alpha=0.01$, the fix is a larger $n$, not accepting a test that's this underpowered.

Trade-offs & pitfalls

• Alpha and beta are symmetric-looking numbers with asymmetric real-world stakes; picking $\alpha=0.05$ by convention without asking what each error costs is the most common mistake.
• Tightening alpha without growing the sample size silently starves the test of power. Teams sometimes do this and then are surprised the "safer" test never finds anything.
• Failing to reject $H_0$ is not proof of no effect. It can just as easily mean the test was underpowered for the true effect size, so a null result should be read alongside the achieved power, not treated as a settled "no."

Three common sampling biases in user research and product analytics are selection bias, survivorship bias, and non-response bias. Each systematically skews who or what ends up in a dataset relative to the population the team actually wants to understand, and each has a distinct, practical mitigation.

Structured elaboration

Selection bias

The recruitment or logging mechanism itself favors certain users over others, so the sample never represented the target population to begin with.

• Product example: recruiting research participants exclusively through an in-app "give feedback" banner over-represents highly engaged power users and misses casual or lapsed users entirely.
• Mitigation: diversify recruitment channels (email, support tickets, targeted outreach to inactive users, paid panels), and when analyzing logged data, weight or stratify results by known usage segments so heavy users do not dominate the conclusion.

Survivorship bias

The dataset only contains entities that "survived" some earlier filter, silently excluding the ones that dropped out, which are often the most informative cases for understanding why people leave.

• Product example: analyzing feature usage only among currently active accounts misses everyone who abandoned the product during onboarding, so the data looks healthier than the funnel actually is.
• Mitigation: instrument and retain events from the full funnel, including users who never became active, and deliberately recruit recent churners or drop-outs for exit interviews rather than only studying who remains.

Non-response bias

Among people invited to respond (a survey, an in-app prompt), the ones who actually respond differ systematically from the ones who do not, in ways related to the very thing being measured.

• Product example: a satisfaction survey sent right after a purchase mostly reaches people who are already positive enough to engage further, biasing an NPS score upward relative to the full user base.
• Mitigation: keep surveys short to raise response rates, follow up with non-responders (or a random subsample of them), and compare respondent demographics or behavior to the overall population to check how representative the response set actually is; apply response weighting when a gap is found.

Worked example

A team wants to know why a new onboarding flow has a 40% completion rate. If they only interview users pulled from a "rate your experience" prompt shown after successful completion, they have compounded two biases at once: survivorship (only completers are in the sampling frame at all) and non-response (only the completers who bothered to respond to the prompt are actually interviewed). The resulting interviews will describe the experience of the roughly 40% who finished and felt strongly enough to respond, and say nothing about the 60% who dropped out, which is the group the team most needed to understand. Fixing this requires reaching users at the point of drop-off (an exit-intent prompt, or a follow-up email to users who started but did not finish), not just the users who reached the end.

Trade-offs & pitfalls

• These three biases often compound in the same study (as above), so checking for one is not enough; ask separately who was even eligible to be seen, who dropped out along the way, and who actually responded.
• Fixing selection bias with broader recruitment can raise the cost and time of a study; weighting analysis by known segments is often a cheaper partial fix when full re-recruitment is not practical.
• Comparing respondents to the overall population only catches non-response bias on variables you can actually measure for non-respondents (demographics, usage tier); it cannot catch bias on unmeasured traits.
• Passive telemetry (event logs) is not automatically unbiased just because it does not rely on someone opting in; the underlying feature or platform itself can have selection effects, such as only being available on newer devices.

Direct answer

Confidence interval width is driven by the standard error, and the standard error scales with variability divided by the square root of the sample size: more variable data means wider intervals, and more data means narrower ones, but only at a diminishing rate. Doubling your sample size does not halve the interval, it shrinks it by a factor of about $1/\sqrt{2} \approx 0.71$, a roughly 29% reduction. You need four times the data to actually halve the width.

Structured elaboration

The formulas. For a mean, standard error is:

$$SE = \frac{\sigma}{\sqrt{n}}$$

For a proportion:

$$SE = \sqrt{\frac{p(1-p)}{n}}$$

Both have the same shape: a measure of the underlying variability in the numerator, sample size under a square root in the denominator. A (roughly) 95% CI is approximately $\text{estimate} \pm 1.96 \times SE$, so CI width is directly proportional to $SE$.

Why square root, not n. An average of $n$ independent observations has variance $\sigma^2/n$ (variances of independent things add, and dividing by $n$ to get an average divides the variance by $n^2$, leaving $\sigma^2/n$ after undoing one factor of $n$). Standard deviation, not variance, is what sets the scale of the interval, and standard deviation is the square root of variance, which is where the $\sqrt{n}$ in the denominator comes from.

The consequence for doubling n. If $n \to 2n$:

$$\frac{SE_{2n}}{SE_n} = \frac{\sigma/\sqrt{2n}}{\sigma/\sqrt{n}} = \frac{1}{\sqrt{2}} \approx 0.707$$

So the interval shrinks to about 71% of its original width, not 50%. To actually halve the width you need $SE_{kn} = SE_n/2 \Rightarrow \sqrt{k} = 2 \Rightarrow k = 4$: quadruple the sample.

Non-technical intuition. Each new observation gives you a little more information, but because the estimate is an average, the added observations partly "cancel out" existing noise rather than stacking on top of it the way a raw sum would. Think of it like averaging opinions to estimate a room's mood: your first 10 opinions cut a lot of your uncertainty, but going from 1,000 to 2,000 opinions barely moves the needle, because you'd already averaged out most of the noise with the first 1,000. The information is still growing, just at a shrinking rate per additional observation.

Worked example

Take $\sigma = 10$ and compute the standard error at $n = 100, 200, 400$:

(All values computed directly from the formula and verified.) Going from n=100 to n=200 (doubling) drops the margin from 1.96 to 1.39, a 29.3% reduction, matching $1/\sqrt{2} \approx 0.707$ exactly. Going from n=100 to n=400 (quadrupling) drops the margin from 1.96 to 0.98, exactly half.

Trade-offs & pitfalls

• The most common wrong intuition in an interview is "2x the users means roughly 2x the precision." It's a very natural mistake because it's how totals work, but estimates of an average behave differently, and getting this wrong leads to badly underestimating how long an experiment needs to run to hit a target precision.
• Because of the diminishing $1/\sqrt{n}$ return, past a certain point it's often cheaper to reduce $\sigma$ itself (removing or capping extreme outliers, using a less noisy proxy metric, or CUPED-style variance reduction (using each user's own pre-experiment behavior, such as their historical conversion rate, as a control variate to subtract out variance that has nothing to do with the treatment)) than to keep collecting more raw sample size to hit a target interval width.
• This scaling only holds under the independence assumption behind $SE = \sigma/\sqrt{n}$; correlated observations (e.g. multiple sessions from the same user counted as independent) inflate the effective variance and make the naive formula overstate precision.