Statistical Inference and Hypothesis Testing Questions

Conditional probability updates a belief about an event given new evidence; Bayes' theorem does this formally by combining the detector's known accuracy with how rare fraud actually is. Even with a 98% true positive rate and only a 1% false positive rate, a 0.1% base rate of fraud means most flagged transactions are still false alarms: the posterior probability a flagged transaction is truly fraudulent comes out to about 8.9%.

Structured elaboration

The formula

$P(\text{Fraud}\mid \text{Flag}) = \frac{P(\text{Flag}\mid \text{Fraud})\,P(\text{Fraud})}{P(\text{Flag}\mid \text{Fraud})\,P(\text{Fraud}) + P(\text{Flag}\mid \text{Not Fraud})\,P(\text{Not Fraud})}$

Each term has a name:

• P(Fraud): the prior, the base rate before seeing the flag.
• P(Flag given Fraud): the likelihood, here the true positive rate (sensitivity).
• P(Flag given Not Fraud): the false positive rate.
• The denominator, P(Flag), is the total probability of a flag from either a truly fraudulent or a truly legitimate transaction. This is what "explains away" most of the apparent accuracy.

Why intuition misleads here (base-rate neglect)

It is tempting to read "98% true positive rate" as "a flag is 98% likely to be fraud." That conflates P(Flag given Fraud) with P(Fraud given Flag), which are different conditional probabilities. When the event being detected is rare, even a small false positive rate produces far more false alarms than true ones in absolute terms, because there are so many more legitimate transactions to draw false positives from.

Worked example

$P(\text{Fraud}\mid \text{Flag}) = \frac{0.98 \times 0.001}{0.98 \times 0.001 + 0.01 \times 0.999} = \frac{0.00098}{0.01097} \approx 0.0893$

So about 8.9% of flagged transactions are actually fraudulent; roughly 91% are false alarms (computed directly from the formula above, no simulation needed).

Implications for the decision threshold: the false positive rate drives the posterior almost as much as the base rate does, because it applies to the much larger pool of legitimate transactions. Holding the true positive rate fixed at 98% and varying only the false positive rate:

(each row computed from the same formula with only the false positive rate changed; verified with python3)

Trade-offs & pitfalls

• A model can have excellent sensitivity and still produce a low-precision alert stream when the target event is rare; sensitivity alone is not enough to judge a detector meant for a rare class.
• Tightening the threshold to shrink the false positive rate raises the posterior (fewer false alarms per true one), but it comes at the cost of missing more true fraud (lower true positive rate); the two move in opposite directions.
• Operationally, an 8.9% posterior does not mean "ignore the flag." It means route flagged transactions to a cheaper secondary check (rules, a human reviewer, step-up authentication) rather than an automatic block, so the cost of a false positive stays low.
• Base rates drift over time as fraud patterns and volumes change; a posterior computed once goes stale if the prior is not periodically re-estimated.

Direct answer

A frequentist 95% confidence interval is a statement about the procedure: if you repeated the same sampling and interval-construction process many times, 95% of the resulting intervals would contain the true, fixed parameter. It says nothing about the probability that this one interval, from this one dataset, contains the parameter. A Bayesian 95% credible interval is a direct probability statement about the parameter itself: given the observed data and a prior, there is a 95% probability the parameter lies in that range. The two coincide numerically when the prior is flat or the data dominate, but their meaning is different even then.

Structured elaboration

Why they can differ. The Bayesian interval blends the observed data with the prior, weighted by their relative precision (inverse variance). A confident (low-variance) prior pulls the credible interval toward the prior mean and narrows it; a weak (high-variance, close to flat) prior lets the data dominate and the credible interval approaches the frequentist CI.

Worked example

Suppose an experiment produces an effect estimate $\hat\theta = 2.0$ with standard error $SE = 1.0$.

Frequentist 95% CI:

Bayesian 95% credible interval, with a prior belief centered at zero and moderately informative, $\theta \sim N(0, 1^2)$, combined with the data via the standard normal-normal conjugate update, conjugate meaning the prior and the posterior come from the same distribution family (normal in, normal out), which is what lets the update be a closed-form formula instead of numerical integration (treating $SE=1.0$ as the data's known standard deviation):

(All values verified by direct computation of the conjugate-update formulas.) The frequentist interval, $(0.04, 3.96)$, is entirely positive and would typically be read as "a positive effect." The Bayesian interval, $(-0.39, 2.39)$, is pulled toward zero by the prior and includes zero, because the prior said the effect was probably small and the single data point wasn't overwhelming enough to fully override that. Neither interval is "wrong": they're answering different questions (what would repeated sampling do, versus what should you believe given this data and this prior).

Why this matters for stakeholders

A credible interval lets you say, in plain language, "there's a 95% probability the true lift is between X and Y," which is exactly what most non-technical stakeholders assume a confidence interval means anyway. That mismatch, people reflexively read frequentist CIs as if they were credible intervals, is the single most common misinterpretation in practice, and it's worth correcting explicitly rather than exploiting it. If the team has real prior information (results from a similar past experiment, a strong mechanistic reason to expect a small effect), a Bayesian framing makes that information visible and auditable instead of leaving it as an unstated bias in how someone reads the frequentist number.

Trade-offs & pitfalls

• Never claim a frequentist CI has a 95% chance of containing the parameter for the specific interval in hand; that statement describes the long-run behavior of the method, not this instance.
• A Bayesian interval is only as trustworthy as its prior. A poorly chosen or overconfident prior can bias the interval in a way that's easy to miss if the prior isn't stated and justified alongside the result.
• With large samples the data dominate and a weak prior makes the two intervals nearly identical, so in high-traffic product experiments the distinction is often more about how you communicate the result than about a different numeric answer.

Direct answer

Frame non-inferiority as a one-sided test on the difference minus the margin: the null is that the new engine is worse than the current one by at least the margin $\delta$, and you reject that null (concluding non-inferiority) if the data rule it out. The Two One-Sided Tests (TOST) procedure runs this as two one-sided tests simultaneously; for a pure non-inferiority claim you only need the lower-bound side. Sample sizes for small margins are large, often far larger than a standard superiority test, because you're trying to rule out a small effect with precision rather than just detect any effect at all.

Structured elaboration

Hypotheses

With $p_{\text{new}}$ and $p_{\text{curr}}$ the true conversion rates and margin $\Delta = -0.005$ (a drop of more than 0.5 percentage points counts as "worse"):

Non-inferiority (one-sided):

Equivalence (two-sided, TOST), if the goal were "not meaningfully different in either direction":

TOST procedure

Run two one-sided tests against the two margin boundaries:

1. Test $p_{\text{new}} - p_{\text{curr}} > -\Delta$ (reject if the standardized statistic exceeds $z_{1-\alpha}$)
2. Test $p_{\text{new}} - p_{\text{curr}} < +\Delta$ (reject if the statistic is below $z_{\alpha}$)

If both one-sided tests reject, conclude equivalence within $\pm\Delta$. For pure non-inferiority (only care about the "not worse" direction), run only test 1, using a one-sided $\alpha$ (commonly 0.025, matching the one-sided error rate of a standard 0.05 two-sided test).

Test statistic (large-sample z-test for a difference of proportions):

Sample size

Choose one-sided $\alpha$, desired power $1-\beta$, a planning baseline rate $p_0$, and the true effect you want to be powered to detect ($\delta_{\text{target}}$, typically 0 - assuming the new engine performs identically to the old one and you want to reliably rule out that it's actually $\Delta$ worse):

Worked example

Planning inputs: baseline $p_0 = 0.10$, margin $\Delta = -0.005$, one-sided $\alpha = 0.025$, power $= 80\%$, assuming true rates are equal ($\delta_{\text{target}} = 0$, i.e. $p_1 = p_0 = 0.10$).

from scipy import stats
alpha, power = 0.025, 0.80
z_alpha, z_beta = stats.norm.ppf(1-alpha), stats.norm.ppf(power)
p0, delta_margin = 0.10, 0.005
var_term = 2 * p0*(1-p0)
n_per_group = (z_alpha+z_beta)**2 * var_term / delta_margin**2
# n_per_group = 56511.93... -> round up to 56512 per arm, ~113,024 total

Recommending this to stakeholders: "Ruling out a 0.5-point drop in conversion with 80% power requires roughly 56,500 users per arm, over 100,000 total, because we're trying to detect the absence of a small effect precisely, not just the presence of any effect." That single number is often the deciding factor in whether non-inferiority testing is practical for a given margin.

Trade-offs & pitfalls

• Margin choice drives everything. A tight margin (e.g., 0.5 points) demands a much larger sample than a loose one; the margin should be set from business impact (how much conversion loss is tolerable) or prior data, not picked for statistical convenience.
• One-sided $\alpha$ still needs justification. Using 0.025 one-sided matches a conventional two-sided 0.05 test's error budget, but confirm this convention with stakeholders since it directly trades off against required sample size.
• Analysis population matters more here than in superiority testing. Per-protocol analysis (excluding non-compliant users) is often preferred for non-inferiority because intention-to-treat analysis tends to bias results toward apparent non-inferiority (dilution of a real difference), which is the wrong direction to be conservative in.
• Asymmetric costs change the design. If a regression in conversion is much costlier than a similar-sized loss elsewhere, consider a stricter margin or higher power rather than treating this like a standard A/B test.

Direct answer

Use one-way ANOVA instead of running six pairwise t-tests across four regions because ANOVA tests a single global null hypothesis (all four region means are equal) with one test, controlling the overall false-positive rate; running six separate pairwise t-tests at 0.05 each lets the false-positive rate compound well past 5%. A significant F-statistic means the variance between region means is larger than would be expected from within-region noise alone, but it only tells you some region differs, not which one, so it's followed by a post-hoc test like Tukey's HSD to identify the specific pairs.

Structured elaboration

Why not just run pairwise t-tests. With 4 regions there are $\binom{4}{2}=6$ possible pairs. Running each pairwise comparison at $\alpha=0.05$ independently, the chance of at least one false positive across all six, assuming the tests were independent, is:

So even with genuinely equal region means, there's roughly a 26.5% chance of at least one "significant" pairwise result purely by chance, more than five times the nominal 5% rate. ANOVA avoids this by testing one combined hypothesis:

Interpreting the F-statistic. $F$ is the ratio of between-group variance to within-group variance:

A large $F$ means the region means are more spread out relative to each other than the natural noise within each region would predict, and a significant p-value (compared to the $F$-distribution with the appropriate numerator/denominator degrees of freedom) means it's unlikely this spread arose by chance alone. Critically, rejecting $H_0$ only supports "at least one region differs from at least one other," not a specific pairwise claim, which is exactly why a post-hoc step is needed.

Assumptions to check. Independent observations; approximately normal residuals within each group (or large enough samples per group for the CLT); homogeneity of variance across groups, checkable with Levene's or Bartlett's test. Report an effect size like $\eta^2$ alongside the F-test, since a significant F with a tiny effect size may not be practically meaningful.

Post-hoc methods. Lead with Tukey's HSD: it's the standard default for all-pairwise comparisons after a significant ANOVA, controlling the family-wise error rate across all pairs simultaneously while providing simultaneous confidence intervals (its Tukey-Kramer variant handles unbalanced group sizes). That's the one answer most interviews are actually testing for. Beyond that baseline, a few named alternatives are worth knowing exist but are depth rather than the first answer: Bonferroni correction (dividing $\alpha$ by the number of comparisons) is simpler but more conservative, costing power; Holm-Bonferroni is uniformly more powerful than plain Bonferroni while still controlling the family-wise error rate; and Benjamini-Hochberg trades some family-wise protection for more power when the concern is false discovery rate across many comparisons rather than strict family-wise control.

Worked example

Satisfaction scores for 5 customers in each of 4 regions:

One-way ANOVA gives $F \approx 22.38$, $p \approx 0.000006$ (verified with scipy.stats.f_oneway), strongly rejecting $H_0$: the region means are not all equal. The group means (74.0, 81.6, 70.4, 77.0) suggest Region 2 is notably higher and Region 3 notably lower, but confirming exactly which pairs differ, and by how much, requires the post-hoc step: running Tukey's HSD on this data (rather than six uncorrected pairwise t-tests, which at $n=5$ per group and this much separation would likely also individually reach significance, but without the family-wise error control that makes the conclusion trustworthy).

Trade-offs & pitfalls

• A significant F only earns you "something differs," not "which pairs differ." Reporting a significant ANOVA as if it identifies a specific region without running the post-hoc step is a common overreach.
• Running naive uncorrected pairwise t-tests after a significant ANOVA "just to see" reintroduces exactly the inflated false-positive problem ANOVA was meant to solve; the whole point of the post-hoc method is that it controls for the multiple comparisons being made.
• If the equal-variance assumption clearly fails (e.g. Levene's test is significant), switch to Welch's ANOVA, which doesn't assume equal variances, rather than proceeding with standard ANOVA and hoping it's robust enough; if normality is a serious concern instead, the Kruskal-Wallis test (the rank-based, nonparametric analogue of one-way ANOVA) is the fallback.

Direct answer

A $(1-\alpha)$ confidence interval and a two-sided hypothesis test at level $\alpha$ are two views of the same procedure: the CI is exactly the set of null values that the test would fail to reject. So a 95% CI for a mean difference that excludes zero is equivalent to a two-sided test of "difference = 0" rejecting at $\alpha = 0.05$. That controls the long-run Type I error rate at 5%, not the probability that this particular result is a false positive.

Structured elaboration

The formal duality

For a parameter $\theta$, a $(1-\alpha)$ CI is built by inverting the test: it contains every value $\theta_0$ for which the two-sided test of $H_0: \theta = \theta_0$ against $H_a: \theta \neq \theta_0$ fails to reject at level $\alpha$.

Equivalently: $\theta_0$ falls outside the CI if and only if the test rejects. This is why testing "difference = 0" and checking whether 0 is inside the 95% CI give identical accept/reject conclusions, for any $\theta_0$, not just zero.

What "excludes zero" tells you (and doesn't)

• What it tells you: under the sampling model and its assumptions (independence, correctly estimated standard error, and either normal errors or a large enough sample for the CLT to apply), if the true difference were actually zero, an interval this far from zero would arise no more than 5% of the time across repeated samples. That is what "Type I error controlled at 5%" means: a property of the long-run procedure, not a 5% chance that this specific null is true.
• What it does not tell you: it says nothing about effect size in practical terms, and it is silent on Type II error or power for other effect sizes. A narrow CI that barely excludes zero and a wide CI that excludes zero by a mile both "reject," but they carry very different information.

Reporting to stakeholders

Lead with the point estimate and interval together, translate the interval's low end into business terms (does it still matter if the true effect is at the boundary?), and only mention the p-value as a secondary confirmation, since it adds no information beyond "0 is outside the CI" for a two-sided test at matching $\alpha$.

Worked example

Suppose an A/B test on checkout conversion produces a mean difference of 0.014 (1.4 percentage points) with standard error 0.006, and you want a 95% CI.

This excludes 0, so the two-sided test at $\alpha = 0.05$ rejects. Confirming with the test statistic directly:

from scipy import stats
diff, se = 0.014, 0.006
z_crit = stats.norm.ppf(0.975)          # 1.959963984540054
lo, hi = diff - z_crit*se, diff + z_crit*se   # (0.0022402, 0.0257598)
z_stat = diff/se                         # 2.3333333333333335
p_value = 2*(1 - stats.norm.cdf(abs(z_stat)))  # 0.01963065725729063

Both routes agree: CI excludes 0, p < 0.05, reject $H_0$. To stakeholders: "Conversion improved by 1.4 percentage points (95% CI: 0.22 to 2.58 points), a statistically significant result at the 5% level."

Trade-offs & pitfalls

• The overlapping-CI fallacy. Two group-level CIs that overlap do not imply the difference between the groups is non-significant, and non-overlapping CIs are a stricter (overly conservative) condition than a significant difference test. The duality above is specifically about the CI of the difference versus the null value, not about comparing two separate CIs by eye.
• "Excludes zero" is not "matters." With enough sample size, trivially small differences produce CIs that exclude zero. Always report the interval's practical range, not just significance.
• Assumption sensitivity. The duality and the stated Type I error rate both depend on the SE being estimated correctly (proper variance formula, no unmodeled clustering/dependence). Get the SE wrong and both the CI and the p-value are wrong together.
• Multiple comparisons. If you are constructing many CIs or testing many hypotheses, the 5% per-interval guarantee does not extend to "5% chance any of them is wrong" - that requires a correction (e.g., Bonferroni, Benjamini-Hochberg) on top of the per-test $\alpha$.