Statistical Inference and Hypothesis Testing Questions

Quick answer

With three independent classifiers each 70% accurate, a majority vote is correct whenever at least 2 of the 3 are individually correct. Summing the binomial probabilities of exactly 2 and exactly 3 correct gives about 78.4%, an improvement over any single classifier's 70%, but that gain depends entirely on the independence assumption holding, which it usually doesn't in practice.

Combinatorial setup

Let $p = 0.70$ be each classifier's accuracy, and assume the three classifiers' correctness is independent. Majority vote is correct if at least 2 of 3 are correct:

Combining algebraically:

Worked calculation

from math import comb

p = 0.70
p_exact2 = comb(3, 2) * p**2 * (1 - p)
p_exact3 = comb(3, 3) * p**3
p_majority = p_exact2 + p_exact3
print(f"P(exactly 2)={p_exact2:.4f}, P(exactly 3)={p_exact3:.4f}")
print(f"P(majority correct)={p_majority:.4f}")
# P(exactly 2)=0.4410, P(exactly 3)=0.3430
# P(majority correct)=0.7840

$P(\text{majority correct}) \approx 0.784$, about a 8.4-percentage-point improvement over any individual classifier's 70% accuracy.

Why independence matters

The 78.4% figure is an upper bound, valid only if the three classifiers' errors are truly independent. In practice, classifiers trained on the same or overlapping data, using similar features or architectures, tend to make correlated errors: when one gets an example wrong, the others are more likely to get it wrong too, since they're picking up on the same signal (or the same blind spot). Positive error correlation shrinks the majority-vote gain, in the extreme case (perfectly correlated errors) collapsing the ensemble accuracy back down to exactly 70%, since all three either agree correctly or agree incorrectly together. Negative correlation, or genuinely diverse models that fail on different examples, pushes the realized gain closer to (or even above) the independence-assuming 78.4%.

Trade-offs and pitfalls

• Never report the independence-assuming number as the expected real-world ensemble accuracy without a caveat. State it as a ceiling, and validate the actual gain empirically on held-out data.
• Diversity, not just accuracy, drives ensemble value. Three classifiers of the same architecture trained on the same data with different random seeds will be far more correlated than three classifiers built on different feature sets, different algorithms, or different training subsets (bagging); the latter is what makes ensembling actually work in practice.
• Pairwise error correlation is measurable from validation data (co-occurrence of errors across the three classifiers on the same held-out set), and is the natural next step to move from the theoretical 78.4% to a realistic estimate.

Approach

Comparing two proportions under $H_0: p_a = p_b$ uses a pooled-proportion z-test. Estimate the pooled proportion across both groups, use it to compute the standard error under the null, form the z-statistic from the observed difference, and convert to a two-sided p-value using the standard normal CDF:

def proportion_ztest(success_a, n_a, success_b, n_b) -> float:
    """
    Two-sided z-test p-value for a difference in two proportions (pooled variance).

    Assumptions:
    - Two independent samples of Bernoulli trials.
    - Large-sample normal approximation holds (rule of thumb: n*p and n*(1-p) >= 5
      in each group under the pooled proportion).
    - H0: p_a == p_b (two-sided alternative).

    Edge cases:
    - Raises ValueError if n_a or n_b <= 0, or if success counts are out of range.
    - If the pooled variance is 0 (e.g. success_a == n_a and success_b == n_b, both
      groups all-success), returns p=1.0 when proportions are equal, else 0.0.
    """
    import math

    for name, v in (("success_a", success_a), ("n_a", n_a), ("success_b", success_b), ("n_b", n_b)):
        if not isinstance(v, (int, float)):
            raise TypeError(f"{name} must be numeric")
    if n_a <= 0 or n_b <= 0:
        raise ValueError("n_a and n_b must be > 0")
    if not (0 <= success_a <= n_a) or not (0 <= success_b <= n_b):
        raise ValueError("success counts must be between 0 and their respective n")

    p_a = success_a / n_a
    p_b = success_b / n_b
    pooled_p = (success_a + success_b) / (n_a + n_b)
    var = pooled_p * (1 - pooled_p) * (1 / n_a + 1 / n_b)

    if var == 0:
        return 1.0 if p_a == p_b else 0.0

    z = (p_a - p_b) / math.sqrt(var)

    def std_norm_cdf(x):
        return 0.5 * (1 + math.erf(x / math.sqrt(2)))

    p_value = 2 * (1 - std_norm_cdf(abs(z)))
    return float(min(max(p_value, 0.0), 1.0))

Key points

• Uses math.erf for the normal CDF, so the function has zero third-party dependencies; if scipy is available, scipy.stats.norm.cdf is a drop-in swap for std_norm_cdf.
• Pools the two sample proportions under the null (rather than using each group's own variance) because that's the correct variance estimate for a test of equality; a confidence interval on the difference, by contrast, should use the unpooled variance.
• Validates types and ranges up front so a bad call fails loudly rather than silently returning a nonsense p-value.

Complexity

$O(1)$ time and space: every operation is a fixed number of arithmetic operations regardless of $n_a$ or $n_b$ (the counts, not the raw data, are the inputs).

Edge cases

• n_a or n_b equal to 0 raises ValueError rather than dividing by zero.
• Success counts outside [0, n] raise ValueError.
• Both groups at 100% success (var == 0, no variability to test): returns 1.0 if the proportions are equal, 0.0 if the sample sizes differ in a way that still leaves the proportions equal (this only occurs at the boundary and both proportions are identically 1.0, so it always returns 1.0 in practice for this branch).
• Very small n_a/n_b where n*p < 5: the function still returns a number, but the normal approximation is unreliable there; a docstring note (and, in production, a runtime warning) flags that Fisher's exact test is the safer choice for small samples.

Verified with a pinned worked example: success_a=1240, n_a=20000 (6.20% conversion) vs. success_b=1360, n_b=20000 (6.80% conversion) gives pooled_p=0.0650, z=-2.4338, p-value = 0.0149, which matches scipy.stats.norm.cdf-based computation to within 1e-9 when cross-checked.

Direct answer

Use one-way ANOVA instead of running six pairwise t-tests across four regions because ANOVA tests a single global null hypothesis (all four region means are equal) with one test, controlling the overall false-positive rate; running six separate pairwise t-tests at 0.05 each lets the false-positive rate compound well past 5%. A significant F-statistic means the variance between region means is larger than would be expected from within-region noise alone, but it only tells you some region differs, not which one, so it's followed by a post-hoc test like Tukey's HSD to identify the specific pairs.

Structured elaboration

Why not just run pairwise t-tests. With 4 regions there are $\binom{4}{2}=6$ possible pairs. Running each pairwise comparison at $\alpha=0.05$ independently, the chance of at least one false positive across all six, assuming the tests were independent, is:

So even with genuinely equal region means, there's roughly a 26.5% chance of at least one "significant" pairwise result purely by chance, more than five times the nominal 5% rate. ANOVA avoids this by testing one combined hypothesis:

Interpreting the F-statistic. $F$ is the ratio of between-group variance to within-group variance:

A large $F$ means the region means are more spread out relative to each other than the natural noise within each region would predict, and a significant p-value (compared to the $F$-distribution with the appropriate numerator/denominator degrees of freedom) means it's unlikely this spread arose by chance alone. Critically, rejecting $H_0$ only supports "at least one region differs from at least one other," not a specific pairwise claim, which is exactly why a post-hoc step is needed.

Assumptions to check. Independent observations; approximately normal residuals within each group (or large enough samples per group for the CLT); homogeneity of variance across groups, checkable with Levene's or Bartlett's test. Report an effect size like $\eta^2$ alongside the F-test, since a significant F with a tiny effect size may not be practically meaningful.

Post-hoc methods. Lead with Tukey's HSD: it's the standard default for all-pairwise comparisons after a significant ANOVA, controlling the family-wise error rate across all pairs simultaneously while providing simultaneous confidence intervals (its Tukey-Kramer variant handles unbalanced group sizes). That's the one answer most interviews are actually testing for. Beyond that baseline, a few named alternatives are worth knowing exist but are depth rather than the first answer: Bonferroni correction (dividing $\alpha$ by the number of comparisons) is simpler but more conservative, costing power; Holm-Bonferroni is uniformly more powerful than plain Bonferroni while still controlling the family-wise error rate; and Benjamini-Hochberg trades some family-wise protection for more power when the concern is false discovery rate across many comparisons rather than strict family-wise control.

Worked example

Satisfaction scores for 5 customers in each of 4 regions:

One-way ANOVA gives $F \approx 22.38$, $p \approx 0.000006$ (verified with scipy.stats.f_oneway), strongly rejecting $H_0$: the region means are not all equal. The group means (74.0, 81.6, 70.4, 77.0) suggest Region 2 is notably higher and Region 3 notably lower, but confirming exactly which pairs differ, and by how much, requires the post-hoc step: running Tukey's HSD on this data (rather than six uncorrected pairwise t-tests, which at $n=5$ per group and this much separation would likely also individually reach significance, but without the family-wise error control that makes the conclusion trustworthy).

Trade-offs & pitfalls

• A significant F only earns you "something differs," not "which pairs differ." Reporting a significant ANOVA as if it identifies a specific region without running the post-hoc step is a common overreach.
• Running naive uncorrected pairwise t-tests after a significant ANOVA "just to see" reintroduces exactly the inflated false-positive problem ANOVA was meant to solve; the whole point of the post-hoc method is that it controls for the multiple comparisons being made.
• If the equal-variance assumption clearly fails (e.g. Levene's test is significant), switch to Welch's ANOVA, which doesn't assume equal variances, rather than proceeding with standard ANOVA and hoping it's robust enough; if normality is a serious concern instead, the Kruskal-Wallis test (the rank-based, nonparametric analogue of one-way ANOVA) is the fallback.

Quick answer

Compare the expected cost of each possible decision and pick the cheaper one: predicting positive costs $(1-p)\times C_{FP}$ in expectation, predicting negative costs $p \times C_{FN}$. With $p=0.20$, $C_{FP}=1$, $C_{FN}=10$, predicting positive costs 0.80 in expectation versus 2.00 for predicting negative, so the Bayes-optimal decision here is to predict positive, even though $p=0.20$ is well under the naive 0.5 cutoff. That's the whole point of cost-sensitive thresholding: the optimal cutoff moves away from 0.5 whenever the two error types aren't equally costly.

Expected cost calculation

Given $P(y{=}1|x) = 0.20$, $C_{FP} = 1$, $C_{FN} = 10$:

p, cost_fp, cost_fn = 0.20, 1, 10
ec_pos = (1 - p) * cost_fp
ec_neg = p * cost_fn
print(f"E[cost|predict positive]={ec_pos}, E[cost|predict negative]={ec_neg}")
# E[cost|predict positive]=0.8, E[cost|predict negative]=2.0

Since $0.80 < 2.00$, the Bayes-optimal decision is to predict positive, despite $p=0.20$ being far below 0.5.

The general Bayes decision rule

For a two-class problem with $p = P(y{=}1|x)$, predict positive whenever its expected cost is no higher than predicting negative's:

Rearranging to isolate $p$ gives the optimal threshold form: predict positive iff

cost_fp, cost_fn = 1, 10
threshold = cost_fp / (cost_fp + cost_fn)
print(f"t* = {threshold:.4f}")
# t* = 0.0909

$t^* \approx 0.091$ here. Since $p=0.20 \ge 0.091$, predicting positive is optimal, matching the direct comparison above.

How costs map to the threshold

• $C_{FP} = C_{FN}$ (equal costs): $t^* = 0.5$, recovering the familiar default MAP decision rule (predict the more probable class).
• $C_{FN} \gg C_{FP}$ (missing a true positive is far more expensive, e.g. failing to flag fraud): $t^*$ drops well below 0.5, so the model predicts positive far more liberally, exactly what happened here with $t^* \approx 0.09$.
• $C_{FP} \gg C_{FN}$ (a false alarm is far more expensive, e.g. an aggressive account-suspension model): $t^*$ rises above 0.5, making the model more conservative about predicting positive.

Trade-offs and pitfalls

• This calculation needs a well-calibrated $P(y{=}1|x)$, not just a ranking score. Many classifiers (especially tree ensembles, uncalibrated neural nets) output scores that rank examples correctly but aren't valid probabilities; calibrate first (e.g. Platt scaling or isotonic regression) before applying a cost-derived threshold.
• Costs are themselves often uncertain business estimates, not measured constants. $C_{FP}=1$, $C_{FN}=10$ should be treated as a starting point to sanity-check with the business, and revisited as a range rather than a single number if the downstream decision is sensitive to it.
• A single global threshold assumes costs are the same for every example, which breaks down when the cost of a false negative genuinely varies by segment (e.g. transaction size in a fraud model); segment-specific thresholds are the natural extension.

Conditional probability updates a belief about an event given new evidence; Bayes' theorem does this formally by combining the detector's known accuracy with how rare fraud actually is. Even with a 98% true positive rate and only a 1% false positive rate, a 0.1% base rate of fraud means most flagged transactions are still false alarms: the posterior probability a flagged transaction is truly fraudulent comes out to about 8.9%.

Structured elaboration

The formula

$P(\text{Fraud}\mid \text{Flag}) = \frac{P(\text{Flag}\mid \text{Fraud})\,P(\text{Fraud})}{P(\text{Flag}\mid \text{Fraud})\,P(\text{Fraud}) + P(\text{Flag}\mid \text{Not Fraud})\,P(\text{Not Fraud})}$

Each term has a name:

• P(Fraud): the prior, the base rate before seeing the flag.
• P(Flag given Fraud): the likelihood, here the true positive rate (sensitivity).
• P(Flag given Not Fraud): the false positive rate.
• The denominator, P(Flag), is the total probability of a flag from either a truly fraudulent or a truly legitimate transaction. This is what "explains away" most of the apparent accuracy.

Why intuition misleads here (base-rate neglect)

It is tempting to read "98% true positive rate" as "a flag is 98% likely to be fraud." That conflates P(Flag given Fraud) with P(Fraud given Flag), which are different conditional probabilities. When the event being detected is rare, even a small false positive rate produces far more false alarms than true ones in absolute terms, because there are so many more legitimate transactions to draw false positives from.

Worked example

$P(\text{Fraud}\mid \text{Flag}) = \frac{0.98 \times 0.001}{0.98 \times 0.001 + 0.01 \times 0.999} = \frac{0.00098}{0.01097} \approx 0.0893$

So about 8.9% of flagged transactions are actually fraudulent; roughly 91% are false alarms (computed directly from the formula above, no simulation needed).

Implications for the decision threshold: the false positive rate drives the posterior almost as much as the base rate does, because it applies to the much larger pool of legitimate transactions. Holding the true positive rate fixed at 98% and varying only the false positive rate:

(each row computed from the same formula with only the false positive rate changed; verified with python3)

Trade-offs & pitfalls

• A model can have excellent sensitivity and still produce a low-precision alert stream when the target event is rare; sensitivity alone is not enough to judge a detector meant for a rare class.
• Tightening the threshold to shrink the false positive rate raises the posterior (fewer false alarms per true one), but it comes at the cost of missing more true fraud (lower true positive rate); the two move in opposite directions.
• Operationally, an 8.9% posterior does not mean "ignore the flag." It means route flagged transactions to a cheaper secondary check (rules, a human reviewer, step-up authentication) rather than an automatic block, so the cost of a false positive stays low.
• Base rates drift over time as fraud patterns and volumes change; a posterior computed once goes stale if the prior is not periodically re-estimated.