Statistical Inference and Hypothesis Testing Questions
Reasoning about uncertainty in data and drawing formal conclusions from samples. Covers probability rules and common distributions, the Central Limit Theorem, sampling, standard error, confidence intervals, and Bayesian reasoning, together with the significance-testing framework: null and alternative hypotheses, p-values, statistical power, Type I and Type II errors, effect sizes, and choosing the right test (t-test, chi-square, non-parametric). Emphasizes correctly interpreting statistical results and avoiding common misreadings of significance in business and product contexts rather than memorizing formulas.
What does a 95% confidence interval actually mean? Give the correct frequentist interpretation, contrast it with the common misreading that there's a 95% chance the true value falls inside this particular interval, and explain in plain language how you would describe a confidence interval to someone who isn't a statistician.
Sample Answer
Direct answer
A 95% confidence interval is a range built from your sample using a procedure that, if repeated many times, would produce an interval containing the true, fixed, unknown parameter about 95% of the time. It is a statement about the reliability of the procedure across hypothetical repetitions, not a 95% probability that this one specific, already-computed interval contains the true value. The true value either is or isn't in this particular interval; there's no probability left to talk about once the interval is fixed.
Structured elaboration
The correct frequentist interpretation
Before data is collected, the interval's endpoints are random, since they depend on which sample happens to be drawn; the true parameter is a fixed, unknown constant. "95% confidence" describes the long-run behavior of the interval-producing procedure: across many hypothetical repeated samples, about 95% of the intervals it produces would bracket the true value, and about 5% would miss it. Once actual data is collected and one specific interval is computed, say [41.2, 48.7], that interval either contains the true value or it doesn't. The randomness was in the sampling and calculation, not in the now-fixed true parameter or the now-fixed interval.
The common misreading, and why it's wrong
"There's a 95% chance the true value is in this interval" sounds almost identical to the correct statement but treats the true parameter as the random thing with a 95% probability of being in some location, when in the frequentist framework the true parameter isn't random at all, only the sampling procedure is. This misreading is extremely common, surveys of both students and working researchers find majorities endorse it, precisely because the language is subtle: "confidence" was chosen specifically to avoid claiming "probability" about a fixed parameter, but in everyday usage the two words feel interchangeable.
Why the distinction matters practically
It matters most when someone tries to use a single computed interval to make a probabilistic claim. "There's a 95% chance the true conversion lift is between 2% and 6%" is not licensed by a frequentist confidence interval, that statement would require a Bayesian credible interval, which is a genuine probability statement about the parameter given a prior. Swapping one framework's language onto the other's number is a real, if common, error, not just pedantry; it can lead to overconfident decisions when the sampling assumptions behind the interval, random sampling, correct variance model, are shakier than they look.
Plain-language version for a non-statistician
"We can't know the exact true number, only estimate it from our sample. This range is our best estimate plus a margin that accounts for sampling luck: if we ran this same measurement over and over on different random samples, about 19 times out of 20 the range we'd compute would contain the real number. So this range is the set of values consistent with what we observed, not a guarantee, but a calibrated one."
Worked example
Suppose (illustrative numbers, chosen to demonstrate the interpretation, not derived from a specific dataset) an A/B test reports a 95% confidence interval for the lift in conversion rate of [1.2%, 5.8%], entirely above zero.
- Correct statement: using this measurement procedure, if the experiment were rerun many times, about 95% of the resulting intervals would contain the true lift; this particular interval happens to be [1.2%, 5.8%].
- Correct practical use: because the interval excludes 0, the data are inconsistent with "no effect" at the 5% significance level, equivalent to a two-sided test rejecting the null of zero lift at alpha = 0.05.
- Incorrect statement to avoid: "there's a 95% probability the true lift is between 1.2% and 5.8%," which treats the fixed true lift as the random quantity.
- Incorrect statement to avoid: "95% of future users will see a lift in this range," which confuses a confidence interval for the mean parameter with a prediction interval for individual outcomes, a different and typically much wider interval.
Trade-offs and pitfalls
- The CI-versus-hypothesis-test duality, excludes zero if and only if significant at that alpha, is genuinely useful, but leaning on it too heavily can obscure that the interval also communicates magnitude and precision, which a bare p-value doesn't. Report the interval, not just significant or not significant.
- A narrow interval is not automatically good news; it just means the estimate is precise. A narrow interval tightly around a practically meaningless effect size is still a practically meaningless result.
- All of this assumes the sampling and modeling assumptions behind the interval are valid, random or representative sampling, correctly specified variance. A confidence interval computed on a biased or non-random sample gives a precise-looking interval around the wrong quantity; precision is not the same as accuracy.
- When talking to non-technical stakeholders, resist compressing the explanation down to "95% chance it's in this range," that's exactly the misreading. Use the "if we repeated this, most of the ranges we'd get would contain the truth" framing instead, even though it takes a few more words.
You're presenting A/B test results to a product manager who asks: what's the difference between a p-value, a confidence interval, and effect size? Explain each concept in plain language, state what each does and does not tell you, and give an example sentence you would use to summarize results to a non-technical stakeholder.
Sample Answer
Direct answer
The p-value tells you whether the observed difference is unlikely to be pure chance under "no effect." The confidence interval (CI) tells you the range of effect sizes the data are consistent with. Effect size tells you how big the difference actually is, in units the business cares about. You need all three together: a tiny p-value with a tiny effect size is not a reason to act, and a wide confidence interval is a warning that the point estimate alone is not precise enough to bet on.
Structured elaboration
P-value
- What it is: the probability of seeing data this extreme (or more) if there were truly no difference between the groups.
- What it tells you: whether "no effect" is a poor explanation for what you observed.
- What it does NOT tell you: the probability the treatment works, or how large the effect is. A p-value of 0.001 and a p-value of 0.04 can come from effects of the same practical size, just with different sample sizes or noise.
Confidence interval
- What it is: a range of effect sizes that are plausible given the data and the model, at a chosen confidence level (typically 95%).
- What it tells you: both the size of the estimated effect and how precisely it's been measured. A narrow interval means the data pin the effect down tightly; a wide one means there's a lot of remaining uncertainty.
- What it does NOT tell you: it is not literally "a 95% probability the true value is in this specific interval." the 95% describes the long-run behavior of the procedure across repeated experiments, not a probability statement about this one realized interval.
Effect size
- What it is: the actual magnitude of the difference, absolute (percentage points) or relative (percent lift).
- What it tells you: whether the change is worth the engineering cost and rollout risk, independent of whether it's statistically significant.
- What it does NOT tell you: on its own, whether the estimate is reliable. An effect size without a confidence interval could be pure noise.
Worked example
A checkout test: baseline click-through p0=0.080, treatment p1=0.086, n=20,000 per arm.
Pooled test statistic:
z=2pˉ(1−pˉ)/np1−p0=2.1748⇒p≈0.029695% CI for the absolute difference (unpooled standard error):
(p1−p0)±1.96np0(1−p0)+np1(1−p1)=[0.0006, 0.0114](all values computed directly from these formulas with scipy.stats.norm)
Summary sentence for the PM: "Click-through rose from 8.0% to 8.6%, a 7.5% relative lift (p = 0.030). We're 95% confident the true absolute lift is somewhere between 0.06 and 1.14 percentage points. It's a real improvement, though the interval is wide enough that the low end is a modest win, not a blockbuster."
Trade-offs & pitfalls
- Presenting only the p-value invites the "significant equals big and certain" misread. Always pair it with the interval and the effect size in the business's own units.
- A p-value just under 0.05 with a confidence interval that barely excludes zero (as in this example, lower bound 0.0006) is a different story than a p-value of 0.0001 with a tight interval far from zero. Treat "significant" as a single bit of information, not the whole picture.
- Wide confidence intervals are common with realistic sample sizes and should be surfaced, not hidden. Narrowing the interval requires either more data or a less noisy metric, not a different way of describing the same data.
Define Type I and Type II errors in hypothesis testing. Give concrete business examples where each error would be costly. Explain how the significance level alpha and beta relate to these errors and how business costs can inform the choice of alpha.
Sample Answer
Direct answer
A Type I error is rejecting a true null hypothesis (a false positive: concluding there's an effect when there isn't one); a Type II error is failing to reject a false null (a false negative: missing a real effect). The significance level alpha (α) is the Type I error rate you're willing to accept; beta (β) is the Type II error rate, and power (1−β) is the chance of detecting a true effect. The right alpha is not a universal default. It should reflect how expensive a false positive is relative to a false negative for the specific decision being made.
Structured elaboration
The error matrix
| H0 actually true (no real effect) | H0 actually false (real effect exists) | |
|---|---|---|
| Test rejects H0 | Type I error (false positive), probability α | Correct decision, probability 1−β (power) |
| Test fails to reject H0 | Correct decision, probability 1−α | Type II error (false negative), probability β |
Business examples
- Type I error (false positive) is costly: shipping an expensive personalization system because a test showed a "significant" lift that was actually noise. You pay ongoing engineering and infrastructure cost, plus the opportunity cost of the team not working on something that actually helps.
- Type II error (false negative) is costly: killing a checkout redesign because the test result wasn't significant, when the redesign truly would have lifted conversion. You lose the revenue and any competitive advantage from never shipping it.
Setting alpha and beta from cost, not convention
For a fixed sample size, α and β trade off against each other: tightening α (fewer false positives) pushes power down (more false negatives) unless you also grow n. The textbook defaults (α=0.05, power =0.80) are a starting point, not a business decision. Set α lower when a false positive is expensive to reverse (a payments flow, a compliance-adjacent surface, a change that's hard to roll back); accept a higher α, or invest in a larger sample, when missing a true win is the more expensive mistake.
Worked example
A team wants to detect a 2 percentage-point lift on a checkout metric, baseline p0=0.20, p1=0.22, with n=2,000 users already collected per arm (fixed. This cohort won't grow before the decision is needed).
Holding n fixed, power at two different alpha values (two-sided z-test for two proportions):
| α | z1−α/2 | Power (1−β) | β |
|---|---|---|---|
| 0.05 | 1.9600 | 0.342 | 0.658 |
| 0.01 | 2.5758 | 0.153 | 0.847 |
(computed from the standard two-proportion power formula with scipy.stats.norm)
Insisting on α=0.01 for this "high-stakes" surface, without collecting more data, drops power from 34% to 15%: an 85% chance of missing a real 2-point lift entirely. If the team genuinely needs α=0.01, the fix is a larger n, not accepting a test that's this underpowered.
Trade-offs & pitfalls
- Alpha and beta are symmetric-looking numbers with asymmetric real-world stakes; picking α=0.05 by convention without asking what each error costs is the most common mistake.
- Tightening alpha without growing the sample size silently starves the test of power. Teams sometimes do this and then are surprised the "safer" test never finds anything.
- Failing to reject H0 is not proof of no effect. It can just as easily mean the test was underpowered for the true effect size, so a null result should be read alongside the achieved power, not treated as a settled "no."
A metric you must report is strongly right-skewed: most observations are small, but a small fraction of extreme values (large orders, slow requests, long task times) dominates the tail. Explain when you would report the mean versus the median and why skew pulls them apart. What additional statistics would you report to honestly communicate the tail, which visualizations would you use, and how would you present the numbers to non-technical stakeholders so they act on the right one?
Sample Answer
Report the median when a metric is strongly right-skewed and you want a single number that describes the typical case, because the median ignores how far outliers sit and just finds the middle. Report the mean when you actually care about totals or per-unit averages (like total cost, or a number you would multiply by a count to get a budget), because the mean is exactly what you get by dividing a total by a count, but say clearly that it is being pulled upward by the tail.
Structured elaboration
Why skew pulls mean and median apart
The mean is mean=n1∑i=1nxi, a total divided by a count, so every value (including extreme ones) pulls it in proportion to its size. The median is the middle value once the data are sorted, so it only cares about rank, not magnitude: a single extreme outlier can be ten times as large as the next value and it still only counts as "one value at the top," not something that inflates the middle.
What else to report
- Median plus interquartile range (IQR: the 25th to 75th percentile band), to describe the typical case and its normal spread.
- A tail percentile (90th or 95th), to be explicit about how bad the tail actually gets, since the median alone hides it.
- The count or share of extreme values, if there is a natural definition of "extreme" for the metric (an SLA breach threshold, for example).
- The mean, labeled explicitly as sensitive to the tail, when totals matter (total infrastructure cost is a sum, so the mean per-request cost times request count is the number that reconciles with the actual bill; the median per-request cost does not).
Visualizations
- A histogram (or a histogram with a log-scaled x-axis) to show the shape of the tail directly.
- A boxplot, which visually separates the median and IQR from the outlier points beyond the whiskers.
- A cumulative distribution plot (ECDF), which lets a reader answer "what fraction of cases finish within X" directly from the curve.
Presenting to non-technical stakeholders
Lead with a plain-language framing tied to the median and a tail statistic together ("half of tickets close within 3.5 hours; the slowest one in ten takes over a day"), rather than a single averaged number that hides the tail. Naming both numbers, and briefly why they differ, prevents the mean from being read as "what usually happens" when it is actually being dragged around by a small number of extreme cases.
Worked example
Ten support tickets, resolution time in hours: 1, 2, 2, 3, 3, 4, 5, 6, 8, 48 (nine ordinary tickets and one that took two full days to resolve).
mean=101+2+2+3+3+4+5+6+8+48=1082=8.2 hours
The median (the average of the 5th and 6th sorted values, 3 and 4) is 3.5 hours.
(verified with Python's statistics module: mean = 8.2, median = 3.5)
Eight of the ten tickets resolve in under 8 hours, and a ninth resolves at exactly 8 hours, yet the mean (8.2 hours) sits at or above nine of those ten values, entirely because of the one 48-hour ticket. Reporting "average resolution time: 8.2 hours" alone would make the typical support experience look meaningfully worse than it is for the vast majority of tickets; reporting "median 3.5 hours, with 1 ticket in 10 taking over a day" gives stakeholders both the typical case and an honest view of the tail.
Trade-offs & pitfalls
- Reporting only the mean on a skewed metric systematically misrepresents "typical," and can trigger the wrong reaction (over-investing in fixing a rare tail case while believing it describes most users).
- Reporting only the median can hide a real, costly tail problem from stakeholders who need to know it exists: a small share of very slow requests can still dominate total infrastructure cost or user pain, even if they do not move the median.
- Deciding to drop or cap ("winsorize") outliers changes what the mean means; that decision should be explicit and defensible, not a silent data-cleaning step.
- A percentile chosen to make a story sound better (cherry-picking p50 vs p90 vs p99 depending on which looks best) is a credibility risk. Pick and state the percentiles the audience actually needs before looking at how they turn out.
Explain the difference between statistical significance and practical (or clinical/business) significance. Provide an example where a tiny lift is statistically significant due to very large sample size but offers no practical business value, and describe how you would present the result to stakeholders.
Sample Answer
Direct answer
Statistical significance answers "is this difference unlikely to be pure chance," using the p-value or a confidence interval that excludes the null. Practical (business) significance answers "is this difference big enough to justify acting on it," using the effect size translated into business terms like revenue or cost. At large enough sample sizes, a real but tiny effect can be statistically significant while being economically marginal, so both questions have to be asked, not just one.
Structured elaboration
Why huge samples make this split visible
The standard error of an estimated difference shrinks as 1/n, so with enough data even a vanishingly small true difference eventually produces z large enough to cross the significance threshold. Statistical significance is a statement about whether the effect is distinguishable from zero; practical significance is a statement about whether the size of that effect, however precisely measured, changes what you should do.
How to present both together
- Report the point estimate and its confidence interval in business units (percentage points, dollars), not just the p-value.
- Translate the interval's endpoints, not just the point estimate, into a revenue or cost range. The point estimate can look meaningful while the low end of the interval does not.
- Compare that range to the cost of building, maintaining, and operationally supporting the change.
- State a recommendation explicitly (ship, hold, gather more data), rather than leaving "statistically significant" to imply "ship it."
Worked example
A platform-scale experiment: baseline conversion p0=0.10000, treatment p1=0.10020 (0.02 percentage points absolute, 0.2% relative lift), n=20,000,000 users per arm.
z=2pˉ(1−pˉ)/np1−p0=2.1072⇒p≈0.0351Statistically significant at α=0.05. The 95% CI for the absolute lift (unpooled SE):
(p1−p0)±1.96np0(1−p0)+np1(1−p1)=[0.0014, 0.0386] percentage points(computed directly from these formulas with scipy.stats.norm)
At an assumed $40 average order value, the incremental conversions and revenue implied by this arm-sized sample span the whole confidence interval, not just the point estimate:
| Lower bound (0.0014 pp) | Point estimate (0.02 pp) | Upper bound (0.0386 pp) | |
|---|---|---|---|
| Incremental conversions | ~280 | ~4,000 | ~7,720 |
| Incremental revenue | ~$11,200 | ~$160,000 | ~$308,800 |
If the personalization system needed to deliver this lift costs, say, $150,000 a year to build and maintain, the point estimate clears that bar but the low end of the honest interval does not. A single p-value or a single point estimate would have hidden that ambiguity.
Trade-offs & pitfalls
- Reporting "p<0.05, ship it" without the interval hides exactly the case above: a real, statistically-confirmed effect whose economic value is genuinely uncertain, not just small.
- Practical significance thresholds (minimum detectable effect, or a minimum dollar bar) should be set from the business case before running the test, not chosen after seeing a p-value, or the threshold itself becomes a form of after-the-fact rationalization.
- Recommending "don't ship" on a statistically significant but practically marginal result is a legitimate outcome. It should be framed as "the cost of building this exceeds even the upper end of the plausible benefit," not as a rejection of the statistics.
Unlock Full Question Bank
Get access to all 20 Statistical Inference and Hypothesis Testing interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.