Statistical Inference and Hypothesis Testing Questions

Direct answer

A confidence interval at $x^*$ captures uncertainty about the average outcome for that $x$ value (a population quantity); a prediction interval at $x^*$ captures uncertainty about one new individual observation at that $x$ value. The prediction interval is always wider because it has to account for both the uncertainty in estimating the regression line and the irreducible noise around that line, while the confidence interval only accounts for the first source.

Structured elaboration

Setup. Simple linear regression $y=\beta_0+\beta_1x+\varepsilon$, $\varepsilon\sim N(0,\sigma^2)$ i.i.d. (homoscedastic Gaussian errors). Let $\hat y(x^*)=\hat\beta_0+\hat\beta_1x^*$, $s^2=\mathrm{RSS}/(n-2)$, and $S_{xx}=\sum_i(x_i-\bar x)^2$.

Confidence interval for the mean response $E[y\mid x^*]$:

This only reflects how precisely you've pinned down the regression line itself, since $E[y\mid x^*]$ is a fixed number, not a random draw.

Prediction interval for a new observation $y_{\text{new}}$ at $x^*$:

The extra "$1$" inside the square root is exactly $\sigma^2$ worth of individual-observation noise that a single future draw carries and that never goes away, regardless of how much data you have.

How the widths change:

The practical takeaway: growing the sample only tightens the CI meaningfully; it barely tightens the PI, because the PI's floor is set by $\sigma$ itself, not by how well you know $\beta_0,\beta_1$.

Worked example

Simulated revenue-per-user regression (pinned seed, $n=40$):

import numpy as np
from scipy import stats

rng = np.random.default_rng(7)
n = 40
x = rng.uniform(1, 10, n)
y = 5.0 + 3.0 * x + rng.normal(0, 4.0, n)   # true b0=5, b1=3, sigma=4

xbar, Sxx = x.mean(), np.sum((x - x.mean())**2)
b1 = np.sum((x - xbar) * (y - y.mean())) / Sxx
b0 = y.mean() - b1 * xbar
resid = y - (b0 + b1 * x)
s = np.sqrt(np.sum(resid**2) / (n - 2))

xstar = 8.0
yhat = b0 + b1 * xstar
tcrit = stats.t.ppf(0.975, n - 2)
se_mean = s * np.sqrt(1/n + (xstar - xbar)**2 / Sxx)
se_pred = s * np.sqrt(1 + 1/n + (xstar - xbar)**2 / Sxx)

print(f"fitted: b0={b0:.3f}, b1={b1:.3f}, s={s:.3f}  (xbar={xbar:.3f}, Sxx={Sxx:.2f})")
print(f"yhat(x*=8.0) = {yhat:.3f},  t_crit(df={n-2}, .975) = {tcrit:.3f}")
print(f"95% CI  for mean response: [{yhat-tcrit*se_mean:.3f}, {yhat+tcrit*se_mean:.3f}]   half-width = {tcrit*se_mean:.3f}")
print(f"95% PI  for new observation: [{yhat-tcrit*se_pred:.3f}, {yhat+tcrit*se_pred:.3f}]  half-width = {tcrit*se_pred:.3f}")

Output:

fitted: b0=2.027, b1=3.569, s=3.218  (xbar=5.238, Sxx=282.42)
yhat(x*=8.0) = 30.577,  t_crit(df=38, .975) = 2.024
95% CI  for mean response: [29.091, 32.063]   half-width = 1.486
95% PI  for new observation: [23.895, 37.259]  half-width = 6.682

The prediction interval is about 4.5 times wider than the confidence interval at the same $x^*$, entirely because of the extra "1" term, since both intervals use the same $\hat y(x^*)$, $s$, $t_{n-2}$, $n$, and $(x^*-\bar x)^2/S_{xx}$.

Trade-offs & pitfalls

The most common mistake in practice is reporting a confidence interval when the actual question is about a single future user (e.g. "what revenue should we expect from a new customer with these attributes") - that calls for a prediction interval, and using the CI there systematically understates risk. The reverse mistake, reporting a PI when summarizing the average effect across a segment, needlessly widens the story and can make a real, precisely-estimated average look uncertain. Both formulas also assume homoscedastic Gaussian errors; if the residual spread grows with $x$ (heteroscedasticity) or is skewed, these closed-form intervals are wrong in both center and width, and a weighted-least-squares model or a bootstrap-based interval that estimates residual spread locally is the more defensible choice.

Direct answer

The null hypothesis (H0) is the "nothing changed" default you assume true until the data gives strong enough evidence to reject it. The alternative hypothesis (H1, or Ha) is the specific claim you're trying to find evidence for. A two-tailed test's alternative allows the effect to go either direction, so an increase or a decrease both count as evidence against H0. A one-tailed test's alternative commits in advance to only one direction, and only evidence in that direction can ever reject H0, however extreme the result is in the other direction, a one-tailed test treats it as not significant.

Structured elaboration

Defining the two hypotheses

• H0: the status-quo claim, typically "no difference" or "no effect," for example mu_new equals mu_old, or p_new equals p_old. It's what you'd believe by default absent evidence otherwise.
• H1 (or Ha): the claim there is reason to suspect and want the data to support. Hypothesis testing is structured to make H0 the thing you disprove, not the thing you prove; you never "accept H0," you either reject it or fail to reject it.

One-tailed versus two-tailed

A two-sided alternative is symmetric: H1 says the parameter is simply not equal to the null value, so evidence in either direction, higher or lower, counts against H0. A one-sided alternative picks a direction in advance, for example H1 says the new value is greater than the old one, and the entire significance level (say 5%) is allocated to that one tail of the distribution. That gives more power to detect an effect in that specific direction for the same sample size, at the cost of being structurally blind to an effect in the other direction.

Worked example

Testing whether a new checkout flow changes conversion rate, where p_old is the current rate and p_new is the new flow's rate:

Testing whether it changes the metric in either direction, appropriate when you'd act on either a lift or a drop, for example rolling back a regression just as readily as shipping an improvement:

Testing whether it increases the metric, appropriate only when a decrease and "no effect" would be handled identically, for example neither would ship:

Which to choose, and why

Choose the two-sided test whenever a result in either direction would change the decision, which is the common case in product work: a checkout redesign that decreases conversion is just as actionable, roll it back, as one that increases it, so a one-sided test built only to detect increases could let a real, harmful regression show up as "not significant" simply because the test structurally can't reject in that direction. Choose a one-sided test only when a result in the "wrong" direction is genuinely not actionable, and that decision was made before seeing the data. For example, a pure cost-reduction change to backend infrastructure where the only actions are "doesn't hurt the metric" versus "measurably hurts it," never "significantly helped," might reasonably justify a directional test. That's a narrow case, though, and switching to one-sided after peeking at a result that's "almost significant" two-sided is a form of p-hacking, not a legitimate use of the one-sided test.

Trade-offs and pitfalls

• One-sided tests have more power for a fixed alpha and sample size, but that power gain exactly matches giving up the ability to detect the opposite effect. It isn't a free lunch, it's a bet on direction made in advance.
• Choosing the tail after seeing which direction the data leans is a common way to quietly gain extra apparent significance without disclosing it. Always fix the direction, or commit to two-sided, before collecting data, ideally in writing.
• Default to two-sided unless there is a specific, pre-committed, defensible reason not to. When a one-sided test is used, say so explicitly in the write-up along with the pre-registered direction and rationale, so a reader can judge whether the choice was made honestly.

Direct answer

Model each minute's conversions as Binomial(n_t, p_t) and reduce the detection problem to a sequential likelihood-ratio test between "still at baseline p0" and "shifted to p1" (the smallest lift worth catching), computed as a running log-likelihood ratio. CUSUM accumulates that ratio into a one-sided statistic that resets at zero and alerts when it crosses a calibrated threshold; SPRT uses the same log-likelihood ratio but with two boundaries (accept shift, accept null, keep sampling) and gives a formal false-alarm and miss-rate guarantee via Wald's approximation. The two share one building block: CUSUM is "keep watching forever with one boundary," SPRT is "decide and reset" with both boundaries.

Structured elaboration

Detection statistic

Per-minute log-likelihood ratio for a shift from baseline rate p0 to the smallest detectable rate p1:

where n_t is impressions in minute t and k_t is conversions. Under H0 (still at p0), LLR_t has negative expectation and drifts down on average; under H1 (shifted to p1), it drifts up. That asymmetry is what makes accumulation work.

CUSUM

Alert the first minute S_t reaches a threshold h. The reset to zero means CUSUM only accumulates evidence for a shift, so a long stable period doesn't leave it stuck at a large negative deficit; it stays responsive the moment a real shift starts, unlike a raw cumulative sum with no floor.

SPRT

Same LLR_t, accumulated without the reset, with two boundaries:

Stop and declare a shift when cumulative LLR reaches A; stop and declare no shift when it drops to B; otherwise keep sampling. Alpha is the target false-alarm probability, beta the target miss probability (Wald's approximation; validate the actual operating characteristics empirically since grouped, discrete data only approximates the continuous-time theory).

Setting thresholds to control the false alarm rate

Pick alpha and beta from the business requirement first (for example, "at most one false page per ten days," "catch a 20% relative conversion drop within a few minutes on average with 95% probability"), derive A from Wald's formula, and reuse it as the CUSUM threshold h for a shared calibration. Then validate against replayed historical "known-quiet" traffic: run the detector over weeks of past data with no real incidents and measure the empirical false-alarm rate. If it exceeds target, raise the threshold and accept slower detection, rather than hand-tuning per launch.

Seasonality and traffic bursts

• p0 cannot be a single constant. Replace it with a seasonally varying baseline estimated from an exponentially weighted, time-of-day and day-of-week bucketed model, refit hourly, so the null the detector compares against tracks expected diurnal patterns instead of flagging every evening peak.
• A traffic burst changes n_t, not necessarily the rate. Because LLR_t scales with n_t, a burst with the rate unchanged doesn't bias the mean of LLR_t, but it inflates its variance episode to episode. Standardize the statistic using an online estimate of LLR_t's variance, computed with Welford's algorithm (a numerically stable way to update a running mean and variance one data point at a time, without storing the full history or accumulating floating-point error), so bursty-but-stable traffic doesn't trip the threshold more often than quiet traffic at the same calibrated alpha.
• Add a short cooldown window after any alert (10 to 15 minutes) so one real shift doesn't re-page every minute it stays elevated.

Scaling

• Stream ingestion (Kafka or Kinesis) feeds a stateful stream processor (Flink or Beam) keyed by (product, segment, minute), computing n_t and k_t with exactly-once semantics and checkpointed state for S_t per key.
• Run the detector per segment in parallel (geo, platform, experiment arm) plus one aggregate detector; segment-level alerts should roll up so a real global incident doesn't spam dozens of separate pages.
• Persist the LLR and CUSUM time series to a time-series store for replay-based recalibration and post-incident review; recalibrate the seasonal baseline model on a schedule, not manually.

Worked example

Baseline p0 = 0.050, minimum shift to catch p1 = 0.060 (a 20% relative lift), alpha = 0.01, beta = 0.05:

Six minutes of traffic, with a real shift beginning at minute 3:

Using threshold h = A = 4.5539: S_t hits 18.883 at minute 3, already above h, so the detector alerts at minute 3, the same minute the shift began. Minutes 1 and 2, still at baseline, each produce a large negative LLR and correctly reset S_t to 0 rather than accumulating toward a false alarm.

Trade-offs and pitfalls

• The threshold-versus-detection-speed trade-off is irreducible: a lower h (or larger alpha) detects the shift of interest faster on average but pages more often on noise. That is a business decision about the cost of a missed incident versus the cost of alert fatigue, not a purely statistical one.
• Picking p1, the alternative you calibrate against, implicitly defines what "small" means. A detector tuned to catch a 20% relative drop is intentionally slow, or blind, to a 2% drop. Don't oversell a single detector as catching any shift.
• CUSUM and SPRT assume a known p0 at each instant. If the seasonal baseline model has estimation error, that error leaks into every LLR_t and inflates the true false-alarm rate above what Wald's formula predicts, which is why empirical backtesting against quiet historical periods is not optional.
• A common wrong turn: setting h once at launch and never revisiting it. Traffic mix, seasonality, and business risk tolerance all drift; recalibrate on a schedule using replayed data, not a one-time tune.

Direct answer

For IID Bernoulli trials, the maximum likelihood estimator of $p$ is simply the sample mean, $\hat p = \bar X = \frac{1}{n}\sum X_i$. It's unbiased, has variance $p(1-p)/n$, and the standard normal-approximation 95% CI is $\hat p \pm 1.96\sqrt{\hat p(1-\hat p)/n}$. That CI can behave badly (even producing bounds outside $[0,1]$) when $n$ is small or $p$ is near 0 or 1, because the normal approximation to a discrete, bounded variable breaks down exactly in that regime.

Structured elaboration

Deriving the MLE

The likelihood for $n$ IID Bernoulli($p$) draws is:

Taking logs:

Differentiating with respect to $p$ and setting to zero:

Solving:

Unbiasedness

so $\hat p$ is unbiased for every finite $n$.

Variance and standard error

Since the $X_i$ are IID with $Var(X_i) = p(1-p)$:

Normal-approximation 95% CI

By the CLT, $\hat p$ is approximately $N(p, \, p(1-p)/n)$ for large $n$, giving the Wald interval:

Limitations

• For small $n$, the discrete binomial distribution is poorly approximated by a continuous normal.
• For $p$ near 0 or 1, the sampling distribution of $\hat p$ is skewed (bounded at 0 or 1), so a symmetric normal interval can overshoot the boundary, producing a lower bound below 0 or an upper bound above 1, which is nonsensical for a probability.
• Coverage of the nominal 95% Wald interval is often noticeably below 95% in these regimes ("the interval doesn't actually cover the truth 95% of the time").
• Better alternatives: the exact Clopper-Pearson interval (guaranteed coverage, can be conservative), the Wilson score interval (better calibrated, doesn't leave $[0,1]$), or Agresti-Coull (a simple, well-behaved adjustment).

Worked example

Two pinned-seed simulations to show both the well-behaved and the breakdown regime:

import numpy as np
rng = np.random.default_rng(seed=11)

# Well-behaved: n=50, p_true=0.3
n = 50
x = rng.binomial(1, 0.3, n)
p_hat = x.mean()                              # 0.22 (11 successes / 50)
se = np.sqrt(p_hat*(1-p_hat)/n)               # 0.0586
ci = (p_hat - 1.96*se, p_hat + 1.96*se)       # (0.105, 0.335)

# Breakdown regime: small n, p near 0
x2 = np.array([0,0,0,0,0,0,0,0,0,1])          # p_hat = 0.1, n=10
p_hat2 = x2.mean()
se2 = np.sqrt(p_hat2*(1-p_hat2)/10)           # 0.0949
ci2 = (p_hat2 - 1.96*se2, p_hat2 + 1.96*se2)  # (-0.086, 0.286)

In the well-behaved case ($n=50$, true $p=0.3$), the estimate is $\hat p = 0.22$ with a 95% Wald CI of $(0.105, 0.335)$, a sensible interval fully inside $[0,1]$. In the breakdown case ($n=10$, $\hat p = 0.1$, one success out of ten), the Wald CI comes out to $(-0.086, 0.286)$, a lower bound that is a negative probability, which is exactly the failure mode the limitations section warns about. This is a mechanical consequence of the formula, not a coding bug: the normal approximation simply isn't valid there.

Trade-offs & pitfalls

• The MLE being unbiased doesn't mean the Wald CI built from it is well-calibrated. These are separate properties; don't conflate "unbiased point estimate" with "trustworthy interval."
• Reporting a Wald CI that includes values outside [0,1] without flagging it is a common interview red flag. A senior answer catches this and names the fix (Wilson, Clopper-Pearson, or bootstrap) rather than just reporting the number.
• Clopper-Pearson guarantees coverage but is conservative (intervals wider than necessary), which matters if you're using CI width to size a downstream decision.
• For a hypothesis test of $p = p_0$ (rather than just a CI), you'd typically use the same test statistic, but note that the Wald-based test and the score-based test (which doesn't plug in $\hat p$ under the null) can disagree in exactly the same small-$n$/extreme-$p$ regime.

Direct answer

A frequentist 95% confidence interval is a statement about the procedure: if you repeated the same sampling and interval-construction process many times, 95% of the resulting intervals would contain the true, fixed parameter. It says nothing about the probability that this one interval, from this one dataset, contains the parameter. A Bayesian 95% credible interval is a direct probability statement about the parameter itself: given the observed data and a prior, there is a 95% probability the parameter lies in that range. The two coincide numerically when the prior is flat or the data dominate, but their meaning is different even then.

Structured elaboration

Why they can differ. The Bayesian interval blends the observed data with the prior, weighted by their relative precision (inverse variance). A confident (low-variance) prior pulls the credible interval toward the prior mean and narrows it; a weak (high-variance, close to flat) prior lets the data dominate and the credible interval approaches the frequentist CI.

Worked example

Suppose an experiment produces an effect estimate $\hat\theta = 2.0$ with standard error $SE = 1.0$.

Frequentist 95% CI:

Bayesian 95% credible interval, with a prior belief centered at zero and moderately informative, $\theta \sim N(0, 1^2)$, combined with the data via the standard normal-normal conjugate update, conjugate meaning the prior and the posterior come from the same distribution family (normal in, normal out), which is what lets the update be a closed-form formula instead of numerical integration (treating $SE=1.0$ as the data's known standard deviation):

(All values verified by direct computation of the conjugate-update formulas.) The frequentist interval, $(0.04, 3.96)$, is entirely positive and would typically be read as "a positive effect." The Bayesian interval, $(-0.39, 2.39)$, is pulled toward zero by the prior and includes zero, because the prior said the effect was probably small and the single data point wasn't overwhelming enough to fully override that. Neither interval is "wrong": they're answering different questions (what would repeated sampling do, versus what should you believe given this data and this prior).

Why this matters for stakeholders

A credible interval lets you say, in plain language, "there's a 95% probability the true lift is between X and Y," which is exactly what most non-technical stakeholders assume a confidence interval means anyway. That mismatch, people reflexively read frequentist CIs as if they were credible intervals, is the single most common misinterpretation in practice, and it's worth correcting explicitly rather than exploiting it. If the team has real prior information (results from a similar past experiment, a strong mechanistic reason to expect a small effect), a Bayesian framing makes that information visible and auditable instead of leaving it as an unstated bias in how someone reads the frequentist number.

Trade-offs & pitfalls

• Never claim a frequentist CI has a 95% chance of containing the parameter for the specific interval in hand; that statement describes the long-run behavior of the method, not this instance.
• A Bayesian interval is only as trustworthy as its prior. A poorly chosen or overconfident prior can bias the interval in a way that's easy to miss if the prior isn't stated and justified alongside the result.
• With large samples the data dominate and a weak prior makes the two intervals nearly identical, so in high-traffic product experiments the distinction is often more about how you communicate the result than about a different numeric answer.