Statistical Inference and Hypothesis Testing Questions

Three common sampling biases in user research and product analytics are selection bias, survivorship bias, and non-response bias. Each systematically skews who or what ends up in a dataset relative to the population the team actually wants to understand, and each has a distinct, practical mitigation.

Structured elaboration

Selection bias

The recruitment or logging mechanism itself favors certain users over others, so the sample never represented the target population to begin with.

• Product example: recruiting research participants exclusively through an in-app "give feedback" banner over-represents highly engaged power users and misses casual or lapsed users entirely.
• Mitigation: diversify recruitment channels (email, support tickets, targeted outreach to inactive users, paid panels), and when analyzing logged data, weight or stratify results by known usage segments so heavy users do not dominate the conclusion.

Survivorship bias

The dataset only contains entities that "survived" some earlier filter, silently excluding the ones that dropped out, which are often the most informative cases for understanding why people leave.

• Product example: analyzing feature usage only among currently active accounts misses everyone who abandoned the product during onboarding, so the data looks healthier than the funnel actually is.
• Mitigation: instrument and retain events from the full funnel, including users who never became active, and deliberately recruit recent churners or drop-outs for exit interviews rather than only studying who remains.

Non-response bias

Among people invited to respond (a survey, an in-app prompt), the ones who actually respond differ systematically from the ones who do not, in ways related to the very thing being measured.

• Product example: a satisfaction survey sent right after a purchase mostly reaches people who are already positive enough to engage further, biasing an NPS score upward relative to the full user base.
• Mitigation: keep surveys short to raise response rates, follow up with non-responders (or a random subsample of them), and compare respondent demographics or behavior to the overall population to check how representative the response set actually is; apply response weighting when a gap is found.

Worked example

A team wants to know why a new onboarding flow has a 40% completion rate. If they only interview users pulled from a "rate your experience" prompt shown after successful completion, they have compounded two biases at once: survivorship (only completers are in the sampling frame at all) and non-response (only the completers who bothered to respond to the prompt are actually interviewed). The resulting interviews will describe the experience of the roughly 40% who finished and felt strongly enough to respond, and say nothing about the 60% who dropped out, which is the group the team most needed to understand. Fixing this requires reaching users at the point of drop-off (an exit-intent prompt, or a follow-up email to users who started but did not finish), not just the users who reached the end.

Trade-offs & pitfalls

• These three biases often compound in the same study (as above), so checking for one is not enough; ask separately who was even eligible to be seen, who dropped out along the way, and who actually responded.
• Fixing selection bias with broader recruitment can raise the cost and time of a study; weighting analysis by known segments is often a cheaper partial fix when full re-recruitment is not practical.
• Comparing respondents to the overall population only catches non-response bias on variables you can actually measure for non-respondents (demographics, usage tier); it cannot catch bias on unmeasured traits.
• Passive telemetry (event logs) is not automatically unbiased just because it does not rely on someone opting in; the underlying feature or platform itself can have selection effects, such as only being available on newer devices.

Direct answer

Use a z-test only when the population standard deviation is genuinely known in advance, which is rare in practice. Use a t-test whenever the standard deviation has to be estimated from the sample itself, which is the normal situation, and this holds regardless of sample size. Sample size affects a different thing: how close the t and z critical values are to each other and how much you can lean on the Central Limit Theorem if the underlying data isn't very normal.

Structured elaboration

Known vs. unknown variance. This is the formal criterion. If $\sigma$ is known (rare outside quality-control settings with a long-established process variance), use z. If $\sigma$ is estimated from the sample as $s$ (the normal case), use t with $df = n-1$; the t-distribution's heavier tails are exactly the correction for the added uncertainty of estimating $\sigma$ rather than knowing it.

Sample size's actual role. As $n$ grows, $t_{n-1}$ converges to $z$, so at large $n$ the choice barely changes the numeric answer, which is why "just use z for n≥30" survives as a practical shortcut even though it's not the formal reason. Separately, larger $n$ also makes the Central Limit Theorem a stronger justification for treating the sampling distribution of the mean as approximately normal even when the raw data isn't, which matters for the validity of either test, not for the t-vs-z choice itself.

Comparing two means: pooled vs. Welch's t. If assuming the two groups have equal population variances, use the standard (pooled) two-sample t-test. If variances might differ, and there's rarely a strong reason to assume they're equal, use Welch's t-test, which does not assume equal variances and adjusts the degrees of freedom accordingly. Welch's costs very little power when variances actually are equal but protects against inflated Type I error when they aren't, which is why it's the safer default.

Robustness. t-tests are reasonably robust to mild-to-moderate non-normality once $n$ is moderate (roughly 30+ per group), thanks to the CLT. They're not robust to strong skew or heavy outliers at small $n$, where a few extreme points can dominate both the mean and the variance estimate.

Worked example: how close t and z actually are, by sample size

(All values from scipy.stats.t.ppf(0.975, df), verified directly.) At $df=5$ the t critical value is about 31% larger than z, meaningfully widening the interval or raising the bar for significance; by $df=60$ the gap has shrunk to about 2%. This is the practical justification behind "large n, t and z are basically the same," even though the theoretically correct reason to pick t is always "σ is estimated," not "n is small."

When variances are unknown and sample sizes are small: the actual procedure

1. Look at the data: a histogram or Q-Q plot per group, and check for obvious outliers.
2. If approximate normality looks plausible, default to Welch's t-test (not pooled, unless there's a specific reason to believe variances are equal, such as both groups measuring the identical underlying process).
3. If normality looks clearly violated, or the sample is extremely small (single digits per group) with visible skew, switch to a nonparametric alternative like the Mann-Whitney U test, or use a bootstrap for the confidence interval and p-value instead of the t-distribution's analytic formula.

Trade-offs & pitfalls

• Defaulting to the pooled t-test "because it's the classic one" without checking the equal-variance assumption is a common shortcut that inflates false positives when variances genuinely differ; Welch's is essentially free insurance against this.
• Small samples with heavy skew or outliers can pass a superficial normality check while still producing an unreliable t-test; this is where nonparametric or bootstrap alternatives earn their keep, not just as a formality but as a real fix.
• The "n≥30 use z" heuristic is useful as a rule of thumb but wrong as a justification; it should never be given as the reason to choose z over t in an interview answer, since the real criterion is whether σ is known.

Direct answer

Marginal (statistical) independence means two variables carry no information about each other on their own: $P(X,Y) = P(X)P(Y)$. Conditional independence means that once you know a third variable $Z$, the remaining information one gives about the other disappears: $P(X,Y\mid Z) = P(X\mid Z)P(Y\mid Z)$. The two can point in opposite directions: a confounder can make two marginally independent variables look dependent once you fail to condition on it, while conditioning on a collider can manufacture dependence between two variables that were genuinely independent to begin with. Getting this backwards in a feature-selection pipeline silently corrupts both predictive and causal conclusions.

Structured elaboration

Definitions

Neither implies the other. Two variables can be marginally dependent but conditionally independent given a confounder (the confounder explains away the association), or marginally independent but conditionally dependent given a collider (conditioning induces a spurious association).

Confounder vs. collider: opposite implications for adjustment

Worked example

Setting: $X$ = a user's click count (standardized), $Y$ = purchase amount (standardized), genuinely generated independently of each other. $Z$ = whether the user got flagged into a "power user" analysis cohort, where the flagging rule looks at both clicks and purchases (a collider: an effect of both $X$ and $Y$, not a cause of either).

import numpy as np
rng = np.random.default_rng(7)
n = 5000

X = rng.normal(0, 1, size=n)   # clicks, independent by construction
Y = rng.normal(0, 1, size=n)   # purchase amount, independent by construction

# Z is a collider: selected into the cohort if clicks + purchases (plus noise) is high
selected = (X + Y + rng.normal(0, 0.5, size=n)) > 1.0

corr_marginal = np.corrcoef(X, Y)[0, 1]                    # 0.0146
corr_conditional = np.corrcoef(X[selected], Y[selected])[0, 1]  # -0.5323

Marginally, $\text{corr}(X,Y) = 0.015$, essentially zero, matching how the data was generated: clicks and purchases are independent. But restricted to the 1,246 users selected into the "power user" cohort, $\text{corr}(X,Y) = -0.532$, a strong negative association that does not reflect any real relationship. It's an artifact of the selection rule: among users who cleared the combined bar, a user with fewer clicks almost had to have made up for it with a larger purchase (and vice versa) to have been selected at all. This is the same structure behind the classic "why do successful people in a field seem to trade off looks and talent" collider illustration, applied to a product-analytics cohort filter.

Implications for feature selection and causal interpretation

• If $Z$ is a confounder, omitting it from a predictive or causal model leaves a spurious $X$-$Y$ association in the data unexplained, and any causal claim about $X \to Y$ is biased.
• If $Z$ is a collider, including it as a feature or, more subtly, restricting the training sample to a cohort defined by $Z$ (like "power users," "converted users," or "users who completed onboarding") induces exactly the spurious association shown above, and a model trained on that filtered population will learn a relationship between $X$ and $Y$ that reverses or vanishes once applied to the unfiltered population.

Trade-offs & pitfalls

• You cannot tell confounder from collider by looking at correlations alone - the same numeric pattern (variable $Z$ correlated with both $X$ and $Y$) can be either structure. You need a causal assumption about the direction of the arrows, typically encoded in a DAG (directed acyclic graph: a diagram whose arrows show which variable causes which, with no cycles), before deciding whether to adjust for $Z$.
• Selection-induced collider bias is easy to introduce by accident. Any pipeline step that filters rows based on an outcome-adjacent variable ("only keep users who converted," "only keep completed sessions") is implicitly conditioning on a collider if that filter variable is itself downstream of both features under study.
• Feature-importance tools don't distinguish real signal from collider-induced signal. A feature that shows strong importance only within a filtered subpopulation is a signal to check the sampling/filtering logic, not necessarily a signal that the feature is causally meaningful.
• Testing conditional independence from observational data is itself imperfect - standard partial-correlation tests assume linear/Gaussian relationships and can miss nonlinear (in)dependence; nonparametric conditional independence tests exist but are harder to calibrate and much more data-hungry.

Direct answer

Naive p-values assume the sample size was fixed in advance; checking results repeatedly and stopping as soon as significance appears ("optional stopping") inflates the true Type I error far above the nominal level, because you're implicitly running many correlated tests and keeping the lucky one. The Sequential Probability Ratio Test (SPRT) and martingale-based always-valid p-values are the two formal fixes: SPRT pre-specifies likelihood-ratio stopping boundaries with guaranteed error rates under a correctly specified model, while always-valid p-values use test martingales to guarantee the error bound holds no matter when or how often you look, even under adversarial stopping.

Structured elaboration

SPRT (Wald)

Sequentially accumulate the likelihood ratio between the alternative and null models as data arrives, and stop the first time it crosses one of two pre-set boundaries:

Stop and accept $H_1$ if $LR_t \geq A$; stop and accept $H_0$ if $LR_t \leq B$; otherwise keep sampling. Wald's boundaries, calibrated to target Type I rate $\alpha$ and Type II rate $\beta$:

Martingale-based always-valid p-values

Informally, a martingale is a running quantity where, given everything observed so far, your best guess for its next value is exactly its current value - like tracking your wealth in a fair betting game with no built-in edge: some rounds you win, some you lose, but there's no way to pick a stopping time that systematically nets you ahead of where you already are. Formally, construct a nonnegative process $M_t$ that is a martingale under $H_0$ (i.e. $E_0[M_t] = 1$ for all $t$, often built as a mixture or "betting" likelihood ratio). Define $p_t = 1/M_t$. By Ville's inequality:

Ville's inequality bounds the running maximum of $M_t$ over all of time, not just its value at one fixed $t$. That's exactly what makes this safe against optional stopping: whatever (random, data-dependent) time $\tau$ you eventually decide to stop at, $M_\tau$ is just one particular value the process takes on, so it can never exceed $\sup_{t \geq 0} M_t$. If the running maximum stays below $1/\alpha$ with probability at least $1-\alpha$, then the value at whatever moment you happened to stop is automatically below $1/\alpha$ with at least that same probability, no matter how you chose $\tau$ - which means $P_0(p_\tau \leq \alpha) \leq \alpha$ for any stopping time $\tau$, including one chosen by looking at the data as it arrives. This is the formal antidote to optional stopping: the guarantee holds regardless of when or why you decided to stop, because the martingale property $E_0[M_t]=1$ gives you no systematic edge to exploit by choosing your stopping point.

Assumptions

Choosing stopping boundaries

For SPRT, calibrate $A$ and $B$ to the target Type I rate $\alpha$ and Type II rate $\beta$ using Wald's approximation (shown numerically below). For always-valid p-values, the decision threshold is simply $1/\alpha$ on the martingale scale; the design choice is which mixture prior over effect sizes (or which "betting fraction" schedule) maximizes the growth rate of $M_t$ under plausible true alternatives, since that determines expected sample size to reach a decision.

Estimating long-run Type I error under model misspecification

Analytic guarantees are conditional on the model being correctly specified; under misspecification neither method's stated error rate is automatically trustworthy. Practical mitigations:

• Prefer nonparametric or "universal inference" martingale constructions that make fewer distributional assumptions than a fully parametric SPRT. Universal inference builds a valid test by splitting the data in two: fit the alternative hypothesis's parameters on one half, then evaluate the likelihood ratio using those fitted parameters on the other half, which sidesteps needing a fully pre-specified point alternative the way the plain SPRT above does, at the cost of using only half the data for each piece.
• Use mixture-SPRT (a prior over plausible alternatives rather than one fixed point alternative) to reduce sensitivity to the exact alternative specified.
• Run fully specified stress simulations under a range of plausible dependence structures and alternative data-generating processes (with pinned parameters and seeds) to empirically estimate the realized Type I rate before trusting the nominal $\alpha$ in production.
• Track the empirical false-rejection rate across live tests over time and recalibrate thresholds if it drifts above nominal.

Worked example

Target Type I rate $\alpha = 0.05$, Type II rate $\beta = 0.20$ (80% power):

alpha, beta = 0.05, 0.20
A = (1-beta)/alpha   # 16.0
B = beta/(1-alpha)   # 0.21052631578947367

In log-likelihood-ratio terms (numerically more convenient to accumulate than the raw ratio), the running sum $\log LR_t$ stops once it crosses:

To see how this translates into an expected sample size: Wald's approximation says the expected number of observations to reach a decision under $H_1$ is roughly $\log A$ divided by the expected per-observation log-likelihood-ratio increment under $H_1$. If a given test's per-observation increment averages 0.4 under the true alternative (a stand-in value; the real number comes from the specific $f_0, f_1$ in use):

import math
logA = math.log(16.0)          # 2.772588722239781
mean_llr_increment = 0.4       # stand-in per-observation E[log LR | H1]
expected_n_h1 = logA / mean_llr_increment   # 6.931471805599452

This is the concrete payoff of SPRT's efficiency claim: under a correctly specified model with this much per-observation signal, the test expects to reach a decision in about 7 observations, far fewer than a fixed-sample design calibrated to the same $\alpha, \beta$ would need, precisely because SPRT stops as soon as the evidence is sufficient rather than waiting for a pre-committed sample size.

Always-valid (martingale) worked example

The always-valid method uses the same kind of per-observation evidence as SPRT, but only needs a single threshold, $1/\alpha$, because it isn't calibrated to a target power $\beta$ the way SPRT's two boundaries $A$ and $B$ are:

Reusing the same stand-in per-observation signal as above ($E[\log LR \mid H_1] = 0.4$), the expected number of observations to cross this single threshold is:

import math
alpha = 0.05
threshold = 1/alpha                    # 20.0
log_threshold = math.log(threshold)    # 2.995732273553991
mean_llr_increment = 0.4               # same stand-in as the SPRT calc above
expected_n_martingale = log_threshold / mean_llr_increment   # 7.489330683884977

To make $M_t$ itself concrete, trace a hand-picked sequence of per-observation log-likelihood-ratio increments (a real run would compute these from the actual data and betting/mixture construction; here they stand in for a plausible realization under $H_1$): 0.5, 0.3, 0.6, 0.4, 0.5, 0.7. Accumulating them into $\log M_t$:

import math
log_threshold = math.log(1/0.05)   # 2.995732273553991, same threshold as above

increments = [0.5, 0.3, 0.6, 0.4, 0.5, 0.7]
cum = 0.0
running_log_Mt = []
for inc in increments:
    cum += inc
    running_log_Mt.append(round(cum, 4))
# running_log_Mt = [0.5, 0.8, 1.4, 1.8, 2.3, 3.0]
crossed_at = next(i+1 for i, v in enumerate(running_log_Mt) if v >= log_threshold)   # 6

At round 6, $\log M_6 = 3.0$ exceeds $\log(1/\alpha) = 2.9957$, so $M_6 = e^{3.0} \approx 20.09$ has just crossed the $1/\alpha = 20$ threshold: stop and reject $H_0$, with the guarantee $P_0(p_\tau \leq 0.05) \leq 0.05$ holding even though the stopping decision was made by watching the process evolve round by round, exactly the peeking that would invalidate a fixed-sample p-value.

Trade-offs & pitfalls

• SPRT is efficient but brittle. Under a correctly specified model, SPRT minimizes expected sample size to a decision - but a misspecified likelihood (wrong noise model, unmodeled dependence) silently breaks the stated error guarantees with no visible symptom until you audit the realized error rate.
• Always-valid p-values trade some power for robustness to when you stop, not to model misspecification. They solve the optional-stopping problem specifically; they don't rescue you from a wrong likelihood model any more than SPRT does.
• Mixture/mixture-SPRT approaches reduce brittleness but add a design choice (the mixing prior) that itself needs justification - a poorly chosen mixture prior can meaningfully hurt expected time-to-decision even while error guarantees remain valid.
• Neither method is a substitute for pre-registration discipline. Formal always-valid tools let you monitor continuously, but teams still benefit from pre-specifying the primary metric and minimum meaningful effect before looking at data, since post-hoc metric switching is a separate failure mode these tools don't address.

Direct answer

Use one-way ANOVA instead of running six pairwise t-tests across four regions because ANOVA tests a single global null hypothesis (all four region means are equal) with one test, controlling the overall false-positive rate; running six separate pairwise t-tests at 0.05 each lets the false-positive rate compound well past 5%. A significant F-statistic means the variance between region means is larger than would be expected from within-region noise alone, but it only tells you some region differs, not which one, so it's followed by a post-hoc test like Tukey's HSD to identify the specific pairs.

Structured elaboration

Why not just run pairwise t-tests. With 4 regions there are $\binom{4}{2}=6$ possible pairs. Running each pairwise comparison at $\alpha=0.05$ independently, the chance of at least one false positive across all six, assuming the tests were independent, is:

So even with genuinely equal region means, there's roughly a 26.5% chance of at least one "significant" pairwise result purely by chance, more than five times the nominal 5% rate. ANOVA avoids this by testing one combined hypothesis:

Interpreting the F-statistic. $F$ is the ratio of between-group variance to within-group variance:

A large $F$ means the region means are more spread out relative to each other than the natural noise within each region would predict, and a significant p-value (compared to the $F$-distribution with the appropriate numerator/denominator degrees of freedom) means it's unlikely this spread arose by chance alone. Critically, rejecting $H_0$ only supports "at least one region differs from at least one other," not a specific pairwise claim, which is exactly why a post-hoc step is needed.

Assumptions to check. Independent observations; approximately normal residuals within each group (or large enough samples per group for the CLT); homogeneity of variance across groups, checkable with Levene's or Bartlett's test. Report an effect size like $\eta^2$ alongside the F-test, since a significant F with a tiny effect size may not be practically meaningful.

Post-hoc methods. Lead with Tukey's HSD: it's the standard default for all-pairwise comparisons after a significant ANOVA, controlling the family-wise error rate across all pairs simultaneously while providing simultaneous confidence intervals (its Tukey-Kramer variant handles unbalanced group sizes). That's the one answer most interviews are actually testing for. Beyond that baseline, a few named alternatives are worth knowing exist but are depth rather than the first answer: Bonferroni correction (dividing $\alpha$ by the number of comparisons) is simpler but more conservative, costing power; Holm-Bonferroni is uniformly more powerful than plain Bonferroni while still controlling the family-wise error rate; and Benjamini-Hochberg trades some family-wise protection for more power when the concern is false discovery rate across many comparisons rather than strict family-wise control.

Worked example

Satisfaction scores for 5 customers in each of 4 regions:

One-way ANOVA gives $F \approx 22.38$, $p \approx 0.000006$ (verified with scipy.stats.f_oneway), strongly rejecting $H_0$: the region means are not all equal. The group means (74.0, 81.6, 70.4, 77.0) suggest Region 2 is notably higher and Region 3 notably lower, but confirming exactly which pairs differ, and by how much, requires the post-hoc step: running Tukey's HSD on this data (rather than six uncorrected pairwise t-tests, which at $n=5$ per group and this much separation would likely also individually reach significance, but without the family-wise error control that makes the conclusion trustworthy).

Trade-offs & pitfalls

• A significant F only earns you "something differs," not "which pairs differ." Reporting a significant ANOVA as if it identifies a specific region without running the post-hoc step is a common overreach.
• Running naive uncorrected pairwise t-tests after a significant ANOVA "just to see" reintroduces exactly the inflated false-positive problem ANOVA was meant to solve; the whole point of the post-hoc method is that it controls for the multiple comparisons being made.
• If the equal-variance assumption clearly fails (e.g. Levene's test is significant), switch to Welch's ANOVA, which doesn't assume equal variances, rather than proceeding with standard ANOVA and hoping it's robust enough; if normality is a serious concern instead, the Kruskal-Wallis test (the rank-based, nonparametric analogue of one-way ANOVA) is the fallback.