Statistical Inference and Hypothesis Testing Questions

Direct answer

Frequentist A/B testing treats the true conversion rates as fixed unknowns and describes uncertainty through the long-run behavior of a procedure (p-values, confidence intervals) over repeated sampling. Bayesian A/B testing treats the parameters themselves as random, combines a prior belief with the observed data via Bayes' rule, and produces a posterior distribution you can make direct probability statements about. The practical trade-off is interpretability and sequential flexibility (Bayesian) versus simplicity, cheap computation, and no dependence on a prior (frequentist).

Structured elaboration

Credible interval vs. confidence interval

Role of the prior

Priors encode existing knowledge or act as regularization. An informative prior (built from historical baselines) can stabilize estimates when data are sparse, but a badly misspecified prior can bias the posterior, especially with a small sample; a weakly informative or hierarchical prior is the usual compromise, letting data dominate once there's enough of it while still borrowing strength when there isn't. Frequentist methods have no explicit prior, though penalization or shrinkage techniques play an analogous stabilizing role.

Computational trade-offs

• Frequentist tests (z-test, t-test) are closed-form and essentially free to compute at any scale, which is why they're the default for real-time dashboards across many simultaneous experiments.
• Bayesian inference can require Markov Chain Monte Carlo (MCMC) sampling for complex models, but for the common case of a Beta-Binomial conjugate model (Beta prior, Binomial likelihood for a conversion rate), the posterior is closed-form and just as cheap as the frequentist calculation.
• Bayesian sequential updating (recomputing the posterior as new data streams in) does not by itself protect against inflated error rates from repeated peeking; the peeking problem still needs a stopping rule that accounts for it (e.g. a pre-specified decision rule on the posterior, not "stop whenever it looks good").

Worked example

A team runs many small, sequential experiments on low-traffic feature variants with sparse conversion events, and wants to say "there's a 95% probability variant A beats variant B" to a product manager, plus borrow strength across related past experiments on similar features.

Using a Beta-Binomial model: Beta$(\alpha_0,\beta_0)$ prior on each variant's conversion rate, updated with observed successes/failures to a Beta posterior. Concretely, suppose variant A converts 22 of 50 visitors and variant B converts 14 of 50, with a weakly informative Beta(1,1) (uniform) prior on each arm:

import numpy as np
rng = np.random.default_rng(42)
a_A, b_A = 23, 29
a_B, b_B = 15, 37

samples = 1_000_000
pA = rng.beta(a_A, b_A, samples)
pB = rng.beta(a_B, b_B, samples)
p_a_beats_b = (pA > pB).mean()   # 0.9503

Both posteriors are closed-form Betas, so sampling is just the simplest way to compare two distributions; an exact numerical integral would give the same answer. With these two posteriors, the direct statement "$P(p_A > p_B \mid \text{data}) = 0.95$" answers exactly the question the PM is asking, which the frequentist p-value does not, and now it traces back to real counts (22 conversions of 50 for A, 14 of 50 for B) instead of being asserted.

Trade-offs & pitfalls

• Bayesian output is easier to explain to a non-technical stakeholder ("95% probability A is better") than a p-value or a frequentist confidence interval, but that ease of interpretation depends on the audience trusting the prior, which is a modeling choice, not a fact.
• An informative prior chosen after seeing early results (rather than from genuinely prior information) quietly reintroduces the same p-hacking risk Bayesian methods are sometimes assumed to avoid.
• Reaching for full MCMC when a conjugate model would do adds compute cost and diagnostic burden (convergence checks, effective sample size) with no benefit; match the model complexity to the actual question.
• Neither framework is immune to multiple-comparisons risk when running "many small, sequential" experiments and reporting only the ones that look good; that discipline (pre-specification, correction, or hierarchical pooling) is needed regardless of which paradigm you use.

State the null as "no difference in the metric between the arms" and let the alternative's direction match what the team will actually act on. A one-sided test is defensible only when a decrease in the metric would never change the decision (you would never ship a change that hurts the metric, regardless of a "significant" drop); otherwise use two-sided. The relative cost of a false positive (shipping a change that does not really help, or worse) versus a false negative (missing a real improvement) should directly set alpha and target power, and therefore the sample size and rollout gate.

Structured elaboration

1. Formulating H0 and H1

Let p be the true population rate of the metric of interest (for example, conversion or click-through). H0 states there is no effect:

$H_0: p_{\text{treatment}} = p_{\text{control}}$

If the team will only ship on evidence of improvement:

$H_1: p_{\text{treatment}} > p_{\text{control}} \quad \text{(one-sided)}$

If a regression is also actionable (you would roll back on a significant drop, or the change touches something safety- or compliance-sensitive):

$H_1: p_{\text{treatment}} \neq p_{\text{control}} \quad \text{(two-sided)}$

The population is the unit actually randomized (usually users or sessions in the experiment's traffic), not "everyone who could ever use the product." Say which population the inference is scoped to.

2. One-sided vs two-sided: how to decide

The one-sided vs two-sided choice must be locked before looking at results. Choosing it after peeking at the sign of the effect inflates the true Type I error rate above the stated alpha.

3. Costs, alpha, power, and the rollout

• Alpha (P(reject H0 given H0 true)) is the cost of a false positive: shipping a change that does nothing or hurts, paid in engineering effort, UX regression risk, or reduced trust in future experiments. Lower alpha (e.g. 0.01 instead of 0.05) when shipping is expensive to reverse.
• Power, 1 minus beta (P(reject H0 given H1 true)), is protection against a false negative: missing a real improvement. Higher target power (0.8 to 0.9) when the upside is large or a missed win is costly to the roadmap.
• Both feed the required sample size directly, so this calibration is not just philosophical: it changes how long the test needs to run.

Worked example

A team is testing a checkout change against baseline click-through p0 = 0.10, and wants to detect a 5% relative lift (p1 = 0.105) with alpha = 0.05 and power = 0.80. The required sample size per arm:

$n = \frac{(z_{crit} + z_{\beta})^2 \left[p_0(1-p_0) + p_1(1-p_1)\right]}{(p_1 - p_0)^2}$

With z_beta = 0.8416 (power 0.80):

• Two-sided (z_crit = z_{0.025} = 1.9600): n is about 57,760 per arm
• One-sided (z_crit = z_{0.05} = 1.6449): n is about 45,498 per arm

(both values computed directly from the closed-form expression above with python3; the two differ only in z_crit)

The one-sided design needs about 21% fewer users to reach the same power, which is the concrete return on committing to "we only act on improvement." If the team is not actually willing to make that commitment, that saved sample size is not real, because they will end up wanting to look at the other tail anyway.

Trade-offs & pitfalls

• Picking "one-sided" purely to shrink the sample size, without a genuine commitment to ignore harm, is the most common misuse; it quietly weakens protection against shipping something worse.
• Alpha and power are symmetric-looking numbers with asymmetric consequences: they should be set from the business cost of each error type, not left at textbook defaults (0.05 / 0.80) by convention.
• Rejecting H0 tells you the observed difference is unlikely under "no effect," not that the effect is large enough to matter. Rollout decisions need the estimate and its confidence interval, not just the p-value.
• Failing to reject H0 is not evidence of no effect; it may just mean the test was underpowered for the true effect size.

Direct answer

Use a z-test only when the population standard deviation is genuinely known in advance, which is rare in practice. Use a t-test whenever the standard deviation has to be estimated from the sample itself, which is the normal situation, and this holds regardless of sample size. Sample size affects a different thing: how close the t and z critical values are to each other and how much you can lean on the Central Limit Theorem if the underlying data isn't very normal.

Structured elaboration

Known vs. unknown variance. This is the formal criterion. If $\sigma$ is known (rare outside quality-control settings with a long-established process variance), use z. If $\sigma$ is estimated from the sample as $s$ (the normal case), use t with $df = n-1$; the t-distribution's heavier tails are exactly the correction for the added uncertainty of estimating $\sigma$ rather than knowing it.

Sample size's actual role. As $n$ grows, $t_{n-1}$ converges to $z$, so at large $n$ the choice barely changes the numeric answer, which is why "just use z for n≥30" survives as a practical shortcut even though it's not the formal reason. Separately, larger $n$ also makes the Central Limit Theorem a stronger justification for treating the sampling distribution of the mean as approximately normal even when the raw data isn't, which matters for the validity of either test, not for the t-vs-z choice itself.

Comparing two means: pooled vs. Welch's t. If assuming the two groups have equal population variances, use the standard (pooled) two-sample t-test. If variances might differ, and there's rarely a strong reason to assume they're equal, use Welch's t-test, which does not assume equal variances and adjusts the degrees of freedom accordingly. Welch's costs very little power when variances actually are equal but protects against inflated Type I error when they aren't, which is why it's the safer default.

Robustness. t-tests are reasonably robust to mild-to-moderate non-normality once $n$ is moderate (roughly 30+ per group), thanks to the CLT. They're not robust to strong skew or heavy outliers at small $n$, where a few extreme points can dominate both the mean and the variance estimate.

Worked example: how close t and z actually are, by sample size

(All values from scipy.stats.t.ppf(0.975, df), verified directly.) At $df=5$ the t critical value is about 31% larger than z, meaningfully widening the interval or raising the bar for significance; by $df=60$ the gap has shrunk to about 2%. This is the practical justification behind "large n, t and z are basically the same," even though the theoretically correct reason to pick t is always "σ is estimated," not "n is small."

When variances are unknown and sample sizes are small: the actual procedure

1. Look at the data: a histogram or Q-Q plot per group, and check for obvious outliers.
2. If approximate normality looks plausible, default to Welch's t-test (not pooled, unless there's a specific reason to believe variances are equal, such as both groups measuring the identical underlying process).
3. If normality looks clearly violated, or the sample is extremely small (single digits per group) with visible skew, switch to a nonparametric alternative like the Mann-Whitney U test, or use a bootstrap for the confidence interval and p-value instead of the t-distribution's analytic formula.

Trade-offs & pitfalls

• Defaulting to the pooled t-test "because it's the classic one" without checking the equal-variance assumption is a common shortcut that inflates false positives when variances genuinely differ; Welch's is essentially free insurance against this.
• Small samples with heavy skew or outliers can pass a superficial normality check while still producing an unreliable t-test; this is where nonparametric or bootstrap alternatives earn their keep, not just as a formality but as a real fix.
• The "n≥30 use z" heuristic is useful as a rule of thumb but wrong as a justification; it should never be given as the reason to choose z over t in an interview answer, since the real criterion is whether σ is known.

Quick answer

Cohen's $d$ is the standardized mean difference: $d = (\bar x_1 - \bar x_2) / s_{pooled}$, the gap between the two groups' averages expressed in units of their pooled standard deviation, so $d = 0.3$ means the two groups differ by three-tenths of a standard deviation on whatever metric is being compared. To find the sample size needed to detect that effect with 80% power at $\alpha = 0.05$ in a two-sample t-test, use the standard normal approximation that trades off the critical value for significance against the value needed for power, both scaled by how small the effect is. The result, worked by hand below, is about 175 people per group, roughly 350 total.

Assumptions

• Observations are independent, both between and within groups (no repeated measurement of the same unit).
• The outcome is approximately normally distributed in each group, or the sample is large enough that the sampling distribution of the mean difference is close to normal via the Central Limit Theorem.
• The two groups have equal (or near-equal) variance, since $d$ is defined relative to a single pooled standard deviation; if variances differ meaningfully, the formula needs a Welch-style adjustment.
• The test is two-sided; a one-sided test would use $z_{1-\alpha}$ instead of $z_{1-\alpha/2}$ and requires a smaller sample for the same power.
• Group sizes are equal ($n_1 = n_2$); unequal allocation changes the formula's leading constant.

Formula and derivation

The approximate required sample size per group for a two-sided, two-sample t-test is:

where $z_{1-\alpha/2}$ is the standard normal critical value for the significance level, and $z_{1-\beta}$ is the standard normal value corresponding to the desired power. Intuitively: the numerator captures how far apart the rejection boundary and the true effect need to sit to both control false positives and reliably detect a real effect; the denominator shrinks the required sample as the effect gets easier to see (bigger $d$).

Plugging in $\alpha = 0.05$ (two-sided) and power $= 0.80$:

import numpy as np
from scipy import stats

alpha, power, d = 0.05, 0.80, 0.3
z_a = stats.norm.ppf(1 - alpha / 2)
z_b = stats.norm.ppf(power)
n_approx = 2 * (z_a + z_b) ** 2 / d ** 2
print(f"z_a={z_a:.4f}, z_b={z_b:.4f}, n_per_group={n_approx:.2f}")
# z_a=1.9600, z_b=0.8416, n_per_group=174.42

Round up: 175 per group, about 350 total.

Trade-offs and pitfalls

• This is a normal approximation, not the exact answer. The exact calculation uses the noncentral t-distribution, since with finite samples the test statistic isn't exactly normal. Confirming with software shows the approximation slightly underestimates the requirement here:

import numpy as np
from scipy import stats

d, alpha = 0.3, 0.05

def power_two_sample_t(n, d, alpha=0.05):
    df = 2 * n - 2
    nc = d * np.sqrt(n / 2)
    tcrit = stats.t.ppf(1 - alpha / 2, df)
    beta = stats.nct.cdf(tcrit, df, nc) - stats.nct.cdf(-tcrit, df, nc)
    return 1 - beta

for n in [175, 176]:
    print(f"n={n} per group -> exact power={power_two_sample_t(n, d, alpha):.4f}")
# n=175 per group -> exact power=0.7991
# n=176 per group -> exact power=0.8014

At exactly 175 per group the true power is 79.91%, just under the 80% target; 176 is the first sample size that actually clears it. The normal approximation is close enough to plan with, but a careful answer flags that the exact figure (computed via software such as R's power.t.test or Python's statsmodels) can differ by a handful of units, and that's the number to use for a final commitment.

• Power is far more sensitive to the assumed $d$ than to $\alpha$ or the power target, because $n$ scales with $1/d^2$; a $d$ that's overestimated by 20% overstates the achievable power substantially at the planned $n$.
• Equal-variance and equal-group-size assumptions rarely hold exactly in practice; if allocation is unequal (e.g. 70/30 due to cost or ethics), the required total sample size goes up even for the same effect.

Quick answer

Each user receiving 10 independent emails, each clicked with probability $p = 0.3$, is a Binomial$(n=10, p=0.3)$ model. The probability of at least 3 clicks is easiest to get via the complement of 0, 1, or 2 clicks, which comes out to about 61.7%. The expected number of clicks is $np = 3$, with variance $np(1-p) = 2.1$.

Setup

$X \sim \text{Binomial}(n=10,\ p=0.3)$, with PMF:

(a) $P(X \ge 3)$

Computing the complement is less arithmetic than summing 8 terms directly:

from math import comb

n, p = 10, 0.3
p0 = comb(n, 0) * p**0 * (1 - p)**10
p1 = comb(n, 1) * p**1 * (1 - p)**9
p2 = comb(n, 2) * p**2 * (1 - p)**8
p_ge3 = 1 - (p0 + p1 + p2)
print(f"P(X=0)={p0:.6f}, P(X=1)={p1:.6f}, P(X=2)={p2:.6f}")
print(f"P(X>=3)={p_ge3:.4f}")
# P(X=0)=0.028248, P(X=1)=0.121061, P(X=2)=0.233474
# P(X>=3)=0.6172

$P(X \ge 3) \approx 0.6172$: roughly 61.7% of users click at least 3 of the 10 emails.

(b) Expected value and variance

For a Binomial random variable:

Standard deviation $= \sqrt{2.1} \approx 1.449$.

Interpretation

An average user clicks 3 of the 10 emails, with a standard deviation of about 1.45 clicks, so a typical user's click count lands somewhere between about 1.5 and 4.5. The fact that over 60% of users clear the "at least 3" bar despite the average also being 3 reflects the right-skew-free, moderately spread-out shape of this particular Binomial (n=10, p=0.3 isn't close enough to either 0 or 1 to be heavily skewed).

Trade-offs and pitfalls

• Independence across the 10 emails is the load-bearing assumption. If a user who clicks one email is more likely to click the next (engagement momentum) or less likely (fatigue), the true distribution is overdispersed or underdispersed relative to Binomial, and the variance formula above will be wrong.
• $p=0.3$ is itself an estimate, not a known constant. In practice $p$ is estimated from historical click data, and the Binomial calculation above doesn't account for uncertainty in that estimate; a more careful treatment would put a Beta prior on $p$ and integrate over it (a Beta-Binomial), widening the resulting probability estimates.