Statistical Inference and Hypothesis Testing Questions

Direct answer

Solve the margin-of-error formula backward for $n$: given a target margin $E$, a confidence level (95%, so $z^{*}=1.96$), and an estimate of the standard deviation from a pilot sample, the required sample size is $n = (z^{*} s / E)^2$, rounded up. With the pilot's $s \approx 4$ and a target margin of 0.5, that comes out to about 246 customers, treated as a starting plan to be revisited once real data comes in.

Structured elaboration

Setting up the formula. A 95% CI for a mean has half-width (margin of error) $E = z^{*} \cdot SE = z^{*} \cdot \frac{s}{\sqrt n}$. Solving for $n$:

Using $z^{*}=1.96$ for planning (not $t$) is a standard simplification: at the sample sizes this formula tends to produce, $t$ and $z$ are close enough that using $z$ up front and refining with $t$ afterward is fine.

Why the pilot is only a starting point. The formula needs a standard deviation, but the true population $\sigma$ for customer lifetime value is unknown; that's exactly what the 40-customer pilot supplies as an estimate. Because it's an estimate from a small sample, it's noisy, and CLV in particular is often right-skewed with a long tail of high-value customers, which can make a 40-person pilot understate the true variance if none of the top-tail customers happened to land in it.

Worked example

Target margin $E = 0.5$, 95% confidence ($z^{*}=1.96$), pilot standard deviation $s = 4$ from $n_{pilot}=40$:

(Verified by direct computation.) If historical response/completion rates for this kind of data pull run around 80%, plan to sample or invite more than 246 to end up with 246 completed observations: $246 / 0.8 = 307.5 \Rightarrow 308$ invites.

Iterative refinement in practice.

1. Collect the initial planned batch (or an early tranche of it).
2. Recompute the sample standard deviation from the larger, more reliable batch. If it's meaningfully different from the pilot's 4, recompute $n$ with the updated $s$; a larger true $s$ means the original plan undershoots the target margin.
3. Once close to the target $n$, switch the final margin-of-error check to the $t$-distribution with $df = n-1$ for the precise value; at $n \approx 246$, $t_{0.975,245} \approx 1.970$ versus $z=1.96$, a difference of about 0.01 in the critical value, small enough that it rarely changes the plan.
4. Most product sample-size calculations stop at step 3. Two further refinements apply only in specific cases: if the population of eligible customers is small relative to $n$ (e.g. a niche segment with only a few thousand total customers), apply a finite-population correction (a downward adjustment to $n$ that accounts for sampling a large fraction of a small, finite population rather than an effectively infinite one), which shrinks the required sample size.
5. If sampling is clustered (e.g. by region or cohort) rather than a simple random sample, inflate $n$ by a design effect, or DEFF (a multiplier that accounts for the extra correlation between units sampled from the same cluster, which reduces how much independent information the same number of clustered units carries compared to a true simple random sample), to account for the extra correlation clustering introduces.

Trade-offs & pitfalls

• Treating the pilot-based $n=246$ as final rather than a planning estimate is the main risk: if the true variance is higher than the pilot suggested (common with skewed CLV data and a small pilot), the study will land with a wider-than-intended interval unless the sample size gets revisited.
• Non-response and attrition are easy to forget until data collection is already underway; inflating for expected response rate up front avoids a late scramble to recruit more.
• The margin-of-error formula assumes simple random sampling; ignoring clustering or stratification in the actual collection design while using the unadjusted formula will understate the sample size actually needed.

Approach

Comparing two proportions under $H_0: p_a = p_b$ uses a pooled-proportion z-test. Estimate the pooled proportion across both groups, use it to compute the standard error under the null, form the z-statistic from the observed difference, and convert to a two-sided p-value using the standard normal CDF:

def proportion_ztest(success_a, n_a, success_b, n_b) -> float:
    """
    Two-sided z-test p-value for a difference in two proportions (pooled variance).

    Assumptions:
    - Two independent samples of Bernoulli trials.
    - Large-sample normal approximation holds (rule of thumb: n*p and n*(1-p) >= 5
      in each group under the pooled proportion).
    - H0: p_a == p_b (two-sided alternative).

    Edge cases:
    - Raises ValueError if n_a or n_b <= 0, or if success counts are out of range.
    - If the pooled variance is 0 (e.g. success_a == n_a and success_b == n_b, both
      groups all-success), returns p=1.0 when proportions are equal, else 0.0.
    """
    import math

    for name, v in (("success_a", success_a), ("n_a", n_a), ("success_b", success_b), ("n_b", n_b)):
        if not isinstance(v, (int, float)):
            raise TypeError(f"{name} must be numeric")
    if n_a <= 0 or n_b <= 0:
        raise ValueError("n_a and n_b must be > 0")
    if not (0 <= success_a <= n_a) or not (0 <= success_b <= n_b):
        raise ValueError("success counts must be between 0 and their respective n")

    p_a = success_a / n_a
    p_b = success_b / n_b
    pooled_p = (success_a + success_b) / (n_a + n_b)
    var = pooled_p * (1 - pooled_p) * (1 / n_a + 1 / n_b)

    if var == 0:
        return 1.0 if p_a == p_b else 0.0

    z = (p_a - p_b) / math.sqrt(var)

    def std_norm_cdf(x):
        return 0.5 * (1 + math.erf(x / math.sqrt(2)))

    p_value = 2 * (1 - std_norm_cdf(abs(z)))
    return float(min(max(p_value, 0.0), 1.0))

Key points

• Uses math.erf for the normal CDF, so the function has zero third-party dependencies; if scipy is available, scipy.stats.norm.cdf is a drop-in swap for std_norm_cdf.
• Pools the two sample proportions under the null (rather than using each group's own variance) because that's the correct variance estimate for a test of equality; a confidence interval on the difference, by contrast, should use the unpooled variance.
• Validates types and ranges up front so a bad call fails loudly rather than silently returning a nonsense p-value.

Complexity

$O(1)$ time and space: every operation is a fixed number of arithmetic operations regardless of $n_a$ or $n_b$ (the counts, not the raw data, are the inputs).

Edge cases

• n_a or n_b equal to 0 raises ValueError rather than dividing by zero.
• Success counts outside [0, n] raise ValueError.
• Both groups at 100% success (var == 0, no variability to test): returns 1.0 if the proportions are equal, 0.0 if the sample sizes differ in a way that still leaves the proportions equal (this only occurs at the boundary and both proportions are identically 1.0, so it always returns 1.0 in practice for this branch).
• Very small n_a/n_b where n*p < 5: the function still returns a number, but the normal approximation is unreliable there; a docstring note (and, in production, a runtime warning) flags that Fisher's exact test is the safer choice for small samples.

Verified with a pinned worked example: success_a=1240, n_a=20000 (6.20% conversion) vs. success_b=1360, n_b=20000 (6.80% conversion) gives pooled_p=0.0650, z=-2.4338, p-value = 0.0149, which matches scipy.stats.norm.cdf-based computation to within 1e-9 when cross-checked.

Direct answer

When treatment causes users to drop out before the primary metric can even be measured, comparing outcomes only among the users who stuck around is comparing two different, treatment-selected populations, not the two randomized groups - this is differential attrition, and it biases the naive per-protocol effect. Intention-to-treat (ITT) analyzes everyone as originally randomized regardless of what happened after, preserving the randomization's unbiasedness for the effect of offering the treatment; per-protocol analyzes only compliant, observed users and targets a different, generally biased quantity unless attrition is unrelated to potential outcomes. When attrition itself is informative, worst-case (Lee) bounds or explicit missing-data modeling are needed to say anything defensible about the effect size.

Structured elaboration

Why differential attrition biases effect estimates. If who drops out depends on treatment assignment and on their would-be outcome (not simply on ignorable, fully-observed covariates), then the observed treated group is a selected subset. If, for example, the onboarding treatment disproportionately loses users who would have performed poorly, the treated group that remains looks artificially better - even if the treatment itself did nothing after week one. The relevant missing-data taxonomy: MCAR (missing completely at random: missingness unrelated to anything, including the outcome itself, rare in practice - here that would mean users drop out for reasons totally unconnected to onboarding or their eventual behavior, like a random app crash unrelated to the product), MAR (missing at random: missingness explained by observed covariates, fixable via modeling - here that would mean a specific onboarding step is disproportionately abandoned by users on an older app version, something you can see and condition on), MNAR (missing not at random: missingness depends on the unobserved outcome itself, not fully fixable without extra assumptions - here that would mean the users most likely to have failed onboarding are exactly the ones who never generate a completion event, so their absence itself carries information about the very outcome you're trying to measure).

ITT vs per-protocol:

Methods for estimating the effect under missingness: In practice, start with Lee bounds, since they need no missingness model at all and give an honest worst-case range; reach for multiple imputation or IPCW only when the Lee bounds are too wide to act on and the team is willing to assume MAR.

1. Multiple imputation - impute missing outcomes from a model conditioned on baseline covariates and early engagement signals; valid under MAR, still biased under MNAR unless the imputation model captures the missingness-outcome link.
2. Lee (2009) trimming bounds - when the outcome is only meaningfully defined among "survivors" and survival rates differ by arm, trim the arm with the higher survival rate down to the lower arm's rate (removing the most extreme observations first for the lower bound, the least extreme for the upper bound), giving a worst-case-assumption-free interval on the effect rather than a point estimate.
3. Inverse-probability-of-censoring weighting - model the probability of remaining observed as a function of pre-treatment covariates, and reweight observed users by its inverse; consistent under MAR (meaning as sample size grows, the estimate converges to the true effect) and degrades under MNAR just like MI.

Worked example

Simulated onboarding experiment ($n=20{,}000$, pinned seed): a latent "quality" trait drives potential outcomes on both arms, and the treatment's onboarding friction disproportionately drops low-quality users before the metric is reached (differential attrition), while control has a flat 10% attrition unrelated to quality:

import numpy as np
rng = np.random.default_rng(17)
n = 20000
quality = rng.normal(0, 1, n)
treat = rng.integers(0, 2, n)
y0 = 50 + 8*quality + rng.normal(0, 5, n)
y1 = y0 + 3.0                                    # true ATE = 3.0
y = np.where(treat == 1, y1, y0)

p_drop_treat = 1 / (1 + np.exp(-(-1.5 - 1.2*quality)))   # low quality -> more likely to drop, treatment only
p_drop_control = np.full(n, 0.10)
dropped = rng.uniform(0, 1, n) < np.where(treat == 1, p_drop_treat, p_drop_control)
observed = ~dropped

surv_treat = observed[treat == 1].mean()
surv_control = observed[treat == 0].mean()
print(f"survival rate: treatment={surv_treat:.3f}, control={surv_control:.3f}")
print("true ATE (ground truth, known only in simulation) = 3.000")
print()

naive = y[observed & (treat == 1)].mean() - y[observed & (treat == 0)].mean()
print(f"naive per-protocol estimate (observed-only comparison)          = {naive:.3f}")

y_zero_imputed = np.where(observed, y, 0.0)
itt = y_zero_imputed[treat == 1].mean() - y_zero_imputed[treat == 0].mean()
print(f"ITT estimate (dropouts hard-coded as metric=0)                  = {itt:.3f}")
print()

# control has the higher survival rate, so trim control down to treatment's survival rate
trim_frac = 1 - surv_treat / surv_control
y_control_obs = np.sort(y[observed & (treat == 0)])
n_trim = int(round(trim_frac * len(y_control_obs)))
treat_mean = y[observed & (treat == 1)].mean()

# best case for control (drop its lowest-outcome survivors first) -> smallest gap -> lower bound
control_mean_best_case = y_control_obs[n_trim:].mean()
# worst case for control (drop its highest-outcome survivors first) -> largest gap -> upper bound
control_mean_worst_case = y_control_obs[:len(y_control_obs) - n_trim].mean()
lower_bound = treat_mean - control_mean_best_case
upper_bound = treat_mean - control_mean_worst_case

print(f"Lee bounds (trim control arm down to treatment's survival rate, trim fraction={trim_frac:.3f}):")
print(f"  [{lower_bound:.3f}, {upper_bound:.3f}]")

Output:

survival rate: treatment=0.765, control=0.897
true ATE (ground truth, known only in simulation) = 3.000

naive per-protocol estimate (observed-only comparison)          = 5.184   <- biased upward (73% inflated)
ITT estimate (dropouts hard-coded as metric=0)                  = -2.628  <- badly biased the OTHER way

Lee bounds (trim control arm down to treatment's survival rate, trim fraction=0.147):
  [2.605, 7.766]   <- correctly BRACKETS the true ATE of 3.000

Two lessons in one simulation: the naive per-protocol comparison overstates the effect by 73% because it compares a treatment arm quietly enriched for high-quality survivors against an unenriched control. And a common "fix," coding every dropout's outcome as zero for an ITT analysis, is not automatically safe either - here it manufactures a large negative bias because it punishes the treatment arm disproportionately hard for its higher dropout rate, without that punishment reflecting anything real about the metric. Zero-imputation is only valid when zero is the correct value for a non-reacher (e.g., "did not convert"), not a stand-in for "unmeasured." The Lee bounds, needing only the two survival rates and the observed-outcome distributions, correctly bracket the true effect without assuming why people dropped out.

Trade-offs & pitfalls

The most damaging mistake is running only the per-protocol comparison and treating it as "the" effect - it silently answers a selection-biased question. A close second is reflexively coding missing outcomes as zero for ITT without checking whether zero is the semantically correct value for that metric; as shown above this can introduce a bias larger than the one it was meant to fix. Multiple imputation and IPCW both assume MAR (missingness explained by what you measured); if the true driver of dropout is unobserved, both remain biased, which is why Lee bounds (or an explicit sensitivity analysis varying an assumed MNAR departure) matter for reporting a defensible range instead of a false-precision point estimate. Finally, ITT dilutes an effect that is real but experienced only by compliers - reporting ITT alone without the compliance rate can understate how large the mechanistic effect is for users who actually get through onboarding.

Quick answer

Maximum A Posteriori (MAP) estimation with a Gaussian prior on the linear regression weights, maximizing $P(w \mid X, y) \propto P(y \mid X, w)\,P(w)$, produces exactly the ridge regression estimator (ordinary least squares with an added penalty term that shrinks the coefficients toward zero to control overfitting) once you take logs and differentiate. The regularization strength $\lambda$ that appears in ridge is the ratio of the noise variance to the prior variance, $\lambda = \sigma^2/\tau^2$, which is what makes ridge a special case of Bayesian shrinkage rather than an arbitrary penalty term.

Derivation

Model setup. Linear model $y = Xw + \epsilon$, with Gaussian noise $\epsilon \sim \mathcal{N}(0, \sigma^2 I)$ and a Gaussian prior on the weights, $w \sim \mathcal{N}(0, \tau^2 I)$.

Negative log-likelihood (up to a constant that doesn't depend on $w$):

Negative log-prior:

MAP objective. Maximizing the posterior is equivalent to minimizing the sum of the negative log-likelihood and negative log-prior:

Solve by setting the gradient to zero:

This is exactly the ridge regression closed-form solution. Compare with the OLS/MLE estimator (equivalent to a flat, improper prior on $w$):

The only difference is the $\lambda I$ term added before inversion.

How $\lambda$ relates to the prior variance

$\lambda = \sigma^2/\tau^2$: a larger prior variance $\tau^2$ (weak belief that weights are near zero, close to a flat prior) means smaller $\lambda$, so weaker regularization, converging to OLS as $\tau^2 \to \infty$. A smaller prior variance (strong belief that weights should be small) means larger $\lambda$, more aggressive shrinkage toward zero.

Worked numeric check

Confirming the closed forms agree, on a synthetic dataset with pinned seed and parameters:

import numpy as np

rng = np.random.default_rng(seed=46)
n, p = 50, 5
X = rng.normal(size=(n, p))
true_w = np.array([1.5, -2.0, 0.5, 0.0, 3.0])
y = X @ true_w + rng.normal(scale=1.0, size=n)

sigma2, tau2 = 1.0, 0.5
lam = sigma2 / tau2  # = 2.0

w_map = np.linalg.solve(X.T @ X + lam * np.eye(p), X.T @ y)
w_ols = np.linalg.solve(X.T @ X, X.T @ y)

print(f"lambda={lam}")
print(f"w_map = {np.round(w_map, 4)}")
print(f"w_ols = {np.round(w_ols, 4)}")
print(f"||w_map||={np.linalg.norm(w_map):.4f}, ||w_ols||={np.linalg.norm(w_ols):.4f}")
# lambda=2.0
# w_map = [ 1.2037 -1.9539  0.278   0.0782  2.8459]
# w_ols = [ 1.2483 -2.0261  0.2999  0.1019  2.9643]
# ||w_map||=3.6673, ||w_ols||=3.8145

Every $w_{MAP}$ coefficient sits closer to zero than the matching $w_{OLS}$ coefficient, and the overall norm shrinks from 3.81 to 3.67, exactly the shrinkage ridge regularization is supposed to produce.

Bias-variance implications

• Increasing $\lambda$ (equivalently, decreasing $\tau^2$, tightening the prior) shrinks the weight estimates toward zero, which reduces variance (less sensitivity to noise in the training data) but introduces bias (the estimator no longer targets the true $w$ on average, unless the true $w$ actually is near zero).
• In the ill-conditioned case, when $X^\top X$ is singular or near-singular (collinear features, $p$ close to or exceeding $n$), adding $\lambda I$ before inversion stabilizes the solution, which is a numerical benefit on top of the statistical bias-variance trade-off.
• Choosing $\lambda$ in practice: cross-validation is the standard, interview-safe approach. The rest of this bullet is depth beyond what most interviews require: a fully Bayesian version instead places a hyperprior (a prior distribution on the prior's own variance parameter) on $\tau^2$ (or equivalently $\lambda$) and either integrates it out or estimates it via empirical Bayes (choosing $\tau^2$ by maximizing the marginal likelihood, i.e. the prior variance that makes the observed data most probable, rather than holding out a validation set).

Trade-offs and pitfalls

The core interview answer is the derivation above: MAP with a Gaussian prior gives ridge, with $\lambda = \sigma^2/\tau^2$. The rest of this section covers extensions beyond what most interviews expect, useful only if the conversation goes further.

• This derivation assumes an isotropic prior, the same $\tau^2$ for every weight; a non-isotropic Gaussian prior (different variance per feature, or per feature group) generalizes this to weighted ridge or, in the extreme, feature-specific regularization like Automatic Relevance Determination (ARD, which lets the model learn a separate shrinkage strength per feature and drive irrelevant features' weights toward zero).
• The intercept is usually excluded from the prior/penalty in practice (it's centered separately), a detail that's easy to gloss over in the pure math but matters in implementation.
• A Laplace, not Gaussian, prior on $w$ gives L1 (LASSO) regularization instead, which induces sparsity rather than pure shrinkage; the derivation follows the identical MAP logic but the log-prior term becomes $\lVert w \rVert_1$ rather than $\lVert w \rVert_2^2$, and the resulting objective no longer has a closed-form solution.

Direct answer

Compute per-page conversion rate as unique converting users over unique visiting users, separately for the current period and the previous period, then run a two-proportion z-test per page comparing the two rates. Because this test runs once per page, correct for multiple comparisons (Benjamini-Hochberg) before deciding which pages have a real, not just noisy, drop.

Structured elaboration

1) Aggregate per-page, per-period counts from the event log. Using the given schema page_events(page_id, user_id, event_type, event_time), label each event into the current period (T1) or prior period (T0), then compute unique visiting users and unique converting users per page per period:

WITH events AS (
  SELECT page_id, user_id, event_type, event_time
  FROM page_events
  WHERE event_time >= DATE_SUB(CURRENT_DATE(), INTERVAL 14 DAY)
),
labelled AS (
  SELECT
    page_id,
    user_id,
    CASE
      WHEN event_time >= DATE_SUB(CURRENT_DATE(), INTERVAL 7 DAY) THEN 'T1'
      ELSE 'T0'
    END AS period,
    MAX(CASE WHEN event_type = 'conversion' THEN 1 ELSE 0 END) AS converted_flag
  FROM events
  GROUP BY page_id, user_id, period
)
SELECT
  page_id,
  period,
  COUNT(DISTINCT user_id) AS users,
  SUM(converted_flag) AS conversions
FROM labelled
GROUP BY page_id, period;

Grouping by (page_id, user_id, period) before taking MAX(converted_flag) ensures a user who converted multiple times, or visited multiple times, in the same period is counted once, which is what makes "conversion rate" a rate over people rather than a count of raw events.

2) Test each page for a significant drop. For each page, let $p_1$ = current-period rate, $p_0$ = prior-period rate, with sample sizes $n_1, n_0$. Under $H_0: p_1 = p_0$, pool the proportion and compute a one-sided z-statistic (one-sided because the question specifically asks about drops, not any change):

3) Correct for testing many pages at once. Running this test independently across, say, 6-50 pages means some will cross a raw $p < 0.05$ threshold by chance even with zero real drops anywhere. Rank all pages by p-value and apply Benjamini-Hochberg: find the largest rank $i$ such that $p_{(i)} \le (i/m)\alpha$, and flag every page at or below that rank, which controls the expected proportion of false discoveries among the flagged pages rather than the count.

4) Guard against low-power pages. Filter out (or flag separately) pages with too few visitors for the normal approximation to hold (rule of thumb $n \cdot p \ge 5$ and $n(1-p) \ge 5$); for those, either pool a longer window or fall back to Fisher's exact test.

Worked example

Pinned synthetic dataset (random.seed(7)), 6 pages with roughly 5% baseline conversion, where only page_D has a genuine underlying drop (from 5.2% to 3.8%, true effect injected) and the rest fluctuate by noise only:

import math
from scipy.stats import norm

pages = {
    "page_A": (5000, 250, 5000, 250),   # (n_T0, conv_T0, n_T1, conv_T1)
    "page_B": (3000, 135, 3000, 132),
    "page_C": (1200, 72, 1200, 66),
    "page_D": (4000, 208, 4000, 152),   # real drop
    "page_E": (800, 56, 800, 52),
    "page_F": (2500, 120, 2500, 127),
}

results = []
for page, (n0, c0, n1, c1) in pages.items():
    p0, p1 = c0 / n0, c1 / n1
    pooled = (c0 + c1) / (n0 + n1)
    se = math.sqrt(pooled * (1 - pooled) * (1/n1 + 1/n0))
    z = (p1 - p0) / se
    pval = norm.cdf(z)  # one-sided: P(drop this large or larger)
    results.append((page, p0, p1, z, pval))

results.sort(key=lambda r: r[4])
m, alpha = len(results), 0.05
thresholds = [(i / m) * alpha for i in range(1, m + 1)]
below = [pval <= t for (*_, pval), t in zip(results, thresholds)]
k_max = max((i for i, b in enumerate(below, 1) if b), default=0)
flagged = {results[i - 1][0] for i in range(1, k_max + 1)}

for page, p0, p1, z, pval in results:
    print(f"{page}: p0={p0:.4f}, p1={p1:.4f}, z={z:.3f}, p={pval:.5f}")
print(f"BH threshold at rank 1 of {m}: {thresholds[0]:.4f}")
print(f"flagged: {sorted(flagged)}")

Running this gives, sorted by p-value: page_D: p0=0.0520, p1=0.0380, z=-3.020, p=0.00126, followed by page_C: p=0.299, page_E: p=0.345, page_B: p=0.426, page_A: p=0.500, page_F: p=0.676. At rank 1 of 6, the BH threshold is $(1/6)(0.05) = 0.0083$; page_D's p-value of 0.00126 clears it, and no other page's p-value clears its own (looser) rank threshold, so only page_D is flagged, correctly recovering the one page with a genuine injected drop and correctly not flagging the five pages that only fluctuated by noise.

Trade-offs & pitfalls

A related but distinct bug is implementing BH as a pointwise per-rank filter (flag rank $i$ only if its own p-value clears $(i/m)\alpha$) instead of true BH (find the largest rank that clears its threshold, then flag every rank at or below it); the two agree whenever the sorted p-values cross the threshold line cleanly, but a pointwise filter silently under-flags whenever a middle rank fails its own bar while a later, larger rank still clears its own looser one. A separate common wrong turn is running the raw per-page z-test and flagging every page with $p < 0.05$ without the BH step at all; across dozens of pages that reliably produces false alarms, and a team that chases every one erodes trust in the alerting system. Another is computing conversion as events-per-event rather than unique-users-per-unique-users, which lets a handful of users retrying a broken flow inflate the apparent event count on a page and mask or fabricate a rate change. Low-traffic pages are the trickiest case in practice: they have the least power to detect a real drop and the most volatility from small-number noise, so a fixed significance threshold applied uniformly across high- and low-traffic pages either misses real problems on quiet pages or over-alerts on them; segmenting the alerting logic by minimum traffic volume, or pooling several periods for low-traffic pages before testing, avoids both failure modes.