Statistical Inference and Hypothesis Testing Questions

Direct answer

For a two-sided test comparing two proportions with baseline $p_0=0.05$, target $p_1=0.06$ (20% relative lift), $\alpha=0.05$, and power $=0.80$, the required sample size is about 8,158 per arm, computed from the standard normal-approximation formula for a two-proportion test. Unequal allocation and continuous metrics use the same underlying logic (compare the standardized difference against critical values from the desired error rates) but change the variance term and, for unequal group sizes, the optimal split between arms.

Structured elaboration

Formula (equal allocation, two-sided)

where $\bar p = (p_0+p_1)/2$, $z_{1-\alpha/2}=1.9600$ for $\alpha=0.05$, and $z_{1-\beta}=0.8416$ for power $=0.80$.

Numeric steps

(all steps reproduce exactly with plain arithmetic; verified with python3)

So you need about 8,158 users per arm, 16,320 total.

Unequal allocation

With allocation ratio $r = n_1/n_0$ (treatment vs. control), the sample size for the control arm becomes:

with $\bar p = (p_0 + r\,p_1)/(1+r)$. The allocation ratio that minimizes the total sample size $n_0+n_1$ for a fixed variance target (Neyman allocation) is:

For this example, $p_0(1-p_0)=0.0475$ and $p_1(1-p_1)=0.0564$, so $r^{*}=\sqrt{0.0564/0.0475}=1.090$, close to 1: with proportions this close together, equal allocation is already near-optimal, and there's little total-sample-size benefit to skewing the split. Unequal allocation becomes worth doing when one arm is deliberately capped (e.g. a risky treatment held to 10% of traffic); the cost is a larger total sample size than the balanced design would need for the same power.

Continuous metrics

Replace the proportion-variance terms with the outcome variance $\sigma^2$ (estimated from historical data or a pilot):

where $\Delta$ is the absolute mean difference you want to detect. Use a pooled or historical estimate of $\sigma$, and consider winsorizing or a log transform first if the metric is heavy-tailed, since $\sigma^2$ from raw heavy-tailed data can be dominated by a handful of extreme values and understate how many "typical" observations you'll actually need.

Trade-offs & pitfalls

• For small $p_0$ or small $\Delta$, the normal approximation underlying this formula can be inaccurate; check that $n p_0$ and $n(1-p_0)$ are both comfortably above about 5-10, or fall back to an exact binomial/simulation-based power calculation.
• The unequal-allocation Neyman ratio only minimizes total sample size; it does not account for a fixed traffic budget per arm or for practical constraints like a hard cap on treatment exposure, so it should be treated as a starting point, not an automatic answer.
• Continuous-metric sample sizes are only as good as the variance estimate feeding them; an outdated or unrepresentative $\sigma$ estimate (from before a product change, or from a different user segment) will make the planned sample size wrong in either direction.
• None of these formulas account for multiple comparisons, expected attrition, or metric seasonality; pad the computed $n$ for real-world dropout and plan the run to cover at least one full weekly cycle regardless of what the raw sample-size number implies about duration.

Direct answer

Compute per-page conversion rate as unique converting users over unique visiting users, separately for the current period and the previous period, then run a two-proportion z-test per page comparing the two rates. Because this test runs once per page, correct for multiple comparisons (Benjamini-Hochberg) before deciding which pages have a real, not just noisy, drop.

Structured elaboration

1) Aggregate per-page, per-period counts from the event log. Using the given schema page_events(page_id, user_id, event_type, event_time), label each event into the current period (T1) or prior period (T0), then compute unique visiting users and unique converting users per page per period:

WITH events AS (
  SELECT page_id, user_id, event_type, event_time
  FROM page_events
  WHERE event_time >= DATE_SUB(CURRENT_DATE(), INTERVAL 14 DAY)
),
labelled AS (
  SELECT
    page_id,
    user_id,
    CASE
      WHEN event_time >= DATE_SUB(CURRENT_DATE(), INTERVAL 7 DAY) THEN 'T1'
      ELSE 'T0'
    END AS period,
    MAX(CASE WHEN event_type = 'conversion' THEN 1 ELSE 0 END) AS converted_flag
  FROM events
  GROUP BY page_id, user_id, period
)
SELECT
  page_id,
  period,
  COUNT(DISTINCT user_id) AS users,
  SUM(converted_flag) AS conversions
FROM labelled
GROUP BY page_id, period;

Grouping by (page_id, user_id, period) before taking MAX(converted_flag) ensures a user who converted multiple times, or visited multiple times, in the same period is counted once, which is what makes "conversion rate" a rate over people rather than a count of raw events.

2) Test each page for a significant drop. For each page, let $p_1$ = current-period rate, $p_0$ = prior-period rate, with sample sizes $n_1, n_0$. Under $H_0: p_1 = p_0$, pool the proportion and compute a one-sided z-statistic (one-sided because the question specifically asks about drops, not any change):

3) Correct for testing many pages at once. Running this test independently across, say, 6-50 pages means some will cross a raw $p < 0.05$ threshold by chance even with zero real drops anywhere. Rank all pages by p-value and apply Benjamini-Hochberg: find the largest rank $i$ such that $p_{(i)} \le (i/m)\alpha$, and flag every page at or below that rank, which controls the expected proportion of false discoveries among the flagged pages rather than the count.

4) Guard against low-power pages. Filter out (or flag separately) pages with too few visitors for the normal approximation to hold (rule of thumb $n \cdot p \ge 5$ and $n(1-p) \ge 5$); for those, either pool a longer window or fall back to Fisher's exact test.

Worked example

Pinned synthetic dataset (random.seed(7)), 6 pages with roughly 5% baseline conversion, where only page_D has a genuine underlying drop (from 5.2% to 3.8%, true effect injected) and the rest fluctuate by noise only:

import math
from scipy.stats import norm

pages = {
    "page_A": (5000, 250, 5000, 250),   # (n_T0, conv_T0, n_T1, conv_T1)
    "page_B": (3000, 135, 3000, 132),
    "page_C": (1200, 72, 1200, 66),
    "page_D": (4000, 208, 4000, 152),   # real drop
    "page_E": (800, 56, 800, 52),
    "page_F": (2500, 120, 2500, 127),
}

results = []
for page, (n0, c0, n1, c1) in pages.items():
    p0, p1 = c0 / n0, c1 / n1
    pooled = (c0 + c1) / (n0 + n1)
    se = math.sqrt(pooled * (1 - pooled) * (1/n1 + 1/n0))
    z = (p1 - p0) / se
    pval = norm.cdf(z)  # one-sided: P(drop this large or larger)
    results.append((page, p0, p1, z, pval))

results.sort(key=lambda r: r[4])
m, alpha = len(results), 0.05
thresholds = [(i / m) * alpha for i in range(1, m + 1)]
below = [pval <= t for (*_, pval), t in zip(results, thresholds)]
k_max = max((i for i, b in enumerate(below, 1) if b), default=0)
flagged = {results[i - 1][0] for i in range(1, k_max + 1)}

for page, p0, p1, z, pval in results:
    print(f"{page}: p0={p0:.4f}, p1={p1:.4f}, z={z:.3f}, p={pval:.5f}")
print(f"BH threshold at rank 1 of {m}: {thresholds[0]:.4f}")
print(f"flagged: {sorted(flagged)}")

Running this gives, sorted by p-value: page_D: p0=0.0520, p1=0.0380, z=-3.020, p=0.00126, followed by page_C: p=0.299, page_E: p=0.345, page_B: p=0.426, page_A: p=0.500, page_F: p=0.676. At rank 1 of 6, the BH threshold is $(1/6)(0.05) = 0.0083$; page_D's p-value of 0.00126 clears it, and no other page's p-value clears its own (looser) rank threshold, so only page_D is flagged, correctly recovering the one page with a genuine injected drop and correctly not flagging the five pages that only fluctuated by noise.

Trade-offs & pitfalls

A related but distinct bug is implementing BH as a pointwise per-rank filter (flag rank $i$ only if its own p-value clears $(i/m)\alpha$) instead of true BH (find the largest rank that clears its threshold, then flag every rank at or below it); the two agree whenever the sorted p-values cross the threshold line cleanly, but a pointwise filter silently under-flags whenever a middle rank fails its own bar while a later, larger rank still clears its own looser one. A separate common wrong turn is running the raw per-page z-test and flagging every page with $p < 0.05$ without the BH step at all; across dozens of pages that reliably produces false alarms, and a team that chases every one erodes trust in the alerting system. Another is computing conversion as events-per-event rather than unique-users-per-unique-users, which lets a handful of users retrying a broken flow inflate the apparent event count on a page and mask or fabricate a rate change. Low-traffic pages are the trickiest case in practice: they have the least power to detect a real drop and the most volatility from small-number noise, so a fixed significance threshold applied uniformly across high- and low-traffic pages either misses real problems on quiet pages or over-alerts on them; segmenting the alerting logic by minimum traffic volume, or pooling several periods for low-traffic pages before testing, avoids both failure modes.

Direct answer

Decide pooled versus Welch by testing whether the two sample variances are plausibly equal, using Levene's test, which is robust to non-normality, rather than assuming equal variances by default. If Levene's p-value is above 0.05, treat the variances as equal and use the pooled-variance t-test with n1 + n2 - 2 degrees of freedom; otherwise use Welch's t-test, which does not assume equal variances and uses the Welch-Satterthwaite approximation for degrees of freedom.

Approach

1. Run Levene's test on the two samples to check the equal-variance assumption.
2. If the assumption holds (p > 0.05): pooled t-test, pooled variance, degrees of freedom = n1 + n2 - 2.
3. Otherwise: Welch's t-test, separate variances in the standard error, Welch-Satterthwaite degrees of freedom.
4. Compute the two-sided p-value from the Student-t distribution at that degrees of freedom, and a 95% confidence interval for the mean difference using the same standard error and critical value.

import numpy as np
from scipy import stats


def two_sample_ttest_with_ci(x, y, alpha=0.05):
    x = np.asarray(x, dtype=float)
    y = np.asarray(y, dtype=float)
    n1, n2 = len(x), len(y)
    if n1 < 2 or n2 < 2:
        raise ValueError("Each sample must have at least two observations.")

    # Levene's test decides pooled vs Welch (robust to non-normal data,
    # unlike Bartlett's test which assumes normality)
    stat_var, p_var = stats.levene(x, y)

    mu1, mu2 = x.mean(), y.mean()
    diff = mu1 - mu2
    s1_sq = x.var(ddof=1)
    s2_sq = y.var(ddof=1)

    if p_var > 0.05:
        test_type = "pooled"
        sp_sq = ((n1 - 1) * s1_sq + (n2 - 1) * s2_sq) / (n1 + n2 - 2)
        se = np.sqrt(sp_sq * (1 / n1 + 1 / n2))
        df = n1 + n2 - 2
    else:
        test_type = "welch"
        se = np.sqrt(s1_sq / n1 + s2_sq / n2)
        num = (s1_sq / n1 + s2_sq / n2) ** 2
        den = (s1_sq ** 2) / ((n1 ** 2) * (n1 - 1)) + (s2_sq ** 2) / ((n2 ** 2) * (n2 - 1))
        df = num / den  # Welch-Satterthwaite

    t_stat = diff / se
    p_value = 2 * stats.t.sf(np.abs(t_stat), df)
    t_crit = stats.t.ppf(1 - alpha / 2, df)
    ci = (diff - t_crit * se, diff + t_crit * se)

    return {
        "test_type": test_type,
        "t_statistic": float(t_stat),
        "degrees_of_freedom": float(df),
        "p_value": float(p_value),
        "confidence_interval_95": (float(ci[0]), float(ci[1])),
        "variance_test_pvalue": float(p_var),
    }

Key points

• Levene's test on the inputs, not an eyeball comparison of sample variances, drives the branch. That keeps the decision reproducible and automatable.
• Welch is the safer unconditional default in practice (it is what R's t.test() uses unless asked otherwise). This function only chooses pooled when Levene's test gives no evidence against equal variances.
• The confidence interval reuses the exact same standard error and degrees of freedom as the hypothesis test, so the CI and the p-value are always consistent: the CI excludes 0 exactly when p < alpha.

Worked example

Pinned seed np.random.default_rng(42), x drawn from Normal(52, 4) with n=30, y drawn from Normal(49, 9) with n=25:

Because y's variance (48.82) is roughly 5 times x's variance (9.65), Levene's test correctly rejects equal variances (p = 0.0018), the function selects Welch, and the fractional Welch-Satterthwaite degrees of freedom (31.85, not the pooled 53) reflects that the smaller, more variable sample limits how much the variance estimate can be trusted. Cross-checked against scipy.stats.ttest_ind(x, y, equal_var=False): t = 1.3823, p = 0.1765, matching the hand-rolled formulas.

Complexity

O(n1 + n2) time to compute means and variances, O(1) additional space beyond the inputs. Levene's test itself is O(n1 + n2).

Edge cases

• n1 < 2 or n2 < 2 raises, since sample variance is undefined.
• Zero-variance samples (all identical values) make Levene's test degenerate and can collapse the pooled-branch standard error toward 0, producing an enormous or undefined t-statistic. Guard this explicitly in production code.
• Heavy-tailed or strongly skewed data violate the normal-sampling-distribution assumption behind the t-distribution regardless of which branch is chosen. A bootstrap confidence interval or a nonparametric test such as Mann-Whitney U is the safer tool then, not a variance-equality fix.

Direct answer

Frequentist A/B testing treats the true conversion rates as fixed unknowns and describes uncertainty through the long-run behavior of a procedure (p-values, confidence intervals) over repeated sampling. Bayesian A/B testing treats the parameters themselves as random, combines a prior belief with the observed data via Bayes' rule, and produces a posterior distribution you can make direct probability statements about. The practical trade-off is interpretability and sequential flexibility (Bayesian) versus simplicity, cheap computation, and no dependence on a prior (frequentist).

Structured elaboration

Credible interval vs. confidence interval

Role of the prior

Priors encode existing knowledge or act as regularization. An informative prior (built from historical baselines) can stabilize estimates when data are sparse, but a badly misspecified prior can bias the posterior, especially with a small sample; a weakly informative or hierarchical prior is the usual compromise, letting data dominate once there's enough of it while still borrowing strength when there isn't. Frequentist methods have no explicit prior, though penalization or shrinkage techniques play an analogous stabilizing role.

Computational trade-offs

• Frequentist tests (z-test, t-test) are closed-form and essentially free to compute at any scale, which is why they're the default for real-time dashboards across many simultaneous experiments.
• Bayesian inference can require Markov Chain Monte Carlo (MCMC) sampling for complex models, but for the common case of a Beta-Binomial conjugate model (Beta prior, Binomial likelihood for a conversion rate), the posterior is closed-form and just as cheap as the frequentist calculation.
• Bayesian sequential updating (recomputing the posterior as new data streams in) does not by itself protect against inflated error rates from repeated peeking; the peeking problem still needs a stopping rule that accounts for it (e.g. a pre-specified decision rule on the posterior, not "stop whenever it looks good").

Worked example

A team runs many small, sequential experiments on low-traffic feature variants with sparse conversion events, and wants to say "there's a 95% probability variant A beats variant B" to a product manager, plus borrow strength across related past experiments on similar features.

Using a Beta-Binomial model: Beta$(\alpha_0,\beta_0)$ prior on each variant's conversion rate, updated with observed successes/failures to a Beta posterior. Concretely, suppose variant A converts 22 of 50 visitors and variant B converts 14 of 50, with a weakly informative Beta(1,1) (uniform) prior on each arm:

import numpy as np
rng = np.random.default_rng(42)
a_A, b_A = 23, 29
a_B, b_B = 15, 37

samples = 1_000_000
pA = rng.beta(a_A, b_A, samples)
pB = rng.beta(a_B, b_B, samples)
p_a_beats_b = (pA > pB).mean()   # 0.9503

Both posteriors are closed-form Betas, so sampling is just the simplest way to compare two distributions; an exact numerical integral would give the same answer. With these two posteriors, the direct statement "$P(p_A > p_B \mid \text{data}) = 0.95$" answers exactly the question the PM is asking, which the frequentist p-value does not, and now it traces back to real counts (22 conversions of 50 for A, 14 of 50 for B) instead of being asserted.

Trade-offs & pitfalls

• Bayesian output is easier to explain to a non-technical stakeholder ("95% probability A is better") than a p-value or a frequentist confidence interval, but that ease of interpretation depends on the audience trusting the prior, which is a modeling choice, not a fact.
• An informative prior chosen after seeing early results (rather than from genuinely prior information) quietly reintroduces the same p-hacking risk Bayesian methods are sometimes assumed to avoid.
• Reaching for full MCMC when a conjugate model would do adds compute cost and diagnostic burden (convergence checks, effective sample size) with no benefit; match the model complexity to the actual question.
• Neither framework is immune to multiple-comparisons risk when running "many small, sequential" experiments and reporting only the ones that look good; that discipline (pre-specification, correction, or hierarchical pooling) is needed regardless of which paradigm you use.

Direct answer

Revenue up while retention and NPS decline for two quarters is a classic sign that the business is winning short-term monetization at the expense of long-term health, most often through mix shift (different, sometimes lower-fit users or a different pricing structure) rather than the product genuinely improving. The investigation should treat "revenue growth" and "retention/NPS decline" as two outcomes potentially driven by a shared, upstream cause (pricing, acquisition mix, or a product change), not evaluate them independently.

Structured elaboration

Hypotheses to test, roughly in order of how commonly they explain this exact pattern

1. Pricing or monetization change increased ARPU but also increased friction or perceived value mismatch, driving both the revenue bump and dissatisfaction.
2. Acquisition mix shifted toward higher-paying but lower-fit users (e.g. a paid channel or enterprise push brought in users who convert to revenue quickly but churn or complain more).
3. A specific feature or experiment increased short-term monetization while degrading the core experience for a subset of users.
4. One-time or concentrated revenue events (large enterprise deals, a promotion) inflate aggregate revenue while the core, ongoing user base's underlying health is actually flat or declining.
5. Measurement or attribution issue: revenue recognition change or duplicate counting inflating the topline number without a real behavioral change at all.

Metrics and cohorts to analyze

Statistical tests and analyses to run

• Cohort retention heatmap crossed with ARPU decile: is the revenue growth concentrated in cohorts whose retention is also declining, or in a separate, healthy cohort?
• Difference-in-differences comparing cohorts before and after the suspected driver (e.g. a pricing change), against a comparable unaffected baseline, to isolate its effect on both revenue and retention.
• Regression of retention on covariates including acquisition channel and revenue tier, to test whether channel or tier explains the retention decline once controlled for, versus it being broad-based across the whole user base.
• Trend significance test on the NPS and retention time series themselves (e.g. comparing quarter-over-quarter means with appropriate variance estimates) to confirm the decline isn't within normal seasonal noise before treating it as a real signal worth acting on.

Decisions that might follow

Worked example

Suppose a cohort retention heatmap crossed with ARPU decile shows that the top 20% of ARPU users (by revenue) have 30-day retention of 38%, versus 61% for the bottom 80%. Splitting further by acquisition channel shows the top-ARPU decile is 70% sourced from a single paid channel launched two quarters ago, versus that channel being only 15% of the broader user base. This pattern (a new, concentrated acquisition channel simultaneously driving the ARPU decile up and dragging blended retention down) points toward hypothesis 2: the revenue growth and the retention/NPS decline share a common cause in acquisition mix, rather than the product itself degrading for its existing base. That reframes the fix as a channel-quality and targeting problem, not a product-quality problem, though the team should still verify NPS specifically within that channel's cohort to rule out a genuine experience gap for those users too.

Trade-offs & pitfalls

• Don't average away the story. A blended, company-wide retention number can look like a modest decline while masking a severe drop in one segment offset by stability elsewhere; always cross revenue and retention cuts by the same cohort dimensions.
• Correlation between the revenue and retention trends is not proof of a shared cause. Test the specific hypothesized mechanism (e.g. via difference-in-differences around the suspected driver) rather than asserting a link because two lines moved in opposite directions in the same period.
• NPS is a noisy, low-response-rate metric. A shift in who responds to the survey (not just how they feel) can produce an apparent decline; check response rate and respondent composition alongside the score itself.
• A quarter or two of concentrated enterprise deals is easy to over-interpret as "the business is healthy." Report revenue concentration (e.g. share from the top 10 accounts) alongside the headline number so a one-time deal isn't mistaken for durable growth.
• Acting on hypothesis 1 (pricing) versus hypothesis 2 (acquisition mix) implies very different fixes, so resist moving to remediation before the cohort-level analysis actually distinguishes between them; treating a mix-shift problem as a product or pricing problem wastes a cycle and doesn't fix the underlying issue.