Statistical Rigor & Avoiding Common Pitfalls Questions

Demonstrate deep understanding of statistical concepts: power analysis, sample size calculation, significance levels, confidence intervals, effect sizes, Type I and II errors. Discuss common mistakes in test interpretation: peeking bias (checking results too early), multiple comparison problem, regression to the mean, selection bias, and Simpson's Paradox. Discuss how you've implemented safeguards against these pitfalls in your testing processes. Provide examples of times you've caught flawed analyses or avoided incorrect conclusions.

MediumTechnical

97 practiced

Given tables:

users(user_id, assignment_group TEXT),events(user_id, purchased BOOLEAN)

Write a Postgres SQL query (CTE style) that computes: 1) point estimate of treatment effect on purchase rate (difference in proportions) and 2) approximate 95% confidence interval using the normal approximation for the difference. Explain any assumptions you make about missing data or duplicates.

Sample Answer

Approach: compute per-user purchase indicator (deduplicate events by user using MAX(purchased)), then aggregate by assignment_group to get proportions, compute difference and 95% CI using normal approximation: se = sqrt(pA*(1-pA)/nA + pB*(1-pB)/nB), CI = diff ± 1.96*se. Assumptions: exclude users with NULL assignment_group; treat multiple events per user by taking MAX(purchased) (user purchased if any event shows purchased=TRUE); users with no events are treated as purchased=FALSE.

sql

WITH per_user AS (
  -- one row per user: whether they ever purchased (TRUE/FALSE)
  SELECT
    u.user_id,
    u.assignment_group,
    COALESCE(MAX(e.purchased::int), 0) AS purchased_int
  FROM users u
  LEFT JOIN events e USING (user_id)
  WHERE u.assignment_group IS NOT NULL   -- exclude unknown assignments
  GROUP BY u.user_id, u.assignment_group
),
group_stats AS (
  SELECT
    assignment_group,
    COUNT(*) AS n,
    SUM(purchased_int) AS purchases,
    AVG(purchased_int::numeric) AS p  -- proportion
  FROM per_user
  GROUP BY assignment_group
),
pairs AS (
  -- assume two groups: "treatment" and "control"
  SELECT
    t.p AS p_t, t.n AS n_t,
    c.p AS p_c, c.n AS n_c
  FROM group_stats t
  CROSS JOIN group_stats c
  WHERE t.assignment_group = 'treatment'
    AND c.assignment_group = 'control'
)
SELECT
  p_t - p_c AS diff,
  p_t AS p_treatment,
  p_c AS p_control,
  n_t, n_c,
  -- standard error for difference in proportions
  sqrt( (p_t*(1-p_t)/n_t) + (p_c*(1-p_c)/n_c) ) AS se,
  (p_t - p_c) - 1.96 * sqrt( (p_t*(1-p_t)/n_t) + (p_c*(1-p_c)/n_c) ) AS ci_lower,
  (p_t - p_c) + 1.96 * sqrt( (p_t*(1-p_t)/n_t) + (p_c*(1-p_c)/n_c) ) AS ci_upper
FROM pairs;

Notes/assumptions:- Normal approximation is reasonable when n*p and n*(1-p) are both >= ~5 in each group.- If assignment groups differ from 'treatment'/'control', adjust the WHERE clause or join appropriately.- Alternative: use bootstrapping or exact (Fisher) methods if counts are small or proportions near 0/1.

MediumTechnical

84 practiced

Implement the Benjamini–Hochberg procedure in Python. Signature: def benjamini_hochberg(p_values, q=0.05): -> returns list of booleans of same length indicating rejected hypotheses. Ensure proper handling of equal p-values and preserve order of input when returning rejects.

Sample Answer

Approach: Sort p-values with original indices, apply the Benjamini–Hochberg rule to find the largest index k where sorted_p[i] <= (i/m) * q (i uses 1-based ranking). Then reject all hypotheses with p <= sorted_p[k]. Preserve original order by mapping rejections back to input indices. Ties are handled naturally by comparing p-values; if multiple equal p-values straddle the threshold, all equal p-values ≤ threshold are rejected.

python

def benjamini_hochberg(p_values, q=0.05):
    """
    Benjamini–Hochberg FDR control.
    p_values: iterable of floats in [0,1]
    q: target FDR level (0 < q <= 1)
    Returns: list of booleans (same length as p_values), True = reject H0
    """
    if q <= 0 or q > 1:
        raise ValueError("q must be in (0, 1].")
    m = len(p_values)
    if m == 0:
        return []

    # Attach original indices and sort ascending by p
    indexed = sorted(enumerate(p_values), key=lambda x: x[1])
    # Find largest i (1-based) where p_(i) <= (i/m)*q
    thresh_index = -1
    for rank, (orig_idx, p) in enumerate(indexed, start=1):
        if p <= (rank / m) * q:
            thresh_index = rank  # candidate to reject up to this rank

    # If none satisfy, return all False
    rejects = [False] * m
    if thresh_index == -1:
        return rejects

    # Determine p threshold (p_(k)); reject all with p <= p_k
    p_threshold = indexed[thresh_index - 1][1]
    # Mark rejects for all original positions with p <= p_threshold
    for i, p in enumerate(p_values):
        if p <= p_threshold:
            rejects[i] = True
    return rejects

Key points:- Time: O(m log m) for sorting; Space: O(m).- Edge cases: empty list, q out of bounds, p-values equal to threshold handled by <=.- Alternative: Return indices of rejections or adjusted p-values (q-values) if needed.

EasyTechnical

82 practiced

Explain differences between p-values, confidence intervals, and effect sizes when reporting A/B test results. Give an example where p<0.05 but the effect size is practically negligible and describe precisely how you would present this to a product manager, including alternative recommendations.

Sample Answer

P-values, confidence intervals (CIs), and effect sizes answer different questions and are all needed for responsible A/B reporting.

- P-value: probability of observing data as extreme (or more) than observed assuming the null hypothesis is true. It tests evidence against null (statistical significance) but says nothing about magnitude or business importance.- Confidence interval: a range, usually 95%, of plausible values for the true effect. It shows precision and direction; if the CI excludes zero, the effect is statistically significant at α=0.05.- Effect size: the estimated magnitude of the difference (e.g., absolute lift, relative lift, Cohen’s d). This is what determines practical/ business significance.

Concrete example:Suppose baseline conversion = 10%, treatment = 10.1% on 200k users per arm. p = 0.03 (statistically significant), but effect size = +0.1 percentage points (relative lift = 1%), and 95% CI = [0.02 pp, 0.18 pp]. Although p<0.05, the lift is tiny — likely below the minimum detectable or meaningful lift for product metrics and may not justify rollout costs.

How I’d present to a PM:- Start with headline: “The test is statistically significant (p=0.03) but the estimated lift is only +0.1 pp (1% relative); 95% CI [0.02, 0.18] — meaning the true lift is likely very small.”- Explain implications in plain terms: “Even at the high end (0.18 pp), revenue/DAU impact is minimal and may be smaller than implementation/maintenance costs or measurement noise.”- Provide numbers: translate lift into business impact (e.g., projected monthly revenue change, confidence in direction) and show uncertainty band.- Recommendations: 1. Don’t fully roll out yet — run targeted rollout to high-value segments where impact could be larger. 2. Increase experiment duration or sample in segments to tighten CI and test heterogeneity. 3. Consider alternative interventions expected to deliver larger effects. 4. If implementing, run an A/A sensitivity or monitor metrics post-release with alerting and safety rollbacks.

Key takeaway: use p-values to detect signals, CIs to judge precision and direction, and effect sizes (converted to business metrics) to decide action.

MediumTechnical

89 practiced

You're running an experiment that measures CTR, conversion, and revenue. Describe how to choose a single primary metric that aligns with business objectives, define appropriate guardrail metrics, and design how statistical alpha should be allocated across primary and secondary outcomes to avoid cherry-picking results.

Sample Answer

Goal alignment — pick the one metric that best maps to the business objective. If the objective is profitability or monetization, choose revenue-per-active-user (ARPU) or revenue per session as the primary. If the objective is user growth, choose new-user conversion rate. State it: “Primary = incremental ARPU per DAU over 14 days.” Justify: it directly captures the business value we care about and aggregates upstream behaviors (CTR → conversion → purchase).

Guardrail metrics — pre-specify metrics that detect regressions or unintended side-effects:- CTR (to monitor engagement changes)- Conversion rate (click→purchase) to separate engagement from monetization- Retention (7/30-day) to detect long-term harm- UX/quality signals (error rate, page load time)For each guardrail define directionality and thresholds (e.g., conversion drop >1.5% absolute or p<0.01 flags hold).

Statistical alpha & multiplicity control — prevent cherry-picking by pre-registering primary metric and testing plan:- Allocate full alpha (α = 0.05) to the declared primary; power the experiment for that metric first (sample size based on minimum detectable effect and business-cost tradeoff).- Treat secondary/guardrail tests as confirmatory only if primary is significant, or control family-wise error using methods like Holm-Bonferroni or an alpha-splitting scheme. Example: keep α_primary = 0.05; for three secondaries use Holm with family α = 0.05, or pre-allocate α_secondary_total = 0.01 across them to be conservative.- Alternatively use hierarchical testing: test primary at 0.05; only if significant proceed to test secondary metrics sequentially (controls FWER).- For continuous monitoring, use alpha spending rules (O’Brien-Fleming) or adopt Bayesian decision thresholds (posterior probability > 97.5%) to avoid inflating Type I error.

Operational rules:- Pre-register the hypothesis, primary metric, guardrails, sample size, and analysis scripts.- Run pre-specified A/B analysis (ITT), report confidence intervals and uplift in business units (e.g., $/DAU).- If primary is non-significant but a secondary is, treat as hypothesis-generating — require follow-up experiments before productization.

Example allocation (concrete): power for detecting +2% ARPU uplift at α=0.05. Pre-specify α_primary = 0.05; reserve α_secondary_total = 0.01 split via Holm among CTR, conversion, retention. Stop and investigate if any guardrail breaches threshold regardless of significance.

This approach aligns statistical rigor with business impact, prevents post-hoc cherry-picking, and keeps teams focused on the metric that matters most.

EasyTechnical

89 practiced

Define Type I and Type II errors in the context of A/B testing for conversion rate. Give a concrete numeric example (e.g., baseline conversion 5%, treatment observed 6.5% with p=0.04) and explain which error each decision (launch vs not launch) risks. Discuss the business consequences of both error types and how you would balance them when prioritizing growth experiments.

Unlock Full Question Bank

Get access to hundreds of Statistical Rigor & Avoiding Common Pitfalls interview questions and detailed answers.

Join thousands of developers preparing for their dream job.