Hypothesis Testing and Inference Questions

Fundamental framework and application of hypothesis testing and statistical inference. Topics include formulating null and alternative hypotheses, understanding Type I and Type II errors, interpreting p values and confidence intervals, selecting and applying common tests such as t tests, chi square tests, analysis of variance, and non parametric alternatives, checking test assumptions, and discussing statistical versus practical significance. Candidates should explain power, significance levels, effect sizes, and common pitfalls such as misinterpreting p values or violating independence assumptions. At more advanced levels, discuss limitations of null hypothesis significance testing, alternatives such as Bayesian inference, and guidance for when different approaches are appropriate.

MediumTechnical

32 practiced

Implement McNemar's test in Python. The input should be a 2x2 contingency table of disagreement counts: [[n00, n01], [n10, n11]]. Return the chi-square statistic and two-sided p-value, and explain when McNemar is appropriate for classifier comparison.

Sample Answer

Use McNemar’s test when you have paired binary outcomes (e.g., two classifiers evaluated on the same samples) and want to test whether the proportion of discordant pairs differs (i.e., whether classifier A and B disagree asymmetrically). The test uses only off-diagonal counts n01 and n10 from the 2x2 contingency table.

Approach: validate input, extract n01 and n10, compute test statistic with optional continuity correction, and compute two-sided p-value from chi-square(1).

python

import math
from scipy.stats import chi2

def mcnemar(table, continuity=True):
    """
    Perform McNemar's test on a 2x2 contingency table.
    table: [[n00, n01],
            [n10, n11]]
    continuity: apply Yates' continuity correction when True
    Returns: (chi2_statistic, two_sided_pvalue)
    """
    # validate
    if (not isinstance(table, (list, tuple)) or len(table) != 2
        or any(len(row) != 2 for row in table)):
        raise ValueError("table must be 2x2 list-like: [[n00,n01],[n10,n11]]")
    n00, n01 = table[0][0], table[0][1]
    n10, n11 = table[1][0], table[1][1]
    for x in (n00, n01, n10, n11):
        if not (isinstance(x, (int, float)) and x >= 0):
            raise ValueError("counts must be non-negative numbers")
    b = n01
    c = n10
    if b + c == 0:
        # No discordant pairs -> statistic 0, p=1
        return 0.0, 1.0

    if continuity:
        # Yates' continuity correction
        stat = (abs(b - c) - 1)**2 / (b + c)
    else:
        stat = (b - c)**2 / (b + c)

    p = 1 - chi2.cdf(stat, df=1)
    return float(stat), float(p)

Key points:- Use McNemar when observations are paired and binary; only discordant counts (n01, n10) matter.- With small b+c (e.g., <25) consider exact binomial test (exact McNemar) instead of chi-square approximation.- continuity=True applies Yates correction to reduce Type I error for small samples.

MediumTechnical

28 practiced

You observe different predicted positive rates across demographic groups. Explain how to set up a chi-square test of independence to check whether predicted outcome is independent of group membership. Show how to construct the contingency table and discuss alternatives when counts are small.

EasyTechnical

25 practiced

Explain the multiple testing problem and describe how Bonferroni correction works. Give a short ML example where multiple testing arises and discuss the downsides of Bonferroni in that context.

MediumTechnical

26 practiced

You have daily aggregated conversion rates for control and variant over 14 days. Sample sizes per day vary and days may be correlated. Explain which statistical test or approach you would use to evaluate lift, justify your choice, and describe how you would check independence assumptions.

Sample Answer

I would treat the daily aggregated data as binomial counts (conversions / impressions) with within-day correlation and possible day-to-day dependence. My preferred approach is a generalized linear mixed model (GLMM) or a marginal model via GEE:

- GLMM: logistic regression with fixed effect for variant (lift) and a random intercept for day to capture unobserved day-level variability (and overdispersion).- GEE: population-averaged logistic model with an exchangeable or AR(1) working correlation and robust (sandwich) SEs if you're focused on average treatment effect and want inference robust to misspecified correlation.

Justification:- Aggregated counts are binomial — logistic link is appropriate.- Random day effects/GEE handle varying sample sizes per day and intra-day correlation.- GLMM models heterogeneity and gives subject-specific estimates; GEE gives marginal effect and robust SEs (useful if days are correlated but you don't trust random-effect distribution).

How to check independence/time-correlation assumptions:- Plot daily conversion residuals by day and compute ACF/PACF; look for significant lags → if present prefer AR(1) structure in GEE or include lagged covariates.- Fit model and inspect Pearson/deviance residuals vs time; test for overdispersion (observed variance > binomial).- Use Durbin-Watson or Ljung-Box on residuals for serial correlation.- Sensitivity: fit GLMM, GEE (exchangeable and AR(1)), and a block bootstrap (resample contiguous day blocks) to compare CIs — if conclusions agree, result is robust.

Example (python/statsmodels, conceptual):

python

import statsmodels.api as sm
# data: df with cols conversions, impressions, variant (0/1), day
df['success'] = df['conversions']
df['fail'] = df['impressions'] - df['conversions']
model = sm.GEE.from_formula("success + fail ~ variant", groups="day", data=df,
                           family=sm.families.Binomial(), cov_struct=sm.cov_struct.Exchangeable())
res = model.fit()
print(res.summary())

Edge notes:- If days small (n=14) be cautious: prefer GEE with robust SEs and bootstrap CIs.- Report both point estimate (odds-ratio or marginal lift) and CI; perform sensitivity to correlation choice.

HardSystem Design

29 practiced

Design CI/CD gating rules for ML model rollout that incorporate offline hypothesis tests on held-out data, online experiment results, and production monitoring alerts. Specify measurable thresholds for promotion, rollback criteria, automated checks, and how to avoid blocking useful changes due to overly conservative statistical gates.

Sample Answer

Requirements (brief):- Promote models that reliably improve business metric(s) without introducing regressions, bias, or operational risk.- Use offline held-out hypothesis tests, online experiments, and production monitoring.- Provide measurable promotion/rollback thresholds, automated checks, and avoid blocking useful changes via overly strict stats.

High-level flow:1. Pre-deploy CI checks (offline).2. Staged rollout with online experiment (canary → ramp).3. Production monitoring and automated rollback.

Key components and rules1) Offline (CI) gates — run on held-out test set and validation slices- Metrics: primary business metric (e.g., conversion rate, revenue per user), secondary metrics (accuracy/AUC, latency), fairness metrics (demographic parity difference), and unit/regression tests.- Statistical criteria: require estimated effect size ≥ Minimum Detectable Effect (MDE) and 95% confidence interval excluding zero OR Bayesian posterior P(improvement > 0) ≥ 0.95. - Example: Δconversion >= 0.5 percentage points and 95% CI lower bound ≥ 0.0; or posterior prob > 0.95.- Power check: ensure test set size provides ≥ 80% power to detect MDE.- Automated checks: schema validation, feature drift on training vs test (PSI < 0.1), model card generated, fairness thresholds (max demographic uplift < 2× baseline), latency < SLA.

2) Online experiment (canary / A/B)- Start with canary: 1% traffic for 12–24 hours, then 5% for 24–72 hours, then 20% if stable.- Experimental evaluation: run sequential A/B tests with pre-specified stopping rules (alpha spending / group sequential or Bayesian stopping).- Decision rule (frequentist): require statistically significant improvement (two-sided p < 0.05 after sequential correction) in primary metric OR Bayesian posterior P(improvement > MDE) ≥ 0.95 across the pooled traffic.- Guardrails: no degradation beyond tolerance in secondary safety metrics (e.g., latency increase ≤ 10%, failure rate increase ≤ 0.1 percentage points). If safety metrics exceed thresholds at any stage, pause ramp and revert to previous model.

3) Production monitoring & rollback- Real-time monitors: primary metric (moving-window), prediction distribution drift (PSI > 0.25), feature missingness spike (> 3σ), latency SLA breaches (> 95th percentile > target), and customer-facing error rates.- Automated rollback triggers: - Absolute rule: if primary metric degrades by > X% (or X absolute points) vs baseline for T consecutive windows (e.g., >1% absolute drop for 3 consecutive hours). - Statistical rule: observed drop outside the precomputed CI of expected variation (e.g., z-score > 3). - Operational rule: latency or error-rate breaches exceeding SLA for > 30 minutes. - Safety/fairness rule: fairness metric crosses threshold (e.g., demographic disparity > 5%).- Rollback action: instantly route traffic back to previous stable model; create an incident ticket with diagnostics.

Avoiding overly conservative gates- Use practical MDE: set MDE to business-meaningful minimum (not near-zero). This prevents rejecting true small-but-useful uplifts.- Compute power and sample-size: only require statistical tests when sample size/power is sufficient; otherwise rely on Bayesian updating with priors.- Use adaptive/sequential testing: alpha-spending or Bayesian sequential allows early detection without inflated Type I error.- Combine evidence: require concordance across offline + online + monitoring — e.g., if offline shows modest uplift but online is inconclusive due to low traffic, allow a prolonged canary instead of hard block.- Use risk-based gates: low-risk model changes (non-customer-facing) get relaxed thresholds; high-risk (payment, safety) get stricter.- Human-in-the-loop overrides: surface dashboards and alerts to SRE/ML owners with documented criteria to approve promotion in ambiguous cases.

Operational details and metrics to set initially (example defaults)- MDE = 0.5 percentage points for conversion, power ≥ 80%- Offline CI: 95% lower bound > 0- Bayesian threshold: P(Δ > MDE) ≥ 0.95- PSI thresholds: <0.1 (good), 0.1–0.25 (watch), >0.25 (block)- Canary ramp: 1% (12h) → 5% (24–72h) → 20% (48h) → 100% (if stable)- Rollback: primary metric down >1% absolute for 3 consecutive 1-hour windows OR z-score >3 OR latency 95p > SLA for 30m

Trade-offs and scalability- Tight gates reduce risk but increase time-to-market; mitigate with risk-based thresholds and Bayesian/seq testing.- Automate diagnostics to reduce cognitive load on humans and speed decisions.- Store all experiment data and decisions to refine thresholds over time.

This design balances statistical rigor with operational practicality: measurable thresholds, staged ramping, automated rollback, and adaptive, risk-based mechanisms to avoid blocking useful changes.

Unlock Full Question Bank

Get access to hundreds of Hypothesis Testing and Inference interview questions and detailed answers.

Join thousands of developers preparing for their dream job.