Hypothesis Testing and Inference Questions

Fundamental framework and application of hypothesis testing and statistical inference. Topics include formulating null and alternative hypotheses, understanding Type I and Type II errors, interpreting p values and confidence intervals, selecting and applying common tests such as t tests, chi square tests, analysis of variance, and non parametric alternatives, checking test assumptions, and discussing statistical versus practical significance. Candidates should explain power, significance levels, effect sizes, and common pitfalls such as misinterpreting p values or violating independence assumptions. At more advanced levels, discuss limitations of null hypothesis significance testing, alternatives such as Bayesian inference, and guidance for when different approaches are appropriate.

HardTechnical

27 practiced

Write Python code to compute Cohen's d (effect size) for two independent samples with unequal sample sizes. Explain the pooled standard deviation formula you use and how to interpret the resulting d in business terms.

Sample Answer

Approach: compute means and sample standard deviations for each group, use the pooled standard deviation weighted by degrees of freedom (unbiased estimator) for independent samples with unequal sizes, then compute Cohen's d = (mean1 - mean2) / sp. If needed for small samples, apply Hedges' g correction.

python

import math
import numpy as np

def cohens_d(x, y, hedge_correction=False):
    """
    Compute Cohen's d for two independent samples x and y (unequal sizes allowed).
    Returns (d, sp) where sp is the pooled standard deviation used.
    """
    x = np.asarray(x, dtype=float)
    y = np.asarray(y, dtype=float)
    x = x[~np.isnan(x)]
    y = y[~np.isnan(y)]
    n1 = x.size
    n2 = y.size
    if n1 < 2 or n2 < 2:
        raise ValueError("Each group must have at least 2 observations.")
    m1, m2 = x.mean(), y.mean()
    s1_sq = x.var(ddof=1)  # sample variance (unbiased)
    s2_sq = y.var(ddof=1)

    # pooled standard deviation (weighted by degrees of freedom)
    sp_sq = ((n1 - 1) * s1_sq + (n2 - 1) * s2_sq) / (n1 + n2 - 2)
    if sp_sq <= 0:
        raise ValueError("Pooled variance is zero or negative.")
    sp = math.sqrt(sp_sq)

    d = (m1 - m2) / sp

    if hedge_correction:
        # Hedges' g correction factor J
        df = n1 + n2 - 2
        J = 1 - (3 / (4 * df - 1))
        d = d * J

    return d, sp

Pooled SD explanation:- Formula: sp = sqrt( ((n1-1)*s1^2 + (n2-1)*s2^2) / (n1+n2-2) ).- Rationale: weights each group's sample variance by its degrees of freedom so groups with more precise variance estimates contribute more; appropriate when comparing independent groups assumed to have similar population variances.

Interpretation in business terms:- Cohen's d is a standardized difference in means measured in pooled SD units. Rules of thumb: ~0.2 small, ~0.5 medium, ~0.8 large.- Translate to business impact: multiply d by a meaningful SD (e.g., historical SD of monthly revenue) to get expected absolute change. Example: if d = 0.5 and monthly revenue SD = $100k, expected average uplift ≈ $50k—useful for forecasting ROI, sizing experiments, and prioritizing initiatives.- Also consider practical significance, sample sizes, confidence intervals, and whether assumptions (independence, similar variances) hold.

HardTechnical

30 practiced

Design an interrupted time series (ITS) analysis to evaluate the impact of a site redesign deployed on a specific date. Outline the statistical model (segmented regression), necessary diagnostics (autocorrelation, seasonality), and how to test for a sustained level or slope change. Describe implementation in Python/statsmodels.

Sample Answer

Approach (brief): use segmented (piecewise) regression to estimate immediate (level) and gradual (slope) changes after the redesign date. Model time trend, binary intervention indicator, and interaction for post-intervention slope; check and account for autocorrelation and seasonality.

Statistical model (segmented regression)Y_t = β0 + β1 * time_t + β2 * intervention_t + β3 * post_time_t + ε_t- time_t: continuous time index (1,2,...)- intervention_t: 0 before, 1 after (including day of deployment)- post_time_t: 0 before, (time_t - time_of_intervention +1) afterInterpretation:- β2 tests immediate level change- β3 tests change in slope (trend)

Diagnostics- Plot series + fitted lines- Residual ACF/PACF plots; Durbin-Watson; Ljung-Box for autocorrelation- Seasonal decomposition (STL) or include seasonal dummies (day-of-week, month)- Check heteroskedasticity (Breusch-Pagan)

Adjustment strategies- If autocorrelation: fit GLSAR or use OLS with HAC (Newey-West) SEs; or ARIMA with regressors (SARIMAX)- If seasonality: include seasonal dummies or Fourier terms, or model with SARIMAX

Python/statsmodels implementation (example using OLS + Newey-West and GLSAR)

python

import pandas as pd
import numpy as np
import statsmodels.api as sm
from statsmodels.stats.diagnostic import acorr_ljungbox
from statsmodels.graphics.tsaplots import plot_acf

# assume df with 'date' and 'y'
df = df.sort_values('date').reset_index(drop=True)
df['time'] = np.arange(1, len(df)+1)
intervention_date = pd.to_datetime("2024-06-01")
df['intervention'] = (df['date'] >= intervention_date).astype(int)
df['post_time'] = df['time'] - df.loc[df['date']>=intervention_date,'time'].min() + 1
df.loc[df['intervention']==0,'post_time'] = 0

# design
X = sm.add_constant(df[['time','intervention','post_time']])
ols = sm.OLS(df['y'], X).fit()
print(ols.summary())

# diagnostics
plot_acf(ols.resid)
print('Ljung-Box:', acorr_ljungbox(ols.resid, lags=[12]))

# robust SEs (Newey-West) for autocorrelation up to lag L
nw = ols.get_robustcov_results(cov_type='HAC', maxlags=7)
print(nw.summary())

# alternative: GLSAR to model AR(1)
from statsmodels.regression.linear_model import GLSAR
glsar = GLSAR(df['y'], X, rho=1)
res_glsar = glsar.iterative_fit(10)
print(res_glsar.summary())

Testing sustained change- Immediate level: H0: β2 = 0 (use t-test from model)- Slope change: H0: β3 = 0- Joint tests: F-test on β2 and β3 simultaneously- Use HAC SEs or GLSAR estimates for correct inference if autocorrelation present

Practical notes- Visualize counterfactual (predict using pre-intervention β0+β1*time)- Check sensitivity (vary intervention date, include covariates)- Report effect sizes with CI and discuss practical significance, not only p-values.

EasyTechnical

26 practiced

Your revenue-per-user data are heavily right-skewed with outliers. The product team asks whether average revenue differs between two groups. Explain when to use a non-parametric test like Mann-Whitney U, what it actually tests, and trade-offs versus a t-test.

MediumTechnical

30 practiced

You have three pricing tiers and want to test whether revenue differs across tiers. Explain when to use one-way ANOVA, outline its assumptions, and describe which post-hoc test you would use to identify which groups differ and why.

MediumBehavioral

34 practiced

Describe a past situation where you had to convince a non-technical stakeholder that a statistically significant result was not actionable due to small effect size or poor data quality. What approach did you take, what tools or visuals did you use, and what was the ultimate business decision?

Unlock Full Question Bank

Get access to hundreds of Hypothesis Testing and Inference interview questions and detailed answers.

Join thousands of developers preparing for their dream job.