Covers fundamental probability theory and statistical inference from first principles to practical applications. Core probability concepts include sample spaces and events, independence, conditional probability, Bayes theorem, expected value, variance, and standard deviation. Reviews common probability distributions such as normal, binomial, Poisson, uniform, and exponential, their parameters, typical use cases, computation of probabilities, and approximation methods. Explains sampling distributions and the Central Limit Theorem and their implications for estimation and confidence intervals. Presents descriptive statistics and data summary measures including mean, median, variance, and standard deviation. Details the hypothesis testing workflow including null and alternative hypotheses, p values, statistical significance, type one and type two errors, power, effect size, and interpretation of results. Reviews commonly used tests and methods and guidance for selection and assumptions checking, including z tests, t tests, chi square tests, analysis of variance, and basic nonparametric alternatives. Emphasizes practical issues such as correlation versus causation, impact of sample size and data quality, assumptions validation, reasoning about rare events and tail risks, and communicating uncertainty. At more advanced levels expect experimental design and interpretation at scale including A B tests, sample size and power calculations, multiple testing and false discovery rate adjustment, and design choices for robust inference in real world systems.
MediumTechnical
69 practiced
Implement in Python a benjamini_hochberg(p_values: List[float], alpha: float = 0.05) -> List[bool] that returns which hypotheses to reject under the Benjamini-Hochberg procedure controlling FDR at level alpha. Explain algorithmic complexity and corner cases (ties, p-values = 0 or 1).
Sample Answer
To implement Benjamini–Hochberg (BH), sort p-values ascending with their original indices, find largest k s.t. p_(k) <= (k/n)*alpha, then reject all hypotheses with rank <= k. This yields FDR control under independence/positive dependence.
python
from typing import List, Tuple
def benjamini_hochberg(p_values: List[float], alpha: float = 0.05) -> List[bool]:
"""
Returns a list of booleans indicating which hypotheses to reject under BH procedure.
"""
n = len(p_values)
if n == 0:
return []
# attach original indices and sort by p-value ascending
indexed = sorted(enumerate(p_values), key=lambda x: x[1])
# compute thresholds and find largest k (1-based) satisfying p <= (k/n)*alpha
reject_up_to = -1
for rank, (idx, p) in enumerate(indexed, start=1):
if p <= (rank / n) * alpha:
reject_up_to = rank
# build result: reject those with rank <= reject_up_to
rejects = [False] * n
for rank, (idx, p) in enumerate(indexed, start=1):
if rank <= reject_up_to:
rejects[idx] = True
return rejects
Key points:- Time: O(n log n) for sort; Space: O(n) for indexed list and output.- Ties: Sorting is stable; tied p-values get deterministic ranks based on original order — BH uses sorted order so ties are fine.- p=0: allowed; will likely be rejected (since 0 <= threshold). p=1: rarely rejected.- If all p > thresholds, no rejections. For conservative behavior use Benjamini–Yekutieli when dependencies are arbitrary.
MediumTechnical
57 practiced
You ran 100 independent hypothesis tests at α = 0.05 and observed 7 significant results. Explain why some positives are expected by chance and describe two methods to control for false positives: Bonferroni correction and Benjamini-Hochberg FDR. Demonstrate the BH procedure on a small sorted list of p-values and explain how it controls the false discovery rate.
Sample Answer
If you run 100 independent tests at α = 0.05, under the global null you expect about 0.05 × 100 = 5 false positives on average. So observing 7 significant results is not surprising — some of those may be true effects, some simply chance.Two common ways to control false positives:1) Bonferroni correction:- Adjust per-test alpha to α' = α / m (m = number of tests). Here α' = 0.05/100 = 0.0005.- Declare a test significant only if p ≤ 0.0005.- Pros: controls family-wise error rate (FWER) — probability of any false positive ≤ α, even under dependence.- Cons: very conservative when m large, reduces power.2) Benjamini–Hochberg (BH) FDR:- Controls the expected proportion of false discoveries (FDR) among rejected hypotheses, less strict than FWER, higher power.- Procedure (for m tests): sort p-values p(1) ≤ p(2) ≤ ... ≤ p(m). Find largest k such that p(k) ≤ (k/m)·α. Reject all hypotheses with p ≤ p(k).Demonstration (sorted p-values, m=8, α=0.05):p: [0.003, 0.01, 0.02, 0.04, 0.06, 0.08, 0.12, 0.2]Compute thresholds (k/8)*0.05: [0.00625, 0.0125, 0.01875, 0.025, 0.03125, 0.0375, 0.04375, 0.05]Compare:- p1=0.003 ≤ 0.00625 → OK- p2=0.01 ≤ 0.0125 → OK- p3=0.02 > 0.01875 → stop — largest k is 2Reject p1 and p2 only.Why BH controls FDR:- BH lets the rejection threshold grow with k, accepting more discoveries while bounding the expected fraction of false rejections at α under independence (and under some dependence structures). Intuitively, by matching each ordered p-value to an increasing acceptance threshold, BH balances discovery and error rate so that, averaged over many experiments, at most α fraction of declared positives are false.
HardTechnical
61 practiced
You train a deep generative model but some training features are missing at random. Compare approaches for handling missingness: marginalizing missing values via Monte Carlo EM, multiple imputation followed by training, and variational inference that treats missing entries as latent variables. Discuss computational trade-offs and convergence diagnostics for each.
Sample Answer
Brief summary: all three treat missingness under MAR but trade off bias, computation, and diagnostics differently. Below I compare Monte Carlo EM (marginalization), multiple imputation + standard training, and variational inference (missing entries as latents), focusing on computational costs and how to detect/assess convergence.1) Monte Carlo EM (marginalize missing values)- What: E-step approximates expected complete-data log‑likelihood by sampling missing entries from current model (or an auxiliary conditional), M-step updates model parameters (often with gradient steps for deep nets).- Compute trade-offs: high cost per EM iteration due to sampling and repeated forward/backward passes; variance from finite Monte Carlo samples requires many samples or control variates. Hard to batch efficiently unless you amortize sampling.- Convergence diagnostics: monitor observed-data log-likelihood (approx via importance sampling), Monte Carlo standard error of the E-step estimate, and parameter traceplots. Watch for slow mixing/poor chains; increase samples or use Hamiltonian/importance sampling. EM guarantees only local ascent; noisy MC may break monotonicity — track expected complete-data objective and smooth with running averages.2) Multiple imputation then training- What: create K completed datasets (e.g., MICE, model-based draws), train separate generative models on each, combine results (pool parameters or predictions).- Compute trade-offs: embarrassingly parallel training across imputations — good for GPUs and distributed clusters. Cost scales linearly with K. Simpler to implement; however imputation model misspecification can bias downstream generative model. Doesn’t fully propagate uncertainty if only single imputed dataset used.- Convergence diagnostics: check between-imputation vs within-imputation variance (Rubin’s rules), sensitivity to K, compare held-out log-likelihoods across imputations, and posterior predictive checks. If estimates vary widely across imputations, increase K or improve imputation model.3) Variational inference treating missing entries as latent variables (amortized)- What: jointly optimize model parameters and variational posterior q(z, x_mis | x_obs) with an ELBO; typically use an encoder network that amortizes inference over missing patterns.- Compute trade-offs: highly scalable and GPU-friendly; single end-to-end optimization avoids outer EM loop. Amortization reduces per-sample cost dramatically for large datasets and complex missing-patterns. But variational approximations can introduce bias (underestimate uncertainty) and optimization may find poor local optima.- Convergence diagnostics: monitor ELBO (and its variance with different seeds/batches), reconstruction/held-out log-likelihood, posterior predictive checks, and calibration of imputed values (compare marginal distributions of imputed vs observed). Use importance-weighted ELBO or hierarchical q to reduce bias. Check training stability across seeds and anneal KL/learning rate schedules.Practical recommendations- For smaller datasets or when precise marginal likelihoods matter: MCEM (with advanced samplers) gives asymptotically unbiased E-step but at high cost.- For engineering scale with many examples and complex models: amortized variational inference is usually best trade-off — efficient, supports minibatches, and integrates naturally with deep generative models.- Multiple imputation is pragmatic for distributed workflows, rapid prototyping, or when you must reuse standard training pipelines; ensure enough imputations and diagnostics.- Always: validate with held-out observed-data likelihood, posterior predictive checks, sensitivity analyses (different missingness models), and measure imputation uncertainty (between-imputation variance or q variance). Consider hybrid strategies: use amortized VI with importance-weighted corrections or MCEM with an amortized sampler to balance bias and cost.
EasyTechnical
54 practiced
Apply Bayes' theorem to a realistic diagnosis example: a medical binary classifier has sensitivity P(test+ | disease) = 0.95 and specificity P(test- | no disease) = 0.90. If disease prevalence is 0.5% in the population, compute P(disease | test+) and explain the implications for model calibration and decision thresholds in low-prevalence settings.
Sample Answer
Apply Bayes’ theorem to compute P(disease | test+). Let D = disease, T+ = positive test.P(D) = 0.005 (prevalence)Sensitivity = P(T+ | D) = 0.95Specificity = P(T- | ¬D) = 0.90 ⇒ P(T+ | ¬D) = 0.10Bayes:P(D | T+) = P(T+ | D) P(D) / [P(T+ | D) P(D) + P(T+ | ¬D) P(¬D)]= 0.95 * 0.005 / [0.95*0.005 + 0.10*0.995]= 0.00475 / (0.00475 + 0.0995)≈ 0.00475 / 0.10425 ≈ 0.0456 ≈ 4.6%Implication: even with high sensitivity and good specificity, the positive predictive value (PPV) is low (~4.6%) when prevalence is 0.5%. Most positives are false positives.Practical consequences for AI models in low-prevalence settings:- Calibration matters: model output probabilities must reflect true risk (use calibration plots, Platt scaling or isotonic regression). An uncalibrated score will mislead clinicians.- Decision thresholds: choose thresholds by expected utility/costs (false positives vs false negatives). Raising threshold increases specificity and PPV at cost of sensitivity.- Use two-stage strategies: high-sensitivity screening followed by a high-specificity confirmatory test to raise PPV.- Evaluate with prevalence-aware metrics: report PPV/NPV for realistic prevalences, and use precision-recall curves rather than only ROC.- External validation: prevalence shift changes PPV; validate/calibrate in the target population.Key takeaway: in low-prevalence contexts, even accurate classifiers produce many false positives; prioritize calibration, thresholding based on costs, and multi-stage workflows.
HardTechnical
68 practiced
Design a statistical testing framework to detect fairness issues across sensitive groups for a binary classifier (e.g., differences in false negative rates across groups). Include which fairness metrics to test, how to select thresholds for binary decisions, sample size considerations for subgroup tests, and how to adjust for multiple subgroup comparisons.
Sample Answer
Requirements and goal: detect statistically significant fairness gaps across sensitive groups for a binary classifier, focusing on error-rate disparities (e.g., False Negative Rate, FNR) while controlling Type I error and accounting for sample size and multiple comparisons.Framework overview (steps)1. Define metrics to test - Primary: False Negative Rate (FNR = FN / (FN+TP)) per group - Secondary: False Positive Rate (FPR), Positive Predictive Value (PPV), calibration (Brier, reliability plots), and subgroup parity (selection rate) - Report absolute difference and ratio (Δ = FNR_group − FNR_baseline; RR = FNR_group / FNR_baseline)2. Threshold policy - If operationally one global threshold: evaluate metrics using that threshold (favours consistency). - If business allows per-group thresholds: explicitly test both global and per-group decisions; per-group thresholds should be decided transparently and subject to fairness constraints. - Also test range of thresholds (ROC/utility sweep) and report worst-case gaps across plausible thresholds.3. Statistical testing procedure - For each group g, test H0: FNR_g = FNR_ref (ref = majority or pooled) vs H1: unequal (two-sided) or one-sided if direction known. - Use test for difference of proportions (Wald/chi-square) or exact (Fisher) when counts small. Better: use two-sample z-test with pooled SE: z = (p1 − p2) / sqrt(p1(1−p1)/n1 + p2(1−p2)/n2) - Prefer logistic regression with group indicator to adjust for covariates: logit(P(y_hat=0 | y=1)) ~ group + covariates; test group coefficient.4. Sample size / power - For comparing proportions, required n per group: n ≈ [ (z_{1−α/2}√(2p(1−p)) + z_{power}√(p1(1−p1)+p2(1−p2)) )² ] / (p1−p2)² - Practical rule: ensure at least 30–50 positive actual labels per group for approximate tests; prefer exact or bootstrap when <50. - Use power analysis: decide minimum detectable effect (e.g., Δ=0.05 FNR) and power (80–90%) to compute sample needs.5. Multiple comparisons - If testing many subgroups, control false discoveries: use Benjamini–Hochberg (FDR) for discovery-focused monitoring; use Bonferroni or Holm for strict family-wise error when decisions are high-stakes. - Alternative: hierarchical testing (first test coarse groups, then subgroups only if coarse test significant) or closed testing procedures.6. Practical considerations - Stratify by outcome: tests for FNR require sufficient positives per group; for FPR, sufficient negatives. - Adjust for covariates that legitimately affect risk via regression or propensity scoring to separate feature-caused differences from unfairness. - Use bootstrap CIs for Δ and permutation tests when distributional assumptions fail. - Produce dashboards with effect sizes, CIs, p-values (adjusted), and practical impact (e.g., number of additional missed positives). - Monitor continuously and re-run tests after model or population shifts.Example summary- Decide global threshold = 0.5. Compute FNR per group (requires positive counts). Run logistic regression conditioned on y=1 with group indicator to get p-value; supplement with Δ and 95% bootstrap CI. If testing 10 groups, apply BH at q=0.05. Require at least 50 positives per group or use exact/bootstrap otherwise. If a significant gap exists, quantify operational impact and propose mitigations (rethresholding, reweighting, collecting more data, model retraining with fairness constraints).
Unlock Full Question Bank
Get access to hundreds of Probability and Statistical Inference interview questions and detailed answers.
