Covers fundamental probability theory and statistical inference from first principles to practical applications. Core probability concepts include sample spaces and events, independence, conditional probability, Bayes theorem, expected value, variance, and standard deviation. Reviews common probability distributions such as normal, binomial, Poisson, uniform, and exponential, their parameters, typical use cases, computation of probabilities, and approximation methods. Explains sampling distributions and the Central Limit Theorem and their implications for estimation and confidence intervals. Presents descriptive statistics and data summary measures including mean, median, variance, and standard deviation. Details the hypothesis testing workflow including null and alternative hypotheses, p values, statistical significance, type one and type two errors, power, effect size, and interpretation of results. Reviews commonly used tests and methods and guidance for selection and assumptions checking, including z tests, t tests, chi square tests, analysis of variance, and basic nonparametric alternatives. Emphasizes practical issues such as correlation versus causation, impact of sample size and data quality, assumptions validation, reasoning about rare events and tail risks, and communicating uncertainty. At more advanced levels expect experimental design and interpretation at scale including A B tests, sample size and power calculations, multiple testing and false discovery rate adjustment, and design choices for robust inference in real world systems.
MediumTechnical
66 practiced
You want to test whether the empirical distribution of user click-through rates follows a log-normal distribution. Compare goodness-of-fit options: Q-Q plot, Kolmogorov-Smirnov test, and chi-square goodness-of-fit. Explain implementation steps, limitations for large samples, and which diagnostic plots you would show stakeholders to convey fit quality.
Sample Answer
Approach summary: Use a mix of visual (Q–Q plot) and statistical tests (KS, chi-square) because each answers different questions — visual reveals where deviations occur, tests give p-values but are sensitive to sample size and assumptions.Q–Q plot- Implementation: fit log-normal (estimate mu, sigma on log-CTR), plot empirical quantiles vs theoretical log-normal quantiles, add y=x and a loess band for uncertainty.- Strengths: shows tail/median misfit and heteroscedastic patterns; intuitive for stakeholders.- Limitations: subjective; needs clear annotation.Kolmogorov–Smirnov (KS)- Implementation: estimate parameters from data (use Lilliefors-corrected or perform parametric bootstrap) then compute max |F_empirical − F_theoretical|.- Strengths: nonparametric, uses full distribution.- Limitations: when parameters are estimated, standard KS p-values are invalid — must use corrected versions or simulate null distribution. Very large samples make KS detect tiny, practically irrelevant differences (high power).Chi-square goodness-of-fit- Implementation: bin data into k meaningful intervals (ensure expected counts ≥5), compute sum (O−E)^2/E, compare to χ² with k−1−p df (p = estimated params).- Strengths: familiar, interpretable by bins; shows which ranges contribute to mismatch.- Limitations: arbitrary binning, loss of power versus KS, sensitive to low counts; with large n, small deviations produce significant χ².Limitations for large samples- All tests become very powerful: tiny, practically unimportant deviations yield low p-values. Emphasize effect size and practical significance (e.g., KS statistic magnitude, QQ deviations in percent).Recommended diagnostic deliverables for stakeholders- Q–Q plot with annotated deviations and shaded CI.- Probability plot or PP-plot to highlight cumulative differences.- Binned histogram overlaid with fitted log-normal PDF and residuals by bin (observed − expected).- Table summarizing test statistics (KS with bootstrap p-value, χ², sample size) plus interpretation: “statistically significant” vs “practically meaningful” and recommended next steps (transform, robust modeling, or mixture models).
MediumTechnical
64 practiced
Explain multiple hypothesis testing and how the Benjamini-Hochberg procedure controls the false discovery rate (FDR). Given 100 p-values, describe step-by-step how to implement BH at alpha=0.05, explain how to compute adjusted p-values, and discuss contexts where BH is preferable to Bonferroni correction.
Sample Answer
Multiple hypothesis testing: when you run many statistical tests, the chance of false positives (Type I errors) increases. Family-wise error rate (FWER) controls the probability of any false positive (e.g., Bonferroni). False discovery rate (FDR) controls the expected proportion of false positives among rejected hypotheses — a less conservative, often more powerful criterion for large-scale testing.Benjamini–Hochberg (BH) procedure (controls FDR at level α):Step-by-step for 100 p-values (m = 100) at α = 0.05:1. Sort p-values ascending: p_(1) ≤ p_(2) ≤ ... ≤ p_(100).2. For each rank k from 1 to 100 compute critical value: (k/m) * α = k/100 * 0.05.3. Find the largest k (call it k*) such that p_(k) ≤ (k/100)*0.05.4. Reject all null hypotheses with p ≤ p_(k*). If none satisfy, make no rejections.Adjusted (BH) p-values (step-up adjusted p-values) — give each test a pBH that can be compared directly to α:- For sorted p-values compute raw adjusted q_k = (m/k) * p_(k).- To ensure monotonicity, set adjusted pBH_(k) = min_{j ≥ k} q_j, and cap at 1.- Then map adjusted p-values back to original hypotheses order.Why BH vs Bonferroni:- Use BH when you expect multiple true effects and want higher power while tolerating some proportion of false discoveries (e.g., genomics, A/B testing with many metrics, exploratory analyses).- Bonferroni is preferred when any false positive is costly (critical clinical trials, regulatory decisions) because it controls FWER.Practical notes:- BH assumes independent or positively dependent tests (works reasonably well otherwise, but alternatives exist for arbitrary dependence).- Report both raw p-values, BH-adjusted p-values (q-values), and the chosen FDR threshold so stakeholders understand trade-offs.
HardTechnical
65 practiced
Explain the delta method and demonstrate how to use it to approximate the variance of a nonlinear function of an estimator. As a concrete example, derive the approximate variance and a 95% confidence interval for the log-odds g(p_hat) = log(p_hat/(1 - p_hat)) where p_hat is a sample proportion with variance p_hat(1-p_hat)/n.
Sample Answer
The delta method approximates the sampling distribution of a smooth function of an estimator using a first-order Taylor expansion. If sqrt(n)(θ̂ - θ) ->d N(0, σ^2) and g is differentiable at θ with g'(θ) ≠ 0, thensqrt(n)(g(θ̂) - g(θ)) ->d N(0, [g'(θ)]^2 σ^2),so Var(g(θ̂)) ≈ [g'(θ)]^2 Var(θ̂) (often implemented by plugging θ̂ for θ).Apply to g(p̂)=log(p̂/(1−p̂)) (log-odds). Compute derivative:g'(p) = d/dp [log(p/(1−p))] = 1/[p(1−p)].Given Var(p̂) = p(1−p)/n, the delta-method variance isVar(g(p̂)) ≈ [1/(p(1−p))]^2 · [p(1−p)/n] = 1/[n p(1−p)].In practice use plug-in p̂:se_g ≈ sqrt(1/[n p̂(1−p̂)]).Thus an approximate 95% CI for the log-odds isg(p̂) ± 1.96 · se_g= log(p̂/(1−p̂)) ± 1.96 · sqrt(1/[n p̂(1−p̂)]).Back-transforming gives an approximate CI for odds by exponentiating the endpoints; for p itself use inverse-logit if needed. Conditions: asymptotic accuracy requires n large with np, n(1−p) reasonably big. For small samples or p̂ near 0 or 1, consider exact methods or profile-likelihood intervals.
EasyTechnical
69 practiced
A spam filter returns a binary feature. Prior probability P(spam)=0.20. The feature is present in 90% of spam emails and 10% of non-spam emails. Compute P(spam | feature) using Bayes theorem, show your algebra and numeric result. Explain each term in Bayes formula and discuss how a change in the prior (class imbalance) affects the posterior probability. When would a data analyst prefer a Bayesian framing for classification problems?
Sample Answer
We want P(spam | feature). Bayes' theorem:P(spam | feature) = P(feature | spam) * P(spam) / P(feature).Compute denominator P(feature) by total probability:P(feature) = P(feature | spam)P(spam) + P(feature | not-spam)P(not-spam).Plug numbers:P(spam)=0.20P(feature | spam)=0.90P(feature | not-spam)=0.10P(not-spam)=1−0.20=0.80P(feature) = 0.90*0.20 + 0.10*0.80 = 0.18 + 0.08 = 0.26P(spam | feature) = (0.90 * 0.20) / 0.26 = 0.18 / 0.26 ≈ 0.6923 (≈69.2%)Explanation of terms:- P(spam): prior probability — belief about spam rate before seeing the feature (class prevalence).- P(feature | spam): likelihood — probability the feature appears when the email is spam (feature sensitivity).- P(feature): marginal probability the feature appears across all emails (normalizing constant).- P(spam | feature): posterior — updated probability an email is spam given the feature.Effect of changing the prior (class imbalance):- If P(spam) increases, P(feature) rises and the posterior P(spam | feature) increases: a common feature in spam becomes more convincing when spam is more common.- If P(spam) is very small, even a high likelihood may yield a modest posterior (false positives become more likely). Priors thus strongly influence posterior, especially with limited or ambiguous evidence.When a data analyst prefers a Bayesian framing:- When prior knowledge matters (known base rates, expert opinion).- With small datasets or rare classes where frequentist estimates are unstable.- For sequential updating as new data arrive.- When you need full posterior distributions or principled uncertainty estimates for decision-making (e.g., cost-sensitive thresholds for false positives vs false negatives).
HardTechnical
81 practiced
Compare multi-armed bandit algorithms for live optimization with randomized A/B tests aimed at unbiased inference. Explain how bandit allocation induces bias in naive effect estimates and describe statistical corrections such as inverse propensity score (IPS) weighting and doubly robust estimators that can recover unbiased estimates when assignment probabilities are logged. What logging and infrastructure are required to support these corrections?
Sample Answer
High-level comparison- Randomized A/B tests: uniform (or stratified) randomization ensures unconfounded assignment — naive difference-in-means gives unbiased estimates for causal effects with simple variance formulas.- Multi-armed bandits (MAB) for live optimization: adaptively skew assignment toward promising arms to increase cumulative reward. This improves business metrics online but breaks randomization: assignment depends on prior observed outcomes and contexts, so naive estimates are biased.Why adaptive allocation induces bias- Adaptive policy causes correlation between assignment and potential outcomes: arms that performed well early are oversampled, and later outcomes reflect this selection. Naive averages over observed outcomes mix the selection mechanism and treatment effect, producing upward-biased estimates for exploited arms (and downward for under-explored ones).Statistical corrections (requires logged assignment probabilities)1) Inverse Propensity Score (IPS) weighting- Idea: weight each observed outcome by 1 / p(a | context, history) where p is the logged probability the policy assigned that action at decision time.- IPS estimator for an arm a: sum_{i:Ai=a} Yi / p_i divided by sum_{i:Ai=a} 1/p_i (or compare weighted means across arms).- Pros: unbiased if propensities are correct and positivity holds. Cons: high variance when propensities are small; needs propensity clipping/regularization.2) Doubly Robust (DR) estimator- Combines outcome modeling and IPS to reduce variance and be robust if either the propensity model or outcome model is correct.- DR term per sample: m̂(context,a) + (I[A=a]/p) * (Y - m̂(context,A)), where m̂ estimates expected outcome given context and action.- DR is unbiased if either p is correct or m̂ is correct, often much lower variance than IPS.- Practical improvements: use cross-fitting to avoid overfitting, propensity clipping, normalization, and compute standard errors via bootstrap or influence functions.Logging and infrastructure requirements- Mandatory fields at decision time (must be recorded atomically): - unique event id and deterministic user/session id - timestamp - context/features used by the policy (or a hash that can reconstruct them) - action taken (arm) - policy-assigned probability p(action | context, history) — the exact randomized probability used - policy version / model id and random seed (for reproducibility) - reward/outcome observed and timestamp of observation (may arrive later) - any intermediate rewards or censoring indicators- Engineering needs: - Synchronous logging at decision time (before outcome) to avoid post-hoc recomputation of propensities. - Durable, joinable event store (streaming + batch warehouse) that preserves ordering and history so you can reconstruct the policy state if needed. - Data pipeline that joins decisions with downstream outcomes reliably, handles delayed/missing rewards, and records attribution windows. - Monitoring: distribution of logged propensities, effective sample sizes, and positivity checks; alerts for tiny probabilities or missing logs. - Tooling for offline evaluation: support for IPS/DR computation, cross-fitting, bootstrap, and variance-stabilizing transforms.Practical tips and pitfalls- Ensure positivity: don’t assign zero probability to arms you want to evaluate later.- Log exact propensities — recomputing them offline from model checkpoints is error-prone.- Use clipping and normalization to control IPS variance; prefer DR with good outcome models and cross-fitting.- For business reporting, report both offline-corrected estimates with confidence intervals and context about coverage (effective sample size) so stakeholders understand uncertainty.This approach lets a data analyst recover unbiased (or nearly unbiased, lower-variance) effect estimates from adaptive experiments while running bandits for online optimization — provided the decision-time logging and pipeline guarantees are met.
Unlock Full Question Bank
Get access to hundreds of Probability and Statistical Inference interview questions and detailed answers.
