Statistical Foundations for Experimentation Questions

Core statistical concepts and inference needed to design analyze and interpret experiments. Topics include hypothesis testing p values confidence intervals Type One and Type Two errors the relationship between sample size variability and interval width statistical power minimum detectable effect and effect size versus practical significance. Candidates should be able to choose and explain common statistical tests such as t tests and chi square tests contrast Bayesian and frequentist approaches at a conceptual level and describe variance estimation and variance reduction techniques. The topic covers corrections for multiple comparisons sequential testing and the risks of peeking and p hacking common misconceptions about p values and limitations of inference such as confounding and selection bias. Candidates should also be able to translate statistical findings into clear language for non technical stakeholders and explain uncertainty and limitations.

MediumTechnical

67 practiced

You have baseline conversion 2% and want to detect a 10% relative lift (0.2% absolute) with alpha 0.05 and power 0.8. Calculate (or show the formula and describe how you would compute) the approximate sample size per group required for a two-proportion z-test, and discuss feasibility and trade-offs if the required sample size is very large.

HardTechnical

48 practiced

You are asked to adopt a multi-armed bandit for serving top-performing variants to maximize conversions while still learning. Design a BI-friendly approach that balances regret minimization and unbiased effect estimation. Discuss exploration strategies (epsilon-greedy, Thompson sampling), how adaptive allocation biases estimates, and how to correct for bias in final analysis.

Sample Answer

Situation/goal: You need a BI-friendly adaptive test that serves high-performing variants (low regret) but still yields unbiased or correctable effect estimates for business reporting and decision-making.

Design summary:- Use an adaptive allocation algorithm in production (Thompson Sampling recommended) to balance exploration/exploitation. Thompson sampling minimizes regret and naturally adapts allocation proportional to posterior belief in each arm’s conversion rate.- Preserve unbiased estimation by (1) keeping a persistent randomized holdout (fixed A/B) of ~1–5% of traffic and/or (2) logging the exact adaptive assignment probability for every impression and using IPW / doubly robust estimators in analysis.

Exploration strategies and trade-offs:- Epsilon-greedy: simple, allocates 1−ε to current best and ε uniformly to others. Pros: easy to implement and explain to stakeholders. Cons: inefficient exploration, higher regret, and abrupt behavior changes that complicate variance estimation.- Thompson Sampling: samples from posterior for each arm and picks max; adapts smoothly, lower regret, efficient. Cons: slightly more complex to implement and requires Bayesian priors; assignment probabilities are implicit and must be recorded or approximated.

How adaptive allocation biases estimates:- Adaptive allocation preferentially routes more traffic to better-performing arms; naive averages will overestimate treatment effect because assignment is correlated with observed outcomes (selection bias).- Time-varying traffic, nonstationarity, and delayed outcomes amplify bias.

Practical bias-correction methods (BI-friendly):1. Always log for each event: timestamp, user id, context features, assigned arm, assignment probability p_t(arm), and observed outcome.2. Inverse Probability Weighting (IPW): weight each outcome by 1 / p_t(assigned). Gives unbiased estimate of average treatment effect if p_t known and positivity holds. High-variance if some p_t are tiny. - Estimate: ATE_hat_IPW = (1/N) * sum_i [Y_i * I{A_i=a}/p_i(a) − Y_i * I{A_i=b}/p_i(b)]3. Stabilized weights / clipping: cap extreme weights (e.g., min p=0.01) or use stabilized IPW to reduce variance.4. Doubly Robust (DR) estimators: combine a model for outcome E[Y|X,A] with IPW; consistent if either model or propensity is correct—recommended for BI because it lowers variance.5. Use the randomized holdout: provides a gold-standard unbiased ATE for dashboards and sanity checks, especially for executive reporting.6. Bootstrap or robust standard errors for CIs; account for clustering (user-level) and time.

Operational recommendations for BI:- Create dashboards showing both: (a) real-time conversion rates from the adaptive system (operational metric), (b) bias-corrected ATEs (IPW/DR) with CIs, and (c) randomized holdout results as a stability anchor.- Surface assignment probability distribution, effective sample size, weight variance, and any clipping used—so analysts understand estimate reliability.- Run periodic offline reanalysis: fit DR estimator adjusting for covariates available in BI data (device, geography, cohort) to improve precision.- Document priors, queuing delays, and any business rules that affect p_t (e.g., caps), because they affect validity.

Key takeaways:- Thompson sampling + logged assignment probabilities + DR/IPW + small randomized holdout = practical balance of low regret and reliable, BI-actionable inference.- For executive reports, prefer holdout-based or DR-corrected estimates with clear CIs and diagnostics; show both adaptive-system KPIs and unbiased estimates side-by-side.

EasyTechnical

51 practiced

You have two metrics: time-to-complete (continuous) and purchase (binary). Explain when you would use a t-test versus a chi-square test or Fisher's exact test in experiment analysis, list assumptions for each, and describe alternatives if assumptions are violated (non-normality, small counts, paired data).

MediumTechnical

68 practiced

In an observational dataset you observe higher conversion among treated users than controls but the rollout targeted high-value customers. Describe a diagnostic checklist to detect confounding or selection bias and list statistical methods to mitigate bias (propensity score matching, stratification, regression adjustment, instrumental variables). State when a randomized experiment is still preferable.

Sample Answer

Start with the causal question: higher conversion in treated users may reflect treatment effect or selection (rollout targeted high‑value customers). Diagnostic checklist first, then mitigation methods and when an RCT is preferable.

Diagnostic checklist (practical BI steps)- Verify rollout criteria: inspect targeting rules/segments (SQL audit of assignment logic).- Compare baseline covariates: age, past spend, recency, purchase frequency, region. Produce summary tables and standardized mean differences (SMD).- Propensity overlap: plot propensity score distributions for treated vs control; check common support.- Pre-treatment trends: plot conversion or revenue trends before treatment (weekly/monthly) for both groups.- Balance tests post-adjustment: SMDs, t-tests, or love plots after matching/weighting.- Falsification/placebo tests: test outcome unlikely affected (e.g., past period) or outcomes unrelated to treatment.- Sensitivity analysis: compute E-value or Rosenbaum bounds to gauge unmeasured confounding impact.- Subgroup/heterogeneity checks: ensure effect not driven by a small high-value segment.

Statistical methods to mitigate bias- Propensity score matching (PSM): match treated to similar controls on estimated propensity; check balance; good when overlap adequate.- Stratification (subclassification): bin by propensity or key covariates and estimate within strata, then aggregate.- Regression adjustment: multivariable regressions (e.g., logistic or linear) controlling for confounders; include interactions if needed.- Inverse probability weighting (IPW): weight observations by inverse propensity to create pseudo-population.- Doubly robust methods: combine IPW + outcome regression for robustness.- Instrumental variables (IV): use an instrument that affects treatment assignment but not outcome except via treatment (rare; must justify validity). Example: randomized rollout timing by region.- Difference-in-differences (DiD): if staggered rollout with pre/post data and parallel trends plausible.- Regression discontinuity: if assignment based on a cutoff score (e.g., loyalty score threshold).

When randomized experiment is preferable- When feasible and ethical: RCTs eliminate selection bias and unobserved confounding, providing the cleanest causal estimate. Prefer RCT if you can randomize assignment (even A/B on a holdout), control implementation, and measure outcomes reliably. If rollout targets high‑value customers and you need credible causal evidence to scale, run a randomized pilot or stratified randomized rollout (blocking by value segment) to get unbiased estimates and support decision-making.

Practical note: in BI work, combine diagnostics, adjusted estimates, clear visuals, and sensitivity discussion when presenting observational results; recommend an RCT when decisions require high-confidence causal claims.

HardSystem Design

57 practiced

Design a monitoring system to detect signs of p-hacking or frequent peeking across the organization. Propose detection heuristics (for example, unusually many early stopping events, p-value density spikes just below 0.05, repeated re-analyses with minor data tweaks), describe automated alerts and escalation, and recommend preventive policies to reduce occurrence.

Sample Answer

Requirements and constraints:- Detect signals of p-hacking / frequent peeking across experiments, dashboards, ad-hoc analyses.- Low false positives, privacy-preserving, integrate with analytics stack (Snowflake/BigQuery, Looker/Tableau).- Produce actionable alerts and dashboards for governance and remedial policies.

High-level architecture:- Ingest: logging of all experiment-related actions (create/stop experiment, analysis run, cohort changes, table exports, dashboard views/filters, code commits) into central event store.- Processing: ETL jobs compute heuristics and aggregate metrics daily/near-real-time.- Detection & Rules Engine: evaluates heuristics, scores risk, produces alerts.- Presentation: BI dashboard for monitoring and drilldowns; automated alerts to Slack/email/incident system.- Audit & Storage: immutable audit logs and anonymized snapshots for investigations.

Detection heuristics (examples):- Early stopping rate: fraction of experiments stopped before planned sample size; flag cohorts with >X% early stops vs baseline.- P-value bunching: density spike immediately below 0.05 (e.g., excess mass in 0.045–0.05 vs 0.03–0.04) using kernel density and z-score.- Re-analysis frequency: repeated re-runs of same experiment/metric within short window with minor data tweaks (filter changes, trimming outliers) >N times.- Sequential peeking: series of intermediate analyses where cumulative sample increases and p-value crosses threshold multiple times.- Selective metric switching: primary metric changed post-hoc or many correlated secondary metrics tested with selective reporting.- Unusual effect-size jumps coincident with analysis changes.- Cross-user correlation: same user repeatedly creates/stops experiments with borderline p-values.

Scoring & thresholds:- Combine heuristics into composite risk score (weighted), with tunable thresholds for warning/high-risk.- Use historical baseline per team to reduce false positives.

Alerts & escalation:- Tier 1 (informational): automated message to experiment owner + link to remediation guide; logged on dashboard.- Tier 2 (requires review): if risk score exceeds threshold or pattern repeats, notify experiment review board (Data Science Lead, Compliance) with audit trail and anonymized analysis snapshots.- Tier 3 (policy breach): persistent or egregious cases escalate to manager/HR with full audit.- Every alert includes: implicated experiment IDs, timeline of actions, p-value/time plots, diffs of analysis versions.

Dashboards & reports:- Organization-level heatmap of risk by team/product, time-series of p-value densities, list of flagged experiments with drill-down (action timestamps, code/SQL diffs, user activity).- Investigator mode: replay analysis steps, compare pre/post tweaks, export immutable report.

Preventive policies & controls:- Pre-registration: require experiment spec (primary metric, sample size, stopping rule) stored before collection; lockable in DB.- Enforce/encourage pre-specified stopping rules (e.g., group sequential with alpha spending) via templates in experimentation platform.- Read-only primary metric in dashboards until pre-registration complete.- Education: mandatory training on statistical best-practices and false discovery risk.- Tooling: provide safe sequential testing methods (Bayesian approaches, alpha-spending) and automated calculators.- Access & governance: limit who can stop experiments; require second-person review for early stops that change conclusions.- Regular audits: weekly sampling of flagged experiments and quarterly org-wide reports.

Privacy & legal:- Anonymize user identities in reports when possible; retain mapping in secure access for investigations only.

Trade-offs:- Aggressive thresholds catch more misuse but raise false positives and overhead; combine automated detection with human review and clear remediation paths.- Focus detection first on high-impact experiments (revenue-affecting) to prioritize effort.

This system gives BI teams actionable visibility into suspicious analysis patterns, automates low-friction prevention, and creates an audit trail for governance while balancing trust and efficiency.

Unlock Full Question Bank

Get access to hundreds of Statistical Foundations for Experimentation interview questions and detailed answers.

Join thousands of developers preparing for their dream job.