Statistical Foundations for Experimentation Questions

Core statistical concepts and inference needed to design analyze and interpret experiments. Topics include hypothesis testing p values confidence intervals Type One and Type Two errors the relationship between sample size variability and interval width statistical power minimum detectable effect and effect size versus practical significance. Candidates should be able to choose and explain common statistical tests such as t tests and chi square tests contrast Bayesian and frequentist approaches at a conceptual level and describe variance estimation and variance reduction techniques. The topic covers corrections for multiple comparisons sequential testing and the risks of peeking and p hacking common misconceptions about p values and limitations of inference such as confounding and selection bias. Candidates should also be able to translate statistical findings into clear language for non technical stakeholders and explain uncertainty and limitations.

HardTechnical

48 practiced

An experiment shows a +10% lift in an activation metric at 7 days but cohort analysis shows -5% retention at 30 days. As the PM, how would you investigate whether the feature causes long-term harm? Propose additional analyses and experiments (longer windows, holdout cohorts, mediation analysis, sequential rollouts), and describe rollout options if initial signals conflict.

Sample Answer

Situation: We ran an experiment that increased a 7‑day activation metric by +10% but cohort analysis shows a -5% retention at 30 days. As PM I need to determine if the feature causally harms long‑term retention and what to do about rollout.

1) Clarify and hypothesize- Confirm metric definitions (activation vs retention), measurement windows, and user eligibility.- Hypotheses: short‑term activation attracts low‑quality users; feature changes behavior that reduces long‑term value; measurement/selection bias.

2) Immediate analyses- Extend experiment analysis to 30, 60, 90 days using existing randomized assignment (intention‑to‑treat). Check survival/retention curves (Kaplan‑Meier) and time‑to‑churn differences.- Segment by user cohorts (acquisition source, power users vs new users, geography, device). Look for heterogeneous treatment effects.- Funnel and engagement: compare downstream engagement (DAU, key actions) to see if activation is superficial.- Check for instrumentation issues, differential attrition, or cross‑contamination.

3) Causal/mediation analyses- Mediation analysis: quantify how much of long‑term retention change is explained by intermediate behaviors (e.g., activation events, session frequency) using causal mediation models.- Survival regression (Cox) controlling for covariates to estimate hazard ratio.- Use instrumental variables if compliance differs (e.g., encouragement design).

4) Additional experiments- Longer randomized run: power the trial for 30/60‑day retention (compute required sample size). Keep original randomization to avoid post‑treatment bias.- Holdout cohorts: maintain permanent holdouts (e.g., 5–10% of traffic) for long‑term monitoring.- Sequential rollout / canary: rollout to small percentiles (1%, 5%, 20%) and monitor short/long metrics with predefined stop/ramp rules.- Variant experiments: modify feature variants that aim to preserve activation but reduce potential harm (e.g., delay certain prompts, change onboarding flow).

5) Rollout decision framework (if signals conflict)- If extended RCT shows no long‑term harm: proceed phased rollout with monitoring and permanent holdouts.- If harm persists uniformly: pause and rollback; prioritize fixes guided by mediation results.- If harm only in subsegments: restrict rollout to safe segments; A/B test mitigations targeted at affected cohorts.- If mixed signals but uncertain: keep feature in limited rollout with holdouts and require runbook thresholds (statistical and business) for further expansion.

6) Communication & governance- Surface analytics, hypotheses, planned experiments, and risk thresholds to stakeholders.- Define KPIs, monitoring dashboards, and automatic alerts for retention regressions.

This approach balances evidence (causal RCTs + mediation), risk management (holdouts, canary), and iterative product fixes so we avoid shipping long‑term harm while preserving short‑term gains where safe.

HardTechnical

93 practiced

Your team wants to aggregate results from 20 similar past experiments to estimate an overall effect. Describe how to perform a meta-analysis, handle between-experiment heterogeneity (fixed vs random effects), weight individual estimates, and diagnose publication bias or p-hacking that might inflate the aggregated effect estimate.

Sample Answer

Approach: treat the 20 experiments as independent estimates of the same underlying quantity, then pool them with appropriate weighting and check robustness to heterogeneity and bias.

Step-by-step:1. Extract standardized effect sizes (e.g., Cohen’s d, log-odds, or mean difference) and their standard errors from each experiment. If raw data available, compute consistent metrics; otherwise convert reported statistics.2. Calculate inverse-variance weights wi = 1 / SEi^2 so more precise studies contribute more.3. Choose model: - Fixed-effect: assumes one true effect; pooled estimate = sum(wi * ei) / sum(wi). Use when heterogeneity is negligible. - Random-effects: assumes study effects vary around a mean; incorporates between-study variance τ^2. Use DerSimonian–Laird or REML to estimate τ^2, then weights become 1 / (SEi^2 + τ^2). Prefer this in product settings where contexts differ.4. Diagnose heterogeneity: compute Q-statistic, I^2 (percent variance due to heterogeneity), and τ^2. If I^2 > ~40% or Q significant, favor random-effects and explore sources.5. Explore heterogeneity: subgroup analyses, meta-regression (e.g., experiment size, platform, user segment), and leave-one-out sensitivity checks.

Detecting publication bias / p-hacking:- Funnel plot asymmetry visual check.- Egger’s regression test and Begg’s test for asymmetry.- Trim-and-fill to estimate adjusted pooled effect.- p-curve and p-uniform methods to assess evidential value (whether significant p’s are right-skewed).- Compare pooled estimates from preregistered vs non-preregistered studies; run selection-model sensitivity analyses.- Pre-specify inclusion criteria and run worst-case bias scenarios to see how robust product decisions are.

Actionable PM guidance:- Prefer random-effects for product experiments with varied contexts; report both fixed and random pooled estimates plus heterogeneity metrics.- Use meta-regression to surface moderators that inform product strategy (which segments see larger effects).- If bias suspected, down-weight non-preregistered or small studies, and recommend replication tests for high-impact decisions.

HardTechnical

61 practiced

A feature was released early because an experiment reached p<0.05 in a short run, but the full rollout later showed negative impact on key metrics. As PM, analyze possible root causes (peeking, seasonality, non-representative early sample, metric leakage), list signals and diagnostics you would inspect, and recommend policy changes to experiment monitoring and launch procedures to avoid recurrence.

Sample Answer

Situation: A feature was promoted early because an A/B test hit p<0.05 on a short run; after full rollout, key metrics declined. Below I analyze plausible root causes, list diagnostics/signals to inspect, and recommend concrete policy changes to experiment monitoring and launch procedures.

Possible root causes and why they matter- Peeking (sequential testing): Stopping when p<0.05 inflates false-positive risk — short runs are especially vulnerable.- Seasonality/time effects: Early results captured a favorable short window (weekday/weekend, holiday, marketing campaign) not representative of long-term baseline.- Non-representative early sample: Early adopters or specific geos/devices may respond differently than broader population.- Metric leakage/segmentation issues: Instrumentation bug or metric definition changed post-launch (e.g., double-counting events).- Novelty/short-term uplift: Short-term curiosity increases engagement which decays later.- External confounders: Concurrent experiments, promotions, or backend changes biased early results.

Signals & diagnostics to inspect- Statistical diagnostics: - Look at p-value trajectory and test for alpha spending; run sequential analysis (e.g., O’Brien-Fleming, Pocock) to estimate true Type I risk. - Compute Bayesian credible intervals or sequential p-adjusted estimates.- Time-series and heterogeneity: - Hourly/daily treatment vs. control lift plots across the test period. - Segment lifts by geo, device, user cohort (new vs returning), traffic source. - Compare pre-test baselines across segments to detect imbalance.- Instrumentation & integrity: - Audit event logs for dropped/duplicated events pre/post rollout. - Verify randomized assignment ratios over time and by segment. - Check for overlapping experiments and interactions.- External context: - Correlate with marketing calendar, deployments, outages, or holiday spikes.- Post-rollout analysis: - Compare short-run effect decay curve: immediate lift then regression to mean? - Monitor downstream metrics (retention, revenue, error rates) for delayed harms.

Recommended policy and procedural changes- Enforce minimum sample size and minimum duration rules based on power calculations and business cycles (e.g., at least 2 full weekly cycles for consumer-facing metrics).- Adopt sequential testing procedures: pre-specify stopping rules using alpha-spending or use Bayesian approaches to allow valid interim looks.- Require cross-segment consistency: success must hold across core cohorts or pass a heterogeneity test before rollout.- Instrumentation/governance: - Mandatory analytics smoke tests and event audits before rollout sign-off. - Canary rollouts with phased ramp (e.g., 5% → 25% → 100%) and conditional progression gates based on health metrics.- Launch checklist additions: - Explicit investigation of seasonality/confounders and confirmation no overlapping experiments. - Post-launch monitoring window with rollback criteria and automated alerts on key metric degradations.- Experiment registry & accountability: - Central registry of experiments, owners, hypotheses, stopping rules, and power calculations. - Postmortem requirement for experiments that fail to replicate, with learnings fed back into planning.

Example immediate actions (first 48–72 hours)- Pause further rollout or begin controlled rollback if harm is significant.- Run the diagnostics above and assemble a cross-functional war room (Analytics, Eng, QA, Marketing).- Publish a short postmortem outlining cause, impact, fixes, and timeline for policy adoption.

These steps reduce false positives from peeking, catch seasonality or sample bias earlier, ensure measurement integrity, and create clear operational gates so signal in an experiment more reliably predicts full-rollout outcomes.

HardTechnical

66 practiced

For a highly skewed metric like session revenue, compare analytic confidence intervals (normal approximation, log-transform) with bootstrap-based intervals in terms of bias, variance, coverage, and computational cost. Explain which approach you would choose for automated daily experiment reporting and why.

Sample Answer

Situation: You're reporting daily experiment metrics where session revenue is highly right-skewed. The choice of CI method affects bias, variance, coverage, and operational cost.

Comparison (bias / variance / coverage / cost)- Normal approximation on raw revenue: - Bias: Likely biased for mean-based CIs when distribution is skewed (sample mean is unbiased, but the normal-CI relies on asymptotic symmetry; small/ moderate samples produce asymmetric sampling distributions). - Variance: Uses analytic variance (low computational cost) but underestimates uncertainty when tails are heavy. - Coverage: Poor — nominal 95% CIs will undercover (actual coverage < nominal) for skewed distributions. - Cost: Minimal CPU; ideal for real-time but risky for correctness.

- Log-transform + normal approximation (CI on log, then retransform): - Bias: Reduces skewness, so CIs on log scale are closer to normal. Retransforming to original scale introduces retransformation bias (needs correction, e.g., smearing estimator or delta method). - Variance: Typically smaller and more stable on log scale; retransformation increases variance estimate complexity. - Coverage: Often much better than raw normal CIs if retransformation bias is corrected; still depends on sample size and zeros (sessions with $0 revenue) complicate log. - Cost: Low; analytic formulas suffice.

- Bootstrap (percentile, bootstrap-t, or BCa): - Bias: Can correct for estimator bias (BCa in particular adjusts for bias and skewness). - Variance: Captures empirical sampling variability including skew/tails; variance estimates are realistic. - Coverage: Best among these for skewed metrics when configured correctly (BCa/bootstrap-t > percentile > analytic). - Cost: High — many resamples (e.g., 5k–10k) per daily metric; but embarrassingly parallelizable. Computational burden grows with number of metrics and segments.

Recommendation for automated daily experiment reporting- Use a hybrid approach: 1. Primary automated monitoring: log-transform-based analytic CIs with proper handling of zeros (e.g., log(1+revenue) or model zeros separately) and apply a retransformation correction (smearing or delta-method-adjusted mean/CI). This is fast, stable, and substantially better than raw normal CIs for day-to-day dashboards and alerting. 2. Backstop / confirmatory checks: Run bootstrap BCa (or bootstrap-t) overnight or on demand for experiments that cross alert thresholds or at decision points. This provides higher-fidelity CIs and corrects remaining bias/coverage issues. 3. Operational mitigations: parallelize bootstraps, cache resamples, or use fewer resamples with importance sampling for speed; consider Bayesian hierarchical models (posterior intervals) as a scalable alternative for many segments.

Why: The log-transform path yields a pragmatic balance — low latency and much-improved statistical validity for daily automation. Reserving computationally expensive bootstrap for confirmatory use ensures final decisions rely on robust intervals without overwhelming production resources.

HardTechnical

68 practiced

Explain how Bayesian credible intervals and posterior probabilities can be used for product decisions, for example reporting P(effect > 1%) = 0.92. Describe how priors influence results, how to perform sensitivity analyses to priors, and how you would explain posterior-based decisions and uncertainty to non-technical stakeholders.

Sample Answer

Bayesian credible intervals and posterior probabilities turn test results into direct statements about uncertainty that are easy to use for product decisions. For example, "P(effect > 1%) = 0.92" means that, given our data and model, there's a 92% probability the true uplift exceeds 1% — not a statement about long-run frequency but about our belief now. A 95% credible interval [0.3%, 2.1%] would mean we believe the true effect lies in that range with 95% probability.

How priors influence results- Priors encode pre-existing knowledge (or lack of it). A strong prior centered near zero will shrink noisy estimates toward zero; a weak or flat prior lets the data dominate. With large samples the prior’s influence shrinks; with small samples it can be decisive.- Example: with 500 users, an informative prior that expects small effects might change P(effect>1%) from 0.92 to 0.75; with 50k users the same prior would move probability negligibly.

Sensitivity analyses to priors- Run multiple plausible priors (weakly informative, skeptical, enthusiastic) and report how posterior quantities change.- Do prior predictive checks to ensure priors imply realistic outcomes before seeing data.- Report robustness: e.g., “P(effect>1%) is 0.92 under our baseline prior, 0.88 under a skeptical prior, and 0.95 under an optimistic prior.”- Optionally compute Bayes factors or decision-theoretic expected loss under different priors to see how decisions would change.

Explaining posterior-based decisions to non-technical stakeholders- Use plain language: “We are 92% confident the change gives at least 1% lift given what we know.”- Frame decisions in business terms: translate probabilities into expected revenue, conversion counts, or risk. E.g., "92% chance of >1% lift corresponds to expected incremental revenue of $X over Y period."- Show visual aids: probability density with shaded region >1%, credible interval bars, and a short scenario table (best/middle/worst case).- Tie to an explicit decision rule: define threshold (e.g., roll out if P(effect>1%)>0.9 or expected value positive after costs), and present trade-offs (cost of false positive vs false negative).- Be transparent about assumptions: state prior choice, robustness checks, and what additional data would shrink uncertainty.

Bottom line: use posterior probabilities and credible intervals to quantify belief and expected value, test sensitivity to priors so stakeholders see robustness, and present results in business-relevant visuals and decision rules so uncertainty informs — not paralyzes — product action.

Unlock Full Question Bank

Get access to hundreds of Statistical Foundations for Experimentation interview questions and detailed answers.

Join thousands of developers preparing for their dream job.