Experimentation and Product Validation Questions

Designing and interpreting experiments and validation strategies to test product hypotheses. Includes hypothesis formulation, experimental design, sample sizing considerations, metrics selection, interpreting results and statistical uncertainty, and avoiding common pitfalls such as peeking and multiple hypothesis testing. Also covers qualitative validation methods such as interviews and pilots, and using a mix of methods to validate product ideas before scaling.

HardTechnical

53 practiced

Some features cause users in treatment to influence control users (network effects). Describe how you'd detect SUTVA violations and propose experimental designs to handle interference such as cluster randomization, graph-clustering randomization, and randomization inference. Include how you'd estimate treatment effects under interference.

Sample Answer

Detecting SUTVA violations- Exploratory checks: compute correlation between a user’s outcome and fraction of their neighbors assigned to treatment in the A/B holdout (or pre-period). A significant relationship suggests interference.- Balance tests: after randomization, test whether network exposure (e.g., number/proportion of treated neighbors) is balanced across arms — imbalance implies potential interference or bad randomization.- Event-timing & dose–response: check whether outcomes change with temporal proximity to treated neighbors or with increasing treated-neighbor fraction.- Fisher-style sharp null tests: permute treatment across users preserving network structure and compute a test statistic sensitive to spillovers (e.g., difference in means conditional on neighbor treatment); large observed statistic -> reject no-interference.

Designs to handle interference1. Cluster randomization- Partition users into disjoint clusters (e.g., households, geographic units) and randomize clusters. Minimizes cross-arm edges if clusters map to high within-cluster connectivity.- Trade-offs: reduces effective sample size; requires intracluster correlation adjustment.

2. Graph-clustering randomization- Use community-detection (Louvain, METIS, spectral clustering) to create clusters that cut few edges. Randomize clusters to treatment/control. Optionally enforce cluster size caps and balance covariates.- Useful when social graph is known; yields lower spillover leakage than arbitrary clusters.

3. Two-stage / partial population randomization- Randomize groups into treated vs. control clusters; within treated clusters, randomize a fraction p of users to actually receive the feature. This identifies both direct and spillover effects by varying exposure density.

Estimation under interference- Define exposure mapping: map each unit to an exposure condition e_i (e.g., treated & k treated neighbors, control & >0 treated neighbors). Estimands become contrasts between exposure conditions (direct, indirect, total effects).- Horvitz–Thompson / Hajek estimators: weight observed outcomes by inverse probability of units’ exposure condition under the randomization to get unbiased estimates.- Regression with exposure indicators: Y_i = α + β_direct * 1{treated_i} + β_spill * f(neighbor_treatment_i) + controls + cluster-robust SEs. Use randomization-based SEs or bootstrap respecting clusters.- Randomization inference (Fisher): test sharp nulls for direct and spillover effects by permuting assignments consistent with experimental design; compute p-values and confidence intervals via inversion.- Model-assisted estimators: use generalized propensity scores for exposure patterns or G-computation when network is large; combine with doubly robust techniques for efficiency.- Adjust for interference in power/sample-size: account for design effect = 1 + (m-1)ρ where m=avg cluster size and ρ=intracluster corr; for graph designs, estimate effective sample via simulation on observed graph.

Practical workflow1. Map network and estimate cross-arm edge fraction; run pre-experiment exploratory interference checks.2. Choose design: cluster or graph-cluster; if feasible, use two-stage randomization to separately estimate direct/spillover.3. Pre-register exposure definitions and estimands.4. Analyze using exposure-mapping estimators, HT/Hajek weights, regression with cluster-robust or randomization-based inference; report both direct and indirect effects plus sensitivity analyses (vary exposure thresholds, cluster definitions).5. Communicate trade-offs: lower bias vs. lower power; show simulated power and expected bias under plausible spillover models.

This approach provides principled detection and estimation when SUTVA fails, combining design choices (to reduce interference) with randomization-based inference and exposure-based estimands to correctly quantify direct and spillover effects.

MediumTechnical

61 practiced

Design a sample-size calculation for a continuous metric (e.g., daily time-on-site per user) where standard deviation is large relative to the mean. Describe how the metric's variance influences sample size, show the formula for detecting a small standardized effect (Cohen's d = 0.2) with 80% power and alpha=0.05, and explain practical implications.

Sample Answer

Approach (short): For a two-arm A/B test on a continuous metric, sample-size depends on the variance of the metric and the absolute effect you want to detect. Using standardized effect (Cohen’s d = Δ/σ) makes the dependence on σ explicit and easy to reason about when σ is large relative to the mean.

Formula (two independent samples, equal n, two-sided α=0.05, power 80%):n_per_group = ((Z_{1-α/2} + Z_{1-β})^2 * 2 * σ^2) / Δ^2Rewrite in terms of Cohen’s d = Δ/σ:n_per_group = 2 * (Z_{1-α/2} + Z_{1-β})^2 / d^2

Plug numbers: Z_{1-α/2}=1.96, Z_{1-β}=0.84 → (1.96+0.84)=2.80; (2.80)^2=7.84So n_per_group = 2 * 7.84 / d^2 = 15.68 / d^2

For Cohen’s d = 0.2:n_per_group = 15.68 / (0.2)^2 = 15.68 / 0.04 = 392=> ~392 users per arm

Interpretation and variance influence:- n scales with σ^2 (if you use absolute Δ): doubling σ quadruples required n. Using d removes σ from the formula but Δ needed = d * σ, so high σ means the same d corresponds to a larger absolute change.- With large σ relative to mean, detecting small absolute changes requires large samples. Example: mean time-on-site = 60s, σ = 120s (high variability). For d=0.2, Δ = 0.2 * 120 = 24s; need ~392 per arm to detect a 24s change.

Practical implications & recommendations:- If σ is large, consider reducing noise before up-weighting sample size: - Transform metric (log) to stabilize variance if distribution is skewed. - Use within-subject or paired designs when possible (reduces variance). - Improve metric definition (per-session vs per-user, filter bots). - Increase experiment duration or use more users only after cost/benefit analysis.- Check assumptions: central-limit theorem helps for large n; for heavy tails consider robust tests or bootstrap for power estimation.- Report minimal detectable effect in both standardized (d) and absolute units (Δ = d·σ) so stakeholders understand business impact.

This gives a transparent rule: for d=0.2, expect ~400 users per arm; if σ is larger, absolute Δ grows linearly with σ and sample size grows with σ^2 unless you reduce variance.

EasyTechnical

57 practiced

Calculate the required sample size per variant to detect an absolute lift of 2 percentage points from a baseline conversion rate of 10% to 12% with 80% power and 5% two-sided alpha. Show the normal-approximation formula you used, compute numeric answer, and state the main assumptions behind the calculation.

HardTechnical

78 practiced

How would you evaluate long-term outcomes (e.g., 12-month LTV) when experiment duration and business needs force you to make decisions earlier? Discuss surrogate endpoints, bridging models, survival analysis, and ways to estimate long-term incremental value from shorter-term experimental data. Explain key assumptions and validation checks.

Sample Answer

Start by framing the problem: you need a 12‑month incremental LTV decision but only have short‑term experiment data (e.g., 4 weeks). There are three complementary approaches: surrogate endpoints, bridging models, and survival/renewal analysis. Use them together and surface assumptions + validation.

1) Surrogate endpoints- Identify short‑term metrics that causally mediate long‑term value (e.g., first‑week purchases, retention at 7 days, trial conversion). Build models estimating how treatment affects the surrogate and how surrogate predicts 12‑month LTV.- Key assumption: surrogate fully captures treatment effect on long LTV (no unmeasured pathways). Validate via historical cohorts: compute proportion of treatment effect explained by surrogate (Prentice criteria or proportion mediated).

2) Bridging (forecasting) models- Train models mapping early behavior + user features to long‑term LTV (regression, gradient boosting, or hierarchical Bayesian models). Use cohort history to learn mappings and apply to experiment users.- Include uncertainty: produce prediction intervals and propagate treatment assignment uncertainty to estimate incremental LTV.- Assumptions: stationarity of user behavior and treatment effect; no new confounders. Validate by backtesting: simulate “stop early” on past cohorts and compare predicted vs actual 12‑month outcomes.

3) Survival / renewal analysis- Model event timing (repeat purchases, churn) with survival models (Cox, parametric like Weibull) or renewal processes (Gamma‑Poisson/Beta‑Binomial). Fit to historical inter‑purchase distributions and allow treatment to shift hazard/scale parameters.- Use these to extrapolate future transaction rates and monetization per event to compute LTV.- Assumptions: hazard form stable and treatment effect multiplicative/shiftable. Validate via time‑split holdouts and goodness‑of‑fit (Kaplan‑Meier overlays, residuals).

Practical workflow- Combine: predict individual 12‑month LTV = model(early behavior, covariates) + survival‑based extrapolation of future events. Use hierarchical/Bayesian model to pool information and quantify posterior uncertainty.- Report: point estimate of incremental LTV, 95% CI, probability treatment ROI > threshold, sensitivity analyses under alternative assumptions (e.g., weaker surrogate, non‑stationary retention).- Validation checks: backtest on historical cohorts with “early stopping,” mediation proportion, calibration plots, coverage of prediction intervals, and stress tests (assume worst‑case drift).

Communicate assumptions and risk to stakeholders, provide recommended decision rule (e.g., require >X% probability incremental LTV > cost) and plan for continued monitoring post‑rollout.

EasyTechnical

60 practiced

What makes a good primary metric for an A/B test? Compare ratio metrics (e.g., conversion rate) and absolute metrics (e.g., revenue per user), discuss sensitivity to sample size, stability over time, and trade-offs when choosing guardrail metrics for growth experiments.

Unlock Full Question Bank

Get access to hundreds of Experimentation and Product Validation interview questions and detailed answers.

Join thousands of developers preparing for their dream job.