Experiment Design and Practical Considerations Questions

Defining metrics to measure (primary and secondary). Estimating sample size and duration needed. Choosing between between-subjects and within-subjects designs. Considering confounding variables and how to control for them. Planning for randomization strategy. Discussing trade-offs between statistical rigor and practical constraints.

MediumSystem Design

78 practiced

Design an experiment to evaluate a new neural ranking model prioritized for different user segments (new users, light users, power users) across multiple countries. Describe the randomization scheme, sample allocation per segment, how you would control for differing baseline rates across segments, the method to test for interaction effects, and how to control family-wise Type I error while reporting segment-level results.

EasyTechnical

66 practiced

You face a high-variance metric and limited traffic. Describe the trade-offs between running a longer experiment to reach statistical power versus making a faster, less rigorous decision. Propose two pragmatic approaches (for example, sequential analysis, Bayesian credible intervals, or prioritizing different metrics) to proceed and enumerate the risks and mitigation strategies for each.

EasyTechnical

74 practiced

Explain the role of statistical power, significance level (alpha), and effect size in sample size estimation for experiments. Provide the standard formula for sample size for a two-sample test of proportions (normal approximation), and describe qualitatively how increasing or decreasing each parameter (power, alpha, effect size, baseline variance) affects required sample size. Give examples of realistic effect sizes in product experiments.

Sample Answer

**Brief definitions (role perspective)**- Statistical power (1 − β): probability test detects a true effect — we typically target 80–90% in experiments.- Significance level (α): Type I error rate (false positive rate), commonly 0.05 (two-sided) in research/product settings.- Effect size: the true difference you care about detecting (for proportions, absolute difference Δ = p2 − p1).These three drive sample size together with baseline variance (for proportions, p(1−p)).

**Standard formula (two-sample test of proportions, equal group sizes, normal approximation)**

text

n_per_group = 
  [ z_{1−α/2} * sqrt(2 * p̄ * (1−p̄))  +  z_{1−β} * sqrt(p1*(1−p1) + p2*(1−p2)) ]^2
  / (p2 − p1)^2

Where p̄ = (p1 + p2)/2, z are standard normal quantiles.

**Qualitative effects on required n**- Increase power (higher 1−β) → larger z_{1−β} → n increases (need more data to reduce false negatives).- Decrease α (more stringent) → larger z_{1−α/2} → n increases (harder to claim significance).- Smaller effect size Δ → denominator shrinks → n grows quadratically (small effects need much more data).- Higher baseline variance p(1−p) (peak near p=0.5) → numerator terms grow → n increases.

**Realistic product experiment effect sizes**- Very small absolute lifts: 0.1–0.3 percentage points (at massive traffic, e.g., monetary or engagement metrics).- Typical A/B lifts targeted: 0.5–2 pp for conversion, or 2–10% relative lifts depending on baseline.- Large UX changes may yield 3–10 pp.

As a research scientist, choose power/α based on cost of errors and downstream impact, convert relative goals to absolute Δ using baseline p, then compute n and validate with simulations for non-normal settings.

MediumTechnical

104 practiced

You are evaluating an experiment with one primary metric and 20 secondary metrics. Propose a principled approach to control false positives and discuss the practical pros and cons of Bonferroni, Holm, Benjamini-Hochberg (FDR), and hierarchical testing strategies. Recommend a strategy for exploratory vs. confirmatory reporting.

Sample Answer

**Approach summary**- Pre-specify one primary metric; treat 20 secondaries as a family for multiplicity control. Use confirmatory rules for hypotheses intended to make product decisions, and looser exploratory controls for hypothesis generation.

**Principled workflow**1. Gate on primary: test primary at alpha = 0.05. Only if primary is significant proceed to secondary analysis (protects type I error for business decision).2. For secondaries, choose method based on goal (confirmatory vs exploratory).

**Methods: pros/cons**- Bonferroni - Pros: Simple, strong family-wise error rate (FWER) control even under dependence. - Cons: Very conservative with 20 tests — low power. Adjusted alpha = 0.05/20 per secondary (or 0.05/21 if including primary).- Holm (step-down) - Pros: Uniformly more powerful than Bonferroni, controls FWER, still nonparametric about dependence. - Cons: Still conservative when many correlated tests; implementation slightly more complex.- Benjamini–Hochberg (BH; FDR) - Pros: Controls expected proportion of false discoveries (FDR), much higher power for exploratory studies, interpretable when many effects possible. - Cons: Allows some false positives; control is weaker for strong decision-making, dependence assumptions matter (use BH or BY variant).- Hierarchical (gated) testing - Pros: Matches scientific workflow: test global or primary first, then families or grouped secondaries; can dramatically improve power while controlling FWER if gates are pre-specified. - Cons: Requires careful pre-specification of gates and families; complex when effects are heterogeneous.

**Recommendation (Research Scientist)**- Confirmatory reporting (decisions, claims): use primary gate + Holm for secondaries (or hierarchical family testing by pre-specified groups). This balances FWER control and power.- Exploratory reporting (discovery, follow-ups): use BH FDR at q = 0.05 and clearly label discoveries as exploratory; follow with replication experiments with confirmatory controls.- Always report: raw p-values, adjusted p-values, method used, dependency assumptions, and effect sizes with CIs. Transparently separate confirmatory vs exploratory results and preregister when possible.

MediumTechnical

67 practiced

A primary metric is measured per-session but often aggregated per-user; sessions per user vary widely. Explain how intra-user correlation (ICC) influences experiment power and sample size. Provide the design effect formula used to adjust sample size in clustered data and describe how you would estimate ICC from historical logs and account for varying cluster sizes.

Sample Answer

**High-level effect of ICC on power**- When observations (sessions) are nested in users, positive intra-user correlation (ICC) reduces the independent information per session: sessions from same user are more similar, so effective sample size is smaller and power falls.- Practically, you must inflate the per-session sample size (or equivalently reduce observed information) by a design effect.

**Design-effect formulas**- For equal cluster size m:

text

DE = 1 + (m - 1) * ICC

- For varying cluster sizes with mean m_bar and coefficient of variation CV = sd(m)/m_bar, an often-used approximation is:

text

DE = 1 + (m_bar - 1) * ICC * (1 + CV^2)

- Exact expression using cluster counts n_i (k clusters):

text

DE = 1 + ICC * ( (sum_i n_i^2) / (k * m_bar) - 1 )

**How to estimate ICC from historical logs**- Fit a two-level random-intercept model (sessions nested in users). For continuous metric: - model: y_ij = beta_0 + u_i + e_ij, with u_i ~ N(0, sigma_u^2), e_ij ~ N(0, sigma_e^2). - estimate ICC = sigma_u^2 / (sigma_u^2 + sigma_e^2). Use lmer (R) or mixedlm (statsmodels) for ML/REML estimates.- For non-normal metrics use generalized mixed models and compute latent-scale ICC or use cluster-robust variance estimates.- Use block bootstrap or profile likelihood to get CI for ICC; check time-heterogeneity (early vs late sessions).

**Accounting for varying cluster sizes in power/sample-size**- Compute m_bar and CV from logs; compute DE using the CV-adjusted formula and inflate required per-session sample size by DE, or equivalently divide target effective N by DE to get required number of sessions.- Alternatively, perform simulation: resample users with their session counts and simulate treatment effects using estimated variance components to obtain empirical power — this captures skewed, heavy-tailed session counts and metric non-normality.

**Practical notes**- ICCs even modest (0.05–0.2) can materially increase needed sessions when users have many sessions.- Report sensitivity: show required sample sizes across a plausible ICC range and via simulation for robustness.

Unlock Full Question Bank

Get access to hundreds of Experiment Design and Practical Considerations interview questions and detailed answers.

Join thousands of developers preparing for their dream job.