Experimentation Methodology and Rigor Questions

Focuses on rigorous experimental methodology and advanced testing approaches needed to produce reliable, actionable results. Topics include statistical power and minimum detectable effect trade offs, multiple hypothesis correction, sequential and interim analysis, variance reduction techniques, heterogenous treatment effects, interference and network effects, bias in online experiments, two stage or multi component testing, multivariate designs, experiment velocity versus validity trade offs, and methods to measure business impact beyond proximal metrics. Senior level discussion includes designing frameworks and practices to ensure methodological rigor across teams and examples of how to balance rapid iteration with safeguards to avoid false positives.

MediumTechnical

59 practiced

Scenario: Your analytics team reports a statistically significant 2% lift in add-to-cart for a new recommendation model after 3 days, but retention and revenue show no improvement after two weeks. Outline how you would investigate whether the initial signal was a false positive, short-term novelty effect, or leading indicator for future gain.

MediumTechnical

81 practiced

Explain how cluster-randomized (e.g., geo or user-group) experiments affect variance and power compared to individual randomization. Provide the formula for the design effect using intra-cluster correlation (ICC) and show how it scales required sample size.

Sample Answer

**Brief answer / intuition**

Cluster randomization inflates variance relative to independent (individual) randomization because outcomes within the same cluster are correlated. That reduces effective sample size and therefore power for a given nominal sample size.

**Design effect formula**

Let m = average cluster size, ICC = intra‑cluster correlation (ρ). The design effect (DE) is:

text

DE = 1 + (m - 1) * ICC

Plain English: each subject contributes less independent information when peers are correlated; DE quantifies that loss.

**How it scales sample size and variance**

- Variance under clustering ≈ DE × variance under individual randomization.- Required nominal sample size (N_clustered) to achieve the same power as N_individual is:

text

N_clustered = DE * N_individual

or equivalently, number of clusters needed = (DE * N_individual) / m.

**Concrete example**

If ICC = 0.02 and m = 50:DE = 1 + 49 * 0.02 = 1.98 ≈ 2. So you need ~2× more subjects (or twice as many clusters adjusted by cluster size) to retain power.

**Practical notes for applied scientists**

- ICC often small (0.001–0.05) but with large m it matters. - When designing experiments prefer more clusters with smaller m rather than few large clusters to reduce DE. - Estimate ICC from historical data; perform cluster-aware power calculations (use cluster-level variance components) and consider mixed‑effects models for analysis.

HardTechnical

81 practiced

Design a two-stage experiment to test: first, whether a new onboarding modal increases activation (stage 1), and second, if for those activated it increases 30-day retention (stage 2). Explain randomization scheme, sample size considerations (power for both stages), and how to analyze the two-stage data to estimate overall business impact.

Sample Answer

**Overview & goal**Design a sequential randomized experiment: stage 1 tests whether modal increases activation; stage 2 tests whether, among those who activated, the modal increases 30‑day retention. Estimate both causal effects and overall business impact (activation rate × retention lift).

**Randomization scheme**- Randomize users at entry into two arms: Treatment (modal) vs Control (no modal) — 1:1.- Track activation event (binary within first N days). For stage 2, use randomization-preserving comparison: analyze retention among those who activated in each arm. Do NOT re-randomize activated users (avoids post‑treatment bias).

**Power & sample size**- Stage 1: power for difference in activation proportions. Use standard two-proportion formula:

text

n_per_arm = (Z_{1-α/2}+Z_{power})^2 * (p1*(1-p1)+p0*(1-p0)) / (p1-p0)^2

- Stage 2: retention comparison is also two-proportion but effective sample = expected activations per arm = n_per_arm * p_activation_arm. Compute sample so expected activated users provide desired power for retention lift δ_r. Iterate: choose n so both stage-1 and expected stage-2 power targets met.- If retention measured as time-to-event or continuous, use appropriate survival or mean-difference sample formulas.

**Analysis**- Primary stage-1: ITT estimate of activation difference (risk difference, CI, p‑value).- Stage-2: compare 30‑day retention among activated users by original assignment. Use ITT-onactivated (i.e., conditional on activation) but be aware of selection: because activation is post-randomization, conditional comparison estimates the effect of assignment among compliers. Report both: - Conditional ITT (difference among activated). - CACE/complier effect via instrumental variable using assignment as instrument for receiving modal (two-stage least squares for continuous outcome or IV for binary).- Overall business impact: decompose

text

Δ_overall = Δ_activation + p_activation_control * Δ_retention_given_activation

More precisely:

text

Overall change in retained users = p0 * (r1 - r0) + (p1 - p0) * r1

where p = activation rate, r = retention rate conditional on activation; subscripts 0/1 control/treatment. Estimate via plug-in with delta method or bootstrap to get CIs.

**Practical considerations**- Pre-specify analysis plan, alpha, covariate adjustment (logistic regression or TMLE) to increase power.- Monitor activation rates; if far from expected, re-run power calc and consider adaptive reallocation or extended sampling.- Use robust SEs and bootstrap for overall-impact CI.

MediumTechnical

117 practiced

Describe the practical differences between frequentist and Bayesian approaches for experiment analysis in product teams. Include considerations on interpretability for stakeholders, stopping rules, incorporation of prior knowledge, and how decisions (deploy/roll back) are made under each paradigm.

Sample Answer

**Brief framing (why it matters)** As an applied scientist I choose an analysis paradigm that maps to business constraints: stakeholder needs for clear statements, legal/regulatory requirements about peeking, and available prior data.

**Core practical differences** - Frequentist: reports p-values/confidence intervals based on long-run error rates. Good for standard A/B pipelines and regulatory settings. - Bayesian: reports posterior distributions and direct probabilities (e.g., P(delta > 0.5%) = 0.92). Aligns better with decision-makers who want "probability of lift."

**Interpretability for stakeholders** - Frequentist: easy shorthand ("p < 0.05") but commonly misinterpreted. Requires explanation: p-value ≠ probability the variant is better. - Bayesian: more intuitive statements ("92% chance variant increases revenue"), but requires explaining priors and credible intervals.

**Stopping rules** - Frequentist: must predefine sample size or use corrected sequential methods (alpha spending, Bonferroni, Wald) to control Type I error. Unplanned peeking inflates false positives. - Bayesian: optional stopping is formally allowed if model and priors are correctly specified — posterior updates remain valid — though practice should still guard against biased data collection and model misspecification.

**Incorporation of prior knowledge** - Frequentist: limited; can use hierarchical models or meta-analysis but not explicit priors. - Bayesian: naturally incorporates historic experiments or expert belief, improving power for small effects. Must be transparent about prior choice (weakly informative vs. skeptical).

**Decision-making (deploy / rollback)** - Frequentist: define action thresholds tied to Type I/II tradeoffs (e.g., deploy if p < 0.01 and lift > X); use power calculations to set sample size. Decisions framed as hypothesis tests. - Bayesian: decision framed via expected loss/utility — choose action that maximizes expected business value (e.g., compute expected revenue under deploy vs. keep). Can incorporate asymmetric costs (bad deploy > missed opportunity).

**Practical recommendation** For product teams I often use a hybrid: run Bayesian analysis for interpretable posteriors and decision-utility calculations, but maintain frequentist safeguards (pre-registration, minimum sample sizes, and offline simulations) to satisfy stakeholders and control operational risk.

MediumTechnical

56 practiced

Implement a simple CUPED adjustment function in Python: def apply_cuped(control_pre, treatment_pre, control_post, treatment_post): -> returns adjusted difference-in-means and variance. Assume inputs are lists or numpy arrays. Provide concise comments explaining steps. You may use numpy.

Sample Answer

**Approach**- Use CUPED: regress post-outcome on pre-outcome in pooled data to estimate theta, then adjust post outcomes: adjusted = post - theta*(pre - pre_mean). Compute difference-in-means and variance.

python

import numpy as np

def apply_cuped(control_pre, treatment_pre, control_post, treatment_post):
    # convert to arrays
    c_pre = np.asarray(control_pre)
    t_pre = np.asarray(treatment_pre)
    c_post = np.asarray(control_post)
    t_post = np.asarray(treatment_post)

    # pooled arrays for regression
    pre = np.concatenate([c_pre, t_pre])
    post = np.concatenate([c_post, t_post])

    # simple OLS slope theta = cov(pre, post) / var(pre)
    pre_mean = pre.mean()
    cov = ((pre - pre_mean) * (post - post.mean())).sum()
    var_pre = ((pre - pre_mean) ** 2).sum()
    theta = cov / var_pre if var_pre != 0 else 0.0

    # adjust outcomes per group
    def adjust(group_pre, group_post):
        return group_post - theta * (group_pre - pre_mean)

    c_adj = adjust(c_pre, c_post)
    t_adj = adjust(t_pre, t_post)

    # difference in means and pooled variance of the estimator
    diff = t_adj.mean() - c_adj.mean()
    n_c = len(c_adj); n_t = len(t_adj)
    var = c_adj.var(ddof=1)/n_c + t_adj.var(ddof=1)/n_t

    return diff, var

**Notes**- Complexity O(n). Handles zero variance in pre by falling back to theta=0.- As Applied Scientist, validate assumptions (linearity, stable pre measurement) and consider stratified CUPED or regularization for small samples.

Unlock Full Question Bank

Get access to hundreds of Experimentation Methodology and Rigor interview questions and detailed answers.

Join thousands of developers preparing for their dream job.