Experimentation Strategy and Advanced Designs Questions

When and how to use advanced experimental methods, and how to prioritize experiments to maximize learning and business impact. Candidates should understand factorial and multivariate designs, interaction effects, blocking and stratification, sequential testing and adaptive designs, and the trade-offs between running many factors at once versus sequential A/B tests in terms of speed, power, and interpretability. The topic includes Bayesian and frequentist analysis choices, techniques for detecting heterogeneous treatment effects, and methods to control for multiple comparisons. At the strategy level, candidates should be able to estimate expected impact, effort, confidence, and reach for proposed experiments, apply prioritization frameworks to select experiments, and reason about parallelization limits, resource constraints, tooling, and monitoring. Candidates should also be able to communicate complex experimental results, recommend staged follow-ups, and design experiments to answer higher-order questions about interactions and heterogeneity.

MediumTechnical

70 practiced

Explain regression discontinuity (RD) as a quasi-experimental design and give a growth use-case where RD would be preferable to randomized assignment. Describe the assumptions necessary for causal interpretation and how you'd test them.

Sample Answer

Regression discontinuity (RD) is a quasi-experimental design that estimates causal effects by exploiting a known cutoff on a continuous assignment variable (the running variable). Units just above and just below the threshold are assumed to be comparable, so differences in outcomes at the cutoff reflect the treatment effect.

Key idea:- Treatment = 1 if running variable X ≥ c (cutoff), else 0.- Estimate local average treatment effect (LATE) at X = c using local linear regressions on either side.

Growth use-case where RD beats randomization:- A freemium product grants a growth-hack feature (e.g., advanced analytics) to users whose monthly spend ≥ $50. Randomizing would be operationally complex or unfair to high-spending customers. RD uses the natural cutoff (spend = $50) to estimate the causal lift on retention or ARPU for marginally-eligible users.

Assumptions for causal interpretation:1. Sharp or fuzzy RD correctly specified: - Sharp: treatment deterministically assigned by cutoff. - Fuzzy: cutoff changes probability of treatment (instrumental variable).2. Continuity of potential outcomes in X at the cutoff (no jump in potential outcomes absent treatment).3. No manipulation of the running variable around the cutoff (no precise sorting).4. Covariates are smooth at the cutoff.

How to test assumptions:- McCrary density test to detect discontinuities in the running variable (tests manipulation).- Covariate balance: plot/pre-test baseline covariates against X and test for discontinuities at c.- Visual inspection: scatterplot of outcome vs. X with fitted local polynomials on each side to see an immediate jump.- Sensitivity checks: vary bandwidths, polynomial orders, and use robust bias-corrected CIs; use placebo cutoffs away from c.- For fuzzy RD, report first-stage jump in treatment probability and use two-stage least squares for LATE, checking instrument strength.

Practical notes for a data analyst:- Preprocess: clean X, remove ties at cutoff if necessary, and document treatment assignment rules.- Report: local effect at cutoff, bandwidth choice rationale, robustness checks, plots, and limitations (external validity beyond near-cutoff users).

EasyTechnical

64 practiced

You must present a complex factorial experiment with interactions to a non-technical executive. Draft a concise structure (3–5 bullets) for the slide or verbal explanation that covers the question tested, headline results, business impact scenarios, uncertainty, and recommended next actions.

EasyTechnical

72 practiced

Draft a concise pre-analysis plan checklist for a product A/B test. Include: metric definitions, exclusion rules, analysis method, stopping rules, and how to handle missing data. The checklist should be clear enough to be attached to a ticket or experiment record.

Sample Answer

Pre-analysis plan checklist — A/B Test (to attach to ticket)

1) Experiment metadata- Test name, start date, owner, hypothesis, primary/secondary metric hierarchy, sample size target.

2) Metric definitions (exact SQL-friendly)- Primary metric: e.g., daily_active_conversion_rate = count(distinct user_id where conversion_event between t0 and t0+7)/count(distinct user_id in cohort)- Secondary metrics: avg_revenue_per_user (ARPU), retention_7d, error_rate.- Denominators and windows explicitly specified.

3) Exclusion rules- Exclude internal/staging accounts (account_type IN ('internal','bot'))- Exclude users with < N historical events if using behavioral eligibility- Exclude duplicate enrollments; use first assignment only- Minimum data availability window: user must have at least X days of exposure

4) Analysis method- Intent-to-treat (ITT) on assigned users- Aggregation level: user-week (or user)- Statistical test: two-sided Welch’s t-test on metric per-user; supplement with bootstrap for skewed revenue- Confidence level: 95%; report effect size and CI- Adjust for multiple comparisons only if >3 primary tests (method: Benjamini-Hochberg)

5) Stopping rules- No peeking rule: do not stop early based on interim p-values- Pre-specified optional interim analysis at 50% sample with alpha spending (O’Brien-Fleming) — must be approved and documented- If safety or major regression detected, stop immediately and run safety analysis

6) Missing data handling- Define missing: absent events vs. null revenue- For missing covariates: impute with predefined defaults or indicator variables; document method- For missing outcomes: treat as zero for conversion/revenue if logically absent; otherwise exclude and report sensitivity analysis (worst/best-case bounds)

7) QA & reproducibility- Attach SQL queries, cohort logic, and analytic notebook to experiment record- Pre-register expected sample size and power calculation- Log any deviations from plan with rationale and timestamp

Sign-off: Analyst / Product / PM / Engineering (names & dates)

EasyTechnical

73 practiced

Describe the difference between a p-value and a confidence interval in the context of analyzing an A/B test. Explain what each communicates to a non-technical stakeholder and one common misinterpretation to avoid for each.

MediumTechnical

75 practiced

Design a constrained multivariate experiment to test 3 headline options and 2 hero-image variants on the landing page, where traffic is limited and you want to keep interpretability high. Describe assignment, sample allocation, and analysis plan.

Sample Answer

Goal: Test 3 headlines (H1, H2, H3) × 2 hero images (I_A, I_B) on a traffic-constrained landing page while keeping results interpretable and statistically useful.

Design choice (constrained factorial / fractional with blocking):- Use a blocked 2×3 factorial with traffic-aware allocation. Treat headline and image as separate main factors and prioritize estimating main effects and headline×image interaction if possible.

Assignment and randomization:- Randomize at session or visitor level using deterministic hashing of visitor_id into experiment buckets to avoid cross-contamination.- Block by major traffic segment (e.g., desktop vs mobile, or geography) to control heterogeneity; randomize within each block.

Sample allocation:- With limited traffic, allocate more to estimating headlines: use a 3:3:3:2:2:2 proportional split across the 6 cells? Simpler and interpretable: equal allocation to 6 cells (1/6 each). If extreme constraint, run a main-effects fractional design: test all 3 headlines with one balanced image assignment (e.g., orthogonal Latin square) so headline comparisons get higher effective sample.- Minimum sample planning: compute minimum detectable effect (MDE) for primary metric (conversion rate) and ensure each cell meets power target (e.g., 80% power, alpha=0.05). If impossible, reduce to headline-only test and run image A/B sequentially after.

Analysis plan:- Pre-register primary metric and analysis window. Primary analysis: two-way ANOVA / logistic regression depending on outcome (continuous KPI vs binary conversion) with terms: headline, image, headline×image, and blocking covariates (device, day). Use robust standard errors clustered by visitor.- If low power for interaction, focus on main effects: estimate headline marginal effects (average across images) and image marginal effects (average across headlines) with 95% CIs.- Adjust for multiple comparisons using Benjamini-Hochberg for headline pairwise tests to control false discovery while preserving power.- Perform sequential monitoring only with an alpha-spending plan (O’Brien–Fleming) if checking early.- Diagnostics: check balance across covariates, inspect per-block effects, and run uplift and heterogeneity analyses (by device) only as exploratory.- Reporting: report estimated lift, CI, p-values, sample sizes per cell, and recommendation (pick best headline if significant main effect; only recommend headline-image combo if interaction is significant and powered).

This keeps design interpretable (clear main effects), robust to low traffic (option to reduce to main-effects design), and statistically principled.

Unlock Full Question Bank

Get access to hundreds of Experimentation Strategy and Advanced Designs interview questions and detailed answers.

Join thousands of developers preparing for their dream job.