Experimentation Strategy and Advanced Designs Questions

When and how to use advanced experimental methods, and how to prioritize experiments to maximize learning and business impact. Candidates should understand factorial and multivariate designs, interaction effects, blocking and stratification, sequential testing and adaptive designs, and the trade-offs between running many factors at once versus sequential A/B tests in terms of speed, power, and interpretability. The topic includes Bayesian and frequentist analysis choices, techniques for detecting heterogeneous treatment effects, and methods to control for multiple comparisons. At the strategy level, candidates should be able to estimate expected impact, effort, confidence, and reach for proposed experiments, apply prioritization frameworks to select experiments, and reason about parallelization limits, resource constraints, tooling, and monitoring. Candidates should also be able to communicate complex experimental results, recommend staged follow-ups, and design experiments to answer higher-order questions about interactions and heterogeneity.

HardSystem Design

75 practiced

Design an experimentation platform for a consumer product with 200M monthly active users that supports factorial, multivariate, sequential, and Bayesian experiment types. Outline core components (assignment service, feature flags, event pipeline, analysis engines, metadata store), how to ensure deterministic low-latency assignment at scale, approaches to prevent interference, and how to incorporate privacy and compliance requirements.

Sample Answer

Requirements & goals:- Serve 200M MAU with <5ms assignment latency; support factorial, multivariate, sequential (adaptive), and Bayesian experiments; strong isolation, reproducible analysis, privacy/compliance (GDPR/CCPA), auditability.

Core components (product-level view):- Assignment Service: lightweight deterministic hash-based bucketing service (client-side SDK + server-side API) that maps user-id + experiment config → treatment. Exposes stable allocation IDs, exposure events, and logs overrides.- Feature Flags: config store (consistent, versioned) with targeting rules, rollout percentages, and experiment wiring. Integrated with CI/CD for rapid rollout.- Event Pipeline: high-throughput streaming (Kafka/Kinesis) for impressions, conversions, and metadata with at-least-once delivery, enrichment workers, and de-duplication.- Analysis Engines: - Frequentist engine for A/B and factorial (batch + streaming) - Multivariate engine using ANOVA / GLMs - Sequential & Bayesian engine supporting bandits and Bayesian posterior updates (online and batched) - Notebook / SQL layer for ad-hoc analysis- Metadata Store: versioned experiment definitions, user-experiment assignment history, feature flag versions, and audit logs (immutable append-only store).

Deterministic low-latency assignment:- Use stateless assignment algorithm: stable hashing (HMAC(user_id || experiment_id || version) mod N) with consistent hashing for bucketing. Implement client SDKs to compute locally to avoid network hop; server API for controlled overrides and logging.- Cache configs via CDN/edge (TTL + SSE/Push) to ensure clients have current rules; use short-lived config versions to allow rollback.- Provide sticky assignments by embedding assignment version ID in cookies/local storage and in server-side session.

Preventing interference:- Namespace experiments and enforce exclusion rules in feature-flag config (mutual-exclusion groups).- Support blocking factors and pre-specified interaction analysis for factorial designs.- Use randomized orthogonalization: stratified randomization by key covariates to balance traffic.- Quarantine sensitive experiments: traffic holdouts, ramping policies, and automated guardrails (statistical and business metric thresholds) to stop harmful interactions.- Instrumentation for cross-experiment telemetry and causal inference (logging of concurrent experiments per user).

Privacy & compliance:- Minimize PII: assignment keys use pseudonymous IDs; hashing with salt managed per-region.- Data residency: partition pipelines/storage by region; use encryption at rest and in transit.- Consent & privacy controls: SDK respects consent flags, opt-out propagation to assignment service and event pipeline; allow deletion/rectification APIs for subject access requests.- Auditability & governance: immutable experiment manifests, change approvals, role-based access, and automated compliance reports.- Differential privacy / aggregation: for reporting on small cohorts apply DP noise or thresholding; enforce k-anonymity for exports.

Operational & product considerations:- SLAs, monitoring (latency, assignment correctness, drift), capacity planning, and cost controls.- UX: self-serve experiment builder, templates for factorial/multivariate/ Bayesian, pre-checks that surface likely interference and privacy impact.- Roadmap: add automated causal discovery, synthetic holdouts, richer Bayesian priors, and tighter SDK telemetry.

This design balances performance, reproducibility, and safety for large-scale experimentation while giving product teams self-serve capabilities and governance.

HardTechnical

75 practiced

Quantitatively compare throughput and time-to-insight for two approaches: (A) run a single 10-armed multivariate test; (B) run 9 sequential A/B tests (baseline vs each variant) with early stopping. Assume baseline conversion 5%, traffic 100k visitors/day, desired power 80%, alpha 0.05. Show approximate sample size and time calculations, and discuss trade-offs in power, speed, and interpretability.

Sample Answer

Situation/goal: compare throughput and time-to-insight for (A) one 10-armed simultaneous test vs (B) nine sequential baseline-vs-variant A/Bs (with early stopping). Assumptions: baseline p0=5% (0.05), traffic 100k visitors/day, target power 80% (β=0.2), two-sided α=0.05. I'll evaluate detecting an absolute lift Δ=1 percentage point (relative +20% → p1=0.06) which is a realistic product-level MDE.

Step 1 — per-pair sample size (two-proportion z-test, equal allocation):Zα/2=1.96, Zβ=0.84.pooled-ish std ≈ sqrt(2*p*(1-p)) with p≈0.055 → ≈0.3224.Required n per arm ≈ ((1.96*0.3224 + 0.84*0.3224)/Δ)^2 ≈ (0.9027/0.01)^2 ≈ 8,150 visitors per arm.

(A) 10-armed multivariate (equal allocation)- per-arm n ≈ 8,150 → total N ≈ 81,500 visitors.- traffic split 10 ways → ~10,000 visitors/arm/day → time ≈ 0.815 days (~19.6 hours).- BUT multiple comparisons: controlling family-wise error (e.g., Bonferroni α'=0.05/9≈0.00556 for 9 comparisons) raises Zα/2→≈2.79, increasing per-arm n: n_arm ≈ ((2.79*0.3224 + 0.84*0.3224)/0.01)^2 ≈ 13,700 → total ≈ 137k → time ≈ 1.37 days.

(B) 9 sequential A/B tests (baseline vs one variant)- Each A/B requires ≈8,150 per arm → ~16,300 total per test.- With a 50/50 split of site traffic to the test, per-arm daily visitors = 50k → time per test ≈ 0.163 days (~3.9 hours).- Running 9 sequentially (no overlap) → 9 * 0.163 ≈ 1.47 days.- With early stopping (group sequential or sequential probability ratio tests), expected sample size often reduces substantially if effects are large — typical average savings 20–50% depending on true effect and stopping rules → expected total time could drop to ~0.7–1.2 days.- Sequential A/Bs can reuse the same baseline population over time (practical caveats: temporal drift, learning effects).

Trade-offs — power, speed, interpretability:- Speed/throughput: Parallel multivariate gives simultaneous results and fastest time-to-insight per variant when no heavy multiple-comparison correction is applied. After strict FWER correction, multivariate can require more total samples and be slower than naive sequential runs.- Power: For a fixed overall alpha, simultaneous multivariate with proper FWER control reduces per-variant power (requires larger N). Sequential A/Bs allow allocating full alpha per test (or use alpha recycling strategies) and—together with early stopping—can achieve similar or higher expected power with fewer expected samples for strong effects.- Interpretability: Sequential A/Bs are simpler to interpret and attribute cause to a single change. Multivariate tests are efficient at evaluating many variants at once and detecting interactions between independent factors, but analysis must account for multiple comparisons and may obscure which specific feature drove change if variants bundle multiple differences.- Practical risks: Sequential tests risk temporal confounders (seasonality, product changes). Multivariate reduces calendar-time bias but increases statistical complexity and potential sample-size cost when controlling FWER.

Recommendation (PM perspective):- If time is critical and you can tolerate per-comparison α control via FDR (not strict FWER), run multivariate to get simultaneous insight.- If interpretability and conservative Type I control matter (and you can run tests over ~1–2 days), run sequential A/Bs with prespecified stopping rules — this is often preferable for product decisions and faster in practice when early stopping will trigger for strong signals.

HardTechnical

75 practiced

Formally define Expected Value of Information (EVI) in the context of product experimentation and demonstrate with a simple numeric example how to use EVI to prioritize between two experiments that differ in cost, reach, and uncertainty in effect size.

Sample Answer

Expected Value of Information (EVI) (in product experimentation) = the expected increase in decision utility you get by running an experiment (i.e., learning) compared with deciding now based on current beliefs. Formally, if U(d, θ) is utility of decision d when true parameter is θ and D0 is the set of decisions available now, EVI = Eθ[ max_{d∈D_after} U(d,θ) | data ] − max_{d∈D0} Eθ[ U(d,θ) ]. For an A/B test where decision is “roll out” vs “don’t roll out”, EVI is the expected uplift in profit from making that decision after observing experiment results.

Example (numeric, simplified):Assumptions:- Value per incremental conversion = $10.- Baseline conversions irrelevant; consider incremental conversions only.Experiment A:- Cost = $10k, reach (population you could roll out to) = 10,000 users- Prior on true lift (absolute conversion rate) ~ Normal(mean=+0.02, sd=0.03)Experiment B:- Cost = $50k, reach = 100,000 users- Prior ~ Normal(mean=+0.01, sd=0.01)

Step 1 — expected benefit if you knew true lift ℓ and rolled out = ℓ * reach * $10.Step 2 — without running experiments, you’d roll out only if E[ℓ]*reach*$10 > 0.- A: E[ℓ]=0.02 → expected benefit = 0.02*10,000*10 = $2,000 → less than cost $10k → don’t roll out now.- B: E[ℓ]=0.01 → expected benefit = 0.01*100,000*10 = $10,000 → equals cost $50k → don’t roll out.

Step 3 — compute EVI (approx) = probability(ℓ>0) * E[ℓ | ℓ>0] * reach * $10 − 0 (since decision now is “don’t roll out”).Use Normal tail formula:- A: z = (0−0.02)/0.03 = −0.667 → P(ℓ>0)=0.747. E[ℓ|ℓ>0] for normal = μ + σ * φ(α)/(1−Φ(α)) where α = (0−μ)/σ = −0.667. Numerically E[ℓ|ℓ>0] ≈ 0.02 + 0.03*(φ(−0.667)/0.747) ≈ 0.02 + 0.03*(0.318/0.747) ≈ 0.0338. So EVI_A ≈ 0.747*0.0338*10,000*10 ≈ $25,250. Net EVI_A = EVI − cost = $25,250 − $10,000 = $15,250.- B: α = (0−0.01)/0.01 = −1 → P=0.841. E[ℓ|ℓ>0] ≈ 0.01 + 0.01*(φ(−1)/0.841) ≈ 0.01 + 0.01*(0.242/0.841) ≈ 0.0129. EVI_B ≈ 0.841*0.0129*100,000*10 ≈ $108,489. Net EVI_B = $108,489 − $50,000 = $58,489.

Decision: Compare net EVI per dollar or absolute net EVI. Both net EVIs are positive, but Experiment B yields much larger Net EVI despite higher cost because of larger reach and still substantial probability of positive lift. So prioritize B.

Key reasoning and trade-offs:- EVI combines uncertainty (sd), expected effect (mean), reach (scale of impact), and cost.- Larger reach magnifies value of learning; lower uncertainty reduces EVI (less to learn).- Use Net EVI (EVI − cost) or ROI = EVI / cost to prioritize when budget constrained.- For rigorous prioritization, integrate experiment power/precision, multiple decision alternatives, and opportunity cost (time-to-run).

MediumTechnical

63 practiced

Design a 2x2 factorial experiment to test a 10% discount (yes/no) and a new product-page layout (A/B) for an e-commerce funnel. Define primary and secondary metrics, explain how you'd check for interaction effects, outline sample-size considerations (power for main effects vs interaction), and write a brief pre-registration plan.

Sample Answer

Design overview:- 2x2 factorial with factors: Discount (No vs 10%) and Layout (A current vs B new). Randomize users at session entry into one of four cells: (No,A), (No,B), (10%,A), (10%,B). Run simultaneously to avoid time confounds.

Primary & secondary metrics:- Primary: Purchase conversion rate (users who complete checkout / users exposed). This aligns with revenue objective.- Secondary: Average order value (AOV), revenue per user (RPU), add-to-cart rate, bounce rate, and time-to-purchase. Monitor QA metrics: click-through on promo, page load time, and checkout errors.

Interaction effects:- Fit a generalized linear model (logistic for conversion) with main effects for Discount and Layout and their interaction term (Discount*Layout). Significant interaction coefficient indicates the effect of discount depends on layout (or vice versa). Visualize with cell-level conversion rates and interaction plots.

Sample-size considerations:- Power main effects: power calculations for two-way ANOVA/logistic model. For conversion baseline p0 and minimum detectable effect (MDE) for main effects (e.g., 1.5–2 ppt), compute N per cell to achieve 80–90% power. Interaction MDE is typically larger; detecting an interaction requires ~4× the sample of main-effect detection depending on variance. Practically, plan sample size to power main effects first; if interaction detection is critical, increase total sample accordingly or run longer. Use simulations (bootstrap or logistic sims) with baseline conversion and anticipated effects to get precise Ns. Adjust for multiple testing (Bonferroni or hierarchical testing) if evaluating many secondary metrics.

Pre-registration plan (brief):- Hypotheses: 1) H1: 10% discount increases conversion (main effect). 2) H2: Layout B increases conversion (main effect). 3) H3: There is an interaction between Discount and Layout.- Primary metric: conversion rate. Analysis model: logistic regression with factors and interaction; report odds ratios and absolute lift with 95% CIs.- Sample size and stopping rule: fixed sample N_per_cell = X (specify based on simulation) and no peeking; if early stopping allowed, use pre-specified alpha-spending (e.g., O’Brien–Fleming).- Data exclusions: exclude bots, internal traffic, and users with prior exposure in last 7 days; define minimum exposure (e.g., viewed product page).- Multiple comparisons: prioritize primary hypotheses; secondary metrics exploratory with FDR control.- Duration: run for full sales cycle (min 2 weeks and one full business cycle) to cover weekday/weekend effects.- Logging & reproducibility: commit analysis script, randomization seed, and dashboard queries to repo before unblinding.

Operational notes:- Ensure consistent promo messaging upstream (e.g., marketing) and technical implementation isolates discount calculation to cell assignment.- Monitor for implementation bugs and early harmful effects; have kill-switch if checkout errors or revenue drops exceed threshold.

MediumTechnical

60 practiced

Explain adaptive sequential designs such as multi-armed bandits (e.g., Thompson sampling) compared to fixed-horizon A/B tests. For each approach list strengths and weaknesses and provide concrete product scenarios where one approach is preferable to the other.

Sample Answer

Adaptive sequential designs (like multi-armed bandits / Thompson Sampling) and fixed-horizon A/B tests both evaluate product changes but solve different business problems.

Summary:- Fixed-horizon A/B test: predefine sample size and duration, randomize 50/50, analyze at end with hypothesis testing and statistical guarantees (p-values, CIs).- Adaptive bandits (Thompson Sampling): continuously update allocation probabilities based on observed performance, favoring better variants while still exploring.

Strengths / Weaknesses:

1) Fixed-horizon A/B- Strengths: Clear inferential guarantees, simple to implement, easy to communicate to stakeholders, unbiased effect estimates if pre-registered, good for regulatory or scientific decisions.- Weaknesses: Wastes exposure to poor performers, slower to act, vulnerable to peeking without corrections, less efficient when many variants or non-stationarity.

2) Bandits (Thompson Sampling)- Strengths: Maximizes short-term reward (better user experience, revenue), efficient with many variants, adapts to changes, reduces regret.- Weaknesses: Biased estimates of effect sizes unless adjusted, harder to provide classical p-values/confidence intervals, more operational complexity and monitoring, can under-explore if priors are wrong.

Product scenarios:

Prefer fixed-horizon A/B when:- You need an unbiased estimate for decision-making (pricing, compliance, major UX changes).- Stakeholders require a clear p-value/CI or results will be used in reporting/regulatory contexts.- Effect sizes are small and precision matters.

Prefer bandits when:- Objective is to maximize user engagement or revenue during the experiment (homepage layout, recommendation systems, ad allocation).- Rapid iteration with many variants (personalization, growth experiments) or when traffic is limited and you want to reduce regret.- Environment is non-stationary and you want the system to adapt (seasonal offers, live tuning).

Practical hybrid approach:- Use bandits in production to allocate traffic, but run a parallel, smaller fixed experiment or use off-policy evaluation / re-weighting to obtain unbiased estimates for reporting and long-term decisions.

As a PM, align choice to business metric priority: maximize cumulative value (bandit) vs. estimate causal effect for strategic decisions (fixed A/B).

Unlock Full Question Bank

Get access to hundreds of Experimentation Strategy and Advanced Designs interview questions and detailed answers.

Join thousands of developers preparing for their dream job.