Experimentation Strategy and Advanced Designs Questions

When and how to use advanced experimental methods, and how to prioritize experiments to maximize learning and business impact. Candidates should understand factorial and multivariate designs, interaction effects, blocking and stratification, sequential testing and adaptive designs, and the trade-offs between running many factors at once versus sequential A/B tests in terms of speed, power, and interpretability. The topic includes Bayesian and frequentist analysis choices, techniques for detecting heterogeneous treatment effects, and methods to control for multiple comparisons. At the strategy level, candidates should be able to estimate expected impact, effort, confidence, and reach for proposed experiments, apply prioritization frameworks to select experiments, and reason about parallelization limits, resource constraints, tooling, and monitoring. Candidates should also be able to communicate complex experimental results, recommend staged follow-ups, and design experiments to answer higher-order questions about interactions and heterogeneity.

MediumTechnical

67 practiced

Discuss the trade-offs between running many factors simultaneously (factorial/multivariate) versus sequential A/B tests in terms of: speed of learning, statistical power for main effects and interactions, interpretability of results, and operational complexity. Provide guidelines for when to choose each approach.

Sample Answer

Start with definitions: a factorial/multivariate test randomizes participants across all combinations of multiple factors simultaneously; sequential A/B tests change one factor at a time (or test one hypothesis per experiment).

Speed of learning- Factorial: learns about all main effects in one experiment — faster when you need multiple answers in parallel.- Sequential A/B: slower overall if you need many factors, because tests are done one after another; but each individual test can conclude sooner if narrowly scoped.

Statistical power- Main effects: factorials efficiently reuse sample to estimate main effects — higher power per sample for main effects vs running separate A/Bs.- Interactions: factorials are the only practical way to estimate interactions; however detecting interactions requires much larger sample sizes (power drops for interaction terms).- Sequential A/B: maximizes power for the single effect under test; interactions are generally missed unless explicitly tested later.

Interpretability- Factorial: can be more complex to interpret when interactions exist — you must report marginal and conditional effects.- Sequential A/B: simpler story per test; easier stakeholder communication.

Operational complexity- Factorial: higher engineering and experiment-design complexity (randomization matrix, traffic splitting, multiple metrics adjustment, multiple comparisons).- Sequential A/B: simpler implementation and rollout; easier rollback and iterative learnings.

Guidelines- Choose factorial when: you have modest-to-large traffic, want answers for multiple factors quickly, and care about interactions or efficient use of samples.- Choose sequential A/B when: traffic is limited, effects are large and isolated, you want simple interpretation and fast rollout for a single decision, or operational overhead must be minimal.- Hybrid: run a small factorial for high-priority factors to screen effects, then follow up with focused A/Bs to validate and measure effect sizes precisely.

Always pre-specify hypotheses, allocate sample size for interaction detection if needed, and control for multiplicity when reporting.

HardSystem Design

74 practiced

Design an experiment platform that supports 50 concurrent experiments, cross-experiment exclusion, deterministic bucketing, reliable event logging, and auditor-friendly reproducibility. Provide a high-level system diagram and describe key components: assignment service, feature config store, event ingestion, metrics database, and analysis layer. Discuss scaling and data integrity considerations.

Sample Answer

Requirements:- 50 concurrent experiments, cross-experiment exclusion, deterministic bucketing, reliable event logging, auditor-friendly reproducibility (full determinism + immutable audit trail), low-latency assignment.

High-level architecture (text diagram):Client SDKs → Assignment Service ← Feature Config Store (git-backed + DB) │ ├→ Event Collector → Stream (Kafka) → Raw Event Store (S3) → Event Processor → Metrics DB (timeseries/OLAP) │ │ └→ Experiment Metadata + Audit Logs (WAL) ------------------------> Analysis Layer (notebooks, BI, causal engine)

Key components:1. Assignment Service- Stateless microservice with consistent hashing + HMAC-based bucketing using secure seed per experiment.- Inputs: user_id, context, active experiments list.- Enforces cross-experiment exclusion via ordered priority groups and deterministic conflict-resolution rules stored in config.- Returns deterministic assignment + assignment token (experiment_id, seed, bucket, timestamp, checksum).- Cache with TTL (Redis) for hot users; rate-limit + SDK local cache for availability.

2. Feature Config Store- Git-backed repo for configs (experiment specs, traffic allocation, exclusion groups) + validation CI.- Served via fast DB (Postgres) and replicated read-only caches.- All changes produce immutable versioned configs; each config has UUID and checksum for audit.

3. Event Ingestion / Logging- Client sends events with assignment token; server-side collector validates token checksum before accepting.- Events appended to Kafka with partitioning by user_id.- Synchronous durable write (acks) to ensure reliability; local SDK retries/backoff.- Events also write a WAL/audit stream (append-only) and are periodically flushed to S3 (raw immutable archive).

4. Metrics Database- Nearline: real-time aggregate pipeline (Flink/Spark Streaming) consumes Kafka, computes aggregations, writes to OLAP store (ClickHouse or Druid) for fast queries.- Batch: event processor compacts raw S3 into Parquet, loads into data warehouse (BigQuery/Snowflake) for reproducible analyses.

5. Analysis Layer- Notebook + BI connected to metrics DB and raw event store.- Causal engine performs assignment-aware statistical tests (adjusting for multiple experiments and exclusions) using recorded seeds for permutation tests.- Reproducibility: every analysis references experiment config UUIDs, raw event snapshot timestamp, and assignment tokens; automated re-run pipeline uses same seeds to reproduce allocations.

Scaling considerations- Kafka topic partitioning by user_id to scale ingestion; autoscale consumers.- Assignment service auto-scales behind LB; rely on Redis cache and local SDK caching to limit calls.- Metrics DB uses sharding/time-partitioning; pre-aggregations for high-cardinality metrics.- S3 + Parquet for cheap long-term storage and re-processing.

Data integrity & reproducibility- Deterministic bucketing: HMAC(user_id, experiment_seed) → numeric bucket.- All writes are idempotent (use event_id); collector deduplicates via Kafka compacted topics and event-store checks.- Immutable config versions + assignment tokens + raw event archive enable auditors to recompute any assignment and metric.- End-to-end checksums & lineage metadata stored in metadata DB; automated nightly validation jobs compare streaming aggregates to batch results and surface drift.

Trade-offs- Strong reproducibility needs more storage (raw archives) and compute (replay pipelines).- Synchronous event acks add client latency; mitigated via SDK batching and local buffering.

This design balances deterministic assignments, cross-experiment exclusion, reliable logging, and auditor-friendly reproducibility while scaling to many concurrent experiments.

MediumTechnical

66 practiced

Compare Bayesian decision thresholds and frequentist p-value thresholds for stopping experiments when mistakes have asymmetric costs (e.g., false negative cost >> false positive). How would you choose priors and decision rules in a Bayesian framework to reflect asymmetric loss?

Sample Answer

Start from the loss-driven principle: a Bayes decision rule chooses the action that minimizes posterior expected loss. For a binary decision (declare effect vs not), let C_FP be cost of a false positive and C_FN cost of a false negative. If π = P(effect | data) is the posterior probability effect > 0, the expected losses are:- Choose positive: EL_pos = (1 − π) * C_FP- Choose negative: EL_neg = π * C_FNChoose positive when EL_pos < EL_neg ⇔ π > C_FP / (C_FP + C_FN). Define λ = C_FN / C_FP (asymmetry). Threshold on posterior π* = 1 / (1 + λ). So when false negatives are much more costly (λ large), π* is small — you require less posterior evidence to act.

Priors: pick priors to reflect realistic prevalence and effect sizes. If missing true effects is very costly, avoid overly skeptical priors that shrink toward zero; instead:- Use informative priors based on historical data or meta-analytic hierarchical priors to reflect plausible effect sizes.- If prior uncertainty matters, perform sensitivity analysis over a small family (weakly informative, skeptical, enthusiastic) and report how π and decisions change.- Consider decision-oriented priors: choose priors that make posterior probabilities calibrated for decision thresholds via prior predictive checks.

Stopping rules and sequential monitoring: Bayesian procedures allow optional stopping without type I inflation when decisions are based on posterior expected loss (not p-values). At each interim, compute π and compare to π*. You can also compute expected value of additional information (EVPI) to decide whether to continue sampling. In contrast, frequentist p-value thresholds require corrections (alpha spending) and are symmetric in error trade-offs unless you set different α and β targets; mapping an asymmetric loss into a single p-value cutoff is indirect and depends on effect size and sample size.

Practical steps:- Elicit C_FP and C_FN from stakeholders; convert to λ and π*.- Choose a baseline prior (hierarchical or weakly informative) and run prior predictive checks to ensure sensible behavior.- Pre-specify sequential decision rule: stop for action if π > π*, stop for futility if π < π_low (derived similarly), otherwise continue while EVPI > sampling cost.- Report sensitivity of decisions to priors and show operating characteristics (simulated false positive/negative rates) under plausible data-generating scenarios.

This yields transparent, loss-aligned stopping rules that directly reflect asymmetric costs, unlike a single frequentist p-threshold which hides the cost trade-off.

EasyTechnical

74 practiced

What is a pre-analysis plan (pre-registration) for experiments and why is it important in a business context? List the key elements you would include in a pre-analysis plan for a major pricing experiment.

MediumTechnical

52 practiced

You must test five binary product features but can only run an experiment with 8 experimental arms due to traffic constraints. Propose a fractional factorial design that estimates main effects and selected two-way interactions, explain the aliasing structure, and describe how you'd interpret aliased interaction estimates.

Sample Answer

Proposal (design): Use a 2^(5-2) fractional factorial (8 runs), resolution IV. Label features A,B,C,D,E. Choose generators D = A·B and E = A·C. This yields 8 experimental arms given by all combinations of A,B,C with D and E set by those generators.

Design table (conceptually): run all 2^3 combinations of A,B,C; set D=AB, E=AC.

Aliasing / defining relation:- Generators: D·AB = I and E·AC = I → defining word set: I = ABD = ACE.- Multiply generators to obtain other words: (ABD)(ACE) = BCDE = I.From these you can derive alias chains:- Main effects are aliased with three-way interactions: A ↔ BDE, B ↔ ADE, C ↔ ACE? (explicitly: A is aliased with B·D·E, etc.)- Two-way interactions are aliased with other two-way interactions: AB is aliased with D (by definition) and with C·E? More usefully: - AB ↔ D - AC ↔ E - AD ↔ BE - AE ↔ BD - BC ↔ DESo this is a Resolution IV design: main effects are clear of other main effects and two-way interactions, but two-way interactions are aliased with other two-way interactions.

How to estimate and interpret:- Fit a linear model with main effects and the specific two-way interactions you can form from the 8 runs. Estimates for main effects are unbiased under the usual sparsity assumption (higher-order interactions negligible).- Two-way interaction estimates must be interpreted as combined (aliased) effects. Example: the estimate for AD actually represents AD + BE. If AD̂ is significant, it could be AD, BE, or both.Practical interpretation strategy:1. Use domain knowledge to decide which of each aliased pair is plausible (e.g., product knowledge may make BE unlikely → attribute to AD).2. Look at signs and magnitudes: if AD̂ and AÊ patterns align with expectations of A interacting with others, that supports interpreting as AD/AE rather than their aliased partners.3. Run a targeted follow-up (foldover) for ambiguous pairs: a simple foldover design flips the signs of generators (e.g., set D' = -AB) to break aliases and resolve which interaction is real.4. If traffic/cost prevents follow-up, report aliased estimates transparently and prioritize features for follow-ups or A/B validation.

Why this design:- Uses available traffic efficiently (8 arms)- Guarantees estimation of all main effects unconfounded by two-way interactions- Allows measuring selected two-way interaction contrasts, accepting that they are aliased in pairs; solvable via domain judgment or a small confirmatory experiment (foldover) if needed.

Edge considerations:- Pre-specify which two-way interactions matter most and consider alternative generator choices to alias less-important pairs.- Pre-register analysis plan and assume higher-order interactions are negligible.

Unlock Full Question Bank

Get access to hundreds of Experimentation Strategy and Advanced Designs interview questions and detailed answers.

Join thousands of developers preparing for their dream job.