Experiment Design and Practical Considerations Questions

Defining metrics to measure (primary and secondary). Estimating sample size and duration needed. Choosing between between-subjects and within-subjects designs. Considering confounding variables and how to control for them. Planning for randomization strategy. Discussing trade-offs between statistical rigor and practical constraints.

MediumSystem Design

65 practiced

Design a scalable metrics pipeline to compute daily experiment metrics for 50M users with near-real-time monitoring: include data ingestion, mapping exposures to treatment assignment, event joins, aggregation, serving to dashboards, and alerting. Note steps to ensure idempotency and correctness across retries and late-arriving data.

Sample Answer

Requirements:- Functional: compute daily experiment metrics for 50M users, map exposures → treatment assignments, join user events to exposures, provide near-real-time (seconds–minutes) dashboards and alerts.- Non-functional: low-latency streaming, correctness with retries & late data, idempotency, scalable to 50M daily users.

High-level architecture:Client events & exposure logs → Ingest (Kafka or Kinesis) → Stream processing (Flink/Beam/KSQ) for mapping and joins → Append-only event store (BigQuery / Snowflake / Delta Lake) + OLAP aggregate store (ClickHouse / Druid) → Serving/API for dashboards → Alerting & monitoring.

Components & responsibilities:1. Ingestion: Kafka topics partitioned by user_id hash; producers include schema/version; use producer idempotence and exactly-once writes where supported.2. Stream processing: - Use Apache Flink (or Beam Dataflow) for event-time processing and stateful joins. - First stage: validate, enrich, protobuf/avro schema registry. - Map exposures to treatment assignment by joining exposure stream against assignment store (also streamed/CDC). - Use keyed state by user_id, with event-time windows and watermarking to handle late events. - Emit deduplicated, canonical "user-event-with-assignment" records.3. Storage: - Append-only raw stream -> partitioned Parquet in object store (S3/GCS) for audit and reprocessing. - Near-real-time aggregates: incremental rollups into OLAP store (per-experiment, per-variant, per-day) using upserts or materialized view support. - Batch reprocessing job daily to recompute from raw for correctness.4. Serving & Dashboards: - Precomputed aggregates served via API to dashboards (Grafana/Looker). For ad-hoc queries, query data warehouse.5. Alerting & Monitoring: - SLAs: freshness, event counts vs expectations, join match-rate, backlog lag. - Alert on anomalies using statistical tests on metrics (z-score, EWMA) and pipeline health metrics (consumer lag, failed tasks).

Idempotency & correctness:- Produce canonical event IDs (user_id + event_type + client_ts + device_seq) and use deduplication in stream processor (Flink state TTL + Bloom filters for memory).- Use exactly-once semantics in stream processor (Flink checkpoints + Kafka transactions) to avoid duplicates on retries.- For storage/upserts, use idempotent upsert keys (user_id + event_id or experiment_id + date + variant) and monotonic update logic (e.g., last-writer-wins with event-time).- Handle late-arriving data: use event-time windows with sufficient allowed lateness (e.g., 24–72h) and trigger incremental re-aggregation when late data arrives; maintain “finalized” flag after lateness window expires.- Support backfill and full recompute from raw Parquet for correctness; compare streaming vs batch aggregates as a reconciliation job and alert on diffs.

Scalability:- Partitioning by user_id distributes load, scale out Flink/Kafka consumers.- Use compacted Kafka topics for assignment lookups.- Aggregate sharding by (experiment_id, date) keys in OLAP store.

Trade-offs:- Strong consistency vs latency: longer allowed-lateness improves correctness but delays final metrics.- Storage cost vs recompute speed: keep raw data to enable recompute; maintain daily materialized aggregates for fast dashboards.

Summary: combine event-time stream processing with durable raw storage and daily batch reconciliation. Ensure idempotent keys, exactly-once processing, watermarking for late data, and automated reconciliation/alerts to guarantee correctness at scale.

MediumTechnical

76 practiced

Compare Bayesian A/B testing to frequentist A/B testing in terms of interpretability, handling of optional stopping, computational cost, and requirements for priors. Give examples of production scenarios where a Bayesian approach is preferable and where a frequentist approach may be better.

Sample Answer

Interpretability:- Bayesian: gives probability statements about parameters directly (e.g., “P(treatment > control) = 0.92”), which is intuitive for product stakeholders.- Frequentist: outputs p-values and confidence intervals that are often misinterpreted (p-value is probability of data under null, CI is long-run coverage), so you must translate carefully.

Optional stopping / sequential monitoring:- Bayesian: naturally supports optional stopping if the model and priors are specified correctly — posterior updates remain valid as data accrue; decisions can be made continuously using posterior odds or decision thresholds.- Frequentist: classical p-values/CI lose nominal Type I error under arbitrary peeking. You need alpha-spending, sequential test corrections (e.g., O’Brien–Fleming, Bonferroni, or group sequential methods) to control false positive rate.

Computational cost:- Bayesian: can be heavier (MCMC) for complex models, but modern variational inference, analytic conjugate updates, or simple beta-binomial/posterior sampling are cheap and practical in many A/B contexts.- Frequentist: typically lighter — closed-form tests (t-test, z-test) or bootstrap; scales well for straightforward metrics.

Requirements for priors:- Bayesian: requires priors. Use weakly informative or hierarchical priors in production to stabilize estimates (especially low-traffic experiments) and pool information across variants or segments. Priors introduce subjectivity but make assumptions explicit.- Frequentist: no priors required; often preferred when stakeholders demand "prior-free" inference or regulatory neutrality.

Production scenarios:- Prefer Bayesian: - Continuous experimentation platform with rolling launches and early stopping; product teams want direct probability statements and adaptive allocation (multi-armed bandits). - Low-traffic features where shrinkage from priors reduces variance and decreases risk of misleading extremes.- Prefer Frequentist: - Regulatory or legal settings requiring pre-specified hypothesis tests, fixed-sample designs, and well-understood Type I error guarantees. - Simple, high-throughput experiments where computational simplicity and standardized reporting (p-values) are expected.

Summary: Bayesian methods offer interpretability and natural sequential use at the cost of choosing priors and potentially higher compute; frequentist methods give clear Type I error control in fixed-sample frameworks with lower computational burden but require care for optional stopping and stakeholder communication.

MediumTechnical

81 practiced

Discuss practical ways to control for confounding variables in experiments: design-time controls (e.g., randomized blocking), analysis-time controls (e.g., regression adjustment), and quasi-experimental methods (e.g., propensity scores). For each, state assumptions, pros/cons, and when you would prefer one approach over another.

Sample Answer

Definition: A confounder is a variable related to both treatment and outcome that biases causal estimates. Three practical control strategies:

1) Design-time controls (randomization, blocking/stratification)- Assumptions: Random assignment (or blocking captures major covariate structure); no hidden imbalance after randomization.- Pros: Gold standard — balances both observed and unobserved confounders on expectation; simple analysis; transparent.- Cons: Sometimes infeasible or unethical; blocking only controls known covariates and can reduce degrees of freedom if many blocks.- When to prefer: RCTs or field experiments where you can assign treatment. Use blocking when a strong prognostic covariate exists to increase precision.

2) Analysis-time controls (covariate adjustment via regression, ANCOVA)- Assumptions: No unmeasured confounding after conditioning on included covariates (conditional ignorability); correct model specification (functional form, interactions).- Pros: Flexible, can adjust for many covariates; improves precision; works when randomization imperfect or observational.- Cons: Sensitive to model misspecification; can't fix unmeasured confounders; extrapolation risks when overlap is poor.- When to prefer: Randomized trials with chance imbalance, or observational studies where you believe measured covariates capture confounding.

3) Quasi-experimental methods (propensity score matching/weighting, IV, difference-in-differences)- Assumptions: For propensity methods — strong ignorability: all confounders observed and PS model correctly specified; for IV — instrument relevance and exclusion restriction; for DiD — parallel trends.- Pros: Explicitly targets balance (PS) and can improve causal claim credibility in nonrandom settings; IV/DiD handle certain unobserved confounders under their assumptions.- Cons: PS doesn't solve unmeasured confounding; matching can discard data; IV estimates Local Average Treatment Effect and hinges on hard-to-verify assumptions.- When to prefer: Observational data where randomization impossible. Use PS to balance observable covariates; IV/DiD if plausible instruments or natural experiments exist.

Practical guidance: Prefer design-time controls where possible. If observational, combine approaches (e.g., PS matching + regression adjustment, sensitivity analyses). Always check overlap/balance diagnostics, conduct robustness checks, and report assumptions and limitations.

EasyTechnical

79 practiced

Compare between-subjects and within-subjects (paired/crossover) experimental designs for measuring a change to a recommendation algorithm in a consumer app. List advantages, disadvantages, and scenarios where each design is preferable, including carryover, learning effects, and washout considerations.

Sample Answer

Between-subjects vs within-subjects for testing recommendation algorithm changes

Definitions- Between-subjects (A/B test): randomly assign users to control or treatment; each user sees only one variant.- Within-subjects (paired/crossover): same users experience both variants in different periods (or interleaved), so each user acts as their own control.

Between-subjects- Advantages: - Simple to implement and analyze; no period/order effects. - No carryover or learning between conditions. - Good for long-running treatments and hard-to-revert user states.- Disadvantages: - Requires larger sample to detect the same effect (higher variance due to between-user heterogeneity). - Potential imbalance if randomization or segment drift fails.- Prefer when: - Treatment permanently changes user state (account-level changes), or when washout impossible. - High risk of cross-contamination (social features). - Long-term retention/monetization metrics that accumulate.

Within-subjects (paired/crossover)- Advantages: - Greater statistical power: controls for user-level heterogeneity, smaller sample needed. - Efficient for detecting small, immediate changes in engagement.- Disadvantages: - Vulnerable to carryover and learning effects: exposure to one variant can affect response to the next. - Requires careful scheduling, washout, or counterbalancing; analysis must model period/order effects.- Prefer when: - Effects are short-lived and reversible (e.g., UI ranking tweaks, short session metrics). - You can randomize order and include washout periods or use interleaving (e.g., alternating sessions) to mitigate carryover.

Carryover, learning, and washout considerations- Carryover: residual influence of earlier variant biases later measurements (e.g., user learns new pattern). Use washout periods or ensure reversible treatments.- Learning effects: users may adapt to a new algorithm, changing baseline behavior—can confound treatment effects. Model time trends or exclude initial adaptation window.- Washout length: choose based on how long treatment effects persist (empirical testing or domain knowledge). If washout impractical, prefer between-subjects.- Analysis: for crossover, include period and order terms; for interleaved designs consider mixed-effects models or paired tests.

Summary guidance- Use between-subjects for irreversible or long-term outcomes and when contamination risk is high.- Use within-subjects for reversible, short-term outcomes to gain power, but only if you can manage carryover, learning, and provide adequate washout or counterbalancing.

HardTechnical

68 practiced

Design an experiment to measure the long-term retention impact of a new recommendation algorithm where the treatment effect may change over months. Specify primary and secondary metrics, a plan for experiment duration and sample size, and how you will handle churn, staggered entry, and delayed outcomes.

Sample Answer

Goal: estimate the long-run causal effect of a new recommender on user retention when treatment effects can evolve monthly.

1) Clarify objective & hypotheses- Primary: Does the new recommender increase 3- and 6-month retention rate (probability user is active in month 3 and month 6) vs control?- Secondary: cumulative active days (months 0–6), 1-yr LTV, churn hazard, engagement quality (watch/completion rate).

2) Metrics- Primary: binary month-3 and month-6 retention (intention-to-treat).- Secondary: time-to-churn (survival), monthly active days (count), cumulative revenue, engagement per session.- Diagnostic: exposure dose, recommendation CTR, item diversity.

3) Experiment duration & checkpoints- Minimum run: 6 months with pre-specified look at month 3 and final at month 6. Prefer 12 months if business allows to capture slow accrual.- Sequential testing: use group-sequential or alpha-spending (O’Brien–Fleming) to allow interim checks without inflating Type I error.

4) Sample size & power (outline)- For binary retention at month T: use standard two-proportion test with baseline p0 and minimum detectable absolute uplift Δ.- Adjust sample size upward for: - expected attrition: inflate by 1/(1 - expected churn by T) - intra-user correlation across months if using repeated-measures models: effective sample size reduction by design effect.- If planning mixed-effects or survival analysis, simulate data with plausible time-varying treatment trajectories and compute power empirically — recommended for complex dynamics.

5) Analysis plan for time-varying effects- Primary analysis: ITT comparison of month-3 and month-6 retention (two-sided tests).- Model time-varying effect with mixed-effects logistic regression or generalized additive models: retention_it ~ treatment_i * f(month_t) + covariates + random_user This estimates monthly treatment effect curve.- Complement with difference-in-differences across cohorts and nonparametric CATE over time.

6) Handling churn, staggered entry, delayed outcomes- Churn/censoring: treat churn as event in survival analysis (Cox or parametric survival) — censor users at last observed date; report hazard ratios and median survival.- Staggered entry (rolling enrollments): include calendar time fixed effects and model time-since-treatment (relative time) rather than absolute calendar time, or use event-study (stacked DID) approach to estimate dynamic effects.- Delayed outcomes: pre-specify lag windows and measure cumulative effects (e.g., ATE at 1, 3, 6 months). Use landmark analyses and delay-adjusted survival models.- Attrition bias: preserve ITT; for per-protocol or exposure-adjusted estimates use inverse-probability-of-censoring weights or instrumental-variable (random assignment) estimates to correct for noncompliance.

7) Robustness & diagnostics- Check balance over entry cohorts and baseline activity.- Heterogeneity: stratify by baseline engagement, device, geography.- Confidence intervals via bootstrap clustered by user.- Pre-register analysis plan: primary endpoints, sample size, stopping rules, covariates.

8) Practical considerations- Logging: ensure consistent event timestamps, user identifiers, and treatment exposure flags.- Simulation: before launch, simulate plausible user trajectories to validate power and detectability under time-varying effects.

This design yields clear point estimates at pre-specified horizons (3m, 6m), captures evolving treatment dynamics, and robustly handles churn, staggered entry, and delayed effects.

Unlock Full Question Bank

Get access to hundreds of Experiment Design and Practical Considerations interview questions and detailed answers.

Join thousands of developers preparing for their dream job.