Experiment Design and Execution Questions

Covers end to end design and execution of experiments and A B tests, including identifying high value hypotheses, defining treatment variants and control, ensuring valid randomization, defining primary and guardrail metrics, calculating sample size and statistical power, instrumenting events, running analyses and interpreting results, and deciding on rollout or rollback. Also includes building testing infrastructure, establishing organizational best practices for experimentation, communicating learnings, and discussing both successful and failed tests and their impact on product decisions.

HardTechnical

75 practiced

After an experiment shows a strong conversion uplift but also a large increase in refunds and customer complaints, craft a rigorous decision framework for rollout vs partial rollout vs rollback. Include immediate monitoring, segmentation to isolate issues, staged rollout strategies, and stakeholder communication plan.

Sample Answer

Decision framework for rollout vs partial rollout vs rollback

1) Immediate monitoring (first 72 hours)- Real-time dashboard: conversion rate, refunds rate, complaint volume, NPS/CSAT, refund cost, chargeback rate, support SLA breaches.- Alerting: trigger if refunds increase > X pp (e.g., +2pp or +30% relative) or complaints > baseline +50% OR refund cost exceeds expected LTV uplift * safety factor.- Short-window A/A check to ensure no instrumentation/regression issues.

2) Rapid segmentation to isolate issues (day 0–3)- Slice by: customer cohort (new vs returning), geography, device/OS, traffic source, product type, order value, promo codes, experiment variant exposure.- Run causal checks: uplift vs refund trade-off per segment; compute net revenue per user (NRPU) = revenue - refunds - support cost.- Flag segments with negative NRPU despite conversion uplift.

3) Decision matrix- Rollout if: overall NRPU positive with refunds within tolerance, no critical segments flagged, operational capacity to handle support.- Partial rollout (targeted) if: positive aggregate uplift but specific segments show harm → deploy only to safe cohorts (e.g., high-LTV, low-refund channels) while excluding flagged segments.- Rollback if: aggregate NRPU negative, refunds/complaints exceed thresholds, or compliance/brand risk present.

4) Staged rollout strategy- Stage 0: pause expansion; deploy mitigation (fix messaging, add guardrails) A/B test fixes on affected segments.- Stage 1: Canary to 1–5% of traffic in safe cohorts with close monitoring.- Stage 2: Gradual ramp (5→25→50→100%) conditional on passing thresholds at each step (predefined windows and statistical tests).- Use sequential analysis (alpha-spending) to avoid peeking bias.

5) Remediation & experiments- Run quick experiments to test hypotheses behind refunds (UX confusion, incorrect pricing, fulfillment).- Apply fixes, re-run targeted A/B tests before larger rollout.

6) Stakeholder communication plan- Immediate: notify product, ops, customer support, legal, finance with a one-page summary: metrics, thresholds breached, immediate action (pause/partial), next 24h plan.- Daily updates for first week: dashboard links, top segment findings, actions taken, recommended decision.- Weekly review: impact on revenue, customer health, proposed timeline.- Post-mortem: causal analysis, root cause, lessons, rollback/playbook for future.

7) Governance & risk controls- Predefine KPIs and financial thresholds with PO/FM before rollout.- Implement kill-switch and automated exclusion rules.- Log decisions, owners, and timelines.

This framework balances short-term revenue gains with long-term customer and brand health by making data-driven, segment-aware, and risk-controlled rollout decisions.

EasyTechnical

44 practiced

List essential instrumentation events and minimal event schema required to run reliable product experiments. Include fields such as user identifier, experiment id, variant, timestamp, event type and event properties. Explain idempotency and reconciliation checks you would implement.

Sample Answer

Essential instrumentation events- exposure (user was assigned and saw the variant)- impression / view (specific UI/feature seen)- conversion (goal achieved: purchase, signup)- interaction (clicks, form submit)- engagement (time-spent, scroll)- error/telemetry (client errors, dropped requests)- heartbeat / session_start (health & sessionization)

Minimal event schema (JSON example)

json

{
  "event_id": "uuid-v4",         // unique per event (idempotency key)
  "user_id": "user_1234",        // persistent user id (nullable)
  "anon_id": "anon_abcdef",      // fallback for anonymous users
  "experiment_id": "exp_42",
  "variant": "control|treatmentA",
  "event_type": "exposure|conversion|click",
  "event_time": "2025-05-12T15:04:05Z",
  "session_id": "sess_9876",
  "page": "/pricing",
  "properties": { "price": 9.99, "currency": "USD" },
  "sdk_version": "web-1.2.0",
  "client_ts": 1683900000
}

Why fields matter- event_id enables dedup and exactly-once semantics.- user_id/anon_id allow user-level aggregation and dedup across devices.- experiment_id + variant are required for assignment joins.- event_time and client_ts allow ordering, latency checks, and windowing.- session_id and properties support richer metrics and covariate controls.

Idempotency- Generate event_id client-side (UUID) or server-side; ingest pipeline deduplicates on event_id and source.- Use upserts or a dedupe layer (e.g., streaming dedupe with a bounded retention window) to avoid double counting.- For exposures, use "exposure_confirmed" pattern: record assignment server-side and emit exposure when rendered; dedupe by (user_id, experiment_id, exposure_window).

Reconciliation checks- Traffic split sanity: compare observed exposures per variant to expected allocation; alert if deviations exceed threshold.- Exposure→Conversion ratio checks: sudden drops suggest instrumentation loss.- Unique users count: reconcile unique user_ids/anon_ids across event types; flag conversions without prior exposure if experiment requires exposure-first.- Time-window & late-arrival handling: accept late events up to configured lateness, then backfill and re-run metrics if necessary.- Missing fields & schema validation: reject or route malformed events to a dead-letter queue with sample for debugging.- End-to-end checksum: periodically compare client-side counts (e.g., sampled logs) against pipeline counts.- Sampling validation: ensure sampling flags propagate; validate that sampled subsets are representative.

Implementation best practices- Enforce strict schema via ingestion (e.g., JSON schema + typed events), store raw and processed lanes.- Keep raw event_id and timestamps to enable reprocessing and audit.- Provide an "assignment log" mapping user_id → (experiment_id, variant, assignment_ts) stored atomically so analyses can join on authoritative assignment.- Monitor latency, duplicate rate, missing-field rate, and variant imbalance; automate alerts and a lightweight reconciliation job that recomputes key aggregates daily.

HardTechnical

41 practiced

Your primary metric is time-to-first-purchase. Explain how you would analyze A/B test data using survival analysis: Kaplan-Meier estimates, log-rank test, Cox proportional hazards model, and how to handle censoring and reporting lift in hazard or median time terms.

Sample Answer

First clarify the objective and data:- Metric: time-to-first-purchase measured from exposure/randomization to event (purchase).- Define event indicator (1=purchase observed, 0=censored). Censor at end-of-follow-up or loss-to-follow-up.- Use intent-to-treat population and preserve randomization.

1) Kaplan–Meier (KM)- Estimate survival function S(t)=P(T>t) per arm to visualize probability of not yet purchased over time.- Plot KM curves with confidence bands and number at risk table; useful to inspect early/late separation and median times.- Code (lifelines):

python

from lifelines import KaplanMeierFitter
kmf = KaplanMeierFitter()
kmf.fit(durations=times_control, event_observed=events_control, label='control')
ax = kmf.plot_survival_function()
kmf.fit(times_treatment, events_treatment, label='treatment')
kmf.plot_survival_function(ax=ax)

2) Log-rank test- Nonparametric test comparing whole survival curves (null: no difference in hazard functions).- Use when proportional hazards roughly hold; returns p-value for overall difference.- In Python: lifelines.statistics.logrank_test.

3) Cox proportional hazards model- Fit Cox model to estimate hazard ratio (HR) between treatment and control, possibly adjusting for covariates (age, channel, prior behavior).- Interpret HR: HR<1 means treatment reduces instantaneous hazard of remaining event-free (i.e., increases rate of purchase). Example: HR=0.8 → 20% higher instantaneous purchase rate in treatment (1/0.8 interpretation depends on coding).- Code:

python

from lifelines import CoxPHFitter
df = pd.DataFrame({...})  # columns: duration, event, treatment, covariates
cph = CoxPHFitter()
cph.fit(df, 'duration', event_col='event')
cph.print_summary()

- Check proportional hazards assumption (Schoenfeld residuals). If violated, use time-varying covariates, stratified Cox, or split time windows.

4) Handling censoring- Right-censoring is standard; include censored observations in KM and Cox.- If administrative censoring differs by arm, ensure follow-up windows are equal or adjust via inverse-prob weighting.- For heavy loss-to-follow-up, perform sensitivity analyses (best/worst-case, multiple imputation).

5) Reporting lift and practical metrics- Primary inferential: report HR with 95% CI and p-value (e.g., HR=0.75, 95% CI [0.62–0.90], p=0.002).- Translate to user-friendly metrics: - Percent change in instantaneous purchase rate: (1 − HR)*100% (if HR<1 and treatment coded 1 reduces time). - Median time-to-event: report medians per arm and difference/ratio with CIs from KM (e.g., median_control=14 days, median_treatment=10 days → ~29% reduction). - Absolute probability improvement at specific time points: ΔS(t)=S_control(t)−S_treatment(t) → difference in cumulative purchase probability by day 7/30.- Example phrasing: "Treatment reduced median time-to-first-purchase from 14 to 10 days (median ratio 0.71), HR=0.75 (95% CI 0.62–0.90), meaning a 25% higher instantaneous purchase rate."

6) Robustness and presentation- Show KM plots with number-at-risk, HR and medians in table, log-rank p-value, covariate-adjusted Cox results, PH diagnostics.- Sensitivity: subgroup analyses, truncated follow-up, competing risks if other terminal events exist.- Emphasize practical significance (e.g., revenue lift from earlier purchases) and uncertainty.

This workflow preserves censoring information, leverages both descriptive (KM) and inferential (log-rank, Cox) methods, adjusts for confounders, checks assumptions, and reports both statistical and business-relevant effects.

HardSystem Design

60 practiced

Design an event schema and logging pipeline to ensure reliable telemetry for experiments at scale. Cover event contract/versioning, client vs server logging choices, idempotency, schema registry, streaming pipeline (e.g., Kafka), real-time and batch aggregations, and reconciliation processes between raw logs and aggregated metric store.

Sample Answer

Requirements & constraints:- Reliable, low-latency telemetry for A/B experiments at millions of users/day- Strong schema governance, backward/forward compatibility, replayable raw events- Support client (browser/mobile) + server-side events, idempotency, real-time and batch analytics, and reconciliation

High-level architecture:Client/Server → API Gateway / Proxy → Kafka raw-events topic → Stream processor (Flink/Beam) → Real-time metrics store (Druid/ClickHouse/Materialized Views) + OLAP warehouse (BigQuery/Snowflake) for batch → Monitoring & Reconciliation jobs

Event contract & schema/versioning:- Use a compact, typed format (Avro or Protobuf) registered in a Schema Registry (Confluent/Apicurio).- Schema fields (required): event_id (UUID), event_type, user_id (nullable for anonymity), session_id, experiment_id, variant, timestamp_ms (UTC), ingestion_source, client_sdk_version, platform, payload (typed union/record).- Versioning strategy: - Additive changes allowed (new optional fields). - Deprecation via deprecation metadata in registry; breaking changes require new subject name (event_type.v2). - Enforce compatibility rules in CI to prevent accidental breaking changes.

Client vs Server logging choices:- Server-side for deterministic events (conversion, server-rendered actions) and to protect PII.- Client-side for UX/contextual events (clicks, view durations).- Common pattern: canonicalize on server where possible; client includes client_generated_id and server assigns server_ingest_id to reconcile duplicates.

Idempotency & deduplication:- Every producer attaches event_id (UUID v4 or ULID) plus optional dedupe_key. Server assigns ingestion_id if missing.- At stream ingestion, use Kafka log compaction and stream processor stateful dedupe with retention window (e.g., 24–72 hours) using event_id as key. Persist seen-ids in a compacted store (RocksDB/Redis TTL) to prevent double counting.

Streaming pipeline details:- Kafka topics: - raw.events (partitioned by experiment_id or hash(user_id) for locality) - processed.events (cleaned, enriched) - metrics.events (pre-aggregated deltas)- Producers: client SDKs + backend services with retries & exponential backoff; use idempotent Kafka producers when server-producer.- Stream processing: use Flink/Beam to: - Validate against schema via registry, - Enrich (geo, device, feature flags), - Apply dedupe and idempotency, - Emit event-level and windowed aggregations (per experiment/variant) into a real-time store.

Real-time vs batch aggregation:- Real-time: low-latency KPIs (exposure counts, conversions, CTR) computed with stream processing into OLAP optimized stores (Druid/ClickHouse) for dashboards and quick alarms. Emit incremental, append-only metric deltas to enable replay.- Batch: nightly/full-window re-computation in data warehouse (Spark/BigQuery) from raw.events for robust metrics and complex joins (user lifetime metrics). Store both pre-aggregated and raw tables.

Schema registry and validation:- All producers/consumers use Schema Registry client; pipeline rejects non-conforming messages to dead-letter topic with alerting.- CI pipeline for schema changes: test producers/consumers, run synthetic load tests, and require owner approval.

Reconciliation process:- Daily reconciliation job: 1. Recompute canonical aggregate from raw.events in warehouse (golden metrics). 2. Compare to streaming real-time metrics (sum of deltas) on key dimensions (experiment_id, variant, date, KPI). 3. Threshold-based alerts: percentage difference or absolute delta triggers investigation. 4. If drift detected, run targeted replay: reprocess raw.events by timestamp range through batch job with identical enrichment logic to isolate bugs. 5. Maintain audit tables linking event_id → ingestion_id → aggregated buckets for traceability.

Operational considerations:- Observability: instrument counts at each stage (producer success, Kafka lag, schema rejects, dedupe hits), and expose to PagerDuty.- Backfills & replays: keep raw.events immutable for at least 30–90 days; support topic retention and archival (S3) for long-term re-processing.- Privacy: PII hashed/encrypted before logging; provide opt-out gating in SDKs.- Trade-offs: aggressive real-time dedupe reduces double-counting but increases state size; tune retention vs accuracy.

What a Data Scientist gets:- Reproducible, audited raw events for model features and experiment logging- Low-latency experiment signals plus a trusted nightly canonical metric for final decisions- Traceability from metric back to raw events for debugging and fairness/QA checks

This design balances reliability, traceability, and scale while giving analysts and data scientists both fast feedback and authoritative batch-backed metrics for experiment evaluation.

MediumTechnical

56 practiced

In an experiment where only 70% of users actually receive the treatment due to client-side errors, explain the differences between Intent-To-Treat (ITT) and Per-Protocol analyses. How would you estimate the Complier Average Causal Effect (CACE) and what assumptions are required?

Sample Answer

Intent-To-Treat (ITT) estimates the causal effect of assignment to treatment (Z) regardless of whether users actually received it. ITT = E[Y | Z=1] − E[Y | Z=0]. It’s unbiased for policy effect of offering treatment but diluted when only 70% comply.

Per-Protocol estimates effect only among those who followed the protocol (treated as if randomized by actual treatment receipt T). It typically compares E[Y | T=1] − E[Y | T=0] and is biased because T is endogenous (compliance may correlate with potential outcomes).

Complier Average Causal Effect (CACE)- CACE (aka Local Average Treatment Effect) estimates effect for compliers — those who take treatment if and only if assigned.- Use assignment Z as an instrument for actual treatment T. Wald estimator (binary Z) gives: CACE = (E[Y | Z=1] − E[Y | Z=0]) / (E[T | Z=1] − E[T | Z=0]) With 70% compliance: denominator ≈ 0.7.- Equivalently run two-stage least squares (TSLS): first stage T ~ Z (+ covariates), second stage Y ~ T_hat (+ covariates).

Required assumptions- Random assignment: Z independent of potential outcomes (holds by design).- Relevance: Z affects T (first-stage nonzero; here 0.7).- Exclusion restriction: Z affects Y only through T (no direct effect of assignment).- Monotonicity: no defiers (assignment never reduces probability of receiving treatment for anyone).- SUTVA: no interference between units.

Practical notes- Report ITT and CACE with CIs (use robust/clustered SEs).- Check first-stage strength, test sensitivity to exclusion restriction, and consider bounds if assumptions are questionable.

Example (Wald):

python

# compute CACE from grouped means
cace = (YZ1.mean() - YZ0.mean()) / (TZ1.mean() - TZ0.mean())

Unlock Full Question Bank

Get access to hundreds of Experiment Design and Execution interview questions and detailed answers.

Join thousands of developers preparing for their dream job.