Real World Experimental Challenges and Solutions Questions

Discuss practical complications in running experiments at scale: user heterogeneity, segment-specific effects, long-term vs. short-term metrics, novelty effects, network effects, and infrastructure constraints. Know techniques for variance reduction (CUPED), segmentation strategies, and how to detect and correct for data quality issues during experiments.

HardTechnical

61 practiced

Explain the statistical estimand and estimation strategy for measuring spillover effects using a partial population design: randomly assign p% of clusters to treatment and leave the rest as pure control. How do you estimate direct and indirect effects and which estimators are unbiased under partial interference?

Sample Answer

Clarify the design and estimands- Design (partial-population / saturation): Clusters are randomized into two arms: "saturated" clusters (a proportion s of units inside each such cluster are randomly treated) and "pure control" clusters (no treated units). This is the standard partial-population design; interference is allowed only within clusters (partial interference).- Potential outcomes notation: Y_i(z, g) is unit i's outcome if i’s own treatment = z ∈ {0,1} and the cluster-level treatment exposure (e.g., fraction treated) = g. Typical choices: g=0 (no treated peers) and g=s (saturated fraction).- Estimands: - Average Direct Effect (ADE) at saturation s: ADE(s) = E[Y_i(1, s) − Y_i(0, s)] — the effect of switching an individual's own treatment while holding cluster saturation at s. - Average Indirect (Spillover) Effect (AIE) for untreated units: AIE(s) = E[Y_i(0, s) − Y_i(0, 0)] — the effect on an untreated individual's outcome when moving from no treated peers to saturated peers.

Estimation strategy under partial interference- Randomization structure gives identification: clusters randomized to saturation vs control; within saturated clusters individuals are randomly assigned to treatment with known probability s. Under this two-stage randomization and partial interference, the following unbiased estimators are standard: 1. Direct effect (ADE): difference-in-means between treated and control units, but computed only among units that are in saturated clusters so the peer exposure is g=s for both groups: ADE_hat = mean_{i ∈ saturated, Z_i=1} Y_i − mean_{i ∈ saturated, Z_i=0} Y_i. Because individual treatment was randomized within saturated clusters, this is unbiased for ADE(s). 2. Indirect effect (AIE): difference-in-means between untreated units in saturated clusters and units in pure-control clusters: AIE_hat = mean_{i ∈ saturated, Z_i=0} Y_i − mean_{i ∈ control} Y_i. Since cluster assignment to saturation vs control was randomized, and untreated units in saturated clusters are randomized to be untreated with known probability, this estimator is unbiased for AIE(s).

Which estimators are unbiased under partial interference- Under (i) randomization of clusters to saturation vs control, (ii) random assignment of individuals within saturated clusters with known s, and (iii) no interference across clusters (partial interference), the simple difference-in-means estimators above are unbiased for the corresponding potential-outcome contrasts (ADE(s), AIE(s)).- More generally, Horvitz–Thompson (IPW) estimators or Hájek-normalized IPW estimators that weight observations by their assignment probabilities produce unbiased (HT) or consistent (Hájek) estimates and naturally handle unequal cluster sizes or unequal assignment probabilities.

Practical considerations and variance- Use cluster-robust variance estimators (or randomization-based variance formulas) because outcomes within clusters are correlated.- If cluster sizes vary, weighting matters: estimate population-average effects (weight by unit) vs cluster-average effects (weight by cluster).- If you want direct effect averaged over the mixture of cluster types, you can combine estimators appropriately (e.g., marginalize ADE(s) over distribution of cluster-level covariates).- Threats to identification: interference across clusters, noncompliance, differential attrition — each requires adjustment (e.g., IV for noncompliance, sensitivity analyses).

Summary- Define ADE(s)=E[Y(1,s)-Y(0,s)] and AIE(s)=E[Y(0,s)-Y(0,0)].- Estimate ADE by treated vs untreated within saturated clusters; estimate AIE by untreated in saturated vs units in pure-control clusters.- Under randomization and partial interference, difference-in-means (or HT/IPW) are unbiased; use cluster-robust variance estimation and adjust weighting according to the estimand (unit- vs cluster-average).

HardSystem Design

66 practiced

You run many overlapping experiments on the same user population. Propose an approach to design overlapping experiments that allows attribution of effects to individual features, including use of factorial designs, shared-control groups, and orthogonalization. Discuss limitations and when to avoid overlapping tests.

Sample Answer

Requirements and constraints:- Estimate causal effect of individual features while running many concurrent experiments on the same user population.- Maintain statistical power, control Type I error, and allow estimation of interactions.- Minimize user interference and operational complexity.

High-level approach:1. Use factorial design where feasible: - Encode K features as factors (binary or multi-level) and randomize users into full or fractional factorial cells. Full factorial estimates main effects and interactions; fractional reduces cells when K large but trades off identifiability of higher-order interactions.2. Shared-control groups: - For features that are rare or expensive to change, create a common pooled control cohort; treatment arms for multiple features draw from the same control to increase power. Carefully ensure control experience is identical.3. Orthogonalization and blocking: - Randomize so assignments to different experiments are orthogonal (independent) — e.g., use hash(user_id, experiment_salt) to assign across features. If full orthogonality impossible, orthogonalize statistically: regress outcome on other concurrent treatments and use residuals or include other treatments as covariates.4. Analysis plan: - Fit a linear (or generalized) model with terms for each feature and interaction terms you expect: outcome ~ intercept + Σ βi Feature_i + Σ γij Feature_i*Feature_j + covariates - Use hierarchical shrinkage (Bayesian or regularized) to stabilize estimates when many interactions exist. - Pre-specify primary contrasts and correct for multiple comparisons (BH or family-wise when needed).5. Power & sample sizing: - Compute sample size accounting for variance inflation from shared-control and interactions. Fractional factorials reduce required sample but lose some interaction estimates.6. Practical controls: - Use blocking/stratified randomization on major confounders to reduce variance. - Monitor for interference (SUTVA violations) — if users influence each other, treat via cluster randomization.

Limitations & when to avoid overlapping tests:- Avoid overlapping tests when treatments interact strongly and unpredictably and you cannot allocate sufficient sample to estimate interactions reliably.- If SUTVA is violated (peer effects), overlaps can bias estimates; prefer cluster or separate sequential experiments.- Operational complexity: many-factor factorials increase engineering and analytics burden; if product constraints prevent orthogonal assignment, run sequential A/B tests or prioritize features.- Multiple comparisons and model misspecification risk: if many correlated treatments and limited data, prefer focused single-factor experiments.

Trade-offs:- Factorial + shared-control increases throughput and power for main effects but requires careful pre-specification of interactions and stronger analysis (regularization, multiple-testing correction). Fractional designs save sample but sacrifice some interaction identifiability.

Summary checklist to implement:- Pre-specify model and interactions to estimate.- Choose full vs fractional factorial based on K and sample budget.- Implement orthogonal randomization; use shared-control where safe.- Use regression with interaction terms + regularization; correct for multiplicity.- Monitor interference and stop/segregate tests when SUTVA violated or power insufficient.

EasyTechnical

48 practiced

Describe CUPED (Controlled Experiments Using Pre-Experiment Data). Explain the intuition, when it helps in practice, and a simple numeric example: given pre-period user spend per user and experiment period spend per user, how does CUPED reduce variance? Mention limitations and assumptions.

MediumTechnical

42 practiced

Design a ramp and rollback plan to safely roll a new recommendation algorithm from 0.1% to 1% to 10% to 100% of traffic. Describe monitoring metrics (both core and guardrail), stopping criteria at each stage, and how to handle surprising negative signals during a ramp.

Sample Answer

Approach: treat the rollout as staged canary experiments with automated guardrails, pre-defined stopping criteria, and clear rollback actions at each stage. Use both product/core metrics and guardrail/health metrics, plus statistical significance checks.

Ramp plan (timing + steps)- 0.1% (smoke, 24–48h)- 1% (short canary, 48–72h)- 10% (extended canary, 3–7 days)- 100% (full launch, 7–14 days monitoring after)

Core metrics (business/product)- CTR / engagement with recommendations (relative lift, %)- Conversion / downstream actions (e.g., add-to-cart, watch-complete)- Revenue per user or recommendation-driven revenue- Average session length from recommended items

Guardrail (safety/health)- Total page latency / 95th percentile latency (+ acceptable delta e.g., <5%)- Error rate (500s, client errors) absolute threshold (e.g., >0.1% increase)- CPU/memory/backpressure on recommender services- Session abandonment / bounce rate- Quality metrics: relevance score or manual sampling rate of poor recommendations

Stopping criteria (per stage)- Stop immediately and rollback if any critical guardrail breaches: - Error rate increase > 50% relative or absolute >0.1% sustained >15 minutes - Latency 95th > baseline +20% sustained - Infrastructure alerts (CPU, OOM) triggered- Pause and investigate if core metrics degrade beyond tolerated loss: - CTR or conversion drop > 5% relative sustained for 2–4x evaluation windows OR statistically significant negative effect (p<0.05) with pre-allocated power- For 0.1%: require no critical guardrail breaches and no obvious negative signal in manual review- For 1%: require stable infra, no regression in core metrics beyond tolerance, and initial uplift/noise within expected CI- For 10%: require sustained non-negative core metrics over rolling 3–7 day window and automated A/B significance tests- To go to 100%: require positive or neutral lift with business-significant CI and no regressions in guardrails

Handling surprising negative signals- Immediate actions: trigger automated kill-switch routing new requests back to control; notify on-call and data science + infra + PM- Triage steps: 1. Confirm signal (check instrumentation, look for deployment anomalies) 2. Run quick stratified analysis (by region, cohort, device, user segment) to localize cause 3. Rollback to previous stable version if signal persists or is high-impact 4. If localized, roll back only affected shards/segments and continue ramp elsewhere 5. Create postmortem with root cause, corrective tests, and improved guardrails- Communicate: rapid incident update to stakeholders with ETA and mitigation

Additional practices- Use feature flag service supporting percentage rollouts and instant kills- Pre-register hypotheses and minimum detectable effect; use sequential testing to avoid peeking bias- Maintain automated dashboards with real-time alerts, and periodic manual quality sampling- Re-run offline replay tests and simulate traffic if feasible before re-attempting ramp

This plan balances speed and safety: small fast smoke tests, progressively longer validation windows, precise stopping rules, and rapid rollback + targeted triage for surprises.

EasyTechnical

34 practiced

Write an SQL query that computes daily unique users and conversion rate per experiment group given a table events(user_id, event_type, event_time, experiment_bucket). Provide assumptions about event schema and how you dedupe multiple conversion events per day per user.

Sample Answer

Approach: Aggregate daily unique users by event_date and experiment_bucket, and compute conversion rate as (distinct users who converted that day) / (distinct active users that day). Deduplicate conversions per user per day by taking the first conversion event_time per user per day (or using DISTINCT on (user_id, date) for event_type='conversion').

Assumptions:- events(user_id STRING, event_type STRING, event_time TIMESTAMP, experiment_bucket STRING)- event_type values include 'view' (or other active events) and 'conversion'- A user is "active" that day if they have any event (or specifically a 'view') that day- We count at most one conversion per user per day

SQL (ANSI / BigQuery-style):

sql

WITH daily_events AS (
  SELECT
    user_id,
    experiment_bucket,
    DATE(event_time) AS event_date,
    event_type
  FROM events
),
daily_active AS (
  -- unique active users per day per bucket
  SELECT
    event_date,
    experiment_bucket,
    COUNT(DISTINCT user_id) AS daily_active_users
  FROM daily_events
  GROUP BY event_date, experiment_bucket
),
daily_conversions AS (
  -- dedupe: count each user at most once per day if they had any conversion event
  SELECT
    event_date,
    experiment_bucket,
    COUNT(DISTINCT user_id) AS daily_converters
  FROM daily_events
  WHERE event_type = 'conversion'
  GROUP BY event_date, experiment_bucket
)
SELECT
  a.event_date,
  a.experiment_bucket,
  a.daily_active_users,
  COALESCE(c.daily_converters, 0) AS daily_converters,
  SAFE_DIVIDE(COALESCE(c.daily_converters, 0), a.daily_active_users) AS conversion_rate
FROM daily_active a
LEFT JOIN daily_conversions c
  ON a.event_date = c.event_date
  AND a.experiment_bucket = c.experiment_bucket
ORDER BY a.event_date, a.experiment_bucket;

Notes:- SAFE_DIVIDE avoids division-by-zero; use CASE WHEN if not supported.- If "active" should be defined as a specific event_type (e.g., 'view'), filter daily_events accordingly.- For multi-day dedupe (count unique users across experiment period), replace DATE(event_time) with the period and count DISTINCT user_id across period.

Unlock Full Question Bank

Get access to hundreds of Real World Experimental Challenges and Solutions interview questions and detailed answers.

Join thousands of developers preparing for their dream job.