Experiment Analysis & Result Interpretation Questions

Goal: estimate A vs B effect on Revenue Per User (RPU) robustly despite heavy tails and outliers, and present business-impact with uncertainty.

1. Pre-analysis setup

• Define unit (user, session) and attribution window (e.g., 30 days post-exposure). Pre-specify primary metric, crew, exclusion rules, and alpha (usually 0.05).
• Run sample size/power using robust variance estimates (simulate heavy-tailed draws) to ensure sufficient power for mean differences or trimmed/median metrics.

2. Preprocessing options (pre-specified; don’t pick after seeing results)

• Winsorization: cap extremes at chosen quantiles (e.g., 99th/1st) to limit outliers’ influence.
• Truncation: remove top x% (less common; must be justified).
• Log-transform: log(RPU + c) (c small, e.g., 1 or machine epsilon) reduces skew; remember back-transform bias (use smearing estimator or bootstrap).
• Trimmed means: e.g., 10–20% symmetric trimming reduces tail sensitivity while estimating central tendency.
  Choice: pre-register 2 parallel analyses — (A) winsorized mean (99/1) and (B) 20% trimmed mean — plus (C) median and (D) log-transformed mean as robustness checks.

3. Statistical tests and estimators

• Primary test: permutation (randomization) test on the chosen estimator (winsorized mean or trimmed mean). Permutation is nonparametric, respects random assignment, and controls type I without normality.
• Complementary: bootstrap to compute CIs for estimator differences (bias-corrected and accelerated (BCa) preferred).
• Use robust standard errors with M-estimators (Huber) if using parametric regression adjusting for covariates.
• For median: use Hodges–Lehmann estimator and rank-based tests (Mann–Whitney not directly testing medians under differing shapes) or bootstrap CI for median difference.

4. Covariate adjustment

• Use ANCOVA-style regression on transformed/winsorized outcome to improve precision: outcome ~ treatment + pre-period revenue + key covariates. For heavy tails, use robust regression (e.g., Huber/quantile regression for median).

5. Sensitivity analyses

• Vary winsorization/trimming thresholds (95/5, 99/1, 90/10) and log offsets to show stability.
• Run analyses on raw mean as an extreme comparison.
• Check effect heterogeneity: by percentile of pre-period spend, geography, device.
• Check influence diagnostics: remove top N users to show impact.

6. Presenting business impact & uncertainty

• Report both relative (%) uplift and absolute dollars per user (delta RPU) with 95% CI from bootstrap/permutation.
• Translate to expected incremental revenue = delta RPU * exposed user count over relevant period (show lower/upper CI and probability of positive impact).
• Provide scenario table: conservative/expected/optimistic using different estimators (trimmed mean, winsorized, median) so stakeholders see range.
• Visuals: CDFs of RPU by arm, violin/boxplots with trimmed means, forest plot of estimators with CIs, and a revenue-impact waterfall showing conversions to total $ over time.
• State assumptions and practical significance threshold (e.g., minimum detectable lift in $ or % that justifies rollout).

7. Decision rule

• Pre-specify primary estimator (e.g., 99/1 winsorized mean) and require both statistical significance and minimum business threshold (e.g., >$X/user) or supporting robustness across other analyses before recommending rollout.

Summary: Use robust estimators (winsorized/trimmed means, median), nonparametric tests (permutation + bootstrap CIs), covariate adjustment via robust regression, run sensitivity checks across thresholds, and present absolute and relative revenue impacts with CI and scenario tables so business can act with quantified uncertainty.

Calculation (step‑by‑step)

• Observed absolute lift = 0.8 percentage points = 0.008 additional conversions per user.
• For 100,000 users: incremental conversions = 100,000 * 0.008 = 800 conversions.
• Average revenue per conversion = $50 → incremental revenue = 800 * $50 = $40,000.

Confidence interval propagation

• 95% CI for lift: [0.2, 1.4] percentage points = [0.002, 0.014] per user.
• Lower bound incremental conversions = 100,000 * 0.002 = 200 → revenue = 200 * $50 = $10,000.
• Upper bound incremental conversions = 100,000 * 0.014 = 1,400 → revenue = 1,400 * $50 = $70,000.

How to explain the uncertainty to stakeholders

• The point estimate says we expect about $40k incremental revenue per 100k users, but the true value plausibly lies between $10k and $70k given sampling variability.
• “95% CI” means if we repeated this experiment many times, about 95% of similarly computed intervals would contain the true lift; it does not guarantee the true value is inside any single interval.
• Communicate risk: worst‑case (CI lower) still yields positive incremental revenue here ($10k), but assess costs: if implementation or marketing costs exceed $10k per 100k users, ROI could be negative.
  Caveats and recommendations
• CI assumes experiment valid (randomization, no major bias). Check for integrity, sample size adequacy, and heterogeneity across segments.
• Revenue per conversion is an average — lifetime value, returns, or attribution windows may change realized revenue.
• Recommended next steps: segment lift by cohort, run a longer or larger experiment to narrow CI, compute net incremental profit (subtract costs), and perform sensitivity analysis across different ARPU assumptions.

Situation: When an A/B experiment had assignment percentages change mid-test and some users were reassigned, you must avoid biased estimates from naïve aggregation. Goal: produce a defensible intent-to-treat (ITT) analysis and sensitivity checks.

1. Initial diagnostics and assumptions to test

• Recreate assignment timeline: for each user, timestamp of first exposure, assignment history, and any reassignment events.
• Test randomization integrity before and after the rollout change (covariate balance by cohort/time-window).
• Check for differential attrition or exposure: did reassignment correlate with user features or behavior?
• Verify logging completeness (missing events, delayed writes).

2. Who to include in ITT

• Primary ITT: include users based on their initial randomized assignment (first assignment at time of eligibility). Rationale: preserves randomization and estimates effect of offering treatment.
• Secondary ITTs / per-protocol: create variants for (a) first-assignment only, (b) current-assignment at outcome time, (c) always-treated vs never-treated. Report both ITT and per-protocol but emphasize ITT for causal validity.

3. Handling partial-rollout and reassignment in analysis

• Time-varying treatment indicator: appropriate when treatment effect can change over time or when outcomes are measured post-reassignment. Model outcome as function of treatment(t) using panel/cox/Poisson or survival models depending on outcome; include user fixed effects if repeated measures.
• Weighted analysis: appropriate if you want to estimate average treatment effect for the population under a uniform assignment policy. Weight observations by inverse probability of being in the observed assignment path (IPTW) when reassignment depends on observed covariates. Use stabilized weights and check for extreme weights.
• Simpler approach: stratify by rollout period (pre-change, transition, post-change) and estimate ITT within each stratum, then meta-analyze or pool with period indicators and interaction terms.

4. Practical workflow / steps

• SQL: produce per-user table with user_id, first_assignment, assignment_timestamps, final_assignment, first_exposure_time, covariates, outcome_time/value.
• Run balance checks and attrition tables by first_assignment and by period.
• Primary model: regression of outcome on indicator(first_assignment==treatment) + period fixed effects + key covariates; cluster SEs by user.
• Sensitivity: per-protocol model (actual exposure), time-varying model (treatment(t)), IPTW-adjusted model; placebo tests on pre-period outcomes.

5. Reporting and recommendations

• Report primary ITT (first assignment) estimate and 2–3 sensitivity analyses with clear assumptions.
• If reassignment likely non-random, downweight per-protocol and emphasize that causal interpretation relies on additional untestable assumptions.
• Transparently document data-cleaning rules (how you defined first exposure, how you handled multiple accounts), number of excluded records, and diagnostics (balance p-values, weight distributions).

This approach preserves randomized inference via ITT, uses time-varying/weighted methods only when their assumptions are credible, and provides stakeholders with robustness checks so decisions aren’t driven by artifacts of partial rollout or reassignment.

When an A/B test spans a holiday and traffic composition shifts, you need methods that adjust for time-varying confounding (seasonality, shocks). Three practical approaches:

1. Interrupted Time Series (ITS)

• What: Compare pre- and post-intervention trends for control and treatment series (or single series with strong pre-period).
• How: Fit segmented regression (level + slope) with seasonality terms (day-of-week, holiday dummies) and autocorrelation (ARIMA/GEE/GLS).
• Assumptions: No other simultaneous interventions; underlying pre-trend is stable and well-modeled.
• Diagnostics: Check pre-intervention fit (parallelism to projected counterfactual), residual autocorrelation (ACF/PACF), goodness-of-fit, sensitivity to varying pre-period windows.

2. Difference-in-Differences (DiD)

• What: Estimate treatment effect as (post−pre change in treatment) − (post−pre change in control).
• How: Use panel regression with time and unit fixed effects, interaction term Treatment×Post, include time-varying covariates (traffic mix), seasonality dummies, cluster-robust SEs.
• Assumptions: Parallel trends (absent treatment), no spillovers, similar composition after controlling covariates.
• Diagnostics: Pre-trend placebo tests (lead coefficients ≈0), balance checks on covariates over time, event-study plot to visualize dynamic effects.

3. Synthetic Control

• What: Build a weighted combination of donor units (other days/segments) that best match the treated unit’s pre-period trajectory; compare post-period divergence.
• How: Optimize weights on predictors and lagged outcomes; include holiday indicators and traffic features as predictors.
• Assumptions: Donor pool can recreate counterfactual; no interference from treated unit to donors.
• Diagnostics: Pre-fit RMSPE (low); placebo tests by applying synthetic control to donor units (inference via RMSPE ratio); sensitivity to donor set and predictor choice.

Practical workflow:

• Start exploratory plots (daily metrics, traffic composition), decompose seasonality.
• Choose approach: DiD if good control group exists; ITS if single series with long pre-period; synthetic control if single treated unit and many donor units.
• Report robustness: alternative specifications, holiday dummies, covariate adjustment, and pre-period falsification tests. Quantify uncertainty (bootstrap or robust SEs) and present visual diagnostics to stakeholders.

Approach: compute per-user 14-day attributed revenue (sum charge_amount within 14 days after assignment), include users with zero purchases, mark censored users (assignment + 14d > data_cutoff) and exclude or use survival-aware imputation; here we exclude censored users for unbiased ARPU uplift. Then bootstrap users (resample users with replacement within each arm) to get distribution of mean per-user revenue and 95% CI for uplift (treatment_mean - control_mean).

SQL to build per-user revenue and censor flag:

-- params: assignment_table(events) has user_id, assignment_date, arm ('treatment'/'control')
WITH events AS (
  SELECT user_id, charge_amount, event_time
  FROM raw_events
),
assign AS (
  SELECT user_id, MIN(assignment_date) AS assign_date, arm
  FROM assignments
  GROUP BY user_id, arm
),
user_window AS (
  SELECT a.user_id, a.arm, a.assign_date,
         DATE_ADD(a.assign_date, INTERVAL 14 DAY) AS window_end
  FROM assign a
),
user_revenue AS (
  SELECT u.user_id, u.arm, u.assign_date, u.window_end,
         COALESCE(SUM(e.charge_amount), 0) AS revenue_14d,
         CASE WHEN u.window_end <= (SELECT MAX(event_time) FROM events) THEN 0 ELSE 1 END AS censored
  FROM user_window u
  LEFT JOIN events e
    ON e.user_id = u.user_id
   AND e.event_time >= u.assign_date
   AND e.event_time < u.window_end
  GROUP BY u.user_id, u.arm, u.assign_date, u.window_end
)
SELECT * FROM user_revenue;

Notes:

• COALESCE ensures users with no purchases get revenue = 0.
• censored = 1 means incomplete 14d window; exclude these in final analysis or treat with survival methods.

Bootstrap procedure (Python/pandas/numpy):

import numpy as np
import pandas as pd

# user_revenue_df: columns ['user_id','arm','revenue_14d','censored']
df = user_revenue_df[df['censored']==0]  # exclude censored users
control = df[df.arm=='control']
treat = df[df.arm=='treatment']

def bootstrap_mean_diff(ctrl, trt, n_boot=10000, seed=0):
    rng = np.random.default_rng(seed)
    diffs = np.empty(n_boot)
    for i in range(n_boot):
        sample_c = rng.choice(ctrl.revenue_14d.values, size=len(ctrl), replace=True)
        sample_t = rng.choice(trt.revenue_14d.values, size=len(trt), replace=True)
        diffs[i] = sample_t.mean() - sample_c.mean()
    return np.percentile(diffs, [2.5,50,97.5]), diffs

ci, diffs = bootstrap_mean_diff(control, treat)
print("ARPU uplift median and 95% CI:", ci)

Interpretation and caveats:

• Use user-level bootstrap (not event-level) to preserve correlation.
• Excluding censored users gives unbiased estimate if censoring is non-informative; if censored differs by arm, consider inverse-prob weighting or survival models (e.g., Kaplan-Meier for time-to-first-purchase then integrate).
• Report means, CI, sample sizes, and percent censored by arm.
• If many zeros and skewed revenue, consider bootstrap on median or log-transform + bias correction.