Experiment Analysis & Result Interpretation Questions

Approach: compute per-user 14-day attributed revenue (sum charge_amount within 14 days after assignment), include users with zero purchases, mark censored users (assignment + 14d > data_cutoff) and exclude or use survival-aware imputation; here we exclude censored users for unbiased ARPU uplift. Then bootstrap users (resample users with replacement within each arm) to get distribution of mean per-user revenue and 95% CI for uplift (treatment_mean - control_mean).

SQL to build per-user revenue and censor flag:

-- params: assignment_table(events) has user_id, assignment_date, arm ('treatment'/'control')
WITH events AS (
  SELECT user_id, charge_amount, event_time
  FROM raw_events
),
assign AS (
  SELECT user_id, MIN(assignment_date) AS assign_date, arm
  FROM assignments
  GROUP BY user_id, arm
),
user_window AS (
  SELECT a.user_id, a.arm, a.assign_date,
         DATE_ADD(a.assign_date, INTERVAL 14 DAY) AS window_end
  FROM assign a
),
user_revenue AS (
  SELECT u.user_id, u.arm, u.assign_date, u.window_end,
         COALESCE(SUM(e.charge_amount), 0) AS revenue_14d,
         CASE WHEN u.window_end <= (SELECT MAX(event_time) FROM events) THEN 0 ELSE 1 END AS censored
  FROM user_window u
  LEFT JOIN events e
    ON e.user_id = u.user_id
   AND e.event_time >= u.assign_date
   AND e.event_time < u.window_end
  GROUP BY u.user_id, u.arm, u.assign_date, u.window_end
)
SELECT * FROM user_revenue;

Notes:

• COALESCE ensures users with no purchases get revenue = 0.
• censored = 1 means incomplete 14d window; exclude these in final analysis or treat with survival methods.

Bootstrap procedure (Python/pandas/numpy):

import numpy as np
import pandas as pd

# user_revenue_df: columns ['user_id','arm','revenue_14d','censored']
df = user_revenue_df[df['censored']==0]  # exclude censored users
control = df[df.arm=='control']
treat = df[df.arm=='treatment']

def bootstrap_mean_diff(ctrl, trt, n_boot=10000, seed=0):
    rng = np.random.default_rng(seed)
    diffs = np.empty(n_boot)
    for i in range(n_boot):
        sample_c = rng.choice(ctrl.revenue_14d.values, size=len(ctrl), replace=True)
        sample_t = rng.choice(trt.revenue_14d.values, size=len(trt), replace=True)
        diffs[i] = sample_t.mean() - sample_c.mean()
    return np.percentile(diffs, [2.5,50,97.5]), diffs

ci, diffs = bootstrap_mean_diff(control, treat)
print("ARPU uplift median and 95% CI:", ci)

Interpretation and caveats:

• Use user-level bootstrap (not event-level) to preserve correlation.
• Excluding censored users gives unbiased estimate if censoring is non-informative; if censored differs by arm, consider inverse-prob weighting or survival models (e.g., Kaplan-Meier for time-to-first-purchase then integrate).
• Report means, CI, sample sizes, and percent censored by arm.
• If many zeros and skewed revenue, consider bootstrap on median or log-transform + bias correction.

Situation: The experiment shows a statistically significant lift on the primary metric (p=0.04) but a marginal increase in refund rate (guardrail) with p=0.06. Product wants to ship.

Answer (advice and rationale):

• Frame the trade-off: a p=0.04 on primary suggests evidence of benefit; p=0.06 on a guardrail is borderline but may still represent a real risk. Prioritize based on business impact (expected revenue/upside vs cost of refunds, customer lifetime value, reputation).
• Ask risk-tolerance questions: How expensive/irreversible are refunds? Can operations absorb them? What's the customer support/brand cost? Who is the stakeholder that owns refunds?

Additional checks before rollout:

• Validate data quality: re-run ingestion checks, ensure refunds are deduplicated and correctly attributed to the test.
• Check statistical assumptions: ensure randomization held (covariate balance), no instrumentation bugs, and test duration covered multiple weekly cycles.
• Effect size & power: examine the magnitude of the refund lift and confidence intervals — is it practically meaningful? Check whether the test was underpowered to detect guardrail changes.
• Multiple comparisons: were many metrics tested? Apply corrections or control FDR for guardrails.
• Segmentation: run stratified analyses (by region, device, user tenure) to see if refunds concentrate in a particular segment; this can enable a targeted rollout.
• Temporal analysis: inspect refund time-lag (immediate vs delayed) and cumulative lift over time.
• Root-cause: look at qualitative data (support tickets, session replays) to understand why refunds increased.

Recommendation:

• If primary upside substantially outweighs measurable cost of refunds and refunds are reversible/mitigable: consider a phased rollout (e.g., 5–10% ramp) with feature flags, tight monitoring, and an automatic rollback threshold for refunds.
• If refunds are high-cost or concentrated in key segments: do not full-ship; instead iterate on the feature to address causes, or run an experiment focused on the risky segment.
• Communicate trade-offs and monitoring plan to stakeholders, including clear KPIs, a rollback plan, and owner for real-time tracking.

This balances statistical evidence with business risk and provides concrete steps to reduce uncertainty before full deployment.

1. Control for multiple testing

• Pre-specify primary metric(s) and label others as secondary/exploratory. Only control Type I for primary.
• For secondary metrics, use False Discovery Rate (Benjamini–Hochberg) to limit expected proportion of false positives when assessing many outcomes; use Bonferroni or Holm for conservative family-wise error if false positives are costly (legal/financial).
• Consider hierarchical testing: test grouped families in order (primary first, then familywise secondaries) to preserve power.
• Report raw p-values, adjusted p-values, and effect sizes with CIs; emphasize practical significance.

2. Decide whether to ship

• Use a decision rule combining statistical and business criteria:
  • Primary metric: must meet pre-specified significance/power and minimal detectable effect (MDE).
  • Guardrails: critical secondaries (e.g., retention, revenue) must not show harmful effects beyond pre-specified thresholds.
  • Cost-benefit: estimate expected value = delta_primary * revenue_per_unit − costs from adverse secondary impacts. Use Bayesian decision framework to incorporate uncertainty.
• If primary wins but some secondaries move adverse yet non-conclusive after multiple-testing correction, prefer caution: delay full roll-out, partial roll-out, or phased rollout with close monitoring.

3. Follow-up experiment / observational analysis

• Design a confirmatory A/B with pre-registered hypotheses for affected secondaries, powered to detect the observed effects (use observed SDs to compute sample size).
• Use stratified or blocked randomization on covariates correlated with ambiguous metrics to reduce variance.
• Consider factorial design to test interactions if multiple changes occurred.
• Use sequential testing with alpha spending (e.g., O’Brien–Fleming) if you need interim decisions.
• Observational analyses: run regression adjustment, difference-in-differences, and propensity-score methods on larger historical data to assess mechanisms; perform mediation analysis to see if primary effect flows through a secondary.
• Always run subgroup and sanity checks, surface heterogeneous treatment effects, and report pre-specified thresholds and FDR-adjusted results so stakeholders can make an informed, quantified decision.

First, state the goal: determine whether the small but significant uplift is reliable, repeatable, and profitable once rolled out. Prioritized follow-up analyses:

1. Segmentation (high priority)

• Break down by user cohort (acquisition channel, geography, device, spend tier, new vs. returning).
• Metric: treatment vs. control uplift and p-value per segment; also absolute revenue and ARPU.
• Signal to scale: consistent positive uplift across high-value segments (e.g., top 30% revenue cohort) or clear winner segments to target.

2. Decay / retention check (high)

• Track effect over time (daily/weekly revenue per user) for experiment cohort vs. control for at least 4–12 weeks.
• Look for fade-in/fade-out: compute survival curves, retention rate deltas, and time-to-neutralization.
• Signal: persistent or growing cumulative lift after initial period; if uplift decays to zero quickly, deprioritize scaling.

3. Dose-response / exposure analysis (medium)

• If applicable, examine intensity (number of exposures, feature usage) vs. outcome using regression with interaction terms or causal forest.
• Metric: marginal revenue per additional exposure; check diminishing returns.
• Signal: positive marginal returns for realistic exposure levels.

4. Cost-to-implement / business case (high)

• Estimate implementation, operational, and incremental cost per user; compute payback period and ROI.
• Use cohort-level CAC, incremental gross margin per user, and projected scale.
• Signal: payback < acceptable threshold (e.g., 6 months) and positive NPV at expected scale.

5. Long-term holdout / ramp experiment (high)

• Recommend rolling rollout with a permanently held-back control (e.g., 10% holdout) or stepped-wedge design across regions—measure 3–6 months.
• Monitor secondary metrics (churn, LTV, customer support, conversion funnels) to detect unintended consequences.

Decision thresholds / signals to scale:

• Statistically and practically significant cumulative LTV uplift (e.g., > minimum detectable effect that justifies cost), consistent across key segments or concentrated in profitable segments.
• Positive ROI / acceptable payback period at scale.
• No negative impacts on retention, conversion, or other core metrics in long-term holdouts.
• Robustness checks pass (pre-period balance, no selective attrition, consistent dose-response).

If signals mixed: scale to targeted segments with a monitored ramp and maintain a holdout to validate long-term effects.

Step-by-step investigation plan (4–6 week timeframe)

1. Quick sanity checks (0–3 days)

• Verify instrumentation and data pipelines: event counts, missing keys, duplicates, timestamp drift.
• SQL to compare event volumes pre/post rollout and between groups:

SELECT group, COUNT(*) AS events, COUNT(DISTINCT user_id) AS users
FROM retention_events
WHERE event_time BETWEEN '2025-10-01' AND '2025-10-28'
GROUP BY group;

Evidence to stop: gross mismatch in event counts or missing group assignment → rollback until tracking fixed.

2. Validate experiment setup (3–7 days)

• Confirm randomization and assignment stability over time: check user-level group churn.

SELECT user_id, MIN(group) AS first_group, COUNT(DISTINCT group) AS groups_seen
FROM assignments
GROUP BY user_id
HAVING groups_seen > 1;

• Check exposure: were targeted users actually reached (emails/ads/notifications delivered and opened)?
  Evidence to iterate: low exposure rate (e.g., <60%) → fix delivery channels and rerun.

3. Statistical power & sample-size review (7–10 days)

• Re-calc minimum detectable effect (MDE) given observed variance and sample size. If underpowered, compute required weeks/users.
• If current power < 80% for sensible uplift → extend experiment rather than rollback.

4. Cohort & segmentation analysis (10–18 days)

• Break down by acquisition cohort, tenure, geography, device, prior engagement. Look for heterogeneous treatment effects.
• Example: retention by tenure:

SELECT tenure_bucket, group, retention_rate
FROM user_retention_summary

Evidence to iterate: positive effect in specific segment (e.g., new users) → pivot to targeted campaign.

5. Selection bias and leakage checks (18–24 days)

• Verify no unintended exclusion/inclusion (e.g., high-value users excluded).
• Audit overlaps with other campaigns; control for concurrent treatments via regression.

6. Causal sensitivity and robustness (24–30 days)

• Run adjusted models (logistic/cox) controlling for covariates; do permutation tests, check pre-trend equivalence.
• If adjustments reveal effect → investigate confounders; if no effect across methods → likely true null.

7. Synthesize and recommend (30–35 days)

• Rollback if: instrumentation broken, exposure failed, negative impact on key metrics (revenue, NPS), or harm to high-value users.
• Iterate if: underpowered, partial success in segments, delivery issues, or clear actionable learnings (message timing/content).
• Provide clear next steps: fix tracking, target winner segments, increase sample, or sunset feature.

Deliverables: daily health dashboard (week 1), interim report with power and exposure (week 2), final root-cause report with recommended action and timelines (week 5).