Experiment Analysis & Result Interpretation Questions

Framework summary (notation):

• Let delta = true per-user relative revenue uplift (treatment - control). Prior p(delta).
• Let r = baseline revenue per user; N = size of target population; C0 = fixed rollout/engineering cost; Cm = maintenance cost per unit time; T = expected lifetime after rollout. Total cost if ship = C0 + Cm·T.
• Data D from an experiment (n_treatment, n_control) gives likelihood p(D | delta). Posterior p(delta | D) ∝ p(D | delta)p(delta).

1. Posterior expected incremental value (PEIV)
   Compute the posterior expectation of net incremental value if you ship now:
   PEIV = E_{delta | D}[ Net(delta) ] where Net(delta) = N·r·delta − (C0 + Cm·T).
   Because you only ship when net>0, the decision-theoretic expected value of choosing to ship is:
   EV_ship = E_{delta|D}[ max(0, Net(delta)) ].
   If the competitor action is "do nothing", its value is 0. So posterior expected incremental value (value of choosing ship now) = EV_ship.

Practical compute: draw M samples delta_i ∼ p(delta | D) (MCMC or closed form). Estimate
EV_ship ≈ (1/M) Σ_i max(0, N·r·delta_i − C0 − Cm·T).

2. Expected Value of Sample Information (EVSI)
   EVSI measures the expected improvement from running more experimentation (collecting additional data) before deciding versus deciding now.
   Let a be the action set {ship, do_nothing}. Let V_now = max_a E_{delta|D}[Value(a,delta)] = max(0, EV_ship) (since do_nothing = 0).
   Let D_new be potential future data from an additional experiment of size n_new with predictive distribution p(D_new | D). After observing D_new, posterior becomes p(delta | D, D_new) and the optimal expected value becomes V_after(D_new) = max_a E_{delta|D,D_new}[Value(a,delta)]. Then
   EVSI = E_{D_new | D}[ V_after(D_new) ] − V_now.
   Compute by simulation:

• For j in 1..J: simulate D_new^(j) from predictive; compute posterior p(delta|D, D_new^(j)); compute V_after^(j)=max(0, E_{delta|D,D_new^(j)}[Net(delta)]); EVSI ≈ (1/J) Σ_j V_after^(j) − V_now.

Compare EVSI to cost of running the additional experiment (sampling cost + opportunity cost). If EVSI > cost_of_sampling, experiment is worth running.

3. Stopping / decision rule
   Two complementary rules:
   A. Direct posterior-net rule (practical): Ship if EV_ship > 0 and probability of positive net meets confidence threshold:
   P(delta > delta_thresh | D) > α, where delta_thresh = (C0 + Cm·T)/(N·r) and α depends on risk tolerance (e.g., 0.9). This ensures high chance incremental revenue covers fixed costs.
   B. EVSI rule (optimal sequential): Stop experimenting and act (ship/do nothing) when EVSI ≤ cost_per_additional_sample. If EV_ship > 0 then choose ship; else do nothing. This is Bayesian optimal stopping.

Notes and implementation tips:

• Use conjugate models (normal-normal) or logistic/poisson GLMs for count/revenue data; posterior sampling via Stan/PyMC3/NumPyro for complex models.
• Account for discounting if revenue accrues over time, and structural uncertainty in N, r, or costs by including them in the posterior.
• Sensitivity: report PEIV, P(delta>delta_thresh), EVSI, and recommended action; use Monte Carlo to produce credible intervals.

1. Traffic/assignment sanity check

• What I check: total impressions, clicks, and users per variant vs expected (signup funnel and randomization ratios), plus pre-post launch sudden spikes.
• Why it matters: ensures randomization and logging worked; big imbalances indicate bias that invalidates lift estimates.
• If it fails: halt analysis, pull instrumentation logs to find rollout bugs (e.g., variant not served, tracking pixel missing). If bug irrecoverable, exclude affected time windows or users and rerun power calculations; possibly rerun the experiment.

2. Duplicate & sessionization check

• What I check: duplicate user_ids/events, unusually high event rates per user, and consistent session boundaries across analytics and backend.
• Why it matters: duplicates inflate denominators or conversions, biasing metrics and variance estimates.
• If it fails: deduplicate using canonical user keys or session IDs; identify source (replay, retry logic) and apply filters or heuristics (first-event only). Recompute metric variance and adjust confidence intervals.

3. Metric parity and instrumentation drift

• What I check: compare primary/guardrail metrics across sources (client logs, server logs, analytics); validate conversion definitions (e.g., purchase recorded with same order_ids).
• Why it matters: measurement inconsistency can create false positives/negatives.
• If it fails: reconcile by choosing the most reliable source (usually server-side), correct metric pipelines, run sensitivity analyses showing how conclusions change, and transparently report instrumentation issues to stakeholders.

Across all checks I document findings, update experiment postmortem, and recommend monitoring/alerts to prevent recurrence.

Credible intervals vs p-values — interpretation

• Bayesian credible interval (e.g., 95% CI): given the observed data and prior, there is a 95% probability the true parameter lies in this interval. It's a direct probability statement about the parameter.
• Frequentist p-value: probability of observing data at least as extreme as observed assuming the null hypothesis is true. It does NOT give the probability the null is true or an interval for the parameter.

Practical pros/cons for A/B testing

Bayesian (pros)

• Direct decision quantities: posterior distributions give probability treatment is better by X, expected uplift, probability of beating business thresholds — easier to communicate to PMs.
• Natural sequential testing: you can peek and stop without complex alpha correction (if decision rule is Bayesian), enabling faster experiments.
• Incorporates prior knowledge: useful when historical data exists or when small samples; hierarchical models pool information across segments.
• Rich uncertainty quantification: full posterior supports risk-aware decisions (e.g., decision-theoretic expected value).

Bayesian (cons)

• Requires choosing priors — can be controversial; needs sensitivity checks.
• More computationally intensive for complex models (MCMC), though approximate methods (variational, analytic conjugates) mitigate this.
• Not universally accepted in regulated settings where frequentist thresholds are standard.

Frequentist (pros)

• Simple, well-understood workflows: t-tests, z-tests, clear sample-size formulas.
• Widely accepted benchmarks (p<0.05) and tooling; often quicker for simple experiments.
• Inference doesn't require subjective priors.

Frequentist (cons)

• P-values are commonly misinterpreted; they don’t tell probability of benefit.
• Sequential peeking inflates Type I error unless adjusted (alpha spending, Bonferroni, sequential tests).
• Confidence intervals have an indirect interpretation (coverage over repeated samples), which is less intuitive for single-study business decisions.

Practical recommendation for product decisions

• If stakeholders want direct probability statements and you can specify reasonable priors, use Bayesian A/B testing for clearer risk/utility decisions (e.g., “80% chance revenue increases by ≥1%”).
• If you need fast, standard tests and regulatory acceptance, frequentist methods are fine, but enforce pre-registration, proper sample-size planning, and sequential correction.
• Regardless of approach: report effect sizes, uncertainty (posterior or CI), decision thresholds tied to business metrics (expected loss/gain), and run sensitivity checks (priors, stopping rules).

Structure (5–7 slides / 10 min):

1. TL;DR & Recommendation

• One-sentence verdict (Ship/Hold), expected impact (e.g., +$X/mo or +Y% conversion) and confidence level.

2. Experiment summary

• Hypothesis, variant descriptions, primary & guardrail metrics, test dates, sample size per cohort.

3. Reliability checks

• Randomization balance table, exposure rate, test duration vs. required sample/power, any data loss or instrumentation issues.

4. Key results (visual + numbers)

• Primary metric: bar chart with point estimates + 95% CI, absolute baseline vs. variant values, relative lift, p-value, and minimum detectable effect (MDE) context.
• Cumulative lift time-series to show stabilization and no late divergence.

5. Secondary & guardrail metrics

• Small multiples showing conversion funnel, revenue per user, retention, and any safety metrics with CIs; highlight any negative signals.

6. Segmentation & heterogeneity

• Waterfall or forest plot of segment-level effects (mobile vs desktop, country, user cohort) with sample sizes; flag significant interactions.

7. Decision rationale & risks

• Business impact calculation (N * lift * ARPU), confidence & sensitivity analysis, rollout plan or next steps (iterative tests, phased rollout, kill-switch).

Key visuals/numbers to include:

• Bar chart with 95% CI for primary metric
• Time-series (cumulative effect) to detect novelty/novelty decay
• Randomization balance table
• Sample size / power summary and MDE
• Segment forest plot
• Business-impact waterfall (revenue/ops impact)
• Clear call-to-action with contingency plan

Speak in plain language for PMs/execs (impact & risk) and include technical appendix for engineers (stat tests, code/queries, logs).

Situation: The experiment shows a clear win on the primary metric (p = 0.02) but a guardrail metric trends worse with p = 0.06 (borderline non-significant). I treat this as mixed evidence, not a simple go/no-go.

Interpretation and reasoning:

• Statistical significance: Primary metric p=0.02 suggests the uplift is unlikely due to chance. Guardrail p=0.06 is above the conventional 0.05 threshold but close — could be a real effect that the study is underpowered for the guardrail.
• Practical vs statistical: I examine effect sizes and CIs for both metrics. A small but statistically significant uplift might not justify risk if the guardrail degradation is business-critical even if marginal.
• Multiplicity & pre-specification: Verify the guardrail was pre-specified, check multiple comparisons corrections and whether thresholds were adjusted.
• Data quality & assumption checks: Validate instrumentation, look for logging/selection bugs, check randomization balance, and ensure no differential attrition.
• Heterogeneity: Run pre-specified segment analyses (device, geography, user value) to see if degradation concentrates in high-value segments.
• Power analysis: Compute post-hoc power for the guardrail to estimate sample size needed to detect the observed effect with confidence.

Actions I would take before recommending rollout:

1. Re-verify data integrity and randomization.
2. Inspect effect sizes and CIs; quantify expected business impact if guardrail is real.
3. Run sensitivity analyses and subgroup checks to identify affected cohorts.
4. If the guardrail is critical and effect uncertain: extend the experiment or increase sample size to achieve adequate power, or run a targeted follow-up.
5. If rapid rollout is desired but risk exists: recommend a phased rollout (canary) with close monitoring and an automated rollback if guardrail crosses a threshold.
6. Consider Bayesian analysis to estimate probability of harm and communicate risk to stakeholders.

Recommendation: Do not full-rollout immediately. Present balanced evidence (effect sizes, CIs, risk scenarios) and propose either (a) extend the test / run follow-up targeted at the guardrail, or (b) phased rollout with strict monitoring and rollback conditions — choice depends on business risk tolerance and speed-to-market priorities.