Experiment Analysis & Result Interpretation Questions

Statistical risks

• With 40 experiments × 25 metrics = 1,000 tests, chance of false positives explodes: at α=0.05 you expect ~50 false positives by chance. This threatens decision quality and causes chasing noise.
• Correlated metrics (e.g., DAU and sessions) invalidate simple independence assumptions and complicate corrections.
• Guardrail metrics increase cost of Type II errors if over-conservative corrections reduce sensitivity.

Design to control false discoveries while retaining sensitivity

1. Pre-specification and metric families

• Require experiment plan that pre-specifies primary metric(s), secondary metrics, and guardrails before launch.
• Group metrics into families by intent/correlation (e.g., engagement, revenue, quality/guardrails). Corrections are applied per family, not globally.

2. Error control strategy

• For primary metric(s): control family-wise error rate (FWER) conservatively using Holm-Bonferroni (maintains interpretability for critical business decisions).
• For secondary metrics and exploratory signals: control false discovery rate (FDR) using Benjamini–Hochberg (BH) or Storey’s q-values — this preserves power while limiting proportion of false positives.
• For guardrails: treat as a strict family with a low α and use FWER control to avoid shipping harmful changes.

3. Power and MDE planning

• Inflate sample sizes based on the effective α after correction to retain power; compute MDEs per primary family.
• Use correlation structure (multivariate SEs or ANOVA) where possible to reduce multiplicity cost by accounting for dependence.

4. Sequential and multiplicity-aware monitoring

• If continuous monitoring is needed, use alpha-spending or sequential methods (e.g., O’Brien–Fleming) or adjust BH for dependent tests (Benjamini–Yekutieli) and avoid peeking without plan.

5. Practical policies & reporting

• Mandatory fields in experiment ticket: primary metric, metric family, directionality (one/two-sided), minimum sample, planned stopping rule.
• Dashboard displays: raw p-values, adjusted p-values (Holm/BH), q-values, confidence intervals, and effect sizes with business-impact estimate (absolute and relative).
• Flagging rules: auto-alerts for guardrail breaches (FWER-controlled) and “exploratory alerts” for BH-significant secondaries requiring replication.
• Require replication or holdout validation for any secondary/exploratory metric used to make product changes.

6. Operational safeguards

• Daily/weekly QC that checks metric correlations, population balance, and variance anomalies.
• Post-hoc Bayesian analysis or hierarchical models as robustness checks to estimate the posterior probability an effect is meaningful.
• Education for stakeholders: communicate that BH-significant exploratory metrics are candidate leads, not immediate product decisions.

Summary decision rule example

• If primary metric passes Holm-adjusted test and business impact > threshold → ship.
• If guardrail fails under FWER control → stop and investigate.
• If secondary metric passes BH (q < 0.05) → schedule replication experiment or validate on holdout cohort before productization.

This hybrid—FWER for critical decisions/guardrails, FDR for exploratory metrics—controls false discoveries while preserving sensitivity to real effects and creates clear operational policies for product teams.

Situation: I ran an A/B test that shows a statistically significant uplift in conversion, but I detect a sample ratio mismatch (SRM) — the observed allocation deviates from expected (e.g., 60/40 vs expected 50/50).

Investigation steps (what I'd do and why):

1. Confirm SRM statistically and practically

• Recompute expected counts and run a chi-squared or binomial test to quantify SRM and its p-value.
• Check whether the mismatch is transient (one day) or persistent across the experiment.

2. Audit randomization and instrumentation

• Ask engineering to review the assignment logic (hashing, bucketing, feature flags). A coding bug or stale SDK can bias assignment.
• Inspect logs for dropped events, partial loads, or client-side failures that preferentially affect one variant.
• Verify experiment config: correct population, rollout percentages, and no overlapping experiments that change targeting.

3. Examine covariates and traffic sources

• Compare baseline covariates (country, device, browser, new vs returning users) across variants. Large imbalances suggest segmentation or tracking differences.
• Check referral channels and bot traffic: a surge from a channel routed only to one variant will cause SRM.

4. Reproduce and isolate

• Replay deterministic hashing on raw user IDs to verify expected assignment.
• Run a short sanity test: simulate assignments on a historical sample to check distribution.

Determine trustworthiness of uplift:

• If SRM is due to instrumentation or targeting bug that also affects conversion measurement (e.g., missing conversion events in control), the uplift is untrustworthy.
• If SRM is caused by unrelated unequal exposure (bots, 3rd-party cache) but conversion measurement is intact and covariates remain balanced, we can salvage the analysis with caveats.
• Perform sensitivity analyses: restrict to unaffected segments (e.g., only web, or only specific countries) and see if uplift persists. Run permutation tests and bootstrap with the observed allocation to assess robustness.

Corrective actions

• If instrumentation bug: stop inference, fix the bug, and either (a) repair data if trustworthy logs exist (map assignments back using server logs and re-run analysis) or (b) re-run the experiment with corrected assignment. Prefer rerun unless data repair is complete and auditable.
• If external traffic skew (bots/referrals): filter those sessions and re-evaluate. If filtering removes SRM and uplift remains, report adjusted results.
• If overlapping experiments caused allocation drift: disentangle via regression adjustment/stratification or pause conflicting experiments and rerun.
• If unable to repair or isolate causes: discard result, document root cause, and rerun.

Communication and governance

• Immediately notify stakeholders, mark the result as provisional, and document investigation steps and decisions.
• Add monitoring: automated SRM alerts, regular randomization audits, and pre-launch integration tests for assignment logic.
• Update postmortem and preventive actions (QA for SDKs, stricter targeting rules).

Summary: Treat SRM as a red flag that can invalidate inference. Investigate assignment, instrumentation, and traffic; attempt data repair or segment-restricted inference only if auditable; otherwise pause conclusions and rerun with fixes. Prioritize transparent communication and preventive controls.

Situation: Our A/B test shows +6% bookings but lower host review ratings and shorter length-of-stay (LOS). I must determine whether the booking gain is net-positive for platform health and revenue.

Approach (what I’d ask data engineering to deliver)

• Primary metrics to compute (treatment vs. control, with CIs and p-values):

  • Bookings per user, booking conversion rate, revenue per booking, gross booking value (GBV)
  • Average LOS, distribution of LOS (median, pctiles), % of single-night vs. multi-night
  • Host review rating (mean, variance), % of 1–2 star reviews, review submission rate
  • Cancellation rate, dispute/complaint rate, refund volume
  • Longitudinal retention: 7/30/90-day guest retention and repeat-booking rate
  • Host metrics: host churn rate, new host activation, host response time

• Slices to analyze:

  • User cohorts: new vs. returning guests, geography, device, booking intent (business vs. leisure)
  • Listing types: price bands, entire home vs. private room, high-rating vs. low-rating hosts
  • Stay length buckets (1,2,3–6,7+ nights)
  • Time windows: first 7 days after exposure, and longer-term 30/90 days
  • Edge cohorts: hosts with low capacity / high sensitivity to negative reviews

• Instrumentation / data requests:

  • Event-level logs linking search exposure → click → booking → stay completion → review
  • Timestamps to compute time-to-stay and post-stay events
  • Tag treatment assignment in all downstream tables for joinable analysis
  • Funnel dashboards refreshed daily; AB test cohort attribution preserved
  • Data pipeline to join host IDs to financials and support tickets

Analysis plan and interpretation

• Estimate short-term revenue lift vs. potential long-term revenue loss from lower retention/host churn.
• Run mediation analysis: does LOS mediate the rating drop? (e.g., shorter stays lead to more negative ratings)
• Compute LTV impact: project change in 90-day LTV using retention delta and monetization changes.
• Test heterogeneity: identify segments where bookings rise without rating/LOS harm — consider scaling only to those.

Cross-functional coordination & stakeholder communications

• Immediately: brief execs and host ops with a one-page summary (lift, risks, next steps) and pause rollout if host risk > threshold.
• With Data Eng: prioritized tickets (event joins, retention cohorts, automated dashboards) with 48–72h SLAs for initial results.
• With Analytics: run causal/mediation models and provide segment-level recommendations in 3–5 days.
• With Host Ops & Trust: prepare mitigation playbook (e.g., proactive communications, guest guidance templates, support staffing) if ratings drop persists.
• With Engineering: consider feature flags to roll back or serve treatment only to safe segments; instrument A/B rollback and targeted rollouts.
• With Legal/Finance: model financial impact and regulatory risks (if any).

Recommendation framework

• If net LTV and host health remain positive → gradual scaled rollout with monitoring.
• If short-term bookings offset by material long-term host churn or LTV loss → rollback and iterate on ranking to preserve LOS/rating.
• If segmented wins exist → deploy targeted rollout and continue experiments to address root causes (e.g., match quality, pricing or guest behavior nudges).

This ensures decisions are data-driven, actionable, and aligned with host and guest health.

Interpretation:
A statistically significant uplift in clicks with no change in revenue per user suggests the feature changed user behavior (engagement) but didn’t translate to monetization. That’s not necessarily bad — it may be early signal, a neutral outcome, or indicate leakage between metrics (e.g., low-quality clicks).

Immediate diagnostics (confirm validity):

• Verify metric definitions and instrumentation: ensure clicks and revenue per user are measured correctly and consistently across variants.
• Check statistical power for revenue: revenue per user often has higher variance—confirm the experiment had enough sample size/duration to detect a meaningful revenue lift.
• Inspect segmentation: segment by cohort (new vs. returning users), device, geography, traffic source — maybe revenue moved for a subgroup but was diluted overall.
• Funnel analysis: examine downstream conversion steps (click → add-to-cart → purchase). Are clicks not converting? Is time-to-purchase shifted?
• Check engagement quality: are clicks accidental or low-intent (short session duration, high bounce)?

Business evaluation & hypotheses:

• Hypothesis A: feature increases low-value engagement (no business value).
• Hypothesis B: feature increases upper-funnel activity but requires more time/secondary flows to convert (delayed monetization).
• Hypothesis C: measurement or sampling issues.

Next steps (experiments & product):

1. Deep dive analytics: run the funnel and cohort analyses above; compute incremental revenue per click and customer lifetime metrics if available.
2. Power/sample re-estimation: if revenue test underpowered, extend experiment or increase sample to detect target MDE (minimum detectable effect).
3. Add secondary/leading metrics: track Add-to-Cart, checkout starts, average order value, retention (D30/D60), ARPU over longer windows.
4. Qualitative research: user sessions, surveys, or interviews to understand why users click but don’t convert.
5. Iteration experiment: if clicks are high quality but conversion blocked, run targeted variants that optimize post-click experience (e.g., clearer CTA, reduced friction, pricing messaging) and A/B test those.
6. Decision criteria: only scale if we see either (a) subsequent positive movement in revenue/ARPU or retention, or (b) clear product/strategic value (engagement aligned to long-term objectives) with acceptable unit economics.

Communication:
Summarize findings for stakeholders: “Clicks ↑, revenue no change. We’re investigating funnel and power; proposed plan is X (analytics + extended test + targeted follow-up experiments). Recommend not rolling out globally until we either demonstrate monetization or accept engagement-only benefit based on strategy.”

This approach balances rigor (validate results and power), product judgment (assess strategic value of engagement), and quick follow-ups (targeted experiments) before a rollout decision.

Situation: We ran a one-month pilot and saw activation go from 20% (control) to 24% (treatment). To decide whether to roll this out, I’d evaluate both statistical significance and practical significance, verify instrumentation, and communicate a clear recommendation.

Statistical test(s) & approach

• Primary: two-proportion z-test (or chi-square) comparing activation rates between groups. If group sizes are small (< ~1000) or counts are tiny, use Fisher’s exact test.
• Complementary: compute 95% confidence interval (CI) for the difference in proportions and report p-value. Optionally run a Bayesian A/B model to produce P(treatment > control) and credible intervals.

Sample size / power calculation

• Decide detectable lift (business threshold). Here observed absolute lift = 4 pp (20% → 24%). With alpha=0.05, power=0.8, p1=0.20, p2=0.24:
  • Using standard two-proportion formula yields ~1,680 users per arm (≈3,360 total) to reliably detect a 4 pp absolute lift.
• If we want to detect smaller lifts or subgroup effects, increase sample or run longer. If pilot already underpowered, report that and recommend extension.

Instrumentation & validity checks (must pass before trusting results)

• Randomization: verify random assignment logic, equal allocation, and no skew in user attributes (device, geo, cohort).
• Exposure logging: confirm impression/exposure and eligibility metrics—only count users who actually saw the new flow.
• Event definitions: validate activation event schema, timestamps, deduplication and identity joins (anonymous → logged-in).
• Data completeness/time windows: ensure full funnel window (enough time for users to activate) and no truncation.
• Bot/spam filtering and traffic sources: exclude internal traffic, QA, or marketing spikes.
• Sequential testing / peeking: check if multiple peeks occurred; adjust for multiplicity or use sequential methods.
• Sanity checks: pre-period A/A test, balance table, and look for sensor drift.

How to compute & report results

• Report: lift = (24–20)/20 = 20% relative lift (4 pp absolute).
• Primary stats: two-proportion z-test p-value, 95% CI for absolute difference (e.g., 4 pp [1.2 pp, 6.8 pp]), and power/sample-size used.
• Practical significance: translate into business metrics—e.g., users * lift → incremental activations, expected revenue or LTV impact, payback period.
• Risk/uncertainty: list instrumentation caveats, segments where effect differs, external factors (campaigns, seasonality).

Visuals and summary for leadership

• One-slide executive summary: headline result (lift, p-value, CI), recommended action (ship / expand test / iterate), and estimated business impact in dollars or activated users.
• Supporting slides: 1) funnel diagram and sample sizes; 2) CIs and p-values; 3) segmentation (platform, geo, new vs returning); 4) instrumentation checks passed; 5) sensitivity analysis (how long to run to detect smaller lifts).
• If underpowered or noisy: propose next steps — extend duration to reach required n, increase allocation, or iterate on treatment.

Recommendation framing

• If statistically significant and practically meaningful (impact > business threshold and instrumentation clean): recommend rollout with monitoring and a rollback plan.
• If statistically significant but practically marginal: consider A/B/n further experiments or targeted rollout to high-value segments.
• If not statistically significant: either extend test to reach required sample size or iterate on the experience.

This approach balances statistical rigor, instrumentation hygiene, and business-focused presentation so leadership can decide with quantified risk and impact.