Experiment Prioritization & Roadmap Questions

Situation: At my previous company I supported the Growth PM who wanted to launch a redesigned checkout upsell modal immediately because it looked better and matched a new marketing campaign. Leadership wanted quick deployment for the holiday peak.

Task: As the BI analyst, I needed to persuade the PM and executive sponsor to run an A/B test first so we could measure real impact on conversion and revenue and avoid regressing critical metrics.

Action:

• I pulled historic checkout conversion and revenue-per-session baselines and estimated detectable lift given our traffic (power calculation showing we needed ~4 weeks to detect a 3% lift with 80% power).
• I built a short deck and one-page dashboard mock showing potential upside (projected revenue) and downside (worst-case loss scenarios) over the holiday period.
• I framed risks concretely: a negative lift would cost an estimated $40k/week; an untested change could also obscure funnel issues. Benefits: controlled learning, ability to roll back, and capturing statistically valid insight for future tests.
• I proposed a compromise: run a 2-week, 50/50 A/B test with an interim stop rule and monitoring dashboard I’d maintain in real time.

Result: Leadership approved the A/B test. The new modal increased upsell conversion by 4.2% and incremental revenue by $65k in two weeks. Because the test was statistically rigorous, we rolled the modal out confidently and documented learnings that improved subsequent experiments. The approach built trust with the PM and execs—future design changes were routinely A/B tested.

Requirements & constraints

• Inputs: expected impact on north-star (delta%), confidence (power or p-value / prior), estimated engineering cost (hours), expected duration (weeks), dependencies (blocking, shared infra).
• Outputs: ranked list + visual trade-off explorer, filters, experiment details, and feedback that updates future prioritization.
• Non-functional: near-real-time updates from experiment & tracking systems; explainable scores for PMs.

Ranking algorithm (scoring)

• Normalize inputs to 0–1: Impact (I) = predicted % north‑star uplift normalized by historical range; Confidence (C) = calibrated probability of true effect (0–1); Effort (E) = engineering hours normalized; Duration (D) = normalized weeks; Dependency penalty (P) = 0 (none), 0.2 (soft), 0.5 (hard).
• Score = (I * C) / (α * E + β * D + γ + P)
  • α, β control tradeoffs; γ small constant to avoid divide-by-zero.
  • Example weights: α = 0.6, β = 0.2, γ = 0.1 (tuned from org preferences).
• Add priority boosts: strategic alignment flag, seasonal windows, or regulatory necessity.

Why this formula

• Multiplying I*C favors high-impact, high-confidence ideas.
• Dividing by effort+duration favors low-cost quick wins.
• Dependency penalty demotes risky blocked work.

Visualization & UI

• Primary explorer: interactive scatter plot (Impact on X, Effort on Y). Each point:
  • Color = Confidence (continuous heatmap)
  • Size = Duration (larger = longer)
  • Shape/badge = dependency type (diamond for blocked)
  • Tooltip: full metadata, score breakdown, owner, expected launch date.
• Secondary views:
  • Ranked table with sortable columns and explainableScore column showing contributions (I, C, E, D, P).
  • “What-if” controls: sliders for α/β to simulate trade-offs.
  • Timeline/Gantt: scheduled experiments, dependencies, resource allocation.
  • Calibration panel: predicted vs observed impact over time; histogram of residuals.
• Filters: team, owner, strategic tag, timeframe, risk appetite.

Experiment history feedback loop

• Persist actual outcomes: observed north-star delta, achieved power, experiment duration, rework hours, and blockers encountered.
• Calibration & Bayesian update:
  • Use historical prediction vs outcome to fit a calibration model for Confidence and Impact priors (e.g., Bayesian shrinkage: posterior mean = (k * prior + n * obs)/(k + n)).
  • Update normalization ranges and α/β tuning via periodic backtests (optimize for realized north-star per engineering-hour).
• Automated signals:
  • Flag consistent over/underestimation by individual owners or hypotheses types.
  • Reweight Confidence for people/teams with bias until recalibrated.
• Learning dashboard:
  • Show model accuracy metrics (RMSE, calibration plot).
  • Recommend priors for new experiments based on similar past experiments (features: feature area, change type, funnel stage).

Data & implementation notes (BI perspective)

• Data sources: experiment platform (segment/optimizely), tracking events, issue tracker (Jira for dependency/estimates), time tracking.
• ETL: nightly/streaming job to compute normalized inputs & store experiment ledger.
• Computation: scoring in BI layer or preprocessing in warehouse (dbt/SQL), store score and components for explainability.
• Tools: Looker/Power BI/Tableau for visualization; use parameterized dashboards for what-if sliders via datasource parameters or lightweight front-end if needed.
• Governance: record final decisions and A/B tags; enable audit trail.

Edge cases & considerations

• Low-sample or high-uncertainty experiments: enforce minimum sample size or require pilot flag; confidence floor.
• Correlated experiments: adjust impact attribution using causal DAGs or mark as mutually exclusive groups.
• Resource contention: integrate engineering capacity view; recommend schedule to maximize throughput.

Outcome

• PMs get an explainable, tunable ranking plus visual trade-off explorer to prioritize experiments aligned to ROI (north-star uplift per engineering-hour) while learning from historical outcomes to continuously improve predictions and confidence calibration.

Approach: I’d treat this as a funnel optimization program — propose experiments that are measurable, low-risk, and sequenced from highest ROI/lowest effort to higher-effort platform changes. For each experiment I state a hypothesis, primary metric, guardrails, sample-size guidance, and BI/measurement needs.

1. Experiment A — Signup page microcopy & reduced fields

• Hypothesis: Simplifying form + clearer benefits will lift Visit→Signup from 4% → +20% relative.
• Primary metric: Visit→Signup conversion rate.
• Guardrails: Signup→Activate and Activate→Purchase conversion rates (should not drop); signups’ 7‑day retention.
• Sample sizing: baseline p=0.04; to detect +20% relative (p→0.048) at α=0.05, power=0.8 → need ≈150k visits per arm (approx; run BI power calc).
• BI needs: event-level tracking, cohort dashboard showing signup quality (activation, purchase, retention) by variant.

2. Experiment B — Email welcome flow (nurture) for new signups

• Hypothesis: A 3-email onboarding sequence will increase Signup→Activate from 50% → +15% relative.
• Primary metric: Signup→Activate within 7 days.
• Guardrails: Email unsubscribe rate, spam complaints, deliverability; lifetime purchase rate.
• Sample sizing: baseline p=0.50; detect +15% relative (p→0.575) → ≈4k signups per arm.
• BI: track email open/click, activation timestamps, segmentation by channel and device.

3. Experiment C — In-app guided activation (product tour / checklist)

• Hypothesis: Interactive product tour increases Activate→Purchase from 10% → +30% relative.
• Primary metric: Activate→First Purchase within 14 days.
• Guardrails: Time-to-activate, support tickets, NPS.
• Sample sizing: baseline p=0.10; detect +30% relative (p→0.13) → ≈13k activated users per arm.
• BI: funnel dashboards, event instrumentation for tour steps, purchase attribution.

4. Experiment D — Discounted first purchase A/B (price promotion)

• Hypothesis: 20% off first purchase increases Purchase conversion but may reduce AOV.
• Primary metric: Purchase conversion rate.
• Guardrails: Average order value (AOV), gross margin, repeat purchase rate at 30/90 days.
• Sample sizing: baseline p=0.10; choose detectable lift (e.g., +50% to 0.15) → ~3.5k activations per arm.
• BI: revenue attribution, margin analysis, cohort LTV monitoring.

Prioritization rationale:

• Order by expected impact / implementation effort / risk. A and B are low-effort, platform-agnostic, and preserve unit economics → prioritize first. C is higher dev but targets purchase intent → next. D yields fast lift but risks margin and quality — run after guardrail monitoring and only if LTV impact acceptable.

Operational notes (BI-specific):

• Implement consistent unique identifiers across events, SLAs for instrumentation, and prebuilt A/B dashboard template (sample size estimator, sequential testing controls, confidence intervals, and impact on revenue and LTV).
• Use sequential testing or group sequential stopping rules; pre-register primary metric and analysis window.

Hypothesis statement:
Deploying the new "Smart Recommendations" module will increase 7-day retention (proportion of users active on day 7 after sign-up) for new mobile app users. Target population: new users who complete first session in the first 7 days (cohort = users who signed up during experiment). Expected direction & magnitude (MDE): an absolute increase of +2.5 percentage points in 7-day retention (baseline = 18% → target = 20.5%), tested at α=0.05 and 80% power. Success criteria: experiment considered successful if the treatment group shows a statistically significant uplift ≥ 2.5pp vs control (two-sided test p<0.05) and there is no statistically significant decrease in secondary metrics (7-day DAU, onboarding completion, or revenue per user).

How I'd prioritize under limited development resources:

• Score this hypothesis using ICE: Impact (expected uplift * business value), Confidence (data quality, prior signals), Ease (engineering effort, instrumentation). Smart Recommendations typically has high impact but medium effort; estimate: Impact=8, Confidence=6, Ease=4 → ICE = 18.
• Compare to alternatives (e.g., onboarding copy change) by the same framework. Prefer experiments with similar impact but lower effort/shorter time-to-insight.
• Check prerequisites: ensure event instrumentation and cohort tracking exist—if missing, that raises cost and lowers priority.
• If resources constrained, run a lightweight proof-of-concept: offline analysis (historical collaborative-filtering simulation) to validate signal before dev work, or a UI-only A/B test exposing precomputed recommendations to a small segment.
• Prioritize if ICE and alignment with retention OKRs are high, or defer and run lower-effort experiments first while preparing instrumentation.

Situation: As the BI lead supporting company-wide experimentation, I designed a reproducible prioritization framework that balances expected revenue impact, risk, time-to-run, sample-size feasibility, and resource constraints to produce a quarterly experiment roadmap.

Scoring function (normalized 0–100):
Score = 0.35Impact + 0.20Feasibility + 0.15Time + 0.15RiskAdj + 0.15*Resource

Where each component is 0–100:

• Impact (35%): estimated incremental revenue = lift% * baseline revenue. Normalize by dividing by max expected impact across candidates, then *100.
• Feasibility (20%): statistical feasibility score = 100 if required sample size for desired power (80%) and MDE fits available traffic, else 100*(available_sample / required_sample) capped at 0–100.
  Use two-sided z-test sample-size calc: n = 2*(Z_{1-α/2}+Z_{1-β})^2 * p*(1-p) / MDE^2.
• Time (15%): time-to-run + implementation time. Map to score: <=2 weeks→100, 2–6 weeks linear decay, >6 weeks→0.
• RiskAdj (15%): negative score for regulatory/brand/technical risk. High risk = 0, medium 50, low 100.
• Resource (15%): engineering/analytics effort. Low effort→100, medium 50, high 0. Adjust for dependencies.

Decision thresholds:

• Tier A (Score ≥75): Priority — run next quarter, allocate engineering + analytics sprint, track daily.
• Tier B (50–74): Backlog — run if slack capacity; prepare A/B test plan and finalize feasibility.
• Tier C (<50): Defer or pilot qualitatively; revisit next quarter.

Quarterly roadmap process:

1. Intake (week 1–2): collect proposals with templates (hypothesis, metrics, baseline, expected lift, MDE).
2. BI pre-screen (week 2): compute required sample sizes, baseline revenue, and initial score.
3. Prioritization workshop (week 3): cross-functional scoring, adjust weights if strategic priorities change.
4. Roadmap publish (week 4): finalize Tier A/B with owners, allocate resources, set KPI dashboards.
5. Execution & monitoring: BI builds experiment dashboards and QA metrics; weekly check-ins.
6. Post-quarter review: analyze results, update priors for Bayesian uplift models, and re-run scoring for next quarter.

Notes & trade-offs:

• Use conservative MDE and power defaults (80%/95% CI) but allow strategic overrides with leadership sign-off.
• Capture qualitative dependencies and strategic value separately to override score in exceptional cases.
• Automate scoring in a BI dashboard (Looker/Tableau) to enable live what-if scenarios.