Data-Driven Business Decision-Making Questions

Situation: We’re rolling out a new checkout flow to 20% of users via randomized assignment. Here’s a concrete measurement plan.

Metric selection:

• Primary metric: Conversion rate (completed purchases / checkout starts) and revenue per user (RPU) as co-primary if commercial impact is critical.
• Secondary metrics: Average order value (AOV), time-to-complete-checkout, drop-off rate at each step.

Guardrail metrics:

• Error rate/failed payments, page load time, customer support contacts, refund/chargeback rate, and retention at 7/30 days to ensure no hidden harm.

Sample-size / power:

• Precompute sample size using baseline conversion p0, minimum detectable effect (MDE) (e.g., 3–5%), alpha=0.05, power=0.8. For binary conversion, use standard two-proportion power formula or simulation. Ensure 20% rollout yields required n; if not, reduce MDE or increase duration.

Experiment duration:

• Run for at least one business-cycle plus one full weekly cycle (minimum 2–4 weeks) and until the required sample size is reached. Ensure enough users across weekday/weekend and promotional events.

Analysis approach:

• Intention-to-treat comparison between treatment and control; use difference-in-proportions or regression (logistic for conversion, linear for RPU) controlling for pre-experiment covariates (device, geo, traffic source).
• Use sequential monitoring with pre-specified looks (e.g., O’Brien-Fleming) or Bayesian stopping rules to avoid peeking bias.
• Correct for multiple testing (if multiple metrics) using hierarchical testing: test primary metric(s) first; if significant, test secondaries. Alternatively use Benjamini-Hochberg FDR control for many comparisons.
• Address novelty effects: examine time-by-treatment interaction; compute metrics over early vs. later windows (first 7 days vs. subsequent days) and test persistence. Run post-launch monitoring for degradation.
• Sensitivity analyses: per-protocol, excluding bots, stratified by cohort (new vs. returning users), and non-parametric bootstrapping for RPU.
• Reporting: dashboard with metric definitions, confidence intervals, lift %, p-values, and decision recommendation; document assumptions and limitations.

This plan ensures statistical rigor, safety guardrails, and practical monitoring to support a launch decision.

Situation: Three months after deploying a predictive churn model with outreach to flagged accounts, there’s no observable uplift.

Diagnostic plan (prioritized, measurable steps):

1. Quick data health checks (1 week)

• Verify label correctness: confirm churn definition and windows (e.g., 30/60/90 days) match what model used.
• Validate intervention logging: check outreach timestamps, channel, message id, and recipient mapping; ensure treatment flag exists for all flagged accounts.
• Sample audit: manually review 50 flagged accounts’ records to confirm outreach actually occurred.

2. Comparative analyses (2 weeks)

• Intent-to-treat vs control: compare key metrics (churn rate, engagement, purchases) for flagged-but-treated, flagged-but-not-treated (if any), and true control groups over same post-intervention window.
• Pre-post trend check: ensure flagged and control groups had comparable pre-intervention trends (parallel trends test).
• Segmented lift analysis: compute lift by cohort (ARR tiers, tenure, product usage, region, channel) to find subgroups with positive/negative effect.
• Propensity/selection bias: check whether treatment assignment correlated with other variables (e.g., high-touch accounts got different follow-up).

3. Statistical validation (2 weeks)

• Power calculation: confirm sample size is sufficient to detect expected effect; if underpowered, explain lack of significance.
• Regression adjustment / doubly robust estimation: estimate treatment effect controlling for covariates to reduce confounding.

4. A/B re-test design (4–8 weeks)

• Randomized controlled trial: randomly assign flagged accounts to outreach vs no outreach (pure RCT) to eliminate selection bias.
• Factorial test: vary treatment type (email vs phone), message content (discount vs value-add), and timing to identify effective combinations.
• Holdout and rollout: start with small pilot to validate, then scale if positive.

5. Model & feature iteration

• Feature importance drift: compare feature distributions pre/post deployment to detect concept drift.
• Add treatment-interaction features: include recent marketing exposures, contact frequency, and past responsiveness.
• Re-label using uplift framing: consider building uplift model (treatment effect model) instead of churn propensity to target those who respond positively.
• Simplify or retrain: retrain model with newer data including post-intervention outcomes; consider calibration and threshold tuning to balance precision/recall for outreach budget.

6. Operational & treatment design changes

• Evaluate messaging: review copy/timing/channel with marketing; run small qualitative user interviews.
• Frequency and cadence: reduce over-contacting; personalize offers based on segment.
• Escalation path: create alternative treatments for high-risk accounts (dedicated CS rep).

Metrics & reporting

• Primary metric: incremental retention (difference-in-differences churn rate).
• Secondary: revenue per account, engagement, cost per retained account, response rate.
• Dashboard: live RCT dashboard with segmentation, p-values, and power estimates; share weekly with stakeholders.

Expected outcome: identify whether failure is due to data/implementation issues, insufficient power, ineffective outreach, or model targeting; then either fix logging/model or run targeted RCTs and iterate on message and model (move to uplift modeling if appropriate).

Primary KPIs (outcome)

1. Saved Items Adoption Rate — % of active users who save >=1 item in period.
   Instrumentation: client event "save_item" with user_id, item_id, timestamp, context(page, campaign). Backfill via backend save table.
2. Saved Items Retention Lift — Retention rate (D7/D30) of users who used saved vs matched control.
   Instrumentation: join auth + save table; flag users with first_save_date for cohort analysis.

Leading indicators (engagement funnel)
3) Save Conversion Rate — saves per view of save-eligible content (saves / eligible_views).
Instrumentation: events: "view_item" (item_id, eligible_save:boolean), "save_item".
4) Time-to-first-save — median time from account creation/session start to first save.
Instrumentation: compute from user.created_at and first "save_item" timestamp in backend.
5) Revisit Rate of Saved Items — % of users who open saved-items list within 7 days.
Instrumentation: event "open_saved_list" with saved_count, session_id.
6) Share/Purchase from Saved — % of saves that lead to downstream conversion (share, purchase).
Instrumentation: attribute downstream events ("purchase","share") to prior save via item_id and user_id.

Observation period & rollout

• Minimum: 4 weeks to capture short-term adoption + 28-day behavior (use D7/D28 retention). For retention lift include 8–12 weeks if measuring purchases.
• Rollout scope: start with 5–10% randomized A/B test (treatment vs control) across representative user segments to measure causal impact, then expand to 25–50% before full launch.

Notes: include user deduplication, GDPR consent flags in events, and monitor instrumentation quality with an event-validity dashboard during rollout.

Approach — formula and calculation:
Use a normal-approximation power formula for two proportions. Let p1=0.05 (baseline), p2=0.058 (target = +0.8 pp), d = p2 − p1 = 0.008. For two-sided α=0.05, Zα/2=1.96; for power 0.8, Zβ≈0.842. One common formula for per-group sample size n is:

n = [ (Zα/2 * sqrt(2p̄(1−p̄)) + Zβ * sqrt(p1*(1−p1) + p2*(1−p2)))^2 ] / d^2

where p̄ = (p1 + p2)/2 = 0.054.

Plugging values:
sqrt(2p̄(1−p̄)) ≈ 0.3196; sqrt(p1*(1−p1)+p2*(1−p2)) ≈ 0.3196.
Numerator ≈ (1.960.3196 + 0.8420.3196)^2 ≈ 0.8019.
d^2 = 0.000064.
So n ≈ 0.8019 / 0.000064 ≈ 12,529 per group → total ≈ 25,058 users.

Tools: verify with statsmodels (Python: statsmodels.stats.power.NormalIndPower) or R’s power.prop.test, Excel or online A/B calculators.

Assumptions to validate before launch:

• Independence and random assignment (no cross-group contamination).
• Metric correctly defined and tracked (accurate conversion event).
• Baseline conversion estimate is stable and representative.
• No major time-varying confounders (seasonality, marketing campaigns).
• SUTVA holds (no interference between units).
• Allocation ratio = 1:1 (adjust formula if not).
• Test duration long enough to capture user behavior and to reach calculated sample.
• Choice of one- vs two-sided test aligned with business hypothesis.

If any assumption fails, re-estimate sample size (e.g., higher variance → larger n) or redesign the experiment.

Approach (one-line): estimate incremental annual revenue = #reactivated_users × ARPA − reactivation_costs; convert to ROI and show sensitivity & risk-adjusted figures.

Back-of-envelope calculation (formula + example):

• Inputs: total_users (T), active_users (A), ARPA (R), reactivation_cost_per_user (C), target conversion of dormant = 10%.
• Dormant = T − A. Reactivated = 0.10 × Dormant.
• Incremental_revenue = Reactivated × R
• Total_reactivation_cost = Reactivated × C
• Net_uplift = Incremental_revenue − Total_reactivation_cost
• ROI = Net_uplift / Total_reactivation_cost

Example numbers: T=1,000,000; A=200,000 → Dormant=800,000 → Reactivated=80,000.
If R=$30/yr, C=$5 → Revenue = 80,000×30=$2.4M; Cost=80,000×5=$400k; Net=$2.0M; ROI=500%.

Key assumptions (call out & justify):

• ARPA applies equally to reactivated users (assume no immediate churn or lower spend).
• Reactivated users become “active” for a full year.
• Reactivation cost is fully allocated per user (marketing + incentives).
• No cannibalization of current active users’ spend.

Sensitivity analysis (what to vary):

• Conversion rate (run scenarios at 5%, 10%, 20%)
• ARPA (±20% to reflect lower engagement)
• Reactivation cost (±50% for different channels)
  Present as a small table or tornado chart showing Net_uplift under each scenario; highlight break-even cost given assumed ARPA.

Risk-adjusted ROI & presentation to stakeholders:

• Show three cases: conservative (5% conv, ARPA−20%), base (10%, ARPA), aggressive (20%, ARPA+10%). Report Net_uplift, payback period, and ROI.
• Adjust for probability: assign likelihoods (e.g., 25%/50%/25%) to compute expected net uplift.
• Include qualitative risks: data quality, attribution, reactivation short-lived, regulatory limits.
• Recommend pilot: run a segmented A/B test (n≈10–50k) to measure conversion and short-term ARPA; present projected scaling with confidence intervals and suggested go/no-go thresholds.

Deliverable format: one-page dashboard with inputs, scenario table, sensitivity chart, break-even analysis, expected ROI and recommended pilot plan with metrics to validate assumptions.