Estimation and Quantitative Reasoning Questions

Clarify assumptions:

• Monthly retention r applies to probability a user is active in month m given active now (month 1 revenue is guaranteed).
• 12-month PV per user = sum_{m=1..12} ARPU * r^{m-1} / (1+d)^{m-1} = ARPU * [1 - (r/(1+d))^{12}] / [1 - r/(1+d)].
• Inputs: users = 500,000; baseline r0 = 0.75; lift = +1.5ppt → r1 = 0.765; ARPU = $10; monthly discount d = 0.005.

Step-by-step calculation:

1. Compute q = r/(1+d).
   • Baseline q0 = 0.75 / 1.005 = 0.7462687
   • Improved q1 = 0.765 / 1.005 = 0.7611940
2. PV per user over 12 months = 10 * (1 - q^12) / (1 - q).
   • Baseline PV0 = 10 * (1 - 0.7462687^12) / (1 - 0.7462687) ≈ $38.22
   • Improved PV1 = 10 * (1 - 0.7611940^12) / (1 - 0.7611940) ≈ $40.27
3. Incremental LTV per user = PV1 - PV0 ≈ $2.05
4. Incremental 12-month revenue for cohort = 2.05 * 500,000 ≈ $1,025,000

Sensitivity analysis (±0.5ppt retention lift):

• Lift = +1.0ppt (r = 0.76): q = 0.76/1.005 = 0.755; PV ≈ $39.41; incremental/user = $1.19; cohort uplift ≈ $595,000.
• Lift = +2.0ppt (r = 0.77): q = 0.77/1.005 = 0.76517; PV ≈ $40.88; incremental/user = $2.66; cohort uplift ≈ $1,330,000.

Notes and caveats:

• Assumes retention change is uniform across months and users, no behavioral feedback loops, and ARPU unchanged.
• If month-1 retention or ARPU also changes, include those adjustments.
• For decision-making, run cohort-level simulations and test sensitivity to discount rate, churn timing, and heterogeneity across user segments.

Approach (summary)

• Expected impact = (reduction in churn probability p) × N users × mean LTV μ.
• For CI, model uncertainty from two sources: who is retained (binomial) and per-user LTV variability. Use an analytic approximation (delta/normal) and optionally validate with bootstrap.

Point estimate (numbers)

• N = 200,000; p = 0.02; μ = $300 → Expected retained = N·p = 4,000 users.
• Expected monetary impact = 4,000 × $300 = $1,200,000.

Analytic 95% CI (derivation & compute)

• Let impact I = sum_{i in retained} LTV_i. Approximate with normal:
  E[I] = N p μ
  Var[I] = N p σ^2 + μ^2 N p(1−p)
  • first term: variability of LTV among retained
  • second term: variability in number retained (binomial) times squared mean LTV
• Plug numbers: σ = $50
  N p = 4,000
  Var[I] = 4,000×50^2 + 300^2×4,000×0.98
  = 10,000,000 + 352,800,000 = 362,800,000
  SD[I] ≈ sqrt(362,800,000) ≈ $19,050
  95% CI ≈ $1,200,000 ± 1.96×19,050 ≈ [$1,162,660, $1,237,340]

Alternative / validation

• Bootstrap: resample user-level pairs (churn outcome / LTV) or simulate:
  1. sample retained count K ~ Binomial(N, p̂) (or use empirical distribution of p̂ from A/B)
  2. sample K LTVs from empirical LTV distribution
  3. compute I; repeat 10k+ times to get percentile CI.
• Delta method: same analytic variance formula derived above; equivalent to normal approx if N large.

Assumptions & caveats

• Independence between who is retained and per-user LTV (we assumed retention effect is uniform across users). If treatment effect varies by LTV, impact changes.
• p is average marginal reduction; if p̂ has estimation uncertainty (from experiment), incorporate its SE into Var[I] (add term (N μ)^2 Var(p̂)).
• LTV mean and SD assumed known or estimated from historical data; censored LTVs or skewness may invalidate normal approx — use bootstrap then.
• Treatment may change future behavior beyond retention; exclude unless modeled.

Presentation for CFO

• One slide: headline "$1.20M expected impact (95% CI $1.16M–$1.24M)", one-liner assumptions, brief methodology (analytic + bootstrap validation), and sensitivity scenarios (e.g., effect heterogeneity: if p=1.5% → $0.9M; if p=2.5% → $1.5M). Include short recommendation: validate with subgroup analysis and report estimation uncertainty from the A/B test.

Step-by-step plan

1. Define model & baseline

• Build deterministic NPV function NPV(conversion_uplift, adoption_rate, cost_per_user) that maps inputs to discounted cashflows over horizon. Document assumptions (discount rate, churn, time window).

2. Specify plausible ranges and distributions

• For each input, set realistic low/likely/high values (e.g., conversion uplift: -1% to 5%), plus best-fit distributions (normal/lognormal/beta) based on historical A/B tests or expert elicitation.

3. One-way sensitivity (for tornado)

• For each input, vary it across its low and high while holding others at baseline. Record resulting NPVs and compute delta_NPV = NPV(high)-NPV(low) or max absolute change from baseline.
• Rank inputs by absolute delta_NPV.

4. Generate tornado chart (Python example)

import numpy as np
import matplotlib.pyplot as plt

params = ['conversion_uplift','adoption_rate','cost_per_user']
low = [ -0.01, 0.05, 5.0 ]
high= [ 0.05,  0.25, 15.0 ]
baseline = [0.02,0.12,10.0]

def NPV(uplift, adopt, cost):
    # placeholder; replace with model
    return 1000 + 1e5*uplift + 5e3*adopt - 1e4*(cost/10)

npv_low  = [NPV(low[i] , baseline[1], baseline[2]) if i==0 else
            NPV(baseline[0], low[i], baseline[2]) if i==1 else
            NPV(baseline[0], baseline[1], low[i]) for i in range(3)]
npv_high = [NPV(high[i], baseline[1], baseline[2]) if i==0 else
            NPV(baseline[0], high[i], baseline[2]) if i==1 else
            NPV(baseline[0], baseline[1], high[i]) for i in range(3)]
deltas = np.abs(np.array(npv_high)-np.array(npv_low))
order = np.argsort(deltas)[::-1]

plt.barh(np.array(params)[order], deltas[order])
plt.xlabel('Absolute NPV change')
plt.title('Tornado chart')
plt.show()

5. Global sensitivity / uncertainty quantification

• Run Monte Carlo (10k+) sampling from joint distributions to get NPV distribution. Use PRCC or Sobol indices for quantitative driver importance beyond one-way effects.

6. Identify key drivers

• From tornado (one-way) and Sobol/PRCC (global), pick drivers causing greatest NPV variance and highest tail risk (e.g., conversion uplift and adoption).

7. Form conservative recommendation

• Use percentile-based decision rules: compute 5th percentile NPV and probability NPV<0. If 5th percentile positive and P(NPV<0) < threshold (e.g., 10%), recommend rollout; otherwise recommend pilot/mitigations.
• Propose mitigations: reduce cost_per_user via negotiated pricing, staged rollout to validate conversion uplift, guardrails (kill if uplift< X% after Y users).
• Communicate: present tornado + NPV distribution, highlight assumptions, sensitivity of recommendation to key drivers, and an action plan tying metrics, experiments, and stop conditions.

Why this works

• Tornado shows marginal impact; global methods quantify joint uncertainty. Percentile-based, conservative thresholds align stakeholder risk appetite and provide operational mitigation steps.

Assumptions:

• You acquire a constant number n of new paying users each month (month 1..12).
• Each new user starts paying immediately and stays for all remaining months (no churn).
• Monthly price = $10, hosting cost = $1 → gross margin per user per month = $9.
• We want cumulative net contribution over 12 months to cover the $50,000 build cost.

Approach and calculations:

• A user acquired in month m (m = 1..12) contributes $9 per month for (13 − m) months (remaining months in the year).
• Total contribution over 12 months from n users each month:
  Total = n * $9 * sum_{m=1}^{12} (13 − m) = n * $9 * sum_{k=1}^{12} k
• Sum 1..12 = 78, so:
  Total = n * 9 * 78 = n * 702

Set Total ≥ $50,000 to break even:

• n * 702 ≥ 50,000
• n ≥ 50,000 / 702 ≈ 71.23

Answer:

• You must acquire at least 72 paying users per month (rounding up) to break even within 12 months under these assumptions.

Notes / sensitivity:

• If all users were acquired up front (month 1), break-even would be 50,000 / (9 * 12) ≈ 463 users.
• This ignores churn, acquisition costs (CAC), discounts, or variable retention—incorporating those would raise required acquisition.

Start by framing the decision as a Bayesian decision problem: two actions A = {buy data, don’t buy data} and downstream decisions D (e.g., build model version 1 or 2, deploy or not) whose expected utility depends on unknown parameter θ (true predictive signal/value from data). Prior p(θ) encodes current belief about how much the new data will improve model performance (e.g., lift in expected profit per user).

Compute baseline expected utility without buying:
EU_no = ∫ U(D*(θ) | θ) p(θ) dθ
where D*(θ) is the optimal downstream decision given no new information.

If you buy data at cost C, you will observe data x and update to posterior p(θ|x), then choose D to maximize posterior expected utility. Expected utility with purchase:
EU_buy = ∫ [ max_D ∫ U(D|θ) p(θ|x) dθ ] p(x) dx − C

Expected Value of Sample Information (EVSI) = EU_buy − EU_no.
Compare EVSI to cost C (or simply compute EVSI and check EVSI > 0).

Concrete decision-tree example (numeric):

• Business: deploying a model yields profit = $100K if true lift θ > 0.02, else $0.
• Prior: P(θ>0.02)=0.3.
• Without data: choose not to deploy → EU_no = $0.
• Data purchase: cost C = $10K. Data has sensitivity such that posterior will indicate θ>0.02 with probability 0.6 when true θ>0.02, and false positive rate 0.1.
  Compute probability data indicates "good" overall: 0.30.6 + 0.70.1 = 0.25. If data says "good" we deploy (expected profit $100K), else don't.
  EU_buy = 0.25*$100K − $10K = $25K − $10K = $15K. EVSI = $15K > 0 so buy data justified.

Sensitivity analysis:

• Vary prior P(θ>0.02): if prior falls <0.15 then EU_buy drops below cost.
• Vary cost C or data accuracy (true/false positive rates). Present threshold plots: break-even cost C* = (prob_signal_post * profit) − EU_no.
  Practically:
• Quantify prior via historical A/B tests or elicitation from stakeholders.
• Estimate data diagnostic performance (predictive value) from pilot or literature.
• Propagate uncertainty using Monte Carlo over priors and data operating characteristics to produce distribution of EVSI and probability EVSI>0.
  This Bayesian EVSI approach makes the trade-off explicit, lets you test robustness, and gives a clear monetary criterion for whether expensive data acquisition is justified.