Business Case Development and ROI Analysis Questions

Direct answer

Treat the benefit as a curve, not a number. Estimate the full run-rate benefit (what the improvement is worth per month once everyone is using it), multiply it each month by an adoption share that climbs from 0% toward its ceiling, and charge the costs on their real dates, which usually start before any benefit does. Then justify the curve with evidence (pilot data, rollout schedule, comparable launches), show how NPV and payback move when the curve is slower, and track the leading indicators so the ramp can be checked after approval.

Terms used below. Run-rate means the steady monthly amount once everything is fully in use. Loaded hourly cost is what an hour of an employee really costs the company (pay plus benefits, taxes and overhead share). Leading indicators are early measures (for example weekly active users) that move before the money does. NPV (net present value) is the sum of each future cash flow divided by (1 + rate) raised to the number of periods until it arrives, so later money counts for less. The discount rate is that rate (the return you could earn elsewhere, set by finance). IRR (internal rate of return) is the discount rate at which NPV is exactly zero. Payback is the first period in which cumulative cash turns non-negative. A monthly series needs a monthly rate: use (1.10)^(1/12) - 1, not 10% / 12, so twelve months of compounding gives exactly the annual 10%.

How to build the ramp

1. Full run-rate benefit. Hours saved per person x people x loaded hourly cost x the share of saved time that becomes real cash or real extra output. Freed time is only a cash saving if the capacity is redeployed to other paid work, backfilled less (a departing person is replaced by fewer new hires), or sold; say which.
2. Adoption curve = rollout x usage x proficiency. Rollout is the share of teams that have the tool (from the deployment schedule). Usage is the share of those people who use it in a given month. Proficiency is how much of the full gain a typical user gets while still learning (a new user may get half the time saving that an expert gets). For example, 80% rollout x 70% usage x 60% proficiency is 33.6% of the full benefit. Multiply them; each has its own evidence.
3. Anchor points, then interpolate. Pick a few anchors you can defend (end of year 1: 30%, year 2: 65%, year 3: 90%) and interpolate monthly (fill the months in between with evenly spaced steps from one anchor to the next). Anchors are easier for a reviewer to challenge than a formula. An S-shaped curve (slow start, fast middle, flattening) is usually more realistic than a straight line, but a straight line between honest anchors is fine.
4. Costs on their own dates. Upfront spend lands in month 0 and run costs begin immediately; neither waits for adoption.
5. Justify the shape. In order of strength: your own pilot cohort (monthly usage of the first teams), earlier rollouts at your company, then outside benchmarks. Cap the ceiling below 100% (here 90%) because not every role gets full value. State what would make it faster or slower (training, executive mandate, integration with existing tools).

Worked example (illustrative inputs)

There are two examples. Part A is the main one: a monthly 36-month model of a tool whose benefit ramps up by anchors. Part B is a second, simpler yearly example that tests how sensitive a savings ramp is to slower adoption; it uses year-end cash flows, not monthly ones. Hand check for Part A: month 1 adoption is 30% x 1/12 = 2.5%, so benefit is $80,000 x 2.5% = $2,000, less $10,000 to run = -$8,000. Each month is then discounted by (1 + 0.0080)^month, where 0.0080 is the monthly rate (1.10^(1/12) - 1 = 0.797%); the headline NPV is the sum of all 36 of these plus the -$600,000 at month 0.

A tool is worth $80,000 per month at full run-rate, costs $600,000 upfront and $10,000 per month to run, at a 10% annual rate over 36 months.

# Part A: 36 monthly cash flows for a productivity gain that ramps up
full_monthly = 80_000.0   # $/month once every team has adopted
upfront = 600_000.0       # $ spent in month 0
run_cost = 10_000.0       # $/month licences and support
annual_rate = 0.10
r_m = (1 + annual_rate) ** (1 / 12) - 1     # equivalent monthly rate
anchors = {0: 0.0, 12: 0.30, 24: 0.65, 36: 0.90}   # share of full benefit reached

def adoption(m):
    ks = sorted(anchors)
    for a, b in zip(ks, ks[1:]):
        if a <= m <= b:
            return anchors[a] + (anchors[b] - anchors[a]) * (m - a) / (b - a)

def npv(flows, rate):           # flows[0] happens now (t = 0)
    return sum(cf / (1 + rate) ** t for t, cf in enumerate(flows))

def payback_index(flows):       # first period where cumulative cash >= 0
    cum = 0.0
    for t, cf in enumerate(flows):
        cum += cf
        if cum >= 0:
            return t
    return None

ramp = [-upfront] + [full_monthly * adoption(m) - run_cost for m in range(1, 37)]
flat = [-upfront] + [full_monthly * 0.90 - run_cost for m in range(1, 37)]  # day-one 90%
for m in (1, 6, 12, 18, 24, 30, 36):
    print(f"month {m:2d}: adoption {adoption(m):5.1%}  net cash ${ramp[m]:>9,.0f}")
print(f"ramped:   NPV ${npv(ramp, r_m):,.0f}, payback month {payback_index(ramp)}")
print(f"day-one:  NPV ${npv(flat, r_m):,.0f}, payback month {payback_index(flat)}")

# Part B (second example, yearly): five-year linear savings ramp, 10% to 40% of an addressable cost base
# (addressable cost base = the annual cost the tool can actually reduce)
def irr(flows):
    lo, hi = -0.9, 10.0
    for _ in range(200):
        mid = (lo + hi) / 2
        if npv(flows, mid) > 0:
            lo = mid
        else:
            hi = mid
    return mid

base_cost, invest, run_yr, rate = 1_000_000.0, 450_000.0, 50_000.0, 0.10
def case(stretch):                 # stretch 1.0 = plan, 0.8 = 20% slower adoption
    flows = [-invest]
    for t in range(1, 6):
        # savings share rises 7.5 points a year from a 2.5% start: 10%, 17.5%, 25%, ... 40% at stretch 1
        pct = 0.025 + 0.075 * (stretch * t)     # 10% at t=1, 40% at t=5 when stretch=1
        flows.append(base_cost * pct - run_yr)
    return flows
for name, s in (("plan", 1.0), ("20% slower", 0.8)):
    f = case(s)
    # break-even = first year cumulative cash >= 0; discounted break-even = same, but each year's cash
    # is first discounted to today's money
    cum, disc, be, dbe = 0.0, 0.0, None, None
    for t, cf in enumerate(f):
        cum += cf
        disc += cf / (1 + rate) ** t
        if be is None and cum >= 0: be = t
        if dbe is None and disc >= 0: dbe = t
    print(name, [round(x) for x in f])
    print(f"  NPV ${npv(f, rate):,.0f}  IRR {irr(f):.1%}  break-even year {be}  discounted break-even year {dbe}")

Output:

month  1: adoption  2.5%  net cash $   -8,000
month  6: adoption 15.0%  net cash $    2,000
month 12: adoption 30.0%  net cash $   14,000
month 18: adoption 47.5%  net cash $   28,000
month 24: adoption 65.0%  net cash $   42,000
month 30: adoption 77.5%  net cash $   52,000
month 36: adoption 90.0%  net cash $   62,000
ramped:   NPV $228,345, payback month 29
day-one:  NPV $1,333,560, payback month 10
plan [-450000, 50000, 125000, 200000, 275000, 350000]
  NPV $254,174  IRR 25.0%  break-even year 4  discounted break-even year 4
20% slower [-450000, 35000, 95000, 155000, 215000, 275000]
  NPV $94,386  IRR 16.0%  break-even year 4  discounted break-even year 5

Reading it: adoption is only 30% at month 12, so net cash in month 1 is negative and turns positive around month 6. The ramped case gives NPV $228,345 and payback in month 29. A model that assumed 90% of the benefit from month 1 would show NPV $1,333,560 and payback in month 10, overstating NPV by about $1.1M and understating the payback time by 19 months. The ramp, not the headline benefit, is what the approval decision turns on.

Five-year linear ramp. For a savings rate that climbs linearly from 10% of an addressable cost base in year 1 to 40% in year 5 (base $1,000,000 a year, $450,000 upfront, $50,000 a year to run, 10% rate, cash at year end), the savings share is 10%, 17.5%, 25%, 32.5% and 40% of $1,000,000 (the formula 0.025 + 0.075 x t, which is 10% at t = 1 and 40% at t = 5), less $50,000 a year to run, so the cash flows are -$450,000, then $50,000, $125,000, $200,000, $275,000, $350,000. Discounted at 10% these are -$450,000, $45,455, $103,306, $150,263, $187,829 and $217,322, which add to NPV $254,174, IRR is 25.0%, and cumulative cash turns positive in year 4 (-$75,000 at the end of year 3, +$200,000 at the end of year 4). If adoption is 20% slower (the ramp reaches in 5 years only what the plan reaches in 4; the savings share is 8.5%, 14.5%, 20.5%, 26.5% and 32.5%), the savings rates are lower in every year, NPV falls to $94,386 (about 63% lower), IRR to 16.0%, and the discounted break-even moves from year 4 to year 5. The case still clears a 10% rate but with much less cushion, and adoption pace is the assumption it is most sensitive to because the costs are fixed.

Trade-offs and pitfalls

• Ramp the benefit but not the costs: costs start on their real dates, so the early months are negative. Also model the learning dip (productivity can fall slightly during onboarding); a benefit curve that rises smoothly from month 1 and ignores that dip flatters the first year.
• An S-shaped curve is one that starts slowly, speeds up in the middle and flattens near the ceiling. Do not let the ceiling creep: the curve's end point matters as much as its slope.
• Present base, slower and faster ramps together, and name the leading indicator (monthly active users, share of work done in the tool) that tells you in month 6 which one you are on.
• Dividing the annual rate by 12 gives a slightly different (wrong) monthly rate; use the equivalent monthly rate.

Direct answer

You value a prevented event as an expected loss avoided: how often the bad event would have happened without the investment, times how much each one would have cost, times how much the investment reduces either. Then you subtract the new costs the investment creates (such as wrongly blocked good customers), and report the result as a range with its evidence, not a single figure. The dollar number is defensible when every term has a source, the counterfactual (what would have happened otherwise) is measured rather than assumed, and the rare large events are handled separately from the frequent small ones.

The method, step by step

1. Define the event precisely: "successful card fraud with a chargeback" (a chargeback is when a cardholder disputes a payment and the bank takes the money back from the merchant, usually adding a fee), or "an unplanned outage longer than 30 minutes". A vague event cannot be counted.
2. Frequency without the investment: use the customer's own history (24 months of chargebacks, incident logs), and industry loss data only to cross-check it. Note the trend, since a rising base rate (how often the event happens when nothing is done about it) makes the benefit bigger and a one-off spike makes it misleading.
3. Severity per event: direct loss (goods, chargeback fees, service credits) plus response cost (investigation hours) and, where evidenced, churn or penalties. Leave out reputation and brand effects unless there is data; list them as unquantified.
4. Effect of the investment: the measured change in catch rate (the share of fraud attempts the system stops) or in incident frequency or duration. Get this from shadow-mode testing (the model scores live traffic without acting), a holdout group (a slice of traffic deliberately left on the old system so you can compare), or the incident history of teams that already adopted the practice. A vendor's brochure number is a claim, not evidence.
5. Subtract the side effects: false positives that block good customers, extra operating cost, and the cost of the system itself.
6. Handle the tail separately: the tail is the rare, very large events at the far end of the loss range, and expected value hides them. For example, a USD 5 million breach that happens about once in 20 years has an expected loss of USD 250,000 a year, yet in 19 of 20 years nothing happens, so counting it would make a lumpy benefit look steady. Report the frequent-event benefit as the base case, and list the catastrophic scenario (a major breach, a multi-day outage) as upside that is not counted in the headline.
7. Show a range: ranges for each input, combined, then P10 and P90 (the 10th and 90th percentile of the outcomes).
8. Avoid double counting: the same avoided incident cannot be claimed by the fraud team, the support team and Finance.

Worked example 1: fraud detection (illustrative figures)

A company processes 4 million card payments a year. From 24 months of chargeback history, 0.12% are fraud attempts (4,800 a year); each successful fraud costs about USD 180. The current system catches 55% of attempts; a new model catches 70% in shadow testing. The new model also wrongly declines about 2,000 more good orders a year, each costing USD 35 of lost margin.

import random
random.seed(7)

TRANSACTIONS = 4_000_000        # card payments per year
ATTEMPT_RATE = 0.0012           # share of payments that are fraud attempts (from 24 months of chargeback history)
attempts = TRANSACTIONS * ATTEMPT_RATE          # 4,800 per year
LOSS_PER_EVENT = 180            # USD lost per successful fraud (goods + chargeback fee)
CATCH_OLD, CATCH_NEW = 0.55, 0.70
FALSE_DECLINES = 2_000          # extra good customers wrongly declined per year
MARGIN_LOST_PER_DECLINE = 35    # USD margin lost per wrongly declined order

prevented = attempts * (CATCH_NEW - CATCH_OLD)
gross = prevented * LOSS_PER_EVENT
cost_of_false = FALSE_DECLINES * MARGIN_LOST_PER_DECLINE
print(f"Fraud attempts per year: {attempts:,.0f}")
print(f"Extra frauds stopped per year: {prevented:,.0f}")
print(f"Gross loss avoided: ${gross:,.0f}; false-decline cost: ${cost_of_false:,.0f}; net: ${gross - cost_of_false:,.0f}")

# Range instead of a point: a Monte Carlo simulation. Each of 20,000 runs picks a random value for every
# uncertain input from a triangular range (low, high, most likely; values near the most likely come up most often)
# and computes the net benefit; sorting the 20,000 results lets us read off P10 (10% of runs fall below it),
# P50 (the median) and P90.
def draw():
    catch_uplift = random.triangular(0.05, 0.20, 0.15)   # new catch rate minus old catch rate
    loss = random.triangular(120, 260, 180)
    attempt_rate = random.triangular(0.0008, 0.0016, 0.0012)
    declines = random.triangular(1_000, 4_000, 2_000)
    return (TRANSACTIONS * attempt_rate * catch_uplift * loss
            - declines * MARGIN_LOST_PER_DECLINE)

runs = sorted(draw() for _ in range(20_000))
p = lambda q: runs[int(q * len(runs))]
money = lambda x: f"-${-x:,.0f}" if x < 0 else f"${x:,.0f}"
print("Net benefit per year, P10 / P50 / P90:", money(p(0.10)), "/", money(p(0.50)), "/", money(p(0.90)))
print(f"Share of runs below zero: {sum(r < 0 for r in runs) / len(runs):.1%}")

# Reliability version: expected annual cost of downtime = events per year x hours per event x cost per hour
before = 6 * 2.5 * 20_000         # 6 incidents x 2.5 h x $20,000 per hour
after  = 3 * 1.5 * 20_000         # 3 incidents x 1.5 h x $20,000 per hour
print(f"Downtime cost before ${before:,.0f}, after ${after:,.0f}, avoided ${before - after:,.0f} per year")

Output:

Fraud attempts per year: 4,800
Extra frauds stopped per year: 720
Gross loss avoided: $129,600; false-decline cost: $70,000; net: $59,600
Net benefit per year, P10 / P50 / P90: -$17,108 / $35,605 / $94,333
Share of runs below zero: 19.8%
Downtime cost before $300,000, after $90,000, avoided $210,000 per year

Reading it: the model stops 720 extra frauds a year, avoiding USD 129,600, but the false declines cost USD 70,000, so the point estimate of the net benefit is USD 59,600. The honest headline is the range from the simulation: roughly -17,000 to +94,000 (P10 to P90), median about 35,600, with about a fifth of the draws below zero. (The most-likely inputs give 59,600 while the median is lower, because the ranges are lopsided: the average of the catch-rate uplift range is 0.133, not its most-likely 0.15, and the average of the false-decline range is about 2,333, not 2,000. Using those averages, 4,800 x 0.133 x 187 - 2,333 x 35 is roughly 38,000, close to the simulated median of 35,600.) That range changes the conversation: this is a decision worth making if we also bring the false-decline cost down, not a self-evident win.

Worked example 2: a reliability investment

The event is an unplanned outage costing USD 20,000 per hour. Before: 6 incidents a year averaging 2.5 hours. After (from comparable teams' incident records): 3 incidents averaging 1.5 hours. Avoided cost is 300,000 minus 90,000, which is USD 210,000 a year (the last line of the output). Both frequency and duration changed, so the claim needs two pieces of evidence, and the cost per hour needs Finance to confirm it, because revenue per hour is not the same as lost profit per hour.

Trade-offs and pitfalls

• Assuming the base rate holds: attackers adapt, so the baseline fraud rate and the catch rate both drift; re-measure yearly.
• Counting benefits that happen only if the rare disaster occurs as certain: report the tail separately.
• Valuing "an incident avoided" without asking whether it would have happened: if last year's incident count was an outlier, regression to the mean (an unusually bad year tends to be followed by a more normal one even if you do nothing) makes the benefit smaller.
• Omitting the side effects of the control: a fraud filter that blocks good customers has its own cost.
• Using a single number: when the range includes zero, say so and say what evidence would narrow it.

Direct answer

Change the drivers the model actually depends on, each within a range you can defend from evidence, not a blanket plus or minus 20% on the answer. For a churn-reduction model (churn means customers leaving) the drivers are how many of the leavers the model catches, how many of those the retention offer saves, how fast the business adopts the process, and what a saved customer is worth. Set the bounds from a holdout test, the model's backtest, past retention campaigns and outside benchmarks, then give the CFO (chief financial officer) the three cases, the break-even point and the single assumption that matters most. On the inputs below: pessimistic NPV is -$965,422 with no payback, base is $449,115 with payback in month 16, optimistic is $2,981,905 with payback in month 5, and the base case breaks even at a save rate of 11.7%.

Terms: NPV (net present value) discounts future cash to today at the discount rate (10% a year, the annual markdown applied to later money; 36 months); payback is the month cumulative cash turns non-negative; save rate is the share of contacted at-risk customers who stay because of the offer; capture rate is the share of all churners the flagged list contains. A backtest with a time split means training the model on older data and checking its predictions against what happened afterwards, so it is not graded on data it has seen. A randomised holdout is a randomly chosen group of at-risk customers who get no offer, to compare against those who do. A simulation here means drawing many random combinations of the drivers within their ranges and looking at the spread of NPV. The 10th percentile is the value that only 1 outcome in 10 falls below. Proxies are stand-in data from a similar situation. Deflection is the share of support tickets resolved without an agent.

What changes between cases, and how to bound it

Fixed in all cases: 200,000 customers, 12% annual churn (24,000 leavers), 20,000 customers contacted a year at $20 each, $350,000 to build, $10,000 a month to run. Keep the discount rate constant so the cases differ for operational reasons; show a rate sensitivity separately.

Hand trace of the base case (the main example): at full operation, benefit is 1,440 saved a year / 12 x $600 = $72,000 a month and the cost is 20,000 / 12 x $20 = $33,333 plus $10,000 to run, so net cash is $28,667 a month. In the 6-month ramp, month k runs at k/6 of full operation, so net cash is -$3,556 in month 1, $2,889 in month 2, $9,333, $15,778, $22,222 in months 3 to 5 and $28,667 from month 6. Cumulative cash after month 5 is -$350,000 + $46,667 = -$303,333, and it takes 303,333 / 28,667 = 10.6 more months, so cash turns positive in month 16. NPV is these monthly flows divided by the monthly discount factor, 1.10^(1/12) = 1.00797 per month, summed over the 36 months, which the code does.

RATE = 0.10
def npv(flows):                    # flows[0] = month 0
    r = (1 + RATE) ** (1 / 12) - 1
    return sum(cf / (1 + r) ** t for t, cf in enumerate(flows))

cases = {   # churners captured by the flagged list, save rate of the offer, adoption ramp (months to full use)
    "pessimistic": dict(capture=0.30, save=0.08, ramp=12, value=450.0),
    "base":        dict(capture=0.40, save=0.15, ramp=6,  value=600.0),
    "optimistic":  dict(capture=0.50, save=0.22, ramp=3,  value=700.0),
}
CUSTOMERS, CHURN = 200_000, 0.12            # 24,000 churners a year
FLAGGED_PER_YEAR = 20_000
OFFER_COST = 20.0                           # $ per flagged customer contacted
BUILD, RUN_MONTH = 350_000.0, 10_000.0      # month-0 build, $/month run cost

def monthly_flows(c):
    saved_per_year = CUSTOMERS * CHURN * c["capture"] * c["save"]
    flows = [-BUILD]
    for m in range(1, 37):
        use = min(1.0, m / c["ramp"])        # share of full operation reached this month
        benefit = use * saved_per_year / 12 * c["value"]
        cost = use * FLAGGED_PER_YEAR / 12 * OFFER_COST + RUN_MONTH
        flows.append(benefit - cost)
    return saved_per_year, flows

for name, c in cases.items():
    saved, flows = monthly_flows(c)
    cum, pb = 0.0, None
    for t, cf in enumerate(flows):
        cum += cf
        if cum >= 0 and pb is None: pb = t
    print(f"{name:<12} saved/yr {saved:>6,.0f}  NPV(36m) ${npv(flows):>10,.0f}  payback month {pb}")

# Break-even save rate in the base case (everything else held at base)
lo, hi = 0.0, 0.5
for _ in range(60):
    mid = (lo + hi) / 2
    _, fl = monthly_flows(dict(cases["base"], save=mid))
    lo, hi = (mid, hi) if npv(fl) < 0 else (lo, mid)
print(f"base case breaks even at a save rate of {mid:.1%}")

# Side example (not part of the churn model): support automation, agent hours released per year
tickets, minutes, rate_hr = 50_000, 12, 38.0
for name, deflect in (("pessimistic", 0.15), ("likely", 0.25), ("optimistic", 0.35)):
    hours = tickets * deflect * minutes / 60
    print(f"{name:<12} deflection {deflect:.0%}: {hours:,.0f} agent hours = ${hours * rate_hr:,.0f} of capacity per year")

Output:

pessimistic  saved/yr    576  NPV(36m) $  -965,422  payback month None
base         saved/yr  1,440  NPV(36m) $   449,115  payback month 16
optimistic   saved/yr  2,640  NPV(36m) $ 2,981,905  payback month 5
base case breaks even at a save rate of 11.7%
pessimistic  deflection 15%: 1,500 agent hours = $57,000 of capacity per year
likely       deflection 25%: 2,500 agent hours = $95,000 of capacity per year
optimistic   deflection 35%: 3,500 agent hours = $133,000 of capacity per year

Check by hand: leavers saved a year = 24,000 x capture x save: 24,000 x 0.30 x 0.08 = 576, 24,000 x 0.40 x 0.15 = 1,440 and 24,000 x 0.50 x 0.22 = 2,640. The pessimistic case changes every driver at once, so treat it as a stress case, not a 10th percentile (a case that would be beaten 90% of the time); a credible range comes from varying drivers within their evidence, ideally with a simple simulation, and showing which driver moves NPV most. Here the save rate is the one to test first because the break-even is 11.7% against a base of 15%.

Other angles

• Limited history. Use proxies (a comparable product's retention data, public market figures), label them as proxies, widen the ranges, and show break-even (the smallest save rate that justifies the spend) so the CFO can judge plausibility.
• Use the model to scope a pilot. The pessimistic and break-even gap (8% to 11.7%) says what precision the pilot needs: size it so its interval on the save rate is narrower than that gap, then release the full build only if the measured save rate clears break-even.
• Support automation (a secondary example, same shape as the main one). Agent hours released a year are tickets x deflection x handling time: 50,000 tickets x 12 minutes at 15%, 25% and 35% deflection is 1,500, 2,500 and 3,500 hours, worth $57,000, $95,000 and $133,000 of capacity at $38 an hour. Add retraining and tooling cost, and count it as cash only if the hours are redeployed or hiring avoided.

Pitfalls

Presenting a spread that is only the same assumption at three scales; hiding the break-even; mixing an annual rate with monthly flows; and quoting a point estimate in the headline. Lead with the base, show the downside in dollars, and say what evidence would move you.

Direct answer

A p-value is the probability of seeing a result at least as extreme as the one you got if the change actually did nothing; a p-value of 0.04 therefore counts as evidence against a zero effect, not as a 96% chance that the effect is real (strictly: such a result would be rare, about 4 times in 100, if there were no real effect). It does not say the effect is big, or that it will repeat at the size you measured. To turn it into dollars I use the confidence interval (the range of effect sizes consistent with the data), multiply by the number of people who would get the change and the value of what they produce, apply a realisation factor (the share of saved time that becomes real cash; the figure is a judgement agreed with the sponsor, not a measurement), and carry the interval through to a dollar range. Before relying on it I check that the test was valid, that the effect was not inflated by the way it was found, and that it will last.

Worked example (illustrative figures)

An A/B test (a randomised comparison of two versions) gives 1,000 analysts the redesigned dashboard and 1,000 the old one. Productivity is tasks completed per analyst per week, with a standard deviation (typical spread) of 6 tasks; the redesign group completes 0.55 more tasks per week than the control's average of 20. The code computes the standard error, the p-value by the normal approximation, the 95% confidence interval, and then the dollars: 2,000 analysts, 48 working weeks, each task worth USD 30 (half an hour at USD 60 loaded hourly cost, meaning salary plus benefits and overhead per hour). In the code: the standard error (se) is how much the measured difference would wobble from sample to sample, here 6 x sqrt(2/1,000) = 0.268; z is the measured difference in units of standard errors, 0.55 / 0.268 = 2.05; the p-value is read off the normal bell curve at that z (the normal approximation, fine for samples this large); and the 95% confidence interval is the difference plus or minus 1.96 standard errors, 0.55 +/- 0.526.

import math

n = 1_000            # analysts per arm
sd = 6.0             # standard deviation of tasks per analyst per week
diff = 0.55          # treatment minus control, tasks per analyst per week (control mean 20)

se = sd * math.sqrt(2 / n)
z = diff / se
p = math.erfc(abs(z) / math.sqrt(2))          # two-sided p-value, normal approximation
lo, hi = diff - 1.96 * se, diff + 1.96 * se
print(f"standard error {se:.3f}, z {z:.2f}, p {p:.3f}, 95% CI for the gain: {lo:.3f} to {hi:.3f} tasks/week")

ANALYSTS = 2_000          # people who would get the redesign
WEEKS = 48                # working weeks per year
VALUE_PER_TASK = 30       # USD: 0.5 hour x $60 loaded hourly cost
REALISATION = 0.30        # share of freed time that turns into real output or avoided hiring (an assumption to agree with the sponsor)

def dollars(tasks_per_week, realisation):
    return tasks_per_week * ANALYSTS * WEEKS * VALUE_PER_TASK * realisation

print(f"Naive (point estimate, 100% realised): ${dollars(diff, 1.0):,.0f} per year")
print(f"Point estimate, 30% realised:           ${dollars(diff, REALISATION):,.0f} per year")
print(f"Lower CI bound, 30% realised:           ${dollars(lo, REALISATION):,.0f} per year")
print(f"Upper CI bound, 30% realised:           ${dollars(hi, REALISATION):,.0f} per year")

# Winner's-curse shrinkage: halve the observed effect before planning (a deliberately blunt haircut)
print(f"Point estimate halved, 30% realised:    ${dollars(diff * 0.5, REALISATION):,.0f} per year")

# From a model score to dollars: only a randomised holdout measures uplift
contacted = 10_000          # top-scoring decile of 100,000 customers
cost_per_contact = 2.0      # USD
margin_per_order = 80.0     # USD
for uplift in (0.015, 0.03):
    gain = contacted * uplift * margin_per_order
    net = gain - contacted * cost_per_contact
    sign = "-" if net < 0 else ""
    print(f"Measured uplift {uplift:.1%}: margin ${gain:,.0f} vs campaign cost ${contacted * cost_per_contact:,.0f} -> net {sign}${abs(net):,.0f}")

Output:

standard error 0.268, z 2.05, p 0.040, 95% CI for the gain: 0.024 to 1.076 tasks/week
Naive (point estimate, 100% realised): $1,584,000 per year
Point estimate, 30% realised:           $475,200 per year
Lower CI bound, 30% realised:           $20,802 per year
Upper CI bound, 30% realised:           $929,598 per year
Point estimate halved, 30% realised:    $237,600 per year
Measured uplift 1.5%: margin $12,000 vs campaign cost $20,000 -> net -$8,000
Measured uplift 3.0%: margin $24,000 vs campaign cost $20,000 -> net $4,000

Reading it:

• The 30% is an assumption, not a result: it says three in ten freed hours become cash and the rest is absorbed as slack. At 20% the point estimate is $316,800 a year; at 50% it is $792,000, so the factor moves the answer as much as the statistics do, and I would ask the sponsor's manager to defend it.
• The test is "significant" (z of 2.05, p = 0.040), but the 95% interval runs from 0.024 to 1.076 tasks per week. The lower end is nearly zero: the data are compatible with a negligible effect.
• The tempting calculation (point estimate, all of it real) gives USD 1,584,000 a year. That assumes a freed task-minute becomes cash.
• Using a realisation factor of 30% gives USD 475,200 for the point estimate, and a range of USD 20,800 to USD 929,600 across the interval. I would put the range in the case, not the point, and give the conservative end as the planning number.
• Winner's curse (the effect you notice because it crossed the p-value line tends to be overestimated): I would plan on a blunt haircut, such as half the effect, which gives USD 237,600.

Checks before relying on the number

(The first three matter in nearly every case; the others apply when the situation calls for them. Terms: sample ratio is the split of people between the arms compared with the planned split; a guardrail metric is a second measure, such as quality, that must not get worse.)

1. The test was valid: randomisation worked (similar groups at the start), the sample ratio matches the design (no dropped users in one arm), and the metric was fixed before the test.
2. Multiple comparisons (checking many metrics until one looks good): if ten metrics were looked at and this is the one that crossed 0.05, the p-value overstates the evidence.
3. Practical significance: is 0.55 tasks on 20 (2.75%) worth the cost of rolling out and maintaining the redesign? Compare with the cost, not with zero.
4. Duration and novelty: a new design can lift activity for two weeks and fade. Look at the effect by week.
5. Heterogeneity (the effect differing between groups): does the gain come from one team? Targeting the rollout might beat a blanket launch.
6. Does it generalise: were the 2,000 testers representative of the people who will get it?
7. Does the work turn into value: if analysts finish tasks faster but the queue is limited, no extra output is produced. That is what the realisation factor stands for, and the sponsor's manager should agree it.

Related conversions

• From a model metric to dollars: AUC (area under the ROC curve, a ranking-quality score) is not a business outcome. Convert it by scoring customers, splitting into score bins, and measuring the actual repeat-purchase rate in each (a decile is one tenth of customers, ranked by score). Then evaluate the policy "contact the top decile". The only trustworthy number for the benefit is the uplift (the difference in outcomes between contacted and not-contacted customers), measured with a randomised holdout. In the last lines of the output, contacting 10,000 customers at USD 2 each and a USD 80 margin per order loses USD 8,000 at a 1.5-point uplift and gains USD 4,000 at a 3-point uplift. A high score can mean the customer was likely to buy anyway, which is not the same as likely to be moved by the campaign, and only the measured uplift captures that difference.
• Attributing revenue lift to a deployed solution: use an experiment where possible. Where it cannot be done, use a quasi-experiment (a comparison that was not randomised) such as difference-in-differences (the change in the treated group minus the change in a comparable untreated group) against a comparable group that did not get the change, with the attribution window written down in advance (long enough to see repeat behaviour, short enough to exclude unrelated campaigns), and report the confidence level with the figure.

Trade-offs and pitfalls

• Reporting "p = 0.04, so USD 1.6 million": that confuses evidence of an effect with the size of the effect.
• Treating the confidence interval as the range of the dollar benefit without the realisation factor: the interval says nothing about conversion to cash.
• Extrapolating to a population the test did not sample.
• Optimising a proxy: more tasks per week can mean lower quality; check a quality guardrail metric.

Direct answer

Conversion going from 2.0% to 2.5% on 100,000 visitors means 2,000 orders a month become 2,500, so 500 extra orders x $50 = $25,000 extra revenue a month, $300,000 a year. Payback (the time for the benefit to repay the cost) on revenue is $400,000 / $25,000 = 16 months. But revenue is not profit: at an illustrative 30% margin the benefit is $7,500 a month and payback is 53 months. The input I trust least is the 0.5-point lift itself, because it is a forecast of a behaviour change, not a measured fact.

The arithmetic

Use profit (contribution margin: what is left of each sale after product cost, shipping and payment fees) for the decision; revenue flatters the case. A relative view helps communication: 2.0 to 2.5 is a 25% relative lift.

Which input I trust least, and why

Ranking by risk: (1) the lift, since the effect of personalisation varies by segment and often fades after novelty; (2) the margin, which finance can confirm quickly; (3) traffic, which can move with season and marketing spend; (4) average order value (AOV), since personalisation can lower it if it pushes cheaper items. Sensitivity on the revenue-payback basis:

Halving the lift doubles payback, which is why that is the assumption to test first. It can be tested with an A/B test: split visitors at random into two arms (a control group on the current site and a treatment group with the feature) and compare conversion. How many visitors are needed comes from the standard two-proportion sample size formula, n per arm = (z_a + z_b)^2 x [p1(1-p1) + p2(1-p2)] / (p2 - p1)^2. Here p1 = 2.0% and p2 = 2.5% are the two conversion rates; z_a = 1.96 is the cut-off for 5% two-sided significance (accepting a 5% chance of calling a lift real when it is noise); z_b = 0.84 is the cut-off for 80% power (an 80% chance of detecting the lift if it is really there). Plugging in: (1.96 + 0.84)^2 = 7.84; p1(1-p1) + p2(1-p2) = 0.0196 + 0.024375 = 0.043975; (p2 - p1)^2 = 0.000025; so n = 7.84 x 0.043975 / 0.000025 = about 13,800 visitors in each arm. That is about 27,600 in total, which is under two weeks of traffic at 100,000 a month (about 8 days). State that this is an approximation and ask the data science team to do the exact power calculation.

Related versions

• Retention feature: 5-point retention gain on 20,000 customers = 1,000 more retained customers. At $50 a month and 70% margin that is 1,000 x $35 = $35,000 a month; a $500,000 build pays back in 14.3 months if the retained customers stay (in reality retention decays, so a cohort NPV is the better test. NPV, net present value, means discounting each future month's cash back to today's money, because a dollar later is worth less than a dollar now; a cohort is the group of customers acquired together, whose retention you follow over time. The discount rate, here 10% a year, is the return the company requires on money it invests).
• Data-labelling project (paying people to tag training examples so a model improves, here +8% model lift): convert the lift into the business metric it moves (orders or cost avoided), then compute break-even months the same way. Illustrative numbers: the model drives $600,000 a year of contribution margin, so an 8% lift is worth $48,000 a year; a $120,000 labelling budget breaks even in $120,000 / $48,000 = 2.5 years, or 30 months.
• 12-month recommender projection (a recommender suggests products to each shopper): add the targeting cost per user (compute, offers, discounts) as a recurring cost beside the one-time build. Illustrative: at $0.02 per visitor, 100,000 visitors cost $2,000 a month, so the $7,500 monthly profit above becomes $5,500 and profit payback on the $400,000 build stretches from 53 to about 73 months.

Pitfalls

Quoting revenue as the benefit; applying a lab lift to all traffic when it was measured on a segment; ignoring the cost of running the feature; and letting a 12-month projection assume steady-state lift from day one. I would present the base case, a downside at half the lift, and the test that would narrow the range.