Business Acumen and Commercial Context Questions

Direct answer

The value of a latency cut is the extra conversions it produces times what each conversion is worth, minus the cost of delivering the speed-up. You need five inputs: traffic and baseline conversion, contribution per conversion, (most important) a measured relationship between latency and conversion from your own site, which is called the elasticity: the percentage change in conversion for each 100 ms of speed gained, which latency and which users the speed-up applies to, and the cost of delivering it. Such estimates usually go wrong by borrowing someone else's elasticity, extrapolating a straight line, confusing correlation with cause, and forgetting the running cost.

What you need to know

1. Traffic and conversion. Sessions per month (10,000,000 in the example) and baseline conversion (2.5%, so 250,000 conversions a month). Conversion here means a session ending in a purchase.
2. Value per conversion. Contribution per conversion (revenue less variable cost), $40 in the example. Using revenue overstates the benefit.
3. The latency-to-conversion relationship. The relative change in conversion for each 100 ms saved. This is the number everything hangs on and it cannot be assumed.
4. Which latency and which users. Median or 95th percentile (p95, the speed that 95% of loads beat)? Which pages (product page versus checkout) and which devices? A 500 ms to 200 ms improvement on the median may be a 3000 ms to 2900 ms change for the slowest users.
5. Cost to deliver. A caching layer costs $18,000 a month in infrastructure and upkeep, in this example.

How to get the relationship

Best: a randomised experiment that adds an artificial delay (for example 100 or 300 ms) to a small slice of traffic and measures conversion loss. Next best: a natural experiment (a change nobody planned as a test but that you can compare before and after, such as a deploy that happened to change speed), keeping the device mix the same and adjusting for confounders, meaning other things that changed at the same time, such as a sale or a new ad campaign. Weakest: correlation across pages, because fast pages are usually simple pages and fast users have good devices.

Worked example

SESSIONS_PER_MONTH = 10_000_000
CONV_RATE = 0.025
CONTRIBUTION_PER_CONV = 40.0     # dollars per conversion after variable costs
CACHE_COST_PER_MONTH = 18_000    # infrastructure plus upkeep of the caching layer
MS_SAVED = 300

conversions = SESSIONS_PER_MONTH * CONV_RATE
print(f"baseline conversions per month: {conversions:,.0f}")
print("assumed relative conversion gain per 100 ms saved -> value and net per month")
for per_100ms in (0.0025, 0.005, 0.01, 0.015):
    relative_gain = per_100ms * MS_SAVED / 100
    value = conversions * relative_gain * CONTRIBUTION_PER_CONV
    print(f"{per_100ms:.2%} per 100 ms: +{relative_gain:.2%} = {conversions*relative_gain:>7,.0f} conversions, ${value:>8,.0f}/month, net ${value - CACHE_COST_PER_MONTH:>8,.0f}")

needed_conversions = CACHE_COST_PER_MONTH / CONTRIBUTION_PER_CONV
total_gain = needed_conversions / conversions
print(f"break-even: {needed_conversions:,.0f} extra conversions a month = {total_gain:.2%} total = {total_gain / (MS_SAVED / 100):.2%} per 100 ms")

Output:

baseline conversions per month: 250,000
assumed relative conversion gain per 100 ms saved -> value and net per month
0.25% per 100 ms: +0.75% =   1,875 conversions, $  75,000/month, net $  57,000
0.50% per 100 ms: +1.50% =   3,750 conversions, $ 150,000/month, net $ 132,000
1.00% per 100 ms: +3.00% =   7,500 conversions, $ 300,000/month, net $ 282,000
1.50% per 100 ms: +4.50% =  11,250 conversions, $ 450,000/month, net $ 432,000
break-even: 450 extra conversions a month = 0.18% total = 0.06% per 100 ms

• At an assumed 0.5% relative conversion gain per 100 ms, saving 300 ms gives +1.5%, which is 3,750 extra conversions and $150,000 a month; net of the $18,000 cache cost, $132,000 a month, a return on cost of about 7.3 times ($132,000 / $18,000).
• The result is very sensitive to the assumed elasticity: it swings from $57,000 to $432,000 net a month across the range tried, while break-even is only 0.06% per 100 ms. So the decision to build is robust, but the headline benefit number is not and should be quoted as a range.

Where such estimates usually go wrong

• Borrowed statistics. Another company's "100 ms = 1%" is a story about its users and pages.
• Linear extrapolation. Gains tend to shrink below a speed where users stop noticing, and may be larger for slow users; a straight line over 300 ms overstates.
• Correlation as cause. Fast pages convert better partly because of what they are.
• Averages versus tails. The benefit may sit in the p95 population, which the median hides.
• Double counting. Double counting means claiming the same benefit twice, for example counting both higher conversion and lower bounce rate (the share of visitors who leave after one page; it is a symptom of the same effect), or adding SEO gains (better search-engine ranking from faster pages) that the experiment cannot see and that you have not measured.
• Forgetting running cost and opportunity cost. The cache needs maintenance, invalidation bugs (cache invalidation is deciding when stored copies of a page are out of date and must be refreshed) can serve stale prices, and the engineers could be doing something else.
• Pulled-forward demand. Some conversions would have happened anyway, later or through another channel; a faster page may only make a shopper buy today instead of next week, so the net gain is smaller than the count of extra conversions in the window.

What I would present

A range with a stated method ("measured 0.5% per 100 ms from a delay experiment, interval 0.3% to 0.7%"), the cost, the break-even elasticity, and the one thing that would change the call. Spoken version: "We measured that every 100 ms we remove lifts conversion by about 0.5% relative, with a plausible range of 0.3% to 0.7%. Cutting 300 ms is therefore worth about $150,000 a month, between $90,000 and $210,000, against $18,000 a month to run. It stops paying for itself only if the true effect is below 0.06% per 100 ms, far under anything we measured. The one thing that would change my call is if the effect turns out to be concentrated in users the cache does not reach."

Direct answer

Reliability connects to revenue through a short chain: how often the service fails, how long customers are affected (the recovery time), how much revenue flows through the affected part during those minutes, how much of that comes back later, and what extra costs the outage triggers (service-level credits, support load). A longer recovery time multiplies the minutes of impact, so the lost-revenue part of the loss grows in proportion to it; the fixed per-incident costs do not grow, so in the example below tripling recovery time from 30 to 90 minutes raises the total cost per incident about 1.65 times ($17,753 to $29,260), not 3 times.

The chain, step by step

1. Frequency. Number of customer-visible incidents per year (6 in the example).
2. Duration. MTTR, mean time to recovery: the average time from the start of customer impact to restoration, so it includes the time taken to detect the problem as well as the time to fix it. In the example it moves from 30 to 90 minutes.
3. Revenue exposure per minute. Average revenue per minute (annual online revenue / minutes in the year), adjusted for when the outage happens (a peak factor of 1.5, an assumption that your incidents tend to land in busier-than-average hours; if outages struck at uniformly random times the factor would be 1.0, so take it from the hour of day of your own past incidents) and for the share of revenue the outage actually blocks (80%, for example because 20% of orders come through a path the outage does not touch, such as the mobile app on a separate service).
4. Recoverable share. Some customers retry later: here 30% of blocked purchases return.
5. Other costs. SLA credits (refunds promised in a service-level agreement), support tickets and engineer time: $12,000 per incident, illustrative and treated as fixed per incident here (in practice credits often grow with duration).
6. Total. Loss per minute x minutes + other costs, then x incidents per year.

Worked example

Annual online revenue is $120 million.

ANNUAL_ONLINE_REVENUE = 120_000_000
HOURS_PER_YEAR = 8760
PEAK_FACTOR = 1.5            # an outage hits busier-than-average hours
SHARE_AFFECTED = 0.8         # share of revenue flows the outage blocks
RETURN_RATE = 0.3            # share of blocked purchases that happen later anyway
SLA_CREDITS_AND_SUPPORT = 12_000   # dollars per incident, illustrative

avg_per_minute = ANNUAL_ONLINE_REVENUE / HOURS_PER_YEAR / 60
loss_per_minute = avg_per_minute * PEAK_FACTOR * SHARE_AFFECTED * (1 - RETURN_RATE)
print(f"average revenue per minute ${avg_per_minute:,.2f}; lost revenue per outage minute ${loss_per_minute:,.2f}")
for mttr in (30, 90):
    per_incident = loss_per_minute * mttr + SLA_CREDITS_AND_SUPPORT
    print(f"MTTR {mttr:>2} min: ${per_incident:>9,.0f} per incident, 6 incidents a year ${6 * per_incident:>9,.0f}")

Output:

average revenue per minute $228.31; lost revenue per outage minute $191.78
MTTR 30 min: $   17,753 per incident, 6 incidents a year $  106,521
MTTR 90 min: $   29,260 per incident, 6 incidents a year $  175,562

If the factor were 1.0 (incidents at random hours) lost revenue per outage minute would be $127.85 and an incident at 30 minutes would cost $15,836, so the peak-factor assumption moves the per-incident cost by about 12%. Lost revenue per outage minute at the 1.5 factor is $191.78, so extending recovery from 30 to 90 minutes adds 60 x $191.78 = $11,507 per incident (from $17,753 to $29,260) and about $69,000 a year over six incidents (from $106,521 to $175,562).

Where the numbers come from, and the caveats

• Revenue per minute should come from payment logs for the same weekday and hour of past incidents, not from a yearly average.
• The return rate comes from comparing order volume in the hours after past outages with a normal day. For example, if 1,000 orders were blocked during an outage and the following three hours ran 300 orders above a normal day, the return rate is 300 / 1,000 = 30%.
• The share blocked comes from payment logs too: the fraction of normal orders in the affected hours that went through the failed component.
• Revenue that is not minute-by-minute (annual subscriptions) is not lost during an outage; the damage shows up as churn, credits and trust, which need separate estimates.
• The model leaves out longer-term effects (customers who leave), so it is a floor for the damage of a single long outage.

Why this matters for decisions

It turns an engineering metric into a price: "each minute we shave off recovery is worth about $190 per incident", which you can set against the cost of better alerting or runbooks (written step-by-step recovery procedures).

Direct answer

Add up the small incidents into one monthly cost (customer-facing downtime plus responder time), classify them by root cause, and then price the fix for each large cause separately. In the illustration below, 30 short incidents cost about $28,500 a month; a broad $96,000 rewrite pays back in about 4.8 months, but fixing only the single biggest cause costs $12,000 and pays back in about 1.4 months. Fix the causes with the shortest payback first (here the second-largest cause pays back slightly faster than the largest), and keep patching the rest.

Step 1: measure the cumulative cost

Illustrative inputs: 30 incidents a month, each affecting customers for 12 minutes on average; each affected hour costs $4,000 in lost profit; each incident takes 2 responders 45 minutes at $100 per hour loaded.

Two costs are missing and should be noted, not priced: interruptions and context switching for the engineers, and the erosion of trust. Do not guess them into the total.

Step 2: classify by root cause

Tag every incident with its cause. Illustrative result: cache stampede (many requests rebuilding the same cache entry at once) 9, bad configuration push 7, dependency timeout 5, and 9 spread across 6 other causes, none above 2 incidents each. These sum to 30 (9 + 7 + 5 + 9). The top three causes account for 21 of 30, or 70%; the long tail of small causes is the other 30%, and no single fix touches much of it.

Step 3: price the options

The monthly saving is the share of incidents removed times $28,500, which assumes every incident costs the average $950 ($28,500 / 30); check that the causes you fix are not cheaper or dearer than average. The costs of the second and third fixes are illustrative estimates for the same hourly rate. Doing all three narrow fixes costs $12,000 + $8,000 + $16,000 = $36,000 and removes the same 70% of incidents ($19,950 a month), a payback of $36,000 / $19,950 = 1.8 months, against 4.8 months for the broad rewrite. The rewrite only earns its extra $60,000 if it also removes part of the long tail or prevents new incidents, which is the case to test before funding it.

The decision

Rank the narrow fixes by payback: the second cause (bad configuration push, 1.2 months) and the biggest cause (cache stampede, 1.4 months) come first, and they can run in parallel because each needs one engineer; the third (dependency timeout, 3.4 months) follows, while the numbers keep showing payback inside the window the business accepts. A six-month payback threshold is a common-sense choice, not a standard: set it against what else the same engineers could do. Do not start the broad rewrite until the narrow fixes show the incident count falls as predicted, because the 70% assumption is the one that decides the broad case (at 50% it becomes 6.7 months).

What would change my call

• If the incidents share one underlying architectural cause rather than many small causes, the broad fix becomes the narrow fix.
• If the incident rate is rising rather than steady, favour the deeper fix sooner.
• If the same few engineers carry all the on-call load, weigh the burn-out risk as a reason to fix earlier even when payback is slower.

Pitfalls

• Averaging the costs: ten five-minute incidents and one 50-minute incident are not the same risk to customers, so keep the distribution as well as the sum.
• Pricing the fix without a measured effect. Record how many incidents each completed fix removes and revise the next estimate.
• Counting the responders' time at the same rate as customer downtime. They are different bases, shown separately above.

Direct answer

A strong answer is a short story with five parts: the business problem and the baseline (the number before the change), the metric I chose and why, what I did and what it cost, how I measured the effect, and the limits on what I could claim. Below is a model story whose numbers are all derived from assumptions shown in it; a real answer replaces them with your own.

Model story

Situation and baseline. A subscription product charges $30 a month. About 20,000 renewals a month; 4% fail on the first attempt (800 failed payments). The existing retry logic recovered 50% (400), so 400 customers a month were lost to payment failure, worth $12,000 a month in revenue.

Metric choice. I chose recovery rate of failed renewals as the main metric because the change could move it within weeks and finance understands it. Overall churn was too slow and noisy to attribute. As a guardrail (a number that must not get worse), I tracked support contacts about duplicate charges.

Action. I changed when retries happen, spacing them toward typical paydays and sending a card-update email before the second retry.

Measurement. A four-week randomised test (customers assigned to a group by chance) with a control group (customers left on the old logic, for comparison) and a treatment group (customers given the new retry timing). Control: 200 of 400 recovered (50%). Treatment: 232 of 400 (58%). The difference is 8 points. The standard error (the typical size of the sampling noise in the difference) is sqrt(0.5 x 0.5 / 400 + 0.58 x 0.42 / 400) = sqrt(0.001234) = 3.5 points. An approximate 95% interval is the difference plus or minus 1.96 standard errors: 8 +/- 1.96 x 3.5 = 8 +/- 6.9, so 1.1 to 14.9 points. Read it as: the true gain is plausibly anywhere in that range.

Value and cost.

• 8 points x 800 failures = 64 more recovered renewals a month x $30 = $1,920 revenue, x 80% gross margin (the share of revenue left after direct delivery costs) = $1,536 contribution (profit after those costs) a month.
• Cost: three engineering weeks at $4,000 = $12,000 once, plus $200 a month for the email service.
• Net $1,336 a month, so payback is $12,000 / $1,336 = 9.0 months.
• Across the interval: using the rounded interval ends, at 1.1 points: 0.011 x 800 x $30 x 80% = $211, less the $200 service cost, is a net of about $11 a month (no practical payback); at 14.9 points: 0.149 x 800 x $30 x 80% = $2,861, less $200, is $2,661 a month (payback 12,000 / 2,661 = 4.5 months).

Limits on attribution. (1) 400 customers per group is small; the honest range is wide. (2) I counted only the recovered renewal payment; the extra months those customers stay were not measured, so I did not claim them. (3) Four weeks may not represent paydays across seasons. (4) A test shows the effect on this customer base now, not for a different market or price.

How I explained it to a non-technical stakeholder

"We changed when we retry failed cards. In a four-week test, 58 of every 100 failed renewals were recovered instead of 50. That is worth about $1,500 a month of contribution, $1,300 after running costs, and pays back in about nine months. The true gain is somewhere between roughly nothing and $2,700 a month, so we are extending the test." Visual: one chart with the two recovery rates as dots with their intervals (so the overlap is visible), and a small payback line showing the three cases, rather than a table of rates.

What separates strong answers

• A baseline stated before the change, not reconstructed afterwards.
• A comparison group, or a stated reason why none was possible.
• Cost included, not only benefit.
• An explicit statement of what you did not claim.
• A decision that followed (extend the test, ship, stop).

Direct answer

Lead with the customer and money, not the bug. Say: "We found a fault that is making some customers fail to pay us. Fixing it first takes about two weeks and moves the new feature by two weeks. Every week we leave it unfixed puts up to $24,000 of orders at risk, against about $15,000 a week of expected feature revenue, and the feature is only delayed, not lost. If many failed customers retry, the true loss is smaller, so I will measure the retry rate and confirm." Then give the new date, the cost of each option, and the one decision you need from them.

How to build the message

1. Translate the regression into a business symptom. A "reliability regression" means something that used to work got worse: slower pages, more errors, more downtime. Name the symptom the CEO already cares about: failed checkouts, support tickets, churn risk, a contract with an availability promise.
2. Put a number on the cost of leaving it. Use rates and money, not technical measures.
3. Put a number on the cost of the delay. Be honest that a delay is revenue deferred, not revenue destroyed, unless a launch window or a customer commitment makes it truly lost.
4. Offer a choice, with a recommendation. Two or three options, each with date, cost and risk, and your pick.
5. Say what you will report and when, so trust does not depend on the CEO understanding the fix.

Worked example (illustrative numbers)

A shop has 20,000 checkout attempts a week at an average order of $80. After last release, 1.5% of attempts fail with an error.

Some failed customers retry, so $24,000 is an upper bound; say so, and say you will measure the real retry rate. The comparison with the $15,000 a week the feature earns holds only if fewer than about 37.5% of failed checkouts are retried and completed (1 - 15,000 / 24,000 = 37.5%); above that, the fault costs less per week than the feature earns, and the case for fixing first rests on the risk growing and on the feature running on the same checkout. The script in the room:

"A change we shipped last month is making about 1 in 67 checkouts fail. That is roughly $24,000 of orders at risk every week. If we fix it first, the new feature lands two weeks later and about $30,000 of its revenue moves later in the year. If we ship the feature first, the checkout problem stays live and the risk keeps growing, and the feature itself runs on the same checkout. I recommend fixing first. I will send you the failed-checkout rate every Friday until it is back to normal."

(The 1 in 67 is 1 divided by 0.015.)

Trade-offs and pitfalls

• Jargon. Do not say "latency regression", "error budget" or "tech debt" to a CEO. Say "customers waiting", "failures", "money at risk".
• Do not hide the delay inside a vague date. Give a date with a confidence level ("two weeks, and I will confirm by Wednesday once we have traced the cause").
• Do not oversell the fix. If the cause is not yet known, say the range and when you will know.
• What changes the recommendation: if the feature is tied to a dated launch, a contract or a partner event that would be lost by a two-week slip, or if the regression only hits a small, low-value slice, the comparison changes. Say that you checked.