Business Acumen and Commercial Context Questions

Direct answer

Show the return as a before-and-after comparison on incidents the business already feels: faster detection and recovery means fewer minutes of customer impact and less engineer time per incident. Convert the minutes saved into dollars with the cost of downtime, subtract what the observability costs, and state the break-even. In the illustration below, saving 30 minutes per incident returns $7,600 a month net on $9,000 of monthly cost, and the investment breaks even at about 16 minutes saved per incident.

Definitions

Observability is the ability to see what the system is doing from its logs, metrics and traces so that you can find the cause of a problem quickly. MTTR (mean time to restore) is the average time from the start of an incident to recovery. Investing in observability mainly cuts the time to detect and diagnose, so it shortens MTTR.

Hypothetical example (illustrative numbers)

Inputs: 8 incidents a month, 5 of which affect customers; each customer-affecting hour costs $6,000 in lost profit and credits; 4 engineers respond to each incident at $100 per hour. Better dashboards and tracing are expected to cut 30 minutes from the time to restore every incident. Cost: $6,000 a month for the platform plus a quarter of an engineer's time ($3,000 a month), so $9,000 a month. A one-off instrumentation project costs $30,000.

Stating the break-even and the sensitivity

Each minute saved per incident is worth 5 x ($6,000 / 60) + 8 x 4 x ($100 / 60) = $553 a month. The $9,000 of monthly cost is covered at $9,000 / $553 = about 16.3 minutes saved per incident. If the improvement were only 10 minutes, the monthly benefit would be $5,533, below the $9,000 cost, so the investment would not pay. This is the honest statement to make to executives: the case rests on the minutes saved, so commit to measuring that number.

Making the claim credible

1. Baseline MTTR and the time to detect for the last two quarters, taken from the incident tracker, not recalled.
2. After the rollout, compare the same incident classes (not the easy ones), and report the change in median and slowest-decile recovery time.
3. Include benefits that are harder to price, such as incidents caught before customers saw them or fewer escalations at night, as separate, unpriced bullets, not folded into the dollar total.

Pitfalls

• Claiming the full cost of downtime as savings. Only the minutes saved count.
• Treating alerts as value: more alerts without a lower detection time is cost.
• Ignoring the ongoing cost of storing and querying telemetry, which can grow faster than traffic.

Direct answer

Add up the small incidents into one monthly cost (customer-facing downtime plus responder time), classify them by root cause, and then price the fix for each large cause separately. In the illustration below, 30 short incidents cost about $28,500 a month; a broad $96,000 rewrite pays back in about 4.8 months, but fixing only the single biggest cause costs $12,000 and pays back in about 1.4 months. Fix the causes with the shortest payback first (here the second-largest cause pays back slightly faster than the largest), and keep patching the rest.

Step 1: measure the cumulative cost

Illustrative inputs: 30 incidents a month, each affecting customers for 12 minutes on average; each affected hour costs $4,000 in lost profit; each incident takes 2 responders 45 minutes at $100 per hour loaded.

Two costs are missing and should be noted, not priced: interruptions and context switching for the engineers, and the erosion of trust. Do not guess them into the total.

Step 2: classify by root cause

Tag every incident with its cause. Illustrative result: cache stampede (many requests rebuilding the same cache entry at once) 9, bad configuration push 7, dependency timeout 5, and 9 spread across 6 other causes, none above 2 incidents each. These sum to 30 (9 + 7 + 5 + 9). The top three causes account for 21 of 30, or 70%; the long tail of small causes is the other 30%, and no single fix touches much of it.

Step 3: price the options

The monthly saving is the share of incidents removed times $28,500, which assumes every incident costs the average $950 ($28,500 / 30); check that the causes you fix are not cheaper or dearer than average. The costs of the second and third fixes are illustrative estimates for the same hourly rate. Doing all three narrow fixes costs $12,000 + $8,000 + $16,000 = $36,000 and removes the same 70% of incidents ($19,950 a month), a payback of $36,000 / $19,950 = 1.8 months, against 4.8 months for the broad rewrite. The rewrite only earns its extra $60,000 if it also removes part of the long tail or prevents new incidents, which is the case to test before funding it.

The decision

Rank the narrow fixes by payback: the second cause (bad configuration push, 1.2 months) and the biggest cause (cache stampede, 1.4 months) come first, and they can run in parallel because each needs one engineer; the third (dependency timeout, 3.4 months) follows, while the numbers keep showing payback inside the window the business accepts. A six-month payback threshold is a common-sense choice, not a standard: set it against what else the same engineers could do. Do not start the broad rewrite until the narrow fixes show the incident count falls as predicted, because the 70% assumption is the one that decides the broad case (at 50% it becomes 6.7 months).

What would change my call

• If the incidents share one underlying architectural cause rather than many small causes, the broad fix becomes the narrow fix.
• If the incident rate is rising rather than steady, favour the deeper fix sooner.
• If the same few engineers carry all the on-call load, weigh the burn-out risk as a reason to fix earlier even when payback is slower.

Pitfalls

• Averaging the costs: ten five-minute incidents and one 50-minute incident are not the same risk to customers, so keep the distribution as well as the sum.
• Pricing the fix without a measured effect. Record how many incidents each completed fix removes and revise the next estimate.
• Counting the responders' time at the same rate as customer downtime. They are different bases, shown separately above.

Direct answer

Reliability connects to revenue through a short chain: how often the service fails, how long customers are affected (the recovery time), how much revenue flows through the affected part during those minutes, how much of that comes back later, and what extra costs the outage triggers (service-level credits, support load). A longer recovery time multiplies the minutes of impact, so the lost-revenue part of the loss grows in proportion to it; the fixed per-incident costs do not grow, so in the example below tripling recovery time from 30 to 90 minutes raises the total cost per incident about 1.65 times ($17,753 to $29,260), not 3 times.

The chain, step by step

1. Frequency. Number of customer-visible incidents per year (6 in the example).
2. Duration. MTTR, mean time to recovery: the average time from the start of customer impact to restoration, so it includes the time taken to detect the problem as well as the time to fix it. In the example it moves from 30 to 90 minutes.
3. Revenue exposure per minute. Average revenue per minute (annual online revenue / minutes in the year), adjusted for when the outage happens (a peak factor of 1.5, an assumption that your incidents tend to land in busier-than-average hours; if outages struck at uniformly random times the factor would be 1.0, so take it from the hour of day of your own past incidents) and for the share of revenue the outage actually blocks (80%, for example because 20% of orders come through a path the outage does not touch, such as the mobile app on a separate service).
4. Recoverable share. Some customers retry later: here 30% of blocked purchases return.
5. Other costs. SLA credits (refunds promised in a service-level agreement), support tickets and engineer time: $12,000 per incident, illustrative and treated as fixed per incident here (in practice credits often grow with duration).
6. Total. Loss per minute x minutes + other costs, then x incidents per year.

Worked example

Annual online revenue is $120 million.

ANNUAL_ONLINE_REVENUE = 120_000_000
HOURS_PER_YEAR = 8760
PEAK_FACTOR = 1.5            # an outage hits busier-than-average hours
SHARE_AFFECTED = 0.8         # share of revenue flows the outage blocks
RETURN_RATE = 0.3            # share of blocked purchases that happen later anyway
SLA_CREDITS_AND_SUPPORT = 12_000   # dollars per incident, illustrative

avg_per_minute = ANNUAL_ONLINE_REVENUE / HOURS_PER_YEAR / 60
loss_per_minute = avg_per_minute * PEAK_FACTOR * SHARE_AFFECTED * (1 - RETURN_RATE)
print(f"average revenue per minute ${avg_per_minute:,.2f}; lost revenue per outage minute ${loss_per_minute:,.2f}")
for mttr in (30, 90):
    per_incident = loss_per_minute * mttr + SLA_CREDITS_AND_SUPPORT
    print(f"MTTR {mttr:>2} min: ${per_incident:>9,.0f} per incident, 6 incidents a year ${6 * per_incident:>9,.0f}")

Output:

average revenue per minute $228.31; lost revenue per outage minute $191.78
MTTR 30 min: $   17,753 per incident, 6 incidents a year $  106,521
MTTR 90 min: $   29,260 per incident, 6 incidents a year $  175,562

If the factor were 1.0 (incidents at random hours) lost revenue per outage minute would be $127.85 and an incident at 30 minutes would cost $15,836, so the peak-factor assumption moves the per-incident cost by about 12%. Lost revenue per outage minute at the 1.5 factor is $191.78, so extending recovery from 30 to 90 minutes adds 60 x $191.78 = $11,507 per incident (from $17,753 to $29,260) and about $69,000 a year over six incidents (from $106,521 to $175,562).

Where the numbers come from, and the caveats

• Revenue per minute should come from payment logs for the same weekday and hour of past incidents, not from a yearly average.
• The return rate comes from comparing order volume in the hours after past outages with a normal day. For example, if 1,000 orders were blocked during an outage and the following three hours ran 300 orders above a normal day, the return rate is 300 / 1,000 = 30%.
• The share blocked comes from payment logs too: the fraction of normal orders in the affected hours that went through the failed component.
• Revenue that is not minute-by-minute (annual subscriptions) is not lost during an outage; the damage shows up as churn, credits and trust, which need separate estimates.
• The model leaves out longer-term effects (customers who leave), so it is a floor for the damage of a single long outage.

Why this matters for decisions

It turns an engineering metric into a price: "each minute we shave off recovery is worth about $190 per incident", which you can set against the cost of better alerting or runbooks (written step-by-step recovery procedures).

Direct answer

The value of a latency cut is the extra conversions it produces times what each conversion is worth, minus the cost of delivering the speed-up. You need five inputs: traffic and baseline conversion, contribution per conversion, (most important) a measured relationship between latency and conversion from your own site, which is called the elasticity: the percentage change in conversion for each 100 ms of speed gained, which latency and which users the speed-up applies to, and the cost of delivering it. Such estimates usually go wrong by borrowing someone else's elasticity, extrapolating a straight line, confusing correlation with cause, and forgetting the running cost.

What you need to know

1. Traffic and conversion. Sessions per month (10,000,000 in the example) and baseline conversion (2.5%, so 250,000 conversions a month). Conversion here means a session ending in a purchase.
2. Value per conversion. Contribution per conversion (revenue less variable cost), $40 in the example. Using revenue overstates the benefit.
3. The latency-to-conversion relationship. The relative change in conversion for each 100 ms saved. This is the number everything hangs on and it cannot be assumed.
4. Which latency and which users. Median or 95th percentile (p95, the speed that 95% of loads beat)? Which pages (product page versus checkout) and which devices? A 500 ms to 200 ms improvement on the median may be a 3000 ms to 2900 ms change for the slowest users.
5. Cost to deliver. A caching layer costs $18,000 a month in infrastructure and upkeep, in this example.

How to get the relationship

Best: a randomised experiment that adds an artificial delay (for example 100 or 300 ms) to a small slice of traffic and measures conversion loss. Next best: a natural experiment (a change nobody planned as a test but that you can compare before and after, such as a deploy that happened to change speed), keeping the device mix the same and adjusting for confounders, meaning other things that changed at the same time, such as a sale or a new ad campaign. Weakest: correlation across pages, because fast pages are usually simple pages and fast users have good devices.

Worked example

SESSIONS_PER_MONTH = 10_000_000
CONV_RATE = 0.025
CONTRIBUTION_PER_CONV = 40.0     # dollars per conversion after variable costs
CACHE_COST_PER_MONTH = 18_000    # infrastructure plus upkeep of the caching layer
MS_SAVED = 300

conversions = SESSIONS_PER_MONTH * CONV_RATE
print(f"baseline conversions per month: {conversions:,.0f}")
print("assumed relative conversion gain per 100 ms saved -> value and net per month")
for per_100ms in (0.0025, 0.005, 0.01, 0.015):
    relative_gain = per_100ms * MS_SAVED / 100
    value = conversions * relative_gain * CONTRIBUTION_PER_CONV
    print(f"{per_100ms:.2%} per 100 ms: +{relative_gain:.2%} = {conversions*relative_gain:>7,.0f} conversions, ${value:>8,.0f}/month, net ${value - CACHE_COST_PER_MONTH:>8,.0f}")

needed_conversions = CACHE_COST_PER_MONTH / CONTRIBUTION_PER_CONV
total_gain = needed_conversions / conversions
print(f"break-even: {needed_conversions:,.0f} extra conversions a month = {total_gain:.2%} total = {total_gain / (MS_SAVED / 100):.2%} per 100 ms")

Output:

baseline conversions per month: 250,000
assumed relative conversion gain per 100 ms saved -> value and net per month
0.25% per 100 ms: +0.75% =   1,875 conversions, $  75,000/month, net $  57,000
0.50% per 100 ms: +1.50% =   3,750 conversions, $ 150,000/month, net $ 132,000
1.00% per 100 ms: +3.00% =   7,500 conversions, $ 300,000/month, net $ 282,000
1.50% per 100 ms: +4.50% =  11,250 conversions, $ 450,000/month, net $ 432,000
break-even: 450 extra conversions a month = 0.18% total = 0.06% per 100 ms

• At an assumed 0.5% relative conversion gain per 100 ms, saving 300 ms gives +1.5%, which is 3,750 extra conversions and $150,000 a month; net of the $18,000 cache cost, $132,000 a month, a return on cost of about 7.3 times ($132,000 / $18,000).
• The result is very sensitive to the assumed elasticity: it swings from $57,000 to $432,000 net a month across the range tried, while break-even is only 0.06% per 100 ms. So the decision to build is robust, but the headline benefit number is not and should be quoted as a range.

Where such estimates usually go wrong

• Borrowed statistics. Another company's "100 ms = 1%" is a story about its users and pages.
• Linear extrapolation. Gains tend to shrink below a speed where users stop noticing, and may be larger for slow users; a straight line over 300 ms overstates.
• Correlation as cause. Fast pages convert better partly because of what they are.
• Averages versus tails. The benefit may sit in the p95 population, which the median hides.
• Double counting. Double counting means claiming the same benefit twice, for example counting both higher conversion and lower bounce rate (the share of visitors who leave after one page; it is a symptom of the same effect), or adding SEO gains (better search-engine ranking from faster pages) that the experiment cannot see and that you have not measured.
• Forgetting running cost and opportunity cost. The cache needs maintenance, invalidation bugs (cache invalidation is deciding when stored copies of a page are out of date and must be refreshed) can serve stale prices, and the engineers could be doing something else.
• Pulled-forward demand. Some conversions would have happened anyway, later or through another channel; a faster page may only make a shopper buy today instead of next week, so the net gain is smaller than the count of extra conversions in the window.

What I would present

A range with a stated method ("measured 0.5% per 100 ms from a delay experiment, interval 0.3% to 0.7%"), the cost, the break-even elasticity, and the one thing that would change the call. Spoken version: "We measured that every 100 ms we remove lifts conversion by about 0.5% relative, with a plausible range of 0.3% to 0.7%. Cutting 300 ms is therefore worth about $150,000 a month, between $90,000 and $210,000, against $18,000 a month to run. It stops paying for itself only if the true effect is below 0.06% per 100 ms, far under anything we measured. The one thing that would change my call is if the effect turns out to be concentrated in users the cache does not reach."