Direct answer
A technical choice only matters commercially through a chain of cause and effect that ends in revenue, cost or risk. Write the chain, name one metric that should move, estimate the size, and then test it with a comparison rather than assuming it. Example chain: polling to event-driven means customers see an order status change within seconds instead of up to 30 seconds, so fewer of them contact support asking "where is my order?", which lowers support cost and improves repeat purchase.
The chain, link by link
Take a delivery-tracking page. Polling means the browser asks the server "anything new?" on a timer. Event-driven means the server pushes a message when the status changes.
| Link | What changes | How you would know |
|---|
| 1. Technical change | Status updates are pushed on change instead of fetched every 30 s | Update delay: average 15 s under 30 s polling (half the interval) down to a few seconds |
| 2. Load | Far fewer requests | Request count per hour |
| 3. Customer experience | Page shows the true state sooner, so fewer "stale page" moments | Share of sessions where the user refreshes manually |
| 4. Behaviour | Fewer customers contact support to ask for status | Contacts per 1,000 orders |
| 5. Business outcome | Lower support cost, higher satisfaction, possibly more repeat orders | Cost per order, repeat purchase rate |
Size it (illustrative numbers)
- Request load: 50,000 customers watching the page, each polling twice a minute = 100,000 requests a minute = 6,000,000 an hour. If each of those sessions sees about 6 status changes an hour, events number 50,000 x 6 = 300,000 an hour, 5% of the polling volume. Caveat: keeping 50,000 connections open has its own cost (memory, connection limits, and reconnect storms, where a server restart or network blip makes thousands of clients reconnect at once and overload the server), so the saving in requests is not the saving in money; measure the whole bill.
- Support effect: suppose contacts fall from 40 to 34 per 1,000 orders on 100,000 orders a month. That is 6 x 100 = 600 fewer contacts; at an assumed $4 per contact, $2,400 a month. This is a hypothesis to test, not a result.
Check it
- Pick the metric before shipping: contacts per 1,000 orders for tracking-related reasons only (tagging contacts by reason makes the signal sharper).
- Compare, do not just look before/after. Roll out to a random half of regions or users for two weeks and compare against the half still on polling. A plain before/after is confounded by seasonality and promotions: if the new design launches the week before a holiday sale and contacts rise from 40 to 48 per 1,000 orders, you cannot tell whether the design hurt or the sale did, because two causes changed together. The half still on polling, measured in the same weeks, shows the sale's effect on its own.
- Guardrails (metrics that must not get worse even if the main metric improves): error rate, page load time, reconnect rate, infrastructure cost per session.
- Decide in advance what lift is worth the maintenance cost; if the result is within noise (no bigger than the random week-to-week wobble in the numbers), you have learned the change is not worth defending on business grounds, only on engineering ones.
Reading the comparison (illustrative)
Suppose each half of the regions has 50,000 orders over the two weeks. The half still on polling has 2,000 tracking-related contacts (40 per 1,000 orders); the event-driven half has 1,700 (34 per 1,000). The gap is 300 contacts, 15% fewer. As a rough rule, a count like this wobbles by about its square root (about 45 for 2,000, about 41 for 1,700), so the gap between two such counts wobbles by about the square root of 2,000 + 1,700, which is 61. A gap of 300 is about five times that, so it is not noise. If the halves had instead shown 2,000 and 1,950, the gap of 50 is smaller than one wobble (about 63), and the honest reading is "no evidence of an effect".
Trade-offs and pitfalls
- Cost-saving claims are not outcome claims. Lower request count may not lower the bill if fixed capacity is already paid for (servers or a contract already bought, so fewer requests change nothing until you shrink what you pay for).
- Pick a metric the change can really move. A change that only affects 5% of users cannot move a company-wide figure; use the metric for the affected population.
- Be honest if the chain breaks. If customers contact support for reasons other than staleness, the support metric will not move, and the chain's link 4 was wrong.
