On-Call Practices and Runbook Design Questions

Direct answer

Since you can't fix the vendor directly, you run three things in parallel: mitigate the blast radius with tools you do control (circuit breakers, cached or degraded responses, feature flags), communicate honestly about something outside your control, and push on the vendor relationship itself through support escalation and, if needed, contractual SLA terms. The trade-offs are mostly about how aggressively to degrade functionality versus how much broken or stale behavior your customers will tolerate in the meantime.

Structured elaboration

Evidence to gather before contacting vendor support. Precise timestamps with timezone noted, representative request/response examples (method, URL, headers, correlation IDs), correlated logs from your own edge/load-balancer and application layers, and a clear scope-and-impact statement (which services, what percentage of traffic, which customers, what SLA is at risk). Vague "your API seems down" tickets sit in a generic queue; a ticket with reproducible evidence and a quantified impact gets triaged faster.

Escalating through vendor support tiers. Open the highest applicable severity case with the evidence attached and explicitly request an engineer and a bridge, not just an acknowledgment. If there's no meaningful response within your own internal SLA for that severity, escalate through the account manager or a phone-based escalation path, citing the specific business impact and contractual SLA terms rather than repeating the original ticket.

Communication cadence, using the same severity-driven pattern as an internal incident: acknowledge to affected customers quickly with what's known and any workaround, then update on a fixed cadence (for example every 30 minutes) until resolved, closing with a summary once the vendor confirms the fix.

Worked example

A payments provider starts returning errors for a subset of transactions. Mitigation: flip a feature flag that routes non-critical calls to a queued-retry path with a "processing" state shown to the user, instead of failing checkout outright; this preserves the customer experience for the subset of traffic where a short delay is tolerable, while transactions that genuinely require a synchronous response fail fast with a clear error rather than hanging. Communication: post an initial status update within roughly 15 minutes acknowledging degraded checkout with the workaround in place, then update every 30 minutes. Vendor escalation: open a high-severity vendor ticket with timestamped request/response examples and the affected transaction volume, request a bridge; if no vendor engineer engages within your internal escalation window, escalate via the account manager's phone line, citing the contractual SLA and quantified customer impact.

Trade-offs and pitfalls

• Pitfall: treating a vendor outage as "not our incident" and skipping the postmortem. Root cause may be external, but your own blast-radius design (whether a circuit breaker or cached fallback existed at all) is exactly what a postmortem should examine, since that's the part you actually control.
• Pitfall: promising customers a fix ETA you don't control. Communicate "investigating, using workaround X, next update in 30 minutes" rather than a timeline that depends on someone else's incident response.
• Trade-off: aggressive circuit-breaking protects your own systems fastest but produces the most visible outage; cached/degraded responses are gentler on the user experience but carry a correctness risk if the vendor's data changes underneath the cache. Which one is right depends on how stale or wrong data is allowed to be for that specific feature, which is a product decision, not just an engineering one.

Direct answer

Severity is a fixed classification of an incident's technical and business impact right now (how bad is it), and each severity tier maps to a specific acknowledgment SLA, escalation path, and notification list so the response scales automatically with how bad things are. Severity is often confused with priority: severity measures blast radius and impact, while priority additionally weighs urgency and business context, and the two usually move together but can diverge.

Structured elaboration

Severity vs. priority. Severity is a property of the system: what fraction of functionality is broken and for whom. Priority is a property of the response: how urgently the organization needs to act on it right now, which factors in severity plus things like contract SLAs, timing, and who is affected. A Sev2 bug (partial degradation, workaround exists) affecting one enterprise customer with a contractual one-hour response commitment can get treated with P1 urgency even though its technical severity classification stays Sev2. Conversely, a technically Sev1-caliber bug discovered in a staging-only environment has low priority because there is no live customer impact yet. Conflating the two leads to two failure modes: under-resourcing a contractually urgent-but-technically-narrow issue, or paging the whole org for something with real severity but zero current business urgency.

Worked example

Two incidents happen the same week. Incident A: the primary API returns errors for 70% of requests across all customers. That's Sev1 by impact (majority of users, core path) and P1 by urgency (acknowledge in 5 minutes, exec on-call notified). Incident B: a non-critical reporting endpoint used by one enterprise customer returns stale data. By impact alone that's Sev3 (small subset, workaround exists: refresh manually). But that customer's contract has a 30-minute response SLA for any reported defect, so it gets routed with P1 priority: acknowledged within the contract window and staffed immediately, even though the severity label on the incident stays Sev3. The postmortem for B should note this divergence explicitly, since it's exactly the kind of nuance a severity-only view misses.

Trade-offs and pitfalls

• Pitfall: over-classifying everything as Sev1 "to be safe." This burns out on-call and trains people to treat pages as noise, defeating the purpose of having tiers at all.
• Pitfall: assigning severity once at triage and never revisiting it. Initial severity is frequently wrong (scope looks narrow until the second wave of impact shows up); the postmortem should include a severity-accuracy check as a standard field.
• Pitfall: letting priority silently override severity without documenting why, which erodes trust in the severity scale over time because people start reading "severity" as "whatever got the fastest response," rather than a consistent, calibratable measure of impact.

A credible automation case has three parts: the annual cost of doing the task manually (engineer time plus any revenue or SLA impact), the fully loaded cost of building and maintaining the automation, and a residual-risk line for when the automation itself misfires. Turn those into a payback period and an annual ROI, then restate the same numbers for a non-technical audience as "what we spend today versus a one-time investment that pays for itself in N months."

Inputs the model needs

The formula

$$\begin{aligned} \text{Toil}_{\text{annual}} &= F \times (H \times C + R) \\ \text{Automation}_{\text{annual}} &= \frac{D}{N} + M \\ \text{Residual}_{\text{annual}} &= F \times p \times C_f \\ \text{Net savings} &= \text{Toil}_{\text{annual}} - \text{Automation}_{\text{annual}} - \text{Residual}_{\text{annual}} \\ \text{Payback (months)} &= \frac{D}{\text{Net savings}/12} \end{aligned}$$

Worked example: a noisy alert that fires 300 times a year

Assumptions (pinned): F = 300/year, H = 0.4 hours, C = $75/hr, R = $100/incident, D = $20,000, N = 2 years, M = $4,000/year, p = 2%, C_f = $175/misfire.

$$\begin{aligned} \text{Toil}_{\text{annual}} &= 300 \times (0.4 \times 75 + 100) = 300 \times 130 = \$39{,}000 \\ \text{Automation}_{\text{annual}} &= \frac{20{,}000}{2} + 4{,}000 = \$14{,}000 \\ \text{Residual}_{\text{annual}} &= 300 \times 0.02 \times 175 = \$1{,}050 \\ \text{Net savings} &= 39{,}000 - 14{,}000 - 1{,}050 = \$23{,}950 \\ \text{Payback} &= \frac{20{,}000}{23{,}950/12} = \frac{20{,}000}{1{,}995.83} \approx 10.0 \text{ months} \\ \text{ROI} &= \frac{23{,}950}{14{,}000} \times 100 \approx 171\% \end{aligned}$$

Translating it for a non-technical stakeholder

The pitch a CFO or exec sponsor needs is the same numbers, stripped of the formula: "this task currently costs the team about $39k a year in engineer time and SLA impact. A $20k investment, amortized over two years plus $4k a year to keep it working, pays for itself in about ten months and then keeps saving roughly $24k every year after that, with a 2% chance any given automated run still needs a human to clean up after it." Leading with the payback date and the residual-risk number, not just the savings, is what makes the pitch credible instead of salesy.

Trade-offs and pitfalls

If F is seasonal or unstable, a single annual estimate is misleading; revisit the model quarterly rather than trusting a number computed once. Don't drop the residual-failure term to make the case look better: an automation that works 98% of the time but corrupts state on the other 2% can be a net negative if C_f is large, especially for destructive actions. A common wrong turn is pitching automation using only "engineer-hours saved" and skipping maintenance and residual-risk costs entirely; that consistently overstates ROI and makes the next proposal harder to trust once someone notices the gap.

MTTD is how long a problem existed before anything noticed it. MTTA is how long a human took to acknowledge the alert once it fired. MTTR is how long it took to fully resolve once someone was working it. The most common mistake is reporting a single blended average and treating it as typical, when one long outage in the set is doing all the work.

Definitions

$$\begin{aligned} \text{MTTD} &= t_{\text{detect}} - t_{\text{start}} \\ \text{MTTA} &= t_{\text{ack}} - t_{\text{detect}} \\ \text{MTTR} &= t_{\text{resolve}} - t_{\text{ack}} \end{aligned}$$

(Some teams instead measure MTTR from detection to resolve rather than ack to resolve; either is defensible, but the convention has to be fixed and stated, because mixing them across teams silently changes what the number means.)

Worked example: one incident timeline

$$\text{MTTD} = 6\text{ min}, \quad \text{MTTA} = 5\text{ min}, \quad \text{MTTR} = 36\text{ min}$$

Worked example: averaging across three incidents, and where it goes wrong

$$\overline{\text{MTTR}} = \frac{36 + 20 + 54}{3} = \frac{110}{3} \approx 36.7 \text{ min}$$

The mean of 36.7 minutes is being pulled up almost entirely by incident 3's 54-minute outlier: the mean sits above two of the three data points (20 and 36), with only the outlier itself larger. The median of {20, 36, 54} is 36, the middle value itself rather than a value inflated by the outlier, so it is a better single-number stand-in for the typical incident than the mean. Reporting mean MTTR alone, without the incident count or a percentile, makes a single bad incident look like the typical case.

Trade-offs and pitfalls

Comparing MTTR across teams that use different start-point conventions is comparing two different metrics wearing the same name; agree on the convention org-wide before benchmarking teams against each other. A dropping mean MTTR can hide a rising incident count: if you're resolving more small incidents faster while one rare severe incident still takes hours, the mean improves and the tail risk hasn't moved at all. Improving MTTD without improving MTTA or MTTR just means you find out about the same slow response faster; treat the three as stages of one pipeline, not independent wins to report separately.

Compensation and schedule design are two separate levers, and both need to move: pay a base standby stipend for availability, add per-incident pay or time-off-in-lieu for the work actually done, and size the rotation and rest guarantees so the schedule itself isn't relying on money to make an unsustainable load tolerable.

Compensation model components

Worked example: one on-call week

Stipend $250 + 6 hours of actual incident work at $40/hr:

$$\text{Pay} = 250 + (6 \times 40) = 250 + 240 = \$490$$

If the same 6 hours bank as TOIL at a 1.5x rate for after-hours work:

$$\text{TOIL banked} = 6 \times 1.5 = 9 \text{ hours}$$

Schedule practices that reduce the load pay has to compensate for

Primary/secondary tiers so one page doesn't always land on the same person; a cap on consecutive on-call weeks per engineer; shorter rotations (fewer consecutive days of stress, more handoffs) traded against longer rotations (fewer handoffs, more concentrated fatigue), sized to team headcount rather than picked arbitrarily; and follow-the-sun coverage once the team is large and distributed enough to make timezone handoffs cheaper than overnight pages.

Trade-offs and pitfalls

Per-incident pay can invite gaming in both directions, either padding logged hours or under-reporting to avoid looking like a "high maintenance" service; review incident-hour claims against the paging log rather than trusting self-reports alone. Contractors and salaried employees often need different structures (cash versus TOIL), and a single model rarely fits both cleanly. Regional labor law varies significantly, some jurisdictions treat standby time itself as compensable working time, so confirm with legal or HR before setting a global policy rather than assuming one region's rules generalize. There's no dominant answer on rotation length; it has to be sized to team size and incident frequency, not copied from another team's policy.