On-Call Practices and Runbook Design Questions

Sequence the work from cheap and immediate (alert routing and runbook hygiene) to structural (instrumentation, then automation) over roughly two quarters, and prove it's real improvement rather than gaming by tracking reopen rate and severity-classification consistency alongside the headline MTTR number, not just the number alone.

Where the 90 minutes actually goes

Before proposing fixes, break the baseline down into the phases every incident passes through:

The phased program

Worked example: tracking the breakdown phase by phase

$$\text{Reduction} = \frac{90 - 42}{90} \times 100 \approx 53.3\%$$

Guarding against gaming the metric

Pair MTTR with reopen rate and a severity-distribution audit, not just the headline number. If MTTR drops but the 24-hour reopen rate rises, for example from roughly 1 reopened incident a month to roughly 4 out of the same ~40 monthly incidents, that's a sign incidents are being closed to hit the target rather than actually fixed; treat a rising reopen rate as an automatic invalidator of the MTTR win until it's addressed. Separately, audit whether the mix of logged severities shifted (more incidents suddenly classified a notch lower than before); that alone can lower average MTTR without anything getting faster.

Trade-offs and pitfalls

Automation is the highest-leverage phase but also the highest-risk one; gate it behind canary rollout and audit logging so a bad automated fix doesn't become its own incident. A program driven purely by the MTTR number invites exactly the gaming described above; always report it alongside reopen rate and incident volume, never alone. Returns are not evenly spread across the phases, and they don't simply shrink over time: Phase 2 (instrumentation) delivers the largest single incremental win, 20 of the 90 minutes, bigger than either Phase 1's 11-minute alerting and runbook fix or Phase 3's 17-minute automation gain, because Phase 2 targets diagnosis, the single largest chunk of the baseline (35 of the 90 minutes). The lesson isn't that early phases always win biggest; it's that whichever phase targets the largest remaining bottleneck wins biggest, so sequence by where the time actually goes, not just by what's cheapest to ship first.

No single number proves burnout. Track three families of signals together: raw load (pages per person per week), response burden (after-hours percentage, time-to-resolve), and human signals (fatigue self-reports, PTO usage), and watch for the same people repeatedly crossing thresholds across categories, not just one bad week.

What to track

Worked example: reading a four-week rotation block

Suppose the primary on-call received a combined 88 pages across the last 4-week rotation block (one week per engineer).

$$\text{Pages per on-call week} = \frac{88}{4} = 22 \text{ pages/week} \approx 3.1/\text{day}$$

Against the roughly-under-10/week guideline, 22 pages/week is more than double, a sustainability flag on its own. If 39 of those 88 pages fired between 20:00 and 08:00:

$$\text{After-hours share} = \frac{39}{88} \times 100 \approx 44.3\%$$

well above the roughly-25% guideline, corroborating that this isn't just a high-volume rotation, it's specifically disrupting sleep.

Trade-offs and pitfalls

These metrics can be gamed by suppressing alerts; pair volume metrics with an independent audit (a sampled review of closed incidents) so under-alerting doesn't masquerade as improvement. Self-reported fatigue data is noisy and subject to survey fatigue itself; treat it as a leading indicator alongside hard metrics, not as the sole trigger for action. A single bad week (one major outage) will spike every metric at once; look for a sustained pattern across at least a full rotation cycle before concluding the rotation itself, rather than the incident, is the problem.

Allocate people by comparing the two incidents' business impact and required expertise, not by splitting the team evenly, and default to running them as separate incidents with separate commanders unless you find a shared root cause; escalate to a higher-level incident commander as soon as the resource conflict itself (not just the technical severity) becomes the bottleneck.

Allocating people across concurrent incidents

• Score each incident independently first: user-facing impact, revenue impact, data-integrity risk, and blast radius (one team's problem versus platform-wide). Two incidents rarely score identically, and the higher-scored one gets first claim on the strongest responders.
• Check for a shared root cause before splitting resources. If both services depend on the same failing component (a shared database, a shared auth service), that's actually one incident with two symptoms, and it should be run as a single incident with one commander, not two competing efforts pulling on the same underlying fix.
• Staff each independent incident with a minimum viable team: one incident commander, one primary responder, one communications owner. Resist over-staffing the incident that's louder or more visible if the other one is actually higher severity but quieter.
• Protect against double-booking the same expert. If one person is the only one who understands a shared piece of infrastructure, they can advise both incidents briefly but should not be the sole owner of fixing both; pull in a secondary responder even if slower.

When to escalate to a higher-level incident commander

Escalate when any of these is true, not just when severity is high:

• Resource contention itself is blocking progress: both incidents need the same scarce specialist or the same change-freeze exception, and someone above both incident commanders needs to arbitrate.
• Combined blast radius crosses an organizational boundary: the two incidents together affect enough of the business (multiple product lines, a shared customer segment) that unified external communication is needed, even if each incident alone wouldn't trigger that.
• One incident commander is starting to context-switch between both incidents. A single IC trying to run two incidents at once is a bigger risk than the incidents themselves; that's a signal to bring in a second commander or an overall coordinator, not to push through.
• Duration crosses a threshold where sustained dual-incident load starts to fatigue the responding team; a higher-level commander can pull in fresh responders or make the call to deprioritize the lower-severity incident explicitly (and communicate that decision) rather than let it silently starve.

Worked example

Two incidents fire eleven minutes apart: the checkout service is returning 500s for roughly a third of requests (revenue-impacting, high severity), and the internal analytics dashboard is showing stale data (no customer impact, low severity). The correct allocation: full incident-commander-plus-primary-plus-comms team goes to checkout immediately; analytics gets a single responder to investigate on a non-paging basis, because pulling more people onto analytics wouldn't shorten its resolution meaningfully and would strip capacity from checkout. If, twenty minutes in, the checkout investigation discovers the 500s trace back to the same message queue that feeds the analytics pipeline, the two incidents are merged under checkout's commander, because they share a root cause and running them separately would mean two people independently investigating the same queue.
Escalation in this example would trigger only if a third, unrelated incident arrived while checkout was still active and unresolved: three concurrent incidents makes single-IC-per-incident coordination itself the bottleneck, which is exactly the resource-contention trigger above.

Trade-offs and pitfalls

A common mistake is allocating headcount proportional to how loud or visible each incident is (how many people are asking about it in Slack) rather than its actual business impact; loud and low-impact will out-compete quiet and high-impact if you let it. Another is treating "escalate to a higher IC" as an admission of failure, so teams delay it past the point where a fresh coordinator would have resolved the resource conflict faster. The trade-off with merging incidents on a suspected shared root cause is real: merge too eagerly and you lose the separate investigation threads that might have found the divergence faster; the mitigation is to merge the coordination and communication, but keep separate technical workstreams until the shared cause is actually confirmed.

The four golden signals for a web service are latency, traffic, errors, and saturation. Which of them should page a human comes down to one rule: page on symptoms that are actively hurting users right now, and let everything else, including most resource metrics, sit on a dashboard until it either crosses into user-visible territory or a slower trend alert catches it.

The four signals

• Latency: how long requests take, measured at p95/p99 (the 95th and 99th percentile: the response time that 95%, or 99%, of requests come in under) rather than the average, since the average hides the slow tail that users actually feel.
• Traffic: request volume and shape; a sudden drop is often as meaningful as a spike.
• Errors: the rate of failed requests, both hard failures (5xx) and soft ones (200s carrying wrong data).
• Saturation: how full a resource is, CPU, memory, connection pools, queue depth, which predicts trouble before it becomes user-visible.

What should page, versus what belongs on a dashboard

The dividing line is whether the signal is symptom-based (something a user is experiencing right now) or cause-based (something that predicts a symptom later). Symptom-based signals, elevated error rate, elevated p95 latency, should page immediately because every minute of delay is a minute of real user pain. Cause-based signals, CPU at 85%, a queue growing, belong on a dashboard and get an alert only if they're trending toward actually breaching a symptom threshold; otherwise the on-call person gets paged for problems that haven't happened yet and often self-resolve.

SLO burn-rate alerting, derived

A cleaner way to decide the paging threshold than picking a number by feel is to work from the error budget itself. Take a service with a 99.9% monthly availability SLO evaluated over a 30-day, 720-hour window.

$$\text{error budget} = 1 - 0.999 = 0.001 = 0.1\%$$ $$\text{budget in minutes} = 0.001 \times 30 \times 24 \times 60 = 43.2 \text{ minutes per month}$$

That's the total "allowed" downtime for the month. A burn rate of 1x means the service is consuming budget exactly on pace to use all 43.2 minutes by month end. To find the burn rate that should page immediately versus the one that should just open a ticket, fix a target: an alert should fire fast enough that a real outage doesn't quietly eat the whole month's budget before anyone notices.

For a 1-hour detection window, solve for the burn multiplier that consumes 2% of the monthly budget within that hour:

$$m \times \frac{1}{720} = 0.02 \implies m = 0.02 \times 720 = 14.4$$

For a 6-hour detection window, solve for the multiplier that consumes 5% of the budget:

$$m \times \frac{6}{720} = 0.05 \implies m = 0.05 \times \frac{720}{6} = 6$$

So: a 14.4x burn sustained for 1 hour (meaning the error rate is running at 14.4 times what the SLO allows) pages a human immediately, because at that rate the entire month's budget would be gone in about 50 hours. A 6x burn sustained for 6 hours only needs a ticket, since it's real but slow enough to catch on the next business day without waking anyone.

Putting it together for a web service

• Page immediately: 5xx rate or p95 latency crossing a threshold that maps to a fast SLO burn (roughly 14x or higher).
• Page on a slower cadence or ticket: a sustained but slow burn (single digits), or a saturation metric trending toward its symptom threshold.
• Dashboard only, no alert: traffic shape changes, CPU/memory levels that haven't threatened a symptom, anything already explained by a known deploy or maintenance window.

Trade-offs and pitfalls

Paging on every saturation metric produces alert fatigue fast, because resource usage fluctuates constantly without breaking anything; the fix is always tying a page to a symptom or a budget-burn calculation, not a raw resource number. The opposite mistake, paging only on hard 5xx errors, misses slow degradation (elevated latency, soft failures returning 200 with bad data) that erodes user trust just as much; that's exactly what the multi-window burn-rate approach is for, since it catches both fast, dramatic breaches and slow, sustained ones on two different clocks.

Direct answer

My rule is: an alert pages me only if it's actionable, urgent, and real, meaning there's something I can actually do about it, it needs a response within minutes rather than hours, and it reflects genuine or imminent customer/business impact. Anything that fails one of those three tests goes to Slack or a ticket, not my phone. I protect my own sustainability by treating a repeatedly noisy page as a bug in the alert that needs fixing, not as a toughness test I'm supposed to pass.

Structured elaboration

The paging bar, in practice. Before an alert is allowed to page a human overnight, it should answer yes to all three:

• Actionable: is there a specific thing a person can do right now, or does it just need to be visible on a dashboard?
• Urgent: does waiting until business hours meaningfully worsen the outcome?
• Real: does it correlate with actual customer or business impact, not just an internal metric that moves for benign reasons?

Anything that's informational, non-urgent, or has historically resolved itself before a human could act belongs in Slack or a ticket queue, not a page.

Boundaries I set for myself and expect from a team:

• Rotation limits: a cap on consecutive on-call weeks and mandatory rest between rotations, so on-call load is a scheduling property of the team, not a matter of individual endurance.
• Compensation for the bad weeks: comp time or a stipend that scales with how bad the week actually was, so a rotation with three overnight Sev1s isn't treated the same as a quiet one.
• A standing agreement that any alert paging the same person more than once or twice without a code change in response gets flagged as noise to fix, not accepted as the cost of doing business.
• Tracking my own load: if my pages-per-week start trending up, that's something I bring to the team, not something I quietly absorb, because sustainable on-call is a team-level property (rotation depth, alert hygiene) and not an individual willpower contest.

Worked example

In a recent on-call rotation, an alert on a background job queue started paging me nightly around 2am even though the queue reliably drained on its own within a few minutes every time. I didn't just mute it and move on; I posted the pattern in our team channel with the timestamps and outcomes for the last several nights (fires, self-resolves, no human action taken each time), and we agreed together that the alert needed a sustained-duration requirement before it could page overnight, since a brief backlog wasn't actually urgent or actionable at 2am. I made the config change, had a teammate review it since it affected everyone's rotation, and rolled it out. The nightly pages for that specific alert stopped after the change. The underlying job still occasionally backed up during traffic spikes, but that now showed up as a daytime ticket for the team to investigate instead of an overnight page for whoever happened to be on call.

Trade-offs and pitfalls

• Pitfall: "I'm always reachable, page me for anything" sounds committed but actually signals a lack of judgment about what deserves urgency, and it's a fast path to burnout that eventually degrades response quality for the alerts that really matter.
• Pitfall: the opposite extreme, "I basically never get paged after hours," can mean either genuinely excellent alert hygiene or quietly under-covered risk; the honest answer distinguishes which one it is with a concrete example, not just an assertion.
• Trade-off: a team that pays well for on-call (stipends, real comp time) can sustain a slightly higher page volume than one that doesn't, but that changes how much noise is tolerable, not whether the actionable/urgent/real bar applies. A well-compensated but chronically noisy alert is still a bug to fix, not a cost center to accept.