On-Call Practices and Runbook Design Questions

Direct answer

Action items get done when they're tracked in the same system and cadence as regular engineering work rather than in a postmortem doc nobody revisits, each has exactly one named owner and a due date, and a recurring, lightweight review surfaces overdue items instead of letting them go quiet. Lessons reach teams that didn't experience the incident when postmortems are indexed centrally by system and failure pattern, not just filed per-team, and a short summary is actively pushed to adjacent teams rather than waiting for someone to search the archive.

Structured elaboration

Getting action items done

Categorize each root cause as process, technical, or people and communication before assigning it. A technical fix routes to the engineering backlog with the same prioritization as other work; a process fix might mean updating a runbook or a review gate; making this distinction up front stops "add more monitoring" from becoming the default answer to everything. Each item gets one owner, one due date, and one acceptance criterion, since items without a single owner reliably don't get done. A weekly or sprint-cadence review of all open incident action items across teams escalates anything overdue to the owning manager, not just the assignee, and the remediation deadline itself scales with the severity of the incident that produced it.

Spreading lessons beyond the team that had the incident

A central, searchable postmortem index tagged by system, failure mode, and root-cause category, not filed only under the owning team's folder, makes cross-cutting patterns findable. A short, standardized summary, what broke, why, what changed, gets pushed to a cross-team channel or a recurring digest rather than relying on other teams to search for it. If the same root-cause category, such as "connection-pool exhaustion" or "no alert on saturation," shows up across multiple postmortems, that's a systemic gap worth its own initiative, and it only becomes visible if postmortems are tagged consistently enough to query across them.

Worked example

A quarter's incident action-item tracker opens 12 action items from postmortems and closes 9 of them within their assigned deadline by end of quarter, a $9/12 = 0.75$, or 75 percent, on-time closure rate. The weekly review flags the 3 still-open items each week; two turn out to be cross-team items stuck on a dependency from a third team that was never explicitly assigned ownership of its part. That gap itself becomes a new process action item: cross-team action items need an explicit dependency owner, not just an assignee, added to the next postmortem review template.

Trade-offs and pitfalls

• Tracking action items only in the postmortem document itself means they compete for attention against the team's regular backlog and usually lose; they need to live in the same system as everything else.
• Broadcasting every postmortem summary to the whole organization trains people to ignore the channel; scope the push to teams that own adjacent or dependent systems, and keep a searchable index for everyone else.
• A strict remediation deadline drives closure but can pressure a team into a shallow fix just to hit the date; pairing the deadline with an explicit acceptance criterion is what keeps "closed" meaning "verified" rather than just "closed."

No single number proves burnout. Track three families of signals together: raw load (pages per person per week), response burden (after-hours percentage, time-to-resolve), and human signals (fatigue self-reports, PTO usage), and watch for the same people repeatedly crossing thresholds across categories, not just one bad week.

What to track

Worked example: reading a four-week rotation block

Suppose the primary on-call received a combined 88 pages across the last 4-week rotation block (one week per engineer).

$$\text{Pages per on-call week} = \frac{88}{4} = 22 \text{ pages/week} \approx 3.1/\text{day}$$

Against the roughly-under-10/week guideline, 22 pages/week is more than double, a sustainability flag on its own. If 39 of those 88 pages fired between 20:00 and 08:00:

$$\text{After-hours share} = \frac{39}{88} \times 100 \approx 44.3\%$$

well above the roughly-25% guideline, corroborating that this isn't just a high-volume rotation, it's specifically disrupting sleep.

Trade-offs and pitfalls

These metrics can be gamed by suppressing alerts; pair volume metrics with an independent audit (a sampled review of closed incidents) so under-alerting doesn't masquerade as improvement. Self-reported fatigue data is noisy and subject to survey fatigue itself; treat it as a leading indicator alongside hard metrics, not as the sole trigger for action. A single bad week (one major outage) will spike every metric at once; look for a sustained pattern across at least a full rotation cycle before concluding the rotation itself, rather than the incident, is the problem.

Direct answer

Escalate immediately, before finishing your own triage, when the signal suggests the failure mode is adversarial or requires expertise and authority you don't have: active data movement out of the environment, evidence of privilege escalation or persistence, or anything (like a ransom note) that means you're dealing with an attacker's decisions, not a bug. The test is simple: could my next troubleshooting action destroy evidence or tip off an active adversary, and is there plausibly someone on the other side making decisions right now? If either is plausibly yes, escalate now and treat a false alarm as an acceptable cost.

Structured elaboration

Signals that trigger immediate escalation:
The two categories below, exfiltration and privilege escalation, are what an on-call responder most commonly actually sees; ransomware and unexplained vendor compromise are rarer but warrant the same immediate escalation when they happen.

• Data exfiltration (data leaving the environment to somewhere it shouldn't): unusual large outbound transfers to unfamiliar destinations, atypical bulk access to data stores outside normal patterns, DLP (data loss prevention, tooling that flags sensitive data leaving the network) or IDS (intrusion detection system, tooling that flags suspicious network or host activity) hits correlated with credential use outside its normal baseline.
• Privilege escalation (an account gaining access it shouldn't have) or persistence (an attacker planting a way to keep that access after you think you've cleaned up): unexpected new admin accounts, IAM (identity and access management, the system controlling who can access what) role or policy changes nobody on the team made, SSH keys added across multiple hosts, processes or webshells (a script an attacker leaves behind for remote access) that survive a reboot.
• Extortion or ransomware: encrypted files, a ransom note, or backup systems suddenly and specifically inaccessible.
• A vendor or dependency failure that doesn't match any known bug or outage pattern, especially if the vendor confirms unauthorized access on their side.

The self-handle-vs-escalate test, three fast questions to ask before touching anything further:

1. Is this reversible with a config change I already understand, or does it plausibly require forensics, legal, or vendor-security expertise I don't have?
2. Could my next action (reboot, delete, revoke, patch) destroy evidence that security or legal will need?
3. Is there a plausible person on the other side actively making decisions right now?

Any "yes" means escalate now, not after you've tried a few things yourself.

Worked example

An alert fires showing an outbound transfer of several gigabytes to an IP address with no history in that account's traffic baseline, immediately after a service account's credentials were used from an unfamiliar region. This hits two of the three escalate-now questions: it plausibly needs forensics expertise (a compromised credential's blast radius isn't something to guess at), and clumsy remediation could tip off an active attacker or destroy evidence. What not to do: reboot the host (destroys volatile memory that forensics needs) or immediately revoke every credential in sight without coordinating (can alert an attacker who's still active and cause them to accelerate or cover tracks). What to do instead: isolate the host at the network layer rather than powering it off, preserve logs and a memory snapshot, and page security immediately with the specific evidence (timestamps, transfer volume, destination IP, the credential and region involved) rather than a vague "something looks off." Security, not the on-call responder, then decides the sequencing of credential revocation, further isolation, and any legal or disclosure steps.

Trade-offs and pitfalls

• Over-escalating everything ambiguous as a security incident burns the security team's trust and slows down genuinely urgent calls later, the same alert-fatigue dynamic that applies to paging in general applies to security escalation specifically.
• Under-escalating to avoid looking alarmist risks destroying evidence or letting an active attacker continue while you troubleshoot as if it were an ordinary bug.
• The resolution isn't "use better judgment in the moment," because incident stress predictably degrades judgment on exactly these ambiguous calls; it's having a short, memorized decision list (the three questions above) that doesn't require clear thinking under pressure to apply correctly.
• Pitfall: treating "call security" as an admission that you failed. In a genuinely blameless culture, escalating on a plausible-but-unconfirmed signal is the correct, expected behavior, not something a responder should be second-guessed for after the fact if it turns out to be a false alarm.

Don't gate sensitive steps behind standing credentials a responder already holds. Gate them behind short-lived, narrowly scoped credentials issued by the runbook orchestrator at the moment of use, with a pre-authorized fast path for the highest severities so speed during a real incident doesn't require quietly bypassing the audit trail.

Comparing access models

Worked example: sizing the credential TTL

If the median observed time to complete a given remediation step across past incidents is 12 minutes, a 15-minute TTL leaves almost no margin:

$$\text{Margin} = 15 - 12 = 3 \text{ min}$$

a responder who hits a snag is interrupted 3 minutes short of done, exactly when stopping is most disruptive. A TTL of roughly 20 minutes, the median plus a working buffer rather than an open-ended grant, gives room to finish without leaving a long-lived credential outstanding. A 4-hour TTL "to be safe" instead means a credential compromised from a responder's terminal during that window stays valid for the rest of the shift; that's the trade being made for the extra convenience.

Keeping it auditable without slowing the responder down

• Runbooks reference secret IDs, never raw values, so the document itself is safe to read even if it leaks.
• The orchestrator executes the sensitive step server-side where practical, so the responder never sees the decrypted secret at all, only the outcome.
• Every credential issuance logs identity, ticket or incident ID, scope, and TTL to an immutable log, correlated automatically rather than reconstructed after the fact.
• The highest severities get pre-authorized issuance, no waiting on a human approver, precisely because the TTL and logging, not a manual gate, are what keep it auditable.

Trade-offs and pitfalls

A break-glass path needs more audit rigor than the normal path, not less; pair any emergency bypass with mandatory post-incident review and automatic rotation of whatever it touched. Auto-approval for the highest severities removes a human gate exactly during the highest-risk window (a real incident, adrenaline, and possibly an actor exploiting the chaos), so the TTL and logging have to carry that weight instead. Orchestrator-executed remediation is safer for the responder but adds its own risk surface; the automation itself now needs the same change-review rigor as production code, not less because "it's just a script."

The four golden signals for a web service are latency, traffic, errors, and saturation. Which of them should page a human comes down to one rule: page on symptoms that are actively hurting users right now, and let everything else, including most resource metrics, sit on a dashboard until it either crosses into user-visible territory or a slower trend alert catches it.

The four signals

• Latency: how long requests take, measured at p95/p99 (the 95th and 99th percentile: the response time that 95%, or 99%, of requests come in under) rather than the average, since the average hides the slow tail that users actually feel.
• Traffic: request volume and shape; a sudden drop is often as meaningful as a spike.
• Errors: the rate of failed requests, both hard failures (5xx) and soft ones (200s carrying wrong data).
• Saturation: how full a resource is, CPU, memory, connection pools, queue depth, which predicts trouble before it becomes user-visible.

What should page, versus what belongs on a dashboard

The dividing line is whether the signal is symptom-based (something a user is experiencing right now) or cause-based (something that predicts a symptom later). Symptom-based signals, elevated error rate, elevated p95 latency, should page immediately because every minute of delay is a minute of real user pain. Cause-based signals, CPU at 85%, a queue growing, belong on a dashboard and get an alert only if they're trending toward actually breaching a symptom threshold; otherwise the on-call person gets paged for problems that haven't happened yet and often self-resolve.

SLO burn-rate alerting, derived

A cleaner way to decide the paging threshold than picking a number by feel is to work from the error budget itself. Take a service with a 99.9% monthly availability SLO evaluated over a 30-day, 720-hour window.

$$\text{error budget} = 1 - 0.999 = 0.001 = 0.1\%$$ $$\text{budget in minutes} = 0.001 \times 30 \times 24 \times 60 = 43.2 \text{ minutes per month}$$

That's the total "allowed" downtime for the month. A burn rate of 1x means the service is consuming budget exactly on pace to use all 43.2 minutes by month end. To find the burn rate that should page immediately versus the one that should just open a ticket, fix a target: an alert should fire fast enough that a real outage doesn't quietly eat the whole month's budget before anyone notices.

For a 1-hour detection window, solve for the burn multiplier that consumes 2% of the monthly budget within that hour:

$$m \times \frac{1}{720} = 0.02 \implies m = 0.02 \times 720 = 14.4$$

For a 6-hour detection window, solve for the multiplier that consumes 5% of the budget:

$$m \times \frac{6}{720} = 0.05 \implies m = 0.05 \times \frac{720}{6} = 6$$

So: a 14.4x burn sustained for 1 hour (meaning the error rate is running at 14.4 times what the SLO allows) pages a human immediately, because at that rate the entire month's budget would be gone in about 50 hours. A 6x burn sustained for 6 hours only needs a ticket, since it's real but slow enough to catch on the next business day without waking anyone.

Putting it together for a web service

• Page immediately: 5xx rate or p95 latency crossing a threshold that maps to a fast SLO burn (roughly 14x or higher).
• Page on a slower cadence or ticket: a sustained but slow burn (single digits), or a saturation metric trending toward its symptom threshold.
• Dashboard only, no alert: traffic shape changes, CPU/memory levels that haven't threatened a symptom, anything already explained by a known deploy or maintenance window.

Trade-offs and pitfalls

Paging on every saturation metric produces alert fatigue fast, because resource usage fluctuates constantly without breaking anything; the fix is always tying a page to a symptom or a budget-burn calculation, not a raw resource number. The opposite mistake, paging only on hard 5xx errors, misses slow degradation (elevated latency, soft failures returning 200 with bad data) that erodes user trust just as much; that's exactly what the multi-window burn-rate approach is for, since it catches both fast, dramatic breaches and slow, sustained ones on two different clocks.