On-Call Practices and Runbook Design Questions

Readiness is a checklist, not a countdown: get access and tooling working first, have them study and sign off on the runbooks for their services, shadow several live pages, then run one supervised tabletop and one supervised live (or simulated) incident before they take a shift alone with a mentor reachable but not present.

A four-week ramp

Sign-off checklist before the first unsupervised shift

• Access confirmed end to end (paging tool, dashboards, deploy/rollback permissions) with a real test, not just "provisioned."
• Runbooks for their assigned services reviewed and any ambiguous steps flagged and fixed.
• At least one supervised tabletop and one supervised live or injected-fault incident completed.
• Mentor sign-off plus the engineer's own confidence self-assessment, not mentor judgment alone.

Extending the ramp for a complex or high-stakes service

Four weeks is often enough for a straightforward service, but for a complex hybrid-cloud system or one with many downstream dependents, add explicit competency checkpoints tied to named systems, for example certifying someone independently on the database failover path as a separate sign-off from general on-call readiness, rather than declaring them ready across the board at once. Longer term, treat the first 90 days as a structured mentorship arc rather than stopping at week four: scheduled 30/60/90-day check-ins, a second mentor pairing on a different service, and a distinct milestone for graduating from "supervised" to "primary" status rather than just a date on the calendar.

Trade-offs and pitfalls

Rushing readiness to fill a rotation gap is the most common failure mode, and it produces confident-sounding but wrong incident responses, which is worse than an obviously under-prepared response because it takes longer to catch. A checklist with no live-incident component only validates that someone can read, not that they can act under time pressure; keep at least one supervised live or simulated incident before signing off. Sign-off criteria should be service-specific rather than one generic "on-call ready" badge, since readiness on a well-instrumented service doesn't automatically transfer to a fragile legacy one with thin runbooks.

Direct answer

Severity is a fixed classification of an incident's technical and business impact right now (how bad is it), and each severity tier maps to a specific acknowledgment SLA, escalation path, and notification list so the response scales automatically with how bad things are. Severity is often confused with priority: severity measures blast radius and impact, while priority additionally weighs urgency and business context, and the two usually move together but can diverge.

Structured elaboration

Severity vs. priority. Severity is a property of the system: what fraction of functionality is broken and for whom. Priority is a property of the response: how urgently the organization needs to act on it right now, which factors in severity plus things like contract SLAs, timing, and who is affected. A Sev2 bug (partial degradation, workaround exists) affecting one enterprise customer with a contractual one-hour response commitment can get treated with P1 urgency even though its technical severity classification stays Sev2. Conversely, a technically Sev1-caliber bug discovered in a staging-only environment has low priority because there is no live customer impact yet. Conflating the two leads to two failure modes: under-resourcing a contractually urgent-but-technically-narrow issue, or paging the whole org for something with real severity but zero current business urgency.

Worked example

Two incidents happen the same week. Incident A: the primary API returns errors for 70% of requests across all customers. That's Sev1 by impact (majority of users, core path) and P1 by urgency (acknowledge in 5 minutes, exec on-call notified). Incident B: a non-critical reporting endpoint used by one enterprise customer returns stale data. By impact alone that's Sev3 (small subset, workaround exists: refresh manually). But that customer's contract has a 30-minute response SLA for any reported defect, so it gets routed with P1 priority: acknowledged within the contract window and staffed immediately, even though the severity label on the incident stays Sev3. The postmortem for B should note this divergence explicitly, since it's exactly the kind of nuance a severity-only view misses.

Trade-offs and pitfalls

• Pitfall: over-classifying everything as Sev1 "to be safe." This burns out on-call and trains people to treat pages as noise, defeating the purpose of having tiers at all.
• Pitfall: assigning severity once at triage and never revisiting it. Initial severity is frequently wrong (scope looks narrow until the second wave of impact shows up); the postmortem should include a severity-accuracy check as a standard field.
• Pitfall: letting priority silently override severity without documenting why, which erodes trust in the severity scale over time because people start reading "severity" as "whatever got the fastest response," rather than a consistent, calibratable measure of impact.

When an automated remediation makes things worse, the first move is to stop trusting the automation, not to debug it live: disable the trigger (feature flag or scheduler pause) so it can't fire again while you investigate, then treat the automation's own actions as the incident's primary evidence trail.

Investigation approach

• Pull the automation's own audit log first. What did it decide to do, on what input, at what timestamp? Most remediation frameworks log the triggering condition and the action taken; if this one doesn't, that's itself a finding.
• Reconstruct the precondition it evaluated against. Was the health check it used stale (cached metrics, delayed scrape) or narrower than reality (checked one replica's health, not cluster quorum)?
• Check for concurrency. Did two instances of the same remediation run at once, or did it run while a human was mid-deploy? Interleaved writes to the same resource are a common cause of "fix that broke things."
• Diff the assumed environment against the actual one. Runbooks and remediation scripts encode assumptions (resource names, API versions, cluster topology) that drift silently; check whether the automation was written against a topology that's since changed.

Framework for the fix

1. Add a pre-check gate: the remediation must verify the system is in the state it assumes (quorum present, no in-flight deploy, dependency healthy) before acting, and abort loudly if not.
2. Make the action idempotent and reversible: re-running it, or running it against a system already in the target state, should be a no-op, and every destructive step needs a paired rollback.
3. Bound the blast radius: act on one node/instance first (canary), verify success, then proceed, rather than acting cluster-wide in one shot.
4. Add a concurrency guard: a lock or lease so two triggers of the same remediation can't run simultaneously.
5. Gate high-impact actions behind a second signal: require the automation to see the problem confirmed by two independent signals (e.g., an alert plus a direct health check) before taking a destructive action, not just one noisy metric.

Worked example

Suppose the remediation is: "if a node reports high memory for 3 consecutive scrapes, cordon and drain it." The postmortem finds the metrics scraper had a 90-second collection lag during a load spike, so by the time the automation cordoned the third node it was actually reading data that was already 4.5 minutes stale (three 90-second-lagged scrapes), and it drained three nodes in the same 2-minute window because the memory spike was cluster-wide, not node-specific. Losing three nodes at once dropped the cluster below quorum for its replicated service, which is the partial outage.
The fix that follows directly from that trace: (a) the pre-check should compare current live memory, not the lagged scrape, before acting; (b) the automation should check how many nodes it has already drained in the current window and refuse to exceed a cap (e.g., no more than one node per 10 minutes) until a human confirms; (c) it should check that the remaining fleet still satisfies quorum before draining another node.

Trade-offs and pitfalls

Adding pre-checks and rate caps makes the remediation slower to react, which is the right trade for anything that can cause an outage of its own; reserve fully unthrottled auto-remediation for actions that are cheap to reverse (like restarting a single stateless pod) and keep caps and human gates on anything that removes capacity or touches shared state. A common wrong turn is to respond to this incident by simply disabling the automation permanently and reverting to manual remediation: that trades a rare automation bug for a much larger population of slower, inconsistent manual responses. The senior move is to narrow what the automation is trusted to do unsupervised, not to abandon automation.

Runbooks rot because nothing forces them to change when the system they describe changes. Keeping them accurate means giving every runbook a named owner, tying updates to the events that actually invalidate a runbook (a relevant code or infra change, or an incident where it was used), and having a lightweight, recurring check that catches staleness before an incident does, rather than relying on someone remembering to update it.

Ownership

Every runbook has one named primary owner, the team or person who owns the service it covers, recorded in the runbook itself, not a separate spreadsheet that gets forgotten. Ownership isn't honorary: the owner is the one who signs off that the runbook is still accurate at each review point, and the one paged if a stale runbook causes a bad outcome during an incident.

What triggers a review, not just a calendar date

Calendar-only review cadences ("review every quarter") catch some staleness but miss the more common case: a runbook goes stale the day the system changes underneath it, not on a schedule. Two triggers matter more than the calendar:

• Change-linked: any deploy that touches the commands, infra, or thresholds a runbook references should require a runbook update as part of that same change, enforced by a checklist item on the pull request, not a follow-up ticket that competes with the next sprint.
• Incident-linked: every time a runbook is actually used during an incident, the post-incident review includes a specific question, did the runbook match reality, and any gap becomes a tracked follow-up before the incident is closed.

Lifecycle stages

Catching a stale runbook before someone relies on it

A stale runbook is genuinely dangerous exactly because it looks trustworthy right up until the moment it's wrong. The cheapest catch is a light automated check, does every command in the runbook reference a tool, dashboard, or endpoint that still exists, run periodically and flagging anything broken for owner review. The more valuable catch is a periodic game-day: pick a runbook, have someone unfamiliar with the system try to follow it against a staging environment, and see where it breaks. Anything that trips someone up in a drill would have tripped up the on-call engineer at 3am.

Worked example: what enforcement actually looks like

A pull request changes the retry/backoff config for a service. The PR template includes a checkbox, "does this change any runbook referenced by this service's on-call docs?" Because the change alters a value a runbook's remediation step depends on, the author checks yes and links the runbook update in the same PR. Two months later, that runbook gets used during an incident; the post-incident review confirms the values matched, so the runbook stays Active with its last-reviewed date updated, no separate ticket needed because the change-linked trigger already did the work.

Trade-offs and pitfalls

Tying every infra change to a mandatory runbook update adds friction to routine PRs, so the check needs to be scoped narrowly (does this specific change affect a specific runbook) rather than a blanket "update all docs" gate that people learn to click through without reading. The most common failure mode is having a lifecycle model on paper but no one actually enforcing the Needs review to Active transition, so runbooks accumulate silently in Needs review and the label stops meaning anything; the fix is making that queue visible (a dashboard, not a buried label) and reviewing it in the same recurring meeting as on-call handoffs.

Direct answer

My rule is: an alert pages me only if it's actionable, urgent, and real, meaning there's something I can actually do about it, it needs a response within minutes rather than hours, and it reflects genuine or imminent customer/business impact. Anything that fails one of those three tests goes to Slack or a ticket, not my phone. I protect my own sustainability by treating a repeatedly noisy page as a bug in the alert that needs fixing, not as a toughness test I'm supposed to pass.

Structured elaboration

The paging bar, in practice. Before an alert is allowed to page a human overnight, it should answer yes to all three:

• Actionable: is there a specific thing a person can do right now, or does it just need to be visible on a dashboard?
• Urgent: does waiting until business hours meaningfully worsen the outcome?
• Real: does it correlate with actual customer or business impact, not just an internal metric that moves for benign reasons?

Anything that's informational, non-urgent, or has historically resolved itself before a human could act belongs in Slack or a ticket queue, not a page.

Boundaries I set for myself and expect from a team:

• Rotation limits: a cap on consecutive on-call weeks and mandatory rest between rotations, so on-call load is a scheduling property of the team, not a matter of individual endurance.
• Compensation for the bad weeks: comp time or a stipend that scales with how bad the week actually was, so a rotation with three overnight Sev1s isn't treated the same as a quiet one.
• A standing agreement that any alert paging the same person more than once or twice without a code change in response gets flagged as noise to fix, not accepted as the cost of doing business.
• Tracking my own load: if my pages-per-week start trending up, that's something I bring to the team, not something I quietly absorb, because sustainable on-call is a team-level property (rotation depth, alert hygiene) and not an individual willpower contest.

Worked example

In a recent on-call rotation, an alert on a background job queue started paging me nightly around 2am even though the queue reliably drained on its own within a few minutes every time. I didn't just mute it and move on; I posted the pattern in our team channel with the timestamps and outcomes for the last several nights (fires, self-resolves, no human action taken each time), and we agreed together that the alert needed a sustained-duration requirement before it could page overnight, since a brief backlog wasn't actually urgent or actionable at 2am. I made the config change, had a teammate review it since it affected everyone's rotation, and rolled it out. The nightly pages for that specific alert stopped after the change. The underlying job still occasionally backed up during traffic spikes, but that now showed up as a daytime ticket for the team to investigate instead of an overnight page for whoever happened to be on call.

Trade-offs and pitfalls

• Pitfall: "I'm always reachable, page me for anything" sounds committed but actually signals a lack of judgment about what deserves urgency, and it's a fast path to burnout that eventually degrades response quality for the alerts that really matter.
• Pitfall: the opposite extreme, "I basically never get paged after hours," can mean either genuinely excellent alert hygiene or quietly under-covered risk; the honest answer distinguishes which one it is with a concrete example, not just an assertion.
• Trade-off: a team that pays well for on-call (stipends, real comp time) can sustain a slightly higher page volume than one that doesn't, but that changes how much noise is tolerable, not whether the actionable/urgent/real bar applies. A well-compensated but chronically noisy alert is still a bug to fix, not a cost center to accept.