On-Call Practices and Runbook Design Questions

Direct answer

Wire the alert directly to a runbook execution layer, not just a link: the alert carries a runbook identifier, the paging tool surfaces the matching runbook inline with a one-click execute action for pre-approved steps, and every click passes through validation, RBAC (role-based access control, the system that decides which actions a given user is permitted to run), and policy evaluation before anything runs, with the result and a full audit entry logged back to the incident. What actually stops an unsafe action isn't the button in the UI, it's the policy layer sitting between the click and the execution.

Structured elaboration

Component flow

sequenceDiagram
  participant M as Monitoring
  participant PD as PagerDuty
  participant R as Responder
  participant E as Runbook Executor
  participant V as Audit Log
  M->>PD: Alert fires with runbook_id attached
  PD->>R: Page plus incident link
  R->>E: Open runbook, click approved step
  E->>E: Validate params, RBAC, policy check
  E->>V: Write audit entry: who, what, when
  E->>PD: Post execution result to incident

What each layer does

• Alert enrichment: alerting rules carry a runbook identifier and a list of allowed actions as labels, so the alert itself knows which remediation it maps to.
• Paging tool integration: the incident view embeds the runbook with each step clearly labeled manual versus automated, along with its risk level.
• Execution broker: on click, it validates the caller's identity through RBAC, confirms the incident is actually open and acknowledged, and evaluates policy, such as whether this action needs extra approval given current severity or a maintenance window.
• Audit log: append-only, recording who clicked, what parameters were used, what the action did, and the result, written before the action is considered complete rather than best-effort afterward.

What logs, and what actually stops an unsafe action

Nothing about the UI stops an unsafe action; the broker re-validates everything server-side even if the button already looked available. Any action tagged destructive requires approval regardless of who clicks it, a circuit breaker disables an action automatically after repeated failures, and execution always runs with short-lived, scoped credentials so even a successful unsafe action is bounded in what it can touch.

Worked example

A "restart cache node" action is clicked from the incident view. The broker checks: is the incident open, yes; is the clicker on the paging on-call roster for this service, yes; is this action tagged low-risk, yes, a single-node restart; is it within the rate limit, zero executions in the last five minutes against a cap of one. All checks pass, the executor runs the restart with a scoped, time-limited credential, and writes the audit entry, who, incident ID, action, parameters, result, before returning success to the incident view.

Trade-offs and pitfalls

• Client-side checks are fine for UX, greying out an unavailable button, but the broker must re-validate everything server-side; the client should never be the actual enforcement point.
• Real-time policy evaluation on every click adds latency to remediation; keep the policy check itself a fast local lookup rather than a call to an external system on the critical path, so safety doesn't undercut automation's main benefit.
• Logging "success" based on the action returning without error, rather than verifying the target metric actually recovered, is a common gap; a runbook step can exit cleanly and still not have fixed anything.

Direct answer

Severity is a fixed classification of an incident's technical and business impact right now (how bad is it), and each severity tier maps to a specific acknowledgment SLA, escalation path, and notification list so the response scales automatically with how bad things are. Severity is often confused with priority: severity measures blast radius and impact, while priority additionally weighs urgency and business context, and the two usually move together but can diverge.

Structured elaboration

Severity vs. priority. Severity is a property of the system: what fraction of functionality is broken and for whom. Priority is a property of the response: how urgently the organization needs to act on it right now, which factors in severity plus things like contract SLAs, timing, and who is affected. A Sev2 bug (partial degradation, workaround exists) affecting one enterprise customer with a contractual one-hour response commitment can get treated with P1 urgency even though its technical severity classification stays Sev2. Conversely, a technically Sev1-caliber bug discovered in a staging-only environment has low priority because there is no live customer impact yet. Conflating the two leads to two failure modes: under-resourcing a contractually urgent-but-technically-narrow issue, or paging the whole org for something with real severity but zero current business urgency.

Worked example

Two incidents happen the same week. Incident A: the primary API returns errors for 70% of requests across all customers. That's Sev1 by impact (majority of users, core path) and P1 by urgency (acknowledge in 5 minutes, exec on-call notified). Incident B: a non-critical reporting endpoint used by one enterprise customer returns stale data. By impact alone that's Sev3 (small subset, workaround exists: refresh manually). But that customer's contract has a 30-minute response SLA for any reported defect, so it gets routed with P1 priority: acknowledged within the contract window and staffed immediately, even though the severity label on the incident stays Sev3. The postmortem for B should note this divergence explicitly, since it's exactly the kind of nuance a severity-only view misses.

Trade-offs and pitfalls

• Pitfall: over-classifying everything as Sev1 "to be safe." This burns out on-call and trains people to treat pages as noise, defeating the purpose of having tiers at all.
• Pitfall: assigning severity once at triage and never revisiting it. Initial severity is frequently wrong (scope looks narrow until the second wave of impact shows up); the postmortem should include a severity-accuracy check as a standard field.
• Pitfall: letting priority silently override severity without documenting why, which erodes trust in the severity scale over time because people start reading "severity" as "whatever got the fastest response," rather than a consistent, calibratable measure of impact.

Direct answer

A runbook is a fixed set of step-by-step instructions for a known failure mode: it tells you exactly what to type. A playbook is a decision framework for handling an incident more broadly: it tells you how to figure out what to do, who needs to be involved, and which runbook to reach for. Reach for a runbook when you already know the cause and the fix is mechanical; reach for a playbook when you're still diagnosing, coordinating multiple people, or the right response depends on judgment.

Structured elaboration

Minimum fields for each:

• Runbook: title, service, owner, trigger/precondition, required permissions and tools, exact step-by-step commands, verification steps, rollback steps, expected impact, last-reviewed date.
• Playbook: title, scope and severity thresholds, incident commander and stakeholder roles, the decision tree itself, links to the relevant runbooks, communication templates, escalation matrix, last-reviewed date.

Worked example

A database replica's lag exceeds a threshold: this is a runbook. It lists the exact commands to promote a replica, the steps to reconfigure the application's read preference, verification queries to confirm the fix, and the rollback commands if the promotion causes a new problem. Now compare that to a major outage affecting payments: this is a playbook. It guides the incident commander through detecting the actual scope, declaring severity, deciding between routing traffic to a fallback payment path versus draining traffic entirely, coordinating the app, infra, and comms teams, and linking out to the specific runbooks (including the replica-promotion one, if that turns out to be the fix) for whichever technical action the decision tree leads to.

Trade-offs and pitfalls

• Pitfall: writing a "runbook" for something that actually needs judgment, for example "when in doubt, restart the service." That hides a decision a playbook should make explicit, and someone follows it verbatim during exactly the incident it doesn't fit.
• Pitfall: letting a playbook go stale is worse than letting one runbook go stale, because the playbook is what everyone reaches for first during ambiguity; a stale escalation matrix (wrong names or numbers) breaks the whole response, not just one specific fix.
• Trade-off: automating a runbook into a one-click execution is great for high-confidence, low-blast-radius fixes (restarting a stateless service) and dangerous for high-blast-radius ones (promoting a database replica). The more damage a wrong click can do, the more the runbook should require an explicit human confirmation step before executing, not less.
• Both belong in a versioned, reviewed repository rather than an unowned wiki page, with periodic review, and runbooks specifically benefit from occasional dry-run or game-day testing to confirm the steps still work against the current system rather than an outdated one.

Direct answer

Action items get done when they're tracked in the same system and cadence as regular engineering work rather than in a postmortem doc nobody revisits, each has exactly one named owner and a due date, and a recurring, lightweight review surfaces overdue items instead of letting them go quiet. Lessons reach teams that didn't experience the incident when postmortems are indexed centrally by system and failure pattern, not just filed per-team, and a short summary is actively pushed to adjacent teams rather than waiting for someone to search the archive.

Structured elaboration

Getting action items done

Categorize each root cause as process, technical, or people and communication before assigning it. A technical fix routes to the engineering backlog with the same prioritization as other work; a process fix might mean updating a runbook or a review gate; making this distinction up front stops "add more monitoring" from becoming the default answer to everything. Each item gets one owner, one due date, and one acceptance criterion, since items without a single owner reliably don't get done. A weekly or sprint-cadence review of all open incident action items across teams escalates anything overdue to the owning manager, not just the assignee, and the remediation deadline itself scales with the severity of the incident that produced it.

Spreading lessons beyond the team that had the incident

A central, searchable postmortem index tagged by system, failure mode, and root-cause category, not filed only under the owning team's folder, makes cross-cutting patterns findable. A short, standardized summary, what broke, why, what changed, gets pushed to a cross-team channel or a recurring digest rather than relying on other teams to search for it. If the same root-cause category, such as "connection-pool exhaustion" or "no alert on saturation," shows up across multiple postmortems, that's a systemic gap worth its own initiative, and it only becomes visible if postmortems are tagged consistently enough to query across them.

Worked example

A quarter's incident action-item tracker opens 12 action items from postmortems and closes 9 of them within their assigned deadline by end of quarter, a $9/12 = 0.75$, or 75 percent, on-time closure rate. The weekly review flags the 3 still-open items each week; two turn out to be cross-team items stuck on a dependency from a third team that was never explicitly assigned ownership of its part. That gap itself becomes a new process action item: cross-team action items need an explicit dependency owner, not just an assignee, added to the next postmortem review template.

Trade-offs and pitfalls

• Tracking action items only in the postmortem document itself means they compete for attention against the team's regular backlog and usually lose; they need to live in the same system as everything else.
• Broadcasting every postmortem summary to the whole organization trains people to ignore the channel; scope the push to teams that own adjacent or dependent systems, and keep a searchable index for everyone else.
• A strict remediation deadline drives closure but can pressure a team into a shallow fix just to hit the date; pairing the deadline with an explicit acceptance criterion is what keeps "closed" meaning "verified" rather than just "closed."

Spend the first two weeks measuring where pages actually come from, not guessing, then rank the recurring drivers by pages eliminated per engineer-day of effort and fund the top of that list first. Feature work gets whatever capacity is left after the pager-volume target for the quarter is funded, not the other way around.

Step 1: baseline before you prioritize anything

Pull a two-week alert log and count pages by source, time of day, and whether each one required a real action or was noise. Prioritizing off memory or the loudest recent incident produces a list that optimizes for what people remember, not what's actually costing the most on-call time.

Step 2: score every recurring driver

Worked example: allocating a 20-day quarterly toil budget

Take the items in score order until the budget runs out: item 1 (1 day) + item 2 (3 days) + item 3 (4 days) + item 4 (5 days) = 13 of 20 days, leaving 7 days of buffer rather than starting the 15-day redesign this quarter.

Against a baseline of 220 pages/month, that's a reduction of

funded by 13 of 20 available toil-reduction days, with the remaining 7 days as margin for whatever the alert log surfaces next.

Trade-offs and pitfalls

Silencing an alert to hit the score is only a win if it was genuinely non-actionable; verify that before suppressing it, since a silenced alert that was catching a real problem just moves the cost from "pages" to "undetected incidents." The full redesign scores lowest on pages-per-day but may be the only fix that prevents an outage-class failure; the score is an input to prioritization, not the whole decision, and a high-blast-radius item can justify funding even at a low score. Communicate the capacity trade explicitly to product as a quarter-long commitment rather than letting it get silently reprioritized sprint by sprint. Re-score the remaining list after each phase; fixing the top four items usually promotes a previously mid-ranked item to the top as the dominant offender changes.