On-Call Practices and Runbook Design Questions

Direct answer

Base the rubric on customer-facing impact and business consequence, not internal technical severity: P0 is a full outage or a safety or data-loss risk with immediate paging and an aggressive SLA, scaling down through partial-impact and degraded-but-functional states to P3, a backlog item with no paging at all. Each level pairs a concrete definition with a response SLA, a resolution SLA, and exactly who gets paged, so classification is a lookup an on-call engineer can make under pressure, not a judgment call.

Structured elaboration

What determines the level

• Scope: how many customers or what fraction of traffic is affected.
• Reversibility of harm: data being lost or corrupted pushes toward P0 regardless of how few customers are affected, versus a fully recoverable error.
• Whether a workaround exists for the customer.
• Contractual exposure: whether this breaches an SLA the company is financially on the hook for.

Reclassifying as more information arrives

Declare an initial severity fast, from the first available signal, and treat it as provisional. Many incidents start looking like P1 and get upgraded to P0 once data loss is confirmed, or start as P0, a total outage, and downgrade to P1 once a workaround is found. Reclassification in either direction should be cheap and require no approval, because holding onto an inaccurate severity either under-pages a real emergency or burns unnecessary on-call attention.

Worked example

A payment-processing API returns errors for all merchants for several minutes before an automatic circuit breaker reroutes traffic to a backup provider, after which errors drop back to baseline. Applying the table: the initial signal, all merchants affected and revenue-blocking, classifies this as P0 and pages the on-call engineer, the service owner, and the incident commander immediately. Once the reroute confirms the impact is contained and no data was lost, the incident is reclassified to P1, SLA-covered functionality degraded with a workaround in place via the backup provider, for the remainder of the response. That reclassification changes the resolution SLA from continuous work under 4 hours to a target under 24 hours, but it doesn't stand down the already-paged responders mid-incident.

Trade-offs and pitfalls

• Defining severity by an internal technical signal, like an error-rate threshold, instead of customer impact, treats a high error rate on a low-traffic internal endpoint the same as the same error rate on checkout, which it isn't.
• Too many severity levels creates ambiguity at classification time under pressure; four tiers is usually enough resolution to route paging and SLAs correctly without forcing a judgment call between two levels that don't functionally differ.
• Making reclassification require a meeting or approval means on-call will just leave the severity wrong, which quietly corrupts incident metrics later.

A production incident moves through seven stages: detection, triage, investigation, escalation, remediation, recovery validation, and postmortem. Each stage has a specific job and produces a specific artifact, and skipping the artifact (not just the activity) is usually where lifecycle discipline actually breaks down, a team can do all the right actions and still lose the value if nothing gets written down for the next stage or the next incident to build on.

The lifecycle

flowchart LR
  A[Detection] --> B[Triage]
  B --> C[Investigation]
  C --> D[Escalation]
  D --> E[Remediation]
  E --> F[Recovery validation]
  F --> G[Postmortem]
  D -.->|if needed| C

Escalation isn't strictly linear, an investigation can loop back through another round after escalating in a specialist, which is why that arrow feeds back rather than only forward.

Stage by stage

Why the artifacts matter as much as the actions

An incident that's handled well but produces no artifacts is invisible to everyone who wasn't in the room: the next on-call engineer can't learn from it, the postmortem has no evidence trail to work from, and "what actually happened" becomes reconstructed memory instead of a timeline. The scribe role during a live incident exists specifically to make sure the investigation and remediation artifacts get captured in real time, because reconstructing a timeline after the fact from memory is unreliable and slow.

Blameless postmortem practice

The postmortem stage only works if it stays blameless in substance, not just in name: it asks what allowed the incident to happen (a missing safeguard, a gap in monitoring, an undocumented dependency) rather than who made the mistake. A postmortem that names an individual as the cause teaches everyone watching to hide problems rather than surface them, which is the opposite of what the stage exists to produce. The concrete output is a short list of action items, each with a named owner and a tracked ticket, not a narrative that ends with "we'll be more careful."

Trade-offs and pitfalls

Some teams try to skip investigation and jump straight to remediation under pressure, which sometimes works for well-understood failure modes but risks applying the wrong fix when the pattern only looks familiar. The more common failure across the whole lifecycle is letting the postmortem's action items go untracked once the adrenaline of the incident fades, an incident that produces a postmortem with no completed follow-through is functionally the same as one with no postmortem at all, just with more paperwork.

Direct answer

A runbook for an unresponsive primary with a healthy replica has to separate "the primary looks dead" from "the primary is dead." Check whether it truly refuses writes versus is just slow or lock-contended, confirm at least one replica is caught up enough to safely promote, and only fail over with an explicit operator confirmation, since promotion is usually irreversible without a full topology rebuild. If no replica is safely caught up, the decision becomes an RTO (recovery time objective: how long the service can stay down) versus data-loss trade-off that gets escalated, not made unilaterally by whoever is holding the pager.

Structured elaboration

What to check first, before touching anything

• Confirm the primary is actually unresponsive: connectivity, a real write test, and whether this looks like a network partition, a true database hang, or lock contention.
• If a deadlock is suspected: stop new writes at the application or proxy layer, identify the blocking transaction(s) (via the database's active-session view), and safely terminate the offending transaction(s) before even considering failover. A large share of "unresponsive primary" pages turn out to be a stuck writer, not a dead node, and killing the offending transaction is far cheaper than a failover.
• Check replica health: replication lag, whether the replica process is actually running, and whether the replica's own health checks pass.

Decision tree

flowchart TD
  A[Primary unresponsive alert fires] --> B{Primary accepting writes?}
  B -->|Yes, just slow| C[No failover: investigate latency/locks]
  B -->|No| D{Healthiest replica lag under 30s?}
  D -->|Yes| E[Get operator confirmation for destructive failover]
  E --> F[Promote healthiest replica]
  F --> G[Repoint app connection string and DNS]
  D -->|No, all replicas lagging| H[Escalate to DBA: weigh RTO vs data loss]
  H --> I{Accept data loss to restore now?}
  I -->|Yes| F
  I -->|No| J[Wait, restore primary from backup]

Decision points explained

• Fail over only once the primary is confirmed non-writable, not just slow, and a replica exists with lag under an agreed threshold. Failing over while the primary is merely slow risks split-brain: two nodes both accepting writes.
• Wait and investigate when the primary is reachable and still landing writes, even slowly.
• A destructive failover needs an explicit human confirmation step, not silent automation, precisely because it is hard to reverse.
• Coordinate the failover live with the owning application team and a DBA before promoting: they know write patterns (in-flight jobs, batch writers) that a generic runbook can't encode, and the DBA can judge whether the replica is truly safe to promote.

Rollback path if the failover goes wrong

• If the promoted replica can't sustain traffic, or the DNS/connection-string cutover doesn't propagate cleanly, first check whether the original primary has since recovered and is not diverged. If it has and is clean, route traffic back to it.
• If the original primary is diverged or unclear, treat this as a second incident and run it through the same decision tree again, treating the newly promoted node as the current primary.
• Fence the old primary (block it from accepting writes) after promotion so it can't silently rejoin as a second writer.

Worked example

Two replicas exist when the primary stops accepting writes: replica A reports 3 seconds of replication lag, replica B reports 45 seconds. The runbook's threshold is "promote only if lag is under 30 seconds." Replica A clears the threshold and replica B does not, so the on-call engineer gets operator confirmation and promotes replica A, accepting up to 3 seconds of potential write loss rather than 45. If both replicas had shown 45 seconds of lag, the runbook routes to the RTO-versus-data-loss escalation instead of an automatic promotion.

Trade-offs and pitfalls

• Fully automating the failover removes the human check that prevents split-brain during a network partition, where the primary might be up but simply unreachable from the monitoring node. That ambiguity is exactly why the confirmation step exists.
• Waiting longer to confirm the primary is truly dead reduces the risk of an unnecessary failover but extends downtime; the health-check timeout and lag threshold are the levers that tune this trade-off.
• Forgetting to fence the old primary after promotion is the most common way a "successful" failover turns into a second, worse incident.

Direct answer

Wire the alert directly to a runbook execution layer, not just a link: the alert carries a runbook identifier, the paging tool surfaces the matching runbook inline with a one-click execute action for pre-approved steps, and every click passes through validation, RBAC (role-based access control, the system that decides which actions a given user is permitted to run), and policy evaluation before anything runs, with the result and a full audit entry logged back to the incident. What actually stops an unsafe action isn't the button in the UI, it's the policy layer sitting between the click and the execution.

Structured elaboration

Component flow

sequenceDiagram
  participant M as Monitoring
  participant PD as PagerDuty
  participant R as Responder
  participant E as Runbook Executor
  participant V as Audit Log
  M->>PD: Alert fires with runbook_id attached
  PD->>R: Page plus incident link
  R->>E: Open runbook, click approved step
  E->>E: Validate params, RBAC, policy check
  E->>V: Write audit entry: who, what, when
  E->>PD: Post execution result to incident

What each layer does

• Alert enrichment: alerting rules carry a runbook identifier and a list of allowed actions as labels, so the alert itself knows which remediation it maps to.
• Paging tool integration: the incident view embeds the runbook with each step clearly labeled manual versus automated, along with its risk level.
• Execution broker: on click, it validates the caller's identity through RBAC, confirms the incident is actually open and acknowledged, and evaluates policy, such as whether this action needs extra approval given current severity or a maintenance window.
• Audit log: append-only, recording who clicked, what parameters were used, what the action did, and the result, written before the action is considered complete rather than best-effort afterward.

What logs, and what actually stops an unsafe action

Nothing about the UI stops an unsafe action; the broker re-validates everything server-side even if the button already looked available. Any action tagged destructive requires approval regardless of who clicks it, a circuit breaker disables an action automatically after repeated failures, and execution always runs with short-lived, scoped credentials so even a successful unsafe action is bounded in what it can touch.

Worked example

A "restart cache node" action is clicked from the incident view. The broker checks: is the incident open, yes; is the clicker on the paging on-call roster for this service, yes; is this action tagged low-risk, yes, a single-node restart; is it within the rate limit, zero executions in the last five minutes against a cap of one. All checks pass, the executor runs the restart with a scoped, time-limited credential, and writes the audit entry, who, incident ID, action, parameters, result, before returning success to the incident view.

Trade-offs and pitfalls

• Client-side checks are fine for UX, greying out an unavailable button, but the broker must re-validate everything server-side; the client should never be the actual enforcement point.
• Real-time policy evaluation on every click adds latency to remediation; keep the policy check itself a fast local lookup rather than a call to an external system on the critical path, so safety doesn't undercut automation's main benefit.
• Logging "success" based on the action returning without error, rather than verifying the target metric actually recovered, is a common gap; a runbook step can exit cleanly and still not have fixed anything.

An escalation system for a multi-team, multi-timezone org needs three separate mechanisms working together: routing (getting an alert to the right on-duty person without a human deciding that in the moment), noise suppression (so one root cause doesn't fan out into ten pages), and a severity model that decides who gets pulled in and how fast, because a revenue-impacting outage and a data-sensitive incident need different people in the room, not just different urgency.

Core building blocks

• Alert gateway: every alert is deduplicated by fingerprint, enriched with service/team/severity tags, and checked against maintenance windows before it's allowed to page anyone.
• Routing table: maps service ownership and team schedule (including timezone-local business-hours windows) to whoever is currently on duty, kept in the paging tool as the single source of truth rather than a wiki page someone forgets to update.
• Severity model: decides who gets paged first and how many people, based on impact type, not just raw error rate.
• Escalation ladder: a fixed sequence of who gets paged next if no one acknowledges, with a hard time budget at each step.

Severity and routing matrix

Escalation flow

flowchart TD
  A[Alert fires] --> B[Gateway: dedupe, enrich, tag severity]
  B --> C{Severity type}
  C -->|Revenue-impacting| D[Page Primary, ack SLA 5m]
  C -->|Data-sensitive| E["Page Data on-call AND Compliance lead in parallel; notify Legal/Compliance on fixed clock, independent of ack"]
  D --> F{Acked by 5m?}
  F -->|No| G[Escalate to Secondary, ack SLA +10m]
  G --> H{Acked by 15m?}
  H -->|No| I[Escalate to Service Lead, open incident bridge]
  F -->|Yes| J[Primary mitigates]
  H -->|Yes| J
  E --> K{Acked by 5m?}
  K -->|No| L[Escalate both chains in parallel, ack SLA +10m]
  K -->|Yes| M[Data on-call + Compliance mitigate]
  L --> M

Preventing alert-noise from cascading into unnecessary escalations

Most alert storms come from one root cause tripping many downstream checks at once (a database going down pages every service that depends on it). The gateway groups alerts by a correlation key (same root dependency, same time window) before routing, so the escalation ladder above runs once for the incident, not once per symptom. Escalation timers also only start on the first page for a correlated group; late-arriving duplicates reset nothing.

Keeping the matrix trustworthy

A routing matrix that's wrong is worse than no matrix, because it creates false confidence. Two things keep it honest: primary/backup contacts are pulled live from the scheduling tool rather than hand-maintained, and if the on-duty person marks themselves absent (leave, travel) in that same tool, pages route to the next person automatically instead of timing out silently first. A silent timeout during a real on-call absence is exactly the failure mode that erodes trust in the whole system.

Trade-offs and pitfalls

Stricter deduplication reduces noise but risks folding two genuinely unrelated incidents into one correlation group if the correlation key is too broad; the fix is scoping correlation to a real dependency graph, not just a time window. A common wrong turn is building one severity ladder for everything, which either pages security teams for routine downtime or under-escalates a compliance-relevant incident because it didn't look revenue-critical on the dashboard. The severity model has to be impact-type aware, not just impact-size aware.