On-Call Practices and Runbook Design Questions

A runbook being written isn't the same as it being trustworthy under pressure: writing tests whether the author understood the system, while readiness tests whether someone else, half-awake at 3am, can follow it and get the right outcome. Check readiness by having someone who didn't write it actually execute it against a real (or realistic) system, not by reading it for completeness.

What to check before trusting a runbook

• Has anyone other than the author run it? A runbook the author has never handed to someone else is unverified by definition; the author's own mental model fills gaps a stranger will trip on (an assumed tool is installed, an assumed permission is already granted, a step that says "check the dashboard" without saying which one).
• Are the steps executable as written, not just described? "Restart the service" is a description; "run systemctl restart payments-api on each of the 3 hosts listed in the service registry" is executable. If a step requires judgment the runbook doesn't supply (how do you know which hosts?), that's a gap, not an acceptable level of abstraction.
• Does it state what success looks like? A remediation step without a stated verification step (what metric or log line confirms this worked) leaves the responder guessing whether to move to the next step or escalate.
• Is it safe to run when the diagnosis is wrong? Incident responders under pressure sometimes run the wrong runbook, or run the right one when the actual cause differs from what it assumes. Check whether each destructive step is reversible, and whether the runbook states a precondition to verify before acting ("only run this if X").
• Is it current? Check for an owner and a last-verified date; a runbook referencing a deprecated tool, an old cluster name, or a rotation that no longer exists is worse than no runbook, because it costs time before the responder realizes it's wrong.

How to actually verify these, not just check for their presence

1. Tabletop walkthrough: someone unfamiliar with the runbook reads it aloud, step by step, without help from the author, narrating what they'd actually type or click. Gaps surface immediately as "wait, what do I do here?" moments.
2. Staging or canary drill: run the actual remediation against a staging environment or a single canary instance, and time it. This catches steps that look right on paper but fail against the real system (a command with an outdated flag, a permission the on-call role doesn't actually have).
3. Cold-open test: hand it to someone with zero context on this specific service (not zero context on the systems generally) and see if they can act on it in under a target time, without pinging the original author. If they can't, the runbook is only usable by the person who wrote it, which defeats the point.

Worked example

A runbook for "database replica lag alert" says: "Check replica lag, if high, failover to standby." A cold-open drill immediately exposes three gaps: no link to where replica lag is displayed, no threshold for what counts as "high" (the alert already fired, so this should already be answered, but the runbook re-asks the question), and "failover to standby" doesn't say which standby if there are multiple, or what to verify afterward to confirm the failover succeeded rather than made things worse. Fixing it: link the specific dashboard panel, state the alert's own threshold so the runbook doesn't require re-deciding it, name the failover command with the specific standby-selection logic, and add a verification step ("confirm write latency on the new primary is under 50ms and replica lag on remaining replicas is decreasing"). Re-running the cold-open drill after the fix, the same tester completes it without asking a clarifying question, which is the actual pass condition.

Trade-offs and pitfalls

Running live drills has a real cost in engineering time and, for staging drills, some risk if the environment isn't well isolated from production; the return is worth it for any runbook covering a high-severity or destructive action, and can be scaled down to tabletop-only for low-risk, easily reversible ones. A common wrong turn is treating runbook review as a documentation-quality pass (is it well written, does it have headers) rather than an execution test; a beautifully formatted runbook that's never been run by anyone but its author is still unverified.

Direct answer

Run the postmortem in three phases: reconstruct a fact-based timeline from logs, metrics, and deploy history before the meeting even starts; use a structured technique during the meeting, like the "5 whys" or a fishbone breakdown, to separate genuine contributing factors from the actual root cause; and convert findings into specific, owned, time-boxed action items that get tracked to closure rather than filed and forgotten.

Structured elaboration

Data to gather before the meeting

• A precise timeline: when the alert fired, was acknowledged, mitigated, and resolved, with timestamps and who acted at each step.
• Dashboards showing the SLO breach before, during, and after.
• Logs and traces from the affected window, correlated across every service involved, not just the one that alerted first, since the alerting service is often a symptom rather than the cause.
• Deploy and config change history around the incident window, including commit references, feature flags, and infra changes.
• The runbook steps actually executed and their outcomes.

Separating contributing factors from root cause

A factor is a contributing factor if removing it alone would not have prevented the outage. It's the root cause only if removing it would have. Multiple genuine root causes are possible and should be named as such rather than forced into one.

Turning findings into action items that get done

Each item is specific, owned by exactly one person, carries a due date and an explicit acceptance criterion, such as "add an alert that fires when X, verified by injecting a synthetic breach and confirming it pages." Items are prioritized by risk reduced versus effort and tagged by urgency the same way incidents are, so remediation work competes visibly against other roadmap work. They're tracked in the same system as regular engineering work, with a scheduled follow-up check that closure actually happened and actually worked.

Worked example

• 14:02 UTC: a config deploy ships a change to the connection-pool size for checkout-service.
• 14:07 UTC: a checkout-service latency alert fires.
• 14:09 UTC: on-call acknowledges and begins triage on checkout-service logs, the service that alerted.
• 14:18 UTC: correlating logs across checkout-service and its downstream payments-service shows connection-pool exhaustion actually originated in payments-service; the earlier deploy had reduced its pool size, and checkout-service was just the first caller to see timeouts.
• 14:22 UTC: on-call rolls back the payments-service config; latency recovers.
• Root cause: the connection-pool config change on payments-service, since removing it would have prevented the outage. Contributing factor: no alert existed on payments-service's own pool saturation, only on downstream symptoms, which meant triage started on the wrong service before cross-service log correlation found the real source.
• Action items: add a connection-pool saturation alert directly on payments-service, owned by the payments team, verified by injecting a synthetic pool-exhaustion test and confirming it pages before the downstream symptom would; and require connection-pool config changes to go through the same review gate as code changes, owned by the platform team.

Trade-offs and pitfalls

• Stopping at the first plausible "why," usually the service that alerted, instead of correlating across the actual dependency chain is how a downstream root cause gets misattributed to whichever service merely surfaced the symptom first.
• Writing action items as vague intentions, like "improve monitoring," instead of specific, verifiable changes with an owner and an acceptance test, is the single biggest reason action items don't get done.
• A longer, more thorough timeline reconstruction produces a more accurate root cause but delays the meeting; worth it for a top-severity incident, disproportionate for a lower-severity one.

Direct answer

Severity is a fixed classification of an incident's technical and business impact right now (how bad is it), and each severity tier maps to a specific acknowledgment SLA, escalation path, and notification list so the response scales automatically with how bad things are. Severity is often confused with priority: severity measures blast radius and impact, while priority additionally weighs urgency and business context, and the two usually move together but can diverge.

Structured elaboration

Severity vs. priority. Severity is a property of the system: what fraction of functionality is broken and for whom. Priority is a property of the response: how urgently the organization needs to act on it right now, which factors in severity plus things like contract SLAs, timing, and who is affected. A Sev2 bug (partial degradation, workaround exists) affecting one enterprise customer with a contractual one-hour response commitment can get treated with P1 urgency even though its technical severity classification stays Sev2. Conversely, a technically Sev1-caliber bug discovered in a staging-only environment has low priority because there is no live customer impact yet. Conflating the two leads to two failure modes: under-resourcing a contractually urgent-but-technically-narrow issue, or paging the whole org for something with real severity but zero current business urgency.

Worked example

Two incidents happen the same week. Incident A: the primary API returns errors for 70% of requests across all customers. That's Sev1 by impact (majority of users, core path) and P1 by urgency (acknowledge in 5 minutes, exec on-call notified). Incident B: a non-critical reporting endpoint used by one enterprise customer returns stale data. By impact alone that's Sev3 (small subset, workaround exists: refresh manually). But that customer's contract has a 30-minute response SLA for any reported defect, so it gets routed with P1 priority: acknowledged within the contract window and staffed immediately, even though the severity label on the incident stays Sev3. The postmortem for B should note this divergence explicitly, since it's exactly the kind of nuance a severity-only view misses.

Trade-offs and pitfalls

• Pitfall: over-classifying everything as Sev1 "to be safe." This burns out on-call and trains people to treat pages as noise, defeating the purpose of having tiers at all.
• Pitfall: assigning severity once at triage and never revisiting it. Initial severity is frequently wrong (scope looks narrow until the second wave of impact shows up); the postmortem should include a severity-accuracy check as a standard field.
• Pitfall: letting priority silently override severity without documenting why, which erodes trust in the severity scale over time because people start reading "severity" as "whatever got the fastest response," rather than a consistent, calibratable measure of impact.

In the moment, redirect the conversation from the person to the timeline: acknowledge what was said without amplifying it, then immediately steer the group back to reconstructing what happened and why the system allowed it, while making clear that accountability for the fix is not going away.

In-the-moment response

• Interrupt with a redirect, not a confrontation. Something like: "Let's hold on names for a second and walk the timeline: what did the system show at each step?" This isn't ignoring what was said; it's refusing to let the postmortem's structure reward the blame framing by continuing down that thread.
• Reframe the specific claim into a system question. If the IC (Incident Commander, the person directing the response) says "this happened because Priya deployed without checking the dashboard," the redirect is: "So the deploy process didn't require a dashboard check before going out. Is that a gap in the checklist, or did the checklist exist and get skipped? Either answer tells us what to fix." This keeps the factual content (a deploy went out without a check) while stripping the blame framing.
• Do not let it pass silently either. Staying quiet when a peer is singled out in front of the team reads as agreement, and it's the fastest way to make the next engineer afraid to be transparent in their own postmortem. A short, calm correction in the room is better than a private word afterward, because the damage (and the culture signal) happened publicly.
• Follow up with the IC privately, separate from the room. The public redirect handles the moment; a private conversation afterward addresses the pattern, especially if this IC does it repeatedly.

Keeping accountability intact

Blameless does not mean no one owns the fix. The distinction to hold onto:

• Blame assigns fault for what already happened, to a person, and looks backward.
• Accountability assigns ownership for what happens next, to a role or system, and looks forward.
  So the postmortem should still end with a named owner for each remediation item (a person, because someone has to actually do the work) and a deadline, but the framing is "you're the best person to close this gap because you understand the deploy path," not "this is your fault so you have to fix it." The action items get assigned based on who has the context and capability, independent of who gets blamed.

Worked example

During a payments-outage postmortem, the IC says: "Marcus rolled back the config and that's what caused the second outage." The redirect: "Let's look at what the rollback runbook told him to check before rolling back. Did it call out this specific config's downstream dependency?" The team pulls up the runbook and finds it didn't mention that this particular config was read by two other services; the rollback step existed, but the pre-check for downstream impact didn't. The postmortem action items become: (1) add a downstream-dependency check to the rollback runbook for this config, owned by the platform team, due in two weeks, and (2) audit other high-fanout configs for the same missing check, owned by Marcus, since he now has the clearest picture of what that gap looks like, due in one month. Marcus ends up with an action item, but it's framed as "you're positioned to close this" rather than "you caused this," and the runbook gap, not Marcus's judgment, is recorded as the finding.

Trade-offs and pitfalls

The main pitfall is overcorrecting into vagueness, where "blameless" gets used to avoid naming any specific decision point, and the postmortem ends up too soft to actually change anything; the fix is to be precise about the decision and the missing guardrail while staying impersonal about who made the decision. A related pitfall specific to this scenario: correcting an incident commander in front of the team carries real interpersonal risk if done poorly, so the redirect has to stay factual and calm rather than accusatory itself. This is a leadership-culture issue that shows up at scale too: it typically takes deliberate, sustained work, roughly a couple of quarters of consistent leadership behavior, published blameless postmortems, and visible non-punitive handling of pages, to shift a team's on-call culture away from a punitive default, and it has to be reinforced the same way every time, including in the exact moment someone in authority breaks the pattern.

The Incident Commander (IC) owns the incident, not the fix. Their job is coordination and decision-making, deciding priorities, deciding when to escalate, keeping the response moving, while the technical work of diagnosing and fixing the problem belongs to subject matter experts (SMEs) the IC coordinates but doesn't have to be one of. That separation is the whole point: it lets someone stay focused on the shape of the response instead of getting pulled into a single technical rabbit hole.

Roles and how they differ

Why the IC role has to be distinct from the SME role

If the IC is also the person elbow-deep in a stack trace, two things suffer at once: the technical dive doesn't get their full attention, and no one is watching the overall picture (are we escalating too slowly, is communications falling behind, has severity changed). Separating the roles means the IC can pull in a second or third SME without needing to personally understand every system, and can make a call like "stop investigating, roll back now" even when an SME would rather keep digging for the root cause, because the IC's job is minimizing impact, not necessarily finding the deepest explanation in the moment.

Escalation triggers and handoffs

An IC should escalate (bring in a more senior IC, or a specific SME) when the current responder hits the edge of their authority or context: severity increasing beyond what the current team can safely own, the fix requiring a decision (like a risky rollback) above the current IC's authorization level, or the incident running long enough that fatigue is a real risk. Handoffs between ICs during a long-running incident follow the same discipline as an on-call shift handoff: the outgoing IC states current status, open decisions, and what's already been tried, in the incident channel, with the incoming IC explicitly confirming they've taken over before the outgoing IC steps back. An IC handoff that happens silently, with no explicit confirmation, is a common source of dropped context in long incidents.

Trade-offs and pitfalls

On a small team, it's tempting to skip a dedicated IC and let the most senior engineer both fix and coordinate; that works for short, simple incidents but breaks down exactly when it matters most, a complex, multi-team incident, because that's when coordination and deep technical focus can no longer be done well by the same person at once. The other common pitfall is an IC who defers every decision back to SMEs instead of actually deciding, which turns the incident into a discussion instead of a response; the IC's authority to make the call, even an imperfect one, quickly, is the actual value of the role.